# Short Diagnosis

Your model is exhausting its generation budget on internal reasoning loops without encountering a termination signal. In autoregressive inference, the model predicts one token at a time. When prompted with open-ended or reasoning-heavy queries, instruction-tuned models are heavily biased toward extended deliberation, especially if they were fine-tuned on chain-of-thought (CoT) datasets. Without explicit `stop` sequences, structural delimiters, or a constrained `max_tokens` budget, the model will continue generating plausible reasoning steps until it hits a hard limit or runs out of context. The proxy layer (nginx, caddy, traefik, or a custom reverse proxy) often compounds this by buffering WebSocket streams, delaying final chunks, or misreporting stream completion, making the UI appear frozen while the backend is still producing tokens. The result is a generation that consumes its entire token allocation on "thinking" and never transitions to a final answer.

# Why This Happens

Several interacting factors drive this behavior:

1. **Autoregressive Generation Mechanics**: LLMs do not "decide" when to stop. They sample the next token based on probability distributions. If the prompt lacks a clear boundary, the model continues generating until it encounters a rare end-of-sequence token or hits a hard cap.
2. **CoT Training Bias**: Many modern instruction-tuned models are trained on datasets that reward step-by-step reasoning. The model learns that verbose deliberation correlates with higher reward, so it defaults to extended internal monologue unless explicitly constrained.
3. **Missing or Mismatched Stop Sequences**: `stop` sequences in llama.cpp are exact string matches against the generated token stream. If your prompt uses markdown, XML tags, or custom delimiters, but your `stop` parameter doesn't match the exact tokenization boundary, the model never recognizes the termination condition.
4. **Token Budget Misalignment**: `max_tokens` (or `num_predict` in older llama.cpp versions) defines the absolute ceiling. If set too high (e.g., 4096) without a stop sequence, the model has room to wander. Conversely, if set too low, it may cut off mid-reasoning.
5. **Proxy/Streaming Artifacts**: OpenWebUI relies on Server-Sent Events (SSE) or WebSocket streaming. Some proxies buffer large payloads, delay chunk delivery, or strip trailing whitespace/newlines that serve as stop markers. This creates a disconnect between backend generation and frontend display.
6. **KV Cache Pressure**: When `num_ctx` is significantly larger than the actual context used, attention computation becomes inefficient. The model may generate slower, increasing the likelihood of timeout or perceived freezing, especially on consumer hardware.
7. **Temperature/Top-p Interaction**: High temperature (>0.8) or loose top_p (>0.95) increases sampling entropy, which can push the model into repetitive or non-terminating reasoning loops. Lower values keep generation focused but may reduce creativity.

# Settings To Check

Focus on these parameters in OpenWebUI's model configuration or llama.cpp CLI flags. Adjust incrementally and test systematically.

- **`max_tokens` / `num_predict`**: Hard generation limit. Start with 512 for direct answers, 1024–2048 for reasoning tasks. Never leave this unbounded.
- **`stop`**: Critical termination control. Add multiple sequences separated by commas. Examples: `\n\n### Answer:\n`, `\n\n---\n`, `</answer>`, `\n\nFINAL ANSWER:\n`. Ensure they match your prompt structure exactly.
- **`temperature`**: Controls sampling randomness. 0.3–0.6 for factual/concise outputs, 0.7–0.9 for creative tasks. Lower values reduce rambling.
- **`top_p`**: Nucleus sampling threshold. 0.9–0.92 keeps generation focused. Higher values increase diversity but risk loops.
- **`repeat_penalty`**: Discourages repetition. 1.1–1.15 is effective. Increase to 1.2 if you see looping phrases.
- **`mirostat` / `mirostat_tau`**: Adaptive perplexity control. Can stabilize generation but may interfere with stop sequence detection. Disable if termination is unreliable.
- **`num_ctx`**: Context window size. Match this to your actual context needs. Over-provisioning increases latency and token waste. For 7B–13B models, 4096–8192 is usually sufficient.
- **`streaming`**: Must be enabled in OpenWebUI. Ensures tokens are sent incrementally rather than buffered.
- **Proxy Configuration**: 
  - Enable WebSocket/SSE passthrough.
  - Set `proxy_buffer_size` and `proxy_read_timeout` appropriately (e.g., 64KB buffer, 60s timeout).
  - Disable compression if it interferes with chunk boundaries.
  - Verify that trailing newlines or delimiters aren't stripped during proxy translation.

# Prompt Design

Structure prompts to enforce boundaries, separate reasoning from output, and provide explicit termination markers. Use delimiters, format constraints, and role separation.

**Bad Prompt Shape:**
```
Explain quantum entanglement and how it relates to quantum computing. Think carefully and consider all angles.
```
*Why it fails:* No structure, no length constraints, no stop signal. Encourages open-ended reasoning. The model may generate pages of derivation, historical context, and speculative extensions without ever transitioning to a direct answer.

**Good Prompt Shape (Structured Output):**
```
You are a technical explainer. Answer the following question concisely.
Question: Explain quantum entanglement and how it relates to quantum computing.
Format your response exactly as follows:
### Reasoning
[Step-by-step logic, max 3 sentences]
### Answer
[Direct, self-contained explanation, max 150 words]
Stop after the Answer section. Do not add anything else.
```
*Why it works:* Explicit sections, length constraints, clear termination marker. The model recognizes the structure and halts after the designated block. The `stop` sequence in settings should match `\n### Answer:\n` or similar.

**Advanced Pattern (CoT with Forced Termination):**
```
Think step-by-step, but keep your reasoning under 100 tokens. Then output exactly:
FINAL ANSWER: [your response]
Do not add anything after FINAL ANSWER.
```
*Why it works:* Leverages the model's training on structured outputs while capping reasoning length. The explicit `FINAL ANSWER:` marker serves as a reliable stop sequence.

**Prompt Engineering Best Practices:**
- Use XML-style tags if your model supports them: `<reasoning>...</reasoning><answer>...</answer>`
- Add negative constraints: "Do not include historical background." "Do not list alternatives."
- Specify output format: JSON, bullet points, or single paragraph.
- Include token-aware instructions: "Keep your response under 200 tokens."
- Test delimiter compatibility: Some tokenizers split `\n\n` differently. Use `\n\n` or `\n` consistently.

# Benchmark Design

To systematically evaluate and tune your setup, design a controlled benchmark that isolates variables and measures termination reliability.

**Test Suite Construction:**
- 15–20 diverse prompts covering: factual recall, multi-step reasoning, code generation, creative writing, and long-context summarization.
- Include edge cases: ambiguous queries, contradictory instructions, and highly technical domains.
- Hash each prompt to ensure reproducibility.

**Metrics to Track:**
- **Time to First Token (TTFT)**: Measures backend readiness and proxy latency.
- **Total Tokens Generated**: Indicates budget consumption.
- **Stop Sequence Hit Rate**: Percentage of runs that terminate cleanly vs. exhaust `max_tokens`.
- **Answer Completeness**: Manual or LLM-as-judge evaluation (1–5 scale).
- **Repetition/Loop Detection**: Count repeated n-grams in the last 50 tokens.
- **Context Window Pressure**: KV cache utilization and attention computation time.

**Procedure:**
1. Fix all parameters except `max_tokens`, `stop`, and prompt structure.
2. Run each prompt 3 times to account for sampling variance.
3. Record metrics in a structured format (CSV/JSON).
4. Identify failure modes: e.g., 80% hit stop, 20% exhaust tokens, 10% loop.
5. Iterate on `stop` sequences and prompt templates. Retest until termination rate >95%.

**Automation Script Outline:**
```python
import requests
import json
import time

def run_benchmark(prompts, config):
    results = []
    for prompt in prompts:
        start = time.time()
        response = requests.post("http://localhost:8080/v1/chat/completions", json={
            "messages": [{"role": "user", "content": prompt}],
            **config
        })
        data = response.json()
        tokens = data["usage"]["total_tokens"]
        ttft = time.time() - start
        results.append({
            "prompt_hash": hash(prompt),
            "ttft": ttft,
            "tokens": tokens,
            "stop_hit": "stop" in data.get("stop_reason", ""),
            "answer": data["choices"][0]["message"]["content"]
        })
    return results
```
Run this script across parameter variations to build a performance matrix. Use statistical analysis to identify optimal configurations.

# What The Flight Recorder Should Show

Monitor these indicators in OpenWebUI logs, llama.cpp console output, or proxy metrics to diagnose generation behavior.

- **Token Usage Curve**: Should plateau near the answer boundary. A flat line at `max_tokens` indicates exhaustion. A gradual decline suggests healthy termination.
- **Stop Sequence Detection**: Log should show `stop` triggered vs `max_tokens` hit. If `stop` is never triggered, your delimiter isn't matching tokenization boundaries.
- **Generation Speed**: Tokens/sec should be stable. Sudden drops indicate context window thrashing or KV cache pressure. Consumer GPUs may drop from 30 t/s to 5 t/s when context exceeds 80% capacity.
- **Repetition Patterns**: High `repeat_penalty` should suppress loops. If you see repeated phrases, adjust penalty or temperature. Look for n-gram repetition in the last 50 tokens.
- **Proxy Stream Events**: WebSocket frames should arrive continuously. Gaps >2s suggest buffering or network issues. Check proxy logs for `upstream timeout` or `buffer overflow`.
- **Context Window Pressure**: If `num_ctx` is near capacity, attention computation slows, increasing TTFT and token waste. Monitor GPU memory usage and KV cache utilization.
- **Error Logs**: Look for `context full`, `stop sequence not found`, `generation timeout`, or `stream interrupted`. These indicate configuration mismatches or hardware limits.
- **Tokenization Artifacts**: Some models split delimiters differently. Check if `\n\n` becomes `\n \n` or similar. Adjust `stop` sequences to match actual token boundaries.

# Practical Defaults

Start with this baseline for most home-lab use cases. Adjust based on model family and task complexity.

- `max_tokens`: 1024
- `stop`: `\n\n### Answer:\n`, `\n\n---\n`, `</answer>`
- `temperature`: 0.6
- `top_p`: 0.92
- `repeat_penalty`: 1.12
- `num_ctx`: 4096 (adjust to model's native context)
- `streaming`: true
- `mirostat`: off
- `num_gpu_layers`: -1 (offload all layers if VRAM permits)
- `flash_attention`: true (if supported by your llama.cpp build)
- Proxy: Enable WebSocket streaming, set timeout to 60s, buffer size 64KB, disable compression

**Rationale:**
- `max_tokens` at 1024 balances reasoning depth with termination reliability.
- Multiple `stop` sequences increase the probability of clean termination.
- `temperature` 0.6 reduces rambling while preserving creativity.
- `top_p` 0.92 keeps sampling focused without excessive determinism.
- `repeat_penalty` 1.12 suppresses loops without killing coherence.
- `num_ctx` 4096 matches most 7B–13B model contexts, minimizing KV cache pressure.
- `flash_attention` reduces memory bandwidth bottlenecks on modern GPUs.
- Proxy settings ensure uninterrupted streaming and prevent chunk buffering.

**Fallback Strategy:**
If termination is still unreliable, reduce `max_tokens` to 512, increase `repeat_penalty` to 1.15, and add stricter `stop` sequences. If answers are too brief, increase `max_tokens` to 1536 and adjust `temperature` to 0.7.

# When To Increase Budgets

Raise `max_tokens` and adjust `stop` sequences when:

- **Multi-step reasoning**: Math, code generation, or complex analysis requires extended deliberation. Increase to 2048–4096.
- **Long context tasks**: Summarizing documents, RAG with large chunks, or translation. Match `num_ctx` to input length.
- **Creative/verbose outputs**: Essays, reports, or detailed technical documentation. Increase to 2048–3072.
- **Model-specific behavior**: Some models (e.g., Mistral, Llama-3 variants, Qwen) naturally generate longer outputs. Test empirically.
- **Complex prompt structures**: Multi-part questions or nested instructions require more tokens to parse and respond.

**Scaling Guidelines:**
- Increase in increments of 512 tokens.
- Pair with stronger `stop` sequences to prevent unbounded generation.
- Monitor token efficiency: if >60% of budget is consumed on reasoning, consider prompt compression or explicit CoT limits.
- Adjust `temperature` downward when increasing budgets to maintain focus.
- Verify proxy timeout and buffer settings scale with longer generations.

**When Budget Increases Backfire:**
- Excessive `max_tokens` without `stop` sequences leads to token waste and latency.
- High `num_ctx` without matching context usage increases KV cache pressure and slows generation.
- Overly long outputs may degrade answer quality due to attention dilution.

# When To Stop A Run

Implement hard abort conditions to prevent resource waste and maintain predictable behavior.

- **Token Budget Exhaustion**: If `max_tokens` is hit without a stop sequence, abort and log. This indicates missing termination signals or prompt misalignment.
- **Time Limit**: >30s for simple queries, >60s for complex tasks. Use proxy timeout or client-side abort. Long TTFT or generation time suggests context pressure or hardware bottlenecks.
- **Repetition Loop**: Detect if the last 50 tokens repeat a phrase >3 times. Trigger early stop. Adjust `repeat_penalty` or `temperature` for future runs.
- **Quality Degradation**: If output becomes nonsensical, repetitive, or off-topic, abort and retry with lower temperature or stricter prompt structure.
- **User Intervention**: Allow manual stop in OpenWebUI. Respect user intent and avoid forcing generation beyond user tolerance.
- **Context Overflow**: If KV cache pressure causes >50% slowdown, abort and reduce `num_ctx`. Monitor GPU memory usage and adjust accordingly.
- **Proxy Stream Interruption**: If WebSocket frames stop arriving for >5s, abort and check proxy configuration. Network instability or buffer limits may be interfering.

**Implementation Patterns:**
- Use client-side timeouts in OpenWebUI or custom UI wrappers.
- Implement server-side abort hooks in llama.cpp via `--abort` or custom scripts.
- Log abort reasons for post-hoc analysis and configuration tuning.
- Combine multiple heuristics: e.g., abort if `max_tokens` hit AND no stop sequence AND repetition detected.

These heuristics ensure predictable behavior, protect hardware resources, and maintain a positive user experience. Regularly review abort logs to refine prompt templates, stop sequences, and parameter defaults.