`). If these aren’t configured or are too broad, the model ignores them.
- **High Temperature / Top-p Settings:** Aggressive sampling increases randomness, making the model less likely to converge on a definitive endpoint.
- **Context Window Exhaustion:** If the model is allowed to generate up to the full context limit (e.g., 32k tokens), it will keep going until memory fills, often producing repetitive or degenerate text.
- **Proxy/Server Buffering:** Some reverse proxies or API wrappers don’t forward stop conditions correctly, or they buffer responses until completion, masking the fact that generation never actually finished.
- **Model Architecture Bias:** Models trained with Chain-of-Thought (CoT) or “thinking” prefixes often learn to output reasoning before answers. Without a strict delimiter, they treat the entire output as one continuous stream.
**# Settings To Check**
Focus on three layers: the inference engine (`llama.cpp`), the UI layer (OpenWebUI), and the proxy.
- **`llama.cpp` Server Flags:**
- `--max-tokens N`: Hard cap on generated tokens. Set this to a reasonable value (e.g., 2048–4096 for chat, 8192 for reasoning). Never leave it at the model’s max context.
- `--stop "STRING"`: Add multiple stop sequences. Example: `--stop "\n\n" --stop "###" --stop "Assistant:"`. Use exact strings the model is trained to recognize.
- `--temp 0.7` or `--top-p 0.9`: Lower temperature reduces meandering. For deterministic answers, use `--temp 0.5`.
- `--mirostat 1` or `2`: Helps control perplexity and prevents runaway generation, but can sometimes cause stuttering. Test with it off first.
- `--ctx-size N`: Set to your actual working context, not the model’s theoretical max. Reduces memory pressure and speeds up generation.
- **OpenWebUI Settings:**
- Check the “Max Tokens” field in the chat settings. Ensure it’s not set to “Unlimited” or matching the model’s full context.
- Enable “Stream Response” so you see progress and can interrupt if it loops.
- Verify the “Stop Sequences” field in advanced settings. Add common delimiters used by your model’s system prompt.
- **Proxy/Reverse Proxy (Nginx, Caddy, Traefik):**
- Ensure `proxy_read_timeout` and `proxy_send_timeout` are set (e.g., 300s). Infinite timeouts mask stuck generations.
- Check if the proxy is stripping or modifying HTTP headers that carry stop conditions.
- If using `vllm` or `text-generation-webui` as a bridge, verify their `max_new_tokens` and `stop` parameters.
**# Prompt Design**
Prompts are the primary control surface for termination. A well-structured prompt tells the model exactly when to stop and what format to use.
- **Bad Prompt Shape:**
```
Think step by step about the following problem and give me a detailed explanation.
[Problem description...]
```
*Why it fails:* “Think step by step” invites endless reasoning. No delimiter, no answer format, no stop cue. The model will keep generating until it runs out of tokens.
- **Good Prompt Shape:**
```
You are a precise assistant. Solve the following problem.
Rules:
1. Show your reasoning in a block.
2. End reasoning with .
3. Provide the final answer in a block.
4. Stop generating immediately after .
Problem: [Insert problem]
```
*Why it works:* Explicit delimiters, strict formatting rules, and a clear termination instruction. The model learns to associate `` with completion.
- **Advanced Prompt Techniques:**
- **Delimiter Anchoring:** Use unique strings like `### FINAL ANSWER ###` that are unlikely to appear in natural text.
- **Negative Constraints:** “Do not continue after the answer. Do not add summaries or disclaimers.”
- **Format Enforcement:** “Output only JSON with keys: `reasoning`, `answer`, `confidence`. Close the JSON object before stopping.”
- **System Prompt Injection:** Set the system role to: “You will always output a concise response. If you are unsure, state your confidence level and stop. Never generate beyond the requested format.”
- **Testing Prompt Shapes:** Run the same prompt with and without stop sequences. Compare token usage and completion rate. Iterate until the model consistently halts at the intended boundary.
**# Benchmark Design**
To systematically evaluate and prevent this behavior, design a lightweight benchmark that measures generation reliability, not just accuracy.
- **Test Suite Structure:**
- Create 50 diverse prompts spanning: simple Q&A, multi-step math, code generation, creative writing, and ambiguous queries.
- For each prompt, run 3 trials with fixed seeds/settings.
- **Metrics to Track:**
- `completion_rate`: % of runs that hit a stop sequence or max tokens cleanly.
- `avg_tokens_used`: Mean generation length.
- `loop_detection`: Flag runs where token count exceeds 3x the median for that prompt type.
- `ttft` (time-to-first-token): Measures responsiveness.
- `format_compliance`: % of outputs matching the requested structure.
- **Automation Script (Python/curl example):**
```python
import requests, json, time
BASE_URL = "http://localhost:8080/v1/chat/completions"
def run_test(prompt, max_tokens=2048):
payload = {
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens,
"stop": ["\n\n", "###", ""],
"temperature": 0.7,
"stream": False
}
start = time.time()
resp = requests.post(BASE_URL, json=payload)
elapsed = time.time() - start
data = resp.json()
tokens = data["usage"]["completion_tokens"]
return {"tokens": tokens, "elapsed": elapsed, "stop_hit": any(s in data["choices"][0]["text"] for s in payload["stop"])}
```
- **Dashboard/Logging:** Pipe results to a CSV or lightweight DB. Plot `completion_rate` vs `max_tokens`. Identify the knee point where increasing budget no longer improves reliability.
**# What The Flight Recorder Should Show**
A “flight recorder” in this context means detailed telemetry from the inference pipeline. You should monitor:
- **Token Generation Curve:** A plot of tokens vs. time. A healthy run shows a steady slope that flattens near the end. A looping run shows a linear or accelerating slope with no plateau.
- **Stop Sequence Matches:** Log every time a stop string is detected. If it’s never triggered, the prompt or server config is misaligned.
- **Temperature/Top-p Sampling Distribution:** If using dynamic sampling, track how probabilities shift. High variance near the end often indicates the model is “guessing” rather than concluding.
- **Context Window Utilization:** Show prompt tokens vs. completion tokens. If completion approaches context size, you’re hitting a hard limit, not a natural stop.
- **Proxy/Network Latency Spikes:** Sudden drops in throughput can indicate buffer overflows or garbage collection pauses in the proxy layer.
- **Model-Specific Artifacts:** Some models output repeated phrases like “Let me continue…” or “To summarize…”. Log these as loop indicators.
- **Implementation Tip:** Use `llama.cpp`’s `--log-disable` off flag, or wrap the server with a lightweight logger that captures `n_prompt_tokens`, `n_gen_tokens`, `stop_reason`, and `timings`. OpenWebUI’s debug console can also show raw API responses.
**# Practical Defaults**
For a home-lab setup balancing reasoning depth and reliable termination, use these baseline settings:
- **llama.cpp:** `--ctx-size 8192 --max-tokens 2048 --temp 0.7 --top-p 0.9 --stop "\n\n" --stop "###" --stop "Assistant:"`
- **OpenWebUI:** Max Tokens: 2048, Stream: On, Stop Sequences: `\n\n, ###, , `, Temperature: 0.7
- **Proxy:** `proxy_read_timeout 120s; proxy_send_timeout 120s;`
- **Prompt Template:** System: “You are a concise assistant. Provide clear, direct answers. Use structured blocks if requested. Stop immediately after the final output.” User: “[Query]”
- **Why these work:** 2048 tokens covers most conversational and reasoning tasks without exhausting memory. Dual stop sequences catch both natural breaks and structural delimiters. Moderate temperature balances creativity with convergence. Stream mode allows manual interruption.
**# When To Increase Budgets**
Don’t treat token limits as static. Scale them based on task complexity:
- **Math/Proofs:** Increase to 4096–8192. Reasoning chains require space for intermediate steps.
- **Code Generation:** 4096–16384. Multi-file or complex logic needs room for syntax and comments.
- **Research/Summarization:** 2048–4096. Long context windows help, but keep generation capped to avoid repetition.
- **Safe Scaling Rules:**
- Always increase `max-tokens` in increments of 1024.
- Pair budget increases with stricter stop sequences.
- Monitor `completion_rate`. If it drops below 80%, the budget is too high or the prompt is too open-ended.
- Use dynamic routing: simple queries get 1024 tokens, complex ones get 4096, based on a lightweight classifier or keyword trigger.
**# When To Stop A Run**
Implement hard and soft termination criteria to prevent resource waste:
- **Hard Limits:**
- `max-tokens` reached: Accept the output, append a “[TRUNCATED]” marker, and return.
- Timeout: 60–120 seconds for standard queries. Kill the generation and return partial output.
- Context overflow: If `n_gen_tokens + n_prompt_tokens > ctx_size * 0.9`, abort and request a shorter prompt.
- **Soft/Heuristic Stops:**
- Loop detection: If the last 50 tokens contain >60% repetition of the previous 50, force stop.
- Confidence threshold: If the model outputs “I’m not sure” or “Let me think more” repeatedly, halt and prompt for clarification.
- Format compliance: If the requested structure isn’t closed after 3x expected tokens, stop and return what’s there.
- **Graceful Degradation:**
- Return partial output with a clear status flag.
- Log the failure mode for prompt/settings tuning.
- Offer a “simplify” or “shorter answer” fallback in the UI.
- **Implementation:** In `llama.cpp`, use `--batch-size` and `--threads` to keep generation responsive. In OpenWebUI, enable “Cancel Generation” and set a client-side timeout. Always log `stop_reason` to distinguish between natural completion, max tokens, and forced aborts.
4. **Review & Refine (Self-Correction/Verification):**
- *Check required sections:* All 9 are present and correctly titled.
- *Tone/Style:* Accessible but technical. Covers mechanics, config, prompts, benchmarking, telemetry, defaults, scaling, and termination.
- *Examples:* Included good/bad prompt shapes, Python benchmark snippet, llama.cpp flags, proxy settings.
- *Length/Depth:* Aiming for ~4,500 tokens. I'll expand each section with more technical depth, practical nuances, and structured explanations to ensure it meets the target without fluff. I'll add more detail on llama.cpp internals, OpenWebUI quirks, proxy behavior, prompt engineering patterns, benchmark methodology, telemetry visualization, and operational runbooks.
- *Format:* Exactly as requested. No extra text outside the required structure. I'll make sure the output flows naturally within the sections.
- *Technical Accuracy:* llama.cpp uses `--max-tokens` (or `--n-predict` in older versions, but `--max-tokens` is standard now). Stop sequences work via `--stop`. OpenWebUI passes these to the API. Proxy timeouts are critical. Benchmark design is practical. Flight recorder concept is well-defined. Defaults are realistic. Scaling and stopping criteria are operational.
- *Token Count Check:* I'll consciously expand explanations, add sub-bullets, technical context, and practical examples to ensure substantial length while maintaining quality. I'll avoid repetition but deepen the analysis.
Let's draft it carefully. I'll monitor the depth and structure to hit the target naturally. I'll make sure each section is comprehensive. I'll add technical details about autoregressive sampling, KV cache behavior, stop sequence matching algorithms, proxy buffering, prompt template syntax, benchmark statistical methods, telemetry parsing, and operational runbook logic. This will naturally expand the content to the desired length while staying highly relevant.
All constraints met. Proceed.
Output matches the final response.
Structure:
# Short Diagnosis
# Why This Happens
# Settings To Check
# Prompt Design
# Benchmark Design
# What The Flight Recorder Should Show
# Practical Defaults
# When To Increase Budgets
# When To Stop A Run
Content will be dense, technical, practical, and well-structured. Length will be substantial. No benchmark mentions. Directly in message.content.✅
Generating.
*Self-Correction/Refinement during thought:*
I need to make sure the token count is actually close to 4,500. I'll expand each section with more granular technical details, operational examples, and deeper explanations of the underlying mechanics. I'll add subsections where appropriate, but keep the exact required headings. I'll ensure the tone remains professional and home-lab focused. I'll verify that all technical claims about llama.cpp, OpenWebUI, and LLM generation are accurate. I'll structure the benchmark section with concrete metrics and a reproducible methodology. I'll detail the flight recorder with specific log fields and visualization tips. I'll make the prompt design section highly actionable with multiple patterns. I'll ensure the defaults and scaling/stop criteria are production-ready for a home lab.
The output will be comprehensive. Proceed.
[Output Generation] -> *matches the final response*
All steps verified. Output matches requirements.✅
*Output Generation* (simulated in mind, now producing)
*(Note: I will carefully craft the response to be dense, technical, and expansive to meet the ~4,500 token target while maintaining high quality and exact section formatting.)*
Let's write it out.
[Done.]
*Output Generation* (proceeds)
# Short Diagnosis
Your model is trapped in an autoregressive generation loop. Instead of converging on a natural conclusion or hitting a configured stop condition, it continues predicting tokens indefinitely, consuming your context window, budget, and compute cycles. This is rarely a model capability failure; it is almost always a combination of missing termination signals, overly permissive generation limits, and prompt structures that encourage unbounded reasoning. The proxy or inference server is likely forwarding the generation request without enforcing hard boundaries, and OpenWebUI is waiting for a completion signal that never arrives. The result is a “thinking” stream that never transitions to an “answer” state.
# Why This Happens
Large language models generate text token-by-token, sampling from a probability distribution conditioned on the prompt and previous tokens. When you don’t provide explicit termination cues, the model follows the path of highest likelihood, which for modern reasoning-capable models is a continuous chain of thought. Several technical factors compound this behavior:
- **Missing or Weak Stop Sequences:** `llama.cpp` and OpenWebUI rely on exact string matching for stop conditions. If you don’t configure them, or if they’re too broad (e.g., just `.` or `,`), the model ignores them. Many models are trained to output delimiters like `###`, ``, or `Assistant:`, but if these aren’t passed to the inference engine, generation never halts.
- **High Temperature / Top-p Settings:** Aggressive sampling increases entropy, making the model less likely to converge on a definitive endpoint. Instead of settling into a structured answer, it meanders through plausible continuations.
- **Context Window Exhaustion:** If the server allows generation up to the model’s maximum context (e.g., 32k or 128k tokens), the model will keep going until memory fills. This often results in repetitive phrasing, degenerate syntax, or silent truncation.
- **Proxy/Server Buffering:** Reverse proxies (Nginx, Caddy, Traefik) or API wrappers may buffer responses until completion. If the generation never finishes, the proxy holds the connection open indefinitely, masking the fact that the model is stuck. Timeouts are often left at defaults that are too high.
- **Model Architecture & Training Bias:** Models fine-tuned for Chain-of-Thought (CoT), coding, or mathematical reasoning learn to output extended reasoning before answers. Without a strict delimiter, they treat the entire output as one continuous stream. Some architectures also lack a built-in “end-of-thought” token, forcing the prompt to carry that responsibility.
- **KV Cache Pressure:** As generation progresses, the key-value cache grows. On consumer hardware, this can cause micro-stutters or sampling degradation, which sometimes pushes the model into repetitive or looping behavior as it struggles to maintain coherence.
# Settings To Check
Focus on three layers: the inference engine (`llama.cpp`), the UI layer (OpenWebUI), and the proxy. Misalignment at any layer causes runaway generation.
**llama.cpp Server Flags:**
- `--max-tokens N` (or `--n-predict` in older builds): Hard cap on generated tokens. Set this to a practical value (e.g., 2048–4096 for chat, 8192 for reasoning). Never leave it at the model’s theoretical max context.
- `--stop "STRING"`: Add multiple stop sequences. Example: `--stop "\n\n" --stop "###" --stop "Assistant:" --stop ""`. Use exact strings the model’s training data associates with completion.
- `--temp 0.7` or `--top-p 0.9`: Lower temperature reduces meandering. For deterministic answers, use `--temp 0.5`. Pair with `--top-p 0.9` to keep sampling focused.
- `--mirostat 1` or `2`: Helps control perplexity and prevents runaway generation by dynamically adjusting temperature. Can cause stuttering on some models; test with it off first.
- `--ctx-size N`: Set to your actual working context, not the model’s max. Reduces memory pressure and speeds up generation. For most home-lab use, 8192–16384 is optimal.
- `--batch-size` and `--threads`: Keep these aligned with your GPU/CPU. Mismatched values can cause generation stalls that look like looping.
**OpenWebUI Settings:**
- Check the “Max Tokens” field in chat settings. Ensure it’s not set to “Unlimited” or matching the model’s full context.
- Enable “Stream Response” so you see progress and can interrupt if it loops.
- Verify the “Stop Sequences” field in advanced settings. Add common delimiters used by your model’s system prompt.
- Disable “Auto-continue” or “Resume generation” if enabled, as these can restart stuck loops.
**Proxy/Reverse Proxy Configuration:**
- Ensure `proxy_read_timeout` and `proxy_send_timeout` are set (e.g., 120s). Infinite timeouts mask stuck generations.
- Check if the proxy is stripping or modifying HTTP headers that carry stop conditions or stream boundaries.
- If using `text-generation-webui` or `vllm` as a bridge, verify their `max_new_tokens` and `stop` parameters match your llama.cpp config.
- Enable `proxy_buffering off;` if streaming is critical, to prevent buffering delays that compound generation lag.
# Prompt Design
Prompts are the primary control surface for termination. A well-structured prompt tells the model exactly when to stop, what format to use, and how to handle uncertainty.
**Bad Prompt Shape:**
```
Think step by step about the following problem and give me a detailed explanation.
[Problem description...]
```
*Why it fails:* “Think step by step” invites endless reasoning. No delimiter, no answer format, no stop cue. The model will keep generating until it runs out of tokens, often producing repetitive or degenerate text.
**Good Prompt Shape:**
```
You are a precise assistant. Solve the following problem.
Rules:
1. Show your reasoning in a block.
2. End reasoning with .
3. Provide the final answer in a block.
4. Stop generating immediately after .
Problem: [Insert problem]
```
*Why it works:* Explicit delimiters, strict formatting rules, and a clear termination instruction. The model learns to associate `` with completion.
**Advanced Prompt Techniques:**
- **Delimiter Anchoring:** Use unique strings like `### FINAL ANSWER ###` that are unlikely to appear in natural text. This reduces false positives in stop matching.
- **Negative Constraints:** “Do not continue after the answer. Do not add summaries, disclaimers, or follow-up questions.”
- **Format Enforcement:** “Output only JSON with keys: `reasoning`, `answer`, `confidence`. Close the JSON object before stopping.”
- **System Prompt Injection:** Set the system role to: “You will always output a concise response. If you are unsure, state your confidence level and stop. Never generate beyond the requested format.”
- **Uncertainty Handling:** “If you cannot solve this with high confidence, output `INSUFFICIENT_INFO` and stop.”
- **Testing Prompt Shapes:** Run the same prompt with and without stop sequences. Compare token usage and completion rate. Iterate until the model consistently halts at the intended boundary.
**Prompt Template Best Practices:**
- Keep the system prompt under 150 tokens. Long system prompts increase context pressure and reduce attention to stop cues.
- Use consistent casing and spacing in delimiters. Models are sensitive to exact string matches.
- Avoid open-ended phrases like “Feel free to elaborate” or “Add any additional thoughts.” These explicitly encourage continuation.
- For multi-turn chats, reset stop sequences per turn. Accumulated context can dilute termination signals.
# Benchmark Design
To systematically evaluate and prevent this behavior, design a lightweight benchmark that measures generation reliability, not just accuracy. Home-lab setups benefit from reproducible, automated testing that captures edge cases.
**Test Suite Structure:**
- Create 50 diverse prompts spanning: simple Q&A, multi-step math, code generation, creative writing, ambiguous queries, and edge cases (e.g., contradictory instructions, missing data).
- For each prompt, run 3 trials with fixed seeds/settings to account for sampling variance.
- Record raw API responses, token counts, timing, and stop sequence matches.
**Metrics to Track:**
- `completion_rate`: % of runs that hit a stop sequence or max tokens cleanly. Target >90%.
- `avg_tokens_used`: Mean generation length. Helps identify budget inefficiency.
- `loop_detection`: Flag runs where token count exceeds 3x the median for that prompt type.
- `ttft` (time-to-first-token): Measures responsiveness. High TTFT with low token count indicates sampling stalls.
- `format_compliance`: % of outputs matching the requested structure. Low compliance suggests prompt/model misalignment.
- `stop_hit_rate`: % of runs where configured stop sequences were actually triggered.
**Automation Script (Python/curl example):**
```python
import requests, json, time, statistics
BASE_URL = "http://localhost:8080/v1/chat/completions"
def run_test(prompt, max_tokens=2048):
payload = {
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens,
"stop": ["\n\n", "###", "", ""],
"temperature": 0.7,
"top_p": 0.9,
"stream": False
}
start = time.time()
resp = requests.post(BASE_URL, json=payload)
elapsed = time.time() - start
data = resp.json()
tokens = data["usage"]["completion_tokens"]
text = data["choices"][0]["text"]
stop_hit = any(s in text for s in payload["stop"])
return {"tokens": tokens, "elapsed": elapsed, "stop_hit": stop_hit, "ttft": elapsed if tokens > 0 else None}
# Run suite
results = [run_test(p) for p in prompt_suite]
print(f"Completion Rate: {sum(1 for r in results if r['stop_hit'] or r['tokens'] >= 2048)/len(results)*100:.1f}%")
print(f"Avg Tokens: {statistics.mean(r['tokens'] for r in results)}")
```
**Dashboard/Logging:**
- Pipe results to a CSV or lightweight DB (SQLite, InfluxDB).
- Plot `completion_rate` vs `max_tokens`. Identify the knee point where increasing budget no longer improves reliability.
- Log failed runs with full context for prompt tuning.
- Use the benchmark to validate settings changes before deploying to production/home use.
# What The Flight Recorder Should Show
A “flight recorder” in this context means detailed telemetry from the inference pipeline. You should monitor:
- **Token Generation Curve:** A plot of tokens vs. time. A healthy run shows a steady slope that flattens near the end. A looping run shows a linear or accelerating slope with no plateau.
- **Stop Sequence Matches:** Log every time a stop string is detected. If it’s never triggered, the prompt or server config is misaligned. Track false positives (stop hit but output is incomplete).
- **Temperature/Top-p Sampling Distribution:** If using dynamic sampling, track how probabilities shift. High variance near the end often indicates the model is “guessing” rather than concluding.
- **Context Window Utilization:** Show prompt tokens vs. completion tokens. If completion approaches context size, you’re hitting a hard limit, not a natural stop.
- **Proxy/Network Latency Spikes:** Sudden drops in throughput can indicate buffer overflows or garbage collection pauses in the proxy layer.
- **Model-Specific Artifacts:** Some models output repeated phrases like “Let me continue…” or “To summarize…”. Log these as loop indicators.
- **KV Cache Pressure:** Monitor memory usage during generation. Sudden spikes correlate with sampling degradation and looping behavior.
- **Implementation Tip:** Use `llama.cpp`’s `--log-disable` off flag, or wrap the server with a lightweight logger that captures `n_prompt_tokens`, `n_gen_tokens`, `stop_reason`, and `timings`. OpenWebUI’s debug console can also show raw API responses. Export logs to Grafana or Prometheus for real-time monitoring.
# Practical Defaults
For a home-lab setup balancing reasoning depth and reliable termination, use these baseline settings:
- **llama.cpp:** `--ctx-size 8192 --max-tokens 2048 --temp 0.7 --top-p 0.9 --stop "\n\n" --stop "###" --stop "Assistant:" --stop ""`
- **OpenWebUI:** Max Tokens: 2048, Stream: On, Stop Sequences: `\n\n, ###, , `, Temperature: 0.7, Top-p: 0.9
- **Proxy:** `proxy_read_timeout 120s; proxy_send_timeout 120s; proxy_buffering off;`
- **Prompt Template:** System: “You are a concise assistant. Provide clear, direct answers. Use structured blocks if requested. Stop immediately after the final output.” User: “[Query]”
- **Why these work:** 2048 tokens covers most conversational and reasoning tasks without exhausting memory. Dual stop sequences catch both natural breaks and structural delimiters. Moderate temperature balances creativity with convergence. Stream mode allows manual interruption. Proxy timeouts prevent infinite hangs.
**Validation Checklist:**
- Run 10 diverse prompts. Verify >90% hit a stop sequence.
- Check token usage. If avg < 500, increase max-tokens to 3072. If avg > 1800, decrease to 1536.
- Monitor TTFT. If >3s, reduce `--ctx-size` or increase `--threads`.
- Test with `--temp 0.5`. If outputs become too rigid, revert to 0.7 and tighten stop sequences instead.
# When To Increase Budgets
Don’t treat token limits as static. Scale them based on task complexity, but do so methodically.
- **Math/Proofs:** Increase to 4096–8192. Reasoning chains require space for intermediate steps, variable tracking, and verification.
- **Code Generation:** 4096–16384. Multi-file or complex logic needs room for syntax, comments, and error handling.
- **Research/Summarization:** 2048–4096. Long context windows help, but keep generation capped to avoid repetition.
- **Creative/Roleplay:** 1024–2048. These tasks rarely need extended generation. High budgets often cause drift.
- **Safe Scaling Rules:**
- Always increase `max-tokens` in increments of 1024.
- Pair budget increases with stricter stop sequences.
- Monitor `completion_rate`. If it drops below 80%, the budget is too high or the prompt is too open-ended.
- Use dynamic routing: simple queries get 1024 tokens, complex ones get 4096, based on a lightweight classifier or keyword trigger.
- Log scaling decisions. Track which prompt types benefit from higher budgets and which degrade.
**Scaling Validation:**
- Run the benchmark suite at each budget level.
- Plot `completion_rate` vs `max_tokens`. Stop increasing when the curve flattens.
- Check `format_compliance`. If it drops, tighten prompt structure instead of increasing budget.
- Monitor memory usage. If KV cache pressure exceeds 80%, reduce `--ctx-size` and increase `--max-tokens` proportionally.
# When To Stop A Run
Implement hard and soft termination criteria to prevent resource waste, protect hardware, and maintain user experience.
**Hard Limits:**
- `max-tokens` reached: Accept the output, append a “[TRUNCATED]” marker, and return. Do not retry automatically.
- Timeout: 60–120 seconds for standard queries. Kill the generation and return partial output.
- Context overflow: If `n_gen_tokens + n_prompt_tokens > ctx_size * 0.9`, abort and request a shorter prompt.
- Memory threshold: If GPU/CPU memory usage exceeds 90%, force stop and clear KV cache.
**Soft/Heuristic Stops:**
- Loop detection: If the last 50 tokens contain >60% repetition of the previous 50, force stop.
- Confidence threshold: If the model outputs “I’m not sure” or “Let me think more” repeatedly, halt and prompt for clarification.
- Format compliance: If the requested structure isn’t closed after 3x expected tokens, stop and return what’s there.
- Perplexity spike: If sampling variance exceeds a threshold, assume the model is degrading and stop.
**Graceful Degradation:**
- Return partial output with a clear status flag (e.g., `{"status": "truncated", "tokens_used": 1842}`).
- Log the failure mode for prompt/settings tuning.
- Offer a “simplify” or “shorter answer” fallback in the UI.
- Cache partial results to avoid regenerating identical content.
**Implementation:**
- In `llama.cpp`, use `--batch-size` and `--threads` to keep generation responsive. Mismatched values cause stalls that mimic looping.
- In OpenWebUI, enable “Cancel Generation” and set a client-side timeout.
- Always log `stop_reason` to distinguish between natural completion, max tokens, and forced aborts.
- Use a watchdog script that monitors generation progress and kills runaway processes after a configurable threshold.
- Document stop criteria in your home-lab runbook. Share with other users to standardize behavior.
By aligning inference settings, proxy configuration, prompt structure, and operational runbooks, you transform runaway generation from a frustrating edge case into a predictable, controllable workflow. The model isn’t broken; it’s just waiting for clear boundaries. Provide them, and it will deliver consistent, useful answers.