`, CoT). If the prompt doesn't constrain the format or the stop sequences aren't set, the model might get stuck in a reasoning loop or generate filler. - Context window limits: When `n_ctx` is reached, generation stops abruptly or loops. - Temperature/Repetition: Low temp + high repetition penalty can cause degeneration or stalling. High temp can cause rambling. - Proxy/Server config: OpenWebUI -> llama.cpp bridge might have mismatched settings. *# Settings To Check* - `n_ctx` (context window) - `n_predict` / `max_tokens` (generation limit) - `stop` sequences (crucial for ending generation) - `temperature`, `top_p`, `repetition_penalty` - `min_p`, `typical_p` - `penalty_last_n` - `mirostat` settings (if applicable) - OpenWebUI specific: max tokens, stop words, system prompt overrides. - Explain how to verify these in llama.cpp server (`/v1/models`, `/v1/chat/completions` payload). *# Prompt Design* - Explain prompt structure: System, User, Assistant turns. - Explicit output formatting: JSON, markdown, clear delimiters. - Chain-of-thought constraints: Force reasoning to be bounded. - Good vs Bad examples: Show how vague prompts cause rambling, how structured prompts force answers. - Techniques: "Answer directly after reasoning", "Use [ANSWER] tag", "Limit reasoning to X sentences". - Few-shot examples. *# Benchmark Design* - How to test if the fix works: Create a synthetic test suite. - Metrics: Time to first token, total tokens, stop sequence hit rate, answer presence/quality. - Tools: `llama-bench`, custom Python scripts, OpenWebUI evaluation features. - Test cases: Simple QA, complex reasoning, code generation, edge cases (empty context, max context). - How to structure the benchmark for reproducibility. *# What The Flight Recorder Should Show* - Explain what a "flight recorder" means in this context (llama.cpp server logs, OpenWebUI request/response logs, token-by-token streaming logs). - Key indicators: Token generation rate, stop sequence triggers, context window usage, repetition loops, temperature drift. - How to read the logs: Look for `stopping`, `context full`, `repetition penalty triggered`, token distribution. - Example log snippets and what they mean. *# Practical Defaults* - Provide a concrete configuration block for llama.cpp/OpenWebUI. - Explain why each default is chosen. - Context: 4096-8192 (depending on VRAM), max tokens: 2048-4096, temp: 0.7-0.9, top_p: 0.9, rep_penalty: 1.1, stop: ["\nUser:", "<|eot_id|>", "[/ANSWER]", etc.]. - How to apply them. *# When To Increase Budgets* - Define "budgets": context window, max generation tokens, repetition penalty tolerance. - Scenarios: Long document analysis, multi-step reasoning, code generation with large files, iterative refinement. - How to safely increase: Monitor VRAM, use KV cache offloading, chunking strategies. - Warning signs of over-allocation. *# When To Stop A Run* - Hard limits vs soft limits. - Signs of degeneration: Repetition, gibberish, context exhaustion, timeout. - How to implement circuit breakers in OpenWebUI/llama.cpp. - Graceful degradation vs hard kill. - Post-mortem analysis. 3. **Draft Generation (Mental Refinement & Expansion):** I need to ensure each section is substantial. I'll aim for ~400-500 tokens per section, with detailed explanations, technical depth, and practical examples. I'll maintain a consistent, professional tone suitable for a home-lab engineer. *Structure Check:* - # Short Diagnosis - # Why This Happens - # Settings To Check - # Prompt Design - # Benchmark Design - # What The Flight Recorder Should Show - # Practical Defaults - # When To Increase Budgets - # When To Stop A Run I'll write out the content carefully, ensuring technical accuracy regarding llama.cpp, OpenWebUI, and LLM generation mechanics. I'll expand on token budgets, KV cache, stop sequences, prompt engineering techniques, and logging/monitoring. *Self-Correction/Verification during drafting:* - Need to make sure I explicitly address "spends all its tokens thinking and never answers". This is often due to missing stop sequences, `n_predict` being too high, or the model getting stuck in a reasoning loop (common with CoT models). - I should clarify that "thinking" in local models usually means generating tokens that look like reasoning but aren't bounded by a stop token or format constraint. - I'll include specific llama.cpp parameters (`--n-predict`, `--stop`, `--ctx-size`, `--temp`, `--repeat-penalty`). - I'll make sure the prompt design section has clear good/bad examples. - I'll expand the benchmark and flight recorder sections to be highly practical for a home-lab setup. - I'll ensure the total length approaches the target by providing thorough explanations, not fluff. Let's draft systematically. I will monitor token count implicitly by ensuring depth, examples, and technical explanations in each section. I'll avoid repetition and keep it tightly focused on the user's stack (OpenWebUI + llama.cpp). *Section-by-Section Expansion Plan:* - **Short Diagnosis**: 1-2 paragraphs. Core issue: unbounded generation + missing stop conditions + context exhaustion. - **Why This Happens**: 3-4 paragraphs. Autoregressive nature, CoT training artifacts, KV cache limits, temperature/repetition dynamics, proxy translation quirks. - **Settings To Check**: Detailed list with explanations for each llama.cpp/OpenWebUI parameter. How they interact. - **Prompt Design**: Principles, structure, explicit formatting, good/bad examples, few-shot, constraint injection. - **Benchmark Design**: How to build a test harness, metrics, test cases, automation, evaluation criteria. - **What The Flight Recorder Should Show**: Log analysis, token streaming patterns, stop triggers, context usage, repetition flags, how to interpret them. - **Practical Defaults**: Concrete config block, rationale, how to apply in OpenWebUI/llama.cpp. - **When To Increase Budgets**: Context vs generation tokens, VRAM implications, chunking, safe scaling. - **When To Stop A Run**: Degeneration signals, hard/soft limits, circuit breakers, graceful fallback, post-run analysis. I'll write it out now, ensuring technical precision and meeting the length/structure requirements. I'll aim for comprehensive coverage without padding. Each section will be substantive. I'll use markdown formatting appropriately. I'll make sure the tone matches a senior home-lab engineer advising a peer. All constraints met. Proceed. *Self-Correction/Note during thought*: The prompt says "Target final-answer length: roughly 4,500 tokens." I need to be mindful of actually generating enough content. I will expand each section with detailed technical explanations, examples, configuration snippets, and operational advice. I'll ensure the depth matches a professional guide. I'll avoid being overly verbose but will be thorough. I'll structure it carefully. Let's generate. *Output Generation matches the response.* I will carefully craft each section to be dense with actionable information, technical details about llama.cpp/OpenWebUI, prompt engineering, benchmarking, and operational monitoring. I'll ensure the examples are clear and the explanations bridge theory and practice. I'll maintain the exact section headers requested. I'll aim for the target length through substantive expansion. All good. Proceeds. [Output Generation] -> *See response.* *Self-Correction/Verification during generation simulation*: I need to make sure I don't accidentally truncate. I'll deliberately expand on the mechanics of token prediction, KV cache management, stop sequence parsing, prompt formatting, benchmark automation, log parsing, and operational thresholds. I'll include concrete JSON/config examples. I'll ensure the tone is consistent. The structure will strictly follow the required headers. I'll produce the full response now. *Output Generation* (matches the final response) [Done.] *Output Generation* (proceeds) # Short Diagnosis Your model is not actually "thinking" in a human sense; it is autoregressively predicting tokens until a termination condition is met. When it appears to spend all its tokens "thinking" and never delivers a final answer, you are typically witnessing one of three overlapping failures: (1) missing or misconfigured stop sequences, causing the model to generate unbounded reasoning or filler text; (2) an excessively high `n_predict`/`max_tokens` budget combined with a low repetition penalty or temperature, allowing the model to loop through internal monologue patterns; or (3) context window exhaustion (`n_ctx`), where the KV cache fills up and the server either truncates mid-generation, loops, or returns an empty/abrupt completion. In the OpenWebUI → llama.cpp proxy chain, these issues are often amplified by mismatched parameter translation, default system prompts that encourage verbose chain-of-thought, or lack of explicit output formatting constraints. The fix requires tightening generation limits, enforcing stop conditions, structuring prompts to bound reasoning, and monitoring token budgets at the server level. # Why This Happens Large language models generate text token-by-token. Each token prediction is conditioned on the entire preceding context window, which is stored in a Key-Value (KV) cache. The model continues generating until one of three conditions is satisfied: a stop sequence is matched, the maximum token limit (`n_predict`/`max_tokens`) is reached, or the context window (`n_ctx`) is exhausted. When a model appears to "think forever," it is usually trapped in a generation loop where none of these conditions are triggered effectively. Modern open-weight models are frequently trained on datasets containing explicit reasoning traces (e.g., `...`, `Let's think step by step`, or internal monologue formats). If your prompt does not explicitly constrain the reasoning length or format, the model may default to generating extended internal deliberation. Without a stop sequence that matches the end of that reasoning block, the model continues predicting tokens that look like reasoning but are actually degenerative filler. This is especially common with instruction-tuned models that were fine-tuned on long-form CoT (Chain-of-Thought) data but deployed without corresponding prompt scaffolding. Temperature and repetition penalties play a critical role here. A low temperature (e.g., 0.1–0.3) combined with a high repetition penalty can cause the model to become overly deterministic, repeating reasoning patterns or getting stuck in semantic loops. Conversely, a high temperature with a weak repetition penalty encourages rambling. The llama.cpp backend calculates token probabilities at each step; if the probability distribution flattens or peaks on self-referential tokens, generation drifts into unproductive territory. The OpenWebUI proxy layer adds another variable. OpenWebUI translates frontend settings into OpenAI-compatible API payloads, which llama.cpp then parses. If `stop` sequences are not properly escaped, if `max_tokens` is set to a default like 4096 or 8192 without regard to your actual context window, or if the system prompt implicitly encourages verbose output, the proxy will happily stream tokens until the budget is exhausted. Additionally, if your model's native chat template expects specific end-of-turn markers (e.g., `<|eot_id|>`, ``, `\n###`), and those are missing from the stop list, the model will never recognize when to yield control back to the user. Finally, KV cache management in llama.cpp is strict. Once `n_ctx` is reached, the server cannot append new tokens. Depending on your build flags and server configuration, this may result in silent truncation, abrupt termination, or fallback to a looped generation pattern. If your prompt + system message + ongoing generation approaches the context limit, the model will effectively run out of "room to think," and the final answer will never materialize. # Settings To Check To resolve unbounded generation, audit the following parameters in both OpenWebUI and your llama.cpp server configuration. These settings interact multiplicatively, so adjust them systematically. **1. `n_ctx` (Context Window)** - What it does: Defines the maximum number of tokens the KV cache can hold. - Check: Ensure it matches your model's training context (e.g., 4096, 8192, 32768). If set too low, generation will choke. If set too high without sufficient VRAM, llama.cpp will fall back to CPU offloading, drastically slowing token generation and increasing the chance of timeout-induced truncation. - Where to verify: `llama.cpp --ctx-size ` or OpenWebUI model settings → Context Length. **2. `n_predict` / `max_tokens`** - What it does: Caps the number of tokens the model will generate per request. - Check: This is the most common culprit. If set to 4096+ on a model that typically answers in 200–500 tokens, you are giving it a blank check to ramble. Set this to 1.5–2x your expected answer length. - Where to verify: OpenWebUI → Model Settings → Max Tokens, or llama.cpp `--n-predict`. **3. `stop` Sequences** - What it does: Tells the server to halt generation immediately when a specific string or token sequence is predicted. - Check: You must include end-of-turn markers native to your model's chat template, plus explicit answer delimiters. Common defaults: `["\nUser:", "<|eot_id|>", "", "[/ANSWER]", "###", "\n\n"]`. - Where to verify: OpenWebUI → Stop Sequences, or llama.cpp `--stop`. **4. `temperature` & `top_p`** - What they do: Control sampling randomness. Low temp = deterministic; high temp = creative/unstable. - Check: For factual/structured answers, keep `temperature` between 0.6–0.8 and `top_p` at 0.9. If generation feels "stuck," slightly increase temperature to break loops. If it rambles, decrease it. - Where to verify: OpenWebUI → Temperature / Top P. **5. `repetition_penalty` & `penalty_last_n`** - What they do: Penalize recently generated tokens to discourage looping. - Check: Default is usually 1.1. If set too high (>1.2), the model may become erratic or refuse to generate. If too low (<1.0), it will repeat reasoning patterns. `penalty_last_n` should match your expected reasoning length (e.g., 64–128). - Where to verify: llama.cpp `--repeat-penalty`, `--penalty-last-n`. **6. `min_p` / `typical_p`** - What they do: Alternative sampling methods that filter low-probability tokens more aggressively than `top_p`. - Check: If you're using `min_p`, keep it between 0.05–0.15. High values can cause premature termination; low values allow degeneration. - Where to verify: llama.cpp `--min-p`. **7. OpenWebUI Proxy Translation** - Check: OpenWebUI sometimes overrides server defaults. Verify that your model's custom settings aren't silently injecting a verbose system prompt or disabling stop sequences. Use the "Raw Request" viewer in OpenWebUI to inspect the actual JSON payload sent to llama.cpp. # Prompt Design Prompt structure is your primary control mechanism for bounding generation. LLMs do not "decide" to stop; they follow patterns. If your prompt lacks explicit formatting constraints, the model will default to its training distribution, which often favors verbose reasoning. **Core Principles:** 1. **Explicit Output Delimiters:** Force the model to wrap its final answer in a recognizable tag. This gives you a reliable stop sequence. 2. **Bounded Reasoning:** If you want chain-of-thought, specify a maximum length or structure. Unbounded CoT is the #1 cause of token exhaustion. 3. **Direct Instruction First:** Place the core request early. LLMs weight early tokens more heavily in instruction-following models. 4. **Few-Shot Scaffolding:** Provide 1–2 examples showing the exact input→reasoning→answer pattern you expect. **Bad Prompt Shape:** ``` User: Explain how transformers work. Give a detailed breakdown. ``` *Why it fails:* No length constraint, no output format, no stop trigger. The model will generate until `max_tokens` or context exhaustion. "Detailed breakdown" is subjective and encourages rambling. **Good Prompt Shape:** ``` System: You are a concise technical assistant. Always wrap your final answer in [ANSWER]...[/ANSWER]. Limit reasoning to 3 sentences. User: Explain how transformers work. Assistant: [REASONING] Transformers use self-attention to weigh token relationships... [ANSWER] The transformer architecture replaces recurrence with parallel attention layers, enabling efficient long-range dependency modeling and scalable training. [/ANSWER] ``` *Why it works:* Explicit delimiters, bounded reasoning, clear stop sequence (`[/ANSWER]`), and a few-shot pattern that demonstrates the expected structure. **Advanced Prompt Techniques:** - **Constraint Injection:** Add phrases like `"Respond in exactly 3 bullet points."` or `"Do not exceed 150 words."` LLMs respect explicit numerical constraints better than vague ones. - **Role Priming:** `"You are a terminal that outputs only JSON. No explanations."` drastically reduces filler. - **Negative Prompting (via System):** `"Never use phrases like 'In conclusion', 'To summarize', or 'Here is the answer'. Output directly."` - **Template Alignment:** Match your prompt to the model's native chat template. If using a Llama 3 model, use `<|start_header_id|>user<|end_header_id|>` formatting if bypassing OpenWebUI's auto-template. **Prompt Validation Checklist:** - Does it contain a unique stop delimiter? - Is reasoning explicitly bounded? - Are there clear examples of the desired output shape? - Does it avoid open-ended adjectives ("detailed", "comprehensive", "thorough")? # Benchmark Design To verify that your configuration changes actually resolve the issue, you need a reproducible benchmark harness. Home-lab environments benefit from lightweight, automated testing that measures both performance and output quality. **1. Test Suite Construction** Create a JSONL file with 20–30 diverse prompts: - 5 simple factual QA (expected: <100 tokens) - 5 multi-step reasoning (expected: 200–400 tokens) - 5 code generation (expected: structured, bounded) - 5 edge cases (empty context, max context, adversarial phrasing) - 5 "trap" prompts designed to trigger rambling (e.g., "Explain everything you know about X") **2. Metrics to Track** - `time_to_first_token` (ms): Indicates KV cache load and initial computation. - `total_tokens_generated`: Should align with expected ranges. - `stop_sequence_hit`: Boolean. Did generation terminate cleanly? - `answer_present`: Regex check for your delimiter (e.g., `\[ANSWER\]`). - `token_per_second`: Throughput indicator. Drops suggest VRAM pressure or CPU fallback. - `degeneration_score`: Manual or heuristic rating (0–1) based on repetition/gibberish. **3. Automation Script (Python + `requests`)** ```python import requests, json, time, re URL = "http://localhost:8080/v1/chat/completions" HEADERS = {"Content-Type": "application/json"} STOP = ["[/ANSWER]", "\nUser:", "<|eot_id|>"] def run_benchmark(prompts): results = [] for p in prompts: payload = { "model": "local-model", "messages": [{"role": "user", "content": p}], "max_tokens": 1024, "stop": STOP, "temperature": 0.7 } start = time.time() resp = requests.post(URL, headers=HEADERS, json=payload) data = resp.json() tokens = data.get("usage", {}).get("completion_tokens", 0) hit_stop = bool(re.search(r"\[ANSWER\]", data["choices"][0]["message"]["content"])) results.append({ "prompt_id": p["id"], "tokens": tokens, "hit_stop": hit_stop, "tps": tokens / (time.time() - start) }) return results ``` **4. Evaluation Criteria** - Pass rate: >90% of prompts must hit stop sequences and contain the answer delimiter. - Token budget adherence: <10% variance from expected ranges. - Degeneration threshold: Any prompt generating >800 tokens without a stop hit is a configuration failure. - Iterate: Adjust `max_tokens`, `stop`, and `temperature` based on failure patterns. Re-run until metrics stabilize. **5. Tooling Recommendations** - `llama-bench` for raw throughput testing. - OpenWebUI's built-in evaluation tab for qualitative scoring. - `litellm` or `vllm` proxy for load testing if scaling beyond single-model. - Log aggregation via `journalctl` or `docker logs` piped to a simple dashboard. # What The Flight Recorder Should Show In a home-lab context, a "flight recorder" refers to the aggregated telemetry from llama.cpp server logs, OpenWebUI request/response traces, and token-streaming metrics. Proper logging is essential for diagnosing generation failures post-mortem. **Key Log Indicators:** 1. **Stop Sequence Triggers:** Look for `stopping: matched stop sequence` in llama.cpp logs. If absent, your stop list is misconfigured or the model never predicted the delimiter. 2. **Context Window Usage:** `context full` or `kv cache full` indicates `n_ctx` exhaustion. Generation will truncate or loop. 3. **Repetition Penalty Activation:** `repetition penalty triggered` or high `repeat_penalty` logs suggest the model is fighting its own output. If frequent, lower the penalty or adjust `penalty_last_n`. 4. **Token Generation Rate:** Sudden drops in tokens/sec often indicate VRAM pressure, CPU offloading, or thermal throttling. Consistent rates suggest healthy inference. 5. **Temperature/Top-p Drift:** If using dynamic sampling, logs may show probability distribution flattening. This precedes degenerative output. **Example Log Snippet & Interpretation:** ``` [llama.cpp] prompt: 128 tokens, cache: 128/4096 [llama.cpp] generating 1024 tokens... [llama.cpp] stopping: matched stop sequence "[/ANSWER]" [llama.cpp] total time: 2.4s, t/s: 426.6 ``` *Healthy run.* Stop sequence hit, context well within limits, high throughput. ``` [llama.cpp] prompt: 3800 tokens, cache: 3800/4096 [llama.cpp] generating 4096 tokens... [llama.cpp] context full, truncating [llama.cpp] total time: 18.2s, t/s: 225.0 ``` *Failure.* Context exhaustion. The model ran out of KV cache space before completing. Solution: Increase `n_ctx`, reduce prompt length, or lower `max_tokens`. **How to Enable Detailed Logging:** - llama.cpp: `--log-disable` (disable) or run with `LOG_LEVEL=debug` environment variable. - OpenWebUI: Enable "Debug Mode" in settings to view raw API payloads and response headers. - Stream tokens to a file: `llama-server ... | tee inference.log` for post-analysis. - Use `htop`/`nvtop` alongside logs to correlate token generation with VRAM/CPU usage. **Flight Recorder Best Practices:** - Log every request ID, timestamp, model version, and parameter set. - Store raw completions for qualitative review. - Set up alerting for `context full` or `stop sequence miss` rates >5%. - Rotate logs daily to prevent disk exhaustion. # Practical Defaults Below is a battle-tested configuration block for llama.cpp + OpenWebUI that balances reasoning capability with strict output control. Adjust based on your hardware and model architecture. **llama.cpp Server Flags:** ```bash llama-server \ --model /path/to/model.gguf \ --ctx-size 8192 \ --n-predict 2048 \ --temp 0.75 \ --top-p 0.9 \ --repeat-penalty 1.1 \ --penalty-last-n 128 \ --stop "\nUser:" \ --stop "<|eot_id|>" \ --stop "[/ANSWER]" \ --stop "###" \ --log-disable \ --port 8080 ``` **OpenWebUI Model Settings:** - Max Tokens: `1500` - Temperature: `0.7` - Top P: `0.9` - Stop Sequences: `["[/ANSWER]", "\nUser:", "<|eot_id|>", "###"]` - System Prompt: `"You are a precise assistant. Wrap final answers in [ANSWER]...[/ANSWER]. Limit reasoning to 2-3 sentences. Output directly."` **Rationale:** - `ctx-size 8192`: Sufficient for most conversational and document-analysis tasks without excessive VRAM pressure. - `n-predict 2048`: Hard cap prevents runaway generation. Most useful answers fit within 500–1000 tokens. - `temp 0.75` / `top-p 0.9`: Balances creativity with determinism. Low enough to avoid rambling, high enough to avoid repetition loops. - `repeat-penalty 1.1`: Gentle nudge against looping without causing erratic token selection. - `penalty-last-n 128`: Covers typical reasoning spans. Longer spans risk penalizing legitimate repetition (e.g., code indentation). - Stop sequences: Multi-layered. Catches end-of-turn, explicit answer delimiters, and common markdown separators. **Application Notes:** - Apply these defaults globally in OpenWebUI, then override per-model if needed. - Test with a simple prompt first. If answers are too terse, increase `max_tokens` to 2000 and `temp` to 0.8. - If generation still loops, add `--min-p 0.1` to llama.cpp to filter low-probability tokens more aggressively. # When To Increase Budgets "Budgets" refer to context window (`n_ctx`), generation limits (`max_tokens`), and repetition tolerance. Increasing them is necessary but carries hardware and stability trade-offs. **Context Window (`n_ctx`):** - Increase when: Processing long documents, multi-turn conversations with history, or codebases requiring full-file context. - Safe scaling: Double in steps (4096 → 8192 → 16384). Monitor VRAM usage. Each 4K context increase typically adds 0.5–1.5 GB VRAM depending on model size. - Warning signs: OOM crashes, severe CPU offloading, token generation dropping below 10 t/s. - Mitigation: Use `--mlock`, enable KV cache quantization (`--cache-type q8_0`), or implement prompt chunking. **Generation Tokens (`max_tokens`):** - Increase when: Complex multi-step reasoning, long-form content generation, or code with extensive comments. - Safe scaling: Add 500-token increments. Pair with explicit length constraints in prompts. - Warning signs: Answers become verbose, stop sequences are missed, token budget exhaustion becomes frequent. - Mitigation: Enforce stricter stop sequences, add `"Keep responses concise"` to system prompt, or implement post-generation truncation. **Repetition Tolerance (`repeat-penalty` / `penalty_last_n`):** - Increase tolerance (lower penalty) when: Generating code, mathematical derivations, or structured data where repetition is semantically valid. - Safe scaling: Reduce `repeat-penalty` to 1.05–1.08. Increase `penalty_last_n` to 256. - Warning signs: Model starts repeating identical phrases or loops on reasoning steps. - Mitigation: Switch to `min_p` sampling, which handles structured repetition better than global penalties. **General Rule:** Never increase budgets without a corresponding constraint mechanism. More tokens without better stop sequences or prompt formatting guarantees degeneration. # When To Stop A Run Knowing when to terminate a generation is as important as knowing how to start one. Unbounded runs waste compute, degrade user experience, and can mask configuration errors. **Hard Stop Conditions:** 1. **Token Budget Exhaustion:** If `max_tokens` is reached without a stop sequence hit, terminate immediately. This indicates a prompt/configuration mismatch. 2. **Context Window Full:** `n_ctx` reached. Kill the run, log the event, and return a truncated response with a warning. 3. **Timeout Threshold:** If time-to-first-token > 5s or total generation > 30s for a simple prompt, abort. Suggests hardware bottleneck or model mismatch. 4. **Degeneration Detection:** Regex match for repeated phrases (>3 occurrences), gibberish token sequences, or probability collapse. Implement via a lightweight post-generation filter. **Soft Stop Conditions:** 1. **Stop Sequence Hit:** Clean termination. Parse and return the answer. 2. **Confidence Threshold:** If using logits, terminate when the top token probability exceeds 0.95 for 3 consecutive steps (indicates model certainty). 3. **User Interrupt:** Always honor manual stop. Implement graceful fallback to save partial context. **Circuit Breaker Implementation:** ```python # Pseudocode for OpenWebUI/llama.cpp integration if tokens_generated > max_tokens and not stop_hit: terminate_run(reason="budget_exhaustion") log_warning("Prompt likely missing stop delimiter or overly open-ended") elif context_usage > 0.95 * n_ctx: terminate_run(reason="context_full") log_warning("Increase n_ctx or reduce prompt length") elif repetition_score > 0.8: terminate_run(reason="degeneration") log_warning("Lower temperature or adjust repeat_penalty") ``` **Post-Run Analysis:** - Always log the termination reason. - Store the final 50 tokens for pattern analysis. - If >20% of runs terminate abnormally, revert to baseline defaults and audit prompt templates. - Implement a "retry with constrained prompt" fallback for failed runs. **Operational Best Practices:** - Set global timeouts in OpenWebUI (e.g., 20s max response time). - Use streaming responses to detect early degeneration. - Maintain a "safe mode" configuration that enforces strict limits for untrusted or automated prompts. - Regularly review termination logs to identify recurring failure patterns. By combining precise parameter tuning, disciplined prompt engineering, automated benchmarking, and robust termination logic, you will eliminate the "thinking forever" behavior while preserving the model's reasoning capabilities. The key is treating generation as a bounded, observable process rather than an open-ended conversation.