`, `### ANSWER:`). - Provide bad examples (open-ended, vague, encouraging endless reflection). - Provide good examples (structured, bounded, explicit output format). - Discuss system prompt engineering for reasoning models. - Explain how to use few-shot examples to guide behavior. *# Benchmark Design* - How to test/measure this behavior in a home lab. - Create a synthetic test suite: simple QA, complex reasoning, edge cases. - Metrics to track: tokens used, time to first token, time to final answer, truncation rate, stop sequence hit rate. - Tools: `llama-bench`, custom Python scripts, OpenWebUI logs. - How to structure a benchmark to isolate prompt vs. settings issues. *# What The Flight Recorder Should Show* - Explain what a "flight recorder" (request/response logger, proxy logs, llama.cpp verbose output) should capture. - Key fields: prompt tokens, completion tokens, total tokens, stop reason, latency, temperature, stop sequences matched. - How to interpret logs: look for `stop_reason: length` vs `stop_reason: stop`. - Example log entries showing healthy vs. stuck runs. *# Practical Defaults* - Provide a concrete configuration block for llama.cpp and OpenWebUI. - `n_ctx`: 8192 or 16384 - `n_predict`: 4096 - `stop`: `["", "### ANSWER:", "\n\nUser:", "Assistant:"]` - `temperature`: 0.7, `top_p`: 0.9, `repeat_penalty`: 1.1 - Explain why these work as a baseline. *# When To Increase Budgets* - Scenarios requiring more tokens: multi-step math, code generation, long document analysis, complex planning. - How to scale safely: increase `n_predict` in steps, monitor latency, adjust context window if needed. - Warning signs of over-allocation: diminishing returns, memory pressure, slower throughput. *# When To Stop A Run* - Hard limits vs. soft limits. - Time-based cutoffs, token-based cutoffs, pattern-based cutoffs (repetition, looping phrases). - How to implement in OpenWebUI/llama.cpp (streaming cutoffs, proxy timeouts, custom middleware). - Graceful degradation: fallback to "I couldn't complete the reasoning within the limit, here's what I have so far." 3. **Draft Generation (Mental Refinement & Expansion):** I need to ensure each section is substantial, technical yet accessible, and hits the token target. I'll expand with concrete examples, configuration snippets, architectural explanations, and troubleshooting workflows. I'll maintain a professional, support-engineer tone. *Structure Check:* - # Short Diagnosis - # Why This Happens - # Settings To Check - # Prompt Design - # Benchmark Design - # What The Flight Recorder Should Show - # Practical Defaults - # When To Increase Budgets - # When To Stop A Run I'll write each section carefully, ensuring depth. I'll use markdown formatting appropriately. I'll aim for ~500 tokens per section on average, with some sections naturally longer (Prompt Design, Settings, Flight Recorder). I'll include code blocks, JSON examples, and clear explanations. Let's draft systematically. *(Self-Correction/Verification during drafting)* - Need to make sure I explicitly address "llama.cpp behind a proxy" and "OpenWebUI". - Need to include good/bad prompt shapes as requested. - Need to hit ~4,500 tokens. I'll expand on technical details, provide comprehensive examples, and explain the underlying mechanics thoroughly. - I'll avoid fluff but ensure depth: explain tokenization, context windows, stop sequences, reasoning model behavior, proxy payload handling, OpenWebUI configuration paths, benchmark methodology, log analysis, and operational thresholds. I'll now generate the full response. Structure: - Short Diagnosis (~300) - Why This Happens (~600) - Settings To Check (~700) - Prompt Design (~800) - Benchmark Design (~500) - What The Flight Recorder Should Show (~500) - Practical Defaults (~400) - When To Increase Budgets (~300) - When To Stop A Run (~400) Total: ~4,500. I'll expand each with technical depth, examples, and actionable advice. Proceed. Output matches the required format exactly. All sections present. Technical but accessible. Includes prompt examples. Covers llama.cpp, proxy, OpenWebUI. Meets length target through comprehensive explanation.✅ Generating. [Output Generation] -> *Proceeds* # Short Diagnosis Your model is likely trapped in an unbounded reasoning loop or chain-of-thought (CoT) generation phase, exhausting the `max_tokens` (or `n_predict`) allocation before it ever reaches a structured final answer. In llama.cpp, when the generation limit is hit, the API returns `stop_reason: length`, which silently truncates the output. OpenWebUI then displays whatever partial text was streamed, often leaving you with pages of internal monologue and no conclusion. This behavior is especially common with modern reasoning-tuned models, overly permissive system prompts, missing stop sequences, or proxy configurations that strip or ignore termination signals. The fix requires aligning three layers: generation limits, explicit stop triggers, and prompt architecture that forces a clean transition from thinking to answering. # Why This Happens Large language models generate text autoregressively, predicting one token at a time based on the preceding context. When a model is prompted to "think step by step," "reason carefully," or is fine-tuned on chain-of-thought data, it enters a deliberate reasoning mode. Without explicit boundaries, the model treats reasoning as an open-ended process. It will continue generating explanatory tokens until one of three conditions is met: 1. It naturally decides the reasoning is complete and transitions to an answer. 2. It hits the `max_tokens` / `n_predict` ceiling and is forcibly truncated. 3. It encounters a stop sequence that matches its output. In home-lab deployments, condition #2 is the most common failure mode. Reasoning models often allocate 60–80% of their token budget to internal deliberation. If `n_predict` is set to 2048 or 4096, the model may spend 3,500 tokens just formulating hypotheses, checking assumptions, and drafting intermediate steps, leaving zero budget for the final response. When llama.cpp hits the limit, it stops streaming. OpenWebUI receives a complete HTTP 200 response with a truncated payload, and the UI renders the partial output as if it were intentional. Proxies compound this issue. If you're routing through nginx, traefik, or a custom reverse proxy, payload buffering, timeout thresholds, or header modifications can interfere with streaming termination. Some proxies drop trailing whitespace or strip custom stop sequences if they're not properly encoded. Additionally, if the proxy enforces a response size limit or timeout before the model finishes, the connection may close abruptly, leaving OpenWebUI with an incomplete stream. Another architectural factor is the model's training distribution. Models fine-tuned on CoT datasets (e.g., DeepSeek-R1, QwQ, or community reasoning merges) have learned to prioritize thorough deliberation over brevity. They often use implicit or explicit delimiters like ``, `### REASONING:`, or `Let's work through this:`. If your system prompt doesn't define how to close that block, the model defaults to its training prior: keep reasoning until it feels "done," which rarely aligns with your token budget. Finally, repetition penalties and temperature settings can cause looping. If `repeat_penalty` is too low, the model may cycle through similar phrasing. If `temperature` is too high, it may generate tangential explorations that never converge. Both behaviors inflate token consumption without advancing toward a final answer. # Settings To Check To resolve token exhaustion and missing answers, audit the following configuration layers. Each setting interacts with the others, so changes should be tested incrementally. **1. llama.cpp Generation Limits** - `n_predict` (or `max_tokens` in API mode): Controls the maximum number of tokens the model will generate per request. Default is often 2048. For reasoning models, start at 4096. If you consistently see truncation, increase in 1024-token increments. - `n_ctx` (context window): Must be larger than `n_predict + prompt_tokens`. If `n_ctx` is 8192 and your prompt uses 3000 tokens, you only have 5192 tokens for generation. Mismatched context windows cause silent truncation or degraded quality. - `stop` sequences: Critical for forcing termination. In llama.cpp CLI, use `--stop "### ANSWER:" --stop ""`. In the OpenAI-compatible API, pass `"stop": ["", "### ANSWER:", "\n\nUser:"]`. Without these, the model relies solely on `n_predict`. **2. OpenWebUI Configuration** - Navigate to Settings → Advanced → Generation Parameters. Verify `Max Tokens` matches your intended budget. OpenWebUI overrides llama.cpp defaults if set. - Check `Stop Sequences` in the UI. Add explicit delimiters that match your prompt structure. - Disable `Stream` temporarily to see if the full response is being generated but cut off by UI rendering. If the full response appears in non-streaming mode, the issue is streaming timeout or proxy buffering. **3. Proxy Configuration** - If using nginx/traefik, ensure `proxy_buffering off;` and `proxy_read_timeout 300s;` (or higher) are set. Reasoning models can take 30–90 seconds to generate 4000 tokens on consumer hardware. - Verify that the proxy isn't stripping trailing newlines or custom delimiters. Some proxies normalize whitespace, which can break stop sequence matching. - Check for payload size limits (`client_max_body_size`, `limit_req`, etc.) that might truncate large JSON responses. **4. Sampling Parameters** - `temperature`: Keep between 0.6–0.8 for reasoning tasks. Higher values increase tangential exploration. - `top_p`: 0.85–0.95 balances creativity and focus. - `repeat_penalty`: 1.1–1.15 prevents looping without suppressing valid repetition. - `min_p`: If supported, set to 0.05–0.1 to filter low-probability tokens that cause meandering. **5. Model-Specific Flags** - Some reasoning models require `--cache-type-k q8_0` or specific KV cache settings to maintain coherence over long generations. - If using a GGUF with embedded chat templates, verify that OpenWebUI isn't double-applying templates, which can inject extra tokens and shift the generation window. # Prompt Design Prompt architecture is the most effective lever for controlling reasoning behavior. Models follow structural cues more reliably than abstract instructions. Below are patterns that work, patterns that fail, and how to bridge the gap. **Bad Prompt Shapes** ```text User: Explain quantum entanglement. Think carefully and be thorough. ``` *Why it fails:* "Think carefully and be thorough" is an open-ended directive. The model interprets this as permission to explore indefinitely. No delimiter, no output format, no termination signal. Token budget is consumed by unbounded elaboration. ```text User: What's the best way to optimize this Python script? [pastes 50 lines] Reason step by step. ``` *Why it fails:* "Reason step by step" triggers CoT mode but provides no closing mechanism. The model will generate intermediate analysis, but without a forced transition, it may never output the optimized code. **Good Prompt Shapes** ```text System: You are a precise technical assistant. When reasoning, wrap your internal thoughts in and tags. After closing , immediately output your final answer under ### ANSWER:. Keep reasoning concise and focused. Never output text after ### ANSWER: except the direct response. User: Explain quantum entanglement. ``` *Why it works:* Explicit delimiters create a bounded reasoning zone. The model learns to treat `` as a hard boundary. The `### ANSWER:` marker forces a structural transition. The system prompt sets expectations for brevity and format. ```text System: Use the following format: [Step-by-step reasoning. Max 300 tokens.] ### ANSWER: [Direct, structured response. No additional commentary.] User: What's the best way to optimize this Python script? [pastes code] ``` *Why it works:* Token budget guidance ("Max 300 tokens") acts as a soft constraint. The format template is few-shot compatible. The model recognizes the pattern and adheres to it consistently. **Advanced Prompt Techniques** - **Explicit Budgeting:** Add `[Reasoning budget: ~40% of response]` to the system prompt. Some models respect proportional guidance. - **Forced Termination:** Include `If you reach the end of your reasoning, output immediately.` This reduces hesitation at the boundary. - **Output Schema:** For structured tasks, append `Return JSON with keys: {"reasoning_summary": "...", "final_answer": "..."}`. This bypasses free-form monologue entirely. - **Negative Constraints:** `Do not repeat steps. Do not add disclaimers after the answer. Stop generating once the answer is complete.` **Few-Shot Examples** Providing 1–2 examples dramatically improves compliance: ```text Example 1: User: Calculate 15% of 240. Assistant: 15% = 0.15 0.15 * 240 = 36 ### ANSWER: 36 Example 2: User: Why does the sky appear blue? Assistant: Rayleigh scattering causes shorter wavelengths (blue) to scatter more than longer wavelengths (red) in Earth's atmosphere. ### ANSWER: The sky appears blue due to Rayleigh scattering, which preferentially scatters shorter blue wavelengths of sunlight across the atmosphere. ``` Few-shot examples teach the model the exact transition pattern. OpenWebUI supports system prompt injection, so embed these in the system message or use a custom chat template. # Benchmark Design To systematically diagnose and optimize token usage, build a lightweight benchmark suite tailored to your home-lab environment. **Test Categories** 1. **Simple QA:** Factual questions requiring <200 tokens. Measures baseline efficiency. 2. **Multi-Step Reasoning:** Math, logic puzzles, or code debugging requiring 500–1500 tokens. Tests CoT behavior. 3. **Long Context Analysis:** Summarization or extraction from 2000+ token documents. Tests context window utilization. 4. **Edge Cases:** Ambiguous prompts, contradictory instructions, or open-ended creative tasks. Tests stop sequence reliability. **Metrics to Track** - `prompt_tokens`: Tokens consumed by input + system prompt. - `completion_tokens`: Tokens generated by the model. - `total_tokens`: Sum of both. - `stop_reason`: `stop` (hit delimiter) vs `length` (hit max tokens). - `time_to_first_token` (TTFT): Measures initial processing latency. - `generation_time`: Total time to complete or truncate. - `truncation_rate`: Percentage of runs ending with `stop_reason: length`. **Implementation** Use a Python script with `httpx` or `requests` to hit your llama.cpp OpenAI-compatible endpoint. Log responses to JSONL. Example structure: ```json { "test_id": "reasoning_math_01", "prompt_tokens": 45, "completion_tokens": 382, "stop_reason": "stop", "ttft_ms": 120, "gen_time_ms": 4500, "output_preview": "... \n### ANSWER: 42" } ``` Run each test 3–5 times to account for hardware variance (CPU cache, RAM pressure, background processes). Aggregate results to identify patterns: if `stop_reason: length` exceeds 20%, your budget or stop sequences are misaligned. **Benchmark Workflow** 1. Baseline with default settings. 2. Introduce explicit stop sequences. 3. Adjust `n_predict` in 1024-token steps. 4. Test prompt variants (with/without delimiters, with/without few-shot). 5. Record which configuration yields `stop_reason: stop` consistently with acceptable latency. # What The Flight Recorder Should Show A "flight recorder" in this context refers to a comprehensive request/response logger that captures every parameter, token count, timing metric, and termination state. This is essential for diagnosing silent failures. **Required Log Fields** - `request_id`: Unique identifier for tracing. - `timestamp`: ISO 8601 format. - `model`: GGUF filename or model ID. - `prompt_length`: Character and token count of input. - `system_prompt`: Hash or truncated preview. - `generation_params`: JSON of all sampling settings (`temperature`, `top_p`, `repeat_penalty`, `stop`, `n_predict`). - `response_tokens`: Completion token count. - `stop_reason`: `stop`, `length`, `eos`, or `error`. - `matched_stop_sequence`: Which delimiter triggered termination (if any). - `latency_breakdown`: `ttft_ms`, `gen_ms`, `total_ms`. - `output_truncated`: Boolean flag. - `proxy_headers`: Request/response headers if routing through a reverse proxy. **Interpreting Logs** - **Healthy Run:** `stop_reason: stop`, `matched_stop_sequence: ""`, `output_truncated: false`, `gen_ms` proportional to token count. - **Budget Exhaustion:** `stop_reason: length`, `matched_stop_sequence: null`, `output_truncated: true`, `completion_tokens == n_predict`. - **Proxy Interference:** `stop_reason: error` or abrupt stream cutoff, `gen_ms` unusually low, `output_truncated: true` but `completion_tokens < n_predict`. - **Looping/Repetition:** `stop_reason: stop` but `matched_stop_sequence` is a generic newline or whitespace, `output` contains repeated phrases, `gen_ms` high relative to token count. **Example Log Entry (Healthy)** ```json { "request_id": "req_8f3a2c", "timestamp": "2024-06-15T14:22:10Z", "model": "reasoning-7b-q4_k_m.gguf", "prompt_length": 142, "generation_params": {"temperature": 0.7, "top_p": 0.9, "n_predict": 4096, "stop": ["", "### ANSWER:"]}, "response_tokens": 312, "stop_reason": "stop", "matched_stop_sequence": "", "latency_breakdown": {"ttft_ms": 95, "gen_ms": 3800, "total_ms": 3895}, "output_truncated": false } ``` **Example Log Entry (Stuck)** ```json { "request_id": "req_9b1d4e", "timestamp": "2024-06-15T14:25:33Z", "model": "reasoning-7b-q4_k_m.gguf", "prompt_length": 142, "generation_params": {"temperature": 0.7, "top_p": 0.9, "n_predict": 2048, "stop": []}, "response_tokens": 2048, "stop_reason": "length", "matched_stop_sequence": null, "latency_breakdown": {"ttft_ms": 110, "gen_ms": 8200, "total_ms": 8310}, "output_truncated": true } ``` The second log shows classic budget exhaustion: no stop sequences, `n_predict` too low, forced truncation. The fix is immediate: add delimiters, increase `n_predict`, and verify stop sequence matching. # Practical Defaults Start with these baseline configurations. They balance reasoning depth, termination reliability, and home-lab hardware constraints. **llama.cpp CLI / API** ```bash --ctx-size 16384 --predict 4096 --stop "" --stop "### ANSWER:" --stop "\n\nUser:" --temperature 0.7 --top-p 0.9 --repeat-penalty 1.1 --min-p 0.05 --cache-type-k q8_0 ``` **OpenWebUI Settings** - Max Tokens: 4096 - Stop Sequences: ``, `### ANSWER:`, `\n\nUser:` - Temperature: 0.7 - Top P: 0.9 - Repeat Penalty: 1.1 - Stream: Enabled (with proxy buffering disabled) **System Prompt Template** ```text You are a precise technical assistant. When reasoning, wrap your internal thoughts in and tags. Keep reasoning concise and focused on the core problem. After closing , immediately output your final answer under ### ANSWER:. Do not add disclaimers, summaries, or additional commentary after the answer. If the question is simple, skip reasoning and output directly under ### ANSWER:. ``` These defaults work for 7B–13B reasoning models on consumer GPUs/CPUs. Adjust `--ctx-size` based on your RAM/VRAM. If you frequently analyze long documents, increase to 32768 but monitor KV cache memory usage. The stop sequences ensure clean termination, while the sampling parameters prevent meandering. # When To Increase Budgets Token budgets should scale with task complexity, not arbitrarily. Increase `n_predict` when: 1. **Multi-Step Math/Logic:** Problems requiring >3 intermediate steps often need 1500–2500 tokens for reasoning alone. 2. **Code Generation/Refactoring:** Debugging or rewriting functions requires context analysis, hypothesis generation, and validation. Budget 2000–4000 tokens. 3. **Document Analysis:** Summarizing or extracting insights from 1000+ token inputs requires proportional reasoning space. Allocate 1.5x the input length. 4. **Creative/Exploratory Tasks:** Brainstorming, scenario planning, or comparative analysis benefits from broader exploration. Use 3000–5000 tokens. **Scaling Protocol** - Increase in 1024-token increments. - Monitor `stop_reason`. If it shifts from `length` to `stop`, you've found the sweet spot. - Track latency. If generation time increases linearly but quality plateaus, you've exceeded useful budget. - Watch hardware metrics. KV cache grows with context. If VRAM/RAM usage exceeds 85%, reduce `n_ctx` or switch to a smaller quantization. **Warning Signs of Over-Allocation** - Diminishing returns: Additional tokens produce repetitive or tangential text. - Latency spikes: Generation time per token increases due to cache pressure. - Memory thrashing: System swaps or GPU OOM errors. - Stop sequence mismatch: Model ignores delimiters because it's "lost" in over-generation. # When To Stop A Run Not all runs should be allowed to consume full budgets. Implement hard and soft cutoffs to maintain system responsiveness and prevent resource exhaustion. **Hard Cutoffs** - **Token Limit:** Enforce `n_predict` as an absolute ceiling. Never exceed hardware limits. - **Time Limit:** Set proxy or application timeouts to 60–120 seconds. Reasoning models on consumer hardware rarely need >90 seconds for 4000 tokens. - **Memory Threshold:** Monitor VRAM/RAM. If usage exceeds 90%, terminate gracefully and return a partial response with a warning. **Soft Cutoffs** - **Repetition Detection:** If the same phrase appears >3 times within 200 tokens, trigger early termination. - **Stagnation Monitoring:** If token generation rate drops below 5 tokens/second for >10 seconds, the model is likely looping or stuck. - **Pattern Matching:** If the output contains `I think`, `Let me consider`, or `Another way to look at this` repeatedly without advancing, force a transition to `### ANSWER:`. **Implementation Strategies** - **OpenWebUI:** Use the `Max Tokens` field as a hard limit. Enable `Stop Generation` button for manual intervention. - **Proxy Layer:** Configure `proxy_read_timeout` and `proxy_send_timeout` to drop stalled connections. Return a custom error payload: `{"error": "Generation timed out. Reasoning incomplete."}` - **Custom Middleware:** Write a lightweight Python/Node.js stream processor that monitors token output in real-time. If repetition or stagnation thresholds are hit, inject a stop sequence or close the stream. - **Graceful Degradation:** When cutoffs trigger, return: `I couldn't complete the full reasoning within the allocated budget. Here's the partial analysis: [truncated output]. Please refine the prompt or increase the token limit for complete results.` **Operational Best Practices** - Log all cutoff events to identify recurring patterns. - Adjust budgets based on cutoff frequency. If >30% of runs hit hard limits, increase `n_predict` or optimize prompts. - Use streaming to provide partial value. Even truncated reasoning can be useful if structured properly. - Maintain a fallback prompt: `If you cannot complete the reasoning, output what you have under ### PARTIAL_ANSWER: and stop.` By aligning generation limits, stop sequences, prompt architecture, and monitoring tools, you transform unbounded reasoning into controlled, predictable output. The model will think efficiently, terminate cleanly, and deliver actionable answers without wasting tokens or stalling your home-lab workflow.