# Short Diagnosis

You are experiencing **Reasoning Exhaustion**. 

This occurs when a model (particularly "Reasoning" models like DeepSeek-R1 or models fine-tuned for Chain-of-Thought) is tasked with a complex problem and allocates its entire allowed "token budget" to the internal monologue (the "thinking" process) before it ever reaches the final output. Because the model sees the thinking process as part of the completion, if your `max_tokens` limit is reached while it is still "thinking," the generation is cut off, and you receive an empty or incomplete response.

# Why This Happens

To understand the fix, you have to understand how these models "think."

1.  **Chain of Thought (CoT) Architecture:** Modern reasoning models are trained to generate a long string of internal logic before producing a final answer. They aren't "thinking" in a separate hidden space; they are literally writing out their thoughts as text.
2.  **The Token Budget Conflict:** Every model has a `max_tokens` (or `max_new_tokens`) limit. This is a hard ceiling. If a model needs 1,000 tokens to "think" through a math problem and 200 tokens to write the answer, but your limit is set to 800, the model will hit the wall at token 800—right in the middle of its internal logic.
3.  **Proxy Interference:** When using a proxy (like LiteLLM, Ollama, or a custom FastAPI wrapper), there are often secondary timeouts. If the model takes 60 seconds to "think" before it starts writing the answer, the proxy might kill the connection because it thinks the model has hung.
4.  **Lack of "Stop" Signals:** If the model isn't explicitly told where the "thinking" ends and the "answering" begins, it may continue to spiral into deeper and deeper levels of abstraction, trying to be "perfect" rather than "concise."

# Settings To Check

Since you are using `llama.cpp` behind a proxy, you need to check settings at three different layers:

### 1. The `llama.cpp` / Inference Layer
*   **`max_tokens` (or `n_ctx` vs `max_new_tokens`):** Ensure your `max_new_tokens` is high enough. For reasoning models, 4096 is often the bare minimum. If you are doing complex coding or math, you may need 8192.
*   **`repeat_penalty`:** If the model gets stuck in a loop (repeating the same thought over and over), increase this slightly (e.g., 1.1 or 1.2).
*   **`temperature`:** For reasoning, keep this low (0.6 to 0.8). High temperature (1.0+) makes the model more likely to "wander" into infinite thought loops.

### 2. The Proxy Layer
*   **Timeout Settings:** Check your proxy's `request_timeout`. If the model is "thinking" for 2 minutes before outputting, but your proxy times out at 60 seconds, the connection will drop. Increase this to 300 seconds (5 minutes) for heavy reasoning tasks.
*   **Streaming:** Ensure `stream=True` is enabled. This allows you to see the "thinking" happen in real-time. If it's not streaming, you'll just see a spinning wheel for 2 minutes and then a "Connection Error."

### 3. The OpenWebUI Layer
*   **Model Parameters:** Check the "Model Settings" in OpenWebUI. Ensure the "Max Tokens" field isn't overriding your `llama.cpp` defaults with a small number (like 512).

# Prompt Design

The way you frame the request determines how much of the "budget" the model spends on thinking. You want to provide a "container" for the thoughts so the model knows when to stop.

### The "Bad" Prompt (Open-Ended)
> "Solve this complex physics problem: [Problem Description]"

**Why it fails:** The model has no instructions on how to structure its output. It may spend 3,000 tokens exploring every possible variable before it even tries to write the solution.

### The "Good" Prompt (Structured)
> "Solve the following physics problem. 
> 
> Please follow this format:
> <thought>
> [Your step-by-step reasoning, calculations, and internal monologue go here]
> </thought>
> 
> Final Answer:
> [Provide a concise, clear final answer here]
> 
> Problem: [Problem Description]"

**Why it works:** By providing tags like `<thought>`, you are giving the model a structural "home" for its reasoning. More importantly, it signals to the model that the "Final Answer" is a distinct section.

### The "Pro" Prompt (Constraint-Based)
> "Solve the following physics problem. 
> 
> Constraints:
> 1. Keep your internal reasoning (Chain of Thought) concise.
> 2. Do not exceed 500 words of reasoning.
> 3. Provide the final answer in a bolded summary at the end.
> 
> Problem: [Problem Description]"

# Benchmark Design

To debug this, you need to run a "Stress Test" to see where the breakdown occurs. Create a benchmark with three variations of the same hard problem (e.g., a complex logic puzzle):

1.  **Test A (Baseline):** No instructions. Just the problem.
2.  **Test B (High Budget):** Use the "Good" prompt above but set `max_tokens` to 8192.
3.  **Test C (Low Budget):** Use the "Good" prompt but set `max_tokens` to 1024.

**Metrics to track:**
*   **Reasoning-to-Answer Ratio:** (Tokens in `<thought>` / Total Tokens).
*   **Completion Success:** Did the model actually reach the "Final Answer" section?
*   **Time to First Token (TTFT):** How long did it take to start "thinking"?
*   **Time to Final Answer:** How long did it take to get to the actual result?

# What The Flight Recorder Should Show

If you are looking at your `llama.cpp` logs or your proxy's "Flight Recorder" (the raw output stream), here is what you are looking for:

*   **The "Stall":** If the logs show tokens being generated at a steady rate (e.g., 20 tokens/sec) but the text is just repeating the same logic, you have a **Repetition Loop**. (Fix: Increase `repeat_penalty`).
*   **The "Hard Cut":** If the logs suddenly stop mid-sentence without any "End of Sentence" token, you hit the **`max_tokens` limit**. (Fix: Increase `max_tokens`).
*   **The "Proxy Drop":** If the `llama.cpp` logs show the model is still generating, but the OpenWebUI interface says "Error," the **Proxy Timeout** is the culprit. (Fix: Increase proxy timeout).
*   **The "Thought Spiral":** If the model generates 2,000 tokens of "I should consider X... but wait, I should consider Y... but if I consider Y, then X is true... but wait...", it is **Over-Reasoning**. (Fix: Use a more concise prompt or a lower temperature).

# Practical Defaults

If you want a "set it and forget it" configuration for a home-lab running a reasoning model (like DeepSeek-R1-Distill-Llama-70B), use these:

| Parameter | Recommended Value | Reason |
| :--- | :--- | :--- |
| **Max Tokens** | 4096 (Minimum) / 8192 (Safe) | Gives the model room to "think" and "speak." |
| **Temperature** | 0.6 - 0.7 | Balances logic with enough variety to avoid loops. |
| **Repeat Penalty** | 1.1 | Prevents the model from getting stuck in a logic loop. |
| **Context Length** | 8192+ | Ensures the "thought" doesn't push the "answer" out of the window. |
| **Stop Sequences** | `</thought>`, `\n\nFinal Answer:` | Helps the proxy/UI know when a section is done. |

# When To Increase Budgets

Don't just set everything to "Maximum." High token budgets increase latency and VRAM pressure. Increase your `max_tokens` only when:

1.  **Coding Tasks:** You need the model to write a full script. The "thought" might be short, but the "answer" is long.
2.  **Complex Math/Logic:** You need the model to "work out" the problem. The "thought" will be very long, but the "answer" is short.
3.  **Multi-Step Planning:** If you ask the model to "Plan a 7-day itinerary for a trip to Japan including flights, hotels, and daily schedules," the output is naturally massive.

# When To Stop A Run

In a home-lab environment, you don't want a runaway process eating your GPU cycles for 20 minutes on a hallucination. Implement these "Stop" triggers:

1.  **The 1,000-Token Thought Rule:** If the model has generated more than 1,000 tokens inside the `<thought>` tags without producing a "Final Answer" header, it is likely spiraling.
2.  **The Repetition Trigger:** If the same 5-word phrase appears 3 times in a row, kill the generation.
3.  **The Time Limit:** Set a hard timeout at the proxy level (e.g., 180 seconds). If it hasn't finished by then, it's better to get a "Timeout" error than to let it run forever.
4.  **Semantic Drift:** If you notice the model has moved from "Solving the math problem" to "Discussing the history of mathematics," the prompt has failed to constrain the model. Stop the run and refine the prompt.

# Advanced Prompt Engineering: Few-Shot Reasoning

While the "Good" prompt structure provided earlier helps define the *shape* of the output, it doesn't always guarantee the *quality* of the logic. To truly master reasoning models, you should employ **Few-Shot Reasoning**. 

Most users provide a "Zero-Shot" prompt (no examples). For complex tasks, providing 2-3 examples of a "Thought -> Answer" pair is the single most effective way to stabilize a model's behavior. This teaches the model the specific "depth" of reasoning you expect.

### Example of a Few-Shot Reasoning Prompt:

> "You are a logic expert. Solve the following problems by thinking step-by-step.
> 
> Example 1:
> Problem: If John has 3 apples and gives 1 to Mary, then buys 2 more, how many does he have?
> <thought>
> 1. Initial count: 3 apples.
> 2. Action: Gives 1 to Mary. Calculation: 3 - 1 = 2.
> 3. Action: Buys 2 more. Calculation: 2 + 2 = 4.
> 4. Final count: 4.
> </thought>
> Final Answer: John has 4 apples.
> 
> Example 2:
> Problem: A train leaves Station A at 60mph. Another leaves Station B at 40mph. They are 200 miles apart. When do they meet?
> <thought>
> 1. Relative speed: 60mph + 40mph = 100mph.
> 2. Distance to cover: 200 miles.
> 3. Time = Distance / Speed. Calculation: 200 / 100 = 2 hours.
> 4. Conclusion: They meet in 2 hours.
> </thought>
> Final Answer: They will meet in 2 hours.
> 
> Problem: [INSERT YOUR COMPLEX PROBLEM HERE]
> <thought>"

By ending the prompt with `<thought>`, you force the model to begin its internal monologue immediately in the correct format.

# The Impact of Quantization on Logic

Since you are running `llama.cpp`, you are likely using quantized models (GGUF format). It is important to understand that **reasoning is more sensitive to quantization than standard chat.**

1.  **Logic Collapse:** In a standard chat model, a 4-bit quantization (Q4_K_M) might cause a slight loss in vocabulary nuance. In a reasoning model, 4-bit quantization can cause "logic collapse," where the model understands the individual steps but loses the ability to connect Step A to Step B. This often manifests as the "infinite loop" or "thought spiral" mentioned earlier.
2.  **The "Sweet Spot":** For reasoning tasks, if your VRAM allows it, try to stay at **Q6_K or Q8_0**. The jump from 4-bit to 6-bit often provides a significant stability boost in multi-step logic, even if the "knowledge" of the model doesn't change much.
3.  **Contextual Drift:** Lower bitrates make it harder for the model to maintain "attention" over long sequences. If the model's "thought" process is very long, a heavily quantized model may "forget" the original constraints of the prompt by the time it reaches the end of its internal monologue.

# Context Window & KV Cache Dynamics

One of the most overlooked reasons for "Reasoning Exhaustion" is the **KV Cache** and the **Context Window**.

When a model "thinks," every token it generates is added to its active memory (the KV Cache). If you have a 32k context window and the model spends 20,000 tokens "thinking," you only have 12,000 tokens left for the actual answer and the history of the conversation.

*   **The "Lost in the Middle" Phenomenon:** Research shows that LLMs are less likely to recall information located in the middle of a very long context. If the model's "thought" block is massive, the original instructions (at the very beginning of the prompt) can become "blurry" to the model's attention mechanism.
*   **The Solution:** If you find the model is "drifting" (starting to talk about irrelevant things after a long thought), you need to either:
    *   Shorten the thought process via prompting (see "Prompt Design").
    *   Use a model with a larger native context window (e.g., 128k).
    *   Use a "System Prompt" that is very concise, as System Prompts often receive higher attention weights than user prompts.

# Deep Dive into llama.cpp Optimization

To ensure your `llama.cpp` backend is handling these long reasoning chains correctly, check these specific flags:

*   **`--n-gpu-layers` (or `-ngl`):** Ensure you are offloading as many layers as possible to your GPU. If the model is "thinking" on the CPU, the tokens per second (TPS) will drop significantly. If the TPS is too low, your proxy might time out before the model finishes a single logical step.
*   **`--ctx-size`:** Ensure this matches your intended use. If you set this too low, the model will literally run out of "room" to think.
*   **`--flash-attn`:** If your hardware supports it (e.g., NVIDIA Ampere or newer), always enable Flash Attention. It significantly reduces the memory overhead of the KV Cache, allowing for longer "thought" processes without crashing your VRAM.
*   **`--threads`:** For CPU offloading, ensure this matches your physical cores (not logical threads) to prevent thread contention, which can cause stuttering in token generation.

# Proxy Layer Deep Dive (LiteLLM / Ollama / Custom)

When using a proxy, the "Reasoning Exhaustion" often looks like a network error, but it's actually a configuration mismatch.

### 1. The Timeout Chain
Your request travels: `OpenWebUI` -> `Proxy` -> `llama.cpp`.
If `llama.cpp` takes 90 seconds to generate a "thought" block, but your Proxy has a `timeout` of 60 seconds, the Proxy will send a `504 Gateway Timeout`. 

**Fix:** Set your proxy timeout to at least 300 seconds.

### 2. Streaming vs. Non-Streaming
If you are not using `stream=True`, the proxy waits for the *entire* response to be finished before sending it to OpenWebUI. For a reasoning model that might generate 2,000 tokens of thought, this could take a minute of "silence."

**Fix:** Always enable streaming. This allows you to see the model's "thought" process in real-time. If you see it moving, you know the model hasn't crashed; it's just working.

### 3. Request Body Limits
Some proxies have a maximum request body size. While this usually affects inputs, some also limit the *response* size. Ensure your proxy isn't capping the output at 1,024 or 2,048 tokens.

# Model Selection Strategy

Not all "Reasoning" models are created equal. Your choice of model dictates how much "budget" you need to allocate.

*   **DeepSeek-R1-Distill-Llama-70B:** This is currently a gold standard for home-lab reasoning. It is highly structured. It tends to produce very long, high-quality thought blocks. You **must** give this model a high `max_tokens` (8192+).
*   **DeepSeek-R1-Distill-Qwen-32B:** A great middle-ground. It is faster and often more concise in its reasoning than the 70B version.
*   **Llama-3-70B-Instruct (Non-Reasoning):** If you use a standard instruction model for a complex logic task, it will try to answer immediately. It won't "think" as much, but it is much more likely to give a wrong answer because it didn't "work out" the steps first.
*   **Small Models (8B - 14B):** These models often struggle with long reasoning chains. They are prone to "looping" (repeating the same logic over and over). If you use these, you must use very strict prompting to keep them on track.

# The "Reasoning Debugging" Flowchart

When a model fails to give an answer, follow this diagnostic path:

1.  **Did the UI show an error message?**
    *   *Yes (Timeout/504):* Increase Proxy Timeout.
    *   *Yes (Connection Error):* Check if `llama.cpp` crashed due to Out of Memory (OOM).
2.  **Did the UI show a "cut off" sentence?**
    *   *Yes:* You hit `max_tokens`. Increase the limit in OpenWebUI/Proxy.
3.  **Did the model just stop after a long period of "thinking"?**
    *   *Yes:* The model likely hit a "Stop Sequence" or a logic loop. Check your `repeat_penalty` and prompt structure.
4.  **Did the model give a wrong answer but "thought" correctly?**
    *   *Yes:* This is a "Knowledge" or "Quantization" issue. Try a higher-bit quantization or a larger model.
5.  **Did the model give a wrong answer and "thought" incorrectly?**
    *   *Yes:* This is a "Prompting" issue. Use Few-Shot examples to guide the logic.

# Summary Cheat Sheet for Home-Lab Users

| Problem | Likely Culprit | Immediate Fix |
| :--- | :--- | :--- |
| **Empty Response / Error** | Proxy Timeout | Increase Proxy Timeout to 300s |
| **Response cuts off mid-sentence** | `max_tokens` limit | Increase `max_tokens` to 8192 |
| **Model repeats the same phrase** | Repetition Loop | Increase `repeat_penalty` to 1.15 |
| **Model "drifts" into nonsense** | Context Window / Drift | Use a more concise prompt; check `ctx_size` |
| **Model is too slow to be usable** | GPU Offloading | Increase `-ngl` (number of layers) |
| **Model gives wrong logic** | Quantization / Model Size | Move from Q4_K_M to Q6_K or a larger model |
| **Model won't start "thinking"** | Prompt Structure | End prompt with `<thought>` tag |

By implementing these structural changes—specifically moving to a structured `<thought>` prompt, increasing your token budgets, and ensuring your proxy isn't killing the connection—you will turn your "smart but silent" model into a reliable reasoning engine.