# Short Diagnosis

Your model is not broken; it is trapped in an autoregressive generation loop. When you prompt a local LLM to "think," "reason," or "explain thoroughly" without a hard termination signal, the model interprets the instruction as a permission to continue generating tokens indefinitely. Because LLMs predict the next token purely based on statistical likelihood, they will keep producing reasoning steps, clarifications, or meta-commentary until they hit a hard limit (context window full, `max_tokens` reached, or a stop sequence triggered). In your setup, the combination of OpenWebUI's streaming interface, llama.cpp's proxy layer, and permissive generation settings is allowing this loop to consume your entire token budget before a final answer is ever emitted. The fix requires aligning prompt structure, sampling parameters, and proxy limits so the model knows exactly when to stop reasoning and deliver the conclusion.

# Why This Happens

Large language models generate text autoregressively: they predict one token at a time, conditioning on all previous tokens. This architecture has three inherent properties that directly cause the "thinking loop" behavior you're seeing:

1. **No Native Stop Mechanism**: LLMs do not understand "I'm done thinking." They only understand patterns. If your prompt encourages open-ended reasoning without explicit boundaries, the model's probability distribution will keep favoring continuation tokens (`\n`, `next step`, `furthermore`, `in summary`, etc.).
2. **Context Window Pressure**: As the conversation grows, the model must attend to prior turns. If the system prompt, history, and current query consume most of the context window, the model may get stuck trying to reconcile conflicting instructions or re-reading its own reasoning, which degrades output quality and prolongs generation.
3. **Proxy & UI Layering**: OpenWebUI sits behind a proxy that forwards requests to llama.cpp. Each layer can silently alter or ignore stop sequences, temperature, or token limits. If the proxy doesn't enforce a hard `num_predict` or `max_tokens`, or if it strips custom stop sequences, the model will run until the underlying server times out or exhausts VRAM.

Additionally, sampling parameters play a critical role. High temperature (`>0.8`) increases randomness, which can cause the model to wander into tangential reasoning. Low `top_p` or `top_k` can force the model into narrow, repetitive loops. Without a well-crafted prompt structure and explicit termination signals, the model defaults to "keep going" behavior.

# Settings To Check

You need to audit three layers: OpenWebUI UI settings, proxy configuration, and llama.cpp inference parameters. Here's what to verify and adjust:

**OpenWebUI Settings**
- `Max Tokens`: Set to a realistic ceiling (e.g., 1024–2048). This is your primary safety net.
- `Stream`: Keep enabled for UX, but ensure the proxy respects token limits. Streaming doesn't change generation behavior, but it can mask timeouts if the proxy doesn't enforce hard stops.
- `System Prompt`: Keep it concise. Overly long system prompts consume context and can confuse the model's expected behavior. Use a clear role + task definition + output format constraint.
- `Stop Sequences`: Add explicit delimiters. Common choices: `\n\n`, `###`, `Answer:`, `</answer>`. Test which one reliably terminates your model's reasoning loop.

**Proxy Configuration**
- `num_predict` / `max_new_tokens`: Ensure the proxy forwards this value to llama.cpp. If the proxy has its own token limit, it must match or be slightly lower than OpenWebUI's setting.
- `timeout`: Set a reasonable request timeout (e.g., 30–60 seconds). Prevents the proxy from hanging indefinitely while the model loops.
- `stop_sequences` passthrough: Verify the proxy isn't stripping or escaping your stop sequences. Some proxies normalize whitespace or remove `\n\n` before forwarding.
- `context_length` / `n_ctx`: Match this to your model's native context (e.g., 4096, 8192). Don't exceed your VRAM capacity, or enable sliding window/flash attention if supported.

**llama.cpp Inference Parameters**
- `temperature`: 0.5–0.7 for balanced reasoning. Lower for deterministic tasks, higher for creative writing.
- `top_p`: 0.9–0.95. Keeps sampling diverse but focused.
- `repeat_penalty`: 1.1–1.2. Prevents the model from getting stuck in repetitive reasoning loops.
- `logit_bias` (optional): Penalize continuation tokens if you notice the model favoring phrases like "next," "further," or "in conclusion" excessively.
- `--ctx-size` / `--n-predict`: Must align with proxy and UI settings. Mismatches cause silent truncation or unexpected loops.

**Verification Workflow**
1. Run a simple prompt with `max_tokens=128` and `temperature=0.5`. If it stops cleanly, the issue is parameter drift or missing stop sequences.
2. Check proxy logs for token counts and stop sequence matches.
3. Gradually increase `max_tokens` while monitoring completion rate. Note the threshold where loops begin.

# Prompt Design

Prompt structure is the most effective lever for preventing reasoning loops. You need to explicitly define the reasoning boundary, force structured output, and provide clear termination signals. Below are concrete examples and design principles.

**Bad Prompt Shapes (Encourage Loops)**
- `"Explain how transformers work. Think step by step. What are the limitations?"`
  - *Why it fails*: Open-ended, encourages unbounded reasoning, no output format, no stop signal.
- `"Analyze this code and tell me everything you think about it."`
  - *Why it fails*: Vague scope, invites meta-commentary, no length constraint.
- `"Reason thoroughly about the ethical implications of AI alignment, then give your final thoughts."`
  - *Why it fails*: "Reason thoroughly" is a strong continuation trigger. "Final thoughts" is too late in the generation process to reliably stop the model.

**Good Prompt Shapes (Force Termination)**
- `"You are a technical writer. Explain how transformers work in exactly three paragraphs. Paragraph 1: core architecture. Paragraph 2: attention mechanism. Paragraph 3: limitations. End with a single-sentence summary. Do not add extra commentary."`
  - *Why it works*: Explicit structure, paragraph limits, clear ending instruction, no open-ended reasoning permission.
- `"Analyze the following code. Output exactly two sections: <issues> and <fixes>. Under <issues>, list at most three bugs. Under <fixes>, provide corrected code blocks. Stop after </fixes>."`
  - *Why it works*: XML-style delimiters, item limits, explicit stop tag.
- `"Reason step-by-step inside <reasoning> tags. Then output the final answer inside <answer> tags. Stop immediately after </answer>."`
  - *Why it works*: Separates reasoning from conclusion, uses stop sequences that can be enforced in settings, prevents post-answer rambling.

**Prompt Engineering Principles**
1. **Role + Task + Format + Constraint**: Always define who the model is, what it must do, how to structure it, and what to avoid.
2. **Explicit Delimiters**: Use markdown, XML, or custom tags to segment reasoning and answers. These also work well with `stop` sequences.
3. **Length Boundaries**: Specify paragraph counts, bullet limits, or token/word estimates. LLMs respond well to structural constraints.
4. **Negative Instructions**: Explicitly state what not to do ("Do not add examples," "Do not summarize," "Do not continue reasoning after the answer").
5. **Few-Shot Anchors**: Provide 1–2 examples of the exact input/output shape you want. This dramatically reduces open-ended generation.

**Example: Chain-of-Thought with Hard Stop**
```
You are a debugging assistant. Follow these steps:
1. Identify the error in the provided code.
2. Explain the root cause in one sentence.
3. Provide the corrected code.
Output format:
<error>...</error>
<cause>...</cause>
<corrected_code>...</corrected_code>
Do not add any text outside these tags. Stop immediately after </corrected_code>.
```

# Benchmark Design

A structured benchmark validates your configuration, identifies breaking points, and ensures reproducibility. For a home-lab setup, you don't need enterprise-grade evaluation suites; you need a focused, repeatable test harness that measures completion reliability, token efficiency, and answer quality.

**Test Suite Construction**
- Create 30–50 prompts spanning:
  - Simple Q&A (factual, direct)
  - Multi-step reasoning (math, logic, debugging)
  - Code generation (functions, scripts)
  - Creative/structured output (summaries, JSON, markdown)
- Ensure prompts are deterministic (no random elements) and have clear ground-truth or evaluation criteria.

**Variable Matrix**
Test combinations of:
- `max_tokens`: 256, 512, 1024, 2048
- `temperature`: 0.3, 0.6, 0.8
- `stop` sequences: none, `\n\n`, custom tags
- `repeat_penalty`: 1.0, 1.15, 1.3

**Metrics to Track**
- `completion_rate`: % of prompts that produce a final answer before hitting limits
- `avg_tokens_used`: Mean tokens generated per prompt
- `loop_detection`: Count of prompts where token generation rate drops below threshold or repetition penalty triggers
- `quality_score`: Manual or LLM-as-judge rating (1–5) based on accuracy, structure, and adherence to constraints
- `time_to_first_token`: Proxy/UI latency indicator
- `context_overflow_rate`: % of prompts failing due to context window limits

**Execution Workflow**
1. Run each prompt sequentially with a fixed seed (if supported) or deterministic sampling.
2. Log all parameters, outputs, token counts, and timestamps to JSONL.
3. Automate parsing to calculate metrics. Use a simple Python script or OpenWebUI's export feature.
4. Identify the "sweet spot": highest completion rate with acceptable quality and reasonable token usage.
5. Iterate: adjust `stop` sequences, temperature, or prompt structure based on failure modes.

**Home-Lab Practicalities**
- Run benchmarks during low-load periods to avoid VRAM/CPU contention skewing results.
- Use a dedicated test account in OpenWebUI to avoid polluting conversation history.
- If your proxy supports request tracing, enable it to capture exact token counts and stop sequence matches.
- Re-run benchmarks after any configuration change to ensure stability.

# What The Flight Recorder Should Show

Your "flight recorder" refers to detailed generation logs, proxy traces, and inference metrics. When properly configured, it should reveal exactly where the model stalls, loops, or terminates. Here's what to expect and how to interpret it:

**Expected Data Points**
- `tokens_generated`: Total output tokens per request
- `tokens_per_second`: Generation throughput (indicates VRAM/CPU bottleneck)
- `context_window_usage`: % of `n_ctx` consumed before generation starts
- `stop_sequence_match`: Boolean flag indicating if a stop sequence was triggered
- `temperature` / `top_p` / `repeat_penalty`: Active sampling parameters
- `proxy_latency_ms`: Network + processing overhead
- `error_code`: e.g., `context_overflow`, `timeout`, `max_tokens_reached`, `none`

**Interpretation Guide**
- **High token usage + low completion rate**: Missing or ineffective stop sequences. The model is reasoning indefinitely. Add explicit delimiters and enforce them in proxy/UI settings.
- **Spiking latency + context >80%**: VRAM bottleneck. The model is struggling to attend to long histories. Reduce `n_ctx`, enable sliding window, or prune conversation history.
- **Repetitive token patterns**: `repeat_penalty` too low or `temperature` too high. Increase `repeat_penalty` to 1.15–1.2, lower temperature for deterministic tasks.
- **Proxy timeouts without token limit hits**: Proxy isn't forwarding `max_tokens` or has its own timeout. Align proxy limits with llama.cpp settings.
- **Frequent `context_overflow`**: `n_ctx` too low for your use case. Increase if VRAM allows, or implement prompt chunking.
- **Clean stops at `max_tokens`**: Working as intended. Adjust `max_tokens` downward if answers are truncated, upward if reasoning is cut off prematurely.

**Logging Setup**
- Enable llama.cpp's `--log-level debug` or `--log-prefix` to capture generation traces.
- Use OpenWebUI's debug mode to inspect request payloads and responses.
- If using a custom proxy (e.g., Nginx, Traefik, or a Python wrapper), log JSONL traces with timestamps, token counts, and stop sequence matches.
- Parse logs with a simple script to generate a dashboard of completion rates, token efficiency, and failure modes.

# Practical Defaults

For a typical home-lab setup (8B–13B model, 16–24GB VRAM, OpenWebUI + llama.cpp proxy), these defaults balance reliability, quality, and token efficiency. They are designed to prevent reasoning loops while preserving useful output.

**Core Inference Parameters**
- `max_tokens`: 1024
- `temperature`: 0.6
- `top_p`: 0.9
- `stop`: `["\n\n", "###", "Answer:"]`
- `repeat_penalty`: 1.15
- `n_ctx`: 4096 (match model's native context)
- `stream`: true

**Proxy Configuration**
- `num_predict`: 1024 (matches OpenWebUI)
- `timeout`: 45 seconds
- `context_length`: 4096
- `stop_sequences_passthrough`: enabled
- `log_level`: info (debug only when troubleshooting)

**OpenWebUI Settings**
- `System Prompt`: Concise role + task + format constraint (max 150 tokens)
- `Max Tokens`: 1024
- `Stop Sequences`: `\n\n`, `###`, `Answer:`
- `Stream`: enabled
- `History Pruning`: enabled (keep last 4–6 turns to preserve context headroom)

**Why These Work**
- `max_tokens=1024` provides enough room for structured reasoning and answers without encouraging loops.
- `temperature=0.6` balances creativity and focus. Lower for code/facts, higher for creative writing.
- `stop` sequences act as hard boundaries. Test which delimiter your model responds to most reliably.
- `repeat_penalty=1.15` prevents repetitive reasoning without killing fluency.
- `n_ctx=4096` fits most 8B–13B models comfortably on 16–24GB VRAM. Adjust based on your hardware.
- History pruning ensures the model isn't drowning in prior turns, which often triggers re-reasoning loops.

**Tuning Notes**
- If answers are consistently truncated, increase `max_tokens` to 1536 and add explicit length constraints to prompts.
- If the model still loops, tighten `stop` sequences or lower `temperature` to 0.4–0.5.
- For code-heavy workloads, add `stop: ["\n\n```", "```"]` to terminate code blocks cleanly.

# When To Increase Budgets

Higher token budgets, temperature, or context lengths are justified only when the task complexity demands it. Increasing budgets without fixing prompt structure or stop sequences will only delay the loop, not solve it.

**Complex Multi-Step Reasoning**
- Increase `max_tokens` to 2048–4096.
- Use structured reasoning tags: `<reasoning>...</reasoning><answer>...</answer>`.
- Set `stop` to `["</answer>", "\n\n"]`.
- Keep `temperature` at 0.5–0.6 to maintain logical consistency.

**Long Document Analysis**
- Increase `n_ctx` to 8192 (if VRAM allows) or use chunking with a summarization pipeline.
- Set `top_p` to 0.95 for broader context attention.
- Add explicit extraction constraints: "Output exactly 5 bullet points. Do not summarize the document."
- Monitor context overflow; if frequent, implement sliding window or progressive summarization.

**Code Generation & Debugging**
- Increase `max_tokens` to 2048.
- Lower `temperature` to 0.2–0.4 for deterministic syntax.
- Add `stop` sequences: `["\n\n```", "```", "Answer:"]`.
- Use few-shot examples of the exact code structure you expect.

**Creative Writing & Brainstorming**
- Increase `max_tokens` to 1536–2048.
- Raise `temperature` to 0.7–0.9 for diversity.
- Add structural constraints: "Write exactly 3 paragraphs. End with a concluding sentence."
- Monitor for tangential rambling; if present, tighten `stop` sequences or lower temperature.

**Critical Rule**: Always pair budget increases with explicit prompt boundaries and stop sequences. A higher `max_tokens` without structural constraints is just a longer leash for the same loop.

# When To Stop A Run

Knowing when to abort a generation is as important as configuring it. Unchecked runs waste tokens, degrade performance, and can destabilize your home-lab environment.

**Hard Limits (Automatic)**
- `max_tokens` reached: Generation terminated by design.
- Context window full: `n_ctx` exhausted. Implement history pruning or chunking.
- Proxy timeout: Request exceeded configured timeout (e.g., 45–60 seconds).
- Error codes: `context_overflow`, `gpu_memory_limit`, `invalid_stop_sequence`.

**Soft Limits (Watchdog/Manual)**
- No new tokens generated in >5 seconds: Indicates a stall or VRAM bottleneck. Check GPU utilization and context length.
- Repetition detected: Model enters a loop (e.g., repeating phrases, backtracking). Increase `repeat_penalty` or lower `temperature`.
- Quality degradation: Output shifts from structured to rambling, hallucinates, or violates constraints. Abort and refine prompt.
- Resource saturation: CPU/RAM/VRAM >90% for extended periods. Pause other workloads, reduce `n_ctx`, or switch to CPU offloading if necessary.

**Operational Triggers**
- User abort: OpenWebUI's cancel button or proxy interrupt.
- Batch queue overflow: If running automated benchmarks, pause when system load exceeds safe thresholds.
- Log anomalies: Sudden spikes in token generation rate, frequent stop sequence misses, or proxy errors.

**Implementation Tips**
- Use OpenWebUI's built-in cancel function for immediate aborts.
- Configure proxy timeouts to match your hardware capabilities (e.g., 30s for 8B, 60s for 13B).
- Implement a watchdog script that monitors token generation rate and aborts if it drops below a threshold (e.g., <2 tokens/sec for >10 seconds).
- Log all aborts with reasons to refine future configurations.

By aligning prompt structure, sampling parameters, proxy limits, and abort criteria, you transform the "thinking loop" from a frustrating bug into a predictable, controllable workflow. Test systematically, log rigorously, and adjust incrementally. Your home-lab setup will become reliable, efficient, and ready for production-like workloads.