# Quick Diagnosis

The inconsistency you are feeling—the "sometimes slow" first token and the "jumping" GPU memory—is the hallmark of a system struggling with **KV Cache management** and **Prompt Processing (Pre-fill) overhead**. When you use `llama.cpp`, the performance isn't a flat line; it is a dynamic curve influenced by the length of your prompt, the size of the context window, and how the memory is being allocated.

To get your bearings immediately, we need to distinguish between three distinct metrics:

1.  **TTFT (Time to First Token):** This is the "pre-fill" phase. If this is slow, your bottleneck is either the sheer size of the prompt (too many tokens for the GPU to process in one pass) or your `n_batch` / `n_ubatch` settings are too low.
2.  **TPS (Tokens Per Second):** This is the "decoding" phase. If this is slow, your bottleneck is your memory bandwidth (the speed at which the weights move from VRAM to the compute cores).
3.  **VRAM Volatility:** If your memory "jumps," it usually means `llama.cpp` is dynamically allocating the KV Cache or you are hitting a "context overflow" where the system is forced to swap memory or re-allocate buffers.

**The "Quick Fix" Checklist before you start your weekend deep-dive:**
*   **Check `n_ctx`:** Ensure your context window is explicitly set. If it's too small, the model will "forget" or slow down as it hits the limit. If it's too large, it might be reserving too much VRAM, leaving no room for the weights.
*   **Check `n_batch`:** For modern GPUs, `n_batch` should usually be 512 or 2048. If it's 128, your pre-fill (TTFT) will be agonizingly slow.
*   **Check `flash_attn`:** Ensure Flash Attention is enabled. It significantly reduces the memory footprint of the KV cache and speeds up pre-fill.

---

# Data Collection Plan

To stop "guessing" and start "knowing," you need a structured data collection pipeline. You have the right tools (Flight Recorder, ServerTop, SQLite), but you need a standardized protocol.

### 1. The Environment Baseline
Before running any tests, record your hardware state.
*   **GPU Temperature:** Use `nvidia-smi -l 1` or `nvtop`.
*   **Memory Clock:** Ensure your GPU isn't thermal throttling.
*   **System Load:** Ensure no background processes (like a browser with 50 tabs) are stealing PCIe bandwidth or CPU cycles.

### 2. The "Control" Variables
To make your data scientific, you must hold these constant during every test:
*   **Model File:** Use the exact same `.gguf` file.
*   **Quantization:** Do not switch between Q4_K_M and Q8_0 during testing.
*   **Temperature/Top_P:** Keep these at default (e.g., 0.7, 0.9).
*   **Seed:** Use a fixed seed (e.g., 42) so the model produces the same output for the same prompt.

### 3. The "Independent" Variables (What you will change)
You will systematically vary:
*   **Prompt Length:** 128, 512, 1024, 2048, 4096 tokens.
*   **MTP Status:** Enabled vs. Disabled.
*   **Context Window:** 4096 vs. 8192 vs. 16384.

### 4. Logging Protocol
Use your Flight Recorder proxy to capture every request. Every entry in your SQLite database should ideally contain:
*   `timestamp`
*   `prompt_tokens`
*   `completion_tokens`
*   `ttft_ms` (Time to First Token)
*   `tps` (Tokens Per Second)
*   `vram_used_mb`
*   `mtp_enabled` (Boolean)

**Example Curl for manual testing (to ensure clean logs):**
```bash
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your_model_name",
    "prompt": "Write a detailed technical explanation of quantum entanglement...",
    "max_tokens": 500,
    "temperature": 0.7,
    "stream": false
  }'
```

---

# MTP Comparison Plan

Multi-Token Prediction (MTP) is designed to predict multiple future tokens simultaneously, theoretically increasing throughput. However, it can introduce overhead or "hallucination" in the prediction path. To see if it's actually helping *you*, you need a side-by-side comparison.

### The "A/B" Test Methodology
Run the following test 5 times for each configuration to account for variance.

**Configuration A: Standard (No MTP)**
*   Run a 500-token prompt.
*   Record `Total Time` and `Total Tokens Generated`.
*   Calculate `Effective TPS = Total Tokens / Total Time`.

**Configuration B: MTP Enabled**
*   Run the *exact same* 500-token prompt.
*   Record `Total Time` and `Total Tokens Generated`.
*   Calculate `Effective TPS`.

### What to look for:
1.  **The Throughput Gain:** If MTP is working, your `Effective TPS` should be higher in Configuration B.
2.  **The Quality Penalty:** Compare the outputs. Does MTP cause the model to "drift" or lose coherence faster?
3.  **The VRAM Overhead:** Check if MTP causes a significant jump in VRAM. MTP requires extra weights/heads; if your VRAM is near capacity, MTP might trigger swapping, which will destroy your TPS.

---

# Reasoning Budget Plan

"Reasoning" in local LLMs often refers to the "Pre-fill" phase where the model processes the instructions. If you are doing agentic work, the "Reasoning Budget" is the time the model spends "thinking" before it starts outputting the final answer.

### Measuring the "Thinking" Cost
To analyze this, you need to separate **Prompt Processing Time** from **Generation Time**.

1.  **The "Static Prompt" Test:**
    *   Provide a very long, complex instruction (e.g., a 1,000-word coding task).
    *   Measure the time until the first token appears. This is your **Reasoning Budget**.
    *   If this time is > 5 seconds, your `n_batch` is likely too low, or your GPU is struggling with the attention matrix size.

2.  **The "Chain of Thought" (CoT) Test:**
    *   Ask the model to "Think step-by-step."
    *   Measure the TPS during the "Thinking" phase vs. the "Final Answer" phase.
    *   Often, the TPS remains constant, but the *number of tokens* generated increases. Your "Budget" here is the total tokens allowed for reasoning before the final output.

**Goal:** You want to find the "Sweet Spot" where the Reasoning Budget (TTFT) is acceptable (under 2-3 seconds for most tasks) while maintaining a high TPS for the final output.

---

# Long Output Test Plan

This is where most users see their performance "fall off a cliff." As the KV Cache fills up, the memory management becomes more complex.

### The "Context Stress Test"
Run a series of prompts that generate long outputs (e.g., 1,000+ tokens) at different context lengths.

*   **Test 1:** `n_ctx` = 2048. Generate 1,000 tokens.
*   **Test 2:** `n_ctx` = 4096. Generate 1,000 tokens.
*   **Test 3:** `n_ctx` = 8192. Generate 1,000 tokens.
*   **Test 4:** `n_ctx` = 16384 (or your max). Generate 1,000 tokens.

### Data Points to Capture:
*   **TPS Decay:** Does the TPS drop significantly as you move from 2048 to 8192? If so, your GPU's memory bandwidth is the bottleneck.
*   **VRAM Ceiling:** At which context length does the VRAM usage hit 90%? This is your "Safe Zone."
*   **The "Cliff":** Identify the point where TPS drops by more than 50%. This is the limit of your current hardware/quantization setup.

---

# SQLite Queries

Assuming your Flight Recorder or a custom script logs to a table named `inference_logs`, use these queries to extract your insights.

### 1. Average TTFT vs. Prompt Length
This helps you see how much "Reasoning Budget" you are spending as prompts get longer.
```sql
SELECT 
    prompt_tokens, 
    AVG(ttft_ms) as avg_ttft, 
    COUNT(*) as sample_count
FROM inference_logs
GROUP BY prompt_tokens
ORDER BY prompt_tokens ASC;
```

### 2. TPS Variance Analysis
Identify if the model is "inconsistent." A high standard deviation in TPS for the same prompt length indicates a problem (like thermal throttling or background noise).
```sql
SELECT 
    prompt_tokens, 
    AVG(tps) as avg_tps, 
    (MAX(tps) - MIN(tps)) as tps_spread
FROM inference_logs
GROUP BY prompt_tokens;
```

### 3. MTP Efficiency Comparison
Compare the "Effective TPS" of MTP vs. Non-MTP.
```sql
SELECT 
    mtp_enabled, 
    AVG(tps) as avg_tps, 
    AVG(ttft_ms) as avg_ttft
FROM inference_logs
GROUP BY mtp_enabled;
```

### 4. VRAM Usage vs. Context
Find the "Safe Zone" for your VRAM.
```sql
SELECT 
    n_ctx, 
    MAX(vram_used_mb) as peak_vram,
    AVG(tps) as avg_tps
FROM inference_logs
GROUP BY n_ctx;
```

---

# Dashboard Screenshots To Capture

When you present your findings (or just for your own records), capture these four specific charts. If you don't have a dashboard tool, you can plot these in Excel/Google Sheets using your SQLite data.

1.  **The "Context Cliff" Chart:**
    *   *X-Axis:* Context Length (`n_ctx`)
    *   *Y-Axis:* Tokens Per Second (TPS)
    *   *What to look for:* A steady line that suddenly drops. This tells you your hardware's maximum efficient context.

2.  **The "Prompt Penalty" Chart:**
    *   *X-Axis:* Prompt Tokens
    *   *Y-Axis:* Time to First Token (TTFT)
    *   *What to look for:* A linear increase. If it's exponential, your `n_batch` is too low.

3.  **The "MTP Delta" Chart:**
    *   *X-Axis:* Prompt Length
    *   *Y-Axis:* TPS
    *   *Series:* Two lines (MTP Enabled vs. MTP Disabled).
    *   *What to look for:* Does the MTP line stay above the Standard line? If they overlap, MTP is providing no benefit to your specific use case.

4.  **The "VRAM Heatmap":**
    *   *X-Axis:* Prompt Length
    *   *Y-Axis:* Output Length
    *   *Color/Z-Axis:* VRAM Usage.
    *   *What to look for:* Identify the "Red Zone" where VRAM exceeds 90%.

---

# Weekend Checklist

### Saturday: The Baseline & MTP Phase
*   [ ] **Morning:** Clear all caches. Run 5 "Standard" tests (No MTP) with 128, 512, 1024, 2048, and 4096 prompt tokens. Record all metrics.
*   [ ] **Afternoon:** Run 5 "MTP" tests with the same prompt lengths. Record all metrics.
*   [ ] **Evening:** Run the "Reasoning Budget" test. Take a 1,000-word technical prompt and measure TTFT. Compare this against your "acceptable" threshold (e.g., < 3 seconds).

### Sunday: The Stress & Context Phase
*   [ ] **Morning:** Run the "Long Output Test." Test `n_ctx` at 4096, 8192, and 16384. Generate 1,000 tokens for each.
*   [ ] **Afternoon:** Run "Agentic Work" simulations. Give the model a multi-step task (e.g., "Plan a trip, then write the itinerary, then create a budget"). Measure the total time and TPS for the entire chain.
*   [ ] **Evening:** Run the SQLite queries. Generate your charts. Compare your "Actual" results against your "Desired" performance.

---

# Interpreting Results

Once you have your data, use this matrix to decide how to configure your model for different tasks:

| Task Type | Priority Metric | Target Profile | Optimization Strategy |
| :--- | :--- | :--- | :--- |
| **Coding** | High TPS & High Context | **High Context / High Batch** | Use `n_ctx` of 8k+, `n_batch` of 2048. MTP is usually helpful here for speed. |
| **RAG** | Low TTFT | **Fast Pre-fill** | Focus on `n_batch` and `flash_attn`. You need the model to "read" the retrieved docs quickly. |
| **Agentic** | Low Latency / Consistency | **Balanced** | Keep `n_ctx` moderate (4k-8k). You need reliable, fast responses for every "step" the agent takes. |
| **Chat** | High TPS | **Max Throughput** | Use MTP. You want the conversation to feel fluid and instantaneous. |
| **Creative** | High Quality / Long Output | **Max Context** | Prioritize `n_ctx` and `n_ubatch`. MTP might be less important than the model's ability to maintain a long narrative. |

**How to know if MTP is "Fooling" you:**
If your **Effective TPS** (Total Tokens / Total Time) is higher with MTP, but your **TTFT** is significantly slower, MTP is helping the *generation* but hurting the *start*. For Chat, this is usually a win. For RAG, this is a loss.

---

# Common Mistakes

1.  **The "Context Overflow" Trap:**
    Setting `n_ctx` to 32,768 when your GPU only has 24GB of VRAM. `llama.cpp` might try to allocate the KV cache, fail, and then fall back to system RAM (slow) or crash. **Rule:** Always leave at least 2-4GB of VRAM free for the OS and overhead.

2.  **Ignoring `n_ubatch`:**
    Many users set `n_batch` but forget `n_ubatch`. `n_ubatch` controls how many tokens are processed in a single GPU kernel call. For modern architectures (Ada Lovelace, Hopper), `n_ubatch` should be large (e.g., 512) to keep the CUDA cores saturated.

3.  **Over-Quantization:**
    If you are using Q2_K or Q3_K_M to save VRAM, your TPS might be high, but your "Reasoning" quality will crater. If the model starts looping or giving nonsensical answers, your "Reasoning Budget" is being spent on garbage.

4.  **Thermal Throttling:**
    If your TPS starts high and slowly drops over a 10-minute generation, your GPU is overheating. Check your fan curves and power limits.

5.  **The "MTP Hallucination" Effect:**
    MTP can sometimes cause the model to "jump the gun" and skip logical steps because it's trying to predict the end of the sentence too early. If you notice the model skipping "Step 2" and going straight to "Step 4," disable MTP for that specific model.

6. **The "MTP Hallucination" Effect**
    MTP (Multi-Token Prediction) works by training the model to predict a sequence of future tokens rather than just the next one. While this can significantly increase throughput, it can occasionally lead to "over-eager" predictions. In complex reasoning tasks—like writing a legal brief or a complex piece of software—the model might "jump" to a conclusion before it has fully "processed" the logical constraints of the prompt. If you notice the model skipping steps or providing a summary when you asked for a detailed breakdown, MTP might be over-prioritizing the "end" of the sequence over the "path" to get there.

---

# Advanced Optimization Techniques

To move beyond basic usage and into "Lab" territory, you need to master the knobs that actually govern the underlying CUDA kernels and memory management.

### 1. Flash Attention & Memory Efficiency
Flash Attention is non-negotiable for high-context work. It optimizes the attention mechanism by reducing the number of memory reads/writes between the GPU's fast SRAM and the slower VRAM.
*   **Why it matters:** Without it, the memory required for the attention matrix grows quadratically ($O(N^2)$) with the sequence length. With it, the memory footprint is significantly reduced, allowing for much larger context windows.
*   **How to check:** Ensure your `llama.cpp` build supports it and that the flag is active in your startup command.

### 2. KV Cache Quantization
The KV (Key-Value) Cache is the "memory" of the current conversation. It stores the representations of all previous tokens so the model doesn't have to re-calculate them for every new token.
*   **The Problem:** At high context lengths (e.g., 32k+), the KV Cache can consume more VRAM than the model weights themselves.
*   **The Solution:** Use `cache_type_q4_0` or `cache_type_q8_0`. This quantizes the cache.
*   **The Trade-off:** Q4 quantization of the cache can slightly degrade the "coherence" of very long conversations, but it can often double your available context window in the same VRAM footprint.

### 3. Batching Dynamics (`n_batch` vs. `n_ubatch`)
These two settings are often confused, but they serve different purposes in the pre-fill (TTFT) phase.
*   **`n_batch`:** This is the maximum number of tokens the model will process in a single "pass" during the pre-fill phase. If you have a 2,000-token prompt and `n_batch` is 512, the model will take 4 passes to "read" your prompt.
*   **`n_ubatch`:** This is the number of tokens processed in a single "micro-batch" within the GPU kernel.
*   **Optimization Rule:** For modern GPUs (RTX 30-series, 40-series, A100, H100), you want `n_batch` to be as high as your VRAM allows (e.g., 2048 or 4096) to minimize the number of passes. Set `n_ubatch` to a value that keeps the GPU cores saturated (usually 512 or 1024).

---

# Hardware-Specific Nuances

Your hardware dictates the "ceiling" of your performance. You must understand your specific architecture to interpret your benchmarks correctly.

### 1. NVIDIA (CUDA)
NVIDIA cards are the gold standard for `llama.cpp` due to highly optimized kernels.
*   **Tensor Cores:** Ensure your model is running in a way that utilizes Tensor Cores (FP16 or BF16).
*   **PCIe Bandwidth:** If you are running a model that is too large for your VRAM and it "spills over" into system RAM, your performance will be limited by your PCIe slot speed (e.g., Gen 3 x16 vs. Gen 4 x16). This is why "jumping" memory often feels like a massive slowdown—it's the transition from VRAM to System RAM.

### 2. AMD (ROCm)
If you are using ROCm, the kernels are different.
*   **Memory Bandwidth:** AMD cards often have massive memory bandwidth (e.g., 700GB/s+ on 7900 XTX). You should see very high TPS on smaller models (7B-13B) but may see sharper "cliffs" when the context window grows large.

### 3. Apple Silicon (Unified Memory)
On a Mac, the "VRAM" is actually your system RAM.
*   **The Advantage:** You can run massive models (70B+) because the memory is shared.
*   **The Bottleneck:** System RAM is significantly slower than dedicated VRAM. Your TPS will be lower, but your "Context Ceiling" is much higher.
*   **Optimization:** On Mac, `n_batch` and `n_ubatch` are less about "fitting" and more about "throughput." Keep them high to maximize the utilization of the Unified Memory Architecture.

---

# Agentic Workflow Benchmarking

If you are building agents (e.g., using AutoGPT, LangChain, or CrewAI), standard TPS is a secondary metric. You need to measure **Turn-around Time (TAT)** and **Tool-Call Reliability**.

### 1. The "Tool-Call" Latency Test
Agents often need to output a specific JSON format to call a tool.
*   **Test:** Give the model a prompt: "Check the weather in Tokyo and tell me the temperature."
*   **Metric:** Measure the time from "Enter" to the completion of the JSON block.
*   **Why:** If the TTFT is too high, the agent will feel "laggy" to the user. If the TPS is too low, the agent will take too long to "think" through the tool parameters.

### 2. The "Multi-Turn Context Accumulation" Test
Agents accumulate history. Every turn adds to the prompt length.
*   **Test:** Run a 10-turn conversation where the agent performs a task.
*   **Data Point:** Record the TPS for Turn 1, Turn 5, and Turn 10.
*   **Analysis:** If the TPS drops by more than 30% by Turn 10, your context management is inefficient, and the agent will become exponentially slower as the "memory" grows.

---

# Advanced SQL Queries for Deep Analysis

Use these queries to find the "hidden" patterns in your data that a simple average won't show.

### 1. Identifying the "Context Cliff"
This query finds the point where your TPS starts to degrade significantly.
```sql
SELECT 
    prompt_tokens, 
    AVG(tps) as avg_tps,
    (AVG(tps) - LAG(AVG(tps)) OVER (ORDER BY prompt_tokens)) as tps_drop
FROM inference_logs
WHERE mtp_enabled = 0
GROUP BY prompt_tokens
ORDER BY prompt_tokens;
```
*Look for the first prompt_tokens value where `tps_drop` becomes a large negative number.*

### 2. TTFT vs. VRAM Correlation
Does your VRAM usage actually correlate with your "Time to First Token"?
```sql
SELECT 
    vram_used_mb, 
    AVG(ttft_ms) as avg_ttft,
    COUNT(*) as samples
FROM inference_logs
GROUP BY vram_used_mb
ORDER BY vram_used_mb;
```
*If `avg_ttft` spikes suddenly at a certain `vram_used_mb`, you have found your hardware's "Hard Ceiling."*

### 3. MTP Efficiency Coefficient
Calculate exactly how much faster MTP is making your model.
```sql
SELECT 
    mtp_enabled,
    AVG(tps) / NULLIF(AVG(ttft_ms), 0) as throughput_efficiency_ratio
FROM inference_logs
GROUP BY mtp_enabled;
```

---

# The "Perfect" Config Cheat Sheet

Based on common hardware profiles, here are the starting points for your weekend tests. Adjust based on your specific results.

| Hardware Profile | Model Size | `n_ctx` | `n_batch` | `n_ubatch` | `cache_type` | MTP |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **8GB VRAM (RTX 3060/4060)** | 7B-8B (Q4_K_M) | 4096 | 512 | 128 | q4_0 | Off |
| **12GB VRAM (RTX 3060 12G)** | 7B-13B (Q4_K_M) | 8192 | 1024 | 256 | q4_0 | On |
| **16GB VRAM (RTX 4070 Ti)** | 13B-14B (Q4_K_M) | 8192 | 2048 | 512 | q4_0 | On |
| **24GB VRAM (RTX 3090/4090)** | 30B-34B (Q4_K_M) | 16384 | 2048 | 512 | q4_0 | On |
| **48GB+ VRAM (Dual 3090/A6000)** | 70B (Q4_K_M) | 16384 | 4096 | 1024 | q4_0 | On |
| **64GB+ Unified (Mac Studio)** | 70B (Q4_K_M) | 32768 | 2048 | 512 | f16 | On |

---

# Troubleshooting the "Inconsistency"

If your benchmarks show wild swings in TPS (e.g., 50 TPS one second, 5 TPS the next), check these three "Invisible" culprits:

### 1. Memory Fragmentation
If you are running `llama.cpp` on a system with other apps open, the VRAM might be fragmented. Even if you have 4GB "free," it might not be 4GB of *contiguous* space.
*   **Fix:** Close all browsers and other GPU-accelerated apps (like Discord or Steam) before running your benchmark suite.

### 2. The "System Swap" Ghost
On Windows and Linux, if your VRAM fills up, the OS might try to move parts of the model to "Shared GPU Memory" (System RAM). This is a performance killer.
*   **How to detect:** Watch `nvidia-smi`. If "Used Memory" stays constant but "Memory Usage" (in the OS) spikes, you are hitting the swap.
*   **Fix:** Lower your `n_ctx` or use a more aggressive quantization (e.g., Q3_K_L).

### 3. Thermal Throttling
If your GPU hits 85°C+, it will downclock its cores.
*   **How to detect:** Run a 10-minute generation and watch the clock speeds in `nvidia-smi -l 1`. If the MHz drops significantly, you need better airflow.

---

# Model Selection Guide (Based on Benchmarks)

Once you have your data, use this guide to decide which model to use for which task.

### 1. Coding (The "Precision" Model)
*   **Requirement:** High `n_ctx` (to see the whole file), High TPS (to see code appear fast).
*   **Recommendation:** DeepSeek-Coder-V2 or CodeLlama.
*   **Benchmark Target:** You want a "Context Cliff" that doesn't happen until at least 16,384 tokens.

### 2. RAG (The "Retrieval" Model)
*   **Requirement:** Low TTFT (to process the retrieved chunks quickly).
*   **Recommendation:** Mistral-7B-v0.3 or Llama-3-8B.
*   **Benchmark Target:** Focus on the "Prompt Penalty" chart. You want the TTFT to stay under 2 seconds even with 2,000 tokens of retrieved context.

### 3. Agentic Work (The "Reliability" Model)
*   **Requirement:** High consistency, low "MTP Hallucination."
*   **Recommendation:** Llama-3-70B (if VRAM allows) or Mixtral-8x7B.
*   **Benchmark Target:** Look for the lowest "TPS Spread" in your SQLite queries. You want a model that is predictably fast.

### 4. Creative Writing (The "Flow" Model)
*   **Requirement:** High TPS, High Context.
*   **Recommendation:** Command R or Mixtral.
*   **Benchmark Target:** Focus on the "Long Output Test." You want a model that maintains a steady TPS for 1,000+ tokens without a "cliff."

---

# The "Lab Notebook" Methodology

To truly master your local setup, you should treat this like a scientific experiment. Create a "Lab Notebook" (a simple Markdown file or a Notion page) and record the following for every model you test:

1.  **The "Golden Config":** The exact command line used (including `n_batch`, `n_ctx`, `cache_type`, etc.).
2.  **The "Context Cliff" Point:** The exact token count where your TPS dropped by >50%.
3.  **The "MTP Delta":** The percentage increase in TPS provided by MTP for that specific model.
4.  **The "Reasoning Budget":** The average TTFT for a 1,000-token prompt.
5.  **Subjective Quality Score:** A 1-10 rating on how well the model followed instructions at that specific context length.

**Why do this?**
Because as `llama.cpp` updates and new models are released, your "Golden Config" will change. By having a notebook, you can quickly see if a new version of a model is actually faster or if it's just "feeling" faster because of a different quantization method.

### Final Advice for the Weekend:
Don't try to optimize everything at once. Start with the **Data Collection Plan**. Get your SQLite database populated with 20-30 samples of different prompt lengths. Once you have the data, the "Quick Diagnosis" will become obvious. You will see the "cliff," you will see the "MTP delta," and you will know exactly which knobs to turn.

Stop guessing. Start measuring.

### Quantization Nuances: The GGUF Spectrum

To truly master your "Reasoning Budget," you must understand how different quantization methods affect the model's internal weights and, consequently, its performance. In the GGUF ecosystem, not all quants are created equal.

**1. K-Quants (Q4_K_M, Q5_K_M, etc.)**
These are the "gold standard" for most users. They use a mix of different bit-depths for different layers of the model.
*   **Q4_K_M:** Generally considered the "sweet spot." It offers a significant reduction in VRAM usage with a negligible hit to perplexity (the model's "intelligence").
*   **Q5_K_M:** If you have the extra VRAM, this is often the best choice for coding and complex reasoning. The jump in intelligence over Q4 is noticeable in long-form logic.
*   **Q8_0:** Use this only if you are running a small model (e.g., 7B or 8B) and need absolute maximum precision. For larger models, the VRAM cost usually outweighs the marginal gains in logic.

**2. IQ (Importance Matrix) Quants**
If you are pushing the limits of your VRAM, look into IQ quants (e.g., IQ4_XS, IQ3_M). These use an "Importance Matrix" (imatrix) generated during the quantization process to identify which weights are most critical to the model's performance.
*   **The Benefit:** An IQ4_XS quant can sometimes outperform a standard Q4_K_M in terms of logic because it preserves the "important" weights more effectively.
*   **The Trade-off:** They can be slightly slower to load and may have slightly higher overhead during the initial pre-fill.

**3. The "Reasoning Budget" vs. Quantization**
A lower quantization (like Q3_K_L) will give you a higher TPS (Tokens Per Second) because the weights are smaller and move faster across the memory bus. However, your "Reasoning Budget" will suffer because the model's ability to maintain a coherent logical chain over 1,000+ tokens will degrade. 
*   **Rule of Thumb:** For Chat and Creative Writing, you can go lower (Q4_K_S). For Coding and Agentic work, never go below Q4_K_M or Q5_K_M.

---

### The Math of the KV Cache

To stop the "jumping" VRAM, you need to understand the math behind the KV Cache. This is the most common reason for performance instability.

The KV Cache size is roughly calculated as:
`2 * layers * heads * head_dim * precision_bytes * n_ctx`

**Example for a Llama-3-8B model:**
*   Layers: 32
*   Heads: 32
*   Head Dim: 1280
*   Precision: 2 bytes (for FP16)
*   Context: 8192

`2 * 32 * 32 * 1280 * 2 * 8192 = 4,294,967,296 bytes (approx 4GB)`

**The Insight:**
If you set your `n_ctx` to 32,768, that KV Cache alone will take up **16GB of VRAM**. If your GPU only has 16GB of VRAM, the model weights won't even fit. This is why your memory "jumps"—the system is trying to allocate a massive block of memory for the cache, failing, and then trying to find a way to swap it.

**Practical Application:**
Use this formula to calculate your "Safe Context." Subtract the size of your model weights (e.g., a Q4_K_M 8B model is ~5.5GB) from your total VRAM, then see how much room is left for the KV Cache. 
*   **Pro Tip:** Use `cache_type_q4_0` to reduce that `precision_bytes` multiplier from 2 down to 0.5, effectively quadrupling your available context window for the same VRAM cost.

---

### Mixture of Experts (MoE) vs. Dense Models

If you are running models like Mixtral or DeepSeek-V2, the rules of "TPS" change slightly.

**1. Active vs. Total Parameters**
MoE models have a high "Total Parameter" count (e.g., 57B) but a much lower "Active Parameter" count (e.g., 13B). 
*   **VRAM Impact:** You must fit the *Total* parameters into your VRAM. This means MoE models require massive amounts of memory.
*   **TPS Impact:** Your TPS will be closer to a 13B model because only a fraction of the weights are being "touched" for each token.

**2. The "MTP" Interaction in MoE**
MTP can be particularly effective in MoE models because it helps the "router" (the part of the model that decides which experts to use) stay on track. However, because MoE models are already memory-bandwidth heavy, ensure your `n_ubatch` is optimized to keep the experts saturated.

---

### OpenWebUI Integration & Configuration

To make your weekend findings useful, you need to translate your "Golden Config" into the OpenWebUI interface.

**1. Model Files and Parameters**
Don't just change the global settings. Create a "Model File" in OpenWebUI for each specific use case (e.g., "Coding-Optimized," "RAG-Fast," "Creative-Long").
*   **Coding-Optimized:** Set `n_ctx` to 8192, `n_batch` to 2048, and `temperature` to 0.2.
*   **RAG-Fast:** Set `n_ctx` to 4096, `n_batch` to 1024, and `temperature` to 0.7.

**2. Mapping llama.cpp Flags**
Ensure your OpenWebUI connection to `llama.cpp` (via the OpenAI-compatible API) is passing the correct flags. If you are using a wrapper like `ollama` or a direct `llama.cpp` server, verify that:
*   `--ctx` matches your OpenWebUI `n_ctx`.
*   `--batch-size` matches your `n_batch`.
*   `--flash-attn` is enabled.

---

### Prompt Engineering for Stress Testing

To get accurate data for your SQLite reports, you need "Stress Test" prompts that force the model to use its full reasoning budget.

**1. The Logic Stress Test (Tests Reasoning Depth)**
> "A farmer has 17 sheep. All but 9 die. How many are left? Now, explain the logic of that answer using a 3-step deductive reasoning chain, then write a short poem about the remaining sheep in the style of Robert Frost."
*   *Why:* This forces the model to handle a trick question, a logic chain, and a creative task in one go.

**2. The Context Stress Test (Tests Context Cliff)**
Paste a long technical manual (3,000+ words) and ask:
> "Based on the text provided above, identify the three most critical safety warnings and explain how they relate to the maintenance schedule described in section 4."
*   *Why:* This tests the model's ability to "needle in a haystack"—finding specific info in a large context.

**3. The Agentic Loop Test (Tests Consistency)**
> "You are a project manager. I need you to plan a 5-day software sprint. First, list the goals. Second, break them into daily tasks. Third, assign a priority level to each task. Fourth, identify potential bottlenecks. Do this in a structured JSON format."
*   *Why:* This tests the model's ability to maintain a complex structure over a long output.

---

### Advanced Troubleshooting: The "Freeze" and the "Loop"

If your benchmarks show that the model "freezes" (stops generating) or "loops" (repeats the same sentence forever), here is how to diagnose it:

**1. The "Context Wrap" Freeze**
If the model stops mid-sentence, it has likely hit the `n_ctx` limit. 
*   **The Fix:** Either increase `n_ctx` (if VRAM allows) or implement a "Context Window Management" strategy where you truncate the oldest parts of the conversation.

**2. The "Repetition Loop"**
If the model starts repeating "The data is... The data is... The data is...", it's a sign of "Context Poisoning."
*   **The Cause:** The model's own previous output is now part of the prompt, and it's getting stuck in a high-probability loop.
*   **The Fix:** Increase your `repetition_penalty` (try 1.1 or 1.2) or slightly increase your `temperature`.

**3. The "Kernel Hang"**
If the entire `llama.cpp` process stops responding and your GPU usage stays at 100% but TPS is 0.
*   **The Cause:** This is usually a CUDA kernel crash or an Out-of-Memory (OOM) error that didn't trigger a clean exit.
*   **The Fix:** Reduce your `n_batch` or `n_ubatch`. Sometimes a batch size that is too large for the specific model architecture will cause the kernel to overflow.

---

### Final Synthesis: The Philosophy of the Local Lab

The goal of this weekend isn't just to find the "fastest" settings; it's to find the **"Optimal Utility"** settings. 

A model that runs at 100 TPS but fails to follow a complex coding instruction is useless. A model that runs at 2 TPS but perfectly executes a complex agentic workflow is a success. 

By following this guide, you are moving from a **Consumer** (who just hopes the model works) to a **Lab Operator** (who knows exactly why the model is behaving the way it is). You now have the tools to:
1.  **Quantify** your hardware's limits.
2.  **Isolate** the impact of MTP.
3.  **Map** your model's performance to your specific needs (Coding, RAG, Chat).
4.  **Iterate** on your configuration as you upgrade your hardware or try new models.

**Your final output should be your "Golden Config" table.** When you find a configuration that hits your target TPS while maintaining high-quality reasoning, save it. That is your baseline. Every time you try a new model or a new quantization, compare it against that baseline. 

**Stop guessing. Start measuring.**