# Quick Diagnosis

Before you start your weekend deep dive, we need to stabilize your mental model of what "inconsistent" actually means in the context of `llama.cpp`. Inconsistency usually stems from one of three architectural bottlenecks: **Prefill Latency**, **KV Cache Dynamics**, or **System Resource Contention**.

### 1. The "First Token" Lag (Prefill vs. Decoding)
When you send a prompt, `llama.cpp` performs a "Prefill" phase. It processes the entire input prompt to build the initial Key-Value (KV) cache. 
*   **The Symptom:** A 500-word prompt takes 3 seconds to start, but then generates at 50 tokens per second (TPS).
*   **The Cause:** Prefill is compute-bound (matrix multiplications). If your GPU is saturated or your prompt is large, the "Time to First Token" (TTFT) will spike.
*   **The Fix:** Check if your `n_batch` and `n_ubatch` settings are optimized. If they are too low, prefill is slow. If they are too high, you might hit VRAM limits.

### 2. The "GPU Memory Jump" (KV Cache Allocation)
You mentioned memory jumps. This is often the result of how `llama.cpp` manages the KV Cache.
*   **The Symptom:** Memory usage stays flat for three prompts, then suddenly spikes by several gigabytes.
*   **The Cause:** `llama.cpp` often allocates memory dynamically or in chunks. If you are using a large `n_ctx` (context window), the system might be allocating the cache for the *next* expected block of tokens.
*   **The Fix:** Use `flash_attn` if supported by your hardware. It significantly reduces the memory footprint of the attention mechanism. Also, ensure `n_ctx` is set to a fixed value rather than letting it grow dynamically, which causes unpredictable "jumps."

### 3. The MTP (Multi-Token Prediction) Variable
MTP is designed to predict multiple future tokens simultaneously. 
*   **The Symptom:** Higher throughput (TPS) but potentially lower "intelligence" or "coherence" in complex reasoning.
*   **The Cause:** MTP speeds up the *decoding* phase by predicting $N$ tokens at once. However, it adds overhead to the *prefill* phase and consumes more memory because it maintains additional hidden states for the "lookahead" tokens.
*   **The Fix:** You cannot tell if MTP is "helping" unless you isolate it. You must run the exact same prompt twice: once with MTP enabled and once with it disabled, measuring only the **Decoding TPS** and **Perplexity/Accuracy**.

---

# Data Collection Plan

To stop "feeling" like the model is inconsistent and start *knowing*, you need a structured telemetry pipeline. You have the tools; now we need the protocol.

### 1. The Instrumentation Stack
*   **Flight Recorder:** Use this to capture the raw request/response lifecycle. It should log: `Prompt_Length`, `Completion_Length`, `TTFT` (Time to First Token), and `Total_Duration`.
*   **ServerTop:** Use this to capture the "Hardware Truth." You need to sample GPU Utilization, VRAM usage, and Power Draw at 1-second intervals.
*   **SQLite:** This is your "Source of Truth." Every request from OpenWebUI should be piped into a table.

### 2. The Logging Schema
Ensure your SQLite table includes these specific columns:
*   `request_id` (UUID)
*   `timestamp` (ISO8601)
*   `model_name` (e.g., Llama-3-8B-Instruct-Q4_K_M)
*   `prompt_tokens` (Integer)
*   `completion_tokens` (Integer)
*   `ttft_ms` (Milliseconds)
*   `tps_decoding` (Tokens per second)
*   `gpu_mem_used_mb` (From ServerTop)
*   `mtp_enabled` (Boolean)
*   `reasoning_steps` (Integer - if using a CoT model)

### 3. Standardized Test Prompts
Do not use "random" chat for benchmarking. You need three "Standardized Stressors":
1.  **The "Short Burst" (Chat):** "Explain the concept of photosynthesis in two sentences." (Tests low-latency response).
2.  **The "Context Heavy" (RAG/Coding):** Paste a 2,000-word technical documentation snippet and ask: "Based on the text, how do I configure the API key?" (Tests Prefill and KV Cache stability).
3.  **The "Long Form" (Creative/Reasoning):** "Write a 1,000-word short story about a clockmaker who can freeze time." (Tests Decoding stability and memory endurance).

### 4. Collection Command (Example)
Use a `curl` script to hit your `llama.cpp` server. This allows you to bypass the OpenWebUI overhead for pure benchmarking.

```bash
# Example: Benchmarking a specific prompt with MTP enabled
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-8b",
    "prompt": "Write a detailed technical explanation of how transformer attention works.",
    "max_tokens": 500,
    "temperature": 0.7,
    "stream": false,
    "extra_body": {
      "n_ctx": 4096,
      "n_batch": 512,
      "use_mtp": true
    }
  }' > response_$(date +%s).json
```

---

# MTP Comparison Plan

To determine if Multi-Token Prediction is actually helping your specific hardware/model combo, you must perform a **Controlled A/B Test**.

### Step 1: The Baseline (Non-MTP)
Run the "Long Form" test prompt 5 times.
*   Record `Average_TTFT`.
*   Record `Average_TPS`.
*   Record `Max_VRAM_Usage`.
*   **Note:** Observe if the TPS drops as the completion gets longer (this indicates KV cache pressure).

### Step 2: The MTP Variant
Run the *exact same* prompt 5 times with MTP enabled.
*   Record the same metrics.
*   **Crucial:** Check the "Quality Score." Does the model start repeating itself or lose the thread faster? MTP can sometimes "hallucinate" the trajectory of a sentence.

### Step 3: The Comparison Matrix
Create a table to compare:

| Metric | Non-MTP (Baseline) | MTP Enabled | Delta (%) |
| :--- | :--- | :--- | :--- |
| **TTFT (ms)** | (e.g., 1200ms) | (e.g., 1500ms) | +25% (Slower) |
| **Decoding TPS** | (e.g., 40 tps) | (e.g., 65 tps) | +62% (Faster) |
| **VRAM Peak** | (e.g., 6GB) | (e.g., 7.5GB) | +25% (Higher) |
| **Coherence** | 10/10 | 8/10 | -20% (Lower) |

**The Verdict:**
*   If **TPS Gain > 30%** and **Coherence Loss is negligible**, MTP is a win for Creative Writing and Chat.
*   If **TTFT increases significantly** and **Coherence drops**, MTP is a net negative for Agentic work or Coding (where accuracy is paramount).

---

# Reasoning Budget Plan

When using models for "Reasoning" (like DeepSeek-R1 or Llama-3-70B-Instruct), you need to measure the **Reasoning Density**. This is the ratio of "Thinking Tokens" to "Output Tokens."

### 1. Defining the Budget
*   **Agentic Work:** High Reasoning Budget. You want the model to "think" for 500 tokens to produce 50 tokens of high-quality tool calls.
*   **Coding:** Medium Reasoning Budget. You want it to plan the logic, then execute the code.
*   **Chat/Creative:** Low Reasoning Budget. You want immediate, fluid output.

### 2. How to Measure
In your SQLite database, track `thinking_tokens` (tokens inside `<thought>` tags or similar).
*   **Metric:** `Reasoning_Efficiency = (Thinking_Tokens / Completion_Tokens)`.
*   **Goal:** If you are doing RAG, you want a high `Reasoning_Efficiency` during the "Analysis" phase but a low one during the "Summarization" phase.

### 3. The "Cost" of Reasoning
Monitor your GPU power draw during the "Thinking" phase. Reasoning often involves high-intensity computation. If your GPU hits a thermal limit during a long "thought" block, your TPS will tank for the remainder of the generation. This is a "Reasoning Budget" failure.

---

# Long Output Test Plan

Long outputs are where local LLMs usually break. You need to test for **Contextual Decay** and **Memory Exhaustion**.

### 1. The "Needle in a Haystack" (Contextual Integrity)
1.  Create a 4,000-token prompt.
2.  In the middle (at token 2,000), insert a random fact: *"The secret password is 'BlueberryMuffin77'."*
3.  At the end, ask: *"What is the secret password?"*
4.  **Repeat this at different context lengths:** 2k, 4k, 8k, 16k.
5.  **Success Metric:** Does the model still find the password? If it fails at 8k but works at 4k, your KV cache management or model attention is degrading.

### 2. The "Drift" Test (Coherence)
1.  Ask the model to write a story.
2.  Force it to continue for 1,000 tokens.
3.  **Check for:**
    *   **Repetition:** Does it start repeating the same sentence?
    *   **Contradiction:** Does a character who died in paragraph 2 reappear in paragraph 10?
    *   **Style Drift:** Does it start with a formal tone and end with slang?

### 3. The "VRAM Ceiling" Test
1.  Start a generation.
2.  Monitor `ServerTop` VRAM.
3.  If VRAM hits 95%+, does the system swap to system RAM (causing a massive TPS drop) or does the process crash?
4.  **Goal:** Identify your "Safe Context Limit." If 8k context causes a swap, your safe limit is 7k.

---

# SQLite Queries

Use these queries to analyze your collected data. Replace `bench_table` with your actual table name.

### 1. Average Performance by Prompt Length
This helps you see exactly where the "slowdown" begins.
```sql
SELECT 
    CASE 
        WHEN prompt_tokens < 512 THEN 'Short (<512)'
        WHEN prompt_tokens BETWEEN 512 AND 2048 THEN 'Medium (512-2k)'
        ELSE 'Long (>2k)' 
    END AS category,
    AVG(ttft_ms) as avg_ttft,
    AVG(tps_decoding) as avg_tps,
    AVG(gpu_mem_used_mb) as avg_vram
FROM bench_table
GROUP BY category;
```

### 2. Identifying "Outlier" Inconsistencies
Find requests that were significantly slower than the average for their size.
```sql
WITH Stats AS (
    SELECT 
        prompt_tokens, 
        AVG(ttft_ms) as avg_ttft 
    FROM bench_table 
    GROUP BY prompt_tokens
)
SELECT b.* 
FROM bench_table b
JOIN Stats s ON b.prompt_tokens = s.prompt_tokens
WHERE b.ttft_ms > (s.avg_ttft * 1.5); -- Find requests 50% slower than average
```

### 3. MTP Impact Analysis
Compare MTP vs. Non-MTP performance directly.
```sql
SELECT 
    mtp_enabled,
    AVG(tps_decoding) as avg_tps,
    AVG(ttft_ms) as avg_ttft,
    COUNT(*) as sample_count
FROM bench_table
GROUP BY mtp_enabled;
```

### 4. P95 Latency (The "Frustration" Metric)
Average speed is nice, but P95 (the speed 95% of users experience) is what matters for UX.
```sql
SELECT 
    prompt_tokens,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY ttft_ms) as p95_ttft
FROM bench_table
GROUP BY prompt_tokens;
```

---

# Dashboard Screenshots To Capture

When you present your findings (or just for your own records), capture these four specific views:

### 1. The "TTFT vs. Prompt Length" Scatter Plot
*   **X-Axis:** Prompt Tokens.
*   **Y-Axis:** TTFT (ms).
*   **What to look for:** A linear increase is normal. An exponential spike indicates a memory bottleneck or a "context overflow" where the model is struggling to attend to the whole prompt.

### 2. The "TPS Decay" Line Chart
*   **X-Axis:** Completion Tokens (Cumulative).
*   **Y-Axis:** Tokens Per Second.
*   **What to look for:** A steady line is perfect. A downward slope indicates that the KV cache is becoming too large for the GPU's high-speed memory, forcing it to rely on slower memory paths.

### 3. The "VRAM Heatmap"
*   **X-Axis:** Request ID.
*   **Y-Axis:** VRAM Usage (MB).
*   **What to look for:** "Staircase" patterns. If VRAM jumps up and stays there, `llama.cpp` is pre-allocating. If it jumps up and then back down, it's dynamic allocation (which can cause fragmentation).

### 4. The "MTP Delta" Bar Chart
*   **Group 1:** Non-MTP TPS.
*   **Group 2:** MTP TPS.
*   **What to look for:** The "Efficiency Gap." If the gap is small but the TTFT is much higher with MTP, MTP is a waste of your resources.

---

# Weekend Checklist

### Saturday: Data Collection (The "Grind")
- [ ] **Morning (Setup):** Configure `llama.cpp` with fixed `n_ctx` and `n_batch`. Ensure your SQLite logging script is active.
- [ ] **Mid-Day (Baseline):** Run 10 "Short Burst" prompts. Record results.
- [ ] **Afternoon (Context Stress):** Run 10 "Context Heavy" prompts (2k, 4k, 8k). Record results.
- [ ] **Evening (MTP Test):** Repeat the "Context Heavy" prompts with MTP enabled. Record results.
- [ ] **Night (Long Form):** Run 5 "Long Form" story prompts. Monitor VRAM and TPS decay.

### Sunday: Analysis (The "Science")
- [ ] **Morning (SQL Analysis):** Run the queries provided above. Identify your "Break Point" (the prompt length where TTFT spikes).
- [ ] **Mid-Day (MTP Verdict):** Compare the MTP vs. Non-MTP delta. Decide if you will keep it on for specific use cases.
- [ ] **Afternoon (Use Case Mapping):** Based on your data, categorize your models. (e.g., "Model X is too slow for Chat but perfect for RAG").
- [ ] **Evening (Optimization):** Adjust `n_batch`, `n_ubatch`, and `flash_attn` based on your findings. Re-run one test to verify the improvement.

---

# Interpreting Results

How do you turn these numbers into a "Practical Guide" for your daily work? Use this decision matrix:

### 1. For Coding (The "Precision" Use Case)
*   **Requirement:** High accuracy, medium context, low tolerance for "hallucinated" logic.
*   **Ideal Profile:** Low MTP (or off), High `n_batch` (for fast prefill), High `n_ctx`.
*   **Your Data Signal:** If your "Context Heavy" prompts show high accuracy but slow TTFT, keep MTP off. You need the model's full "attention" on the code logic.

### 2. For RAG (The "Retrieval" Use Case)
*   **Requirement:** Massive context, high retrieval accuracy, moderate speed.
*   **Ideal Profile:** Flash Attention enabled, `n_ctx` set to maximum supported by VRAM.
*   **Your Data Signal:** Look at the "Needle in a Haystack" test. If the model fails at 8k, you must use a RAG strategy that chunks data into 4k pieces.

### 3. For Agentic Work (The "Reasoning" Use Case)
*   **Requirement:** High reasoning density, reliable tool calling, low latency for "thought" blocks.
*   **Ideal Profile:** High Reasoning Budget, MTP off (to ensure logic remains linear).
*   **Your Data Signal:** If your "Reasoning Efficiency" is high but the model gets stuck in loops, you need a model with a higher parameter count or a better "Thinking" fine-tune.

### 4. For Chat (The "Fluidity" Use Case)
*   **Requirement:** Ultra-low TTFT, high TPS, personality.
*   **Ideal Profile:** MTP enabled (to maximize TPS), `n_batch` optimized for speed.
*   **Your Data Signal:** If your "Short Burst" TTFT is < 500ms and TPS is > 50, this is your "Daily Driver" for casual interaction.

### 5. For Creative Writing (The "Flow" Use Case)
*   **Requirement:** High TPS, long-form coherence, stylistic variety.
*   **Ideal Profile:** MTP enabled, high `n_ctx`.
*   **Your Data Signal:** If the "Long Form" test shows good coherence and high TPS, MTP is your best friend here.

---

# Common Mistakes

### 1. The "Vanity Metric" Trap
**Mistake:** Only looking at "Average TPS."
**Why it's bad:** A model might have a high average TPS but a terrible TTFT. If it takes 10 seconds to start and then flies at 100 TPS, it still *feels* slow to a human.
**Correction:** Always prioritize **TTFT** for Chat and **TPS** for Creative Writing.

### 2. Ignoring Thermal Throttling
**Mistake:** Benchmarking for 2 hours straight and seeing performance drop.
**Why it's bad:** Your GPU might be downclocking due to heat.
**Correction:** Run benchmarks in "bursts." Run 5 prompts, wait 2 minutes for the GPU to cool, then run 5 more. If the second set is slower, you have a cooling problem, not a software problem.

### 3. Over-allocating `n_ctx`
**Mistake:** Setting `n_ctx` to 32k because the model "supports" it.
**Why it's bad:** `llama.cpp` allocates the KV cache based on `n_ctx`. If you set it to 32k, you are "stealing" VRAM from the model weights, which might force the weights into system RAM (slow) or cause an Out-Of-Memory (OOM) error.
**Correction:** Set `n_ctx` to the *maximum you actually need* for your most common task, plus a 10% buffer.

### 4. Mixing "Reasoning" and "MTP"
**Mistake:** Enabling MTP on a model that is already doing heavy "Chain of Thought" (CoT).
**Why it's bad:** MTP tries to predict the *next* tokens, but CoT models are trying to "think" through the *current* logic. These two can conflict, leading to "muddled" reasoning where the model skips steps in its logic to satisfy the MTP prediction.
**Correction:** If using a "Reasoning" model (like R1), keep MTP off unless you are purely using it for non-critical creative tasks.

### 5. The "Prompt Bloat" Oversight
**Mistake:** Not realizing that your "System Prompt" is being re-processed every single time.
**Why it's bad:** If your system prompt is 1,000 tokens long, every "Short Burst" chat will have a massive prefill cost.
**Correction:** Use **Prompt Caching**. Ensure your `llama.cpp` server has prompt caching enabled. This allows the model to "remember" the system prompt, reducing TTFT to near-zero for subsequent turns in a conversation.

6. The "Prompt Caching" Oversight
**Mistake:** Not realizing that your "System Prompt" is being re-processed every single time.
**Why it's bad:** If your system prompt is 1,000 tokens long, every "Short Burst" chat will have a massive prefill cost. Even if the user only types "Hello," the model has to "read" those 1,000 tokens of instructions before it can respond.
**Correction:** Use **Prompt Caching**. Ensure your `llama.cpp` server has prompt caching enabled. This allows the model to "remember" the system prompt, reducing TTFT to near-zero for subsequent turns in a conversation. In your SQLite logs, you should see a massive drop in `ttft_ms` for the second and third turns of a conversation compared to the first.

### Advanced Hardware Optimization

To move from "it feels slow" to "it is optimized," you must master the low-level parameters of `llama.cpp`. These are the knobs that determine how your hardware interacts with the model's math.

#### 1. The `n_gpu_layers` Sweet Spot
This is the most critical setting for local inference. It determines how many layers of the neural network are offloaded to your GPU.
*   **The Goal:** You want the *entire* model to fit in VRAM. If a model has 32 layers and you set `n_gpu_layers` to 32, the entire weight set is in VRAM.
*   **The Danger:** If you set it to 33 (or the max) and the model is too large for your VRAM, `llama.cpp` will "split" the model. Some layers will stay in system RAM. This causes a massive performance hit because the system must shuttle data across the PCIe bus for every single token generated.
*   **How to Check:** Monitor your VRAM usage. If you see your VRAM hit 98-100% and your TPS drops to < 5, you are likely spilling into system RAM. Reduce `n_ctx` or use a smaller quantization to ensure the whole model fits with room for the KV cache.

#### 2. `n_batch` vs. `n_ubatch`
These control the "Prefill" phase (how many tokens are processed at once during the initial prompt).
*   **`n_batch`:** The total number of tokens the model can process in one "pass."
*   **`n_ubatch`:** The number of tokens processed in a single "micro-batch."
*   **Optimization:** For high-end GPUs (RTX 3090/4090), you can often set `n_batch` to 512 or 1024. This makes the prefill phase much faster. However, if you have lower VRAM, a high `n_batch` will cause an Out-Of-Memory (OOM) error because the "intermediate" math requires extra memory.
*   **Rule of Thumb:** If your TTFT is slow but your TPS is fast, increase `n_batch`. If your model crashes during prefill, decrease `n_batch`.

#### 3. `mmap` and Memory Mapping
`llama.cpp` uses `mmap` to map the model file into memory.
*   **Pros:** It allows for faster startup and allows the OS to manage memory more efficiently.
*   **Cons:** On some systems (especially Windows), `mmap` can cause "stuttering" as the OS pages memory in and out.
*   **The Fix:** If you experience "jumps" in memory or inconsistent speeds during the first few seconds of a prompt, try disabling `mmap` (or setting it to false) to force the model to load fully into memory at startup.

#### 4. Flash Attention
If your hardware supports it (Ampere architecture and newer, like RTX 30-series, 40-series, or A100s), **always** enable Flash Attention.
*   **What it does:** It optimizes the "Attention" mechanism, which is mathematically $O(N^2)$ relative to context length.
*   **The Benefit:** It significantly reduces the VRAM footprint of the KV cache and speeds up the processing of long prompts. It is the single most important optimization for RAG and long-context work.

---

### Quantization Strategy & Trade-offs

You cannot benchmark a model without understanding how its quantization affects its "intelligence" and "speed."

#### 1. The GGUF Hierarchy
*   **Q4_K_M (The Standard):** This is the "Goldilocks" of quantization. It offers a massive reduction in size with very little perceptible loss in intelligence. Use this for 90% of your work.
*   **Q5_K_M (The High-Fidelity):** Use this for coding or complex reasoning where every nuance matters. The file size is larger, but the "perplexity" (error rate) is much lower.
*   **Q8_0 (The Near-Lossless):** Use this only if you have massive VRAM and need the absolute highest quality. It is rarely worth the extra VRAM cost over Q5_K_M.
*   **IQ4_XS / IQ3_M (The "Squeeze"):** Use these only when you are trying to fit a massive model (like a 70B) into a small GPU. Expect a noticeable drop in logic and "hallucination" rates.

#### 2. The Perplexity vs. Speed Curve
When you quantize a model, you are essentially "rounding" the numbers in the neural network.
*   **Small Quantizations (Q3, Q2):** The "rounding" is so aggressive that the model starts to lose its ability to follow complex instructions. It might still be "fast," but the output is garbage.
*   **Large Quantizations (Q8, FP16):** The model is "smart," but it is slow.
*   **The Benchmark Goal:** Your goal is to find the highest quantization level that fits entirely in your VRAM while maintaining a TPS of at least 20-30.

---

### The OpenWebUI & Proxy Overhead

Sometimes the "inconsistency" isn't in `llama.cpp`—it's in the "plumbing."

#### 1. The "Frontend Tax"
OpenWebUI is a powerful wrapper, but it adds overhead. It has to:
*   Parse your input.
*   Manage the chat history.
*   Format the JSON for the API.
*   Stream the response back to your browser.
*   **How to isolate:** Run a `curl` command directly against your `llama.cpp` port. If the `curl` command is consistently fast but the OpenWebUI interface feels "laggy," the bottleneck is your browser's rendering or the OpenWebUI backend's processing of the history.

#### 2. Flight Recorder Analysis
Use your Flight Recorder to look for "Gaps."
*   **Gap A:** Time between "User hits Enter" and "Request reaches llama.cpp." (This is OpenWebUI/Network latency).
*   **Gap B:** Time between "Request reaches llama.cpp" and "First Token." (This is Prefill latency).
*   **Gap C:** Time between "Tokens" (This is Decoding latency).
*   **The Diagnosis:** If Gap A is high, your proxy or OpenWebUI is the problem. If Gap B is high, your `n_batch` or GPU compute is the problem. If Gap C is high, your VRAM is full or your model is too large for your hardware.

#### 3. JSON Serialization
For very long outputs (e.g., 2,000 tokens), the time it takes to turn that text into a JSON object and send it over the network can be non-trivial. If you see a "hang" at the very end of a long generation, it's likely the serialization of the final response.

---

### Agentic Workflow Benchmarking

If you are using the model for agents (e.g., AutoGPT, CrewAI, or custom loops), standard "Chat" benchmarks are useless. You need to measure **Time to Action (TTA)**.

#### 1. The "Loop" Benchmark
Agents work in loops: *Think -> Act -> Observe -> Repeat.*
*   **The Metric:** Measure the total time for one full loop.
*   **The Bottleneck:** Often, the "Think" step is slow because the agent's prompt is huge (containing all previous actions).
*   **The Fix:** Use **Prompt Caching** for the "System Instructions" and "Tool Definitions." This ensures that only the "New Observation" is being processed as new data, keeping the loop fast.

#### 2. Success Rate vs. Latency
In agentic work, a model that is 50% faster but fails to call the correct tool 20% more often is a failure.
*   **The Test:** Run a 10-step agentic task (e.g., "Research this company and write a summary").
*   **The Data:** Record the "Success Rate" (Did it finish the task?) and the "Total Time."
*   **The Decision:** If a slower model (e.g., a 70B Q4) has a 95% success rate and a faster model (e.g., an 8B Q8) has a 60% success rate, the 70B is the only "useful" model for that agent.

---

### Advanced SQLite Analysis

To truly "see" the data, you need to move beyond simple averages. Use these advanced queries to find the "hidden" problems.

#### 1. Standard Deviation of TPS (The "Jitter" Metric)
High variance in TPS is what makes a model feel "inconsistent."
```sql
SELECT 
    model_name,
    AVG(tps_decoding) as avg_tps,
    (SELECT AVG(tps_decoding * tps_decoding) - AVG(tps_decoding) * AVG(tps_decoding) 
     FROM bench_table) as variance_tps
FROM bench_table
GROUP BY model_name;
```
*   **Interpretation:** If the variance is high, it means the model is "stuttering." This is usually caused by thermal throttling or the OS moving the process between CPU cores.

#### 2. Correlation: Context Length vs. TTFT
This helps you find the exact point where your hardware "breaks."
```sql
SELECT 
    prompt_tokens,
    AVG(ttft_ms) as avg_ttft,
    COUNT(*) as sample_count
FROM bench_table
GROUP BY prompt_tokens
ORDER BY prompt_tokens ASC;
```
*   **Interpretation:** Look for the "Elbow." The point where the line stops being linear and starts curving upward exponentially is your hardware's context limit.

#### 3. MTP Efficiency Score
Calculate how much "speed" you are gaining vs. how much "prefill" you are losing.
```sql
SELECT 
    mtp_enabled,
    AVG(tps_decoding) / AVG(ttft_ms) as efficiency_score
FROM bench_table
GROUP BY mtp_enabled;
```
*   **Interpretation:** A higher score means MTP is providing a better "User Experience" (faster tokens for the time spent waiting).

---

### Model Selection Matrix

Based on your weekend benchmarks, use this matrix to assign models to specific tasks. Do not use the same model for everything; it is the fastest way to get frustrated.

| Task Type | Priority Metric | Recommended Config | Why? |
| :--- | :--- | :--- | :--- |
| **Coding** | Accuracy / Logic | Q5_K_M, MTP Off | You need the model to "think" through the logic without MTP "guessing" the next line of code incorrectly. |
| **RAG** | Context Integrity | Flash Attention, High `n_ctx` | You need the model to "see" the whole document. Speed is secondary to retrieval accuracy. |
| **Agentic** | Tool Calling / TTA | Prompt Caching, Q4_K_M | You need the loop to be fast. Prompt caching is mandatory to keep the "Think" step from slowing down. |
| **Chat** | TTFT / Fluidity | MTP On, Q4_K_M | You want the model to feel "snappy." MTP provides the high TPS that makes chat feel like a real-time conversation. |
| **Creative** | TPS / Coherence | MTP On, High `n_ctx` | You want the model to "flow." High TPS allows for long-form generation without the "stutter" of slow decoding. |

---

### Troubleshooting "Ghost" Latency

If your benchmarks show that `llama.cpp` is fast, but the experience is still slow, you are dealing with "Ghost Latency."

#### 1. PCIe Bandwidth Bottlenecks
If you are using a multi-GPU setup or a GPU in a PCIe slot with limited lanes (e.g., x4 instead of x16), the "shuttling" of weights can be the bottleneck.
*   **The Symptom:** High GPU utilization but very low TPS.
*   **The Fix:** Ensure your GPU is in a primary PCIe slot. If using multiple GPUs, ensure they are on a bus that supports enough bandwidth for the model weights.

#### 2. CPU Context Switching
If you have a high core count (e.g., 32+ cores) but are only using a few threads for `llama.cpp`, the OS might be moving the process between cores.
*   **The Fix:** Use `taskset` (Linux) or "Processor Affinity" (Windows) to lock the `llama.cpp` process to specific physical cores. This prevents the "stutter" caused by the OS moving the process around.

#### 3. Disk I/O during Model Loading
If the "inconsistency" only happens when you first start the server, it's a disk issue.
*   **The Fix:** Move your model files to an NVMe SSD. If you are running them off a mechanical HDD or a slow network drive, the initial load and any "paging" will be agonizingly slow.

---

### Human-in-the-loop Evaluation (The "Vibe Check")

Data is king, but "feel" matters. You need a way to quantify the "vibe" of the model.

#### 1. The Side-by-Side (A/B) Test
Take 5 complex prompts. Run them on two different configurations (e.g., Model A with MTP vs. Model B without MTP).
*   **The Judge:** Have a human (or yourself) rate each response on a scale of 1-10 for:
    *   **Coherence:** Did it make sense?
    *   **Instruction Following:** Did it do what I asked?
    *   **Style:** Did it sound like the intended persona?
*   **The Goal:** If the "faster" model consistently scores lower than the "slower" model, the speed is a vanity metric.

#### 2. The "Frustration" Threshold
Define your personal frustration threshold.
*   **Chat:** If TTFT > 1.5 seconds, it feels "heavy."
*   **Coding:** If TPS < 10, it feels "painful."
*   **Creative:** If TPS < 5, it feels "unusable."
*   **The Goal:** Use your SQLite data to ensure your "Daily Driver" models stay below these thresholds.

---

### Scaling Strategy

If you find that your current setup is "consistent" but too slow for your needs, you need to scale.

#### 1. Multi-GPU Tensor Parallelism
If you have two GPUs, `llama.cpp` can split the model across both.
*   **The Benefit:** This allows you to run much larger models (e.g., 70B) at usable speeds.
*   **The Trade-off:** It requires a very fast interconnect (like NVLink) to be truly efficient. If your GPUs are connected via standard PCIe, the "communication overhead" between the two cards can sometimes negate the speed gains.

#### 2. Concurrent Request Handling
If you are running a server for multiple users, `llama.cpp` handles requests by queuing them.
*   **The Problem:** If User A is generating a 2,000-token story, User B's "Hello" will be stuck in the queue.
*   **The Fix:** Use a "Continuous Batching" approach or run multiple instances of the `llama.cpp` server, each with its own dedicated GPU or VRAM slice.

---

### Final Roadmap: From Inconsistent to Optimized

Follow this 4-week plan to turn your home-lab into a professional-grade inference engine.

#### Week 1: The Data Foundation
*   Set up the SQLite logging and Flight Recorder.
*   Run the "Standardized Stressors" (Short, Context Heavy, Long Form).
*   **Goal:** Stop guessing. Have a database of every prompt you've ever run.

#### Week 2: Bottleneck Identification
*   Analyze the SQLite data.
*   Identify your "Elbow" (Context Limit).
*   Compare MTP vs. Non-MTP for your specific models.
*   **Goal:** Know exactly why the model feels slow (is it Prefill? Decoding? VRAM?).

#### Week 3: Parameter Tuning
*   Adjust `n_batch`, `n_ubatch`, and `n_gpu_layers` based on your "Elbow" data.
*   Test different quantization levels (Q4_K_M vs. Q5_K_M).
*   Enable Flash Attention and Prompt Caching.
*   **Goal:** Maximize your TPS and minimize your TTFT.

#### Week 4: Model Selection & Deployment
*   Create your "Model Matrix."
*   Assign specific models to specific tasks (Coding, RAG, Chat).
*   Set up your OpenWebUI to use the "correct" model for the "correct" task.
*   **Goal:** A seamless, consistent experience where the model always feels "right" for the job.

By following this guide, you move from being a "user" of local LLMs to an "engineer" of local inference. You will no longer wonder if MTP is helping; you will have a chart that proves it. You will no longer wonder why the memory jumps; you will have a VRAM heatmap that explains it. You will have a laboratory, not just a chat box.