This guide is designed to move you from "anecdotal frustration" to "empirical data." When running local models, the "feeling" of inconsistency is often caused by three variables: **Prefill Latency (TTFT)**, **Decode Throughput (TPOT)**, and **Dynamic KV Cache Allocation**.

To solve this, we will treat your home lab like a scientific instrument. We will isolate variables, establish a baseline, and then stress-test the MTP (Multi-Token Prediction) configurations.

---

# Quick Diagnosis

Before you start the weekend marathon, check these three "usual suspects" that cause the inconsistencies you described:

### 1. The "Prefill" vs. "Decode" Split
The first token is slow because the model is performing a **Prefill** (processing your entire prompt at once). The subsequent tokens are the **Decode** phase (generating one token at a time).
*   **The Symptom:** If the first token takes 5 seconds but subsequent tokens fly at 50 tokens/sec, your "inconsistency" is actually just the difference between these two phases.
*   **The Fix:** Monitor `llama.cpp` logs specifically for "prompt eval time" vs. "generation time."

### 2. Dynamic KV Cache & VRAM Jumps
If your GPU memory "jumps," you are likely using a dynamic KV cache. As the model generates more tokens, it allocates more memory for the Key-Value pairs.
*   **The Symptom:** The model starts fast, then hits a "wall" where it slows down or the system swaps to system RAM (Unified Memory) because the VRAM is full.
*   **The Fix:** Check your `ctx_size` and `n_ctx`. If your context window is 32k but you only use 2k, `llama.cpp` might still be trying to reserve space or managing it dynamically in a way that causes fragmentation.

### 3. MTP (Multi-Token Prediction) Overhead
MTP allows the model to predict multiple future tokens in a single forward pass.
*   **The Symptom:** It should theoretically increase speed, but if the "lookahead" is too aggressive or the hardware isn't optimized for the extra heads, it can cause "stuttering" where the model pauses to calculate the next block.
*   **The Fix:** We will isolate this in the MTP Comparison Plan.

---

# Data Collection Plan

To stop "fooling yourself," you need a synchronized data stream. You need to capture three things simultaneously: **Inference Metrics**, **System Metrics**, and **Output Quality**.

### Tools Setup
1.  **Flight Recorder Proxy:** Ensure it is logging `request_start`, `response_start`, and `response_end`. This gives you **TTFT (Time to First Token)** and **Total Latency**.
2.  **ServerTop / `nvidia-smi`:** Run a background loop to capture VRAM and GPU Utilization every 1 second.
    *   *Command:* `watch -n 1 nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv`
3.  **SQLite Reports:** Ensure your OpenWebUI or Proxy is writing to a local SQLite DB. We will query this later.

### The "Control" Environment
To get clean data, you must eliminate "noise":
*   **Fixed Context:** Run all tests with a fixed `n_ctx` (e.g., 8192).
*   **Fixed Temperature:** Use `temperature=0` or `1.0` (no sampling randomness) for baseline speed tests.
*   **Fixed Prompt:** Use a standard "Identity" prompt (e.g., "You are a helpful assistant. Summarize the following text...") to ensure the prefill cost is constant.

### Data Points to Capture
For every test run, record:
*   **Prompt Tokens:** (Input length)
*   **Completion Tokens:** (Output length)
*   **TTFT:** (Time to first token)
*   **TPOT:** (Tokens per second during generation)
*   **Peak VRAM:** (Max memory used during the run)
*   **MTP Status:** (Enabled/Disabled)

---

# MTP Comparison Plan

This is where you determine if MTP is actually helping your specific hardware.

### The Methodology
You will run the **exact same prompt** through two configurations:
1.  **Profile A (Baseline):** Standard inference (No MTP).
2.  **Profile B (MTP):** MTP enabled with your current settings.

### The Comparison Matrix
Do not just look at "Tokens per second." You must look at **Effective Throughput**.

| Metric | Profile A (No MTP) | Profile B (MTP) | Analysis |
| :--- | :--- | :--- | :--- |
| **TTFT** | X ms | Y ms | Does MTP slow down the initial "thinking" phase? |
| **TPOT** | Z tokens/s | W tokens/s | Is the generation speed actually faster? |
| **VRAM Delta** | Base | Base + $\Delta$ | How much extra VRAM does MTP consume? |
| **Perplexity/Quality** | Score 1 | Score 2 | Does MTP cause "hallucination loops" or repetitive text? |

### How to avoid "Fooling Yourself"
Users often see a higher "Tokens per second" in MTP and assume it's better. However, if MTP causes the model to repeat the same phrase 5 times, the *useful* throughput is zero.
*   **The Rule:** If MTP increases speed by <15% but decreases output quality (measured by your own "vibe check" or a secondary LLM evaluation), **disable it.**

---

# Reasoning Budget Plan

"Reasoning" is the cost of the model "thinking" before it speaks. In an agentic workflow, you need to know how much "thinking" you can afford.

### Defining the Budget
*   **Low Reasoning (Chat/Creative):** You want high TPOT. You don't care if the model "thinks" for 2 seconds; you want it to start typing immediately.
*   **High Reasoning (Coding/Agentic):** You can tolerate a 10-second TTFT if the resulting code is bug-free.

### Measuring the Budget
Use a "Reasoning Token" count. If you use a model like DeepSeek-R1 or a CoT (Chain of Thought) prompt:
1.  **Count "Thought" Tokens:** Tokens generated inside `<thought>` tags or before the first actual answer.
2.  **Calculate Cost:** `(Thought Tokens / Total Tokens) * Total Latency`.
3.  **The Goal:** Determine the "Sweet Spot." If a 500-token thought process yields a 20% increase in code accuracy, that is a "profitable" reasoning budget. If it yields 0% increase, your model is over-thinking.

---

# Long Output Test Plan

Long outputs (e.g., writing a 2,000-word story or a complex Python script) are where `llama.cpp` performance usually breaks down.

### The "Stress Test" Protocol
Run a prompt that forces a long output (e.g., "Write a detailed technical specification for a distributed database, including 10 different modules").

### What to monitor:
1.  **The "Slowdown" Curve:** Does the TPOT drop as the context fills up? (This indicates KV Cache pressure).
2.  **The "Lost in the Middle" Test:** Provide a 4,000-token context. Place a specific fact in the middle. Ask the model to retrieve it.
    *   *Metric:* Accuracy of retrieval vs. Context Length.
3.  **Memory Exhaustion:** Does the system start swapping to disk? (Look for a massive spike in "System" memory or a sudden drop in GPU utilization).

---

# SQLite Queries

Assuming your Flight Recorder or OpenWebUI logs are stored in a table named `inference_logs` with columns: `id`, `model_name`, `prompt_tokens`, `completion_tokens`, `ttft_ms`, `tpot_avg`, `vram_peak`, `mtp_enabled`, and `timestamp`.

### Query 1: Average Performance by Model
*Use this to see which model is actually the "fastest" across all runs.*
```sql
SELECT 
    model_name, 
    AVG(ttft_ms) as avg_ttft, 
    AVG(tpot_avg) as avg_tpot, 
    COUNT(*) as total_runs
FROM inference_logs
GROUP BY model_name
ORDER BY avg_tpot DESC;
```

### Query 2: MTP Impact Analysis
*This is the core of your weekend project. It compares MTP vs. Non-MTP for the same model.*
```sql
SELECT 
    model_name,
    mtp_enabled,
    AVG(tpot_avg) as avg_tpot,
    AVG(vram_peak) as avg_vram,
    (AVG(tpot_avg) / NULLIF(AVG(tpot_avg) FILTER (WHERE mtp_enabled = 0), 0) - 1) * 100 as speed_increase_pct
FROM inference_logs
GROUP BY model_name, mtp_enabled;
```

### Query 3: Identifying "Jitter" (Standard Deviation)
*A model that is consistently "medium" is often better than a model that is "fast" one second and "slow" the next.*
```sql
SELECT 
    model_name, 
    STDDEV(tpot_avg) as tpot_jitter,
    AVG(tpot_avg) as avg_tpot
FROM inference_logs
GROUP BY model_name
ORDER BY tpot_jitter ASC;
```

---

# Dashboard Screenshots To Capture

When you present your findings (or just for your own records), capture these four specific views:

1.  **The "TTFT Distribution" Histogram:**
    *   *X-axis:* Time (ms). *Y-axis:* Frequency.
    *   *Goal:* You want a tight bell curve. A "bi-modal" distribution (two humps) means your system is intermittently failing or hitting a bottleneck (like swapping to system RAM).

2.  **The "VRAM vs. Tokens" Line Chart:**
    *   *X-axis:* Completion Tokens. *Y-axis:* VRAM Usage.
    *   *Goal:* A linear increase is normal. A "step function" (sudden jump) indicates a dynamic allocation event or a context window overflow.

3.  **The "MTP Speedup" Bar Chart:**
    *   *X-axis:* Model Name. *Y-axis:* Tokens/sec.
    *   *Comparison:* Side-by-side bars for MTP On vs. MTP Off.

4.  **The "Reasoning Budget" Scatter Plot:**
    *   *X-axis:* Thought Tokens. *Y-axis:* Accuracy Score (1-10).
    *   *Goal:* Identify the "Diminishing Returns" point where more thinking doesn't lead to better answers.

---

# Weekend Checklist

### Saturday: Baseline & Data Collection
- [ ] **Morning:** Standardize your environment. Set `n_ctx` to a fixed value (e.g., 8192) and `temperature` to 0.
- [ ] **Midday:** Run 10 "Short" prompts (under 100 tokens) and 10 "Long" prompts (over 1000 tokens) with MTP **OFF**.
- [ ] **Afternoon:** Record all metrics into your SQLite DB. Capture `nvidia-smi` logs for every run.
- [ ] **Evening:** Run the same 20 prompts with MTP **ON**.

### Sunday: Analysis & Optimization
- [ ] **Morning:** Run the SQL queries provided above.
- [ ] **Midday:** Create the "Reasoning Budget" test. Run 5 complex coding prompts and 5 creative writing prompts. Note the "Thought" token count.
- [ ] **Afternoon:** Analyze the "Jitter." If a model has high standard deviation in TPOT, investigate if it's due to background processes or `llama.cpp` thread contention.
- [ ] **Evening:** Final Decision. Create a "Model Use-Case Matrix" based on your data.

---

# Interpreting Results

Use this matrix to decide how to deploy your models based on your weekend data:

| Use Case | Priority Metric | Target Profile | Decision Logic |
| :--- | :--- | :--- | :--- |
| **Coding** | Accuracy / Reasoning | High Reasoning Budget | If MTP causes logic errors, disable it. Prioritize TTFT over TPOT. |
| **RAG** | Context Retrieval | Low Jitter | You need consistent performance. If VRAM jumps cause "Lost in the Middle," reduce `n_ctx`. |
| **Agentic** | Reliability | Low Jitter / High Reasoning | The agent needs to "think" reliably. If MTP is inconsistent, it will break the agent's loop. |
| **Chat** | TPOT (Speed) | High TPOT | If MTP gives you >10% speedup without quality loss, keep it ON. |
| **Creative** | Variety / TPOT | High TPOT | Use higher temperature. If MTP helps the "flow" of text, keep it ON. |

---

# Common Mistakes

1.  **The "Variable Context" Trap:**
    *   *Mistake:* Testing a model with a 2k prompt one time and a 4k prompt the next.
    *   *Correction:* Always use a fixed prompt length for speed benchmarks.

2.  **Ignoring "System Noise":**
    *   *Mistake:* Running benchmarks while your browser is open with 50 tabs or a video is playing.
    *   *Correction:* Close all non-essential apps. Local inference is sensitive to CPU/Memory bus contention.

3.  **Cherry-Picking:**
    *   *Mistake:* Only looking at the "fastest" run and ignoring the "slow" ones.
    *   *Correction:* Look at the **Standard Deviation**. A model that is 40 tokens/sec with a 20% jitter is worse for a production UI than a model that is 30 tokens/sec with 2% jitter.

4.  **Confusing "Prefill" with "Slow Model":**
    *   *Mistake:* Thinking the model is slow because the first token took 3 seconds.
    *   *Correction:* If the subsequent tokens are fast, the model is fine. If you want faster first tokens, you need to optimize your prompt (shorten it) or use a faster prefill strategy (like Flash Attention, if supported).

5.  **Over-reliance on MTP:**
    *   *Mistake:* Assuming MTP is a "magic button" for speed.
    *   *Correction:* MTP is a trade-off. It trades a small amount of "predictive certainty" for "generation speed." If your task requires 100% precision (like math or code), MTP's "guesses" might introduce subtle errors.

6. **Over-reliance on MTP:**
    *   *Mistake:* Assuming MTP is a "magic button" for speed.
    *   *Correction:* MTP is a trade-off. It trades a small amount of "predictive certainty" for "generation speed." If your task requires 100% precision (like math or code), MTP's "guesses" might introduce subtle errors.

---

# Advanced KV Cache Management & Memory Dynamics

To address your specific complaint about "GPU memory jumps," we need to look under the hood at how `llama.cpp` manages the Key-Value (KV) cache. This is the primary reason for inconsistent performance in long-context scenarios.

### The "Jump" Explained
When you run a model, the KV cache stores the "memory" of the conversation. In many implementations, this memory is allocated dynamically. 
*   **The Scenario:** You start a prompt. The model allocates a small block of VRAM. As it generates tokens, it realizes it needs more space. It then requests a larger block from the GPU.
*   **The Result:** This request causes a "hiccup" in the generation (a momentary pause) and a visible "jump" in your `nvidia-smi` monitor. If the GPU is nearly full, this request might trigger a "swap" to system RAM (Unified Memory), which causes a massive drop in tokens per second (TPOT).

### How to Stabilize Memory
To eliminate these jumps, you should move toward **Static Allocation**.
1.  **Set `n_ctx` strictly:** Do not let the model decide the context size. If you know you only need 4096 tokens, set `-c 4096`.
2.  **Use `flash_attn`:** If your hardware supports it, enabling Flash Attention significantly reduces the memory footprint of the attention mechanism.
3.  **Monitor "Page Faults":** If you see your GPU memory jump and your system memory usage spike simultaneously, your `n_ctx` is too large for your VRAM, and the system is "paging" to your SSD/RAM.

---

# Quantization and Precision Analysis

The "inconsistency" you feel might also be a byproduct of the quantization level. Different bits (Q4_K_M, Q8_0, etc.) interact differently with MTP and KV cache management.

### The Bit-Depth Trade-off
*   **Q4_K_M (4-bit):** High speed, lower memory, but can suffer from "drift" in long-context reasoning. MTP is often more "volatile" here because the underlying weights are already compressed.
*   **Q8_0 (8-bit):** Much more stable. If you are seeing "inconsistent" logic, switching from 4-bit to 8-bit (if VRAM allows) is the first thing to try.
*   **FP16/BF16:** The gold standard. If you have the VRAM, always test your baseline in BF16 to see if the "inconsistency" is a result of quantization noise.

### Testing Protocol for Quantization
During your weekend, run one "Golden Prompt" (a complex logic puzzle) through three versions:
1.  **Q4_K_M**
2.  **Q8_0**
3.  **FP16 (if possible)**

Compare the **Accuracy Score** vs. **TPOT**. You will likely find that Q4_K_M is much faster but "hallucinates" more on complex logic, which might be what you are perceiving as "inconsistency."

---

# Hardware-Level Bottleneck Identification

Sometimes the "inconsistency" isn't the model—it's the hardware pipeline.

### 1. PCIe Bandwidth
If you are running a model that is too large for your GPU VRAM, `llama.cpp` will offload layers to the CPU. The speed of this is limited by your PCIe lanes.
*   **The Test:** Run a model that is *just* slightly too large for your VRAM. If the TPOT drops from 50 to 5, you are bottlenecked by PCIe bandwidth.
*   **The Fix:** Either reduce the model size (quantize further) or increase your VRAM.

### 2. Thermal Throttling
Local inference is intensive. If your GPU hits 80°C+, it may downclock.
*   **The Test:** Run a long-output test (1,000+ tokens) and watch your GPU clock speeds in `nvidia-smi`.
*   **The Symptom:** If the clock speed drops mid-generation, your "inconsistency" is a thermal issue.
*   **The Fix:** Improve airflow or set a power limit on the GPU.

### 3. CPU Thread Contention
If you are using `llama.cpp` with high thread counts, your CPU might be fighting with your OS or other background apps.
*   **The Test:** Run the benchmark with `-t 1`, `-t 2`, `-t 4`, and `-t 8`.
*   **The Goal:** Find the "Sweet Spot." Often, using *fewer* threads than you have physical cores results in higher TPOT because it reduces context switching.

---

# Advanced SQLite Queries for Lab Analysis

To truly understand your data, you need to move beyond simple averages. Use these queries to find the "Outliers"—the specific runs that felt "weird."

### Query 4: Identifying "Slow" Outliers
*This finds any run where the TPOT was 50% lower than the model's average.*
```sql
WITH ModelAverages AS (
    SELECT model_name, AVG(tpot_avg) as avg_tpot
    FROM inference_logs
    GROUP BY model_name
)
SELECT l.*
FROM inference_logs l
JOIN ModelAverages ma ON l.model_name = ma.model_name
WHERE l.tpot_avg < (ma.avg_tpot * 0.5)
ORDER BY l.tpot_avg ASC;
```

### Query 5: Context Length vs. Speed Degradation
*This helps you see exactly when the model starts to "choke" as the context fills up.*
```sql
SELECT 
    prompt_tokens, 
    AVG(tpot_avg) as avg_tpot,
    COUNT(*) as sample_count
FROM inference_logs
GROUP BY prompt_tokens
ORDER BY prompt_tokens ASC;
```

### Query 6: MTP Efficiency Score
*Calculates the "Speedup per unit of VRAM" to see if MTP is actually worth the memory cost.*
```sql
SELECT 
    model_name,
    mtp_enabled,
    (AVG(tpot_avg) / NULLIF(AVG(vram_peak), 0)) as efficiency_score
FROM inference_logs
GROUP BY model_name, mtp_enabled
ORDER BY efficiency_score DESC;
```

---

# Automation and Scripting

To make your weekend efficient, don't manually type every prompt. Use a simple Python script to automate the "Golden Set" testing.

### Python Benchmark Runner (Conceptual)
Create a script that iterates through a list of prompts, calls your local server, and logs the results to your SQLite DB.

```python
import requests
import time
import sqlite3
import json

# Configuration
SERVER_URL = "http://localhost:8080/v1/completions"
DB_PATH = "benchmarks.db"
PROMPTS = [
    "Explain quantum entanglement in simple terms.",
    "Write a Python function to sort a list of dictionaries by a nested key.",
    "Write a 500-word story about a robot learning to paint.",
    "Analyze the following text for sentiment: [Insert Long Text]"
]

def run_benchmark(prompt, model_name, mtp_enabled):
    payload = {
        "model": model_name,
        "prompt": prompt,
        "max_tokens": 500,
        "temperature": 0,
        "extra_params": {"mtp": mtp_enabled} # Example param
    }
    
    start_time = time.time()
    response = requests.post(SERVER_URL, json=payload)
    data = response.json()
    end_time = time.time()
    
    # Calculate metrics
    ttft = end_time - start_time # Simplified for example
    completion_tokens = data['usage']['completion_tokens']
    tpot = completion_tokens / (end_time - start_time)
    
    # Log to SQLite
    conn = sqlite3.connect(DB_PATH)
    cursor = conn.cursor()
    cursor.execute("""
        INSERT INTO inference_logs 
        (model_name, prompt_tokens, completion_tokens, ttft_ms, tpot_avg, mtp_enabled)
        VALUES (?, ?, ?, ?, ?, ?)
    """, (model_name, len(prompt), completion_tokens, ttft*1000, tpot, mtp_enabled))
    conn.commit()
    conn.close()

# Run the loop
for p in PROMPTS:
    run_benchmark(p, "llama-3-8b", True)
    run_benchmark(p, "llama-3-8b", False)
```

---

# The "Golden Set" Construction

To avoid "fooling yourself," you must use a **Golden Set**. This is a collection of prompts that are:
1.  **Deterministic:** They should yield similar results every time.
2.  **Diverse:** They must cover different "modes" of the model.

### Your Golden Set should include:
*   **The Logic Test:** "If John has 3 apples and gives 1 to Mary, then Mary gives 2 to Steve, how many apples does Steve have?" (Tests basic reasoning).
*   **The Coding Test:** "Write a Python script to scrape a website and save the titles to a CSV." (Tests syntax and structure).
*   **The Summarization Test:** Provide a 1,000-word news article and ask for a 3-bullet summary. (Tests context window and extraction).
*   **The Creative Test:** "Write a poem about a toaster in the style of Edgar Allan Poe." (Tests stylistic adherence).
*   **The "Refusal" Test:** Ask for something slightly controversial or restricted to see how the safety filters impact latency.

**Rule:** Only compare models/settings using this exact set. If you change the prompt, you change the variables.

---

# Deep Dive into MTP Mechanics

Since you specifically asked about MTP, let's clarify what it is doing to your hardware.

### How MTP Works
Standard inference predicts token $N+1$ based on tokens $1 \dots N$. 
MTP (Multi-Token Prediction) attempts to predict tokens $N+1, N+2, \dots, N+k$ simultaneously.

### Why it feels "Inconsistent"
1.  **The "Lookahead" Penalty:** To predict $N+2$, the model has to perform extra calculations. If your GPU's compute units are saturated, this extra work can cause a "stutter" in the generation of the *current* token.
2.  **The "Drift" Effect:** Because the model is "guessing" the next few tokens, if it guesses a word that is slightly off-path, the subsequent tokens will be based on that error. This can lead to "hallucination loops" where the model gets stuck repeating a phrase because it predicted that phrase two tokens ago.

### When to use MTP vs. Non-MTP
*   **Use MTP when:** You are doing **Creative Writing** or **Chat**. In these modes, "flow" is more important than perfect logic. MTP helps the model maintain a coherent "sentence structure" faster.
*   **Avoid MTP when:** You are doing **Coding** or **Math**. In these modes, a single wrong character (like a missing bracket or a wrong digit) ruins the entire output. The "predictive" nature of MTP can introduce these small but fatal errors.

---

# Final Synthesis: The "Model Use-Case Matrix"

After your weekend of testing, you should be able to fill out this matrix. This is your "Source of Truth."

| Model + Config | Best Use Case | Why? | MTP Status |
| :--- | :--- | :--- | :--- |
| **Llama-3-8B-Q8_0** | **Coding / RAG** | High stability, low jitter, perfect logic. | OFF |
| **Mistral-7B-Q4_K_M** | **Creative Writing** | High TPOT, good "flow" for prose. | ON |
| **DeepSeek-R1-Distill** | **Complex Reasoning** | High "Reasoning Budget" required. | OFF |
| **Phi-3-Mini** | **Fast Chat** | Extremely high TPOT, low VRAM footprint. | ON |

### How to read this matrix:
*   **Coding:** If a model has high "Jitter" (Standard Deviation) in your SQLite results, it is **not** suitable for coding, as it will produce inconsistent syntax.
*   **RAG:** If a model's TPOT drops significantly after 2,000 tokens, it is **not** suitable for RAG, as it will struggle with long context retrieval.
*   **Agentic:** If a model's "Reasoning Budget" (Thought Tokens) doesn't correlate with higher accuracy, it is **not** suitable for agents.

---

# Final Weekend Workflow Summary

1.  **Setup:** Install `nvidia-smi` logging and your SQLite schema.
2.  **Baseline:** Run the "Golden Set" with MTP OFF and Q8_0 quantization.
3.  **MTP Test:** Run the "Golden Set" with MTP ON.
4.  **Stress Test:** Run the "Long Output" test to find the "Slowdown Curve."
5.  **Analyze:** Run the SQL queries to find the "Jitter" and "Efficiency Score."
6.  **Finalize:** Fill out your Use-Case Matrix and stick it on your wall (or keep it in your notes).

By following this, you move from "I think my model is slow" to "My model has a 15% TPOT drop at 3,000 tokens due to KV Cache pressure, and MTP adds 5% speed but introduces a 2% logic error rate in coding tasks." **That is the power of data.**

---

# llama.cpp Parameter Deep Dive: Tuning the Engine

To solve "inconsistency," you must move beyond the default settings. `llama.cpp` provides several levers that directly impact how the model interacts with your hardware.

### 1. `n_batch` and `n_ubatch`
These are the most critical parameters for "Prefill" (the first token) speed.
*   **`n_batch`**: This defines how many tokens are processed in a single forward pass. If your prompt is 1,000 tokens and `n_batch` is 512, the model takes two passes to "read" your prompt.
*   **`n_ubatch`**: This is the "micro-batch" size. It determines how many tokens are processed at once *within* a batch.
*   **The Optimization:** For high-end GPUs, increasing `n_batch` to 2048 or higher can significantly speed up the first token. However, if it's too high, you might run out of VRAM during the prefill phase.
*   **The Test:** Run your "Golden Set" with `n_batch=512` and then `n_batch=2048`. Compare the `ttft_ms`.

### 2. `threads`
This determines how many CPU threads are used for the non-GPU parts of the inference (like the KV cache management and prompt processing).
*   **The Rule of Thumb:** Do not set this to your total core count. Often, setting it to the number of *physical* cores (not logical/hyperthreaded cores) yields the best results.
*   **The Test:** If you have a 16-thread CPU, test `-t 8`, `-t 12`, and `-t 16`. You will often find a "sweet spot" where the speed is highest before the overhead of thread management causes a drop.

### 3. `mmap` vs. `no_mmap`
*   **`mmap`**: Maps the model file into memory. It's faster to start but can be slower if the OS has to swap pages.
*   **`no_mmap`**: Loads the entire model into RAM/VRAM immediately.
*   **The Recommendation:** For a home lab where you want consistent performance, use `no_mmap` (or ensure your system has enough overhead) to ensure the model is fully resident in memory before the first request hits.

---

# MoE (Mixture of Experts) vs. Dense Models

If you are switching between models like **Mixtral (MoE)** and **Llama (Dense)**, your "inconsistency" might be architectural.

### The MoE Behavior
MoE models only activate a fraction of their parameters for each token.
*   **The VRAM Trap:** Even though only a few "experts" are active, the *entire* model must reside in VRAM. This means a Mixtral 8x7B model requires massive VRAM but might have a faster TPOT than a dense model of the same total parameter count.
*   **The Jitter:** MoE models can sometimes exhibit "jitter" because the routing logic (deciding which expert to use) can vary slightly between tokens, leading to minor fluctuations in compute time.

### The Dense Behavior
Dense models use all parameters for every token.
*   **The Predictability:** These are much more "stable" in their performance profiles. If a dense model is slow, it is consistently slow.
*   **The Comparison:** When benchmarking, always compare MoE to MoE and Dense to Dense. Comparing a Mixtral 8x7B to a Llama 3 70B is an "apples to oranges" comparison regarding memory management.

---

# Advanced SQL Queries for Lab Analysis

To truly find the "ghosts in the machine," we need to look at percentiles. Averages hide the spikes; percentiles reveal them.

### Query 7: Percentile Analysis (P95 and P99)
*This identifies the "worst-case" scenarios. If your P99 is 5x higher than your average, you have a significant "hiccup" problem.*
```sql
SELECT 
    model_name,
    AVG(tpot_avg) as avg_tpot,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY tpot_avg) as p95_tpot,
    PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY tpot_avg) as p99_tpot
FROM inference_logs
GROUP BY model_name;
```
*(Note: Percentile syntax may vary slightly depending on your SQLite version/extension; some may require a subquery).*

### Query 8: Correlation between Context Length and Jitter
*This helps you see if the model becomes "unstable" as the conversation gets longer.*
```sql
SELECT 
    prompt_tokens,
    STDDEV(tpot_avg) as tpot_stddev,
    AVG(tpot_avg) as avg_tpot
FROM inference_logs
GROUP BY prompt_tokens
ORDER BY prompt_tokens ASC;
```

---

# Agentic Workflow Benchmarking

If you plan to use these models for agents (e.g., AutoGPT, LangChain agents), "Tokens per second" is the wrong metric. You need to measure **Loop Latency**.

### Defining Loop Latency
Loop Latency is the time it takes for the model to:
1.  Receive a task.
2.  "Think" (Reasoning tokens).
3.  Produce a tool call or action.
4.  Return the result.

### How to Benchmark Agents
1.  **The "Action" Test:** Give the model a task that requires 3 tool calls (e.g., "Check the weather in London, find a restaurant there, and book a table").
2.  **Measure:** Total time from "Start" to "Final Answer."
3.  **The Metric:** `Actions Per Minute (APM)`.
4.  **The Goal:** You want a model that can complete 3-4 actions per minute reliably. If a model is "fast" but takes 10 minutes to complete the loop because it keeps hallucinating the tool syntax, it is a failure for agentic work.

---

# RAG (Retrieval-Augmented Generation) Benchmarking

For RAG, the "inconsistency" usually happens during the **Context Injection** phase.

### The "Needle in a Haystack" Test
To benchmark your RAG setup:
1.  **The Setup:** Create a 4,000-token document.
2.  **The Needle:** Place a specific, unique fact (e.g., "The secret code is Blue-Elephant-99") in the middle (around token 2,000).
3.  **The Query:** Ask the model: "What is the secret code?"
4.  **The Metric:** **Retrieval Accuracy.**
    *   Run this 10 times.
    *   If the model fails 3/10 times, your `n_ctx` is too small, or your KV cache is "dropping" information.

### Context Compression vs. Full Context
*   **Full Context:** Pass the entire retrieved chunk. (High VRAM, High Accuracy).
*   **Compressed Context:** Use a smaller model to summarize the chunk before passing it to the main model. (Low VRAM, Lower Accuracy).
*   **The Benchmark:** Compare the **Accuracy vs. TPOT** for both methods. Often, "Compressed Context" is the only way to maintain high TPOT on consumer hardware.

---

# Troubleshooting Flowchart (Text-Based)

If you see a "Slowdown" or "Jump" during your weekend tests, follow this logic:

**Step 1: Is the TPOT drop constant or intermittent?**
*   *Constant:* Your `n_ctx` is too high for your VRAM. **Action:** Reduce `n_ctx` or use a more aggressive quantization (e.g., Q4_K_S).
*   *Intermittent:* You are hitting a "Page Fault" or "Swap." **Action:** Check system RAM usage. If it spikes, your GPU is offloading to system memory.

**Step 2: Does the "Jump" happen only during the first token?**
*   *Yes:* This is a Prefill bottleneck. **Action:** Increase `n_batch` and `n_ubatch`.
*   *No:* This is a Decode bottleneck. **Action:** Check for thermal throttling or CPU thread contention.

**Step 3: Does the "Jump" happen only during long outputs?**
*   *Yes:* This is KV Cache fragmentation. **Action:** Use `flash_attn` or a fixed `n_ctx` with a larger `cache_16bit` (if available).

---

# Prompt Engineering for Benchmarking

To get "clean" data, you must avoid "Prompt Drift." Use these three specific prompt types for your weekend tests:

### 1. The "Zero-Shot" Logic Prompt
> "A man is looking at a portrait. He says, 'Brothers and sisters have I none, but this man's father is my father's son.' Who is in the portrait?"
*   *Purpose:* Tests pure reasoning without "fluff."

### 2. The "Structured Output" Prompt
> "List the top 5 cities in Europe by population. Return the result in JSON format with keys 'city' and 'population'."
*   *Purpose:* Tests the model's ability to follow constraints. Inconsistency here usually means the model is "drifting" out of the JSON format.

### 3. The "Long-Form" Creative Prompt
> "Write a 500-word story about a clockmaker who discovers a way to pause time for five seconds every hour. Describe the atmosphere of his workshop in detail."
*   *Purpose:* Tests TPOT stability over a long generation and the "flow" of the prose.

---

# Final Summary of Metrics for your Dashboard

When you finish your weekend, your "Final Report" should contain these five numbers for every model you use:

1.  **TTFT (Time to First Token):** How "snappy" the model feels.
2.  **TPOT (Tokens Per Second):** How fast the "typing" is.
3.  **Jitter (StdDev of TPOT):** How "smooth" the experience is.
4.  **Reasoning Efficiency:** (Accuracy / Thought Tokens).
5.  **VRAM Ceiling:** The maximum context length before the model "breaks" or "swaps."

**The Ultimate Goal:** You aren't looking for the "fastest" model. You are looking for the model that provides the **highest Accuracy** at a **TPOT > 20** with a **Jitter < 5%**. If you find that, you have won the home-lab game.