# Quick Diagnosis

When you describe a local `llama.cpp` server that feels inconsistent, exhibits slow first answers, shows unpredictable GPU memory jumps, and leaves you uncertain about whether Multi-Token Prediction (MTP) is actually helping, you are encountering three well-documented phenomena in local LLM inference: cold-start latency, dynamic VRAM allocation behavior, and speculative decoding overhead. Let's break down what is happening under the hood so you can approach your weekend testing with clarity.

**Slow First Answer (Cold Start)**
The first request after server startup or after a long idle period triggers several initialization steps:
1. **Model Loading & Tensor Initialization:** Even if the model is already loaded, the first request often triggers GPU context creation, CUDA/Metal/ROCm runtime initialization, and kernel compilation for your specific hardware.
2. **KV Cache Allocation:** `llama.cpp` allocates the key-value cache dynamically based on `--ctx-size`. The first prompt forces the allocator to reserve VRAM, which can take hundreds of milliseconds to seconds depending on your GPU and quantization.
3. **Prompt Processing:** The entire prompt is processed through the model to build the KV cache. This is computationally expensive and scales linearly with prompt length.
4. **MTP Draft Network Warmup:** If MTP is enabled, the draft model must also initialize its weights and buffers. This adds overhead to the first token generation.

**GPU Memory Jumps**
VRAM usage in `llama.cpp` is not static. It fluctuates due to:
- **Dynamic Context Expansion:** As conversations grow, the KV cache expands. If you exceed `--ctx-size`, `llama.cpp` may shift older tokens to CPU RAM or trigger reallocation, causing visible VRAM spikes.
- **MTP Buffer Allocation:** MTP requires additional VRAM for draft token buffers and verification steps. The memory footprint scales with `--mtp-max-steps`.
- **Batch Processing:** If your proxy or OpenWebUI queues requests, `llama.cpp` may process them in batches, temporarily increasing VRAM usage for intermediate activations.
- **Fragmentation:** Repeated allocation/deallocation of KV cache blocks can fragment VRAM, causing the allocator to request more memory than theoretically required.

**MTP Uncertainty**
Multi-Token Prediction (speculative decoding) works by using a smaller draft model to predict multiple tokens ahead, then verifying them with the main model. If the draft tokens are correct, you save decode steps. If they are wrong, you incur verification overhead. MTP helps when:
- The draft model aligns well with the main model's token distribution.
- The hit rate (percentage of accepted draft tokens) exceeds ~50-60%.
- The verification overhead is lower than the time saved by skipping decode steps.

MTP hurts when:
- The hit rate is low (<40%), causing frequent rejections and extra compute.
- The draft model is too large or poorly quantized, adding VRAM pressure.
- The prompt is highly variable or creative, reducing draft accuracy.

**Immediate Diagnostic Steps**
Before diving into data collection, run these quick checks:
1. Verify your `llama-server` flags: `--ctx-size`, `--gpu-layers`, `--mtp-speculative`, `--mtp-max-steps`, `--cache-type-k`.
2. Check VRAM limits: Ensure `--mlock` and `--no-mmap` are set appropriately for your hardware.
3. Disable background processes that compete for GPU/CPU resources.
4. Run a single fixed prompt twice: once cold, once warm. Compare `time_per_token` and `first_token_time`.

Understanding these mechanics will prevent you from misinterpreting normal behavior as inconsistency. The rest of this guide will show you how to measure, compare, and optimize your setup systematically.

# Data Collection Plan

To diagnose inconsistency and evaluate MTP objectively, you need a structured data collection pipeline. Your stack (Flight Recorder proxy, ServerTop dashboard, SQLite reports) is well-suited for this. Here's how to configure it for reproducible, high-fidelity measurements.

**1. Flight Recorder Proxy Configuration**
The Flight Recorder proxy should intercept all `/v1/chat/completions` requests and log structured metadata. Configure it to capture:
- `request_id`: Unique identifier for each request.
- `timestamp_start` & `timestamp_end`: ISO 8601 timestamps.
- `prompt_tokens`: Number of tokens in the input.
- `completion_tokens`: Number of tokens generated.
- `total_time_ms`: End-to-end latency.
- `first_token_time_ms`: Time to first token (TTFT).
- `time_per_token_ms`: Average generation latency.
- `mtp_hits`: Number of draft tokens accepted.
- `mtp_rejected`: Number of draft tokens rejected.
- `gpu_mem_peak_mb`: Peak VRAM usage during request.
- `status_code`: HTTP response code.
- `streaming`: Boolean indicating streaming vs. non-streaming.

Set the proxy to log to a JSON file or directly insert into SQLite. Enable request replay capability for consistent testing.

**2. ServerTop Dashboard Setup**
ServerTop should visualize real-time metrics. Configure panels for:
- **Request Latency:** TTFT, total time, time_per_token (line chart, 15s resolution).
- **VRAM Usage:** Current vs. peak, with threshold lines at 70%, 85%, 95%.
- **MTP Metrics:** Hit rate, rejection rate, draft tokens/sec.
- **System Load:** CPU usage, RAM usage, GPU utilization, disk I/O.
- **Error Rate:** 4xx/5xx responses, timeouts, OOM events.

Set data retention to 7 days. Enable annotations for test runs (e.g., "MTP ON", "Long Output Test", "Cold Start").

**3. SQLite Schema Design**
Create a normalized schema to store historical data:
```sql
CREATE TABLE requests (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    request_id TEXT UNIQUE,
    timestamp_start TEXT,
    timestamp_end TEXT,
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    total_time_ms REAL,
    first_token_time_ms REAL,
    time_per_token_ms REAL,
    mtp_hits INTEGER DEFAULT 0,
    mtp_rejected INTEGER DEFAULT 0,
    gpu_mem_peak_mb REAL,
    status_code INTEGER,
    streaming BOOLEAN,
    test_label TEXT
);

CREATE TABLE system_metrics (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT,
    gpu_mem_used_mb REAL,
    cpu_mem_used_mb REAL,
    gpu_util_percent REAL,
    cpu_load_avg REAL,
    disk_io_mbps REAL
);

CREATE INDEX idx_requests_timestamp ON requests(timestamp_start);
CREATE INDEX idx_requests_test_label ON requests(test_label);
CREATE INDEX idx_system_metrics_timestamp ON system_metrics(timestamp);
```

**4. Synthetic Load Generation**
Use `curl` to generate controlled traffic. Avoid relying on manual OpenWebUI interactions for benchmarking.

*Cold Start Test:*
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [{"role": "user", "content": "Explain quantum entanglement in three paragraphs."}],
    "max_tokens": 500,
    "temperature": 0.7,
    "stream": false
  }' \
  -o /dev/null -w "TTFT: %{time_starttransfer} Total: %{time_total}\n"
```

*Warm Start Test (repeat 10x):*
```bash
for i in {1..10}; do
  curl -s -X POST http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"local-model","messages":[{"role":"user","content":"Summarize the following text: [FIXED 1000-WORD TEXT]"}],"max_tokens":300,"temperature":0.2,"stream":false}' \
    -o /dev/null -w "Run $i: TTFT=%{time_starttransfer} Total=%{time_total}\n"
done
```

*Streaming Test:*
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Write a detailed technical guide about distributed systems."}],"max_tokens":2000,"temperature":0.8,"stream":true}' \
  | grep -o '"delta":{[^}]*}' | head -20
```

**5. Data Collection Protocol**
- Run each test configuration 30 times to establish statistical significance.
- Record system state before each run (GPU temp, background processes, VRAM free).
- Use fixed seeds (`"seed": 42`) for reproducibility.
- Log all results to SQLite via Flight Recorder.
- Tag each run with `test_label` (e.g., `baseline_cold`, `mtp_on_warm`, `long_output_8k`).

This plan ensures you capture high-resolution, reproducible data without manual bias. The next section will show you how to design a fair MTP comparison using this data.

# MTP Comparison Plan

Multi-Token Prediction (MTP) is a powerful optimization, but it is not universally beneficial. To determine whether MTP helps your specific model and workload, you must design a controlled A/B test that isolates MTP as the only variable. Here's how to do it without fooling yourself.

**1. Understanding MTP in llama.cpp**
MTP uses a draft model (often the same model with fewer layers or a smaller quantization) to predict multiple tokens ahead. The main model then verifies these tokens in a single forward pass. If accepted, you skip individual decode steps. If rejected, you fall back to standard decoding. Key parameters:
- `--mtp-speculative <draft_model.gguf>`: Path to the draft model.
- `--mtp-max-steps`: Maximum number of draft tokens to predict per step (default: 4).
- `--mtp-tensor-split`: GPU/CPU split for draft model (if applicable).

**2. Test Design Principles**
- **Single Variable:** Only change MTP configuration. Keep `--ctx-size`, `--gpu-layers`, `--batch-size`, temperature, top_p, and prompt identical.
- **Fixed Workload:** Use a curated prompt set covering different domains (technical, creative, conversational, code).
- **Repetition:** Run each configuration 30+ times to capture variance.
- **Cold vs Warm:** Test both cold start and warm start conditions separately.
- **Output Length:** Test short (500 tokens), medium (2000 tokens), and long (8000 tokens) generations.

**3. Execution Steps**
*Step 1: Baseline (MTP OFF)*
```bash
llama-server -m main_model.gguf --ctx-size 8192 --gpu-layers 35 --threads 8 --log-disable
```
Run synthetic load tests. Record all metrics to SQLite with `test_label='mtp_off'`.

*Step 2: MTP ON*
```bash
llama-server -m main_model.gguf --mtp-speculative draft_model.gguf --mtp-max-steps 4 --ctx-size 8192 --gpu-layers 35 --threads 8 --log-disable
```
Run identical synthetic load tests. Record with `test_label='mtp_on'`.

*Step 3: Parameter Sweep*
Test `--mtp-max-steps` values: 2, 4, 6, 8. Record as `mtp_on_steps_2`, etc.

**4. Metrics to Compare**
- **Throughput:** `completion_tokens / total_time_ms` (tokens/sec).
- **Latency:** `first_token_time_ms`, `time_per_token_ms`.
- **MTP Efficiency:** `mtp_hits / (mtp_hits + mtp_rejected)` (hit rate).
- **VRAM Impact:** `gpu_mem_peak_mb` difference.
- **Consistency:** Standard deviation and 90th percentile of latency.

**5. Statistical Analysis**
Do not rely on averages. Use:
- Median latency (robust to outliers).
- 90th percentile (represents typical user experience).
- Confidence intervals (95% CI for mean difference).
- Paired t-test or Wilcoxon signed-rank test for significance.

Example calculation:
```
MTP OFF median TPT: 12.5 ms
MTP ON median TPT: 10.8 ms
Hit rate: 62%
VRAM increase: +1.2 GB
Conclusion: MTP improves throughput by ~13.6% with acceptable VRAM cost.
```

**6. When MTP Fails**
- Hit rate < 40%: Draft model mismatch or overly creative prompts.
- VRAM spike > 2GB: `--mtp-max-steps` too high or draft model too large.
- Increased TTFT: Draft network initialization overhead outweighs decode savings.
- Inconsistent TPT: Context fragmentation or cache invalidation.

**7. Decision Framework**
- **Keep MTP ON if:** Hit rate > 55%, TPT improvement > 10%, VRAM increase < 1.5GB, consistency improves.
- **Tune MTP if:** Hit rate 40-55%, try lower `--mtp-max-steps` or different draft quantization.
- **Disable MTP if:** Hit rate < 40%, VRAM pressure causes OOM, or latency variance increases.

This plan ensures you evaluate MTP objectively, avoiding the common trap of judging performance based on a single run or subjective feel. The next section covers how to manage your reasoning budget to prevent VRAM and context issues.

# Reasoning Budget Plan

"Reasoning budget" refers to how you allocate and manage context window, KV cache, VRAM, and token limits across different workloads. Poor budget management is the primary cause of GPU memory jumps, slow first answers, and inconsistent performance. Here's how to plan and enforce it.

**1. KV Cache & Context Window**
The KV cache stores attention keys and values for every token in the context. Its size scales with `--ctx-size` and model dimensions.
- **Static Allocation:** `llama.cpp` pre-allocates VRAM for the KV cache based on `--ctx-size`. If you set `--ctx-size 32768`, it reserves memory for 32k tokens regardless of actual usage.
- **Dynamic Behavior:** As conversations grow, the cache fills. When full, `llama.cpp` may shift older tokens to CPU RAM (`--no-ctx-shift` disables this) or truncate, causing latency spikes.
- **Budget Rule:** Set `--ctx-size` to 1.5x your expected maximum context. For chat: 8k. For RAG: 16k-32k. For coding: 32k-64k.

**2. VRAM Budgeting**
VRAM is consumed by:
- Model weights (quantization-dependent).
- KV cache (context-dependent).
- Activations & buffers (batch size, MTP, draft model).
- OS & driver overhead.

Calculate your budget:
```
Total VRAM = Model Weights + KV Cache + MTP Buffers + Safety Margin (10%)
```
Example for 13B Q4_K_M on 24GB GPU:
- Weights: ~7.5 GB
- KV Cache (16k ctx): ~4.0 GB
- MTP Buffers: ~1.5 GB
- Safety: ~2.0 GB
- Total: ~15.0 GB (safe)

If you exceed budget, you'll see VRAM jumps, CPU offloading, or OOM crashes.

**3. Prompt Caching Strategy**
OpenWebUI and `llama.cpp` support prompt caching to avoid reprocessing identical prefixes.
- **Enable:** `--prompt-cache-all` or `--prompt-cache-ro`.
- **Cache Type:** `--cache-type-k q8_0` (better accuracy, higher VRAM) or `q4_0` (lower VRAM, slight accuracy loss).
- **Invalidation:** Cache breaks if system prompt changes, temperature changes, or context exceeds cache size.
- **Budget Rule:** Reserve 20% of KV cache for prompt cache. Monitor cache hit rate via Flight Recorder.

**4. Token Budgeting by Use Case**
- **Chat:** Prompt: 500-2k tokens. Output: 200-800 tokens. Budget: 4k ctx. MTP: Marginal benefit.
- **RAG:** Prompt: 4k-16k tokens (retrieved context). Output: 200-600 tokens. Budget: 16k-32k ctx. MTP: Low benefit (prompt-heavy). Prompt caching critical.
- **Coding:** Prompt: 2k-8k tokens (codebase). Output: 500-2k tokens. Budget: 32k ctx. MTP: High benefit if hit rate >50%.
- **Agentic:** Multi-turn, tool calls, variable context. Budget: 16k-32k ctx. MTP: Risky (unpredictable context shifts). Prioritize consistency.
- **Creative Writing:** Prompt: 1k-4k tokens. Output: 2k-8k tokens. Budget: 8k-16k ctx. MTP: Moderate benefit, depends on draft model alignment.

**5. Monitoring & Alerts**
Configure ServerTop to alert when:
- VRAM usage > 85%
- Context utilization > 80%
- Prompt cache hit rate < 30%
- MTP hit rate < 45%

**6. Tuning Parameters**
- `--gpu-layers`: Set to max possible without OOM. Use `--verbose` to see layer allocation.
- `--batch-size`: Increase for throughput, decrease for latency. Default: 512.
- `--threads` & `--threads-batch`: Match CPU cores. Avoid oversubscription.
- `--mlock`: Prevents swapping, improves consistency.
- `--no-mmap`: Forces full model load into RAM/VRAM, reduces cold start variance.

**7. Practical Implementation**
Create a configuration matrix:
| Use Case | --ctx-size | --gpu-layers | --cache-type-k | --mtp-max-steps | Expected VRAM |
|----------|------------|--------------|----------------|-----------------|---------------|
| Chat     | 8192       | 35           | q8_0           | 0               | 12 GB         |
| RAG      | 16384      | 30           | q4_0           | 0               | 14 GB         |
| Coding   | 32768      | 25           | q8_0           | 4               | 18 GB         |
| Agentic  | 16384      | 30           | q8_0           | 2               | 15 GB         |
| Creative | 8192       | 35           | q8_0           | 4               | 13 GB         |

Test each configuration with your synthetic load. Adjust based on VRAM headroom and latency targets. This budget plan prevents memory jumps and ensures consistent performance across workloads.

# Long Output Test Plan

Long-form generation (4k+ tokens) stresses your system differently than short responses. It reveals VRAM leaks, KV cache fragmentation, MTP degradation, and thermal throttling. Here's how to test it systematically.

**1. Test Objectives**
- Measure token-per-second consistency over long generations.
- Detect VRAM leaks or fragmentation.
- Evaluate MTP hit rate stability.
- Identify thermal or power throttling.
- Validate context management and cache behavior.

**2. Prompt Design**
Use prompts that force deterministic, long outputs:
- Technical: "Write a comprehensive guide to building a REST API with FastAPI, including authentication, database integration, testing, and deployment. Continue until 8000 tokens."
- Creative: "Write a detailed fantasy novel chapter about a scholar discovering an ancient library. Include dialogue, world-building, and character development. Continue until 6000 tokens."
- Code: "Generate a complete Python module for a distributed task queue with producer, consumer, broker, and monitoring components. Include type hints, docstrings, and error handling. Continue until 5000 tokens."

**3. Execution Protocol**
Run each prompt with:
- `max_tokens`: 4000, 8000, 12000
- `temperature`: 0.7 (consistent)
- `stream`: true (to monitor real-time TPT)
- `seed`: 42 (reproducibility)

Curl example:
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [{"role": "user", "content": "Write a comprehensive guide to building a REST API with FastAPI..."}],
    "max_tokens": 8000,
    "temperature": 0.7,
    "stream": true,
    "seed": 42
  }' \
  | python3 -c "
import sys, json, time
start = time.time()
tokens = 0
for line in sys.stdin:
    if line.startswith('data: '):
        data = json.loads(line[6:])
        if 'choices' in data and data['choices'][0]['delta']:
            tokens += 1
            if tokens % 1000 == 0:
                elapsed = time.time() - start
                print(f'Token {tokens}: {elapsed:.2f}s, TPT: {1000/elapsed:.2f} tok/s')
"
```

**4. Metrics to Monitor**
- **TPT Curve:** Plot tokens/sec over time. Healthy: flat or slight decline. Unhealthy: sharp drops, oscillations.
- **VRAM Trend:** Should stabilize after initial allocation. Leaks show linear increase.
- **MTP Hit Rate:** Should remain consistent. Degradation indicates draft model mismatch or context shift.
- **GPU Utilization:** Should stay high (>80%). Drops indicate CPU bottleneck or throttling.
- **Temperature/Power:** Monitor via `nvidia-smi` or `rocm-smi`. Throttling causes TPT drops.

**5. Failure Modes & Fixes**
- **TPT Drops at 4k+:** KV cache fragmentation. Fix: `--cache-type-k q4_0`, reduce `--ctx-size`, enable `--no-ctx-shift`.
- **VRAM Leak:** Driver or llama.cpp bug. Fix: Restart server, update llama.cpp, reduce `--batch-size`.
- **MTP Collapse:** Draft model fails on long context. Fix: Lower `--mtp-max-steps`, switch to `--mtp-speculative` with smaller model.
- **Thermal Throttling:** GPU cooling insufficient. Fix: Improve airflow, undervolt, reduce `--threads`.

**6. Analysis Protocol**
- Run 3 repetitions per length.
- Calculate median TPT for each 1k-token segment.
- Plot TPT vs. token count.
- Compare MTP ON vs OFF curves.
- Document VRAM peak and stability.

**7. Decision Criteria**
- **Pass:** TPT variance < 15%, VRAM stable, MTP hit rate > 50%, no throttling.
- **Tune:** TPT variance 15-30%, adjust cache type, reduce MTP steps, optimize cooling.
- **Fail:** TPT variance > 30%, VRAM leak, MTP hit rate < 40%, frequent OOM. Disable MTP, reduce context, upgrade hardware.

This plan ensures you stress-test your system realistically and identify bottlenecks before they impact production workloads. The next section provides SQL queries to analyze your collected data.

# SQLite Queries

Your SQLite database is the foundation for objective analysis. Below are production-ready queries to extract insights from your Flight Recorder logs. Assume the schema defined in the Data Collection Plan.

**1. Latency Percentiles (P50, P90, P99)**
```sql
SELECT 
    test_label,
    PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY time_per_token_ms) AS p50_tpt,
    PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY time_per_token_ms) AS p90_tpt,
    PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY time_per_token_ms) AS p99_tpt,
    COUNT(*) AS runs
FROM requests
WHERE status_code = 200
GROUP BY test_label
ORDER BY test_label;
```
*Interpretation:* P90 represents typical user experience. P99 reveals tail latency. Compare MTP ON vs OFF.

**2. MTP Hit Rate Over Time**
```sql
SELECT 
    test_label,
    AVG(CAST(mtp_hits AS REAL) / NULLIF(mtp_hits + mtp_rejected, 0)) AS avg_hit_rate,
    MIN(CAST(mtp_hits AS REAL) / NULLIF(mtp_hits + mtp_rejected, 0)) AS min_hit_rate,
    MAX(CAST(mtp_hits AS REAL) / NULLIF(mtp_hits + mtp_rejected, 0)) AS max_hit_rate
FROM requests
WHERE mtp_hits > 0 OR mtp_rejected > 0
GROUP BY test_label;
```
*Interpretation:* Hit rate < 40% indicates MTP is hurting performance. > 60% is ideal.

**3. VRAM Usage vs Context Size**
```sql
SELECT 
    test_label,
    AVG(prompt_tokens + completion_tokens) AS avg_context,
    AVG(gpu_mem_peak_mb) AS avg_vram_peak,
    MAX(gpu_mem_peak_mb) AS max_vram_peak
FROM requests
GROUP BY test_label
ORDER BY avg_context;
```
*Interpretation:* Linear VRAM growth is normal. Exponential growth indicates fragmentation or leaks.

**4. Token Generation Rate by Output Length**
```sql
SELECT 
    CASE 
        WHEN completion_tokens < 500 THEN 'Short'
        WHEN completion_tokens < 2000 THEN 'Medium'
        WHEN completion_tokens < 8000 THEN 'Long'
        ELSE 'Very Long'
    END AS length_bucket,
    test_label,
    AVG(completion_tokens / (total_time_ms / 1000.0)) AS avg_tok_per_sec,
    STDDEV(completion_tokens / (total_time_ms / 1000.0)) AS std_tok_per_sec
FROM requests
WHERE status_code = 200
GROUP BY length_bucket, test_label
ORDER BY length_bucket, test_label;
```
*Interpretation:* Lower std indicates consistency. Compare buckets to find optimal workload range.

**5. Cold Start vs Warm Start Comparison**
```sql
SELECT 
    CASE 
        WHEN ROW_NUMBER() OVER (PARTITION BY test_label ORDER BY timestamp_start) = 1 THEN 'Cold'
        ELSE 'Warm'
    END AS start_type,
    test_label,
    AVG(first_token_time_ms) AS avg_ttft,
    AVG(time_per_token_ms) AS avg_tpt
FROM requests
GROUP BY start_type, test_label;
```
*Interpretation:* Cold TTFT should be 2-5x warm TTFT. If >10x, check GPU initialization or model loading.

**6. Error Rate & Timeout Analysis**
```sql
SELECT 
    test_label,
    COUNT(*) AS total_runs,
    SUM(CASE WHEN status_code >= 400 THEN 1 ELSE 0 END) AS errors,
    SUM(CASE WHEN total_time_ms > 30000 THEN 1 ELSE 0 END) AS timeouts,
    AVG(total_time_ms) AS avg_latency
FROM requests
GROUP BY test_label;
```
*Interpretation:* Error rate > 5% indicates instability. Timeouts suggest VRAM pressure or throttling.

**7. Export to CSV for Plotting**
```sql
.mode csv
.headers on
.output latency_report.csv
SELECT timestamp_start, test_label, time_per_token_ms, mtp_hits, mtp_rejected, gpu_mem_peak_mb
FROM requests
WHERE test_label IN ('mtp_on', 'mtp_off')
ORDER BY timestamp_start;
```

Run these queries after each test phase. Store results in a markdown report. Use the outputs to configure your dashboard and make tuning decisions. The next section defines what to capture in your dashboard.

# Dashboard Screenshots To Capture

Your ServerTop dashboard should tell a story at a glance. Capture screenshots at key milestones to document behavior and share with your team. Below are the exact panels, configurations, and annotations to capture.

**1. Request Latency Over Time**
- **Panel:** Line chart, Y-axis: `time_per_token_ms`, X-axis: Time.
- **Configuration:** 15s resolution, overlay `first_token_time_ms` as secondary axis.
- **Annotations:** Mark test starts ("MTP OFF Baseline", "MTP ON Test", "Long Output 8k").
- **What to Look For:** Flat lines indicate consistency. Spikes indicate cache misses, VRAM pressure, or thermal throttling.
- **Screenshot Timing:** Capture after 1 hour of mixed workload.

**2. VRAM Usage vs Context Size**
- **Panel:** Scatter plot, X-axis: `prompt_tokens + completion_tokens`, Y-axis: `gpu_mem_peak_mb`.
- **Configuration:** Color by `test_label`. Add linear regression line.
- **Annotations:** Mark `--ctx-size` limit, OOM events.
- **What to Look For:** Linear trend is healthy. Outliers indicate fragmentation. Slope > 0.5 MB/token suggests inefficient cache.
- **Screenshot Timing:** Capture after completing all context size tests.

**3. MTP Hit Rate vs Throughput**
- **Panel:** Bubble chart, X-axis: `mtp_hit_rate`, Y-axis: `tokens_per_second`, Bubble size: `completion_tokens`.
- **Configuration:** Filter `mtp_hits > 0`. Add threshold line at 50% hit rate.
- **Annotations:** Mark `--mtp-max-steps` values.
- **What to Look For:** Bubbles above 50% line with high throughput indicate optimal MTP. Below 40% indicates overhead.
- **Screenshot Timing:** Capture after MTP parameter sweep.

**4. Token/Sec Distribution**
- **Panel:** Histogram, X-axis: `time_per_token_ms`, Y-axis: Count.
- **Configuration:** Overlay kernel density estimate. Add mean, median, p90 lines.
- **Annotations:** Mark cold vs warm runs.
- **What to Look For:** Narrow distribution indicates consistency. Long right tail indicates occasional stalls.
- **Screenshot Timing:** Capture after 30+ runs per configuration.

**5. System Load & Thermal**
- **Panel:** Multi-line chart, Y-axis: GPU util, CPU load, GPU temp, VRAM used.
- **Configuration:** 10s resolution. Add threshold lines (GPU temp > 80°C, VRAM > 90%).
- **Annotations:** Mark throttling events, fan speed changes.
- **What to Look For:** Correlation between temp spikes and TPT drops indicates throttling. VRAM spikes with low util indicates fragmentation.
- **Screenshot Timing:** Capture during long output test.

**6. Error Rate & Timeouts**
- **Panel:** Stacked bar chart, X-axis: Time, Y-axis: Count.
- **Configuration:** Color by status code (200, 408, 500, 503).
- **Annotations:** Mark configuration changes, restarts.
- **What to Look For:** Clusters of errors indicate instability. Timeouts correlate with VRAM pressure.
- **Screenshot Timing:** Capture after full weekend testing.

**Dashboard Configuration Tips:**
- Use consistent time ranges (15m, 1h, 24h).
- Enable "Annotations" layer for test labels.
- Export screenshots as PNG with metadata overlay.
- Store in `/benchmarks/screenshots/` with naming convention: `YYYYMMDD_test_label_panel.png`.

These screenshots provide visual proof of your system's behavior and make it easy to spot anomalies. The next section provides a time-boxed weekend checklist.

# Weekend Checklist

This checklist is designed for a focused 16-hour weekend session. Follow it sequentially to avoid context switching and ensure reproducible results.

**Saturday: Baseline & Setup (8 Hours)**
- [ ] **09:00-09:30:** Clear environment. Close background apps. Update `llama.cpp`, drivers, OpenWebUI.
- [ ] **09:30-10:00:** Configure Flight Recorder proxy. Verify SQLite schema. Test logging with single curl.
- [ ] **10:00-10:30:** Configure ServerTop dashboard. Create panels per previous section. Set retention to 7 days.
- [ ] **10:30-11:30:** Run cold start test (MTP OFF). 10 runs. Verify data in SQLite.
- [ ] **11:30-12:30:** Run warm start test (MTP OFF). 30 runs. Fixed prompt, fixed seed.
- [ ] **12:30-13:30:** Lunch. Review initial data. Check for logging errors.
- [ ] **13:30-15:00:** Run long output test (MTP OFF). 4k, 8k, 12k tokens. 3 runs each. Monitor VRAM/temp.
- [ ] **15:00-16:00:** Run RAG-style test (heavy prompt, short output). 20 runs. Measure prompt caching.
- [ ] **16:00-17:00:** Export Saturday data. Run SQLite queries. Generate baseline report. Capture dashboard screenshots.

**Sunday: MTP & Optimization (8 Hours)**
- [ ] **09:00-09:30:** Restart server with MTP ON. Verify draft model loads. Check VRAM baseline.
- [ ] **09:30-10:30:** Run warm start test (MTP ON). 30 runs. Identical prompts to Saturday.
- [ ] **10:30-11:30:** Run MTP parameter sweep (`--mtp-max-steps` 2, 4, 6, 8). 10 runs each.
- [ ] **11:30-12:30:** Run long output test (MTP ON). 4k, 8k, 12k tokens. 3 runs each.
- [ ] **12:30-13:30:** Lunch. Compare MTP ON vs OFF data. Calculate hit rates, TPT improvements.
- [ ] **13:30-15:00:** Run use-case profiling: Coding, Agentic, Creative. 10 runs each. Tag in SQLite.
- [ ] **15:00-16:00:** Analyze results. Run statistical tests. Identify optimal configuration.
- [ ] **16:00-17:00:** Document findings. Update OpenWebUI settings. Create runbook for future tests.

**Decision Gates:**
- If Saturday data shows >10% error rate, fix logging/VRAM before Sunday.
- If MTP hit rate < 40% across all tests, disable MTP and document why.
- If VRAM exceeds 90% during long outputs, reduce `--ctx-size` or switch cache type.

**Fallback Plans:**
- If server crashes, check `llama-server --verbose` logs. Reduce `--gpu-layers` by 5.
- If data is noisy, increase run count to 50. Control background processes.
- If MTP draft model fails, try smaller quantization (Q4_K_S) or disable MTP.

This checklist ensures you collect comprehensive data, analyze it objectively, and make informed decisions. The next section explains how to interpret your results.

# Interpreting Results

Raw data is useless without interpretation. This section provides a framework for reading your metrics, making tuning decisions, and matching your configuration to specific use cases.

**1. Reading Latency Metrics**
- **TTFT (Time to First Token):** Dominated by prompt processing. Should be < 500ms for warm starts. > 2s indicates VRAM pressure or inefficient prompt caching.
- **TPT (Time Per Token):** Dominated by decode speed. Should be consistent. Variance > 20% indicates fragmentation, throttling, or MTP instability.
- **Total Time:** TTFT + (TPT × completion_tokens). Use for end-to-end SLA planning.

**2. MTP Verdict Framework**
- **Hit Rate > 60%:** MTP is highly beneficial. Keep `--mtp-max-steps` at 4-6. Monitor VRAM.
- **Hit Rate 40-60%:** MTP is marginal. Try lower `--mtp-max-steps` (2-3). Consider smaller draft model.
- **Hit Rate < 40%:** MTP is harmful. Disable it. The verification overhead exceeds decode savings.
- **VRAM Impact:** If MTP adds > 2GB VRAM, it may cause OOM during long contexts. Reduce `--ctx-size` or switch to `--cache-type-k q4_0`.

**3. VRAM & Context Management**
- **Linear VRAM Growth:** Healthy. Indicates efficient KV cache.
- **Step VRAM Growth:** Cache reallocation. Normal during context expansion.
- **Exponential VRAM Growth:** Fragmentation or leak. Restart server, reduce `--batch-size`, update llama.cpp.
- **Context Utilization > 80%:** Risk of truncation or CPU offloading. Increase `--ctx-size` or implement conversation summarization.

**4. Use-Case Matching**
- **Coding:** Requires high context, low latency, high accuracy. Optimal: `--ctx-size 32768`, `--cache-type-k q8_0`, MTP ON if hit rate > 55%. Prioritize consistency over peak speed.
- **RAG:** Heavy prompt, short output. Optimal: `--ctx-size 16384`, `--prompt-cache-all`, MTP OFF. Prompt caching is critical. Tune retrieval chunk size to fit context budget.
- **Agentic:** Multi-turn, tool calls, variable context. Optimal: `--ctx-size 16384`, `--cache-type-k q8_0`, MTP OFF or `--mtp-max-steps 2`. Prioritize stability. Implement context window management (summarization, pruning).
- **Chat:** Short prompts, short outputs. Optimal: `--ctx-size 8192`, MTP OFF or marginal. Warm start matters. Optimize TTFT with prompt caching.
- **Creative Writing:** Long outputs, variable speed. Optimal: `--ctx-size 8192-16384`, MTP ON if draft model matches style. Monitor TPT consistency. Use `--temperature 0.8-1.0`.

**5. Decision Matrix**
| Metric | Target | Action if Below | Action if Above |
|--------|--------|-----------------|-----------------|
| MTP Hit Rate | > 55% | Reduce steps, change draft model | Keep ON, monitor VRAM |
| TPT Variance | < 15% | Fix fragmentation, cooling | Optimal |
| VRAM Peak | < 85% | Reduce ctx, cache type q4_0 | Safe headroom |
| TTFT (Warm) | < 500ms | Enable prompt cache, reduce layers | Optimal |
| Error Rate | < 2% | Check logs, reduce batch size | Stable |

**6. Continuous Improvement**
- Re-run tests after llama.cpp updates.
- Monitor thermal performance over time.
- Adjust configurations based on real user feedback.
- Document all changes in a version-controlled runbook.

This interpretation framework turns raw metrics into actionable insights. The final section covers common mistakes to avoid.

# Common Mistakes

Even experienced operators fall into these traps. Avoid them to ensure your benchmarks are valid and your production setup is stable.

**1. Not Warming Up the Server**
- **Mistake:** Running benchmarks immediately after startup.
- **Fix:** Run 5-10 dummy requests before testing. Measure cold vs warm separately.

**2. Mixing Prompt Lengths**
- **Mistake:** Comparing short and long prompts in the same dataset.
- **Fix:** Stratify by prompt length. Use fixed prompt sets per test.

**3. Ignoring VRAM Fragmentation**
- **Mistake:** Assuming VRAM usage equals theoretical allocation.
- **Fix:** Monitor peak vs current VRAM. Restart server between major tests. Use `--mlock`.

**4. Misinterpreting MTP Hit Rate**
- **Mistake:** Assuming higher hit rate always means better performance.
- **Fix:** Correlate hit rate with TPT and VRAM. Low hit rate + high VRAM = disable MTP.

**5. Using Mean Instead of Median**
- **Mistake:** Reporting average latency, which hides outliers.
- **Fix:** Use median and p90. Report standard deviation.

**6. Not Controlling Background Processes**
- **Mistake:** Running benchmarks with browsers, IDEs, or VMs active.
- **Fix:** Close non-essential apps. Use `nice`/`cpulimit` if necessary.

**7. Misconfiguring `--gpu-layers`**
- **Mistake:** Setting too high, causing CPU offloading spikes.
- **Fix:** Use `--verbose` to see layer allocation. Leave 10% VRAM headroom.

**8. Ignoring Prompt Caching**
- **Mistake:** Not enabling cache for RAG or multi-turn chat.
- **Fix:** Enable `--prompt-cache-all`. Monitor hit rate. Invalidate on system prompt changes.

**9. Over-Tuning `--mtp-max-steps`**
- **Mistake:** Setting to 8+ without testing hit rate.
- **Fix:** Start at 2. Increase only if hit rate > 60% and VRAM allows.

**10. Not Logging System Metrics**
- **Mistake:** Only logging request data, ignoring GPU temp, CPU load, disk I/O.
- **Fix:** Log system metrics at 10s intervals. Correlate with latency spikes.

**11. Comparing Different Quantizations**
- **Mistake:** Testing Q4_K_M vs Q5_K_M and attributing speed differences to MTP.
- **Fix:** Keep quantization constant. Test one variable at a time.

**12. Ignoring Network/Proxy Overhead**
- **Mistake:** Blaming llama.cpp for latency caused by Flight Recorder or OpenWebUI.
- **Fix:** Test with direct curl to llama-server. Compare with proxy overhead.

**Best Practices Summary:**
- Control variables rigorously.
- Log everything.
- Use medians and percentiles.
- Validate with multiple runs.
- Document configurations.
- Iterate based on data, not intuition.

By avoiding these mistakes, you'll build a reliable, high-performance local LLM setup that scales to your needs. This guide provides the foundation for consistent, measurable, and optimized inference. Happy benchmarking.