# Quick Diagnosis

Welcome to the lab. Before we dive into the methodology, let’s map your symptoms to the underlying mechanics of llama.cpp, your proxy layer, and GPU memory management. Inconsistency in local LLM serving is rarely a single bug; it’s usually a cascade of interacting variables. Here’s what’s likely happening:

**1. Slow First Answer (High TTFT)**
Time-to-First-Token (TTFT) is dominated by prompt processing. When you send a request, llama.cpp must:
- Parse the prompt tokens
- Allocate or resize the KV cache
- Run the initial forward pass through all layers
- If MTP is enabled, also run auxiliary head predictions for the first N tokens
- Pass the result through the Flight Recorder proxy, which may buffer or log the response

If your first answer is sometimes slow and sometimes fast, you’re likely experiencing:
- **KV cache reallocation**: If `--ctx-size` is tight or you’re hitting fragmentation, the server may need to allocate new memory blocks mid-request.
- **MTP warmup overhead**: Multi-Token Prediction requires auxiliary heads to initialize. The first few requests often pay a one-time compilation/initialization cost.
- **Proxy buffering**: Flight Recorder might be waiting for a minimum buffer size or flushing logs asynchronously, creating artificial latency spikes.
- **GPU memory pressure**: If VRAM is near capacity, the first request may trigger a swap or fallback to CPU offloading, causing massive TTFT variance.

**2. GPU Memory Jumps**
VRAM isn’t static. It fluctuates based on:
- **KV cache growth**: Each new context token adds to the KV cache. If you’re not using a fixed context size or are dynamically resizing, you’ll see step-changes in memory usage.
- **MTP auxiliary heads**: MTP adds extra parameter buffers for predicting multiple tokens per forward pass. This can cause a sudden VRAM jump when MTP is toggled on, especially on models with many layers.
- **Batch allocation**: llama.cpp allocates memory in chunks. If your batch size (`--batch-size`) or sequence count (`--num-predict`) changes, memory allocation may jump discretely.
- **Proxy/SQLite logging**: Flight Recorder and SQLite may buffer large JSON payloads in memory before flushing to disk. A long response can cause a temporary memory spike in the proxy process, which `nvidia-smi` or ServerTop might attribute to the GPU driver if metrics are aggregated at the system level.

**3. Uncertainty About MTP Effectiveness**
MTP (Multi-Token Prediction) is not a universal speedup. It works by predicting N tokens per forward pass using auxiliary heads trained alongside the main model. It helps when:
- The model was explicitly trained with MTP (e.g., Llama 3.1/3.2 MTP variants)
- You’re generating long outputs (TPS increases, TTFT may stay similar or slightly increase)
- GPU compute is the bottleneck, not memory bandwidth or proxy I/O

It hurts when:
- The model lacks MTP training (auxiliary heads produce garbage, causing retries or poor quality)
- You’re generating short outputs (overhead outweighs throughput gains)
- VRAM is constrained (MTP heads + KV cache push you into swap or CPU fallback)
- Proxy logging or SQLite writes become the bottleneck (MTP increases token throughput, but your logging pipeline can’t keep up)

**Immediate Checks to Run Now:**
1. Verify MTP model compatibility: `llama.cpp` only benefits from `--mtp` on models explicitly trained with it. Check the model card or Hugging Face metadata for `mtp` or `multi-token-prediction` tags.
2. Check launch flags: Ensure you’re not mixing `--mtp` with incompatible flags like `--num-predict` set too low, or `--ctx-size` too small.
3. Monitor proxy buffering: Temporarily disable Flight Recorder logging or set it to async-only. Does TTFT stabilize?
4. Check SQLite size: `ls -lh /path/to/flight_recorder.db`. If it’s >500MB, I/O latency may be masking real performance.
5. Run a warmup: Send 3 identical requests before measuring. Does the 4th request show consistent latency?

Now that we’ve diagnosed the likely culprits, let’s build a systematic data collection and testing plan.

# Data Collection Plan

Consistent benchmarking requires controlled variables, standardized prompts, and reliable logging. Here’s how to set up your data collection pipeline for the weekend.

**1. Standardize Your Testing Environment**
- **Hardware**: Note GPU model, VRAM, CPU cores, RAM, storage type (NVMe vs SATA). Run `nvidia-smi -q` and `lspci | grep -i vga` to document.
- **Software**: Record llama.cpp commit hash, Python version, proxy version, SQLite version.
- **Network**: Use `localhost` or `127.0.0.1` to eliminate network jitter. Disable Wi-Fi if possible.
- **Power/Thermal**: Ensure GPU is plugged in, cooling is adequate, and power limit is set to max (`nvidia-smi -pl 300` if applicable). Thermal throttling causes latency variance.

**2. Configure Flight Recorder Proxy**
Flight Recorder should log:
- Request timestamp, model, prompt length, completion length
- TTFT, total latency, TPS
- VRAM usage at request start/end
- MTP enabled flag
- HTTP status, proxy buffer size

Set logging to append-only mode. Avoid synchronous disk writes during inference. If Flight Recorder supports async flushing, enable it. Example config snippet:
```json
{
  "log_level": "info",
  "async_flush": true,
  "max_buffer_size_mb": 64,
  "output_format": "jsonl",
  "metrics": ["ttft_ms", "tps", "vram_mb", "prompt_tokens", "completion_tokens"]
}
```

**3. Configure ServerTop**
ServerTop should sample at 1-second intervals. Capture:
- GPU memory used/free
- GPU compute utilization (%)
- Memory bandwidth (GB/s)
- Temperature
- PCIe throughput

Run it in the background: `servertop --interval 1 --output servertop_log.csv &`

**4. SQLite Reporting Setup**
Ensure your SQLite database has a clean schema. If you’re using a custom schema, standardize it. Example table structure:
```sql
CREATE TABLE IF NOT EXISTS requests (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT NOT NULL,
    model TEXT NOT NULL,
    prompt_tokens INTEGER NOT NULL,
    completion_tokens INTEGER NOT NULL,
    total_tokens INTEGER NOT NULL,
    ttft_ms REAL,
    tps REAL,
    total_ms REAL,
    vram_used_mb REAL,
    mtp_enabled INTEGER DEFAULT 0,
    status TEXT DEFAULT 'success'
);
CREATE INDEX idx_timestamp ON requests(timestamp);
CREATE INDEX idx_mtp ON requests(mtp_enabled);
```

**5. Standardized Curl Test Scripts**
Use these curl commands for consistent testing. Save them as shell scripts for reproducibility.

*Baseline Chat Test (Non-Streaming, Fixed Output):*
```bash
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Explain quantum entanglement in 3 sentences."}],
    "max_tokens": 64,
    "temperature": 0.7,
    "top_p": 0.9,
    "stream": false
  }' > response.json
```

*Streaming Test (For TTFT/TPS measurement):*
```bash
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Write a 200-word story about a robot learning to paint."}],
    "max_tokens": 256,
    "temperature": 0.8,
    "top_p": 0.95,
    "stream": true
  }' | while IFS= read -r line; do
    if [[ "$line" == data:* ]]; then
      echo "$line" | jq -r '.delta.content // empty'
    fi
  done
```

*Long Context Test:*
```bash
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Summarize the following text: [PASTE 1500-TOKEN DOCUMENT]"}],
    "max_tokens": 512,
    "temperature": 0.1,
    "stream": false
  }' > long_context.json
```

**6. Data Collection Protocol**
- Run each test 15 times per configuration.
- Record timestamps, file sizes, and ServerTop snapshots.
- Export Flight Recorder logs to CSV/SQLite after each batch.
- Never change more than one variable per test batch (e.g., only toggle MTP, keep everything else identical).

# MTP Comparison Plan

Comparing MTP vs non-MTP fairly requires rigorous experimental design. Here’s how to avoid self-deception and get actionable results.

**1. Understand MTP Mechanics in llama.cpp**
MTP predicts N tokens per forward pass using auxiliary heads. It trades compute for throughput. Key flags:
- `--mtp`: Enables MTP
- `--mtp-n`: Number of tokens to predict per step (usually 2-4)
- `--mtp-heads`: Number of auxiliary heads (model-dependent)

MTP only works if the model was trained with it. Using it on a standard model adds overhead without benefit.

**2. Control Variables**
- Same model weights (download once, verify SHA256)
- Same prompt set (use a fixed benchmark suite)
- Same temperature/top_p/max_tokens
- Same batch size (`--batch-size`)
- Same context size (`--ctx-size`)
- Same proxy configuration
- Same hardware state (cold boot vs warm, but document which)

**3. Test Matrix**
| Test Type | Prompt Length | Output Length | Temp | Top_p | MTP | Runs |
|-----------|---------------|---------------|------|-------|-----|------|
| Short Chat | 50 tokens | 64 tokens | 0.7 | 0.9 | 0/1 | 15 |
| Medium Chat | 300 tokens | 256 tokens | 0.7 | 0.9 | 0/1 | 15 |
| Long Context | 1500 tokens | 512 tokens | 0.1 | 0.95 | 0/1 | 15 |
| Agentic/Tool | 200 tokens | 128 tokens | 0.2 | 0.9 | 0/1 | 15 |
| Creative | 100 tokens | 1024 tokens | 0.9 | 0.95 | 0/1 | 15 |

**4. Metrics to Track**
- **TTFT (ms)**: Time from request to first token
- **TPS (tokens/sec)**: Completion tokens / (total_ms - ttft_ms)
- **Total Latency (ms)**: Full request time
- **VRAM Delta (MB)**: Memory used at start vs end
- **P95/P99 Latency**: 95th/99th percentile of total latency
- **Proxy Queue Depth**: Requests waiting in Flight Recorder buffer
- **GPU Compute %**: Average utilization during generation

**5. Statistical Analysis**
- Calculate mean, median, standard deviation, P95, P99 for each metric.
- Use box plots to visualize distribution overlap.
- Perform a paired t-test or Wilcoxon signed-rank test if comparing same prompts.
- Only claim MTP helps if:
  - P95 TPS increases by >10%
  - VRAM delta is stable or decreases
  - TTFT doesn’t increase by >15%
  - Quality metrics (if measured) don’t degrade

**6. Avoiding Self-Fooling**
- **Blind the analysis**: Label runs as A/B instead of MTP/non-MTP until after data collection.
- **Automate runs**: Use a Python script to execute curl commands, parse JSON, and log to SQLite. Manual runs introduce bias.
- **Don’t cherry-pick**: Report all runs, including outliers. Explain them.
- **Check for confounders**: Did GPU thermal throttle? Did SQLite flush? Did proxy buffer grow?
- **Verify model compatibility**: Run `llama.cpp --model your_model.gguf --mtp --mtp-n 2` and check for warnings. If it says “MTP not supported”, disable it.

# Reasoning Budget Plan

Managing context window and token budget is critical for consistent performance, especially in RAG, agentic, and coding workflows. Here’s how to optimize your reasoning budget.

**1. KV Cache Memory Formula**
VRAM used by KV cache ≈ `(2 * n_layers * n_heads * head_dim * ctx_size * 2) / (1024^3)` GB
- `n_layers`: Model depth
- `n_heads`: Attention heads
- `head_dim`: Dimension per head
- `ctx_size`: Context window size
- `2`: FP16/BF16 precision (2 bytes per value)

Example: Llama-3-8B (32 layers, 32 heads, 128 dim, ctx=8192) ≈ `(2 * 32 * 32 * 128 * 8192 * 2) / 1073741824 ≈ 3.2 GB`

**2. Context Window Management**
- Set `--ctx-size` to your maximum expected prompt + output. Don’t leave it at default if you’re hitting fragmentation.
- Use `--cache-reuse` if available to share KV cache across similar prompts.
- Monitor `--batch-size`: Larger batches improve throughput but increase VRAM. Start with 512, adjust based on memory.
- Use `--num-predict` to cap output length. Prevents OOM on runaway generations.

**3. Prompt Compression & RAG Chunking**
- **RAG**: Chunk size 256-512 tokens, overlap 50-100. Use embedding models separately. Keep retrieved context under 60% of `--ctx-size`.
- **Agentic**: Tool calls add tokens. Reserve 20% of context for tool output. Use structured JSON prompts to reduce parsing overhead.
- **Coding**: Syntax-heavy prompts benefit from lower temperature (0.1-0.3). Use `--top_k 40` for deterministic output.
- **Creative**: Higher temperature (0.8-1.0), longer outputs. MTP helps here if VRAM allows.

**4. Token Budget Allocation**
| Workflow | Prompt Budget | Output Budget | Temp | Top_p | Notes |
|----------|---------------|---------------|------|-------|-------|
| Chat | 10-20% | 80-90% | 0.7 | 0.9 | TTFT critical |
| RAG | 60-70% | 20-30% | 0.1 | 0.95 | Context heavy |
| Agentic | 30-40% | 40-50% | 0.2 | 0.9 | Tool calls add tokens |
| Coding | 40-50% | 30-40% | 0.1 | 0.8 | Deterministic |
| Creative | 10-15% | 85-90% | 0.9 | 0.95 | Long outputs |

**5. Practical Tips**
- Use `--logit-bias` to penalize unwanted tokens.
- Set `--repeat_penalty 1.1` to avoid repetition.
- Use `--mirostat` for adaptive temperature if quality varies.
- Monitor `--ctx-size` usage: If you’re consistently at >80%, increase it or compress prompts.
- Avoid dynamic context resizing mid-session. It causes KV cache reallocation and memory jumps.

# Long Output Test Plan

Long outputs stress-test memory stability, throughput consistency, and proxy handling. Here’s how to run them safely and interpret results.

**1. Test Setup**
- Use `max_tokens: 2048` or `4096`
- Set `stream: true` to measure real-time TPS
- Use a neutral prompt: “Write a detailed technical guide on [topic] covering introduction, methodology, examples, and conclusion.”
- Run 10 iterations per configuration (MTP on/off)

**2. Metrics to Track**
- **TPS Decay**: Does throughput drop after 512 tokens? Indicates KV cache pressure or memory bandwidth limits.
- **VRAM Stability**: Should remain flat after initial allocation. Jumps indicate fragmentation or proxy buffering.
- **TTFT Consistency**: Should be similar to short outputs. If it increases, MTP warmup or context resizing is happening.
- **Proxy Buffer Growth**: Flight Recorder may buffer large responses. Monitor buffer size vs latency.
- **Error Rate**: Check for timeouts, OOM, or truncated outputs.

**3. Execution Script**
```bash
#!/bin/bash
for i in {1..10}; do
  echo "Run $i"
  curl -s http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "local",
      "messages": [{"role": "user", "content": "Write a 2000-token technical guide on distributed systems."}],
      "max_tokens": 2048,
      "temperature": 0.7,
      "top_p": 0.9,
      "stream": true
    }' > output_stream_${i}.jsonl
  echo "Completed run $i"
done
```

**4. Analysis**
- Parse JSONL for chunk timestamps. Calculate TPS per chunk.
- Plot TPS over token count. Look for decay curves.
- Check VRAM at start, middle, end. Should be stable.
- If TPS drops >20% after 1024 tokens, consider:
  - Reducing `--ctx-size`
  - Disabling MTP (auxiliary heads may not scale)
  - Using `--cache-type-k v8` or `--cache-type-v v8` for quantized KV cache
  - Increasing `--batch-size` if GPU compute is underutilized

**5. Context Overflow Handling**
- Test what happens when prompt + output > `--ctx-size`.
- llama.cpp should truncate or error. Document behavior.
- Use `--no-prompt-gpu` if CPU offloading is causing slowdowns.
- Monitor `--num-predict` vs actual output. If truncated, increase limit or compress prompt.

# SQLite Queries

Use these queries to analyze your Flight Recorder/ServerTop data. Adjust table/column names to match your schema.

**1. Basic Performance Summary**
```sql
SELECT 
    mtp_enabled,
    COUNT(*) as runs,
    ROUND(AVG(ttft_ms), 2) as avg_ttft_ms,
    ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY ttft_ms), 2) as p95_ttft_ms,
    ROUND(AVG(tps), 2) as avg_tps,
    ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY tps), 2) as p95_tps,
    ROUND(AVG(vram_used_mb), 2) as avg_vram_mb
FROM requests
GROUP BY mtp_enabled;
```

**2. TPS Distribution by Prompt Length**
```sql
SELECT 
    CASE 
        WHEN prompt_tokens < 100 THEN 'Short (<100)'
        WHEN prompt_tokens < 500 THEN 'Medium (100-500)'
        ELSE 'Long (>500)'
    END as prompt_bucket,
    mtp_enabled,
    COUNT(*) as runs,
    ROUND(AVG(tps), 2) as avg_tps,
    ROUND(STDDEV(tps), 2) as stddev_tps
FROM requests
GROUP BY prompt_bucket, mtp_enabled
ORDER BY prompt_bucket, mtp_enabled;
```

**3. Memory Jump Detection**
```sql
WITH vram_deltas AS (
    SELECT 
        id,
        vram_used_mb,
        LAG(vram_used_mb) OVER (ORDER BY timestamp) as prev_vram,
        vram_used_mb - LAG(vram_used_mb) OVER (ORDER BY timestamp) as vram_delta
    FROM requests
)
SELECT 
    id,
    timestamp,
    vram_used_mb,
    vram_delta,
    mtp_enabled
FROM vram_deltas
WHERE ABS(vram_delta) > 50  -- Threshold: 50MB jump
ORDER BY ABS(vram_delta) DESC;
```

**4. Slow Query Identification**
```sql
SELECT 
    id,
    timestamp,
    model,
    prompt_tokens,
    completion_tokens,
    ttft_ms,
    tps,
    total_ms,
    mtp_enabled,
    status
FROM requests
WHERE total_ms > (SELECT PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY total_ms) FROM requests)
ORDER BY total_ms DESC
LIMIT 20;
```

**5. Use-Case Performance Breakdown**
```sql
SELECT 
    CASE 
        WHEN prompt_tokens < 100 AND completion_tokens < 256 THEN 'Chat/Creative'
        WHEN prompt_tokens > 500 THEN 'RAG/Long Context'
        WHEN completion_tokens < 128 AND status = 'success' THEN 'Agentic/Tool'
        ELSE 'Other'
    END as use_case,
    mtp_enabled,
    COUNT(*) as runs,
    ROUND(AVG(ttft_ms), 2) as avg_ttft,
    ROUND(AVG(tps), 2) as avg_tps,
    ROUND(AVG(vram_used_mb), 2) as avg_vram
FROM requests
GROUP BY use_case, mtp_enabled
ORDER BY use_case, mtp_enabled;
```

**6. Proxy Buffer Impact**
```sql
SELECT 
    timestamp,
    ttft_ms,
    total_ms,
    vram_used_mb,
    CASE 
        WHEN total_ms > ttft_ms * 3 THEN 'Proxy/IO Bottleneck Suspected'
        ELSE 'Normal'
    END as bottleneck_indicator
FROM requests
WHERE total_ms > 2000
ORDER BY total_ms DESC;
```

Run these queries after each test batch. Export results to CSV for dashboard plotting.

# Dashboard Screenshots To Capture

Visualizing your data is critical for spotting trends. Here’s what to capture and how to generate it.

**1. Dashboard Metric Definitions**
- **TTFT (ms)**: Time from request start to first token received. Lower is better for chat.
- **TPS (tokens/sec)**: Completion tokens divided by generation time. Higher is better for throughput.
- **P95 Latency (ms)**: 95th percentile of total request time. Indicates tail latency.
- **VRAM Utilization (%)**: GPU memory used vs total. Stable is better than spiking.
- **GPU Compute %**: Average utilization during generation. >70% indicates compute-bound.
- **Proxy Queue Depth**: Requests waiting in Flight Recorder buffer. High depth indicates I/O bottleneck.
- **TPS Decay Rate**: % drop in TPS from token 0 to token 1024. Indicates memory bandwidth limits.

**2. Essential Charts**
- **TTFT vs Prompt Length Scatter Plot**: X=prompt tokens, Y=TTFT ms. Look for linear scaling. Outliers indicate KV cache reallocation.
- **TPS Over Time Line Chart**: X=request index, Y=TPS. Flat line = stable. Downward slope = memory pressure or thermal throttling.
- **VRAM Usage Heatmap**: X=time, Y=request ID, Color=VRAM MB. Step changes = fragmentation. Spikes = proxy buffering.
- **MTP vs Non-MTP Box Plot**: X=config, Y=TPS or TTFT. Non-overlapping boxes with MTP higher = significant improvement.
- **Proxy Buffer Size vs Latency Correlation**: X=buffer size, Y=total_ms. Positive correlation = proxy is bottleneck.

**3. How to Generate**
- **Python/Matplotlib**:
```python
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('flight_recorder_export.csv')
plt.figure(figsize=(10, 6))
plt.scatter(df['prompt_tokens'], df['ttft_ms'], c=df['mtp_enabled'], cmap='viridis')
plt.xlabel('Prompt Tokens')
plt.ylabel('TTFT (ms)')
plt.title('TTFT vs Prompt Length (MTP: Blue, Non-MTP: Orange)')
plt.colorbar(label='MTP Enabled')
plt.grid(True)
plt.savefig('ttft_vs_prompt.png')
```
- **Grafana**: Ingest CSV/SQLite via Prometheus or Loki. Create dashboards with the above charts.
- **Excel/Sheets**: Use pivot tables and scatter plots for quick analysis.

**4. What to Capture**
- Screenshot each chart with clear titles, axis labels, and legends.
- Note the date/time, model, and configuration.
- Annotate anomalies: “VRAM jump at run 7 due to proxy flush”, “TPS decay after 1024 tokens”.
- Save as PNG/PDF with consistent naming: `YYYYMMDD_model_config_chart.png`.

# Weekend Checklist

Follow this step-by-step plan to systematically evaluate your setup.

**Saturday Morning: Setup & Baseline**
- [ ] Document hardware/software specs
- [ ] Verify model supports MTP (check metadata)
- [ ] Configure Flight Recorder with async logging
- [ ] Set up SQLite schema and indexes
- [ ] Run ServerTop in background
- [ ] Execute 5 baseline runs (non-MTP, short prompt, stream=false)
- [ ] Export logs to SQLite, verify data integrity
- [ ] Calculate baseline TTFT, TPS, VRAM

**Saturday Afternoon: MTP Testing**
- [ ] Enable MTP flags (`--mtp --mtp-n 2`)
- [ ] Run 15 short prompt tests
- [ ] Run 15 medium prompt tests
- [ ] Run 15 long context tests
- [ ] Monitor VRAM and ServerTop for jumps
- [ ] Export all logs, merge with baseline
- [ ] Run SQLite queries for summary stats
- [ ] Generate initial charts

**Sunday Morning: Long Output & Stress Testing**
- [ ] Run 10 long output tests (MTP on)
- [ ] Run 10 long output tests (MTP off)
- [ ] Monitor TPS decay, VRAM stability
- [ ] Test context overflow behavior
- [ ] Check proxy buffer growth
- [ ] Export data, run bottleneck queries
- [ ] Generate TPS decay and VRAM heatmap charts

**Sunday Afternoon: Analysis & Decision**
- [ ] Compare MTP vs non-MTP statistically
- [ ] Map results to use cases (chat, RAG, agentic, coding, creative)
- [ ] Identify bottlenecks (proxy, SQLite, GPU, memory)
- [ ] Document findings in lab notebook
- [ ] Decide on final configuration
- [ ] Clean up SQLite (vacuum, archive old logs)
- [ ] Plan next iteration (quantization, batch tuning, proxy optimization)

# Interpreting Results

Now that you have data, here’s how to interpret it without bias and make actionable decisions.

**1. When MTP Helps**
- **Throughput Gain**: P95 TPS increases by >10% with stable VRAM
- **Long Outputs**: TPS remains flat or improves after 1024 tokens
- **Compute-Bound Workloads**: GPU utilization >75%, memory bandwidth not saturated
- **Model Compatibility**: Model card explicitly mentions MTP training
- **Use Cases**: Creative writing, long-form generation, batch processing

**2. When MTP Hurts**
- **TTFT Spike**: First token delayed by >15%, indicates warmup overhead
- **VRAM Pressure**: Memory jumps >10%, triggers swap or CPU fallback
- **Short Outputs**: TPS decreases or stays flat, overhead outweighs benefit
- **Proxy/IO Bottleneck**: Flight Recorder or SQLite can’t keep up with token rate
- **Use Cases**: Chat, agentic tool calls, RAG retrieval, low-latency workflows

**3. Use-Case Recommendations**
- **Chat**: Prioritize low TTFT. Disable MTP if it increases first-token latency. Use `--ctx-size` optimized for conversation length.
- **RAG**: Context-heavy. MTP rarely helps. Focus on prompt compression, chunking, and KV cache efficiency. Use `--cache-type-k v8`.
- **Agentic**: Tool calls add tokens. MTP can help if output length is predictable. Monitor TPS stability. Use structured prompts.
- **Coding**: Deterministic output. Low temperature. MTP may hurt quality if auxiliary heads produce syntax errors. Disable unless throughput is critical.
- **Creative Writing**: Long outputs, high temperature. MTP shines here if VRAM allows. Enable `--mtp --mtp-n 3` for best results.

**4. Decision Matrix**
| Metric | Threshold | Action |
|--------|-----------|--------|
| P95 TPS increase | >10% | Keep MTP |
| TTFT increase | >15% | Disable MTP |
| VRAM delta | >10% | Disable MTP or reduce ctx-size |
| TPS decay >20% | After 1024 tokens | Disable MTP, optimize batch |
| Proxy queue depth | >50 requests | Optimize proxy, async flush |
| GPU compute % | <60% | MTP unlikely to help |

**5. Statistical Confidence**
- Use P95/P99, not mean, to avoid outlier skew.
- Run at least 15 iterations per config.
- If confidence intervals overlap, MTP isn’t statistically significant.
- Document effect size: “MTP improved TPS by 12% (95% CI: 8-16%)”.

# Common Mistakes

Avoid these pitfalls to ensure your benchmarking is accurate and actionable.

**1. Benchmarking with Streaming vs Non-Streaming Inconsistently**
Streaming measures TTFT and chunk TPS. Non-streaming measures total latency. Always use the same mode for comparison. If testing TTFT, use `stream: true` and measure first chunk arrival. If testing throughput, use `stream: false` and measure total time.

**2. Ignoring Prompt Length Impact**
TTFT scales linearly with prompt tokens. TPS scales inversely. Always bucket by prompt length. Comparing a 50-token prompt to a 1000-token prompt invalidates MTP comparison.

**3. Assuming MTP Works on All Models**
MTP requires auxiliary heads trained during pretraining. Using `--mtp` on a standard model adds overhead without benefit. Check model metadata. If it doesn’t say “MTP” or “multi-token-prediction”, disable it.

**4. Not Warming Up the Model**
First few requests pay compilation/initialization costs. Always run 3-5 warmup requests before measuring. Document warmup runs separately.

**5. Letting SQLite Grow Unbounded**
Large SQLite files cause I/O latency. Vacuum weekly. Archive old logs. Use `PRAGMA journal_mode=WAL;` for concurrent writes. Monitor file size.

**6. Proxy Buffering Masking Real Latency**
Flight Recorder may buffer responses, causing artificial latency spikes. Test with proxy logging disabled or async-only. Compare with and without proxy to isolate bottleneck.

**7. Cherry-Picking Runs**
Don’t discard “bad” runs. Report all data. Explain outliers. Use statistical methods, not anecdotal evidence.

**8. Confusing Token Count with Wall Time**
TPS = tokens / seconds. A model can generate 100 tokens in 1 second (100 TPS) or 1000 tokens in 10 seconds (100 TPS). Always normalize by time.

**9. Overlooking CPU/GPU Synchronization Delays**
llama.cpp uses CPU for prompt processing and GPU for generation. If CPU is saturated, TTFT increases. Monitor CPU usage. Use `--threads` to balance.

**10. Not Checking Model Architecture Compatibility**
MTP works best on transformer architectures with consistent layer dimensions. MoE models, hybrid architectures, or quantized models may not support MTP properly. Verify with `llama.cpp --model your_model.gguf --mtp --verbose`.

**11. Ignoring Thermal Throttling**
GPU throttling causes latency variance. Monitor temperature. Use `nvidia-smi -q -d TEMPERATURE`. If >85°C, improve cooling or reduce power limit.

**12. Using Default Context Size**
Default `--ctx-size` may be too small or too large. Set it to your maximum expected prompt + output. Too small causes truncation. Too large wastes VRAM and increases fragmentation.

**13. Not Testing Edge Cases**
Test empty prompts, very long prompts, special characters, JSON payloads, tool calls. Real-world usage includes edge cases. Document behavior.

**14. Assuming Higher TPS Always Means Better UX**
Chat users care about TTFT. Batch processing cares about TPS. Match metrics to use case. Don’t optimize for throughput if latency matters.

**15. Forgetting to Document Configuration**
Small changes (temperature, top_p, batch size) affect results. Log everything. Use version control for configs. Reproducibility is key.

---

You now have a complete, lab-tested framework to diagnose, benchmark, and optimize your local llama.cpp server. Follow the weekend checklist, collect data rigorously, analyze statistically, and make decisions based on evidence, not intuition. MTP is a powerful tool, but only when matched to the right model, workload, and hardware constraints. If you hit snags during testing, revisit the SQLite queries and dashboard charts—they’ll point directly to the bottleneck. Happy benchmarking, and may your VRAM stay stable and your TPS stay high.