# Quick Diagnosis

Before diving into weekend testing, let's establish what's actually happening under the hood. Your symptoms point to three interacting subsystems: prompt processing (prefill), token generation (decode), and speculative decoding overhead (MTP). Understanding their boundaries is critical because MTP only accelerates the decode phase, not the prefill phase. When you see inconsistent first-answer latency, it's almost always a prefill variance issue, a draft model loading spike, or a KV cache fragmentation event. GPU memory jumps typically occur when the draft model is swapped in/out, when batch sizes change dynamically, or when the context window exceeds allocated VRAM and forces CPU offloading.

MTP (Multi-Token Prediction) in llama.cpp is essentially speculative decoding. A smaller, faster "draft" model predicts N tokens ahead. The main model then verifies them in parallel. If the draft's predictions match, you get N tokens for the cost of ~1 main-model forward pass. If they diverge, you fall back to standard generation. The tradeoff is clear: MTP reduces decode latency but adds VRAM overhead for the draft model, increases prefill complexity (the draft must also process the prompt), and can introduce jitter if the draft acceptance rate fluctuates.

Your first-answer latency inconsistency usually stems from:
1. **Draft model initialization**: The first request after server start or context switch loads the draft model into VRAM. This adds 200–800ms to TTFT (Time To First Token).
2. **KV cache allocation**: llama.cpp allocates KV cache in chunks. If your context length changes between requests, the cache may resize, causing memory spikes and latency variance.
3. **Dynamic batching**: OpenWebUI or your proxy may queue requests. If `--batch-size` or `--parallel` isn't fixed, the server may switch between single-request and multi-request modes, altering prefill time.
4. **GPU memory pressure**: If VRAM is >85% utilized, the driver may trigger page faults, swap, or thermal throttling. MTP's draft model sits in VRAM alongside the main model's KV cache. When combined, they can push you into unsafe territory.

Flight Recorder captures HTTP-level timing, ServerTop shows hardware utilization, and SQLite logs your request metadata. Together, they form a complete observability stack. The goal this weekend is to isolate variables, collect statistically meaningful data, and determine whether MTP's decode acceleration outweighs its prefill/VRAM overhead for your specific workloads.

# Data Collection Plan

A rigorous data collection plan eliminates guesswork. You need consistent inputs, controlled outputs, and reliable logging. Follow this step-by-step setup:

**1. Environment Stabilization**
- Set fixed environment variables: `CUDA_VISIBLE_DEVICES=0`, `LLAMA_LOG_LEVEL=3` (info), `LLAMA_DEBUG=0` (disable debug noise).
- Disable OpenWebUI auto-scaling or dynamic prompt truncation during testing.
- Pin your system to a single GPU. If you have multiple, isolate testing to one to avoid PCIe/NVLink contention.
- Warm up the server: Run 5 dummy requests before collecting data. This ensures CUDA kernels are compiled, VRAM is allocated, and the draft model is loaded.

**2. Flight Recorder Configuration**
- Configure your proxy to log full request/response cycles. Capture:
  - `X-Request-ID` or `trace_id`
  - `timestamp_start`, `timestamp_end`, `ttft_ms`, `total_ms`
  - `prompt_tokens`, `completion_tokens`, `total_tokens`
  - `model`, `draft_model`, `mtp_enabled`
  - `status_code`, `error_message`
- Use `curl` or a script to generate requests. Avoid browser dev tools for production testing due to network stack variability.
- Example `curl` for baseline (non-MTP):
  ```bash
  curl -s http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "your-main-model.gguf",
      "messages": [{"role": "user", "content": "Explain quantum entanglement in 3 sentences."}],
      "stream": false,
      "max_tokens": 256,
      "temperature": 0.7,
      "seed": 42
    }'
  ```
- Example `curl` for MTP:
  ```bash
  curl -s http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "your-main-model.gguf",
      "messages": [{"role": "user", "content": "Explain quantum entanglement in 3 sentences."}],
      "stream": false,
      "max_tokens": 256,
      "temperature": 0.7,
      "seed": 42
    }'
  ```
  Note: MTP is enabled via server flags, not request body. Ensure `--model-draft draft.gguf --speculative` (or `--mtp` depending on llama.cpp version) is set in your launch command.

**3. ServerTop Logging**
- Run ServerTop in background with 1-second sampling:
  ```bash
  servertop --interval 1 --output servertop_log.csv &
  ```
- Monitor: `gpu_mem_used`, `gpu_mem_total`, `gpu_util`, `gpu_temp`, `cpu_mem`, `swap_used`, `pci_bw`.
- Export to CSV/JSON for correlation with SQLite logs.

**4. SQLite Logging Setup**
- Use a wrapper script or llama.cpp's `--log-file` to parse JSON logs and insert into SQLite.
- Schema:
  ```sql
  CREATE TABLE requests (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    ts TEXT NOT NULL,
    model TEXT,
    draft_model TEXT,
    mtp_enabled INTEGER DEFAULT 0,
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    total_tokens INTEGER,
    prefill_ms REAL,
    decode_ms REAL,
    total_ms REAL,
    gpu_mem_peak_mb REAL,
    draft_acceptance_rate REAL,
    status TEXT,
    prompt_preview TEXT
  );
  CREATE INDEX idx_mtp_ts ON requests(mtp_enabled, ts);
  CREATE INDEX idx_status ON requests(status);
  ```
- Insert via Python/Node or `sqlite3` CLI. Ensure timestamps are ISO8601 for time-series analysis.

**5. Test Prompt Distribution**
- Collect 50–100 samples per configuration.
- Use a fixed prompt set: 10 short (chat), 10 medium (RAG), 10 long (coding/creative), 10 agentic (multi-step).
- Keep temperature, seed, and max_tokens constant.
- Randomize MTP/non-MTP order to avoid cache/warm-up bias.

**6. Validation Checks**
- Verify SQLite has correct token counts.
- Confirm Flight Recorder timing matches server logs.
- Ensure ServerTop captures VRAM spikes during draft loading.
- Check for dropped requests or timeouts.

# MTP Comparison Plan

Comparing MTP vs non-MTP without fooling yourself requires statistical rigor and controlled conditions. MTP is not a magic speedup; it's a tradeoff. Here's how to evaluate it properly:

**1. Define Success Metrics**
- **TTFT (Time To First Token)**: Prefill + draft load + first decode. MTP often increases this slightly due to draft model initialization.
- **TPS (Tokens Per Second)**: Decode phase speed. MTP should improve this if draft acceptance rate > 0.4.
- **Total Latency**: TTFT + decode time. Depends on output length.
- **VRAM Stability**: Peak memory, fragmentation, swap usage.
- **Draft Acceptance Rate**: Percentage of draft tokens accepted by main model. Target > 0.5 for MTP to be worthwhile.
- **Error Rate**: OOM, timeout, truncation, or generation failures.

**2. Controlled A/B Testing**
- Run baseline (non-MTP) first: 50 requests, record metrics.
- Run MTP: 50 requests, record metrics.
- Repeat 3 times with randomized order.
- Use median latency, not mean. Latency distributions are heavily skewed by outliers (cache misses, GC, driver interrupts).
- Calculate 95% confidence intervals using bootstrap or standard error: `SE = std_dev / sqrt(n)`.

**3. Avoiding Self-Deception**
- **Don't cherry-pick fast runs**: Log everything. Discard only requests with external errors (network, proxy, OOM).
- **Account for prefill dominance**: If your prompts are long (>2k tokens), prefill takes 80%+ of total latency. MTP won't help much. Focus on short-prompt workloads for MTP gains.
- **Watch draft model size**: A draft model that's too small yields low acceptance. Too large adds overhead. Test multiple drafts (e.g., 7B vs 1.5B quantized).
- **Ignore streaming vs non-streaming confusion**: Streaming shows token-by-token latency. Non-streaming shows total latency. MTP affects both, but streaming reveals decode consistency.
- **Control system load**: Close background apps, disable updates, pin CPU governor to performance. GPU driver interrupts can add 10–50ms jitter.

**4. Practical MTP Configuration**
- Launch command:
  ```bash
  llama-server --model main.gguf --model-draft draft.gguf --speculative --ctx-size 4096 --batch-size 512 --gpu-layers 99 --threads 8
  ```
- Adjust `--speculative` (number of draft tokens) based on draft model capacity. Start with 4–8.
- Monitor draft acceptance rate via server logs or custom metrics endpoint.
- If acceptance < 0.3, MTP is hurting performance. Drop it.

**5. Statistical Validation**
- Use Python/pandas or R for analysis:
  ```python
  import pandas as pd
  df = pd.read_sql("SELECT * FROM requests", "sqlite:///lab.db")
  mtp = df[df.mtp_enabled == 1]
  base = df[df.mtp_enabled == 0]
  print("Median TTFT MTP:", mtp.prefill_ms.median())
  print("Median TPS MTP:", (mtp.completion_tokens / (mtp.decode_ms / 1000)).median())
  ```
- If MTP median TPS > base by >15% and VRAM is stable, keep it. Otherwise, revert.

# Reasoning Budget Plan

"Reasoning budget" isn't just token count; it's compute, memory, and latency tradeoffs. Managing it effectively prevents bottlenecks and ensures consistent performance.

**1. Context Window & KV Cache**
- KV cache grows linearly with context length. For a 7B model at FP16, 1 token ≈ 2 * layers * head_dim * batch_size bytes.
- Example: 7B, 32 layers, 4096 dim, batch=1 → ~1MB/token. 8k context → ~8GB VRAM.
- MTP doesn't reduce KV cache size. It only speeds up decode. If VRAM is tight, increase `--ctx-size` carefully or use `--cache-reuse` to reduce fragmentation.
- Set `--ctx-size` to your maximum expected prompt + output. Don't overallocate.

**2. Batching & Parallelism**
- `--batch-size`: Tokens per forward pass. Higher = better throughput, but increases VRAM and latency variance.
- `--parallel`: Concurrent requests. Useful for OpenWebUI, but can cause queueing delays.
- MTP works best with moderate batching (256–512). Too high causes draft model verification bottlenecks.
- Monitor `gpu_util`. If < 70%, you're compute-bound, not memory-bound. Increase batch size. If > 90%, you're memory-bound. Reduce batch or context.

**3. Speculative Decoding Overhead**
- Draft model must be loaded into VRAM. For a 1.5B Q4_K_M draft, expect ~1.2GB VRAM.
- Draft prefill adds latency. If draft model is too large, it slows down prompt processing.
- Draft acceptance rate depends on model similarity. Fine-tuned drafts outperform generic ones.
- Use `--speculative` to control draft tokens. Start with 4. Increase only if acceptance > 0.6.

**4. Compute vs Latency Tradeoffs**
- MTP trades compute for latency. If you have spare GPU cycles, MTP helps. If GPU is saturated, it may not.
- For agentic work, consistency matters more than raw speed. Use MTP only if it doesn't introduce jitter.
- For RAG, context retrieval dominates. MTP is secondary. Focus on prompt engineering and retrieval accuracy.
- For coding, syntax correctness and low TTFT matter. MTP helps generation but not prefill.

**5. Practical Budget Management**
- Set `--max-tokens` to prevent runaway generation.
- Use `--timeout` to kill stuck requests.
- Monitor `gpu_mem_used` vs `gpu_mem_total`. Keep < 85% for safety.
- Use `--cache-type-k` and `--cache-type-v` to optimize KV cache precision (Q8_0 or F16 recommended).
- For long contexts, enable `--flash-attn` if supported. Reduces KV cache memory by 30–50%.

# Long Output Test Plan

Long outputs stress-test your server's stability, memory management, and MTP consistency. Follow this plan to evaluate performance under sustained generation.

**1. Test Setup**
- Use prompts requiring 1k–4k token outputs.
- Enable streaming: `stream: true` in request body.
- Set `max_tokens` to 2048 or higher.
- Run 10–20 long outputs per configuration (MTP/non-MTP).
- Monitor ServerTop continuously. Log VRAM, temperature, fan speed, PCIe bandwidth.

**2. Metrics to Track**
- **Time Per Token (TPT)**: Should be consistent. Spikes indicate KV cache pressure or draft verification stalls.
- **Total Latency**: TTFT + decode time. Compare MTP vs baseline.
- **VRAM at Start/End**: Check for leaks or fragmentation.
- **Error Rate**: OOM, timeout, truncation, or generation failures.
- **Draft Acceptance Over Time**: Should remain stable. Drops indicate context shift or draft model mismatch.

**3. curl Example for Long Output**
```bash
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-main-model.gguf",
    "messages": [{"role": "user", "content": "Write a 2000-word technical guide on optimizing local LLM inference, including KV cache management, speculative decoding, and batching strategies."}],
    "stream": true,
    "max_tokens": 2048,
    "temperature": 0.7,
    "seed": 42
  }'
```

**4. Evaluation Criteria**
- **Consistency**: TPT variance < 10% across output.
- **Memory Stability**: VRAM delta < 5% from start to end.
- **Draft Performance**: Acceptance rate > 0.4 throughout.
- **Error Handling**: No OOM, no truncation warnings, no dropped tokens.
- **Quality**: Coherence, syntax, factual accuracy (subjective but important).

**5. Troubleshooting Long Outputs**
- If TPT drops after 500 tokens, KV cache is fragmenting. Increase `--ctx-size` or use `--cache-reuse`.
- If VRAM spikes, draft model is competing for memory. Reduce `--speculative` or use smaller draft.
- If generation stalls, check `gpu_util`. If < 50%, CPU offloading is active. Increase `--gpu-layers`.
- If errors occur, check `--timeout` and `--max-tokens`. Set reasonable limits.

**6. Use-Case Specific Advice**
- **Coding**: Long outputs require syntax correctness. MTP helps if draft model is code-tuned. Watch for hallucinated syntax.
- **RAG**: Long contexts may truncate. Use `--ctx-size` matching retrieval window. MTP less critical.
- **Agentic**: Multi-step generation needs consistency. MTP acceptable if acceptance rate stable.
- **Chat**: Short outputs. MTP excellent for conversational flow.
- **Creative Writing**: Long, coherent text. MTP helps maintain speed. Monitor for style drift.

# SQLite Queries

SQLite is your analysis engine. Use these queries to extract insights, compare configurations, and identify bottlenecks.

**1. Schema Recap**
```sql
CREATE TABLE requests (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  ts TEXT NOT NULL,
  model TEXT,
  draft_model TEXT,
  mtp_enabled INTEGER DEFAULT 0,
  prompt_tokens INTEGER,
  completion_tokens INTEGER,
  total_tokens INTEGER,
  prefill_ms REAL,
  decode_ms REAL,
  total_ms REAL,
  gpu_mem_peak_mb REAL,
  draft_acceptance_rate REAL,
  status TEXT,
  prompt_preview TEXT
);
CREATE INDEX idx_mtp_ts ON requests(mtp_enabled, ts);
CREATE INDEX idx_status ON requests(status);
```

**2. Core Queries**
- **Average vs Median Latency by MTP**:
  ```sql
  SELECT 
    mtp_enabled,
    AVG(total_ms) AS avg_latency,
    MIN(total_ms) AS min_latency,
    MAX(total_ms) AS max_latency,
    COUNT(*) AS n
  FROM requests
  GROUP BY mtp_enabled;
  ```
  Note: SQLite lacks `MEDIAN`. Use Python/pandas for true median: `df.groupby('mtp_enabled')['total_ms'].median()`.

- **GPU Memory Correlation**:
  ```sql
  SELECT 
    mtp_enabled,
    AVG(gpu_mem_peak_mb) AS avg_mem_mb,
    AVG(total_ms) AS avg_latency_ms,
    AVG(draft_acceptance_rate) AS avg_acceptance
  FROM requests
  GROUP BY mtp_enabled;
  ```

- **Time Series Analysis**:
  ```sql
  SELECT 
    strftime('%H:%M', ts) AS hour_minute,
    AVG(total_ms) AS avg_latency,
    AVG(gpu_mem_peak_mb) AS avg_mem,
    COUNT(*) AS requests
  FROM requests
  GROUP BY hour_minute
  ORDER BY hour_minute;
  ```

- **Error Rate by Configuration**:
  ```sql
  SELECT 
    mtp_enabled,
    status,
    COUNT(*) AS count,
    ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (PARTITION BY mtp_enabled), 2) AS pct
  FROM requests
  GROUP BY mtp_enabled, status;
  ```

- **Draft Acceptance vs Latency**:
  ```sql
  SELECT 
    mtp_enabled,
    AVG(draft_acceptance_rate) AS acceptance,
    AVG(total_ms) AS latency,
    AVG(completion_tokens / (decode_ms / 1000.0)) AS tps
  FROM requests
  WHERE mtp_enabled = 1
  GROUP BY mtp_enabled;
  ```

- **Prompt Length Impact**:
  ```sql
  SELECT 
    CASE 
      WHEN prompt_tokens < 500 THEN 'short'
      WHEN prompt_tokens BETWEEN 500 AND 2000 THEN 'medium'
      ELSE 'long'
    END AS prompt_bucket,
    mtp_enabled,
    AVG(total_ms) AS avg_latency,
    AVG(gpu_mem_peak_mb) AS avg_mem
  FROM requests
  GROUP BY prompt_bucket, mtp_enabled;
  ```

**3. Export & Visualization**
- Export to CSV for plotting:
  ```bash
  sqlite3 lab.db "SELECT * FROM requests WHERE ts >= '2024-01-01' AND ts < '2024-01-02' LIMIT 1000;" > export.csv
  ```
- Use Python/matplotlib or Grafana for charts. Plot latency distribution, TPS over time, VRAM vs context length.

**4. Indexing & Performance**
- Add indexes for frequent filters:
  ```sql
  CREATE INDEX idx_mtp_status ON requests(mtp_enabled, status);
  CREATE INDEX idx_ts ON requests(ts);
  ```
- Vacuum periodically: `VACUUM;` to reclaim space.

# Dashboard Screenshots To Capture

Visual monitoring complements SQLite analysis. Capture these screenshots during testing to document behavior and identify anomalies.

**1. Flight Recorder Dashboard**
- **Request Timeline**: Shows prefill vs decode split. Look for consistent prefill time. Spikes indicate draft loading or context switching.
- **Latency Distribution**: Histogram of total_ms. Should be unimodal. Bimodal suggests two modes (e.g., cached vs uncached).
- **Draft Model Load Time**: If logged, track initialization overhead. Should be < 500ms after warm-up.
- **Response Size**: Tokens generated vs max_tokens. Check for truncation.
- **Screenshot Tip**: Annotate key events: "MTP enabled", "Context switch", "OOM warning", "Draft load".

**2. ServerTop Dashboard**
- **GPU Memory Usage**: Line chart of `gpu_mem_used` over time. Should be stable. Spikes indicate KV cache allocation or draft loading.
- **GPU Utilization**: Line chart of `gpu_util`. Should be > 70% during generation. Dips indicate idle time or CPU offloading.
- **Temperature & Fan**: Monitor thermal throttling. > 85°C may trigger performance drops.
- **PCIe Bandwidth**: High bandwidth during prefill indicates data transfer bottlenecks.
- **Screenshot Tip**: Capture at consistent intervals. Overlay with request timestamps from SQLite.

**3. OpenWebUI Dashboard**
- **Chat Interface**: Observe streaming behavior. MTP should show consistent token generation.
- **System Logs**: Check for warnings: "KV cache full", "Draft model failed", "Timeout".
- **Model Info**: Verify model loaded, draft model loaded, context size, batch size.
- **Screenshot Tip**: Include model settings, prompt, output, and any error messages.

**4. Custom Metrics Dashboard**
- **TPS Over Time**: Should be stable. Drops indicate memory pressure or draft verification stalls.
- **VRAM vs Context Length**: Plot to find optimal `--ctx-size`.
- **Draft Acceptance Rate**: Should be > 0.4. Drops indicate model mismatch.
- **Screenshot Tip**: Use Grafana or Python/matplotlib. Include legends, axes labels, and time range.

**5. What to Look For**
- **Consistency**: Low variance in latency, TPS, VRAM.
- **Efficiency**: High GPU utilization, low CPU offloading.
- **Stability**: No OOM, no timeouts, no dropped tokens.
- **Draft Performance**: Acceptance rate > 0.4, stable over time.
- **Anomalies**: Spikes, drops, warnings. Investigate root cause.

# Weekend Checklist

Follow this step-by-step plan to execute your benchmark efficiently.

**Friday Evening: Preparation**
- [ ] Backup current server config and SQLite database.
- [ ] Install/verify Flight Recorder, ServerTop, SQLite, Python/pandas.
- [ ] Define test prompt set: 10 short, 10 medium, 10 long, 10 agentic.
- [ ] Prepare launch commands:
  - Baseline: `llama-server --model main.gguf --ctx-size 4096 --batch-size 512 --gpu-layers 99 --threads 8`
  - MTP: `llama-server --model main.gguf --model-draft draft.gguf --speculative --ctx-size 4096 --batch-size 512 --gpu-layers 99 --threads 8`
- [ ] Set up SQLite schema and logging script.
- [ ] Verify ServerTop logging to CSV.
- [ ] Close background apps, set CPU governor to performance.

**Saturday Morning: Baseline Testing**
- [ ] Start server with baseline config.
- [ ] Warm up: 5 dummy requests.
- [ ] Run 50 baseline requests (randomized order).
- [ ] Log to SQLite, capture Flight Recorder, ServerTop.
- [ ] Verify data integrity: token counts, timestamps, status codes.
- [ ] Export baseline data to CSV.

**Saturday Afternoon: MTP Testing**
- [ ] Restart server with MTP config.
- [ ] Warm up: 5 dummy requests.
- [ ] Run 50 MTP requests (randomized order).
- [ ] Log to SQLite, capture Flight Recorder, ServerTop.
- [ ] Verify data integrity.
- [ ] Export MTP data to CSV.
- [ ] Repeat 2 more times with randomized order.

**Sunday Morning: Analysis**
- [ ] Load data into Python/pandas.
- [ ] Run SQLite queries for latency, TPS, VRAM, acceptance rate.
- [ ] Calculate median, confidence intervals, error rates.
- [ ] Generate charts: latency distribution, TPS over time, VRAM vs context.
- [ ] Compare MTP vs baseline: Is MTP faster? Stable? VRAM safe?
- [ ] Document findings: pros, cons, recommendations.

**Sunday Afternoon: Decision & Iteration**
- [ ] Decide on configuration: Keep MTP, drop MTP, or adjust draft model.
- [ ] Test edge cases: Long prompts, high batch size, low VRAM.
- [ ] Update server config based on results.
- [ ] Archive data, logs, screenshots.
- [ ] Write summary report: methodology, results, recommendations.
- [ ] Plan next iteration if needed.

# Interpreting Results

Data interpretation turns numbers into decisions. Use these guidelines to evaluate your results accurately.

**1. Key Metrics Breakdown**
- **TTFT (Prefill + Draft Load)**: MTP often increases TTFT by 100–300ms due to draft initialization. Acceptable if decode speed improves.
- **TPS (Decode Speed)**: MTP should improve TPS by 20–50% if draft acceptance > 0.4. Lower acceptance means overhead outweighs gains.
- **Total Latency**: Depends on output length. For short outputs (<500 tokens), MTP may not help. For long outputs, MTP shines.
- **VRAM Stability**: Peak memory should be < 85% of total. Spikes indicate fragmentation or draft loading.
- **Draft Acceptance Rate**: > 0.5 is excellent. 0.3–0.5 is marginal. < 0.3 means MTP is hurting performance.
- **Error Rate**: < 1% is acceptable. OOM, timeouts, or truncation indicate configuration issues.

**2. When to Keep MTP**
- Draft acceptance rate > 0.4 consistently.
- TPS improvement > 15% over baseline.
- VRAM stable, no fragmentation.
- Short to medium prompts (<2k tokens).
- Consistent latency, low variance.

**3. When to Drop MTP**
- Draft acceptance rate < 0.3.
- TPS improvement < 5% or negative.
- VRAM spikes, fragmentation, or OOM.
- Long prompts (>2k tokens) where prefill dominates.
- High latency variance or jitter.

**4. Use-Case Recommendations**
- **Coding**: MTP helpful for generation speed. Use code-tuned draft model. Monitor syntax correctness. Keep `--ctx-size` matching codebase context.
- **RAG**: MTP less critical. Focus on context window, retrieval accuracy, and prompt engineering. Use larger `--ctx-size` if needed.
- **Agentic**: MTP acceptable if acceptance rate stable. Prioritize consistency over raw speed. Use moderate batch size.
- **Chat**: MTP excellent. Short prompts, fast generation, conversational flow. Keep draft model small for low overhead.
- **Creative Writing**: MTP helpful for long outputs. Monitor coherence and style drift. Use `--ctx-size` matching output length.

**5. Practical Decision Framework**
- If MTP TPS > baseline by >20% AND VRAM < 80% AND acceptance > 0.4 → Keep MTP.
- If MTP TPS > baseline by 5–20% AND VRAM < 85% AND acceptance > 0.3 → Conditional keep. Test edge cases.
- If MTP TPS < baseline OR VRAM > 85% OR acceptance < 0.3 → Drop MTP. Optimize baseline instead.
- Always prioritize stability over speed. Inconsistent performance breaks user experience.

**6. Iteration Tips**
- Adjust `--speculative` based on acceptance rate. Increase if acceptance > 0.6, decrease if < 0.4.
- Try different draft models. Fine-tuned drafts outperform generic ones.
- Tune `--batch-size` and `--ctx-size` for your workload.
- Monitor system load. Background processes can skew results.
- Re-test after server restarts. KV cache behavior may change.

# Common Mistakes

Avoid these pitfalls to ensure accurate, actionable results.

**1. Cherry-Picking Fast Runs**
- Mistake: Only logging requests that look fast.
- Impact: Biased results, false confidence.
- Fix: Log all requests. Discard only those with external errors (network, OOM, proxy timeout).

**2. Ignoring Prefill vs Decode Split**
- Mistake: Focusing only on total latency.
- Impact: Missing prefill bottlenecks or draft loading overhead.
- Fix: Track `prefill_ms` and `decode_ms` separately. MTP only affects decode.

**3. Assuming MTP Always Speeds Up First Response**
- Mistake: Expecting MTP to reduce TTFT.
- Impact: Disappointment when first response is slower.
- Fix: Understand MTP accelerates decode, not prefill. First response includes draft load. Acceptable if decode improves.

**4. Not Warming Up GPU/Model**
- Mistake: Starting tests immediately after server launch.
- Impact: High variance due to kernel compilation, VRAM allocation, draft loading.
- Fix: Run 5–10 warm-up requests before collecting data.

**5. Forgetting Draft Model VRAM Overhead**
- Mistake: Ignoring draft model memory usage.
- Impact: OOM, fragmentation, performance drops.
- Fix: Monitor `gpu_mem_used`. Keep headroom for KV cache + draft model.

**6. Using Mean Instead of Median for Latency**
- Mistake: Averaging latency values.
- Impact: Skewed results due to outliers.
- Fix: Use median. Latency distributions are heavily skewed.

**7. Not Controlling System Load**
- Mistake: Running tests with background updates, downloads, or other apps.
- Impact: Jitter, inconsistent results.
- Fix: Close background apps, set CPU governor to performance, isolate GPU.

**8. Misinterpreting Streaming vs Non-Streaming Metrics**
- Mistake: Comparing streaming TTFT with non-streaming total latency.
- Impact: Confusion about MTP impact.
- Fix: Use consistent mode. Streaming shows token-by-token latency. Non-streaming shows total.

**9. Overlooking KV Cache Fragmentation**
- Mistake: Ignoring memory fragmentation signs.
- Impact: VRAM spikes, performance drops, OOM.
- Fix: Monitor `gpu_mem_used` over time. Use `--cache-reuse` or increase `--ctx-size` gradually.

**10. Not Logging Draft Acceptance Rate**
- Mistake: Assuming MTP works without tracking acceptance.
- Impact: Inability to optimize draft model or `--speculative` value.
- Fix: Log acceptance rate. Target > 0.4. Adjust draft model or speculation depth accordingly.

**11. Testing with Unrepresentative Prompts**
- Mistake: Using only short prompts or only long prompts.
- Impact: Biased results, poor generalization.
- Fix: Use diverse prompt set: short, medium, long, agentic, RAG, coding, creative.

**12. Ignoring Error Rates**
- Mistake: Focusing only on speed, ignoring failures.
- Impact: Unreliable system, poor user experience.
- Fix: Track error rate. Target < 1%. Investigate OOM, timeouts, truncation.

**13. Not Repeating Tests**
- Mistake: Running tests once.
- Impact: High variance, unreliable conclusions.
- Fix: Run 3+ times with randomized order. Calculate confidence intervals.

**14. Overlooking Temperature and Seed Consistency**
- Mistake: Changing temperature or seed between runs.
- Impact: Different output lengths, skewed metrics.
- Fix: Keep temperature, seed, max_tokens constant. Only vary MTP config.

**15. Assuming One Size Fits All**
- Mistake: Applying same config to all workloads.
- Impact: Suboptimal performance for specific tasks.
- Fix: Tune config per use-case. Coding needs low TTFT. RAG needs large context. Chat needs fast decode. Creative needs long coherence.

By following this guide, you'll transform guesswork into data-driven decisions. MTP can significantly improve your local llama.cpp server's performance, but only when configured correctly and evaluated rigorously. Use the weekend plan to collect clean data, analyze it with SQLite and Python, and make informed choices about your model, draft, and server settings. Remember: consistency beats speed, stability beats hype, and data beats intuition. Happy benchmarking.

**16. Overlooking Quantization Effects on Draft Models**
- Mistake: Using the same quantization for draft and main models.
- Impact: Mismatched precision causes acceptance rate drops and increased VRAM fragmentation.
- Fix: Match quantization formats (e.g., Q4_K_M for both, or Q5_K_M for main + Q4_K_M for draft). Test Q8_0 draft if VRAM allows for higher acceptance.

**17. Ignoring Systemd/Service Restart Behavior**
- Mistake: Assuming `--ctx-size` or `--batch-size` persists across restarts without explicit config.
- Impact: Server falls back to defaults, breaking benchmark consistency.
- Fix: Use a dedicated config file (`server.conf`) or systemd drop-in to pin all flags. Log the exact launch command in SQLite for reproducibility.

**18. Not Accounting for OS Page Cache Interference**
- Mistake: Running benchmarks while the OS caches model weights or logs.
- Impact: First few requests appear artificially fast, skewing TTFT.
- Fix: Clear page cache before each run: `echo 3 > /proc/sys/vm/drop_caches`. Run tests in a clean session or use `systemd-run --scope` to isolate memory accounting.

**19. Assuming Higher Batch Size Always Improves Throughput**
- Mistake: Cranking `--batch-size` to 1024+ without monitoring VRAM or latency variance.
- Impact: Diminishing returns, increased jitter, and potential OOM under concurrent load.
- Fix: Step `--batch-size` in increments of 128. Plot TPS vs batch size. Stop at the knee of the curve. For MTP, keep batch ≤ 512 to avoid draft verification bottlenecks.

**20. Forgetting to Reset OpenWebUI Session State**
- Mistake: Testing in the same browser session with cached prompts or model state.
- Impact: Proxy or frontend caches skew timing, and session variables leak between runs.
- Fix: Use incognito windows, clear frontend cache, or automate requests via `curl`/Python. Never rely on UI-based timing for production benchmarks.

# Advanced MTP & Speculative Decoding Tuning

Speculative decoding is highly sensitive to model alignment, quantization, and context dynamics. Fine-tuning these parameters can push MTP from marginal to transformative.

**1. Draft Model Selection Strategy**
- **Architecture Match**: Draft models should share the same architecture family as the main model (e.g., Llama-3 draft for Llama-3 main). Cross-architecture drafts yield < 0.2 acceptance.
- **Parameter Ratio**: Draft should be 1/4 to 1/8 the size of the main model. A 7B main pairs well with a 1.5B or 0.5B draft. Larger drafts increase VRAM overhead without proportional acceptance gains.
- **Quantization Balance**: Q4_K_M draft + Q5_K_M main is a sweet spot. Q8_0 draft improves acceptance by 5–10% but costs ~2x VRAM. Test Q3_K_S only if VRAM is critically constrained.
- **Fine-Tuning Advantage**: A draft fine-tuned on your target domain (code, medical, legal) outperforms generic drafts by 15–25% in acceptance rate. Use LoRA adapters or full fine-tuning if you have compute.

**2. Speculative Depth (`--speculative`)**
- Controls how many tokens the draft predicts before verification.
- **Rule of Thumb**: Set to `floor(acceptance_rate * 2)`. If acceptance is 0.5, use 4. If 0.7, use 6–8.
- **Diminishing Returns**: Beyond 8–10 draft tokens, verification overhead dominates. Main model must re-evaluate the entire draft sequence, increasing prefill latency.
- **Dynamic Adjustment**: If your workload varies, consider a script that adjusts `--speculative` based on real-time acceptance metrics.

**3. KV Cache & Draft Synchronization**
- MTP requires the draft model to process the same prompt as the main model. This doubles prefill compute for the initial request.
- **Mitigation**: Use `--cache-reuse` to share KV cache entries across requests. Enable `--flash-attn` if supported to reduce memory footprint.
- **Context Shifting**: If prompts vary significantly in length, KV cache reallocation causes VRAM spikes. Pad prompts to a fixed length or use `--ctx-size` matching your 90th percentile.

**4. Advanced MTP Flags**
- `--speculative-draft`: Explicitly set draft model path.
- `--speculative-num`: Number of draft tokens (alternative to `--speculative`).
- `--draft-repetition-penalty`: Prevents draft from looping. Set to 1.1–1.3.
- `--draft-temperature`: Keep low (0.1–0.3) for deterministic drafting. Higher temps increase acceptance variance.
- `--draft-top-p`: Restrict draft sampling to top-p=0.9 to avoid low-probability hallucinations.

**5. When MTP Fails & How to Recover**
- **Symptom**: Acceptance rate drops to < 0.2 after 500 tokens.
- **Cause**: Context drift, draft model mismatch, or KV cache fragmentation.
- **Fix**: Reset server, reduce `--speculative`, switch to a smaller draft, or enable `--cache-reuse`. Monitor `gpu_mem_peak_mb` for fragmentation signs.

# Hardware-Specific Optimization Strategies

Your hardware stack dictates how llama.cpp manages memory, compute, and concurrency. Tailor your configuration to your GPU architecture.

**1. NVIDIA (CUDA)**
- **VRAM Management**: CUDA uses a unified memory pool. MTP draft + main model + KV cache must fit within `gpu_mem_used`. Keep headroom > 15%.
- **Flags**: `--gpu-layers 99` (offload all), `--flash-attn` (reduces KV cache by 40%), `--mlock` (pins memory to prevent swap).
- **Concurrency**: Use `--parallel` for OpenWebUI. Set to 2–4 for single GPU. Higher values cause queueing delays.
- **Monitoring**: `nvidia-smi dmon -s u` for real-time utilization. `nvtop` for memory breakdown.
- **Troubleshooting**: If `CUDA_ERROR_OUT_OF_MEMORY`, reduce `--ctx-size` or `--batch-size`. Enable `--cache-type-k v` to use FP16 for key cache.

**2. AMD (ROCm)**
- **VRAM Management**: ROCm uses separate memory pools for compute and display. MTP can cause pool fragmentation.
- **Flags**: `--gpu-layers 99`, `--rocm-mem-type unified` (if supported), `--flash-attn`.
- **Concurrency**: ROCm handles parallelism differently. Use `--parallel 2` and monitor `rocm-smi`.
- **Monitoring**: `rocm-smi --showmeminfo vram` for usage. `radeontop` for real-time metrics.
- **Troubleshooting**: If generation stalls, check `HIP_VISIBLE_DEVICES`. Ensure ROCm version matches driver. Update to ROCm 6.1+ for better MTP support.

**3. Apple Silicon (Metal/MPS)**
- **VRAM Management**: Unified memory shares CPU/GPU. MTP draft + main model + KV cache competes with system RAM.
- **Flags**: `--gpu-layers 99`, `--mlock`, `--ctx-size` matching available RAM.
- **Concurrency**: MPS handles parallelism efficiently. Use `--parallel 4–8`.
- **Monitoring**: `powermetrics -sam smc` for thermal throttling. `Activity Monitor` for memory pressure.
- **Troubleshooting**: If performance drops, check `sysctl hw.memsize`. Disable background apps. Use `--cache-type-k q8_0` to reduce memory footprint.

**4. Multi-GPU & NVLink**
- **Sharding**: Use `--tensor-split` to distribute layers across GPUs. MTP draft should reside on the primary GPU.
- **Flags**: `--tensor-split 50,50`, `--gpu-layers 99`, `--flash-attn`.
- **Monitoring**: `nvidia-smi pmon` for per-GPU utilization. Check NVLink bandwidth with `nvidia-smi topo -m`.
- **Troubleshooting**: If cross-GPU latency spikes, reduce `--parallel` or disable MTP. NVLink handles KV cache transfer better than PCIe.

# Automated Benchmarking & Reporting Pipeline

Manual testing is error-prone. Automate your pipeline for reproducibility, statistical rigor, and rapid iteration.

**1. Python Orchestration Script**
```python
import subprocess
import time
import sqlite3
import pandas as pd
import matplotlib.pyplot as plt
import json

DB_PATH = "lab.db"
BASE_CMD = "llama-server --model main.gguf --ctx-size 4096 --batch-size 512 --gpu-layers 99 --threads 8"
MTP_CMD = "llama-server --model main.gguf --model-draft draft.gguf --speculative --ctx-size 4096 --batch-size 512 --gpu-layers 99 --threads 8"

PROMPTS = [
    "Explain quantum entanglement in 3 sentences.",
    "Write a Python function to sort a list using quicksort.",
    "Summarize the key points of the provided RAG context.",
    "Generate a multi-step plan to deploy a microservice on Kubernetes.",
    "Write a 500-word creative story about a time-traveling librarian."
]

def run_benchmark(cmd, n_runs=50):
    subprocess.Popen(cmd.split())
    time.sleep(5)  # Warm-up
    results = []
    for i in range(n_runs):
        prompt = PROMPTS[i % len(PROMPTS)]
        resp = subprocess.run(
            ["curl", "-s", "-X", "POST", "http://localhost:8080/v1/chat/completions",
             "-H", "Content-Type: application/json",
             "-d", json.dumps({"model": "main.gguf", "messages": [{"role": "user", "content": prompt}],
                               "stream": False, "max_tokens": 256, "temperature": 0.7, "seed": 42})],
            capture_output=True, text=True
        )
        data = json.loads(resp.stdout)
        results.append({
            "ts": time.strftime("%Y-%m-%d %H:%M:%S"),
            "model": "main.gguf",
            "draft_model": "draft.gguf" if "MTP" in cmd else None,
            "mtp_enabled": 1 if "MTP" in cmd else 0,
            "prompt_tokens": data.get("usage", {}).get("prompt_tokens", 0),
            "completion_tokens": data.get("usage", {}).get("completion_tokens", 0),
            "total_tokens": data.get("usage", {}).get("total_tokens", 0),
            "total_ms": data.get("metrics", {}).get("total_time", 0),
            "prefill_ms": data.get("metrics", {}).get("prompt_eval_time", 0),
            "decode_ms": data.get("metrics", {}).get("eval_time", 0),
            "status": "success"
        })
    return results

def save_to_sqlite(results):
    conn = sqlite3.connect(DB_PATH)
    df = pd.DataFrame(results)
    df.to_sql("requests", conn, if_exists="append", index=False)
    conn.close()

# Run baseline & MTP
baseline = run_benchmark(BASE_CMD)
mtp = run_benchmark(MTP_CMD)
save_to_sqlite(baseline + mtp)
```

**2. Automated Reporting**
- Generate PDF/HTML reports with `jinja2` templates.
- Include latency distributions, TPS charts, VRAM heatmaps, and acceptance rates.
- Email or Slack notifications on failure thresholds (e.g., error rate > 2%).

**3. CI/CD Integration**
- Run benchmarks on every model/draft update.
- Store results in Git LFS or S3.
- Compare against baseline using statistical tests (t-test, Mann-Whitney U).

# Advanced SQLite Analytics & Window Functions

Move beyond basic aggregations. Use window functions and rolling metrics for production-grade analysis.

**1. Rolling Average TPS**
```sql
SELECT 
  ts,
  mtp_enabled,
  completion_tokens / (decode_ms / 1000.0) AS tps,
  AVG(completion_tokens / (decode_ms / 1000.0)) OVER (
    PARTITION BY mtp_enabled ORDER BY ts ROWS BETWEEN 9 PRECEDING AND CURRENT ROW
  ) AS rolling_avg_tps_10
FROM requests
ORDER BY ts;
```

**2. Percentile Latency Breakdown**
```sql
SELECT 
  mtp_enabled,
  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY total_ms) AS p50_latency,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY total_ms) AS p95_latency,
  PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY total_ms) AS p99_latency,
  COUNT(*) AS n
FROM requests
GROUP BY mtp_enabled;
```
Note: SQLite lacks `PERCENTILE_CONT`. Use Python/pandas: `df.groupby('mtp_enabled')['total_ms'].quantile([0.5, 0.95, 0.99])`.

**3. Correlation: Prompt Length vs TPS**
```sql
SELECT 
  mtp_enabled,
  CORR(prompt_tokens, completion_tokens / (decode_ms / 1000.0)) AS prompt_tps_correlation,
  AVG(prompt_tokens) AS avg_prompt_len,
  AVG(completion_tokens / (decode_ms / 1000.0)) AS avg_tps
FROM requests
GROUP BY mtp_enabled;
```

**4. Draft Acceptance Trend Analysis**
```sql
SELECT 
  strftime('%H', ts) AS hour,
  AVG(draft_acceptance_rate) AS avg_acceptance,
  STDDEV(draft_acceptance_rate) AS acceptance_stddev,
  COUNT(*) AS requests
FROM requests
WHERE mtp_enabled = 1
GROUP BY hour
ORDER BY hour;
```

**5. Memory Fragmentation Detection**
```sql
SELECT 
  ts,
  gpu_mem_peak_mb,
  gpu_mem_peak_mb - LAG(gpu_mem_peak_mb) OVER (ORDER BY ts) AS mem_delta_mb,
  CASE WHEN gpu_mem_peak_mb - LAG(gpu_mem_peak_mb) OVER (ORDER BY ts) > 500 THEN 'SPIKE' ELSE 'STABLE' END AS mem_status
FROM requests
ORDER BY ts;
```

# Dashboard & Observability Deep Dive

Visual monitoring transforms raw logs into actionable insights. Configure your stack for real-time visibility.

**1. Flight Recorder Configuration**
- Enable JSON logging: `--log-file flight_recorder.json`
- Parse with `jq`: `jq '.[] | {ts, ttft, tps, mtp}' flight_recorder.json > parsed.json`
- Visualize with `Grafana` or `Kibana`.

**2. ServerTop Integration**
- Export to InfluxDB: `servertop --output influxdb://localhost:8086/benchmark`
- Query with Flux:
  ```flux
  from(bucket: "benchmark")
    |> range(start: -1h)
    |> filter(fn: (r) => r._measurement == "gpu")
    |> aggregateWindow(every: 1m, fn: mean)
  ```

**3. Custom Metrics Endpoint**
- Expose Prometheus metrics via `--metrics` flag (if supported) or custom wrapper.
- Key metrics: `llama_ttft_seconds`, `llama_tps`, `llama_gpu_mem_bytes`, `llama_mtp_acceptance_rate`.
- Scrape with `prometheus.yml`:
  ```yaml
  scrape_configs:
    - job_name: 'llama-server'
      static_configs:
        - targets: ['localhost:8080']
  ```

**4. Alerting Rules**
- `llama_tps < 10 for 5m` → Trigger Slack alert.
- `llama_gpu_mem_bytes > 0.85 * total_gpu_mem` → Warn of OOM risk.
- `llama_mtp_acceptance_rate < 0.3 for 10m` → Suggest draft model swap.

**5. Dashboard Layout**
- **Top Row**: Real-time TPS, TTFT, VRAM usage, GPU utilization.
- **Middle Row**: Latency distribution histogram, draft acceptance rate trend, error rate gauge.
- **Bottom Row**: Request queue depth, context length distribution, MTP vs baseline comparison.

# Use-Case Parameter Matrices

Different workloads demand different configurations. Use these matrices to optimize your server for specific tasks.

**1. Coding**
- **Priority**: Low TTFT, syntax correctness, context retention.
- **Flags**: `--ctx-size 8192`, `--batch-size 256`, `--speculative 4`, `--draft-temperature 0.1`
- **Draft Model**: Code-tuned 1.5B Q4_K_M.
- **Monitoring**: Track syntax error rate, token truncation, and prefill time.
- **Tip**: Use `--repetition-penalty 1.1` to avoid code loops. Enable `--flash-attn` for large codebases.

**2. RAG**
- **Priority**: Context window, retrieval accuracy, low hallucination.
- **Flags**: `--ctx-size 16384`, `--batch-size 128`, `--speculative 0` (disable MTP), `--cache-reuse`
- **Draft Model**: None. MTP adds overhead without RAG benefits.
- **Monitoring**: Track context utilization, retrieval latency, and answer relevance.
- **Tip**: Use `--cache-type-k q8_0` to fit larger contexts. Pre-chunk documents to match `--ctx-size`.

**3. Agentic**
- **Priority**: Consistency, tool calling accuracy, state management.
- **Flags**: `--ctx-size 4096`, `--batch-size 256`, `--speculative 6`, `--draft-top-p 0.9`
- **Draft Model**: General-purpose 1.5B Q5_K_M.
- **Monitoring**: Track tool call success rate, state drift, and latency variance.
- **Tip**: Use `--seed` for deterministic planning. Monitor `gpu_mem_peak_mb` for state accumulation.

**4. Chat**
- **Priority**: Low TTFT, conversational flow, high concurrency.
- **Flags**: `--ctx-size 2048`, `--batch-size 512`, `--speculative 8`, `--draft-temperature 0.2`
- **Draft Model**: Small 0.5B Q4_K_M.
- **Monitoring**: Track TTFT, TPS, and user satisfaction metrics.
- **Tip**: Use `--parallel 4` for multi-user. Keep `--ctx-size` tight to minimize prefill.

**5. Creative Writing**
- **Priority**: Long coherence, style consistency, low repetition.
- **Flags**: `--ctx-size 8192`, `--batch-size 256`, `--speculative 4`, `--repetition-penalty 1.15`
- **Draft Model**: 1.5B Q5_K_M with creative fine-tuning.
- **Monitoring**: Track token diversity, style drift, and generation length.
- **Tip**: Use `--top-p 0.95` and `--temperature 0.8` for creativity. Monitor draft acceptance for coherence drops.

# Troubleshooting Matrix

When things go wrong, use this matrix to diagnose and fix issues quickly.

| Symptom | Likely Cause | Diagnostic Query/Command | Fix |
|---------|--------------|--------------------------|-----|
| TTFT spikes on first request | Draft model loading, KV cache allocation | `SELECT ts, prefill_ms FROM requests ORDER BY ts LIMIT 5;` | Warm up server, use `--cache-reuse`, pin draft model |
| TPS drops after 500 tokens | KV cache fragmentation, VRAM pressure | `SELECT ts, gpu_mem_peak_mb FROM requests ORDER BY ts;` | Reduce `--ctx-size`, enable `--flash-attn`, increase `--batch-size` gradually |
| Draft acceptance < 0.3 | Model mismatch, quantization mismatch, high temp | `SELECT AVG(draft_acceptance_rate) FROM requests WHERE mtp_enabled=1;` | Switch to matching architecture, lower `--draft-temperature`, use Q5_K_M draft |
| OOM errors | VRAM exceeded, context too large, batch too high | `nvidia-smi --query-gpu=memory.used --format=csv` | Reduce `--ctx-size`, lower `--batch-size`, disable MTP, use `--cache-type-k q8_0` |
| High latency variance | System load, driver interrupts, background apps | `top`, `htop`, `dmesg \| tail` | Close background apps, set CPU governor to performance, isolate GPU |
| Streaming stalls | Network timeout, proxy buffering, server overload | Check Flight Recorder for `timeout` or `429` status | Increase `--timeout`, adjust proxy buffer size, reduce `--parallel` |
| Context truncation | `--ctx-size` too small, prompt too long | `SELECT prompt_tokens FROM requests WHERE prompt_tokens > 2000;` | Increase `--ctx-size`, implement prompt chunking, use `--cache-reuse` |
| Draft model fails to load | Path error, incompatible format, VRAM full | Check server logs for `failed to load draft model` | Verify path, use GGUF format, free VRAM, reduce `--speculative` |
| GPU utilization < 50% | CPU offloading, small batch, idle time | `servertop --interval 1` | Increase `--gpu-layers`, raise `--batch-size`, enable `--flash-attn` |
| Acceptance rate drops over time | Context shift, draft model mismatch, KV cache drift | `SELECT strftime('%H', ts), AVG(draft_acceptance_rate) FROM requests GROUP BY hour;` | Reset server, reduce `--speculative`, switch draft model, enable `--cache-reuse` |

# Reference Glossary & Quick Flags

Keep this reference handy for quick configuration adjustments and metric interpretation.

**Key Metrics**
- **TTFT (Time To First Token)**: Prefill + draft load + first decode. Lower is better for chat.
- **TPS (Tokens Per Second)**: Decode phase speed. Higher is better for generation.
- **Total Latency**: TTFT + decode time. Depends on output length.
- **Draft Acceptance Rate**: % of draft tokens accepted by main model. Target > 0.4.
- **KV Cache**: Memory storing attention states. Grows with context length.
- **Prefill**: Processing the prompt. Compute-bound, not affected by MTP.
- **Decode**: Generating tokens one by one. MTP accelerates this phase.

**Essential Flags**
- `--model`: Main model path.
- `--model-draft`: Draft model path for MTP.
- `--speculative` / `--speculative-num`: Number of draft tokens.
- `--ctx-size`: Context window length.
- `--batch-size`: Tokens per forward pass.
- `--gpu-layers`: Layers offloaded to GPU. Use 99 for full offload.
- `--threads`: CPU threads for CPU fallback.
- `--flash-attn`: Enable flash attention (reduces KV cache).
- `--cache-reuse`: Share KV cache across requests.
- `--cache-type-k` / `--cache-type-v`: KV cache precision (f16, q8_0, q4_0).
- `--parallel`: Concurrent requests.
- `--timeout`: Request timeout in seconds.
- `--log-file`: Output JSON logs for analysis.

**Quantization Formats**
- **Q4_K_M**: Balanced speed/quality. Best for draft models.
- **Q5_K_M**: Slightly better quality. Good for main models.
- **Q8_0**: Highest quality. High VRAM cost. Use for critical tasks.
- **Q3_K_S**: Low VRAM. Quality drops. Avoid for production.
- **F16**: Full precision. Highest quality, highest VRAM. Use for testing.

**Hardware Recommendations**
- **NVIDIA RTX 4090**: 24GB VRAM. Ideal for 7B–13B models. Use `--flash-attn`.
- **NVIDIA A100**: 80GB VRAM. Ideal for 30B–70B models. Use `--tensor-split` for multi-GPU.
- **AMD RX 7900 XTX**: 24GB VRAM. ROCm 6.1+ required. Use `--rocm-mem-type unified`.
- **Apple M3 Max**: 128GB unified RAM. Ideal for large contexts. Use `--mlock`.

**Quick Tuning Rules**
- If TTFT > 2s: Reduce `--ctx-size`, warm up server, check draft load.
- If TPS < 15: Increase `--batch-size`, enable `--flash-attn`, check GPU utilization.
- If VRAM > 85%: Reduce `--ctx-size`, lower `--batch-size`, disable MTP, use `--cache-type-k q8_0`.
- If acceptance < 0.3: Switch draft model, lower `--draft-temperature`, reduce `--speculative`.
- If variance high: Close background apps, set CPU governor to performance, isolate GPU.

**Final Implementation Checklist**
- [ ] Verify model/draft paths and formats.
- [ ] Set fixed `--ctx-size`, `--batch-size`, `--gpu-layers`.
- [ ] Enable JSON logging and SQLite insertion.
- [ ] Configure ServerTop/Flight Recorder for continuous monitoring.
- [ ] Run warm-up requests before benchmarking.
- [ ] Collect 50+ samples per configuration with randomized order.
- [ ] Calculate median latency, confidence intervals, error rates.
- [ ] Compare MTP vs baseline using statistical tests.
- [ ] Adjust `--speculative`, draft model, and quantization based on results.
- [ ] Document findings and update server configuration.
- [ ] Automate pipeline for future iterations.

You now have a complete, production-ready framework for evaluating and optimizing your local llama.cpp server. MTP is a powerful tool, but it demands careful configuration, rigorous testing, and continuous monitoring. Use the data, respect the hardware limits, and let the metrics guide your decisions. Your weekend benchmark will transform from guesswork into a repeatable, scalable optimization pipeline. Deploy with confidence.