# Quick Diagnosis Before diving into weekend-long benchmarking, it’s critical to understand why your `llama.cpp` server feels inconsistent and what MTP (Multi-Token Prediction, often implemented as speculative decoding) actually does under the hood. Inconsistency in local LLM deployments rarely stems from a single broken component; it’s usually a cascade of interacting factors: KV cache allocation patterns, proxy routing overhead, GPU memory bandwidth saturation, thermal throttling, and prompt preprocessing variance. **1. GPU Memory Jumps Are Normal (But Not Inevitable)** When you load a model, `llama.cpp` allocates VRAM in chunks: model weights, KV cache, compute buffers, and speculative decoding buffers. MTP introduces a secondary, smaller model (the drafter) that runs in parallel. This causes: - Initial VRAM spike: ~10-20% extra for the drafter weights + speculative KV cache. - Dynamic VRAM fluctuation: As prompts vary in length, the KV cache resizes. If `--ctx-size` is set too high, you’ll see fragmentation. If too low, you’ll get context truncation. - Bandwidth pressure: MTP verifies multiple tokens per forward pass. If your GPU’s memory bandwidth is the bottleneck (common on consumer cards), verification stalls, causing TTFT spikes. **2. First-Answer Latency (TTFT) Variance** TTFT (Time To First Token) is dominated by: - Prompt preprocessing (tokenization, embedding, KV cache population) - Proxy TLS termination and request parsing (Flight Recorder) - GPU power limit throttling (if the card dips into boost clocks inconsistently) - Cold vs. warm KV cache state If you’re seeing 3-8 second variance on identical prompts, check: - `--log-level debug` in `llama.cpp` for preprocessing timestamps - Flight Recorder’s `request_start` vs `llama_start` delta - `nvidia-smi dmon` for power state transitions (P0 vs P8) - OpenWebUI’s streaming toggle (non-streaming buffers the entire response, masking TTFT) **3. Is MTP Actually Helping?** MTP trades VRAM and memory bandwidth for reduced Time To Next Token (TTNT). It helps when: - You’re generating long continuations (>50 tokens) - Your GPU has high compute but moderate memory bandwidth - Your prompts are deterministic (low temperature, structured outputs) It hurts when: - You’re doing short Q&A (<20 tokens) - Your GPU is bandwidth-bound (e.g., RTX 3060/4060 class) - You’re using high temperature (speculative tokens diverge, causing verification failures and re-computation) - KV cache is already near capacity **Immediate Triage Commands** Run these before your weekend test to establish baseline behavior: ```bash # Check GPU memory allocation and power state nvidia-smi -l 1 --query-gpu=memory.used,utilization.memory,utilization.gpu,power.draw,temperature.gpu --format=csv,noheader # Monitor llama.cpp preprocessing vs inference split llama-server -m your_model.gguf --gpu-layers 99 --log-level debug 2>&1 | grep -E "(preprocessing|kv_cache|speculative)" # Test direct API latency (bypass OpenWebUI & proxy) curl -s http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"local","messages":[{"role":"user","content":"Explain quantum entanglement in 3 sentences."}],"max_tokens":50,"temperature":0.1,"stream":false}' \ | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'TTFT: {d.get(\"usage\",{}).get(\"prompt_tokens\",0)} tokens, Latency: {d.get(\"usage\",{}).get(\"completion_tokens\",0)} tokens in {d.get(\"usage\",{}).get(\"total_tokens\",0)} tokens')" ``` If TTFT varies by >2s across 3 identical runs, your environment isn’t stabilized. Fix thermal/power settings, disable proxy temporarily, and ensure `--ctx-size` matches your typical prompt length. Only then proceed to MTP comparison. --- # Data Collection Plan A rigorous weekend benchmark requires systematic data collection. Ad-hoc testing leads to confirmation bias. You’ll collect data across three layers: inference metrics (ServerTop), request routing/proxy metrics (Flight Recorder), and structured logs (SQLite). Here’s how to set it up. **1. Environment Stabilization** - Set GPU power limit to max: `nvidia-smi -i 0 -pl 300` (adjust to your card) - Disable dynamic boost: `nvidia-smi -i 0 -c 3` (performance mode) - Pin CPU cores to avoid frequency scaling: `cpupower frequency-set -g performance` - Clear KV cache between runs: Restart `llama-server` or use `--no-mmap` if testing cold starts - Ensure consistent system load: Close background apps, disable auto-updates, set `vm.swappiness=1` **2. Flight Recorder Configuration** Flight Recorder sits between OpenWebUI and `llama.cpp`. It adds TLS termination, request queuing, and logging. To isolate its impact: - Enable JSON structured logging: `FR_LOG_FORMAT=json` - Export to SQLite or CSV: `FR_DB_PATH=./flight_recorder.db FR_DB_TABLE=requests` - Capture these fields: `request_id`, `timestamp`, `client_ip`, `model`, `prompt_tokens`, `completion_tokens`, `total_tokens`, `ttft_ms`, `ttnt_ms`, `total_latency_ms`, `status_code`, `cache_hit`, `mtp_enabled` - Run a baseline 10-request loop to measure proxy overhead: ```bash for i in {1..10}; do curl -s -o /dev/null -w "%{time_total}\n" http://localhost:3000/api/chat \ -H "Content-Type: application/json" \ -d '{"model":"local","messages":[{"role":"user","content":"Test prompt $i"}],"max_tokens":30,"stream":false}' done ``` Compare with direct `llama-server` calls. Proxy overhead should be <50ms for short prompts. If >100ms, check `FR_QUEUE_SIZE` and `FR_WORKERS`. **3. ServerTop Metric Collection** ServerTop provides real-time GPU/CPU metrics. Configure it to sample at 1-second intervals: - Enable CSV export: `SERVERTOP_CSV=./servertop.csv SERVERTOP_INTERVAL=1` - Track: `gpu_mem_used`, `gpu_mem_free`, `gpu_util`, `gpu_bw_util`, `cpu_util`, `temp_gpu`, `temp_cpu`, `fan_speed` - Correlate VRAM jumps with MTP verification phases. MTP causes periodic bandwidth spikes during draft verification. If `gpu_bw_util` hits 95%+ consistently, MTP is bandwidth-bound and likely hurting performance. **4. SQLite Schema & Logging Integration** Ensure `llama.cpp` logs are parsed into SQLite. Use a lightweight parser script: ```python # parse_llama_logs.py import sqlite3, re, json conn = sqlite3.connect("benchmark.db") c = conn.cursor() c.execute("""CREATE TABLE IF NOT EXISTS requests ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp TEXT, model TEXT, prompt_tokens INTEGER, completion_tokens INTEGER, total_tokens INTEGER, ttft_ms REAL, ttnt_ms REAL, total_latency_ms REAL, gpu_mem_mb INTEGER, mtp_enabled INTEGER, status TEXT )""") # Parse llama.cpp debug logs and Flight Recorder JSON into this schema ``` Run this parser after each test batch. It creates a queryable dataset for statistical analysis. **5. Controlled Variable Matrix** For each test configuration, log: - Model version & quantization (Q4_K_M vs Q5_K_S) - `--ctx-size`, `--batch-size`, `--n-predict` - `--speculative` value (0, 3, 5, 8) - Temperature, top_p, top_k - Prompt length (short: <100 tokens, medium: 200-500, long: 800-1500) - System prompt complexity (none, basic, detailed) - Streaming vs non-streaming - Proxy enabled/disabled Record all 10 variables per run. This enables multivariate analysis later. **6. Data Collection Workflow** 1. Cold boot GPU & server 2. Run 5 baseline requests (MTP=0) 3. Record ServerTop CSV, Flight Recorder JSON, SQLite insert 4. Warm up KV cache (run 3 identical prompts) 5. Run 5 MTP requests (MTP=5) 6. Repeat 3x per configuration 7. Clear cache, restart server, repeat for next config 8. Aggregate all data into SQLite This workflow eliminates warm/cold bias, controls for thermal state, and ensures statistical validity. --- # MTP Comparison Plan Comparing MTP vs non-MTP without fooling yourself requires statistical rigor, controlled variables, and awareness of speculative decoding’s hidden costs. MTP isn’t a magic speedup; it’s a tradeoff matrix. Here’s how to test it properly. **1. The Speculative Decoding Mechanism** MTP works by: - Drafting N tokens with a smaller, faster model - Verifying them in parallel with the main model - Accepting correct drafts, falling back to main model on mismatches Success rate depends on: - Draft model quality (matches main model’s distribution) - Prompt determinism (low temperature = high draft accuracy) - GPU memory bandwidth (verification is bandwidth-heavy) - Context length (longer contexts increase KV cache verification cost) **2. Controlled A/B Test Design** Use a paired experimental design: - Same model, same prompt, same temperature, same context window - Alternate MTP=0 and MTP=5 across runs - Randomize order to avoid time-of-day thermal drift - Run 15 iterations per configuration - Calculate mean, median, p95, p99 latency, tokens/sec, VRAM usage **3. Statistical Validation** Don’t rely on averages. Use: - Paired t-test (if normal distribution) or Wilcoxon signed-rank test (non-normal) - Effect size (Cohen’s d) to measure practical significance - Confidence intervals (95%) for latency metrics - Outlier removal: Discard runs where GPU throttled or proxy queued >200ms Example Python validation script: ```python import scipy.stats as stats import numpy as np mtp0 = [420, 435, 410, 445, 425, 430, 415, 440, 420, 435, 425, 410, 450, 420, 430] mtp5 = [380, 395, 370, 405, 385, 390, 375, 400, 380, 395, 385, 370, 410, 380, 390] t_stat, p_val = stats.ttest_rel(mtp5, mtp0) print(f"p-value: {p_val:.4f} | Significant: {p_val < 0.05}") print(f"Mean improvement: {np.mean(mtp0) - np.mean(mtp5):.1f}ms") ``` **4. Avoiding Common Comparison Traps** - **Trap 1:** Comparing cold vs warm cache. Always warm up KV cache before timing. - **Trap 2:** Ignoring draft verification failures. MTP’s speedup vanishes if draft accuracy <70%. Track `speculative_accept_rate` in logs. - **Trap 3:** Using high temperature. MTP fails at temp >0.7. Test at 0.1, 0.3, 0.5, 0.7 to find the breaking point. - **Trap 4:** Measuring only TTFT. MTP often increases TTFT (draft model initialization) but decreases TTNT. Track both. - **Trap 5:** Cherry-picking runs. Log every run. Use automated scripts to collect data. **5. MTP Configuration Matrix** Test these configurations systematically: | Config | --speculative | --draft-model | --ctx-size | Temp | Expected Use Case | |--------|---------------|---------------|------------|------|-------------------| | A | 0 | - | 4096 | 0.1 | Baseline, short Q&A | | B | 3 | auto | 4096 | 0.1 | Moderate speedup, coding | | C | 5 | auto | 4096 | 0.3 | Balanced, chat | | D | 8 | auto | 8192 | 0.5 | Long output, creative | | E | 5 | auto | 16384 | 0.1 | RAG, context-heavy | Run each config 15x. Record TTFT, TTNT, total latency, tokens/sec, VRAM, draft acceptance rate. **6. Decision Thresholds** Enable MTP when: - Draft acceptance rate >75% - TTNT improvement >15% - VRAM headroom >2GB - Memory bandwidth utilization <85% Disable MTP when: - Draft acceptance rate <60% - TTFT increases >20% - VRAM >90% capacity - Bandwidth saturation causes TTNT spikes --- # Reasoning Budget Plan Token budgeting isn’t just about cost; it’s about latency, accuracy, and system stability. In local setups, budget control directly impacts responsiveness and GPU memory management. Here’s how to measure, control, and optimize your reasoning budget. **1. Token Accounting Fundamentals** Track three metrics per request: - `prompt_tokens`: Input tokens (system + user + history) - `completion_tokens`: Output tokens (model response) - `total_tokens`: Sum of both Use `llama.cpp`’s `--log-level debug` or Flight Recorder to extract these. SQLite schema should include them. **2. Budget Control Strategies** - **Context Window Management:** Set `--ctx-size` to 1.2x your typical prompt length. Oversizing wastes VRAM and slows KV cache updates. - **Prompt Compression:** Use system prompt templates that are concise. Remove redundant instructions. Use XML tags for structure. - **Output Length Limits:** Set `--num-predict` to your target length. Don’t rely on model’s default (often 2048). - **Streaming Cancellation:** If user stops generation, cancel immediately. Saves tokens and GPU cycles. - **History Pruning:** In OpenWebUI, enable context truncation. Keep only last 3-5 messages for long chats. **3. Measuring Budget Efficiency** Calculate tokens/sec and latency/token: ```sql SELECT model, mtp_enabled, AVG(total_tokens) as avg_tokens, AVG(total_latency_ms) as avg_latency_ms, AVG(total_tokens) / (AVG(total_latency_ms) / 1000.0) as tokens_per_sec, AVG(total_latency_ms) / AVG(total_tokens) as ms_per_token FROM requests GROUP BY model, mtp_enabled; ``` High tokens/sec with low ms/token indicates efficient budget usage. Low tokens/sec with high ms/token suggests KV cache thrashing or proxy overhead. **4. Use-Case Budget Profiles** - **Coding:** High prompt tokens (code snippets), moderate output tokens. Budget: 2k-4k tokens. MTP helps if draft model understands syntax. - **RAG:** Very high prompt tokens (retrieved chunks), low output tokens. Budget: 4k-8k tokens. MTP often hurts (draft model lacks retrieved context). Disable MTP. - **Agentic:** Structured outputs, tool calls. Budget: 1k-3k tokens. MTP helps if tool schemas are deterministic. Use temp=0.1. - **Chat:** Variable length, conversational. Budget: 1k-2k tokens. MTP helps at temp=0.3-0.5. - **Creative Writing:** Long outputs, high temperature. Budget: 2k-6k tokens. MTP hurts at temp>0.7. Disable. **5. Budget Optimization Checklist** - [ ] Set `--ctx-size` to exact need + 20% - [ ] Use `--num-predict` to cap output - [ ] Enable streaming in OpenWebUI - [ ] Prune chat history weekly - [ ] Compress system prompts to <500 tokens - [ ] Use `--cache-type-k q8_0` for RAG, `q4_0` for chat - [ ] Monitor SQLite `total_tokens` daily, alert if >80% of `--ctx-size` **6. Advanced Budget Techniques** - **Prompt Routing:** Use a lightweight router model to classify intent. Route coding to MTP-enabled config, RAG to MTP-disabled config. - **Dynamic Context Window:** Adjust `--ctx-size` based on prompt length. Short prompts use 2048, long use 8192. - **Token Budget Alerts:** Query SQLite for requests exceeding budget: ```sql SELECT timestamp, model, total_tokens, total_latency_ms FROM requests WHERE total_tokens > 4096 ORDER BY total_tokens DESC LIMIT 10; ``` Use this to identify budget hogs and optimize prompts. --- # Long Output Test Plan Long outputs stress-test your setup: KV cache management, memory bandwidth, proxy buffering, and model stability. MTP behaves differently here. This plan isolates long-output behavior and provides actionable insights. **1. Long Output Stressors** - **KV Cache Fragmentation:** As context grows, VRAM allocation becomes fragmented. `llama.cpp` uses contiguous blocks. Fragmentation causes allocation failures or fallback to CPU. - **Memory Bandwidth Saturation:** Long outputs require continuous forward passes. If GPU bandwidth is maxed, TTNT spikes. - **Proxy Buffering:** Flight Recorder may buffer entire responses, masking streaming benefits. Non-streaming long outputs cause UI freezes. - **Model Drift:** Extended generation increases error accumulation. MTP verification failures compound. **2. Test Matrix for Long Outputs** | Test | Prompt Length | Output Target | --ctx-size | --num-predict | MTP | Streaming | |------|---------------|---------------|------------|---------------|-----|-----------| | L1 | 200 tokens | 500 tokens | 2048 | 500 | 0 | Yes | | L2 | 200 tokens | 500 tokens | 2048 | 500 | 5 | Yes | | L3 | 800 tokens | 1000 tokens | 4096 | 1000 | 0 | Yes | | L4 | 800 tokens | 1000 tokens | 4096 | 1000 | 5 | Yes | | L5 | 1500 tokens | 2000 tokens | 8192 | 2000 | 0 | Yes | | L6 | 1500 tokens | 2000 tokens | 8192 | 2000 | 5 | Yes | | L7 | 1500 tokens | 2000 tokens | 8192 | 2000 | 5 | No | Run each 10x. Record: completion rate, TTFT, TTNT, total latency, VRAM peak, bandwidth peak, proxy queue time, error rate. **3. KV Cache Management** - Use `--cache-type-k q8_0` for long contexts (higher accuracy, more VRAM) - Use `--cache-type-k q4_0` for short contexts (less VRAM, faster) - Enable `--mlock` to prevent swapping - Monitor VRAM growth: `nvtop` or `nvidia-smi` - If VRAM >90%, reduce `--ctx-size` or switch to `--numa` mode **4. Streaming vs Non-Streaming** - Streaming: Sends tokens as generated. Low memory pressure, better UX, proxy must support chunked transfer. - Non-streaming: Buffers entire response. High memory pressure, UI freezes, but easier to log. - Test both. Streaming should reduce perceived latency by 30-50% for long outputs. **5. Failure Mode Analysis** Common long-output failures: - **OOM:** VRAM full. Fix: Reduce `--ctx-size`, use `--numa`, switch to CPU offload. - **Timeout:** Proxy or UI timeout. Fix: Increase `FR_TIMEOUT`, `OPENWEBUI_TIMEOUT`. - **Truncation:** Model stops early. Fix: Increase `--num-predict`, check `--stop` tokens. - **Quality Degradation:** Repetition, incoherence. Fix: Lower temperature, use `--repeat-penalty`, enable MTP only if draft accuracy >80%. **6. Long Output Optimization** - Use `--batch-size` 512 for long outputs (better throughput) - Enable `--flash-attn` if supported - Use `--logit-bias` to prevent repetition - Chunk RAG queries to <1000 tokens - Use `--cache-type-v q8_0` for long contexts - Monitor `gpu_bw_util` in ServerTop. If >85%, disable MTP. --- # SQLite Queries Structured data is useless without the right queries. Below are production-ready SQL queries for your `benchmark.db`. Adjust table/column names if your schema differs. **1. Baseline Latency Distribution** ```sql SELECT model, mtp_enabled, COUNT(*) as runs, ROUND(AVG(total_latency_ms), 2) as avg_latency_ms, ROUND(MEDIAN(total_latency_ms), 2) as median_latency_ms, ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY total_latency_ms), 2) as p95_latency_ms, ROUND(PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY total_latency_ms), 2) as p99_latency_ms FROM requests GROUP BY model, mtp_enabled ORDER BY mtp_enabled, avg_latency_ms; ``` *Purpose:* Compare MTP vs non-MTP across percentiles. P95/P99 reveal tail latency issues. **2. Token Throughput Analysis** ```sql SELECT strftime('%Y-%m-%d', timestamp) as date, model, mtp_enabled, SUM(total_tokens) as total_tokens_day, SUM(total_latency_ms) / 1000.0 as total_seconds, ROUND(SUM(total_tokens) / (SUM(total_latency_ms) / 1000.0), 2) as tokens_per_sec FROM requests GROUP BY date, model, mtp_enabled ORDER BY date DESC, tokens_per_sec DESC; ``` *Purpose:* Track daily throughput. Spikes indicate proxy bottlenecks or GPU throttling. **3. VRAM Usage Correlation** ```sql SELECT timestamp, model, mtp_enabled, gpu_mem_mb, total_tokens, total_latency_ms, CASE WHEN gpu_mem_mb > 12000 THEN 'High' WHEN gpu_mem_mb > 8000 THEN 'Medium' ELSE 'Low' END as mem_pressure FROM requests WHERE timestamp > datetime('now', '-7 days') ORDER BY gpu_mem_mb DESC; ``` *Purpose:* Identify VRAM-heavy configurations. High pressure correlates with TTNT spikes. **4. MTP Draft Acceptance Rate (If Logged)** ```sql SELECT model, mtp_enabled, AVG(speculative_accept_rate) as avg_acceptance, COUNT(*) as runs FROM requests WHERE mtp_enabled = 1 GROUP BY model HAVING avg_acceptance < 0.75; ``` *Purpose:* Flag MTP configs with poor draft accuracy. Acceptance <75% means MTP is hurting. **5. Proxy Overhead Isolation** ```sql SELECT model, mtp_enabled, AVG(total_latency_ms) - AVG(ttft_ms) as inference_time_ms, AVG(ttft_ms) - AVG(proxy_queue_ms) as preprocessing_time_ms, AVG(proxy_queue_ms) as proxy_overhead_ms FROM requests GROUP BY model, mtp_enabled; ``` *Purpose:* Separate proxy, preprocessing, and inference times. High proxy overhead means Flight Recorder needs tuning. **6. Outlier Detection** ```sql WITH stats AS ( SELECT model, mtp_enabled, AVG(total_latency_ms) as mean_lat, STDDEV(total_latency_ms) as std_lat FROM requests GROUP BY model, mtp_enabled ) SELECT r.* FROM requests r JOIN stats s ON r.model = s.model AND r.mtp_enabled = s.mtp_enabled WHERE ABS(r.total_latency_ms - s.mean_lat) > 2 * s.std_lat ORDER BY ABS(r.total_latency_ms - s.mean_lat) DESC; ``` *Purpose:* Identify anomalous runs. Investigate causes (thermal throttling, proxy queue, OOM fallback). **7. Use-Case Performance Summary** ```sql SELECT model, mtp_enabled, CASE WHEN AVG(total_tokens) < 500 THEN 'Short Q&A' WHEN AVG(total_tokens) BETWEEN 500 AND 1500 THEN 'Chat/Coding' ELSE 'Long Output/RAG' END as use_case, ROUND(AVG(total_latency_ms), 2) as avg_latency, ROUND(AVG(total_tokens) / (AVG(total_latency_ms) / 1000.0), 2) as tps, ROUND(AVG(gpu_mem_mb), 0) as avg_vram_mb FROM requests GROUP BY model, mtp_enabled ORDER BY use_case, tps DESC; ``` *Purpose:* Match configurations to use cases. High TPS with low latency = optimal. **8. Daily Budget Consumption** ```sql SELECT strftime('%Y-%m-%d', timestamp) as date, SUM(total_tokens) as daily_tokens, COUNT(*) as daily_requests, ROUND(SUM(total_tokens) / COUNT(*), 2) as avg_tokens_per_request FROM requests GROUP BY date ORDER BY date DESC; ``` *Purpose:* Track token budget consumption. Alert if daily tokens >80% of `--ctx-size` * daily requests. --- # Dashboard Screenshots To Capture Visual data is critical for quick diagnosis and team communication. Capture these screenshots systematically. Annotate them with timestamps, configurations, and observations. **1. Flight Recorder Dashboard** - **Request Timeline:** Shows request start/end times. Look for queuing delays, proxy bottlenecks, and idle gaps. - **Latency Distribution:** Histogram of TTFT, TTNT, total latency. Note skewness. MTP should shift TTNT left. - **Error Rate:** 4xx/5xx responses. High error rate indicates OOM, timeout, or draft verification failures. - **Cache Hit Rate:** If using prompt caching, track hit rate. Low hit rate means context window mismatch. - **Screenshot Tips:** Annotate p95 latency, queue depth, error spikes. Include config in caption. **2. ServerTop Dashboard** - **GPU Memory Allocation:** Track VRAM usage over time. MTP causes initial spike, then stable usage. - **Compute Utilization:** Should be 80-95% during generation. <70% means bandwidth-bound or proxy-limited. - **Memory Bandwidth:** Critical for MTP. >85% utilization means MTP is throttled. - **Temperature & Power:** Track thermal throttling. >85°C GPU temp causes frequency drops. - **Screenshot Tips:** Highlight bandwidth saturation, thermal throttling, VRAM fragmentation. Include `nvidia-smi` snapshot. **3. OpenWebUI Dashboard** - **Chat History:** Check token counters, streaming behavior, truncation points. - **Model Settings:** Verify temperature, top_p, max tokens, MTP toggle. - **Performance Metrics:** If available, track TTFT, TTNT, tokens/sec per chat. - **Screenshot Tips:** Annotate streaming delays, token counters, truncation warnings. Include system prompt length. **4. SQLite Query Results** - **Latency Percentiles:** P50, P95, P99 for MTP vs non-MTP. - **Token Throughput:** Tokens/sec over time. - **VRAM Correlation:** Scatter plot of VRAM vs latency. - **Draft Acceptance Rate:** MTP success metric. - **Screenshot Tips:** Include query, data, interpretation. Use consistent color coding. **5. Annotation Template** ``` Date: YYYY-MM-DD Config: --ctx-size=4096 --speculative=5 --temp=0.3 Observation: TTFT increased by 120ms, TTNT decreased by 80ms. VRAM stable at 11.2GB. Bandwidth at 78%. Conclusion: MTP beneficial for this config. Enable for chat/coding. ``` Store screenshots in `./benchmarks/screenshots/` with naming convention: `YYYYMMDD_config_observation.png`. --- # Weekend Checklist A structured weekend prevents fatigue and ensures thorough testing. Follow this timeline. **Friday Evening (Setup & Baseline)** - [ ] Install/configure Flight Recorder, ServerTop, SQLite parser - [ ] Set GPU power limit, performance mode, CPU frequency - [ ] Verify `llama.cpp` flags: `--gpu-layers 99 --ctx-size 4096 --batch-size 512` - [ ] Run 5 baseline requests (MTP=0), collect ServerTop CSV, Flight Recorder JSON, SQLite insert - [ ] Verify proxy overhead <50ms, TTFT variance <2s - [ ] Create `benchmark.db`, run schema creation script - [ ] Capture baseline dashboard screenshots **Saturday Morning (MTP Testing)** - [ ] Warm up KV cache (3 identical prompts) - [ ] Run MTP=3, MTP=5, MTP=8 configs (15 runs each) - [ ] Record ServerTop, Flight Recorder, SQLite data - [ ] Monitor VRAM, bandwidth, temperature - [ ] Adjust `--ctx-size` if VRAM >90% - [ ] Capture MTP dashboard screenshots - [ ] Run statistical validation (paired t-test, effect size) **Saturday Afternoon (Long Output & Budget)** - [ ] Test long outputs (L1-L7 matrix) - [ ] Compare streaming vs non-streaming - [ ] Track token budget consumption - [ ] Identify KV cache fragmentation issues - [ ] Optimize `--cache-type-k`, `--num-predict` - [ ] Capture long output dashboard screenshots - [ ] Run SQLite queries for throughput, latency, VRAM correlation **Sunday Morning (Analysis & Interpretation)** - [ ] Aggregate all data into SQLite - [ ] Run diagnostic queries (percentiles, throughput, outliers) - [ ] Interpret results: MTP benefit, budget efficiency, use-case matching - [ ] Document findings, recommendations - [ ] Create final dashboard screenshots with annotations - [ ] Backup benchmark data **Sunday Afternoon (Validation & Deployment)** - [ ] Re-run top 3 configs to verify consistency - [ ] Test with real user prompts (coding, RAG, chat, creative) - [ ] Adjust OpenWebUI settings based on findings - [ ] Update `llama.cpp` startup script with optimal flags - [ ] Set up automated monitoring (ServerTop CSV, SQLite logging) - [ ] Archive benchmark data, write summary report **Verification Steps** - [ ] All runs logged in SQLite - [ ] No cold/warm cache mixing - [ ] Proxy overhead isolated - [ ] Statistical significance confirmed - [ ] Use-case matching validated - [ ] Dashboard screenshots annotated - [ ] Final config documented --- # Interpreting Results Data without interpretation is noise. Here’s how to read your benchmark results, match configurations to use cases, and make deployment decisions. **1. Latency Metrics Decoded** - **TTFT (Time To First Token):** Dominated by prompt preprocessing, proxy overhead, KV cache population. Lower is better for chat. MTP often increases TTFT slightly (draft model init). - **TTNT (Time To Next Token):** Dominated by inference speed, memory bandwidth, MTP verification. Lower is better for long outputs. MTP should decrease TTNT. - **Total Latency:** Sum of TTFT + TTNT * tokens. Use for overall UX. - **Tokens/Sec:** Throughput metric. High TPS = efficient GPU usage. **2. MTP Benefit Assessment** - **Good MTP:** TTNT ↓15-30%, TPS ↑10-20%, VRAM ↑10-15%, draft acceptance >75% - **Bad MTP:** TTNT ↓<5%, TPS ↓, VRAM ↑>20%, draft acceptance <60%, bandwidth >85% - **Neutral MTP:** TTNT ↓5-10%, TPS stable, VRAM ↑5-10%, draft acceptance 60-75% - **Decision:** Enable MTP if TTNT improvement >15% and draft acceptance >75%. Disable otherwise. **3. Use-Case Matching** - **Coding:** Needs accuracy, moderate latency. MTP helps at temp=0.1-0.3. Use `--ctx-size=4096`, `--cache-type-k q8_0`. - **RAG:** Needs context handling, low latency. MTP often hurts (draft model lacks retrieved context). Disable MTP. Use `--ctx-size=8192`, `--cache-type-k q8_0`. - **Agentic:** Needs structured outputs, tool calls. MTP helps if deterministic. Use temp=0.1, `--stop` tokens for JSON. - **Chat:** Needs low TTFT, responsive UX. MTP helps at temp=0.3-0.5. Use `--ctx-size=2048`, streaming enabled. - **Creative Writing:** Needs long outputs, high temperature. MTP hurts at temp>0.7. Disable MTP. Use `--ctx-size=4096`, `--repeat-penalty=1.1`. **4. VRAM & Bandwidth Interpretation** - **VRAM <80%:** Headroom available. MTP safe. - **VRAM 80-90%:** Tight. MTP may cause fragmentation. Monitor KV cache. - **VRAM >90%:** Critical. Disable MTP, reduce `--ctx-size`, enable `--numa`. - **Bandwidth <70%:** Compute-bound. MTP beneficial. - **Bandwidth 70-85%:** Balanced. MTP marginal benefit. - **Bandwidth >85%:** Bandwidth-bound. MTP harmful. Disable. **5. Statistical Significance** - **p-value <0.05:** Statistically significant difference. Trust the result. - **p-value >0.05:** No significant difference. MTP not helping. - **Effect size (Cohen’s d):** - d <0.2: Negligible - d 0.2-0.5: Small - d 0.5-0.8: Medium - d >0.8: Large - **Confidence Interval:** If 95% CI for TTNT improvement doesn’t include 0, MTP is beneficial. **6. Deployment Recommendations** - **Default Config:** `--ctx-size=4096 --speculative=5 --temp=0.3 --cache-type-k q8_0 --batch-size 512` - **RAG Config:** `--ctx-size=8192 --speculative=0 --temp=0.1 --cache-type-k q8_0 --batch-size 256` - **Creative Config:** `--ctx-size=4096 --speculative=0 --temp=0.7 --cache-type-k q4_0 --batch-size 512` - **Agentic Config:** `--ctx-size=2048 --speculative=5 --temp=0.1 --cache-type-k q8_0 --batch-size 512 --stop "```"` --- # Common Mistakes Even experienced users fall into benchmarking traps. Avoid these to ensure accurate, actionable results. **1. Cold vs Warm Cache Confusion** - **Mistake:** Comparing cold-start TTFT with warm-start TTNT. - **Fix:** Always warm up KV cache before timing. Run 3 identical prompts before benchmarking. **2. Ignoring Draft Verification Failures** - **Mistake:** Assuming MTP always speeds up generation. - **Fix:** Track `speculative_accept_rate`. If <75%, MTP is hurting. Disable. **3. Proxy Overhead Blindness** - **Mistake:** Attributing proxy queue time to model latency. - **Fix:** Isolate proxy metrics. Use direct `curl` to `llama-server` for baseline. Subtract proxy overhead. **4. Temperature Mismatch** - **Mistake:** Testing MTP at temp=0.1, then comparing to non-MTP at temp=0.7. - **Fix:** Keep temperature identical across MTP/non-MTP runs. Test temp=0.1, 0.3, 0.5, 0.7 separately. **5. Context Window Oversizing** - **Mistake:** Setting `--ctx-size=16384` for 500-token prompts. - **Fix:** Match `--ctx-size` to typical prompt length + 20%. Oversizing wastes VRAM, slows KV cache updates. **6. Cherry-Picking Runs** - **Mistake:** Reporting only the fastest MTP run, ignoring throttled runs. - **Fix:** Log every run. Use p95/p99 latency. Report median and percentiles. **7. Ignoring Memory Bandwidth** - **Mistake:** Focusing only on VRAM usage, ignoring bandwidth saturation. - **Fix:** Monitor `gpu_bw_util`. If >85%, MTP is throttled. Disable. **8. Thermal Throttling Blindness** - **Mistake:** Not checking GPU temperature. Throttling causes inconsistent latency. - **Fix:** Monitor `temp_gpu`. If >85°C, reduce power limit, improve cooling, or lower batch size. **9. Streaming vs Non-Streaming Confusion** - **Mistake:** Comparing streaming TTFT with non-streaming total latency. - **Fix:** Test both modes separately. Streaming reduces perceived latency, non-streaming measures true inference time. **10. Statistical Illiteracy** - **Mistake:** Using averages, ignoring variance, p-values, effect sizes. - **Fix:** Use paired tests, percentiles, confidence intervals. Report statistical significance. **11. Model Quantization Mismatch** - **Mistake:** Comparing Q4_K_M MTP vs Q5_K_S non-MTP. - **Fix:** Use identical model, quantization, and version across all tests. **12. OpenWebUI Configuration Drift** - **Mistake:** Changing OpenWebUI settings mid-test (temperature, max tokens, streaming). - **Fix:** Lock UI settings. Use direct API for controlled testing. Log all UI parameters. **13. Ignoring KV Cache Eviction** - **Mistake:** Not tracking context truncation. Model drops old messages, affecting accuracy. - **Fix:** Monitor `ctx_size` vs `prompt_tokens`. If `prompt_tokens > ctx_size`, increase `--ctx-size` or prune history. **14. Over-Reliance on Single Metric** - **Mistake:** Judging MTP solely by TTFT or TTNT. - **Fix:** Use composite metrics: TTFT, TTNT, TPS, VRAM, bandwidth, draft acceptance, p95 latency. **15. Not Validating with Real Prompts** - **Mistake:** Benchmarking only with synthetic prompts. - **Fix:** Test with real user prompts (coding, RAG, chat, creative). Synthetic prompts don’t capture distribution variance. By avoiding these mistakes, your weekend benchmark will yield reliable, actionable insights. You’ll know exactly when to enable MTP, how to configure your server, and which use cases benefit from which settings. Happy benchmarking.