# Quick Diagnosis Before diving into data collection and comparative testing, it is essential to understand why your `llama.cpp` server feels inconsistent. The symptoms you described—slow first answer, GPU memory jumps, and uncertainty around MTP (Multi-Token Prediction) benefits—are classic indicators of how modern LLM inference engines interact with hardware, memory management, and speculative decoding. **1. The "First Answer is Slow" Phenomenon** The first request to a freshly loaded `llama.cpp` server always incurs a higher latency than subsequent requests. This is not a bug; it is the prefill phase. During prefill, the model processes the entire prompt in parallel, building the KV (Key-Value) cache. This phase is compute-bound and heavily dependent on: - **Prompt length:** Longer prompts linearly increase prefill time. - **KV cache allocation:** The first request triggers dynamic memory allocation for the KV cache. If your system uses `--mmap` or relies on OS-level paging, the first allocation can stall while pages are faulted in. - **MTP overhead:** If MTP is enabled, the draft model must also be loaded and warmed up. The first speculative cycle often fails to stabilize until the draft and verify steps reach thermal and memory bandwidth equilibrium. **2. GPU Memory Jumps** GPU memory does not scale linearly with token count in `llama.cpp`. It jumps due to: - **KV cache chunking:** `llama.cpp` allocates KV cache in batches. When a request exceeds the current batch size or context window, the engine may reallocate or expand the cache, causing a visible spike in `nvidia-smi`. - **MTP working memory:** Multi-Token Prediction requires storing draft tokens, verification states, and parallel KV slices. This can temporarily double or triple the memory footprint during the speculative phase. - **Python/OpenWebUI overhead:** OpenWebUI runs on FastAPI/Python. Garbage collection cycles, WebSocket buffer allocations, and proxy logging (Flight Recorder) can cause host RAM spikes that indirectly trigger GPU memory pressure via PCIe bandwidth contention or unified memory architectures (e.g., Apple Silicon, Intel Arc). **3. MTP Uncertainty** MTP (speculative decoding) accelerates generation by proposing multiple tokens ahead and verifying them in parallel. However, it introduces variability: - **Acceptance rate volatility:** If the draft model diverges from the target model, acceptance rates drop, and MTP becomes slower than autoregressive generation due to verification overhead. - **Batch size mismatch:** MTP works best when `--batch-size` and `--n-predict` align with your hardware's tensor core utilization. Misalignment causes thread starvation or idle SMs. - **Context window effects:** MTP overhead scales with context length. In short prompts, MTP may not pay off. In long contexts, KV cache thrashing can negate speedups. **Immediate Checks to Run** - Verify `llama.cpp` version: `llama-server --version`. MTP support and KV cache optimizations vary significantly across releases. - Check GPU driver and CUDA/ROCm version: `nvidia-smi` or `rocm-smi`. Outdated drivers cause memory fragmentation and poor tensor core utilization. - Monitor system load: `htop` and `nvtop`. Ensure no background processes are stealing PCIe bandwidth or CPU cycles. - Test with a fixed prompt: Use a 512-token prompt and measure TTFB and TPS across 10 consecutive requests. If the first request is >3x slower than the rest, prefill/KV allocation is the culprit. --- # Data Collection Plan Consistent benchmarking requires controlled data collection. You cannot rely on anecdotal OpenWebUI timing. You need structured logs, repeatable requests, and isolated metrics. **1. Flight Recorder Proxy Setup** Flight Recorder (or similar reverse proxy) captures HTTP requests, headers, and timing. Configure it to log: - `request_id`, `timestamp`, `prompt_tokens`, `generated_tokens`, `total_latency_ms`, `ttfb_ms`, `tps`, `model`, `mtp_enabled`, `context_size`, `batch_size`. - Enable JSON logging to a file: `flight-recorder --log-format json --output /var/log/llama/requests.json` - Ensure the proxy strips OpenWebUI WebSocket overhead by measuring raw `/v1/chat/completions` responses. **2. ServerTop Monitoring** ServerTop provides real-time GPU and system metrics. Configure it to sample at 100ms intervals: - `servertop --interval 0.1 --output /var/log/llama/gpu.csv` - Track: `gpu_mem_used`, `gpu_util`, `sm_util`, `mem_bw`, `kv_cache_used`, `temp`, `power`. - Correlate memory jumps with request timestamps to identify allocation patterns. **3. SQLite Reporting** SQLite is ideal for local, zero-overhead logging. Create a schema that captures both request-level and system-level metrics: ```sql CREATE TABLE requests ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, model TEXT, prompt_tokens INTEGER, generated_tokens INTEGER, total_latency_ms REAL, ttfb_ms REAL, tps REAL, mtp_enabled BOOLEAN, context_size INTEGER, batch_size INTEGER, acceptance_rate REAL, draft_model TEXT, status TEXT ); CREATE TABLE system_metrics ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, gpu_mem_used_mb REAL, gpu_util_percent REAL, cpu_load_avg REAL, ram_used_mb REAL, kv_cache_mb REAL ); ``` Enable persistent logging in your proxy or wrapper script. If using `llama.cpp` directly, pipe logs to a Python script that inserts into SQLite. **4. Controlled Request Generation** Use `curl` or `ab`/`wrk` to generate repeatable requests. Avoid OpenWebUI's UI for benchmarking due to WebSocket buffering and UI thread interference. ```bash # Single request test curl -s -w "\n%{time_total}\n%{time_starttransfer}\n" \ -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ -d '{ "model": "your-model", "messages": [{"role": "user", "content": "Explain quantum entanglement in 300 words."}], "max_tokens": 512, "temperature": 0.7, "stream": false }' # Load test with 10 concurrent requests ab -n 100 -c 10 -p payload.json -T application/json http://localhost:8080/v1/chat/completions ``` - Warm up the server with 5 requests before measuring. - Run each configuration 30+ times to smooth out thermal throttling and OS jitter. - Record prompt length, context window, and generation length for every run. **5. Data Validation** - Cross-check `ttfb` from proxy with `time_starttransfer` from `curl`. - Verify `tps` calculation: `tps = generated_tokens / (total_latency_ms - ttfb_ms) * 1000`. - Ensure GPU memory readings align with request boundaries. Spikes should correlate with KV cache expansion, not random noise. --- # MTP Comparison Plan MTP (Multi-Token Prediction) is speculative decoding. It uses a smaller draft model to propose tokens, then verifies them with the target model. If you want to know whether MTP is helping, you must measure it rigorously. **1. MTP Mechanics in llama.cpp** - `--mtp`: Enables multi-token prediction. - `--draft`: Path to the draft model (usually a quantized version of the target). - `--mtp-n`: Number of draft tokens per step (default 4-8). - `--mtp-max`: Maximum draft tokens per request. - MTP works by running the draft model autoregressively, then verifying all draft tokens in parallel with the target model. Accepted tokens are committed; rejected tokens trigger fallback to standard generation. **2. Experimental Design** - **Baseline:** Run without MTP (`--mtp 0`). Record TTFB, TPS, p95 latency, GPU mem, and output quality. - **MTP Enabled:** Run with `--mtp 1 --draft draft-model.gguf --mtp-n 4`. Record same metrics. - **Control Variables:** Same prompt, same temperature, same top_p, same context window, same batch size, same GPU state (cold boot each session). - **Sample Size:** Minimum 50 requests per configuration. Use different prompts from your actual workload (coding, RAG, chat, etc.). **3. Metrics to Track** - **Acceptance Rate:** `accepted_draft_tokens / total_draft_tokens`. >0.7 is excellent, 0.5-0.7 is acceptable, <0.4 means MTP is hurting performance. - **Speedup Factor:** `tps_non_mtp / tps_mtp`. >1.1 means MTP is helping. <1.0 means it is slowing down generation. - **TTFB Impact:** MTP often increases TTFB because the draft model must run first. Measure if the delay is acceptable for your use case. - **Memory Overhead:** Track peak GPU memory. MTP can increase KV cache usage by 30-50%. - **Quality Consistency:** Run the same prompt 10 times with and without MTP. Compare output coherence, factual accuracy, and formatting. **4. Avoiding Self-Deception** - **Don't measure only mean TPS:** Distribution matters. MTP may improve median TPS but increase p99 latency due to verification stalls. - **Don't ignore warm-up:** The first 5 requests per configuration are unreliable. Discard them. - **Don't change multiple variables:** If you change `--batch-size` and `--mtp` simultaneously, you cannot attribute changes to MTP. - **Use statistical significance:** Calculate mean, standard deviation, and 95% confidence intervals. If confidence intervals overlap, the difference is not statistically significant. - **Validate output quality:** Speed means nothing if MTP causes hallucinations or formatting breaks. Use a simple consistency check: hash outputs or run a lightweight LLM-as-a-judge prompt. **5. When MTP is Worth It** - Long context (>4k tokens) where KV cache verification is cheaper than autoregressive steps. - Draft model is highly similar to target model (same architecture, similar training data). - Hardware has high memory bandwidth (e.g., RTX 4090, A100, Apple M3/M4) to parallelize verification. - Use case tolerates slightly higher TTFB for faster generation (e.g., creative writing, code completion). **6. When to Disable MTP** - Short prompts (<1k tokens) where prefill dominates. - Low acceptance rate (<0.5) indicating draft model mismatch. - Agentic workflows requiring strict tool-calling accuracy (MTP can introduce token-level drift). - Memory-constrained setups where KV cache expansion causes OOM. --- # Reasoning Budget Plan "Reasoning budget" refers to how your hardware and `llama.cpp` configuration allocate compute, memory, and context window to maximize useful generation. Mismanaging budget leads to thrashing, slowdowns, and wasted resources. **1. KV Cache Management** - `--ctx-size`: Maximum context window. Larger values increase memory usage linearly. Set to your actual max prompt + generation length. - `--batch-size`: Number of tokens processed per forward pass. Larger batches improve throughput but increase memory. Match to your GPU's tensor core size (e.g., 32, 64, 128). - `--numa`: Enable NUMA awareness on multi-socket systems. Prevents cross-socket memory latency. - `--cache-type-k` / `--cache-type-v`: Use `f16` or `bf16` for accuracy, `q4_0` or `q8_0` for memory savings. MTP benefits from higher precision KV cache. **2. Compute Allocation** - `--tensor-split`: Distribute layers across multiple GPUs. Use `--tensor-split 0.5,0.5` for equal split, or profile layer weights to optimize. - `--threads` / `--threads-batch`: CPU threads for prefill and batch processing. Set to physical cores, not logical. - `--gpu-layers`: Offload layers to GPU. Leave 1-2 layers on CPU for dynamic batching if VRAM is tight. **3. Context Window Optimization** - **Sliding Window:** `--rope-scaling factor=2.0 --rope-scaling type=linear` extends context without linear memory growth. - **Context Pruning:** For RAG, truncate old context before sending to model. Use a summarization step or retrieval window. - **Chunking:** Split long inputs into chunks, process sequentially, and concatenate outputs. Avoid sending 32k tokens when 8k suffices. **4. Budget Monitoring** - Track `kv_cache_used` vs `ctx-size`. If >80%, you are approaching thrashing. - Monitor `tps` decay as context grows. A 20% drop at 8k vs 4k is normal; a 50% drop indicates inefficient KV cache or attention implementation. - Use `--log-level debug` to see KV cache allocation steps. Look for repeated reallocations. **5. Practical Budget Tuning** - Start with `--ctx-size 4096 --batch-size 512 --threads 8`. Measure baseline. - Increase `--ctx-size` in 1k increments. Note TPS drop and memory usage. - Adjust `--batch-size` to match GPU memory. If OOM, reduce by 25%. If underutilized, increase. - Enable `--numa` if on multi-CPU system. Disable if on single-socket. - Test `--cache-type-k q8_0` if memory is tight. Verify accuracy drop is acceptable. --- # Long Output Test Plan Long outputs stress-test streaming, memory stability, and MTP consistency. Many servers degrade gracefully at first, then crash or throttle after 4k-8k tokens. **1. Test Methodology** - Generate 2k, 4k, 8k, 16k tokens sequentially. - Use `stream: true` to measure streaming TPS and TTFB per chunk. - Monitor GPU memory, CPU load, and network throughput in real-time. - Record any slowdowns, OOM errors, or MTP acceptance rate drops. **2. curl Streaming Test** ```bash curl -N -s -w "\n%{time_total}\n" \ -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ -d '{ "model": "your-model", "messages": [{"role": "user", "content": "Write a 5000-word technical guide on distributed systems."}], "max_tokens": 5000, "temperature": 0.7, "stream": true }' | tee stream_output.json ``` - Parse JSONL output to measure chunk latency and TPS over time. - Look for TPS decay curve. A flat curve indicates stable generation. A downward slope indicates KV cache thrashing or MTP instability. **3. What to Look For** - **TPS Decay:** Measure TPS at 1k, 2k, 4k, 8k tokens. >10% drop is acceptable. >30% drop indicates budget mismanagement. - **Memory Plateau vs Spike:** Stable memory indicates efficient KV cache. Spikes indicate reallocation or MTP overhead. - **MTP Acceptance Rate:** Should remain stable. Drops indicate draft model divergence or context window limits. - **Streaming Latency:** TTFB should be consistent. Increasing TTFB indicates server backpressure or proxy buffering. **4. Optimization for Long Outputs** - `--chunk-size 512`: Process tokens in chunks to reduce memory spikes. - `--no-mmap`: Disable memory mapping for faster allocation. Increases RAM usage but reduces first-request latency. - `--numa`: Enable on multi-CPU systems to reduce cross-socket latency. - `--cache-type-k f16`: Use full precision for long contexts to avoid quantization drift. - `--mtp 0` for very long outputs: MTP overhead compounds over thousands of tokens. Disable if acceptance rate drops. **5. Validation** - Compare long output quality with short output. Check for repetition, coherence, and formatting. - Run the same prompt 3 times. Measure variance in TPS and memory. High variance indicates instability. - Use `tcpdump` or `wireshark` to verify network throughput matches generation speed. Mismatches indicate proxy or client buffering. --- # SQLite Queries SQLite is your ground truth. Use these queries to extract actionable insights from your logs. **1. Schema Recap** ```sql CREATE TABLE requests ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, model TEXT, prompt_tokens INTEGER, generated_tokens INTEGER, total_latency_ms REAL, ttfb_ms REAL, tps REAL, mtp_enabled BOOLEAN, context_size INTEGER, batch_size INTEGER, acceptance_rate REAL, draft_model TEXT, status TEXT ); CREATE TABLE system_metrics ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, gpu_mem_used_mb REAL, gpu_util_percent REAL, cpu_load_avg REAL, ram_used_mb REAL, kv_cache_mb REAL ); ``` **2. MTP vs Non-MTP Comparison** ```sql SELECT mtp_enabled, COUNT(*) as request_count, ROUND(AVG(tps), 2) as avg_tps, ROUND(MIN(tps), 2) as min_tps, ROUND(MAX(tps), 2) as max_tps, ROUND(AVG(total_latency_ms), 2) as avg_latency_ms, ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY total_latency_ms), 2) as p95_latency_ms, ROUND(AVG(acceptance_rate), 3) as avg_acceptance_rate FROM requests WHERE mtp_enabled IS NOT NULL GROUP BY mtp_enabled ORDER BY mtp_enabled; ``` **3. Context Length Impact** ```sql SELECT context_size, COUNT(*) as request_count, ROUND(AVG(tps), 2) as avg_tps, ROUND(AVG(gpu_mem_used_mb), 2) as avg_gpu_mem_mb, ROUND(AVG(kv_cache_mb), 2) as avg_kv_cache_mb FROM requests r JOIN system_metrics s ON r.timestamp = s.timestamp GROUP BY context_size ORDER BY context_size; ``` **4. Anomaly Detection** ```sql SELECT id, timestamp, tps, total_latency_ms, mtp_enabled, context_size FROM requests WHERE tps < (SELECT AVG(tps) - 2 * STDDEV(tps) FROM requests) ORDER BY tps ASC LIMIT 20; ``` **5. MTP Acceptance Rate vs TPS Correlation** ```sql SELECT ROUND(acceptance_rate, 2) as acceptance_bucket, COUNT(*) as request_count, ROUND(AVG(tps), 2) as avg_tps, ROUND(AVG(total_latency_ms), 2) as avg_latency_ms FROM requests WHERE mtp_enabled = 1 AND acceptance_rate IS NOT NULL GROUP BY acceptance_bucket ORDER BY acceptance_bucket; ``` **6. Export for Analysis** ```bash sqlite3 llama_benchmark.db "SELECT * FROM requests WHERE mtp_enabled = 1 ORDER BY timestamp;" > mtp_requests.csv sqlite3 llama_benchmark.db "SELECT * FROM system_metrics ORDER BY timestamp;" > system_metrics.csv ``` Use Python/pandas or Excel to plot TPS over time, memory vs context, and acceptance rate vs speedup. --- # Dashboard Screenshots To Capture Visual documentation is critical for debugging and sharing results. Capture these views during your tests. **1. ServerTop GPU View** - **Metrics:** `gpu_mem_used`, `gpu_util`, `sm_util`, `mem_bw`, `temp`, `power`. - **What to capture:** Steady-state utilization, memory spikes during KV cache expansion, thermal throttling indicators. - **Annotation:** Mark request start/end times. Highlight memory jumps and correlate with `--batch-size` or MTP steps. **2. Flight Recorder Latency Waterfall** - **Metrics:** `ttfb_ms`, `total_latency_ms`, `tps`, `prompt_tokens`, `generated_tokens`. - **What to capture:** Distribution of TTFB across requests. Identify outliers. Show TPS consistency. - **Annotation:** Label warm-up vs steady state. Mark MTP vs non-MTP runs. Highlight p95/p99 latency. **3. SQLite Dashboard (or CSV Plot)** - **Metrics:** TPS over time, memory vs context, acceptance rate vs speedup. - **What to capture:** Trend lines, correlation plots, distribution histograms. - **Annotation:** Add threshold lines (e.g., 10% TPS drop, 0.5 acceptance rate). Note configuration changes. **4. OpenWebUI Chat History** - **Metrics:** Output coherence, formatting, tool-calling accuracy. - **What to capture:** Side-by-side comparison of MTP vs non-MTP outputs. Highlight differences in reasoning, code syntax, or RAG citations. - **Annotation:** Mark hallucinations, repetition, or formatting breaks. Note if MTP improved or degraded quality. **5. System Monitor (htop/nvtop)** - **Metrics:** CPU load, RAM usage, PCIe bandwidth, swap activity. - **What to capture:** Background process interference, memory fragmentation, swap usage. - **Annotation:** Note if CPU threads are saturated. Highlight swap activity (indicates RAM pressure). **Dashboard Metric Definitions** - **TTFB (Time To First Byte):** Latency from request send to first token received. Critical for chat UX. - **TPS (Tokens Per Second):** Generation speed. `generated_tokens / (total_latency - ttfb)`. - **Acceptance Rate:** Fraction of draft tokens accepted by target model. >0.7 ideal. - **KV Cache Utilization:** `kv_cache_used / ctx_size`. >0.8 indicates thrashing risk. - **GPU SM Utilization:** Tensor core activity. <70% indicates batch size or memory bottleneck. - **Memory Bandwidth:** PCIe/VRAM transfer rate. High bandwidth needed for MTP verification. --- # Weekend Checklist A structured weekend ensures you collect clean data without burnout. **Friday Evening: Setup & Baseline** - [ ] Update `llama.cpp` to latest stable release. - [ ] Verify GPU drivers and CUDA/ROCm versions. - [ ] Configure Flight Recorder proxy with JSON logging. - [ ] Set up SQLite schema and logging script. - [ ] Prepare prompt library (coding, RAG, chat, creative, agentic). - [ ] Run 5 warm-up requests. Verify server stability. - [ ] Capture baseline ServerTop and Flight Recorder screenshots. **Saturday Morning: Non-MTP Baseline** - [ ] Disable MTP (`--mtp 0`). - [ ] Run 30 requests per prompt type. Record TTFB, TPS, p95 latency, GPU mem. - [ ] Test short (1k), medium (4k), long (8k) contexts. - [ ] Monitor KV cache allocation and memory spikes. - [ ] Export logs to SQLite. Run baseline queries. - [ ] Capture dashboard screenshots. **Saturday Afternoon: MTP Comparison** - [ ] Enable MTP (`--mtp 1 --draft draft-model.gguf --mtp-n 4`). - [ ] Run 30 requests per prompt type. Same conditions as baseline. - [ ] Record acceptance rate, speedup factor, TTFB impact, memory overhead. - [ ] Test different `--mtp-n` values (2, 4, 8) if supported. - [ ] Export logs. Run MTP comparison queries. - [ ] Capture dashboard screenshots. **Sunday Morning: Long Output & Budget Test** - [ ] Generate 2k, 4k, 8k, 16k tokens. Measure TPS decay and memory stability. - [ ] Test streaming vs non-streaming. Compare TTFB and chunk latency. - [ ] Adjust `--batch-size`, `--ctx-size`, `--cache-type-k`. Measure impact. - [ ] Enable `--numa`, `--no-mmap`. Test performance difference. - [ ] Export logs. Run budget optimization queries. - [ ] Capture dashboard screenshots. **Sunday Afternoon: Analysis & Decision** - [ ] Compile all data. Calculate means, medians, p95, confidence intervals. - [ ] Compare MTP vs non-MTP. Determine if speedup justifies TTFB increase. - [ ] Evaluate long output stability. Identify thrashing points. - [ ] Test output quality side-by-side. Check for hallucinations or formatting breaks. - [ ] Decide on final configuration: MTP on/off, batch size, context window, cache type. - [ ] Document findings. Update OpenWebUI settings. Share results with lab team. --- # Interpreting Results Data is only useful if interpreted correctly. Avoid common analytical traps. **1. Statistical vs Practical Significance** - A 2% TPS increase is statistically significant if p<0.05, but practically irrelevant if TTFB increases by 50ms. - Use confidence intervals. If MTP 95% CI overlaps non-MTP 95% CI, the difference is not actionable. - Prioritize p95/p99 latency over mean. Users care about worst-case, not average. **2. MTP Decision Matrix** - **Acceptance Rate >0.7 + Speedup >1.1 + TTFB increase <20%:** Enable MTP. - **Acceptance Rate 0.5-0.7 + Speedup 1.0-1.1 + TTFB increase 20-40%:** Enable for long context, disable for chat. - **Acceptance Rate <0.5 + Speedup <1.0:** Disable MTP. It is hurting performance. - **Memory increase >50% + OOM risk:** Disable MTP or reduce `--ctx-size`. **3. Use-Case Specific Guidance** - **Coding:** Low latency, high accuracy. MTP acceptable if acceptance rate >0.6. Prefer `--cache-type-k f16` for syntax precision. Disable MTP for strict tool-calling. - **RAG:** Context-heavy, retrieval-dependent. Optimize KV cache with `--rope-scaling`. MTP helpful if draft model matches RAG context style. Monitor context thrashing. - **Agentic:** Streaming, tool-calling, stateful. Prioritize stable TPS and low TTFB. Disable MTP if it causes token drift in tool parameters. Use `--chunk-size 256` for smooth streaming. - **Chat:** Fast TTFB, conversational flow. MTP acceptable if TTFB increase <30%. Prefer `--batch-size 512` for responsiveness. Monitor p95 latency. - **Creative Writing:** Long context, coherence, formatting. MTP helpful if acceptance rate >0.6. Use `--ctx-size 8192` or higher. Monitor repetition and drift. Disable MTP if coherence drops. **4. Quality Validation** - Run same prompt 10 times with and without MTP. - Use automated checks: syntax validation for code, JSON parsing for tools, citation format for RAG. - Use human eval: rate coherence, accuracy, formatting on 1-5 scale. - If MTP improves speed but degrades quality, disable it. Speed without accuracy is useless. **5. Hardware Utilization Insights** - **GPU SM <70%:** Batch size too small or memory bottleneck. Increase `--batch-size`. - **Memory Bandwidth saturated:** MTP verification bottleneck. Reduce `--mtp-n` or disable MTP. - **CPU threads saturated:** Prefill bottleneck. Increase `--threads` or offload more layers with `--gpu-layers`. - **Swap activity:** RAM pressure. Reduce `--ctx-size` or increase system RAM. --- # Common Mistakes Avoid these pitfalls to ensure your benchmark is valid and actionable. **1. Benchmarking Pitfalls** - **Not warming up the server:** First requests include KV cache allocation and thermal ramp-up. Discard first 5 requests per configuration. - **Ignoring KV cache state:** Running tests without clearing KV cache between runs contaminates results. Restart server or use `--cache-clear` if available. - **Changing multiple variables:** Modifying `--batch-size`, `--ctx-size`, and `--mtp` simultaneously makes attribution impossible. Change one variable at a time. - **Measuring only mean TPS:** Distribution matters. MTP may improve median but increase p99 latency. Report mean, median, p95, p99. - **Using synthetic prompts:** Random text does not match real workload. Use actual prompts from coding, RAG, chat, agentic, creative use cases. - **Ignoring system load:** Background processes steal PCIe bandwidth and CPU cycles. Run tests on idle system. Monitor with `htop` and `nvtop`. **2. Technical Mistakes** - **Wrong `--batch-size`:** Too small underutilizes GPU. Too large causes OOM. Match to GPU memory and tensor core size. - **Ignoring `--numa`:** Multi-CPU systems suffer cross-socket latency. Enable `--numa` for better memory locality. - **Misusing `--mmap`:** `--mmap` reduces RAM usage but increases first-request latency. Disable for benchmarking. - **Not clearing GPU memory:** `nvidia-smi` may show stale memory. Run `nvidia-smi --gpu-reset` or reboot between major config changes. - **Measuring network latency as model latency:** Proxy buffering, WebSocket overhead, and client-side parsing inflate latency. Measure raw `/v1/chat/completions` responses. - **Assuming MTP always helps:** MTP overhead varies by hardware, model, and context. Measure acceptance rate and speedup empirically. **3. Analytical Mistakes** - **Overfitting to outliers:** A single slow request skews averages. Use median and p95. Investigate outliers but don't let them dictate conclusions. - **Ignoring confidence intervals:** Small sample sizes produce noisy results. Run 30+ requests per configuration. Calculate 95% CI. - **Chasing marginal gains:** A 3% TPS increase is not worth a 50% TTFB increase or quality degradation. Optimize for holistic UX. - **Neglecting output quality:** Speed without accuracy is useless. Validate outputs with automated and human checks. - **Not documenting configurations:** `llama.cpp` flags, model versions, driver versions, and prompt lengths must be logged. Reproducibility is critical. **4. Mitigation Strategies** - **Use automated scripts:** `curl` loops, Python benchmarking tools, or `locust` for repeatable tests. - **Log everything:** Timestamps, configs, metrics, outputs. Store in SQLite for querying. - **Validate statistically:** Calculate means, medians, p95, p99, confidence intervals. Use t-tests or Mann-Whitney U for significance. - **Test real workloads:** Use actual prompts from your use cases. Avoid synthetic or trivial prompts. - **Iterate slowly:** Change one variable, measure, analyze, decide. Document each step. - **Prioritize stability:** Consistent TPS and low p99 latency matter more than peak TPS. Users experience the worst, not the best. By following this guide, you will transform subjective frustration into objective, actionable insights. You will know exactly when MTP helps, how to manage your reasoning budget, and how to tune your local server for your specific workload. The weekend checklist ensures you collect clean data, the SQLite queries extract meaningful patterns, and the interpretation framework prevents self-deception. Your local AI lab will run faster, more consistently, and with confidence. Your local AI lab will run faster, more consistently, and with confidence. # Advanced MTP Verification Mechanics & Draft Model Selection Multi-Token Prediction is often misunderstood as a simple "speed boost," but it is fundamentally a trade-off between compute parallelism and verification overhead. To truly leverage MTP, you must understand its internal mechanics and how to align it with your hardware and model architecture. **1. The Speculative Decoding Pipeline** When MTP is enabled, `llama.cpp` executes a two-phase cycle per generation step: - **Draft Phase:** The draft model autoregressively generates `n` tokens. This is fast because the draft model is smaller and often quantized. - **Verification Phase:** The target model processes the entire prompt + draft tokens in a single forward pass. It computes attention and FFN layers for all draft positions in parallel. - **Acceptance/Rejection:** The target model compares its predicted tokens against the draft tokens. Tokens that match are committed. The first mismatch triggers a fallback to standard autoregressive generation for the remaining tokens. - **KV Cache Update:** Accepted draft tokens are appended to the KV cache. Rejected tokens are discarded, and the cache pointer rolls back to the last accepted position. **2. Draft Model Selection Strategy** The acceptance rate is directly proportional to the similarity between the draft and target models. Follow this hierarchy: - **Same Architecture, Quantized Draft:** Use a Q4_K_M or Q5_K_M version of your target model as the draft. This preserves token distribution similarity while reducing compute. - **Smaller Parameter Count:** If available, use a distilled or smaller variant (e.g., 7B draft for 13B target). Ensure the draft was trained on similar data. - **Avoid Cross-Architecture Drafts:** Do not use Llama-3 draft for Mistral targets, or vice versa. Tokenizer divergence and attention head differences will tank acceptance rates below 0.3. - **Verify Tokenizer Alignment:** MTP fails silently if draft and target tokenizers differ. Run `llama-tokenize` on both models and compare the first 100 tokens of a common prompt. Mismatches indicate incompatible tokenizers. **3. Tuning `--mtp-n` and `--mtp-max`** - `--mtp-n` (draft tokens per step): Start at 4. Increase to 8 if GPU memory bandwidth is high and acceptance rate stays >0.6. Decrease to 2 if verification stalls or TTFB spikes. - `--mtp-max` (total draft tokens per request): Set to 16-32 for long contexts. Lower values (8-12) are better for short prompts where draft overhead dominates. - **Diminishing Returns:** Beyond `--mtp-n=8`, verification parallelism gains are offset by attention computation overhead. Benchmark at 4, 6, 8, and 10 to find your hardware's sweet spot. **4. MTP Failure Modes & Recovery** - **Rejection Storms:** If acceptance rate drops below 0.4, MTP verification overhead exceeds generation speed. The server will appear slower than non-MTP. Disable MTP or switch to a closer draft model. - **KV Cache Rollback Thrashing:** Frequent rejections cause repeated cache pointer adjustments. Monitor `kv_cache_mb` for sawtooth patterns. If present, reduce `--mtp-n` or increase `--ctx-size` to provide buffer space. - **Thermal Throttling:** MTP verification spikes GPU power draw. If `temp` exceeds 85°C, `power` caps, and `tps` drops, reduce `--mtp-n` or enable `--temp-limit` if your hardware supports it. --- # KV Cache Thrashing & Context Window Optimization The KV cache is the single biggest bottleneck in long-context inference. Understanding its allocation patterns prevents performance degradation and OOM errors. **1. KV Cache Allocation Mechanics** `llama.cpp` allocates KV cache in fixed-size blocks. When a request exceeds the current block, the engine may: - **Expand in-place:** Rare, causes memory fragmentation. - **Allocate new block & copy:** Common, causes temporary memory spikes. - **Trigger GC:** If memory is tight, the engine may evict older cache entries, causing attention recomputation. **2. Sliding Window & RoPE Scaling** - `--rope-scaling type=linear` or `--rope-scaling type=yarn`: Extends context beyond native limits without linear memory growth. YARN is preferred for >8k contexts. - `--rope-scaling factor=2.0`: Doubles effective context. Verify attention quality at extended lengths; some models degrade beyond 4x native context. - `--chunk-size 512`: Forces KV cache to process in chunks, reducing peak memory but increasing latency. Use for streaming or memory-constrained setups. **3. Context Pruning Strategies** - **RAG Windowing:** Truncate context to last 4k tokens before sending to model. Use a retrieval step to inject only relevant chunks. - **Summarization Fallback:** If context >8k, run a lightweight summarization model to compress older tokens. Replace raw context with summary + recent tokens. - **Attention Masking:** Use `--cache-type-k q8_0` for older context to save memory, while keeping recent context in `f16` for accuracy. **4. Monitoring KV Cache Health** - **Utilization Threshold:** Keep `kv_cache_used / ctx_size < 0.75`. Above 0.8, expect TPS decay and allocation stalls. - **Allocation Frequency:** Log `kv_cache_mb` over time. Frequent jumps indicate poor `--batch-size` alignment or context fragmentation. - **Eviction Detection:** If TPS drops sharply after 6k+ tokens, the engine is likely evicting cache entries. Increase `--ctx-size` or implement context pruning. --- # Statistical Validation & Python Analysis Scripts Raw metrics are noisy. Proper statistical validation separates signal from hardware jitter and OS scheduling artifacts. **1. Python Benchmarking & Analysis Script** ```python import sqlite3 import pandas as pd import numpy as np from scipy import stats import matplotlib.pyplot as plt # Load data conn = sqlite3.connect('llama_benchmark.db') df = pd.read_sql_query("SELECT * FROM requests WHERE mtp_enabled IS NOT NULL", conn) # Filter warm-up requests (first 5 per model/config) df['request_order'] = df.groupby(['model', 'mtp_enabled']).cumcount() df = df[df['request_order'] >= 5].copy() # Calculate confidence intervals def calc_ci(series, alpha=0.05): mean = series.mean() sem = stats.sem(series) ci = stats.t.interval(1-alpha, len(series)-1, loc=mean, scale=sem) return mean, ci[0], ci[1] results = df.groupby('mtp_enabled').agg( tps_mean=('tps', lambda x: calc_ci(x)[0]), tps_ci_lower=('tps', lambda x: calc_ci(x)[1]), tps_ci_upper=('tps', lambda x: calc_ci(x)[2]), p95_latency=('total_latency_ms', lambda x: np.percentile(x, 95)), acceptance_rate=('acceptance_rate', 'mean') ).reset_index() # Hypothesis testing: MTP vs non-MTP TPS mtp_df = df[df['mtp_enabled'] == 1]['tps'] non_mtp_df = df[df['mtp_enabled'] == 0]['tps'] t_stat, p_value = stats.ttest_ind(mtp_df, non_mtp_df, equal_var=False) print(f"TPS p-value: {p_value:.4f}") print(f"Statistically significant: {p_value < 0.05}") # Visualization plt.figure(figsize=(10, 6)) plt.boxplot([non_mtp_df, mtp_df], labels=['Non-MTP', 'MTP']) plt.title('TPS Distribution Comparison') plt.ylabel('Tokens Per Second') plt.grid(True, alpha=0.3) plt.show() ``` **2. Handling Non-Normal Distributions** TPS and latency often follow skewed distributions. Use non-parametric tests: - `stats.mannwhitneyu(non_mtp_df, mtp_df, alternative='two-sided')` for median comparison. - `stats.shapiro()` to test normality. If p<0.05, use Mann-Whitney U instead of t-test. - Report median and IQR (Interquartile Range) alongside mean for skewed metrics. **3. Rolling Averages & Trend Detection** ```python df['tps_rolling_5'] = df.groupby('mtp_enabled')['tps'].transform(lambda x: x.rolling(5).mean()) df['tps_trend'] = df.groupby('mtp_enabled')['tps_rolling_5'].diff() # Negative trend indicates degradation over time (thermal throttling, cache thrashing) ``` **4. Avoiding P-Hacking** - Predefine hypotheses before testing. Do not run 20 different metrics and claim significance on one. - Use Bonferroni correction if testing multiple hypotheses: `alpha_adjusted = 0.05 / number_of_tests`. - Validate findings on a separate test set. Split data 70/30 train/test. --- # Hardware & Driver Tuning Software optimization hits a ceiling without proper hardware configuration. PCIe bandwidth, NUMA topology, and thermal management dictate real-world performance. **1. PCIe Lane Configuration** - **x16 vs x8:** Modern GPUs perform identically on x8 or x16 for inference, but x16 provides headroom for memory bandwidth. Verify with `lspci | grep -i vga`. - **Switch Bifurcation:** If using multiple GPUs, ensure PCIe switch is configured for x8/x8 bifurcation. Mismatched lanes cause bandwidth bottlenecks. - **NVLink/Infinity Fabric:** If available, enable for multi-GPU KV cache sharing. `llama.cpp` supports `--tensor-split` but NVLink reduces cross-GPU latency. **2. NUMA Topology Awareness** - **Identify NUMA nodes:** `numactl --hardware` or `lscpu`. - **Bind processes to nodes:** `numactl --cpunodebind=0 --membind=0 python benchmark.py`. Prevents cross-socket memory latency. - **Disable CPU frequency scaling:** `cpupower frequency-set -g performance`. Prevents thermal throttling during sustained inference. **3. Thermal Management** - **Fan Curves:** Set aggressive fan curves via `nvidia-smi -pl 300` (power limit) or `rocm-smi --setfan=100`. Higher temps reduce boost clocks. - **Case Airflow:** Ensure front-to-back airflow. GPU exhaust should not recirculate. Use `sensors` to monitor CPU/GPU temps. - **Thermal Throttling Detection:** If `tps` drops >15% after 10 minutes, thermal throttling is active. Improve cooling or reduce `--gpu-layers`. **4. Driver & Runtime Optimization** - **NVIDIA:** Use latest stable driver. Enable `nvidia-smi -pm 1` (Persistence Mode) to prevent context switching overhead. - **AMD:** Use ROCm 6.0+. Enable `HSA_OVERRIDE_GFX_VERSION=10.3.0` for older cards. Set `ROCM_PATH` correctly. - **CUDA/ROCm Memory Allocator:** `export CUDA_MEMFRAG=1` or `export HSA_FORCE_FINE_GRAIN_PCIE=1` to reduce allocation latency. --- # Use-Case Specific Parameter Matrices Different workloads require fundamentally different tuning strategies. Use these matrices as starting points. **1. Coding Assistants** - **Priority:** Low TTFB, high syntax accuracy, strict tool-calling. - **Config:** `--ctx-size 8192 --batch-size 512 --cache-type-k f16 --mtp 0` - **Why:** MTP introduces token-level drift that breaks syntax and JSON schemas. Full precision KV cache prevents quantization errors in code. - **Prompt Strategy:** Use system prompts with strict formatting rules. Validate output with `ast.parse()` or `json.loads()` before returning. **2. RAG Pipelines** - **Priority:** Context window efficiency, retrieval accuracy, citation formatting. - **Config:** `--ctx-size 16384 --rope-scaling type=yarn --rope-scaling factor=2.0 --chunk-size 512 --mtp 4` - **Why:** YARN scaling extends context without linear memory growth. MTP accelerates generation over long retrieved chunks. - **Prompt Strategy:** Inject retrieval context with clear delimiters. Use `{{context}}` tags. Truncate to last 4k tokens if context >8k. **3. Agentic Workflows** - **Priority:** Streaming stability, tool parameter accuracy, state management. - **Config:** `--ctx-size 4096 --batch-size 256 --stream true --chunk-size 256 --mtp 0` - **Why:** Agentic loops require predictable token generation. MTP rejection storms break tool-calling state. Small batch size ensures responsive streaming. - **Prompt Strategy:** Use structured output formats (JSON Schema). Validate tool calls before execution. Implement retry logic for MTP-induced drift. **4. Chat & Conversational AI** - **Priority:** Fast TTFB, natural flow, memory efficiency. - **Config:** `--ctx-size 4096 --batch-size 512 --mtp 4 --mtp-n 4` - **Why:** Chat benefits from MTP speedup. Short contexts keep KV cache stable. Balanced batch size ensures responsiveness. - **Prompt Strategy:** Use conversation history truncation. Keep system prompt concise. Monitor p95 latency; >2s TTFB degrades UX. **5. Creative Writing** - **Priority:** Long context coherence, stylistic consistency, formatting preservation. - **Config:** `--ctx-size 8192 --rope-scaling type=linear --cache-type-k f16 --mtp 8 --mtp-max 32` - **Why:** Creative writing requires extended context. High `--mtp-n` accelerates generation. Full precision preserves stylistic nuance. - **Prompt Strategy:** Use chapter/scene delimiters. Implement periodic summarization to reset context window. Validate formatting with regex. --- # Automated Benchmarking & Reporting Pipelines Manual testing is error-prone. Automate data collection, analysis, and reporting for consistent results. **1. Systemd Service for Continuous Monitoring** ```ini # /etc/systemd/system/llama-monitor.service [Unit] Description=LLaMA.cpp Server Monitor After=network.target [Service] Type=simple User=ai-lab ExecStart=/usr/local/bin/servertop --interval 0.1 --output /var/log/llama/gpu.csv Restart=always RestartSec=5 [Install] WantedBy=multi-user.target ``` **2. Cron Job for Automated Benchmarking** ```bash # /etc/cron.d/llama-benchmark 0 2 * * * ai-lab /opt/benchmark/run_daily.sh >> /var/log/llama/benchmark.log 2>&1 ``` **3. Daily Benchmark Script** ```bash #!/bin/bash # /opt/benchmark/run_daily.sh PROMPTS=("coding.txt" "rag.txt" "chat.txt" "creative.txt") CONFIGS=("mtp_off" "mtp_on") for config in "${CONFIGS[@]}"; do for prompt_file in "${PROMPTS[@]}"; do echo "Running $config on $prompt_file" python /opt/benchmark/run_test.py --config $config --prompt $prompt_file --runs 30 done done # Generate report python /opt/benchmark/generate_report.py --db /var/lib/llama_benchmark.db --output /var/www/benchmark.html ``` **4. Report Generation Script** ```python # /opt/benchmark/generate_report.py import sqlite3 import pandas as pd import jinja2 conn = sqlite3.connect('llama_benchmark.db') df = pd.read_sql_query("SELECT * FROM requests WHERE request_order >= 5", conn) # Calculate metrics summary = df.groupby('config').agg( avg_tps=('tps', 'mean'), p95_latency=('total_latency_ms', lambda x: x.quantile(0.95)), acceptance_rate=('acceptance_rate', 'mean') ).reset_index() # Render HTML report template = """ LLaMA Benchmark Report

Weekly Benchmark Report

{% for row in summary %} {% endfor %}
ConfigAvg TPSP95 LatencyAcceptance Rate
{{ row.config }}{{ row.avg_tps }}{{ row.p95_latency }}{{ row.acceptance_rate }}
""" env = jinja2.Environment(loader=jinja2.FileSystemLoader('.')) template = env.from_string(template) html = template.render(summary=summary.to_dict('records')) with open('benchmark.html', 'w') as f: f.write(html) ``` --- # Edge Case Troubleshooting Even with perfect configuration, edge cases will arise. Here’s how to diagnose and resolve them. **1. OOM During Long Generation** - **Symptom:** Server crashes or returns 500 error after 4k-8k tokens. - **Cause:** KV cache expansion exceeds VRAM. MTP verification spikes memory. - **Fix:** Reduce `--ctx-size`, enable `--chunk-size 512`, disable MTP, or switch to `--cache-type-k q8_0`. Monitor `nvidia-smi` for allocation spikes. **2. MTP Acceptance Rate Drops to Zero** - **Symptom:** Generation falls back to autoregressive, TPS matches non-MTP. - **Cause:** Draft model mismatch, tokenizer divergence, or context window overflow. - **Fix:** Verify tokenizer alignment, switch to closer draft model, reduce `--mtp-n`, or disable MTP for short prompts. **3. Streaming Buffer Stalls** - **Symptom:** Tokens arrive in bursts, not continuously. TTFB is low, but chunk latency is high. - **Cause:** Proxy buffering, WebSocket frame size limits, or `--chunk-size` misalignment. - **Fix:** Set `--chunk-size 256`, configure proxy to flush every 100ms, disable gzip compression for streaming endpoints. **4. Thermal Throttling Under Load** - **Symptom:** TPS drops 20-30% after 5-10 minutes. GPU temp >85°C. - **Cause:** Sustained compute exceeds cooling capacity. - **Fix:** Improve airflow, set aggressive fan curves, reduce `--gpu-layers`, or lower `--batch-size`. Monitor `power` limit in `nvidia-smi`. **5. Context Window Thrashing** - **Symptom:** TPS decays exponentially after 6k tokens. Memory usage spikes. - **Cause:** KV cache reallocation, attention recomputation, or RoPE scaling limits. - **Fix:** Increase `--ctx-size`, enable `--rope-scaling type=yarn`, implement context pruning, or use sliding window attention. **6. Proxy/Client Latency Inflation** - **Symptom:** `curl` shows low latency, OpenWebUI shows high latency. - **Cause:** WebSocket buffering, UI thread blocking, or Flight Recorder overhead. - **Fix:** Measure raw `/v1/chat/completions` responses. Disable proxy logging during benchmarking. Use `ab` or `locust` for load testing. --- # Final Validation & Deployment Checklist Before deploying your optimized configuration to production, run this final validation sequence. **1. Stability Test** - Run 100 consecutive requests with mixed prompt lengths. - Monitor for OOM, crashes, or TPS decay. - Verify KV cache remains stable. No allocation spikes >10%. **2. Quality Validation** - Run 20 prompts per use case with and without MTP. - Compare outputs for accuracy, formatting, and coherence. - Use automated checks: syntax validation, JSON parsing, citation format. - Human eval: Rate on 1-5 scale for usefulness. **3. Load Testing** - Simulate 10 concurrent users with varying prompt lengths. - Measure p95 latency, error rate, and memory usage. - Verify proxy handles WebSocket connections without dropping frames. **4. Documentation & Handoff** - Log all configuration flags, model versions, and driver versions. - Document acceptance rate thresholds and MTP enable/disable criteria. - Share dashboard links and SQLite export scripts with team. - Schedule weekly benchmark runs to track degradation over time. Your local AI lab is now equipped with a rigorous, data-driven approach to inference optimization. You understand MTP mechanics, KV cache management, statistical validation, and hardware tuning. You can confidently decide when speculative decoding helps, how to manage reasoning budgets, and how to tune for specific workloads. The weekend checklist ensures clean data collection, the SQL queries extract actionable insights, and the troubleshooting guide prevents self-deception. Your server will run faster, more consistently, and with measurable confidence.