# Quick Diagnosis When your local `llama.cpp` server feels inconsistent, the symptoms you're describing—variable first-token latency, sudden GPU memory jumps, and uncertainty around Multi-Token Prediction (MTP)—are almost always rooted in three overlapping subsystems: KV cache allocation dynamics, speculative decoding overhead, and workload mismatch. Let's break down what's actually happening under the hood so you know exactly where to look. **First-Token Latency (TTFT) Variance** The time to first token is dominated by the prompt processing phase. In `llama.cpp`, this means the model must pass your entire input context through the forward pass before generating a single token. If your context window is large, if you're using dynamic batching, or if the GPU is thrashing between CPU and VRAM due to offloading limits, TTFT will fluctuate. Additionally, if your system is warming up the compute graph, compiling kernels, or allocating KV cache pages on the fly, the first few requests will always be slower. This is normal, but it becomes a problem when it persists across requests. **GPU Memory Jumps** VRAM spikes in `llama.cpp` are rarely random. They typically occur when: 1. The KV cache expands to accommodate a longer context or a new batch. 2. MTP/speculative decoding allocates temporary buffers for draft tokens. 3. The GPU driver or `llama.cpp` reallocates memory due to fragmentation. 4. OpenWebUI or the proxy layer buffers large responses before streaming. If you're seeing jumps that don't settle, you're likely hitting KV cache reallocation thresholds or experiencing memory fragmentation from repeated allocate/deallocate cycles. **MTP Uncertainty** Multi-Token Prediction (or speculative decoding) works by using a draft mechanism to predict multiple tokens ahead of time, then verifying them with the main model. If the acceptance rate is low, MTP adds overhead without speed gains. If the acceptance rate is high but VRAM jumps, the draft buffers are consuming headroom. The key metric isn't "is MTP on?" but "what is the effective tokens-per-second (TPS) after accounting for draft overhead, VRAM cost, and acceptance rate?" Many users enable MTP without tuning `--mtp-draft`, `--speculative`, or context limits, leading to unpredictable behavior. **Initial Sanity Checks Before the Weekend** - Verify your `llama.cpp` version and build flags (CUDA, Vulkan, ROCm, etc.). MTP support varies by backend. - Check your quantization format. Q4_K_M, Q5_K_S, and Q8_0 have different VRAM footprints and compute characteristics. - Ensure you're not mixing CPU offload with full GPU offload inconsistently. - Confirm that OpenWebUI isn't adding hidden latency via template processing or safety filters. - Run `nvidia-smi` or `rocm-smi` during a generation to watch VRAM allocation patterns in real-time. Once you've confirmed the basics, you'll move into structured data collection. The goal isn't to guess what's happening; it's to measure it, isolate variables, and build a reproducible testing pipeline. # Data Collection Plan A reliable benchmark requires controlled inputs, consistent logging, and clear metric definitions. Your stack—Flight Recorder proxy, ServerTop dashboard, and SQLite backend—gives you everything you need if configured correctly. **1. Flight Recorder Proxy Configuration** The proxy should sit between OpenWebUI and `llama.cpp`. Configure it to: - Log every request/response pair with timestamps, prompt length, generation length, and model ID. - Capture GPU metrics at fixed intervals (e.g., every 500ms) using a sidecar script or built-in polling. - Tag requests by workload type (chat, coding, RAG, creative, agentic) using custom headers or query parameters. - Enable raw timing breakdowns: TTFT, inter-token latency, total generation time, and MTP draft/verify cycles. Example proxy header injection for workload tagging: ```http X-Workload-Type: coding X-Test-Profile: baseline-mtp-off X-Context-Window: 8192 ``` **2. ServerTop Dashboard Metric Definitions** ServerTop should visualize real-time telemetry. Define these core metrics explicitly: - `ttft_ms`: Time to first token in milliseconds. - `tps`: Tokens per second during generation phase. - `vram_used_mb`: Current VRAM allocation. - `vram_peak_mb`: Highest VRAM usage in the current session. - `kv_cache_utilization`: Percentage of allocated KV cache slots in use. - `mtp_acceptance_rate`: Ratio of draft tokens accepted by the main model. - `mtp_draft_tokens`: Number of speculative tokens generated per step. - `gpu_compute_pct`: GPU utilization percentage. - `cpu_offload_pct`: Percentage of layers running on CPU (if applicable). - `request_queue_depth`: Number of pending requests. Set sampling intervals to 250ms for latency metrics and 1s for VRAM/GPU metrics. Higher sampling introduces overhead; lower sampling misses spikes. **3. SQLite Backend Schema Assumptions** Your SQLite database should store structured logs. A practical schema looks like: ```sql CREATE TABLE requests ( id INTEGER PRIMARY KEY, timestamp TEXT, workload_type TEXT, prompt_tokens INTEGER, generated_tokens INTEGER, ttft_ms REAL, total_ms REAL, tps REAL, mtp_enabled INTEGER, mtp_acceptance_rate REAL, vram_peak_mb REAL, status TEXT ); CREATE TABLE gpu_snapshots ( id INTEGER PRIMARY KEY, timestamp TEXT, vram_used_mb REAL, vram_free_mb REAL, gpu_util_pct REAL, kv_cache_pct REAL, fragmentation_index REAL ); CREATE TABLE mtp_events ( id INTEGER PRIMARY KEY, request_id INTEGER, step INTEGER, draft_tokens INTEGER, accepted_tokens INTEGER, verify_ms REAL, FOREIGN KEY(request_id) REFERENCES requests(id) ); ``` Adjust column names to match your actual Flight Recorder output, but keep the logical structure. **4. Synthetic Load Generation with curl** Use `curl` to send controlled prompts. This eliminates OpenWebUI template variance and ensures reproducible inputs. Baseline chat prompt: ```bash curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "X-Workload-Type: chat" \ -d '{ "model": "local-model", "messages": [{"role": "user", "content": "Explain quantum entanglement in three paragraphs."}], "max_tokens": 512, "temperature": 0.7, "stream": false }' ``` Coding prompt: ```bash curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "X-Workload-Type: coding" \ -d '{ "model": "local-model", "messages": [{"role": "user", "content": "Write a Python function that parses a CSV file and returns a dictionary of column statistics."}], "max_tokens": 1024, "temperature": 0.2, "stream": false }' ``` RAG-style prompt (simulated context): ```bash curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "X-Workload-Type: rag" \ -d '{ "model": "local-model", "messages": [{"role": "system", "content": "You are a research assistant. Use the following context to answer."}, {"role": "user", "content": " [Insert 2000 tokens of technical documentation] \nQuestion: How does the caching layer handle eviction policies?"}], "max_tokens": 256, "temperature": 0.1, "stream": false }' ``` Run each prompt 5-10 times per profile. Record the `curl` output timing alongside the proxy logs. Use `--max-time 120` to prevent hangs. **5. Environment Control** - Disable background GPU workloads (video encoding, desktop compositing, other AI services). - Fix CPU governor to `performance` mode. - Clear GPU cache between test runs if possible (`nvidia-smi --gpu-reset` or equivalent). - Log OS temperature and thermal throttling events. - Keep `llama.cpp` config files version-controlled. With this foundation, you'll have clean, timestamped, workload-tagged data flowing into SQLite, visualized in ServerTop, and ready for comparative analysis. # MTP Comparison Plan Comparing MTP (Multi-Token Prediction) against non-MTP profiles requires strict experimental control. MTP isn't a simple toggle; it's a trade-off between speculative overhead and generation speed. Here's how to test it without fooling yourself. **1. Define Test Profiles** Create two identical configurations except for MTP settings: - `baseline`: `--mtp-draft 0`, `--speculative 0`, MTP disabled. - `mtp-on`: `--mtp-draft 4`, `--speculative 1`, MTP enabled with moderate draft depth. Keep everything else constant: model file, quantization, `--gpu-layers`, `--batch`, `--ubatch`, context window, temperature, and system load. **2. Warm-Up Protocol** GPU inference engines compile kernels and allocate memory on first use. Always run a 3-request warm-up sequence before recording data: ```bash for i in 1 2 3; do curl -s -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"local-model","messages":[{"role":"user","content":"Warmup prompt."}],"max_tokens":64}' > /dev/null done ``` Discard warm-up metrics. Only record data after VRAM allocation stabilizes and TTFT variance drops below 15%. **3. Control Variables Checklist** - Same prompt text and token count. - Same `max_tokens` for generation. - Same GPU driver version and OS state. - Same `llama.cpp` binary and config. - Same network latency (localhost only). - Same OpenWebUI template (or bypass entirely via direct API). **4. Metrics to Track** - **Effective TPS**: `(generated_tokens) / (total_ms / 1000)`. This is your primary speed metric. - **MTP Acceptance Rate**: `accepted_draft_tokens / total_draft_tokens`. Below 0.6, MTP is likely hurting performance. Above 0.8, it's usually beneficial. - **VRAM Overhead**: `vram_peak_mtp - vram_peak_baseline`. MTP requires extra buffers. If overhead exceeds 15% of your total VRAM, you risk fragmentation. - **TTFT Impact**: MTP shouldn't significantly increase prompt processing time. If TTFT jumps >20%, your draft model or context handling is misconfigured. - **Latency Distribution**: Record p50, p90, p95, p99 for both TTFT and inter-token latency. MTP can reduce variance if acceptance is high, but increase it if verification fails often. **5. Statistical Validation** Never judge MTP on a single run. Run each profile 10 times per workload type. Calculate: - Mean and standard deviation for TPS and TTFT. - 95% confidence intervals. - Paired t-test or Wilcoxon signed-rank test if distributions aren't normal. - Effect size (Cohen's d) to determine practical significance. Example calculation logic: ``` baseline_tps = [42.1, 41.8, 43.0, 42.5, 41.9, 42.3, 42.0, 41.7, 42.6, 42.2] mtp_tps = [48.5, 47.9, 49.1, 48.2, 47.8, 48.6, 48.0, 47.5, 48.9, 48.3] # Compare means, check variance, compute confidence intervals ``` **6. When MTP Actually Helps** - High acceptance rate (>0.75) - Long generation sequences (>512 tokens) - Stable VRAM headroom (>20% free) - Predictable token distributions (coding, structured output, factual QA) - Low temperature settings (0.1-0.4) **7. When MTP Hurts** - Low acceptance rate (<0.6) - Short responses (<128 tokens) - High temperature/creative prompts - VRAM-constrained setups - Models without native MTP heads (relying on external draft models adds latency) Document every parameter change. If you tweak `--mtp-draft` or `--speculative`, treat it as a new profile. MTP tuning is iterative, not binary. # Reasoning Budget Plan A "reasoning budget" in local LLM inference refers to how you allocate compute, memory, and speculative depth across different workloads. You can't optimize for everything simultaneously. Here's how to structure budgets for your use cases. **1. VRAM Allocation Strategy** - Reserve 10-15% of total VRAM as headroom for OS, driver, and fragmentation. - Allocate KV cache based on expected context window: `kv_cache_mb ≈ (context_tokens * hidden_size * layers * quant_bits) / 8`. - Leave room for MTP draft buffers if enabled: typically 0.5-1.5 GB depending on draft depth. - Use `--gpu-layers` to offload strategically. Offloading 90-95% is usually optimal; 100% can cause CPU-GPU sync bottlenecks. **2. Batch Size Tuning** - `--batch`: Controls prompt processing batch size. Higher values speed up TTFT but increase VRAM. Start at 2048, adjust based on VRAM headroom. - `--ubatch`: Controls generation batch size. Keep this low (512-1024) for single-user setups. Higher values help multi-user proxies but increase latency variance. **3. Workload-Specific Budget Profiles** | Workload | Context Window | Max Tokens | Temperature | MTP Draft | GPU Layers | Priority Metric | |----------|----------------|------------|-------------|-----------|------------|-----------------| | Coding | 8192-16384 | 1024-2048 | 0.1-0.3 | 4-6 | 90-95% | Accuracy + Stable TPS | | RAG | 4096-8192 | 256-512 | 0.0-0.2 | 2-4 | 85-90% | Low TTFT + Low VRAM Frag | | Agentic | 4096-8192 | 128-256 | 0.1-0.2 | 2-3 | 90% | Consistent TTFT + Tool Parse Reliability | | Chat | 2048-4096 | 256-512 | 0.7-0.9 | 0-2 | 80-85% | Low Latency + Conversational Flow | | Creative | 4096-8192 | 1024-4096 | 0.8-1.2 | 0-1 | 75-80% | Sustained TPS + VRAM Stability | **4. Dynamic Budget Adjustment** - Monitor `kv_cache_utilization`. If it consistently exceeds 85%, reduce context window or increase VRAM headroom. - If `mtp_acceptance_rate` drops below 0.6, reduce `--mtp-draft` or disable MTP for that workload. - If TTFT variance exceeds 30%, increase `--batch` or reduce prompt preprocessing overhead. - If VRAM fragmentation index rises, restart the server periodically or use `--cache-type-k q8_0` to reduce reallocation frequency. **5. Quantization Budget Trade-offs** - Q4_K_M: Best balance of speed, VRAM, and quality. Recommended for most workloads. - Q5_K_S: Slightly better reasoning, higher VRAM. Good for coding/RAG. - Q8_0: Near-fp16 quality, high VRAM. Only use if VRAM allows and accuracy is critical. - Avoid Q2/Q3 for reasoning tasks; they degrade MTP acceptance and increase hallucination rates. **6. System-Level Budget Controls** - Use `cgroups` or `systemd` resource limits to prevent `llama.cpp` from starving other services. - Set `GPU_MEM_LIMIT` if your driver supports it. - Configure OpenWebUI to respect `max_tokens` and `stop_sequences` to prevent runaway generations. - Implement request queuing with priority tags in Flight Recorder to prevent context thrashing. Your reasoning budget isn't static. It's a living configuration that adapts to observed metrics. Log every budget change alongside performance data so you can correlate configuration shifts with outcome changes. # Long Output Test Plan Long generations expose hidden bottlenecks: KV cache exhaustion, VRAM fragmentation, latency drift, and thermal throttling. This test plan stress-tests your setup and reveals whether your configuration holds up over extended sequences. **1. Test Configuration** - Prompt: A structured request that naturally generates long outputs. Example: "Write a detailed technical documentation guide for a REST API, including endpoints, request/response schemas, error codes, and authentication flows. Generate at least 4000 tokens." - `max_tokens`: 4096 or 8192 (depending on model context limit). - Temperature: 0.7 (balanced creativity/structure). - Disable streaming initially to measure total generation time accurately. - Run 3 iterations per profile (baseline vs MTP). **2. Monitoring Targets** - **VRAM Timeline**: Watch for step-wise increases (KV cache growth) vs. sudden spikes (reallocation/fragmentation). - **TPS Drift**: Plot tokens-per-second over time. Healthy systems show flat or slightly declining TPS. Exponential decay indicates cache thrashing or thermal throttling. - **Inter-Token Latency Variance**: High variance suggests GPU scheduling issues or CPU-GPU sync bottlenecks. - **MTP Acceptance Decay**: If MTP acceptance rate drops as generation progresses, the draft model is losing alignment with the main model's distribution. **3. Automated Test Script** ```bash #!/bin/bash PROFILE="long-output-test" MAX_TOKENS=4096 ITERATIONS=3 for i in $(seq 1 $ITERATIONS); do echo "=== Iteration $i ===" START=$(date +%s%N) curl -s -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "X-Workload-Type: long-output" \ -H "X-Test-Profile: $PROFILE" \ -d "{ \"model\": \"local-model\", \"messages\": [{\"role\": \"user\", \"content\": \"Write a detailed technical documentation guide for a REST API, including endpoints, request/response schemas, error codes, and authentication flows. Generate at least 4000 tokens.\"}], \"max_tokens\": $MAX_TOKENS, \"temperature\": 0.7, \"stream\": false }" > /tmp/long_output_$i.json END=$(date +%s%N) ELAPSED=$(( (END - START) / 1000000 )) echo "Total time: ${ELAPSED}ms" sleep 5 # Allow VRAM to settle done ``` **4. Failure Modes to Watch** - **OOM (Out of Memory)**: `llama.cpp` crashes or returns HTTP 500. Indicates KV cache exceeded VRAM limits. - **Latency Spike at Token N**: Suggests KV cache reallocation threshold hit. Adjust `--ctx-size` or use `--cache-type-k q8_0`. - **TPS Collapse**: Drops below 50% of initial TPS. Check thermal limits, GPU clock throttling, or CPU offload bottlenecks. - **MTP Divergence**: Acceptance rate falls below 0.4 after 2000 tokens. Reduce draft depth or disable MTP for long generations. **5. Recovery & Mitigation** - If VRAM fragmentation is high, implement periodic cache clearing or use `--no-mmap` to force contiguous allocation. - If thermal throttling occurs, improve cooling or reduce `--gpu-layers` to shift compute to CPU temporarily. - If MTP hurts long outputs, create a workload-specific profile that disables MTP for `max_tokens > 2048`. Long output testing reveals whether your configuration is sustainable or just fast for short bursts. Document the degradation curves; they're invaluable for capacity planning. # SQLite Queries Your Flight Recorder proxy logs everything to SQLite. These queries will extract actionable insights from your weekend tests. Adjust table/column names to match your schema. **1. Latency Percentiles by Workload** ```sql SELECT workload_type, PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY ttft_ms) AS p50_ttft, PERCENTILE_CONT(0.90) WITHIN GROUP (ORDER BY ttft_ms) AS p90_ttft, PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY ttft_ms) AS p95_ttft, PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY ttft_ms) AS p99_ttft, AVG(tps) AS avg_tps, STDDEV(tps) AS stddev_tps FROM requests WHERE timestamp BETWEEN '2024-06-14 08:00:00' AND '2024-06-15 20:00:00' GROUP BY workload_type ORDER BY workload_type; ``` *Purpose*: Identifies latency consistency. High p99/p50 ratio indicates variance issues. **2. MTP Effectiveness Analysis** ```sql SELECT mtp_enabled, COUNT(*) AS request_count, AVG(mtp_acceptance_rate) AS avg_acceptance, AVG(tps) AS avg_tps, AVG(vram_peak_mb) AS avg_vram_peak, AVG(ttft_ms) AS avg_ttft FROM requests WHERE mtp_enabled IN (0, 1) GROUP BY mtp_enabled; ``` *Purpose*: Direct comparison of MTP vs baseline. Look for TPS gain vs VRAM cost trade-off. **3. VRAM Fragmentation & Peak Usage Timeline** ```sql SELECT strftime('%H:%M:%S', timestamp) AS time_slot, AVG(vram_used_mb) AS avg_vram, MAX(vram_used_mb) AS peak_vram, AVG(fragmentation_index) AS avg_frag, AVG(kv_cache_pct) AS avg_kv_util FROM gpu_snapshots WHERE timestamp BETWEEN '2024-06-14 10:00:00' AND '2024-06-14 18:00:00' GROUP BY time_slot ORDER BY time_slot; ``` *Purpose*: Reveals memory pressure patterns. High fragmentation with moderate usage indicates allocation thrashing. **4. MTP Draft vs Acceptance Correlation** ```sql SELECT step, AVG(draft_tokens) AS avg_draft, AVG(accepted_tokens) AS avg_accepted, AVG(accepted_tokens * 1.0 / NULLIF(draft_tokens, 0)) AS acceptance_rate, AVG(verify_ms) AS avg_verify_latency FROM mtp_events WHERE request_id IN ( SELECT id FROM requests WHERE workload_type = 'coding' AND mtp_enabled = 1 ) GROUP BY step ORDER BY step; ``` *Purpose*: Shows how MTP performance degrades or stabilizes over generation steps. **5. Workload-Specific Resource Consumption** ```sql SELECT workload_type, AVG(prompt_tokens) AS avg_prompt_len, AVG(generated_tokens) AS avg_gen_len, AVG(total_ms) AS avg_total_time, AVG(vram_peak_mb) AS avg_vram_peak, COUNT(*) AS request_count FROM requests GROUP BY workload_type ORDER BY avg_total_time DESC; ``` *Purpose*: Maps resource usage to workload characteristics. Helps size budgets accurately. **6. Outlier Detection** ```sql SELECT id, timestamp, workload_type, ttft_ms, tps, vram_peak_mb, status FROM requests WHERE ttft_ms > (SELECT AVG(ttft_ms) + 2*STDDEV(ttft_ms) FROM requests) OR tps < (SELECT AVG(tps) - 2*STDDEV(tps) FROM requests) ORDER BY timestamp DESC LIMIT 20; ``` *Purpose*: Flags anomalous requests for root cause analysis (driver crashes, thermal events, cache misses). Run these queries after each test phase. Export results to CSV for spreadsheet analysis or feed them back into ServerTop for historical comparison. # Dashboard Screenshots To Capture ServerTop provides real-time visualization. Capture these specific charts during and after tests. Annotate them with timestamps, profile names, and workload tags. **1. VRAM Allocation Timeline** - *What to capture*: Line chart of `vram_used_mb` over 2-hour test window. - *What to look for*: Step increases (normal KV growth), sudden spikes (reallocation), plateaus (stable), or sawtooth patterns (fragmentation/thrashing). - *Annotation*: Mark test start/end, MTP toggle events, and long-output test boundaries. **2. TTFT Distribution Histogram** - *What to capture*: Histogram of `ttft_ms` for each workload type. - *What to look for*: Tight clustering (consistent), long right tail (occasional stalls), bimodal distribution (warm-up vs steady-state). - *Annotation*: Overlay baseline vs MTP profiles with different colors. **3. TPS Over Generation Steps** - *What to capture*: Scatter plot of inter-token latency or rolling TPS during long outputs. - *What to look for*: Flat line (healthy), gradual decline (cache pressure), sharp drops (OOM/recovery), or oscillation (MTP instability). - *Annotation*: Highlight acceptance rate correlation if MTP is active. **4. MTP Acceptance Rate Gauge** - *What to capture*: Real-time gauge or line chart of `mtp_acceptance_rate`. - *What to look for*: Sustained >0.75 (beneficial), 0.5-0.75 (marginal), <0.5 (harmful). Watch for decay over long generations. - *Annotation*: Note draft depth settings and temperature values. **5. GPU Compute vs CPU Offload Split** - *What to capture*: Stacked bar or dual-axis chart of `gpu_compute_pct` and `cpu_offload_pct`. - *What to look for*: Balanced utilization, GPU saturation without CPU bottleneck, or CPU-GPU sync stalls. - *Annotation*: Mark `--gpu-layers` configuration. **6. Request Queue & Concurrency Depth** - *What to capture*: Line chart of `request_queue_depth`. - *What to look for*: Zero/low (single-user optimal), spikes (proxy bottleneck), sustained high (overprovisioned or misconfigured batching). - *Annotation*: Note concurrent user simulation if applicable. Save screenshots as PNG with embedded metadata. Use them in your weekend report to correlate configuration changes with observable behavior. Visual patterns often reveal issues that aggregate metrics hide. # Weekend Checklist Follow this chronological schedule to maximize productivity and data quality. Adjust times based on your availability, but preserve the sequence. **Saturday Morning: Environment Setup & Baseline** - [ ] Verify `llama.cpp` version, GPU driver, and OpenWebUI compatibility. - [ ] Configure Flight Recorder proxy with workload tagging and SQLite logging. - [ ] Set up ServerTop dashboard with defined metrics and 250ms/1s sampling. - [ ] Create baseline profile config file (MTP off, standard batch sizes). - [ ] Run 3-request warm-up sequence. Verify VRAM stabilizes. - [ ] Execute 10 iterations of chat, coding, and RAG prompts via `curl`. - [ ] Capture baseline dashboard screenshots. - [ ] Export SQLite logs to `/tmp/baseline_sat.csv`. **Saturday Afternoon: MTP Profiling & Tuning** - [ ] Create MTP profile config (`--mtp-draft 4`, `--speculative 1`). - [ ] Run warm-up sequence. Verify acceptance rate stabilizes. - [ ] Execute 10 iterations of same prompts. Record metrics. - [ ] Adjust `--mtp-draft` to 2 and 6. Run 5 iterations each. - [ ] Capture MTP dashboard screenshots. Note acceptance rate trends. - [ ] Export SQLite logs to `/tmp/mtp_sat.csv`. - [ ] Run paired statistical comparison (TPS, TTFT, VRAM overhead). **Sunday Morning: Long Output & Stress Testing** - [ ] Configure long-output test parameters (`max_tokens: 4096`). - [ ] Run 3 iterations baseline, 3 iterations MTP. - [ ] Monitor VRAM timeline, TPS drift, and fragmentation index. - [ ] Capture stress-test dashboard screenshots. - [ ] Export SQLite logs to `/tmp/long_output_sun.csv`. - [ ] Test thermal limits: run continuous generation for 30 minutes. Log GPU clocks and temperatures. **Sunday Afternoon: Analysis & Configuration Finalization** - [ ] Run all SQLite queries. Export results to spreadsheets. - [ ] Calculate confidence intervals and effect sizes for MTP vs baseline. - [ ] Map metrics to workload budgets. Identify optimal profiles. - [ ] Document VRAM headroom requirements and fragmentation thresholds. - [ ] Create final config files for each workload type. - [ ] Write summary report with recommendations. - [ ] Backup SQLite database and dashboard snapshots. **Verification Steps** - Confirm all timestamps align across proxy, ServerTop, and SQLite. - Verify no OOM crashes or driver resets occurred during tests. - Check that MTP acceptance rates are logged correctly. - Ensure workload tags are consistent in all queries. - Validate that statistical comparisons use matched prompt sets. Stick to the checklist. Deviations introduce noise. If a test fails, log the error, restart cleanly, and retry. Consistency beats speed. # Interpreting Results Raw numbers mean nothing without context. Here's how to synthesize your weekend data into actionable decisions for each workload type. **1. Coding Workloads** - *Target Metrics*: TTFT < 800ms, TPS > 45, MTP acceptance > 0.7, VRAM peak stable. - *Interpretation*: If MTP boosts TPS by >15% without VRAM spikes, enable it. Coding benefits from deterministic token distributions. High acceptance rates indicate the draft model aligns well with syntax patterns. - *Decision*: Use MTP with `--mtp-draft 4-6`. Prioritize Q5_K_S or Q8_0 if accuracy degrades on Q4. Allocate 85-90% GPU layers. Keep context window at 8192-16384. **2. RAG Workloads** - *Target Metrics*: TTFT < 600ms, low VRAM fragmentation, consistent TPS, fast context injection. - *Interpretation*: RAG suffers from variable context lengths. If VRAM jumps correlate with prompt length, your KV cache allocation is inefficient. MTP often hurts RAG due to unpredictable retrieval patterns. - *Decision*: Disable MTP. Use `--cache-type-k q8_0` to reduce reallocation. Keep `--batch` high for prompt processing. Allocate 80-85% GPU layers. Use Q4_K_M for speed/VRAM balance. **3. Agentic Workloads** - *Target Metrics*: TTFT < 500ms, low latency variance, reliable tool parsing, multi-turn stability. - *Interpretation*: Agents require predictable latency. High TTFT variance breaks tool-calling loops. MTP can help if acceptance is high, but verification overhead may delay tool execution. - *Decision*: Test MTP cautiously. If acceptance > 0.75 and TTFT variance drops, enable it. Otherwise, disable. Prioritize low `--ubatch` (512). Use structured output templates. Allocate 90% GPU layers. **4. Chat Workloads** - *Target Metrics*: TTFT < 400ms, conversational flow, low overhead, responsive feel. - *Interpretation*: Chat is latency-sensitive but generation-short. MTP rarely helps unless responses consistently exceed 256 tokens. VRAM efficiency matters more than raw TPS. - *Decision*: Disable MTP for standard chat. Use Q4_K_M. Keep context at 2048-4096. Optimize `--batch` for prompt speed. Prioritize low TTFT variance over peak TPS. **5. Creative Writing Workloads** - *Target Metrics*: Sustained TPS > 35, VRAM stability over 4k+ tokens, acceptable quality degradation. - *Interpretation*: Creative workloads stress KV cache and thermal limits. MTP often hurts due to high temperature and unpredictable distributions. Long outputs expose fragmentation. - *Decision*: Disable MTP. Use `--no-mmap` if fragmentation is high. Allocate 75-80% GPU layers to leave headroom. Use Q4_K_M or Q5_K_S. Implement periodic cache clearing if VRAM creeps. **Cross-Workload Decision Matrix** | Metric Threshold | Action | |------------------|--------| | MTP acceptance < 0.6 | Disable MTP for that workload | | VRAM fragmentation > 0.4 | Reduce batch size, use q8_0 cache, or restart server | | TTFT p99/p50 ratio > 3.0 | Increase `--batch`, check CPU-GPU sync, reduce offload | | TPS drop > 30% over long gen | Lower `max_tokens`, improve cooling, reduce `--gpu-layers` | | OOM crashes | Reduce context window, switch to lower quant, increase VRAM headroom | Map your observed metrics to these thresholds. Don't optimize for one workload at the expense of system stability. Create separate config profiles per workload type and switch dynamically via OpenWebUI or proxy routing. # Common Mistakes Even experienced home-lab operators fall into these traps. Avoid them to ensure your weekend testing yields reliable, actionable data. **1. Skipping Warm-Up Phases** GPU inference engines compile kernels, allocate memory, and initialize caches on first use. Testing cold starts inflates TTFT and VRAM metrics. Always run 3-5 warm-up requests and discard the data. **2. Ignoring VRAM Fragmentation** Peak VRAM usage isn't the whole story. Fragmentation causes allocation failures even when free memory exists. Monitor fragmentation index and KV cache utilization. If fragmentation rises, reduce batch sizes or use contiguous cache types. **3. Misinterpreting MTP Acceptance Rate** A high acceptance rate doesn't guarantee speed gains if verification latency is high. Conversely, a moderate rate (0.6-0.7) can still yield net TPS improvements if draft generation is fast. Always measure effective TPS, not just acceptance. **4. Testing with Variable Prompts** Changing prompt length, complexity, or topic between runs introduces noise. Use identical prompts for baseline vs MTP comparisons. Tag workload types but keep inputs controlled. **5. Over-Optimizing Batch Sizes** Large `--batch` values speed up prompt processing but increase VRAM pressure and latency variance. Find the sweet spot where TTFT improves without triggering reallocation or fragmentation. **6. Ignoring Thermal Throttling** GPU clocks drop under sustained load, causing TPS decay. Monitor temperatures and clock speeds. If throttling occurs, improve cooling or reduce `--gpu-layers` temporarily. **7. Assuming Quantization is Neutral** Q4, Q5, and Q8 behave differently with MTP, KV cache, and long contexts. Lower quants increase hallucination rates and reduce MTP acceptance. Test quantization as a variable, not a constant. **8. Failing to Log System State** Driver updates, OS background tasks, and thermal events change performance. Log GPU driver version, OS load, temperature, and `llama.cpp` config hash alongside test data. **9. Chasing Peak TPS Over Consistency** Local LLMs are interactive systems. Consistent latency matters more than peak speed. Optimize for p95/p99 metrics, not maximums. **10. Not Validating Statistical Significance** Single-run comparisons are misleading. Run multiple iterations, calculate confidence intervals, and use effect sizes. If the difference isn't statistically significant, don't change your config. **11. Overlooking Proxy/Template Overhead** OpenWebUI templates, safety filters, and proxy buffering add latency. Bypass them during benchmarking using direct API calls. Re-enable them only after baseline metrics are established. **12. Ignoring Context Window Mismatch** If your prompt exceeds the model's trained context or your `--ctx-size` setting, performance degrades unpredictably. Align context windows with workload requirements. Avoid these mistakes, and your weekend testing will yield clear, reproducible insights. Document everything, trust the data, and iterate systematically. Your local LLM stack will become predictable, efficient, and tailored to your actual workloads.