# Quick Diagnosis When a local `llama.cpp` server feels inconsistent, the symptoms you described—variable first-token latency, unpredictable GPU memory jumps, and unclear MTP (Multi-Token Prediction / speculative decoding) benefits—are almost always rooted in three overlapping subsystems: KV cache management, GPU memory fragmentation, and speculative decoding overhead. Before you dive into weekend testing, here is a rapid diagnostic framework to isolate the root cause. ## 1. First-Token Latency (TTFT) Variability The first token latency is dominated by prompt processing, not token generation. In `llama.cpp`, prompt processing runs through the entire context window in a single forward pass (or batched passes if `--batch-size` is configured). Variability usually stems from: - **Cold vs. Warm GPU State:** The first request after boot or idle triggers GPU driver initialization, memory allocation, and cache warming. Subsequent requests benefit from resident VRAM and cached kernels. - **Context Window Flushing:** If OpenWebUI or your client sends full conversation history on every turn, the prompt length grows. `llama.cpp` must reprocess the entire growing context unless you use sliding window or KV cache retention. - **CPU Bottlenecks:** Prompt preprocessing (tokenization, embedding loading, draft model preparation) runs on CPU. If your CPU is starved or swapping, TTFT spikes. - **MTP Draft Model Loading:** If speculative decoding is enabled, the draft model must be loaded into VRAM. If it's not pre-loaded, the first request triggers a synchronous load, causing a massive TTFT spike. **Quick Check:** Run two identical prompts back-to-back. If the first takes 3-5 seconds and the second takes 0.5 seconds, you're seeing cold-start behavior. Add `--mlock` and `--no-mmap` to `llama.cpp` to force memory residency and eliminate OS paging delays. ## 2. GPU Memory Jumps VRAM usage in `llama.cpp` is not static. It fluctuates based on: - **KV Cache Allocation:** `llama.cpp` pre-allocates KV cache based on `--ctx-size` and `--batch-size`. If you exceed the allocated context, it triggers reallocation or fallback to CPU, causing spikes. - **Draft Model Overhead:** MTP/speculative decoding requires a second model (the draft model) in VRAM. If the draft model is large or not properly offloaded, VRAM jumps when MTP activates. - **GPU Memory Fragmentation:** Repeated allocation/deallocation of KV cache blocks can fragment VRAM. The driver reports "used" memory higher than actual tensor usage. - **Batch Processing:** If `--batch-size` is too small, `llama.cpp` processes prompts in multiple passes, increasing VRAM pressure per pass. If too large, it may exceed VRAM limits. **Quick Check:** Monitor `nvidia-smi` or your GPU dashboard during a long generation. If VRAM jumps at the start of generation and then stabilizes, it's KV cache allocation. If it jumps mid-generation, you're hitting context limits or draft model swapping. Use `--gpu-layers` to control offloading, and ensure `--ctx-size` matches your actual usage. ## 3. MTP (Speculative Decoding) Uncertainty MTP in `llama.cpp` works by using a smaller draft model to predict multiple tokens ahead, which the main model then verifies in parallel. The speedup depends entirely on the **acceptance rate** (how many draft tokens the main model accepts). If the acceptance rate is low, MTP adds overhead without benefit. **Quick Check:** Enable verbose logging (`-v` or `--verbose`) and look for `speculative` or `draft` metrics. If acceptance rate is below 40%, MTP is likely hurting performance. If above 60%, you should see 1.5x-2x generation speedup. VRAM jumps during MTP are normal if the draft model isn't pre-loaded. ## Immediate Diagnostic Commands Run these to establish a baseline before your weekend tests: ```bash # Check GPU memory allocation vs usage nvidia-smi --query-gpu=memory.used,memory.total,memory.free --format=csv -l 1 # Test cold vs warm TTFT curl http://localhost:8080/v1/completions \ -H "Content-Type: application/json" \ -d '{"model":"local","prompt":"Explain quantum computing in one sentence.","max_tokens":50,"temperature":0.7}' # Run again immediately curl http://localhost:8080/v1/completions \ -H "Content-Type: application/json" \ -d '{"model":"local","prompt":"Explain quantum computing in one sentence.","max_tokens":50,"temperature":0.7}' ``` Compare the `time` output. If the difference is >2x, you have a cold-start issue. If VRAM jumps >2GB between runs, you have draft model or KV cache allocation issues. --- # Data Collection Plan To scientifically evaluate your setup, you need structured, timestamped data across three layers: API requests, GPU/CPU metrics, and MTP/speculative events. Your Flight Recorder proxy, ServerTop dashboard, and SQLite backend will form the data pipeline. ## 1. Flight Recorder Proxy Configuration The proxy should intercept all OpenAI-compatible API calls from OpenWebUI, log request/response metadata, and forward to `llama.cpp`. Configure it to capture: - Request ID, timestamp, model name, prompt length, max_tokens, temperature, top_p - Response time (TTFT, total generation time, tokens generated) - MTP/speculative flags (enabled, draft model name, acceptance rate if exposed) - Error codes and retry counts **Proxy Logging Schema (SQLite):** ```sql CREATE TABLE IF NOT EXISTS api_requests ( id INTEGER PRIMARY KEY AUTOINCREMENT, request_id TEXT UNIQUE, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, model TEXT, prompt_length INTEGER, max_tokens INTEGER, temperature REAL, ttft_ms INTEGER, total_time_ms INTEGER, tokens_generated INTEGER, mtp_enabled INTEGER DEFAULT 0, mtp_acceptance_rate REAL, status_code INTEGER ); CREATE INDEX idx_timestamp ON api_requests(timestamp); CREATE INDEX idx_model ON api_requests(model); CREATE INDEX idx_mtp ON api_requests(mtp_enabled); ``` ## 2. ServerTop Dashboard Integration ServerTop should poll GPU/CPU metrics at 1-second intervals and write to SQLite. Configure it to capture: - GPU utilization (%), VRAM used/allocated (MB), temperature, power draw - CPU utilization (%), RAM used/available, swap usage - Disk I/O (if models are on HDD/SSD) - Network latency (if proxy adds overhead) **ServerTop Metrics Schema:** ```sql CREATE TABLE IF NOT EXISTS system_metrics ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, gpu_util_pct REAL, vram_used_mb INTEGER, vram_allocated_mb INTEGER, gpu_temp_c REAL, gpu_power_w REAL, cpu_util_pct REAL, ram_used_mb INTEGER, swap_used_mb INTEGER ); CREATE INDEX idx_metrics_timestamp ON system_metrics(timestamp); ``` ## 3. Synthetic Load Generation with curl To collect consistent data, you need controlled test prompts. Create a shell script that runs varied workloads: ```bash #!/bin/bash # load_test.sh BASE_URL="http://localhost:8080/v1/completions" MODEL="local" # Short prompt, short output curl -s $BASE_URL -H "Content-Type: application/json" \ -d "{\"model\":\"$MODEL\",\"prompt\":\"Summarize this: \",\"max_tokens\":100,\"temperature\":0.7}" \ -o /dev/null -w "Short: %{time_starttransfer}ms TTFT, %{time_total}ms total\n" # Long prompt, long output curl -s $BASE_URL -H "Content-Type: application/json" \ -d "{\"model\":\"$MODEL\",\"prompt\":\"$(cat long_context.txt)\",\"max_tokens\":2000,\"temperature\":0.7}" \ -o /dev/null -w "Long: %{time_starttransfer}ms TTFT, %{time_total}ms total\n" # Reasoning stress test curl -s $BASE_URL -H "Content-Type: application/json" \ -d "{\"model\":\"$MODEL\",\"prompt\":\"Solve step-by-step: A train leaves station A at 60mph. Another leaves station B at 80mph. They are 420 miles apart. When do they meet?\",\"max_tokens\":500,\"temperature\":0.2}" \ -o /dev/null -w "Reasoning: %{time_starttransfer}ms TTFT, %{time_total}ms total\n" ``` Run this script 10-20 times per configuration. Log the output to a CSV for later import into SQLite. ## 4. Data Collection Workflow 1. Start `llama.cpp` with your baseline config. 2. Start Flight Recorder proxy pointing to `llama.cpp`. 3. Start ServerTop polling at 1Hz. 4. Run `load_test.sh` in a loop (use `for i in {1..20}; do ./load_test.sh; sleep 5; done`). 5. Stop services, export SQLite database. 6. Repeat for each configuration change (MTP on/off, different `--gpu-layers`, different `--ctx-size`). This gives you a clean, timestamped dataset for statistical analysis. --- # MTP Comparison Plan MTP (speculative decoding) is not a free lunch. It trades VRAM and CPU overhead for potential generation speedup. To compare MTP vs non-MTP without fooling yourself, you need a rigorous A/B testing protocol. ## 1. Understanding MTP in llama.cpp `llama.cpp` implements speculative decoding using a draft model. The main model verifies draft tokens in parallel. Key parameters: - `--speculative`: Number of draft tokens to generate per step - `--draft-model`: Path to the draft model (usually smaller, e.g., 1B-3B) - `--max-speculative`: Maximum draft tokens allowed - `--draft-penalty`: Penalty for rejected draft tokens (prevents infinite loops) **Speedup Formula:** `Effective Tokens/Sec = Base Tokens/Sec * (1 + (Acceptance Rate * Draft Tokens))` If acceptance rate is 50% and draft tokens = 4, you get ~3x speedup. If acceptance rate is 20%, you get ~0.8x (slower). ## 2. A/B Test Design - **Control Group:** MTP disabled (`--speculative 0` or omit flags) - **Test Group:** MTP enabled (`--speculative 4 --draft-model path/to/draft.gguf`) - **Constants:** Same prompt, same seed (`--seed 42`), same GPU/CPU state, same VRAM allocation, same OpenWebUI settings - **Warm-up:** Run 3 dummy prompts before recording data to eliminate cold-start bias - **Runs:** Minimum 10 runs per configuration, randomized order to avoid thermal throttling bias ## 3. Metrics to Track - **TTFT (ms):** Should be similar or slightly higher with MTP (draft model loading) - **Tokens/Second:** Primary speedup metric - **Acceptance Rate:** % of draft tokens accepted by main model - **VRAM Overhead:** Draft model size + KV cache for draft steps - **CPU Utilization:** Draft model runs on CPU/GPU depending on offload ## 4. Statistical Validation Do not cherry-pick the fastest run. Use: - Mean and median tokens/sec - 95th percentile latency - Standard deviation (consistency matters more than peak speed) - Paired t-test or Wilcoxon signed-rank test if you want formal significance **Example Comparison Table:** | Metric | MTP Off | MTP On | Delta | |--------|---------|--------|-------| | TTFT (ms) | 450 | 520 | +15% | | Tokens/Sec | 28.5 | 41.2 | +44% | | Acceptance Rate | N/A | 62% | - | | VRAM Used (MB) | 6200 | 7800 | +26% | | CPU Util (%) | 12 | 34 | +183% | If acceptance rate < 40%, disable MTP. If > 60% and VRAM allows, keep it. ## 5. Avoiding Self-Deception - **Do not** test MTP on short prompts (<50 tokens). Speedup is negligible. - **Do not** ignore VRAM fragmentation. MTP can cause allocation spikes that trigger CPU fallback. - **Do not** assume higher tokens/sec means better quality. Verify output coherence. - **Do** test with your actual workload (coding, RAG, chat). Synthetic prompts misrepresent acceptance rates. --- # Reasoning Budget Plan "Reasoning budget" refers to how much compute you allocate to a model versus the quality of its output. In local LLMs, this is a trade-off between context size, GPU offloading, batch size, and temperature/top_p settings. ## 1. Budget Dimensions - **Context Size (`--ctx-size`):** Larger context = more VRAM, slower TTFT, better long-form reasoning - **GPU Layers (`--gpu-layers`):** More layers on GPU = faster generation, higher VRAM, potential fragmentation - **Batch Size (`--batch-size`):** Larger batch = faster prompt processing, higher VRAM per pass - **Temperature/Top_P:** Lower = more deterministic, better for coding/RAG. Higher = more creative, worse for precision. ## 2. Test Matrix Run your model with these configurations and log performance: | Config | ctx-size | gpu-layers | batch-size | temp | Use Case | |--------|----------|------------|------------|------|----------| | A | 4096 | 35 | 512 | 0.2 | Coding/RAG | | B | 8192 | 35 | 1024 | 0.3 | Agentic/Tool Use | | C | 4096 | 20 | 256 | 0.7 | Chat/Creative | | D | 16384 | 35 | 2048 | 0.5 | Long Context | ## 3. Reasoning Stress Prompts Use these to evaluate quality vs speed: - **Coding:** "Write a Python function to parse JSON with nested arrays, handle missing keys gracefully, and return a typed dataclass. Include type hints and docstrings." - **RAG:** "Based on the following context: [paste 2k tokens], answer: What are the three main limitations of transformer attention mechanisms? Cite specific sections." - **Agentic:** "You have access to tools: search_web, read_file, execute_code. Plan steps to find the latest CVE for OpenSSL, verify it, and generate a patch recommendation." - **Creative:** "Write a 500-word sci-fi microstory about a AI that discovers it's running on a quantum computer. Use vivid sensory details and a twist ending." ## 4. Budget Decision Framework - **If TTFT > 2s:** Reduce `--ctx-size` or increase `--batch-size` - **If VRAM > 90%:** Reduce `--gpu-layers` or enable `--mlock` - **If Tokens/Sec < 15:** Increase `--gpu-layers` or enable MTP - **If Output Quality Drops:** Lower temperature, increase `--ctx-size`, or switch to a larger model Track quality using a simple rubric (1-5) for accuracy, coherence, and instruction following. Plot quality vs speed to find your Pareto frontier. --- # Long Output Test Plan Long outputs stress KV cache management, VRAM stability, and generation consistency. This test reveals degradation patterns that short prompts hide. ## 1. Test Setup - **Prompts:** 3 categories: narrative (story), technical (code/docs), analytical (report) - **Lengths:** 1k, 2k, 4k, 8k tokens - **Metrics:** Tokens/sec over time, VRAM growth, repetition rate, hallucination markers - **Tools:** `llama.cpp` verbose logging, ServerTop VRAM tracking, custom repetition detector ## 2. Execution Script ```bash #!/bin/bash # long_output_test.sh BASE_URL="http://localhost:8080/v1/completions" MODEL="local" for length in 1000 2000 4000 8000; do echo "Testing $length tokens..." curl -s $BASE_URL -H "Content-Type: application/json" \ -d "{\"model\":\"$MODEL\",\"prompt\":\"Write a detailed technical analysis of distributed consensus algorithms. Cover Paxos, Raft, and ZAB. Compare their fault tolerance, latency, and use cases. Do not stop until you have covered all aspects thoroughly.\",\"max_tokens\":$length,\"temperature\":0.3,\"stream\":false}" \ -o /dev/null -w "Length=$length TTFT=%{time_starttransfer}ms Total=%{time_total}ms\n" sleep 10 done ``` ## 3. Degradation Metrics - **Throughput Curve:** Plot tokens/sec vs token index. Healthy: flat or slight decline. Unhealthy: sharp drop at 2k/4k tokens. - **VRAM Growth:** Should be linear with context. Spikes indicate reallocation or draft model swapping. - **Repetition Rate:** Count repeated n-grams (3-5 words). >5% indicates KV cache corruption or temperature drift. - **Hallucination Markers:** Look for fabricated citations, inconsistent facts, or broken syntax in code. ## 4. Optimization Levers - If throughput drops at 2k: Increase `--batch-size` or reduce `--ctx-size` - If VRAM spikes: Enable `--mlock`, reduce `--gpu-layers`, or use `--no-mmap` - If repetition increases: Lower temperature, increase `--repeat-penalty`, or switch to a model with better long-context training Log all runs in SQLite. Compare degradation curves across configurations. --- # SQLite Queries Your SQLite database is the single source of truth. Use these queries to extract actionable insights. ## 1. Latency Percentiles ```sql -- 50th, 90th, 95th percentile TTFT SELECT percentile_cont(0.50) WITHIN GROUP (ORDER BY ttft_ms) AS p50_ttft, percentile_cont(0.90) WITHIN GROUP (ORDER BY ttft_ms) AS p90_ttft, percentile_cont(0.95) WITHIN GROUP (ORDER BY ttft_ms) AS p95_ttft FROM api_requests WHERE model = 'local' AND status_code = 200; ``` ## 2. MTP Performance Comparison ```sql -- Compare tokens/sec and acceptance rate with/without MTP SELECT mtp_enabled, COUNT(*) AS runs, AVG(tokens_generated * 1000.0 / total_time_ms) AS avg_tokens_per_sec, AVG(mtp_acceptance_rate) AS avg_acceptance_rate, STDDEV(tokens_generated * 1000.0 / total_time_ms) AS stddev_tps FROM api_requests WHERE model = 'local' AND status_code = 200 GROUP BY mtp_enabled; ``` ## 3. VRAM Trends Over Time ```sql -- Average VRAM usage per 10-second bucket during test window SELECT strftime('%H:%M:%S', timestamp) AS time_bucket, AVG(vram_used_mb) AS avg_vram_mb, MAX(vram_used_mb) AS peak_vram_mb FROM system_metrics WHERE timestamp BETWEEN '2024-06-15 10:00:00' AND '2024-06-15 12:00:00' GROUP BY time_bucket ORDER BY time_bucket; ``` ## 4. Long Output Degradation ```sql -- Tokens/sec by request ID for long outputs SELECT request_id, tokens_generated, total_time_ms, (tokens_generated * 1000.0 / total_time_ms) AS tokens_per_sec, ttft_ms FROM api_requests WHERE max_tokens >= 2000 AND status_code = 200 ORDER BY tokens_per_sec ASC; ``` ## 5. Error & Retry Analysis ```sql -- Status code distribution and retry patterns SELECT status_code, COUNT(*) AS occurrences, AVG(total_time_ms) AS avg_time_ms FROM api_requests GROUP BY status_code ORDER BY occurrences DESC; ``` ## 6. Correlation: VRAM vs Latency ```sql -- Join API requests with nearest system metric SELECT a.request_id, a.ttft_ms, a.tokens_per_sec, s.vram_used_mb, s.gpu_util_pct FROM api_requests a JOIN system_metrics s ON s.timestamp = ( SELECT timestamp FROM system_metrics WHERE timestamp <= a.timestamp ORDER BY timestamp DESC LIMIT 1 ) WHERE a.model = 'local' AND a.status_code = 200; ``` Run these queries after each test phase. Export results to CSV for plotting in Python/R or your dashboard. --- # Dashboard Screenshots To Capture ServerTop and Flight Recorder should visualize your data. Capture these specific charts for your weekend report. ## 1. TTFT Distribution Histogram - **Metric:** `ttft_ms` from `api_requests` - **Chart Type:** Histogram with bins of 100ms - **What to Look For:** Bimodal distribution indicates cold/warm state mixing. Single peak <500ms is healthy. Tails >2s indicate bottlenecks. - **Screenshot Note:** Label axes clearly. Annotate p50, p95 lines. ## 2. Token Generation Rate Over Time - **Metric:** `tokens_generated / total_time_ms` per request - **Chart Type:** Line chart with request index on X-axis - **What to Look For:** Flat line = consistent performance. Downward slope = VRAM pressure or thermal throttling. Spikes = MTP acceptance bursts. - **Screenshot Note:** Overlay MTP on/off runs with different colors. ## 3. VRAM Used vs Allocated - **Metric:** `vram_used_mb` vs `vram_allocated_mb` from `system_metrics` - **Chart Type:** Dual-axis line chart - **What to Look For:** Gap between used and allocated indicates fragmentation. Sudden jumps = KV cache reallocation or draft model loading. - **Screenshot Note:** Mark test phases (baseline, MTP, long output). ## 4. MTP Acceptance Rate Scatter - **Metric:** `mtp_acceptance_rate` vs `tokens_per_sec` - **Chart Type:** Scatter plot with trend line - **What to Look For:** Positive correlation = MTP helping. Flat/negative = overhead dominating. Cluster around 0.6-0.8 = optimal. - **Screenshot Note:** Add regression line and R² value. ## 5. CPU/GPU Utilization Heatmap - **Metric:** `cpu_util_pct` and `gpu_util_pct` over time - **Chart Type:** Stacked area or heatmap - **What to Look For:** GPU >80% during generation = good offloading. CPU >50% during prompt processing = bottleneck. Both low = idle or throttling. - **Screenshot Note:** Highlight prompt processing vs generation phases. ## 6. Error Rate & Retry Timeline - **Metric:** `status_code` != 200 count per minute - **Chart Type:** Bar chart - **What to Look For:** Spikes correlate with VRAM limits or timeout settings. Consistent 0 = stable. - **Screenshot Note:** Annotate configuration changes. Capture these at the end of each test phase. They form your visual evidence base. --- # Weekend Checklist Follow this chronological plan to maximize your weekend. Adjust times based on your schedule. ## Saturday Morning: Baseline & Setup - [ ] Install/verify `llama.cpp` (latest release), OpenWebUI, Flight Recorder proxy, ServerTop - [ ] Configure SQLite schema (run provided CREATE TABLE statements) - [ ] Set up proxy to intercept OpenWebUI -> llama.cpp traffic - [ ] Configure ServerTop polling at 1Hz, writing to SQLite - [ ] Run cold/warm TTFT test (2 identical prompts) - [ ] Log baseline VRAM, CPU, GPU metrics - [ ] Capture baseline dashboard screenshots ## Saturday Afternoon: MTP Comparison - [ ] Prepare draft model (download 1B-3B GGUF, verify compatibility) - [ ] Run A/B test: 10 runs MTP off, 10 runs MTP on (randomized order) - [ ] Log acceptance rate, tokens/sec, VRAM overhead - [ ] Run SQLite MTP comparison query - [ ] Capture MTP scatter plot and VRAM trend - [ ] Decide: keep MTP on/off based on acceptance rate >40% threshold ## Saturday Evening: Long Output Stress - [ ] Run long output test (1k, 2k, 4k, 8k tokens) - [ ] Monitor VRAM growth and throughput degradation - [ ] Log repetition rate and hallucination markers - [ ] Run degradation SQLite query - [ ] Capture throughput curve and VRAM chart - [ ] Tune `--batch-size` or `--ctx-size` if degradation >20% ## Sunday Morning: Reasoning Budget & Workload Mapping - [ ] Run test matrix (Configs A-D from Reasoning Budget Plan) - [ ] Evaluate quality vs speed for coding, RAG, agentic, chat, creative - [ ] Score outputs (1-5 rubric) - [ ] Run correlation query (VRAM vs latency) - [ ] Capture utilization heatmap and error timeline - [ ] Identify Pareto frontier configuration ## Sunday Afternoon: Analysis & Optimization - [ ] Compile all SQLite query results - [ ] Generate final dashboard screenshots - [ ] Write summary report: best config per workload, MTP verdict, VRAM stability - [ ] Apply final `llama.cpp` flags to production config - [ ] Document lessons learned and next steps ## Sunday Evening: Validation - [ ] Run final validation suite (5 prompts per workload) - [ ] Verify TTFT <1s, tokens/sec >20, VRAM stable - [ ] Archive SQLite database and screenshots - [ ] Celebrate a stable, optimized local AI stack --- # Interpreting Results Your data will tell you exactly how to configure your stack for each workload. Use this decision matrix. ## 1. Coding - **Ideal Metrics:** TTFT <800ms, tokens/sec >25, acceptance rate >50% (if MTP), VRAM stable - **Config:** `--ctx-size 8192`, `--gpu-layers 35`, `--batch-size 1024`, temp 0.2, top_p 0.9 - **Why:** Needs deterministic output, good context for codebases, fast generation for IDE integration - **Red Flags:** High repetition, broken syntax, slow TTFT on long prompts ## 2. RAG (Retrieval-Augmented Generation) - **Ideal Metrics:** TTFT <500ms, tokens/sec >30, low VRAM fragmentation, high accuracy on context - **Config:** `--ctx-size 4096`, `--gpu-layers 30`, `--batch-size 512`, temp 0.1, top_p 0.8 - **Why:** Prompt includes retrieved context. Fast TTFT critical for user experience. Low temperature prevents hallucination. - **Red Flags:** Ignoring context, fabricating citations, VRAM spikes on large chunks ## 3. Agentic Work (Tool Use, Planning) - **Ideal Metrics:** TTFT <600ms, tokens/sec >20, consistent formatting, low error rate - **Config:** `--ctx-size 8192`, `--gpu-layers 35`, `--batch-size 1024`, temp 0.3, top_p 0.95 - **Why:** Needs reliable JSON/tool-call formatting, moderate creativity for planning, stable VRAM for multi-step loops - **Red Flags:** Broken JSON, infinite loops, high retry rate, VRAM leaks ## 4. Chat - **Ideal Metrics:** TTFT <400ms, tokens/sec >35, conversational flow, low latency - **Config:** `--ctx-size 4096`, `--gpu-layers 25`, `--batch-size 256`, temp 0.7, top_p 0.9 - **Why:** Short prompts, fast response critical. Lower GPU layers save VRAM for concurrent sessions. - **Red Flags:** Robotic tone, slow first token, VRAM fragmentation from many short sessions ## 5. Creative Writing - **Ideal Metrics:** TTFT <1s, tokens/sec >25, low repetition, high coherence over 2k+ tokens - **Config:** `--ctx-size 8192`, `--gpu-layers 35`, `--batch-size 512`, temp 0.8, top_p 0.95, repeat_penalty 1.1 - **Why:** Needs longer context for narrative consistency, higher temperature for creativity, repeat penalty to avoid loops - **Red Flags:** Repetitive phrases, plot holes, slow generation on long outputs ## Decision Thresholds - **TTFT > 2s:** Reduce context, increase batch, check CPU bottleneck - **Tokens/Sec < 15:** Increase GPU layers, enable MTP (if acceptance >40%), check thermal throttling - **VRAM > 90%:** Reduce GPU layers, enable mlock, reduce context - **Acceptance Rate < 40%:** Disable MTP, switch draft model, or accept baseline speed - **Repetition > 5%:** Increase repeat_penalty, lower temperature, check KV cache integrity Plot your results on a 2D graph: X-axis = TTFT, Y-axis = Tokens/Sec. Quadrants reveal your strengths. Top-left = ideal. Bottom-right = needs tuning. --- # Common Mistakes Avoid these pitfalls to ensure your weekend testing yields actionable results. ## 1. Ignoring Warm-Up Phase **Mistake:** Recording data from the first request after boot. **Fix:** Always run 3-5 dummy prompts before logging. GPU drivers, memory allocators, and caches need time to stabilize. ## 2. Misinterpreting MTP Acceptance Rate **Mistake:** Assuming higher draft tokens = faster generation. **Fix:** Speedup depends on acceptance rate. If <40%, MTP adds overhead. Test with your actual prompts, not synthetic ones. ## 3. Conflating VRAM Used vs Allocated **Mistake:** Panicking when `nvidia-smi` shows high "used" memory. **Fix:** `llama.cpp` pre-allocates KV cache. "Used" includes reserved blocks. Monitor actual tensor usage via verbose logging or ServerTop. ## 4. Over-Indexing on First-Token Latency **Mistake:** Optimizing TTFT at the expense of generation speed. **Fix:** TTFT matters for chat/RAG. Generation speed matters for long outputs. Balance based on workload. ## 5. Not Controlling for System Load **Mistake:** Running tests while other apps consume CPU/GPU. **Fix:** Close browsers, IDEs, background services. Use `nice`/`taskset` to isolate `llama.cpp` if needed. ## 6. Ignoring Thermal Throttling **Mistake:** Assuming consistent performance over long tests. **Fix:** Monitor GPU/CPU temps. If >85°C, performance drops. Improve cooling or reduce clock speeds. ## 7. Using Incompatible Draft Models **Mistake:** Loading a draft model with different architecture/tokenizer. **Fix:** Draft model must share tokenizer and architecture family with main model. Verify compatibility before testing. ## 8. Cherry-Picking Fast Runs **Mistake:** Reporting the best TTFT or tokens/sec. **Fix:** Use median and p95. Report standard deviation. Consistency matters more than peak performance. ## 9. Forgetting KV Cache Limits **Mistake:** Setting `--ctx-size` higher than VRAM allows. **Fix:** Calculate KV cache size: `ctx_size * layers * hidden_dim * 2 * 4 bytes`. Stay within 70% VRAM limit. ## 10. Neglecting SQLite Indexing **Mistake:** Running slow queries on unindexed tables. **Fix:** Add indexes on `timestamp`, `model`, `mtp_enabled`. Vacuum database after large imports. ## 11. Assuming Linear Scaling **Mistake:** Expecting 2x VRAM = 2x speed. **Fix:** Performance scales sublinearly due to memory bandwidth, PCIe limits, and CPU bottlenecks. Test empirically. ## 12. Ignoring OpenWebUI Overhead **Mistake:** Blaming `llama.cpp` for latency caused by UI streaming or proxy. **Fix:** Test directly via `curl` first. Then add OpenWebUI. Isolate bottlenecks. ## 13. Not Logging Draft Model VRAM **Mistake:** Forgetting that draft model consumes VRAM separately. **Fix:** Monitor total VRAM = main model + draft model + KV cache. Plan accordingly. ## 14. Overlooking Temperature Drift **Mistake:** Assuming temperature affects only creativity. **Fix:** High temperature increases KV cache variance, potentially causing fragmentation. Test with your actual temp settings. ## 15. Skipping Validation After Tuning **Mistake:** Applying new flags without retesting. **Fix:** Always run validation suite after changes. Document before/after metrics. By avoiding these mistakes, your weekend testing will yield reliable, reproducible results. Your local AI stack will be optimized for your specific workloads, not generic benchmarks. --- This guide provides a complete, actionable framework for diagnosing, testing, and optimizing your local `llama.cpp` setup. Follow the checklist, run the queries, capture the dashboards, and apply the decision matrix. Your weekend will transform an inconsistent server into a predictable, workload-optimized AI lab. # Advanced Configuration Tuning Once baseline stability is achieved, fine-tuning requires surgical adjustments to `llama.cpp` flags. These parameters interact non-linearly, so change only one at a time and re-run your validation suite. ## KV Cache Quantization By default, `llama.cpp` stores KV cache in FP16. Enabling `--cache-type-k f16` or `--cache-type-v f16` can reduce VRAM pressure by 30-40% with minimal quality loss for most 7B-13B models. For extreme VRAM constraints, `--cache-type-k q8_0` and `--cache-type-v q8_0` offer further savings but may degrade long-context coherence. Test with your reasoning prompts; if hallucination markers increase, revert to FP16. ## Flash Attention & Paged Attention `llama.cpp` supports `--flash-attn` for supported architectures. This reduces memory bandwidth usage during attention computation, directly improving tokens/sec. Combine with `--no-kv-offload` if you encounter VRAM fragmentation. For models with rotary embeddings, `--rope-scaling` and `--rope-freq-base` adjustments can extend context without retraining. Document exact values in SQLite for reproducibility. ## Thread Affinity & CPU Binding Prompt processing is CPU-bound. Use `--threads` to match physical cores, not logical. Enable `--threads-batch` for parallel prompt chunking. On Linux, bind `llama.cpp` to specific NUMA nodes: ```bash numactl --cpunodebind=0 --membind=0 ./server -m model.gguf ... ``` This prevents cache thrashing and reduces TTFT variance by up to 40%. Monitor with `htop` to ensure no core exceeds 95% utilization during prompt ingestion. ## Draft Model Selection Criteria MTP performance hinges on draft model compatibility. Ideal candidates: - Same tokenizer family (e.g., Llama-3, Mistral, Qwen) - 1/3 to 1/5 parameter size of main model - Trained on similar data distribution - GGUF quantization matching main model (Q4_K_M or Q5_K_M) Test acceptance rates across your workload matrix. If coding prompts yield <30% acceptance, switch to a code-specialized draft model (e.g., CodeLlama-1B). Log draft model hashes in SQLite to track performance drift. --- # OpenWebUI Integration Optimizations OpenWebUI adds abstraction layers that can mask `llama.cpp` bottlenecks. Configure it to expose raw metrics and reduce overhead. ## Streaming & Chunk Size OpenWebUI defaults to SSE streaming with small chunks. Increase `--chunk-size` in OpenWebUI settings to 16-32 tokens. This reduces HTTP overhead and improves perceived latency. Disable `--stream` in curl tests to isolate generation speed from UI rendering. ## Context Window Management OpenWebUI retains full conversation history by default. Enable `--context-window-size` limit in OpenWebUI settings to match `--ctx-size`. Implement sliding window or summarization for long sessions. Monitor `prompt_length` in SQLite; if it exceeds 70% of `--ctx-size`, TTFT will degrade non-linearly. ## Model Routing & Fallback Configure OpenWebUI to route workloads to optimized profiles: - Coding: `--temp 0.2 --top_p 0.9 --repeat_penalty 1.1` - RAG: `--temp 0.1 --top_p 0.8 --ctx-size 4096` - Creative: `--temp 0.8 --top_p 0.95 --repeat_penalty 1.2` Use OpenWebUI's model presets to switch profiles without restarting `llama.cpp`. Log preset usage in SQLite to analyze workload distribution. ## Websocket vs HTTP For agentic loops, switch OpenWebUI to websocket mode. This eliminates HTTP handshake overhead per tool call. Monitor `status_code` and `total_time_ms` in SQLite; websocket should reduce latency by 15-25% for multi-step agents. --- # Automated Benchmarking Script Manual testing scales poorly. Deploy this Python script to automate data collection, SQLite insertion, and statistical analysis. ```python import requests import sqlite3 import time import json import statistics DB_PATH = "llama_bench.db" BASE_URL = "http://localhost:8080/v1/completions" def init_db(): conn = sqlite3.connect(DB_PATH) c = conn.cursor() c.execute("""CREATE TABLE IF NOT EXISTS benchmarks ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, workload TEXT, prompt_len INTEGER, max_tokens INTEGER, ttft_ms REAL, total_ms REAL, tokens_gen INTEGER, tps REAL, mtp_enabled INTEGER, config_hash TEXT )""") conn.commit() return conn def run_benchmark(workload, prompt, max_tokens, mtp_enabled, config_hash): payload = { "model": "local", "prompt": prompt, "max_tokens": max_tokens, "temperature": 0.3 if workload in ["coding", "rag"] else 0.7, "stream": False } start = time.time() resp = requests.post(BASE_URL, json=payload) end = time.time() data = resp.json() tokens = data.get("usage", {}).get("completion_tokens", 0) ttft = data.get("timing", {}).get("ttft_ms", 0) total = (end - start) * 1000 tps = tokens / (total / 1000) if total > 0 else 0 conn = init_db() c = conn.cursor() c.execute("""INSERT INTO benchmarks (workload, prompt_len, max_tokens, ttft_ms, total_ms, tokens_gen, tps, mtp_enabled, config_hash) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""", (workload, len(prompt), max_tokens, ttft, total, tokens, tps, mtp_enabled, config_hash)) conn.commit() conn.close() return tps # Workload prompts PROMPTS = { "coding": "Write a Python async function to fetch and parse JSON from a URL, handle timeouts, and return typed dataclasses.", "rag": "Based on the provided context, extract three key findings about transformer attention mechanisms. Cite specific sections.", "agentic": "Plan steps to search for CVE-2024-1234, verify impact, and generate a mitigation script. Use JSON tool calls.", "creative": "Write a 300-word sci-fi scene about a quantum AI discovering parallel simulations. Use vivid sensory details." } if __name__ == "__main__": for wl, prompt in PROMPTS.items(): for mtp in [0, 1]: for _ in range(5): tps = run_benchmark(wl, prompt, 500, mtp, "cfg_v1") print(f"{wl} MTP={mtp} TPS={tps:.2f}") time.sleep(2) ``` Run this script after each configuration change. It populates SQLite with statistically valid samples. Use the provided SQL queries to generate reports. --- # Memory Profiling & Fragmentation Deep Dive VRAM fragmentation is the silent killer of local LLM performance. `llama.cpp` allocates KV cache in contiguous blocks. Repeated context flushing and draft model loading fragment these blocks, causing allocation failures and CPU fallback. ## Diagnostic Commands ```bash # Monitor fragmentation ratio nvidia-smi --query-gpu=memory.used,memory.total,memory.free --format=csv -l 1 | awk -F, '{print $1, $2, $3, ($1/$2)*100}' # Check allocation patterns valgrind --tool=massif ./server -m model.gguf --ctx-size 8192 --batch-size 1024 ``` If fragmentation ratio exceeds 15%, enable `--mlock` and `--no-mmap`. These force memory residency and prevent OS paging. For persistent fragmentation, implement KV cache pooling: allocate fixed blocks upfront and reuse them across requests. ## Draft Model Memory Isolation MTP draft models should run on separate GPU memory pools if possible. Use `--gpu-layers` to offload main model layers, and keep draft model entirely on CPU if VRAM is constrained. This prevents draft model allocation from fragmenting main model KV cache. Monitor `vram_allocated_mb` vs `vram_used_mb` in ServerTop; a growing gap indicates fragmentation. ## Context Flushing Strategy Implement explicit context flushing in OpenWebUI or proxy layer. Instead of growing context indefinitely, truncate to last N tokens or summarize history. Log `prompt_length` trends in SQLite. If average prompt length exceeds 60% of `--ctx-size`, TTFT will degrade. Implement sliding window with `--ctx-size` matching your actual usage, not theoretical maximum. --- # Production Hardening & Monitoring Transitioning from lab to production requires robust monitoring, alerting, and failover mechanisms. ## Health Checks & Auto-Restart Deploy a systemd service with health checks: ```ini [Unit] Description=llama.cpp Server After=network.target [Service] ExecStart=/usr/local/bin/server -m /models/main.gguf --ctx-size 8192 --gpu-layers 35 --batch-size 1024 --mlock Restart=always RestartSec=5 ExecStartPre=/usr/bin/curl -f http://localhost:8080/health || exit 1 [Install] WantedBy=multi-user.target ``` Configure `systemd` to restart on VRAM OOM or HTTP 503 errors. Log restart events to SQLite for trend analysis. ## Alerting Thresholds Set up monitoring alerts based on SQLite metrics: - TTFT p95 > 2000ms: Alert on CPU bottleneck or context overflow - Tokens/sec < 15: Alert on GPU throttling or MTP misconfiguration - VRAM used > 90%: Alert on fragmentation or context limit breach - MTP acceptance < 40%: Alert on draft model mismatch or workload shift Use cron jobs to run SQLite queries and trigger alerts via email/webhook. ## Backup & Rollback Strategy Version control all configuration files, draft models, and SQLite schemas. Maintain a rollback script: ```bash #!/bin/bash # rollback.sh cp /etc/llama.cpp/server.conf.bak /etc/llama.cpp/server.conf systemctl restart llama-server sqlite3 llama_bench.db ".dump" > backup_$(date +%F).sql ``` Test rollback procedures weekly. Document configuration hashes in SQLite to enable precise state restoration. --- # Final Validation Protocol Before declaring your setup production-ready, run this comprehensive validation suite. ## 1. Cold/Warm Consistency Run 10 identical prompts. TTFT variance should be <15%. If higher, enable `--mlock`, `--no-mmap`, and verify NUMA binding. ## 2. MTP Acceptance Stability Run 20 prompts across all workloads. Acceptance rate should be consistent (±5%). If volatile, switch draft model or disable MTP for that workload. ## 3. Long Context Degradation Generate 8k tokens. Throughput should degrade <20%. VRAM should remain stable. If degradation exceeds threshold, increase `--batch-size` or reduce `--ctx-size`. ## 4. Error Rate Baseline Run 100 requests. Error rate should be <1%. If higher, check timeout settings, VRAM limits, and proxy configuration. ## 5. Quality Rubric Scoring Score outputs on accuracy, coherence, instruction following, and formatting. Minimum score: 4/5 for coding/RAG, 3.5/5 for creative. If scores drop, adjust temperature, top_p, or repeat_penalty. ## 6. Dashboard Verification Capture all required screenshots. Verify trends match SQLite data. Annotate anomalies. Archive for audit trail. ## 7. Documentation & Handoff Compile configuration files, SQL schemas, curl examples, and dashboard links. Create a runbook for operators. Include troubleshooting steps for common failures. By following this protocol, you transform experimental tuning into repeatable, auditable optimization. Your local AI stack will deliver predictable performance across coding, RAG, agentic, chat, and creative workloads. The data-driven approach eliminates guesswork, ensuring every flag, model, and configuration serves a measurable purpose. Weekend testing becomes a foundation for continuous improvement, not a one-time fix.