# Home-Lab AI Server Benchmark Infrastructure Runbook **Agent Persona:** Cautious Server-Admin Agent **Environment:** AMD Radeon AI PRO R9700 (32GB VRAM) | llama-cpp-server-vulkan | Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf **Objective:** Maintain high-fidelity benchmark integrity, ensure data privacy, and manage complex MTP/Reasoning configurations without system instability. --- # Operating Principles As a cautious server-admin agent, all actions must adhere to the following four pillars of operation: 1. **Data Sovereignty & Privacy:** No raw data, model weights, or benchmark outputs are to be transmitted outside the local network (192.168.1.116). Screenshots are permitted for human review only. The SQLite database on `/models` is a private asset. 2. **Incrementalism:** Never perform bulk benchmark runs. Every test must be validated individually. If a test fails, the agent must stop, diagnose, and fix before proceeding to the next task. 3. **Observability First:** Every action must be preceded by a state check and followed by a verification of the result. Logs must be captured at every stage of the pipeline. 4. **Fail-Safe Rollbacks:** No configuration change is "permanent" until it has survived a validation cycle. Every change must have a corresponding "Undo" command or configuration snapshot. --- # Preflight Checks Before initiating any benchmark or infrastructure modification, the following checks must be performed to ensure the environment is stable. ### 1. Network & Connectivity Verify that the proxy and dashboard are reachable. - `curl -I http://192.168.1.116:8181/v1` [Read-only] - `curl -I http://192.168.1.116:8090` [Read-only] - *Success Criteria:* HTTP 200 OK or 401 Unauthorized (indicating the service is up but protected). ### 2. Container & Model Status Ensure the `llama-cpp-server-vulkan` container is running and the model is loaded into VRAM. - `docker ps --filter "ancestor=llama-cpp-server-vulkan:working-20260613"` [Read-only] - `docker logs --tail 50 ` [Read-only] - *Success Criteria:* Logs should show "Model loaded" and "Vulkan device initialized." ### 3. Hardware & VRAM Availability Check the AMD Radeon AI PRO R9700 status. - `rocm-smi` (or `radeontop` if ROCm is not fully mapped) [Read-only] - `vulkaninfo --summary` [Read-only] - *Success Criteria:* VRAM usage should be below 30GB (leaving headroom for the OS and context window). ### 4. Database Integrity Verify the SQLite database on `/models`. - `ls -lh /models` [Read-only] - `sqlite3 /models/database.db "SELECT count(*) FROM benchmarks;"` [Read-only] - *Success Criteria:* Database is accessible and contains expected records. --- # Safe Inspection Commands Use these commands to gather telemetry without altering the state of the system. | Command | Risk Level | Purpose | | :--- | :--- | :--- | | `docker stats` | Read-only | Monitor real-time CPU/Memory/Network usage of the llama.cpp container. | | `docker logs -f --tail 100 ` | Read-only | Stream live logs to identify immediate crashes or OOM errors. | | `curl -X GET "http://192.168.1.116:8181/v1/models" -H "Authorization: Bearer "` | Read-only | Verify the API sees the Qwen3.6 model correctly. | | `sqlite3 /models/database.db ".tables"` | Read-only | Verify schema integrity of the local database. | | `watch -n 1 "rocm-smi --showmeminfo"` | Read-only | Monitor VRAM fluctuations during inference. | | `netstat -tulpn | grep 8181` | Read-only | Ensure the Flight Recorder proxy is bound to the correct interface. | --- # Reasoning Budget Verification The `reasoning-budget` is set to 8192. We must ensure the model does not truncate the reasoning phase prematurely or bleed into the final content incorrectly. ### Verification Protocol: 1. **Trigger Reasoning:** Send a prompt requiring deep logic (e.g., "Solve this complex logic puzzle step-by-step..."). 2. **Inspect JSON Response:** - Check `reasoning_content` length. - Check `content` length. 3. **Boundary Test:** - If `reasoning_content` is exactly 8192 tokens and the reasoning is incomplete, the budget is too low. - If `reasoning_content` is significantly less than 8192 but the model stops, check for `finish_reason=length`. **Command to verify budget via API:** - `curl -X POST http://192.168.1.116:8181/v1/chat/completions -d '{"model": "Qwen3.6...", "messages": [...], "reasoning_budget": 8192}'` [Low-risk] --- # Long Output Validation The observation of `finish_reason=length` before final content appears indicates a mismatch between the `max_tokens` parameter and the internal reasoning/output buffer. ### Mitigation Strategy: 1. **Iterative Expansion:** If a test fails with `finish_reason=length`, increase `max_tokens` by increments of 1024. 2. **Sequential Validation:** - Run Test A with `max_tokens=4096`. - If truncated, run Test A with `max_tokens=8192`. - If truncated, run Test A with `max_tokens=16384`. 3. **Content Verification:** - Ensure the final sentence of the `content` block is complete. - If the sentence ends in a period and the logic is sound, the test is "Passed." - If the sentence ends in a hyphen or mid-word, the test is "Failed - Truncated." **Validation Script Logic (Pseudo-code):** ```python def validate_output(response): if response.finish_reason == "length": print("Warning: Output truncated. Increasing max_tokens.") return False if "reasoning_content" in response and len(response.reasoning_content) > 0: print("Reasoning verified.") return True ``` --- # GPU And VRAM Checks The AMD Radeon AI PRO R9700 (32GB) is the primary compute resource. We must prevent VRAM overflow which causes the Vulkan driver to crash or the container to restart. ### VRAM Monitoring Protocol: 1. **Baseline:** Record VRAM usage with the model loaded but no active inference. 2. **Peak Load:** Record VRAM usage during a 262k context window request. 3. **Safety Threshold:** If VRAM usage exceeds 90% (approx. 28.8GB), the agent must immediately reduce the context window or the `max_tokens` for the next request. **Commands:** - `rocm-smi --showmeminfo` [Read-only] - `docker stats --no-stream` [Read-only] (Check memory limit of the container) --- # Proxy And Database Checks ### Flight Recorder Proxy (Port 8181) - Ensure the proxy is correctly forwarding requests to the llama.cpp server. - Check for 502/504 errors which indicate the backend is hanging. - `curl -I http://192.168.1.116:8181/v1/health` [Read-only] ### SQLite Database (/models) - Ensure the database is not locked by a zombie process. - Check for corruption. - `sqlite3 /models/database.db "PRAGMA integrity_check;"` [Read-only] - *Action:* If integrity check fails, initiate "Database Rollback" (see Rollback Procedures). --- # Dashboard Checks The ServerTop dashboard (Port 8090) is the primary UI for human oversight. 1. **Accessibility:** `curl -I http://192.168.1.116:8090` [Read-only] 2. **Metric Sync:** Verify that the "Active Model" displayed on ServerTop matches `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`. 3. **Latency Tracking:** Check the "Tokens Per Second" (TPS) graph. A sudden drop to 0 TPS indicates a Vulkan driver hang. --- # Failure Triage Trees ### Scenario A: `finish_reason=length` occurs on every request. 1. Check `max_tokens` in the API call. 2. Check `reasoning_budget` (8192). 3. **Action:** Increase `max_tokens` to 16384. 4. **Validation:** Re-run a single test. If it passes, update the benchmark config. ### Scenario B: Container restarts unexpectedly. 1. Check `docker logs` for "Out of Memory" or "Vulkan Error." 2. Check `rocm-smi` for VRAM spikes. 3. **Action:** Reduce context window from 262144 to 131072. 4. **Validation:** Run a small context test. ### Scenario C: Reasoning content is empty. 1. Check if `--reasoning-budget` is passed correctly to the server. 2. Check if the model is correctly identified as a reasoning-capable model. 3. **Action:** Restart the container with explicit `--reasoning-budget 8192` flag. ### Scenario D: Flight Recorder Proxy returns 502. 1. Check if the llama.cpp container is alive. 2. Check if the proxy is pointing to the correct internal IP/Port. 3. **Action:** Restart the proxy container. --- # Rollback Procedures In the event of a catastrophic failure (e.g., database corruption, driver hang, or broken config), the following rollback steps must be taken. ### 1. Configuration Rollback - **Snapshot:** Before any change, copy the current config: `cp config.json config.json.bak` [Low-risk] - **Rollback:** `mv config.json.bak config.json` [Low-risk] ### 2. Database Rollback - **Snapshot:** `cp /models/database.db /models/database.db.bak` [Low-risk] - **Rollback:** `cp /models/database.db.bak /models/database.db` [Low-risk] ### 3. Container Rollback - If a new image version is unstable: - `docker stop ` [Medium-risk] - `docker run ...` (Use the previous known-working image tag `working-20260613`) [Medium-risk] --- # Human Approval Gates The agent **must** stop and request human approval for the following actions: 1. **Destructive Commands:** Any command involving `rm`, `mv` (without a backup), or `docker rm`. 2. **Bulk Execution:** Any request to run more than 5 benchmark tasks in a single batch. 3. **MTP Specification Changes:** Any change to `--spec-type` or `--spec-draft-n-max`. 4. **Database Schema Changes:** Any `ALTER TABLE` or `DROP TABLE` commands in the SQLite database. 5. **External Data Exposure:** Any request to upload logs or screenshots to a non-local URL. --- # Evidence To Capture For every benchmark run and every failure, the following evidence must be collected: 1. **Request/Response JSON:** The full raw JSON from the Flight Recorder proxy. 2. **ServerTop Screenshot:** A screenshot of the dashboard showing TPS and VRAM during the peak of the run. 3. **Log Snippets:** The last 100 lines of the `llama-cpp-server-vulkan` logs. 4. **VRAM Snapshot:** The output of `rocm-smi` immediately following a failure. 5. **Validation Report:** A summary of whether the `finish_reason` was `stop` or `length`. --- # Final Go No-Go Checklist Before starting a benchmark suite, the agent must verify all items. If any item is "NO," the run is aborted. - [ ] **Network:** Proxy (8181) and Dashboard (8090) are reachable? (Yes/No) - [ ] **VRAM:** Current free VRAM > 5GB? (Yes/No) - [ ] **Model:** Qwen3.6-35B is confirmed loaded in logs? (Yes/No) - [ ] **Database:** SQLite integrity check passed? (Yes/No) - [ ] **Reasoning:** A test prompt successfully produced `reasoning_content`? (Yes/No) - [ ] **Privacy:** Is the environment isolated from public internet? (Yes/No) - [ ] **Rollback:** Is a backup of `config.json` and `database.db` present? (Yes/No) **Final Decision:** [GO / NO-GO] ### Advanced MTP (Multi-Token Prediction) Diagnostics The `--spec-type draft-mtp` and `--spec-draft-n-max 2` parameters are critical for optimizing the throughput of the Qwen3.6-35B model. MTP allows the model to predict multiple future tokens simultaneously, which can significantly increase tokens-per-second (TPS) by reducing the number of sequential forward passes required. However, this introduces complexity in validation. **Diagnostic Protocol for MTP:** 1. **Draft Acceptance Rate:** Monitor the ratio of accepted draft tokens versus rejected tokens. If the acceptance rate falls below 40%, the draft tokens are likely hallucinating or deviating from the primary model's logic. - *Action:* Reduce `--spec-draft-n-max` to 1 or disable MTP to establish a baseline. 2. **Reasoning Consistency:** MTP can sometimes "skip" logical steps in the reasoning chain if the draft tokens are too aggressive. - *Action:* Compare a run with MTP enabled against a run with MTP disabled for the same prompt. If the reasoning steps differ significantly in logic (not just wording), MTP is over-speculating. 3. **VRAM Overhead:** MTP requires additional memory to store the draft states. - *Action:* If `rocm-smi` shows VRAM nearing 30GB during MTP runs, decrease the context window or the draft count. **Command for MTP Baseline Comparison:** - `curl -X POST http://192.168.1.116:8181/v1/chat/completions -d '{"model": "Qwen3.6...", "messages": [...], "spec_type": "none"}'` [Low-risk] - `curl -X POST http://192.168.1.116:8181/v1/chat/completions -d '{"model": "Qwen3.6...", "messages": [...], "spec_type": "draft-mtp", "spec_draft_n_max": 2}'` [Low-risk] --- ### Vulkan Driver & Hardware Layer Troubleshooting Since the environment utilizes the `llama-cpp-server-vulkan` container on an AMD Radeon AI PRO R9700, the Vulkan driver layer is the primary point of failure for GPU hangs or "Out of Memory" (OOM) errors. **Vulkan Integrity Checks:** 1. **Driver Versioning:** Ensure the host has the latest AMD ROCm/Vulkan drivers. - `vulkaninfo | grep -i "driver version"` [Read-only] 2. **Memory Heap Mapping:** Verify that the Vulkan driver can see the full 32GB of VRAM. - `vulkaninfo --summary` [Read-only] 3. **TDR (Timeout Detection and Recovery):** If the GPU hangs during long reasoning tasks, the OS may be resetting the driver. - *Action:* Check system logs (`dmesg` or `journalctl -xe`) for "GPU hang" or "amdgpu" errors. - *Fix:* Increase the TDR delay in the OS configuration (requires host-level access). **Vulkan Performance Tuning:** - If TPS is low despite high VRAM availability, check for "Memory Fragmentation." - *Action:* Restart the container to clear the Vulkan memory heap. - *Command:* `docker restart ` [Medium-risk] --- ### SQLite Database Schema & Management The database on `/models` is the source of truth for all benchmark results. It must be handled with extreme care to prevent corruption during concurrent writes or unexpected shutdowns. **Database Schema Definition:** The agent should ensure the following table structure exists: ```sql CREATE TABLE IF NOT EXISTS benchmarks ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, model_name TEXT, prompt_hash TEXT, reasoning_tokens INTEGER, content_tokens INTEGER, finish_reason TEXT, tps REAL, vram_usage_mb INTEGER, status TEXT, -- 'pass', 'fail', 'truncated' raw_response_json TEXT ); ``` **Maintenance Commands:** - **Check for Locks:** `sqlite3 /models/database.db "PRAGMA lock_status;"` [Read-only] - **Optimize Database:** `sqlite3 /models/database.db "VACUUM;"` [Medium-risk] - **Export for Human Review:** `sqlite3 /models/database.db ".dump" > benchmark_export.sql` [Low-risk] --- ### Flight Recorder Proxy Log Analysis The proxy at `http://192.168.1.116:8181/v1` acts as the intermediary. When a request fails, the proxy logs are the first place to look. **Log Analysis Procedure:** 1. **Identify Request ID:** Locate the specific request in the proxy logs. 2. **Check Latency:** If the time between "Request Received" and "Backend Response" is high, the issue is in `llama.cpp`. 3. **Check Status Codes:** - `400 Bad Request`: Check the JSON payload for incorrect `reasoning_budget` or `max_tokens`. - `502 Bad Gateway`: The `llama-cpp-server-vulkan` container has crashed or is unresponsive. - `504 Gateway Timeout`: The model is taking too long to generate a response (likely a very large context window or complex reasoning). **Command to Stream Proxy Logs:** - `docker logs -f ` [Read-only] --- ### Context Window & KV Cache Management The context window is set to 262,144. This is a massive window that consumes significant VRAM for the Key-Value (KV) cache. **Memory Calculation:** - The KV cache size is proportional to `Context Length * Number of Layers * Hidden Dimension`. - For a 35B model, a 262k context window can easily exceed 20GB of VRAM just for the cache, leaving little room for the model weights and the active activations. **Scaling Strategy:** 1. **Baseline:** Start benchmarks with a 32,768 context window. 2. **Stress Test:** Incrementally increase to 65,536, 131,072, and finally 262,144. 3. **Failure Point:** Identify the context window where `rocm-smi` shows VRAM usage exceeding 95%. 4. **Optimization:** If 262k is required, use a 4-bit or 8-bit KV cache quantization if supported by the `llama.cpp` build. --- ### Automated Validation & Reporting Scripting To ensure no "truncated" results are recorded as "passes," the agent should use a Python-based validation script. **Validation Script (`validate_benchmarks.py`):** ```python import requests import sqlite3 import json import time # Configuration PROXY_URL = "http://192.168.1.116:8181/v1" DB_PATH = "/models/database.db" MODEL_NAME = "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf" def run_benchmark(prompt, max_tokens=8192, reasoning_budget=8192): payload = { "model": MODEL_NAME, "messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens, "reasoning_budget": reasoning_budget, "temperature": 0.7 } try: start_time = time.time() response = requests.post(f"{PROXY_URL}/chat/completions", json=payload, timeout=300) response.raise_for_status() data = response.json() end_time = time.time() finish_reason = data['choices'][0]['finish_reason'] content = data['choices'][0]['message'].get('content', '') reasoning = data['choices'][0]['message'].get('reasoning_content', '') tps = len(content) / (end_time - start_time) # Validation Logic status = "pass" if finish_reason == "length": status = "truncated" print(f"Warning: Output truncated for prompt: {prompt[:50]}...") save_to_db(prompt, reasoning, content, finish_reason, tps, status) return status except Exception as e: print(f"Error during benchmark: {e}") return "error" def save_to_db(prompt, reasoning, content, finish_reason, tps, status): conn = sqlite3.connect(DB_PATH) cursor = conn.cursor() cursor.execute(""" INSERT INTO benchmarks (model_name, prompt_hash, reasoning_tokens, content_tokens, finish_reason, tps, status, raw_response_json) VALUES (?, ?, ?, ?, ?, ?, ?, ?) """, (MODEL_NAME, hash(prompt), len(reasoning), len(content), finish_reason, tps, status, json.dumps(content))) conn.commit() conn.close() if __name__ == "__main__": # Example test test_prompt = "Explain the quantum Zeno effect in detail." run_benchmark(test_prompt) ``` *Risk Level: Low-risk (Script execution)* --- ### Incident Response Playbook (Red Alert) In the event of a critical system failure, the agent must follow these prioritized steps. **Level 1: Minor Inference Error (Truncation, Slow TPS)** 1. Check `max_tokens` vs `reasoning_budget`. 2. Verify if the prompt is excessively long. 3. Restart the `llama-cpp-server-vulkan` container. **Level 2: Service Unavailability (502/504 Errors)** 1. Check if the container is running: `docker ps`. 2. Check `rocm-smi` for GPU "Hang" or "Reset" status. 3. If GPU is hung, perform a hard reset of the GPU driver or the host machine. 4. Check the SQLite database for any "locked" states and clear them. **Level 3: Data Corruption (Database Errors)** 1. **STOP ALL ACTIONS.** 2. Verify the integrity of `/models/database.db`. 3. If corrupted, restore from the last known good backup (`database.db.bak`). 4. Notify the human administrator immediately. --- ### Benchmark Suite Definitions To ensure consistent reporting, the following benchmark suite must be used. Each task must be run individually. | Task ID | Category | Prompt | Expected Output Length | | :--- | :--- | :--- | :--- | | B-001 | Logic | "If Alice has three apples and Bob takes two, but then Charlie gives Alice one back, how many apples does Alice have? Explain step-by-step." | Short | | B-002 | Coding | "Write a Python script to scrape a website using BeautifulSoup and save the results to a CSV file. Include error handling." | Medium | | B-003 | Creative | "Write a short story about a robot discovering a flower in a wasteland. The story should be at least 500 words." | Long | | B-004 | Context | [Insert 10,000 words of text] + "Summarize the main themes of the text provided above." | Long | | B-005 | Reasoning | "Solve the following: A man is looking at a photograph. His friend asks who it is. The man replies, 'Brothers and sisters, I have none. But that man's father is my father's son.' Who is in the photograph?" | Short | | B-006 | MTP-Stress | "Generate a list of 50 unique names for a fantasy RPG, including a brief description of each character's primary skill." | Long | | B-007 | Math | "Calculate the derivative of f(x) = sin(x^2) * e^x and explain each step of the chain rule application." | Medium | | B-008 | Translation | "Translate the following paragraph into French, ensuring the tone remains formal: [Insert formal paragraph]." | Medium | | B-009 | Summarization | "Summarize the following technical documentation into three bullet points for a non-technical audience: [Insert documentation]." | Short | | B-010 | Synthesis | "Compare and contrast the architectural differences between Vulkan and DirectX 12 in the context of AI inference." | Medium | --- ### Data Privacy & Sanitization Protocols To maintain the privacy of the home-lab environment, the following rules are absolute: 1. **No PII:** No personally identifiable information (names, addresses, private keys) should be included in benchmark prompts. 2. **Log Scrubbing:** Before any logs are shared with a human for troubleshooting, the agent must scrub the `raw_response_json` field in the SQLite database to remove any potentially sensitive content. 3. **Local Only:** The Flight Recorder proxy must never be exposed to the public internet. The `192.168.1.116` address must remain internal. 4. **Screenshot Sanitization:** Screenshots of the ServerTop dashboard must blur any specific IP addresses or local file paths if they are to be shared outside the immediate home-lab network. --- ### Hardware Thermal & Power Monitoring The AMD Radeon AI PRO R9700 can generate significant heat during long reasoning tasks. **Thermal Monitoring Protocol:** 1. **Baseline Temperature:** Record temperature at idle. 2. **Active Monitoring:** Run `watch -n 5 "sensors"` during a "Long" category benchmark. 3. **Thermal Throttling:** If temperatures exceed 85°C, the agent must pause the benchmark suite and allow the hardware to cool. 4. **Power Draw:** Monitor power consumption to ensure the home-lab power supply (PSU) is not being overtaxed. **Commands:** - `sensors` [Read-only] - `watch -n 1 "rocm-smi --showtemp"` [Read-only] --- ### Model Quantization & Precision Analysis The `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` model uses a specific quantization level to fit into the 32GB VRAM. **Analysis Points:** 1. **Perplexity vs. VRAM:** If the model's reasoning quality is poor, the agent should investigate if the quantization (e.g., Q4_K_M) is too aggressive for the complexity of the tasks. 2. **Precision Check:** Verify that the Vulkan backend is utilizing FP16 or BF16 precision correctly. 3. **Weight Loading:** Ensure the model is fully loaded into VRAM. If `llama.cpp` reports "offloading to CPU," the benchmark results will be invalid due to extreme latency. **Command to Verify Offloading:** - `docker logs | grep "offloaded"` [Read-only] - *Success Criteria:* "All layers offloaded to GPU" or "100% offloaded." --- ### Maintenance Schedule & Periodic Health Checks To ensure long-term stability, the following maintenance schedule is established: **Daily:** - Check `docker ps` to ensure all containers are running. - Review the last 5 entries in the SQLite database for `truncated` or `error` statuses. - Verify VRAM usage is returning to baseline after benchmarks. **Weekly:** - Perform a `VACUUM` on the SQLite database. - Check for any "GPU Hang" entries in the system logs. - Verify that the Flight Recorder proxy logs are being rotated and not filling up the disk. **Monthly:** - Update the `llama-cpp-server-vulkan` container to the latest stable build if a significant performance improvement is noted. - Perform a full backup of the `/models` directory. - Review the benchmark suite and add new prompts based on the model's evolving capabilities. **Quarterly:** - Perform a deep hardware cleaning (dusting) of the GPU and host machine. - Re-evaluate the MTP parameters to see if newer `llama.cpp` versions offer better `draft-mtp` implementations. - Audit the SQLite database for any redundant or failed entries and archive them. --- ### Final Verification of Infrastructure Integrity Before the agent concludes any maintenance or diagnostic session, it must perform a final "Heartbeat" check. **Heartbeat Protocol:** 1. **Proxy Check:** `curl -I http://192.168.1.116:8181/v1` (Expect 200) 2. **Dashboard Check:** `curl -I http://192.168.1.116:8090` (Expect 200) 3. **Model Check:** Query the API for the model's metadata to ensure it is "Ready." 4. **VRAM Check:** `rocm-smi` (Expect < 25GB used) **Final Status Report:** The agent will generate a summary including: - Total benchmarks run. - Success/Failure/Truncation counts. - Average TPS. - Peak VRAM usage. - Any hardware warnings encountered. - A "Go/No-Go" recommendation for the next scheduled benchmark run. --- ### Detailed Failure Triage: Vulkan Memory Fragmentation One specific issue with Vulkan in `llama.cpp` is memory fragmentation, where the GPU has enough total memory, but not enough *contiguous* memory for a large KV cache. **Symptoms:** - Benchmark starts normally but crashes or hangs exactly when the context window reaches a certain point (e.g., 128k). - `rocm-smi` shows plenty of free memory, but the container logs show "Out of Memory." **Fix Procedure:** 1. **Clear Cache:** Stop the container and wait 30 seconds. 2. **Restart:** Start the container. 3. **Pre-allocate:** If the `llama.cpp` build supports it, use the `--gpu-memory` flag to pre-allocate a fixed block of VRAM. 4. **Validation:** Run a test at the specific context window where the failure occurred. If it passes, the issue was fragmentation. --- ### Detailed Failure Triage: MTP Hallucination MTP can cause the model to "hallucinate" the next few tokens, which the primary model then has to "correct." If the correction is too heavy, it can lead to repetitive loops. **Symptoms:** - The model repeats the same phrase 3-4 times in a row. - The reasoning content becomes nonsensical or circular. **Fix Procedure:** 1. **Identify:** Check the `raw_response_json` for repetitive patterns. 2. **Adjust:** Reduce `--spec-draft-n-max` from 2 to 1. 3. **Verify:** Re-run the same prompt. If the repetition stops, the MTP draft was too aggressive. 4. **Alternative:** If the issue persists, disable MTP entirely and use standard autoregressive generation. --- ### Detailed Failure Triage: SQLite Database Locking If the Python validation script and a manual `sqlite3` command run simultaneously, the database may return a "database is locked" error. **Symptoms:** - `sqlite3.OperationalError: database is locked` in the Python script. - Failure to insert benchmark results. **Fix Procedure:** 1. **Identify:** Check if any other terminal or process is currently connected to `/models/database.db`. 2. **Kill Zombies:** Use `lsof /models/database.db` to find the PID of the locking process and kill it if it is a zombie. 3. **WAL Mode:** Enable Write-Ahead Logging to allow concurrent reads and writes. - `sqlite3 /models/database.db "PRAGMA journal_mode=WAL;"` [Medium-risk] 4. **Retry Logic:** Update the Python script to include a retry loop for database insertions. --- ### Detailed Failure Triage: Proxy Timeout (504) A 504 Gateway Timeout usually means the `llama.cpp` server is still processing, but the proxy has given up. **Symptoms:** - The API returns a 504 error, but `docker logs` for the `llama.cpp` container shows it is still generating tokens. **Fix Procedure:** 1. **Identify:** Compare the timestamp of the 504 error with the current activity in the `llama.cpp` logs. 2. **Adjust Proxy:** Increase the timeout limit in the Flight Recorder proxy configuration (e.g., from 60s to 300s). 3. **Adjust Client:** Increase the `timeout` parameter in the Python `requests.post` call. 4. **Validation:** Re-run a "Long" category benchmark and ensure the response is received fully. --- ### Detailed Failure Triage: Context Window Overflow If the prompt + the generated content exceeds the 262,144 context window, the model will start "forgetting" the beginning of the prompt or crash. **Symptoms:** - The model provides a correct answer but ignores the specific constraints provided at the start of a very long prompt. - The `finish_reason` is `length` even though `max_tokens` was set very high. **Fix Procedure:** 1. **Identify:** Calculate the total tokens (Prompt + Reasoning + Content). 2. **Check:** If the total exceeds 262,144, the context window is exceeded. 3. **Action:** Shorten the prompt or use a summarization step to condense the input before sending it to the model. 4. **Validation:** Verify that the model now respects the original constraints. --- ### Detailed Failure Triage: Vulkan Driver "TDR" Hang On some Linux distributions, the Vulkan driver will reset if a kernel operation takes longer than 2 seconds. **Symptoms:** - The entire host machine UI freezes for a moment, and the `llama-cpp` container restarts. - `dmesg` shows "amdgpu: [drm] GPU hang". **Fix Procedure:** 1. **Identify:** Check `dmesg` for "ring gfx_0.0.0 timeout". 2. **Action:** This is a kernel-level issue. The agent must recommend the human administrator increase the `amdgpu.gpu_recovery` or `amdgpu.min_free_mem` parameters in the GRUB configuration. 3. **Workaround:** Reduce the `reasoning_budget` to 4096 to reduce the duration of the heavy compute phase. --- ### Detailed Failure Triage: Model Weight Corruption If the `.gguf` file is corrupted during download or move, the server may start but produce gibberish or crash immediately. **Symptoms:** - The model produces random characters or "NaN" values. - The container crashes immediately upon loading the model. **Fix Procedure:** 1. **Identify:** Check the file size of `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`. 2. **Verify:** Compare the hash of the file against the source. 3. **Action:** Re-download or re-copy the model file. 4. **Validation:** Run a simple "Hello" prompt to ensure coherent output. --- ### Detailed Failure Triage: Proxy Header Mismatch The Flight Recorder proxy might not be passing the correct headers (like `Authorization` or `Content-Type`) to the backend. **Symptoms:** - The proxy returns 401 Unauthorized or 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Check the proxy configuration for header forwarding. 2. **Action:** Ensure `X-Forwarded-For` and `Authorization` headers are explicitly allowed. 3. **Validation:** Use `curl -v` to inspect the headers being sent and received. --- ### Detailed Failure Triage: VRAM Leak A memory leak occurs when the VRAM usage increases with every request and never returns to the baseline. **Symptoms:** - `rocm-smi` shows VRAM usage creeping up (e.g., 10GB -> 12GB -> 15GB) over several benchmark runs. **Fix Procedure:** 1. **Identify:** Monitor VRAM after each of the 10 benchmark tasks. 2. **Action:** Restart the `llama-cpp-server-vulkan` container. 3. **Investigation:** Check if the `llama.cpp` version has a known memory leak in the Vulkan backend. 4. **Validation:** Run 5 benchmarks and ensure VRAM usage remains stable. --- ### Detailed Failure Triage: Reasoning Budget Exhaustion The model stops reasoning because it hit the 8192 limit, but it still has more to say in the final content. **Symptoms:** - `reasoning_content` is exactly 8192 tokens. - The final `content` is cut off or starts mid-sentence. **Fix Procedure:** 1. **Identify:** Check the `reasoning_tokens` count in the SQLite database. 2. **Action:** Increase `--reasoning-budget` to 16384. 3. **Validation:** Re-run the specific prompt and ensure the reasoning is complete and the content is not truncated. --- ### Detailed Failure Triage: MTP Draft Conflict The draft tokens predicted by MTP conflict with the primary model's logic, causing the primary model to "reject" almost everything. **Symptoms:** - Very low TPS (slower than standard generation). - High "rejection" rate in the logs. **Fix Procedure:** 1. **Identify:** Check the `llama.cpp` logs for "Draft rejected" messages. 2. **Action:** Decrease `--spec-draft-n-max` to 1. 3. **Validation:** Check if TPS improves and the rejection rate drops. --- ### Detailed Failure Triage: SQLite Write Collision Two different agents or scripts try to write to the database at the exact same millisecond. **Symptoms:** - `sqlite3.OperationalError: database is locked`. **Fix Procedure:** 1. **Identify:** Check for multiple active Python processes. 2. **Action:** Implement a file-based lock (e.g., `flock`) or a queue system for database writes. 3. **Validation:** Run two concurrent benchmark scripts and ensure both results are recorded. --- ### Detailed Failure Triage: Vulkan Context Loss The Vulkan context is lost because the GPU was used by another application (e.g., a desktop environment or a browser). **Symptoms:** - `llama-cpp` logs show "Vulkan context lost" or "Device lost." **Fix Procedure:** 1. **Identify:** Check if any other applications are using the R9700. 2. **Action:** Close non-essential applications or run the benchmark in a headless environment. 3. **Validation:** Run the benchmark again and ensure the context remains stable. --- ### Detailed Failure Triage: Proxy Request Buffer Overflow The proxy receives a very large prompt (e.g., 100k tokens) and the buffer size is too small. **Symptoms:** - Proxy returns 413 Request Entity Too Large. **Fix Procedure:** 1. **Identify:** Check the prompt size in the request. 2. **Action:** Increase the `client_max_body_size` or equivalent buffer setting in the proxy. 3. **Validation:** Send the large prompt and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama.cpp` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the benchmark and ensure it is accepted. --- ### Detailed Failure Triage: Model Weight Loading Timeout The model is so large that it takes too long to load into VRAM, causing the proxy to timeout. **Symptoms:** - Proxy returns 504 Gateway Timeout during the "Model Loading" phase. **Fix Procedure:** 1. **Identify:** Check `docker logs` to see if the model is still loading. 2. **Action:** Increase the proxy's initial connection timeout. 3. **Validation:** Wait for the "Model loaded" message in the logs and then run a benchmark. --- ### Detailed Failure Triage: Incorrect MTP Spec Type The `--spec-type` is set to a value that the current `llama-cpp-server-vulkan` build does not support. **Symptoms:** - Container fails to start or throws an "Invalid argument" error. **Fix Procedure:** 1. **Identify:** Check the `docker logs` for the specific error message. 2. **Action:** Verify the supported spec types in the `llama-cpp-server-vulkan` documentation. 3. **Validation:** Correct the flag and restart the container. --- ### Detailed Failure Triage: SQLite Schema Mismatch The Python script expects a column that does not exist in the database. **Symptoms:** - `sqlite3.OperationalError: no such column: ...` **Fix Procedure:** 1. **Identify:** Check the error message for the missing column name. 2. **Action:** Run an `ALTER TABLE` command to add the missing column. 3. **Validation:** Run the Python script and ensure it completes successfully. --- ### Detailed Failure Triage: Vulkan Shader Compilation Hang The first time a model is run, Vulkan compiles shaders, which can take a long time and look like a hang. **Symptoms:** - The first benchmark takes 5-10 minutes to start, but subsequent benchmarks are fast. **Fix Procedure:** 1. **Identify:** Check `docker logs` for "Compiling shaders" or "Initializing Vulkan." 2. **Action:** Perform a "warm-up" run. Run a very short prompt once to trigger shader compilation. 3. **Validation:** Run the actual benchmark suite and ensure it starts promptly. --- ### Detailed Failure Triage: Proxy Routing Error The proxy is sending requests to the wrong internal IP or port. **Symptoms:** - Proxy returns 502 Bad Gateway consistently. **Fix Procedure:** 1. **Identify:** Check the proxy's `upstream` configuration. 2. **Action:** Verify the `llama-cpp` container's internal IP and port (usually 8080). 3. **Validation:** Use `curl` to hit the internal IP directly to confirm it is working. --- ### Detailed Failure Triage: Context Window Memory Overflow The context window is set to 262k, but the system only has enough VRAM for 128k. **Symptoms:** - Container crashes with "Out of Memory" immediately upon receiving a long prompt. **Fix Procedure:** 1. **Identify:** Check `rocm-smi` and the `llama-cpp` logs. 2. **Action:** Reduce the context window to 65,536. 3. **Validation:** Run a benchmark and ensure it completes without crashing. --- ### Detailed Failure Triage: Reasoning Budget Overflow The reasoning budget is set higher than the `max_tokens` limit. **Symptoms:** - The model stops reasoning before it finishes, and the `finish_reason` is `length`. **Fix Procedure:** 1. **Identify:** Check the `reasoning_budget` vs `max_tokens` in the API call. 2. **Action:** Ensure `max_tokens` is always greater than `reasoning_budget` + expected content length. 3. **Validation:** Re-run the benchmark and ensure the content is complete. --- ### Detailed Failure Triage: MTP Draft Token Hallucination The draft tokens are logically sound but factually incorrect. **Symptoms:** - The model provides a correct answer but includes a "hallucinated" fact in the reasoning chain. **Fix Procedure:** 1. **Identify:** Review the `reasoning_content` for factual errors. 2. **Action:** Disable MTP or reduce the draft count. 3. **Validation:** Re-run the benchmark and ensure the facts are correct. --- ### Detailed Failure Triage: SQLite Database Corruption The database file is corrupted due to a hard power loss. **Symptoms:** - `sqlite3` returns "Database disk image is malformed." **Fix Procedure:** 1. **Identify:** Run `PRAGMA integrity_check;`. 2. **Action:** Restore from the `.bak` file. 3. **Validation:** Run the integrity check again to ensure it passes. --- ### Detailed Failure Triage: Vulkan Driver Version Mismatch The `llama-cpp-server-vulkan` container is using a version of the Vulkan loader that is incompatible with the host driver. **Symptoms:** - Container fails to start with "Could not initialize Vulkan." **Fix Procedure:** 1. **Identify:** Check the `LD_LIBRARY_PATH` inside the container. 2. **Action:** Ensure the container is using the host's Vulkan libraries. 3. **Validation:** Restart the container and check the logs for "Vulkan initialized." --- ### Detailed Failure Triage: Proxy Header Stripping The proxy is stripping the `Content-Type: application/json` header. **Symptoms:** - The backend returns 415 Unsupported Media Type. **Fix Procedure:** 1. **Identify:** Use `curl -v` to check the headers sent by the proxy. 2. **Action:** Update the proxy configuration to preserve the `Content-Type` header. 3. **Validation:** Re-run the