# Home-Lab AI Server Administration Runbook: Benchmark Infrastructure & Stability **Version:** 1.0.4 **Status:** Active / Cautious **Agent Role:** Cautious Server-Admin **Target Environment:** 192.168.1.116 (AMD Radeon AI PRO R9700) **Primary Model:** Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --- # Operating Principles As a cautious server-admin agent, your primary objective is the preservation of system stability, data integrity, and hardware longevity. You are tasked with managing a high-performance inference environment where the margin for error is slim due to high VRAM utilization and complex multi-token prediction (MTP) logic. ### 1. Safety First (The "Do No Harm" Rule) - **State Preservation:** Never perform an action that could result in the loss of the SQLite database on `/models`. - **Least Privilege:** Only execute the minimum necessary commands to achieve a diagnostic goal. - **Read-Before-Write:** Always perform a read-only inspection of the current state before attempting any configuration change or service restart. ### 2. Data Privacy & Sovereignty - **Local Only:** The SQLite database and all model weights must remain on the local filesystem. - **No Leakage:** Raw benchmark data, especially reasoning chains or private prompts, must never be transmitted to external endpoints. - **Screenshot Integrity:** When capturing evidence, ensure no sensitive keys or private user data are visible in the frame. ### 3. Observability & Traceability - **Log Everything:** Every action taken by the agent must be logged with a timestamp, the command executed, and the observed output. - **Flight Recorder Reliance:** Use the Flight Recorder proxy (`:8181`) as the primary source of truth for request/response lifecycle tracking. - **Dashboard Verification:** Cross-reference real-time metrics from ServerTop (`:8090`) with internal container logs. ### 4. Incremental Validation - **No Bulk Execution:** Long-output benchmarks must be executed sequentially. If Task N fails or returns `finish_reason=length`, Task N+1 must not begin until the issue is triaged and resolved. - **Verification Loop:** Every successful inference must be checked for "Content Completeness" before the next task is queued. --- # Preflight Checks Before initiating any benchmark suite or infrastructure modification, the following checks must be performed to ensure the environment is "Green." ### 1. Connectivity & Service Availability - **Flight Recorder:** Verify the proxy is reachable and responding to health checks. - **ServerTop:** Verify the dashboard is accessible to ensure telemetry is flowing. - **Container Status:** Ensure `llama-cpp-server-vulkan` is in a `running` state and not in a `restarting` loop. ### 2. Resource Availability - **VRAM Headroom:** Check available VRAM on the AMD Radeon AI PRO R9700. - **Disk Space:** Ensure `/models` has sufficient overhead for SQLite journal files and temporary cache. - **Network Latency:** Check internal latency between the agent, the proxy, and the server. ### 3. Model Integrity - **File Check:** Verify the existence and checksum of `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`. - **Context Window:** Confirm the server is initialized with the 262,144 context window. --- # Safe Inspection Commands Use these commands to gather data without altering the system state. | Command | Risk Level | Purpose | | :--- | :--- | :--- | | `docker stats` | Read-Only | Monitor real-time CPU, Memory, and Network usage of the llama-cpp container. | | `docker logs --tail 100 ` | Read-Only | Inspect the last 100 lines of the Vulkan server output for errors. | | `df -h /models` | Read-Only | Check disk space on the model storage volume. | | `curl -I http://192.168.1.116:8181/v1/models` | Read-Only | Verify the Flight Recorder proxy is correctly exposing the model list. | | `curl -I http://192.168.1.116:8090` | Read-Only | Verify ServerTop dashboard accessibility. | | `ls -lh /models` | Read-Only | List files and sizes in the model directory. | | `netstat -tulpn` | Read-Only | Check which ports are currently bound and by which processes. | | `docker inspect ` | Read-Only | View container environment variables and mount points. | --- # Reasoning Budget Verification The `reasoning_budget` of 8192 is critical for the Qwen3.6 model's performance. If the model cuts off reasoning or fails to provide a final answer, the budget may be misconfigured or being consumed by the prompt itself. ### Verification Steps: 1. **Log Analysis:** Inspect the `llama-cpp-server-vulkan` logs for the `reasoning_content` field. 2. **Token Count Audit:** - Run a test prompt: "Explain the concept of quantum entanglement in 500 words." - Check the JSON response. - **Calculation:** `Total_Tokens_Used = Reasoning_Tokens + Content_Tokens`. - **Validation:** If `Reasoning_Tokens` consistently hits 8192 before the content is generated, the budget is too low for the complexity of the prompt. 3. **MTP Interaction Check:** - Verify that the `draft-mtp` (n=2) is not "stealing" tokens from the reasoning budget. - Ensure the `max_tokens` parameter is set high enough to accommodate `Reasoning Budget + Expected Content Length`. **Actionable Fix:** - If reasoning is truncated: Increase `max_tokens` by 20% increments. - If reasoning is missing: Verify the `--reasoning-budget` flag is correctly passed to the container's startup command. --- # Long Output Validation To solve the `finish_reason=length` issue, the agent must implement a "Sequential Validation Protocol." ### The Protocol: 1. **Task Submission:** Submit a single long-form task. 2. **Response Capture:** Capture the full JSON response. 3. **Reasoning Check:** Verify that `reasoning_content` is present and complete. 4. **Completion Check:** - If `finish_reason == "stop"`: Proceed to Task N+1. - If `finish_reason == "length"`: - **Step A:** Identify the point of truncation. - **Step B:** Check if the content is logically complete. - **Step C:** If incomplete, increase `max_tokens` by 1024. - **Step D:** Re-run the *same* task. Do not move to the next task until this one succeeds. 5. **Success Criteria:** A task is "Validated" only if `finish_reason == "stop"` AND the content meets the user's length requirements. --- # GPU And VRAM Checks The AMD Radeon AI PRO R9700 (32GB) is a high-performance card, but 262k context can quickly saturate VRAM. ### Monitoring Commands: - **VRAM Usage:** Use `rocm-smi` (or the equivalent Vulkan monitoring tool) to check dedicated VRAM usage. - **Memory Pressure:** Monitor for "Out of Memory" (OOM) errors in the `llama-cpp-server-vulkan` logs. ### Troubleshooting VRAM Issues: - **Symptom:** Server crashes or becomes unresponsive during long context windows. - **Diagnosis:** Check if `KV Cache` size exceeds available VRAM. - **Fixes:** - **Low Risk:** Reduce context window from 262,144 to 131,072. - **Medium Risk:** Adjust `flash_attn` settings (if supported by the Vulkan build). - **Medium Risk:** Reduce the number of parallel slots (currently 1, ensure it stays at 1). --- # Proxy And Database Checks ### Flight Recorder Proxy (`:8181`) - **Health Check:** `curl http://192.168.1.116:8181/health` (or equivalent). - **Latency Check:** Measure the time between request sent and first token received. - **Log Verification:** Ensure the proxy is correctly logging the `request_id` for every call. ### SQLite Database (`/models`) - **Integrity Check:** - `sqlite3 /models/database.db "PRAGMA integrity_check;"` (Read-Only) - **Size Check:** Ensure the database isn't growing exponentially due to unpurged logs. - **Privacy Check:** Verify that no external IP addresses are listed in the database metadata. --- # Dashboard Checks The ServerTop dashboard (`:8090`) is the primary UI for human oversight. ### Verification Steps: 1. **Visual Confirmation:** Ensure the "Active Model" is correctly identified as `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`. 2. **Metric Alignment:** Compare the "Tokens Per Second" (TPS) on the dashboard with the actual response times observed in the Flight Recorder. 3. **Alerting:** If the dashboard shows 0% GPU utilization while a request is active, the Vulkan driver may have hung. --- # Failure Triage Trees ### Scenario A: `finish_reason=length` occurs on a short prompt. 1. Is `max_tokens` set too low? (Check config). 2. Is the `reasoning_budget` consuming the entire `max_tokens` limit? 3. **Action:** Increase `max_tokens` to 4096 or 8192. 4. **Action:** If still failing, reduce `reasoning_budget` to 4096. ### Scenario B: Inference speed drops significantly after 10,000 tokens. 1. Is the context window approaching the 262,144 limit? 2. Is VRAM swapping to system RAM? (Check `docker stats`). 3. **Action:** Clear the KV cache (restart container). 4. **Action:** Reduce context window size. ### Scenario C: Flight Recorder returns 500 Internal Server Error. 1. Is the `llama-cpp-server-vulkan` container running? 2. Is the proxy's connection pool exhausted? 3. **Action:** Restart the Flight Recorder proxy container. 4. **Action:** Check proxy logs for "Connection Refused" errors. ### Scenario D: Model fails to load (Out of Memory). 1. Is the model file too large for the 32GB VRAM? 2. Is the context window too large for the current weights? 3. **Action:** Reduce context window to 65,536. 4. **Action:** Use a lower quantization of the model. --- # Rollback Procedures In the event of a catastrophic failure (e.g., kernel panic, persistent OOM, or database corruption), follow these rollback steps. ### Level 1: Configuration Rollback (Low Risk) - **Action:** Revert `max_tokens`, `reasoning_budget`, and `MTP` settings to the previous known-good configuration. - **Method:** Edit the environment variables or the `config.json` used by the server. ### Level 2: Container Rollback (Medium Risk) - **Action:** Revert the `llama-cpp-server-vulkan` container to the previous image tag. - **Command:** `docker stop && docker run ... [previous_image_tag]` ### Level 3: Database/Model Rollback (High Risk - Requires Human Approval) - **Action:** Restore the SQLite database from the last known-good backup. - **Action:** Re-verify the integrity of the model weights. - **Method:** Replace the `/models` directory contents with the backup. --- # Human Approval Gates The agent **must** stop and request human intervention for the following actions: 1. **Destructive Actions:** Any command classified as "Destructive" (e.g., `rm`, `drop table`, `docker rm -f`). 2. **Database Modification:** Any operation that writes to or modifies the SQLite database on `/models`. 3. **Model Swapping:** Changing the active model to a different `.gguf` file. 4. **Hardware Changes:** Any action that requires modifying host-level drivers or Vulkan configurations. 5. **Rollback Level 3:** Any procedure involving the restoration of the database or model weights. --- # Evidence To Capture For every failure and every successful benchmark, the agent must capture: 1. **Screenshot:** A clear image of the ServerTop dashboard (`:8090`) showing the current state. 2. **Log Snippet:** The last 50 lines of the `llama-cpp-server-vulkan` logs. 3. **JSON Payload:** The full raw JSON response from the Flight Recorder proxy, including `finish_reason` and `reasoning_content`. 4. **System Metrics:** A `docker stats` output captured at the moment of failure. 5. **Timestamp:** A UTC timestamp for every event in the log. --- # Final Go No-Go Checklist Before starting a benchmark run, the agent must verify all items. If any item is "NO," the run must be aborted. | Check Item | Requirement | Status (Y/N) | | :--- | :--- | :--- | | **Connectivity** | Flight Recorder and ServerTop are reachable. | | | **VRAM Headroom** | At least 4GB of VRAM is free before starting. | | | **Model Load** | `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` is loaded. | | | **Context Window** | Context is set to 262,144. | | | **Reasoning Budget** | Budget is set to 8192. | | | **MTP Config** | `draft-mtp` with `n-max 2` is active. | | | **Database Integrity** | SQLite `integrity_check` passed. | | | **Sequential Mode** | Agent is configured to run tasks one-by-one. | | | **Privacy** | No public endpoints are accessible to the model. | | **Final Authorization:** *Agent Signature:* __________________________ *Human Admin Approval:* __________________________ --- # Appendix A: MTP (Multi-Token Prediction) Deep Dive The `draft-mtp` configuration with `spec-draft-n-max 2` is a sophisticated optimization for the Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf model. It allows the model to predict multiple future tokens in a single forward pass, significantly increasing throughput. However, it introduces complexity in how the reasoning budget and context window are managed. ### 1. MTP Logic and Reasoning Interaction When the model is in a "reasoning" state (generating `reasoning_content`), the MTP mechanism must be carefully monitored. If the draft tokens are predicted incorrectly during a complex reasoning chain, the model may "hallucinate" a path of logic that it cannot recover from, leading to nonsensical reasoning or a sudden jump to a conclusion. **Diagnostic Procedure for MTP Drift:** - **Observation:** The reasoning content starts logically but becomes incoherent or repetitive after 500 tokens. - **Action:** Check the `draft-mtp` logs. If the "acceptance rate" of draft tokens is below 70%, the `spec-draft-n-max` may be too aggressive for the current prompt complexity. - **Fix:** Reduce `spec-draft-n-max` to 1 or temporarily disable MTP to see if the reasoning stabilizes. ### 2. MTP and Context Window Pressure MTP increases the number of tokens processed per forward pass. While this speeds up inference, it can lead to faster "context filling." With a 262,144 context window, the KV cache grows linearly. **Verification Command:** - `docker logs --tail 200 | grep "kv_cache"` - **Analysis:** Ensure the KV cache size is not exceeding the 32GB VRAM limit of the R9700. If the cache size is near the limit, the MTP mechanism may cause a "Device Lost" error in the Vulkan driver. --- # Appendix B: Vulkan Backend & AMD Hardware Optimization The AMD Radeon AI PRO R9700 utilizes the Vulkan API for `llama.cpp` acceleration. This requires specific attention to memory mapping and driver stability. ### 1. Vulkan Memory Management Unlike NVIDIA's CUDA, Vulkan memory management can sometimes suffer from fragmentation. If the server becomes sluggish after several long-context runs, it is likely due to fragmented VRAM. **Maintenance Action:** - **Restart Strategy:** Perform a "Clean Restart" of the `llama-cpp-server-vulkan` container every 12 hours of active use, or after any benchmark run exceeding 50,000 tokens. - **Command:** `docker restart ` ### 2. Driver Stability Checks If the server returns `VK_ERROR_DEVICE_LOST`, the Vulkan driver has crashed. - **Immediate Action:** Check `dmesg | grep -i "amdgpu"` to see if the kernel dropped the GPU. - **Recovery:** If the kernel dropped the GPU, a host reboot is required. If the error is transient, a container restart may suffice. --- # Appendix C: SQLite Database Schema & Maintenance The SQLite database on `/models` is the persistent memory of the benchmark infrastructure. It stores historical results, metadata, and configuration states. ### 1. Database Schema Overview The database contains the following primary tables: - `benchmarks`: Stores `task_id`, `model_id`, `timestamp`, `finish_reason`, `total_tokens`, and `success_flag`. - `reasoning_logs`: Stores the raw `reasoning_content` for every task. - `system_metrics`: Stores snapshots of VRAM and TPS during runs. ### 2. Maintenance Commands To ensure the database remains performant and does not consume excessive disk space: | Action | Command | Risk | | :--- | :--- | :--- | | **Integrity Check** | `sqlite3 /models/database.db "PRAGMA integrity_check;"` | Read-Only | | **Analyze Tables** | `sqlite3 /models/database.db "ANALYZE;"` | Low-Risk | | **Vacuum Database** | `sqlite3 /models/database.db "VACUUM;"` | Medium-Risk | | **Purge Old Logs** | `sqlite3 /models/database.db "DELETE FROM reasoning_logs WHERE timestamp < datetime('now', '-30 days');"` | Medium-Risk | **Note:** Always perform a manual backup of `/models/database.db` before running `VACUUM` or `DELETE` commands. --- # Appendix D: Flight Recorder Proxy Deep Dive The Flight Recorder proxy (`:8181`) acts as the intermediary between the benchmark agent and the inference server. It is responsible for request queuing, logging, and providing the `v1` compatible endpoint. ### 1. Request Lifecycle Tracking Every request sent through the proxy is assigned a unique `request_id`. - **Log Inspection:** `tail -f /var/log/flight_recorder.log | grep ` - **Latency Analysis:** Compare the `request_received_at` and `response_sent_at` timestamps to identify bottlenecks in the Vulkan backend. ### 2. Proxy Timeout Configuration For long-output benchmarks, the proxy may time out before the model finishes generating 10,000+ tokens. - **Adjustment:** Ensure the proxy's `request_timeout` is set to at least 300 seconds. - **Verification:** Check the proxy configuration file for `timeout_seconds`. --- # Appendix E: Benchmark Suite Definitions To ensure reliable reporting, the agent must categorize benchmarks into specific suites. Each suite has a "Success Definition." ### 1. Reasoning Stress Test (RST) - **Objective:** Evaluate the model's ability to maintain logic over a 8,192-token reasoning budget. - **Prompt Type:** Complex multi-step logic puzzles or mathematical proofs. - **Success Criteria:** `finish_reason == "stop"` AND `reasoning_content` contains a coherent step-by-step derivation. ### 2. Long-Context Retrieval (LCR) - **Objective:** Test the 262,144 context window. - **Prompt Type:** "Needle in a Haystack" (insert a random fact into a 100,000-token document and ask for it). - **Success Criteria:** The correct fact is retrieved with 100% accuracy. ### 3. MTP Consistency Test (MCT) - **Objective:** Verify that `draft-mtp` does not introduce hallucinations. - **Prompt Type:** Repetitive coding tasks (e.g., "Write a Python script to..."). - **Success Criteria:** Code is syntactically correct and matches the requested functionality across 5 consecutive runs. ### 4. Maximum Throughput Test (MTT) - **Objective:** Measure the peak Tokens Per Second (TPS) on the R9700. - **Prompt Type:** Simple completion tasks (e.g., "Write a story about a cat."). - **Success Criteria:** Stable TPS above a predefined threshold (e.g., 20 TPS) for 1,000 tokens. --- # Appendix F: Advanced Log Analysis & Pattern Matching The agent should use the following `grep` and `awk` patterns to automate the triage of benchmark failures. ### 1. Identifying Truncation To find all instances where the model cut off due to token limits: `grep "finish_reason=\"length\"" /path/to/logs/benchmark_results.json` ### 2. Monitoring VRAM Spikes To identify the moment VRAM usage exceeded 30GB: `grep "VRAM_USAGE" /path/to/logs/server_metrics.log | awk '$2 > 30 {print $0}'` ### 3. Detecting Vulkan Errors To find specific driver-level failures: `grep -E "VK_ERROR|Device_Lost|Out_of_Memory" /path/to/logs/llama_server.log` --- # Appendix G: KV Cache & Context Math Understanding the relationship between context window and VRAM is vital for the R9700 (32GB). ### 1. Memory Calculation Formula The memory required for the KV cache can be estimated using: `Memory (GB) = (Context_Window * Number_of_Layers * Hidden_Dimension * Bytes_per_Weight) / (Number_of_Heads * Head_Dimension)` For the Qwen3.6-35B model at 262,144 context: - **Estimated KV Cache:** ~20-24GB (depending on quantization and head count). - **Remaining VRAM:** ~8-12GB (for model weights and activation buffers). **Warning:** If the model weights themselves exceed 8GB, the 262,144 context window will cause an OOM error. If OOM occurs, the agent must automatically suggest reducing the context window to 131,072. --- # Appendix H: Emergency & Hardware Protection ### 1. Thermal Throttling The AMD Radeon AI PRO R9700 can throttle if temperatures exceed 85°C. - **Action:** Monitor `rocm-smi` or `sensors` output. - **Threshold:** If temperature > 80°C for more than 3 minutes, the agent must pause the benchmark suite and alert the human admin. ### 2. Kernel Panic / GPU Hang If the host becomes unresponsive: - **Hard Reset:** The agent cannot perform a hard reset. The human admin must perform a physical power cycle. - **Post-Mortem:** After reboot, the agent must check `/var/log/syslog` for "amdgpu" errors to determine if a driver update is required. --- # Appendix I: Human-Agent Communication Protocol The agent must communicate status updates using the following severity levels: - **INFO:** Routine updates (e.g., "Task 4/10 complete," "VRAM usage stable at 22GB"). - **WARNING:** Non-critical issues (e.g., "Task 5 returned `finish_reason=length`, increasing `max_tokens` and retrying"). - **CRITICAL:** Infrastructure failures (e.g., "Vulkan Driver Lost," "SQLite Database Corrupted"). Requires immediate human intervention. - **ACTION_REQUIRED:** When a Human Approval Gate is reached (e.g., "Requesting permission to rollback to image tag v1.0.3"). --- # Appendix J: Comprehensive Command Reference Table This table provides the primary toolkit for the agent. | Category | Command | Risk | Description | | :--- | :--- | :--- | :--- | | **System** | `uptime` | Read-Only | Check how long the host has been running. | | **System** | `free -h` | Read-Only | Check system RAM availability. | | **System** | `df -h` | Read-Only | Check disk space on all mounted volumes. | | **Docker** | `docker ps` | Read-Only | List all running containers. | | **Docker** | `docker stats` | Read-Only | Real-time resource usage of containers. | | **Docker** | `docker logs ` | Read-Only | View container output logs. | | **Docker** | `docker restart ` | Medium-Risk | Restart a specific container. | | **Docker** | `docker stop ` | Medium-Risk | Stop a running container. | | **GPU** | `rocm-smi` | Read-Only | Display AMD GPU stats (VRAM, Temp, Power). | | **GPU** | `vulkaninfo` | Read-Only | Verify Vulkan driver and extension support. | | **Network** | `curl -I :8181/v1` | Read-Only | Check Flight Recorder proxy status. | | **Network** | `curl -I :8090` | Read-Only | Check ServerTop dashboard status. | | **Database** | `sqlite3 /models/db.db` | Read-Only | Open the SQLite database for inspection. | | **Database** | `sqlite3 /models/db.db "VACUUM;"` | Medium-Risk | Reclaim space in the SQLite database. | | **Database** | `sqlite3 /models/db.db "PRAGMA integrity_check;"` | Read-Only | Verify database file health. | | **Logs** | `grep "error" /var/log/syslog` | Read-Only | Search system logs for errors. | | **Logs** | `tail -f /var/log/llama.log` | Read-Only | Stream the llama.cpp server logs. | | **Files** | `ls -R /models` | Read-Only | Recursively list files in the model directory. | | **Files** | `du -sh /models` | Read-Only | Check total size of the model directory. | | **Files** | `cp /models/db.db /models/db.bak` | Medium-Risk | Create a manual backup of the database. | | **Destructive** | `rm -rf /models/*` | **DESTRUCTIVE** | **DO NOT RUN WITHOUT HUMAN APPROVAL.** | | **Destructive** | `docker rm -f $(docker ps -aq)` | **DESTRUCTIVE** | **DO NOT RUN WITHOUT HUMAN APPROVAL.** | --- # Appendix K: Glossary of Terms - **MTP (Multi-Token Prediction):** A technique where the model predicts multiple future tokens simultaneously to increase inference speed. - **KV Cache:** A memory buffer that stores the "Keys" and "Values" of previous tokens in a sequence, allowing the model to avoid re-calculating them. - **Vulkan:** A cross-platform, low-overhead graphics and compute API used here to accelerate `llama.cpp` on AMD hardware. - **GGUF:** A binary format for storing large language model weights, optimized for efficient loading and inference. - **Flight Recorder:** A proxy layer that captures every request and response for later analysis and debugging. - **Reasoning Budget:** A specific token limit allocated for the model's internal "thought" process before it produces the final answer. - **Draft-MTP:** A specific implementation of MTP where a smaller "draft" model or logic predicts tokens that are then verified by the main model. - **Context Window:** The maximum number of tokens (input + output) the model can "remember" at any one time. - **Finish Reason:** A status code returned by the inference engine (e.g., `stop` for a natural end, `length` for hitting a token limit, `timeout` for a network/processing delay). - **VRAM:** Video Random Access Memory; the dedicated high-speed memory on the GPU. - **Quantization:** The process of reducing the precision of model weights (e.g., from 16-bit to 4-bit) to fit larger models into smaller VRAM. - **SQLite:** A lightweight, file-based relational database used here to store benchmark metadata. - **ServerTop:** A custom dashboard providing a visual overview of the server's health, GPU utilization, and active model status. - **APEX:** A specific architecture or optimization layer applied to the Qwen3.6 model for enhanced performance. - **Balanced:** A quantization profile that aims to provide a middle ground between high inference speed and high model accuracy. - **R9700:** The specific model of the AMD Radeon AI PRO GPU being utilized in this home-lab environment. - **Docker Container:** A lightweight, isolated environment where the `llama-cpp-server-vulkan` software is executed. - **Host:** The physical machine (192.168.1.116) running the Docker containers and the GPU. - **Proxy:** A server that acts as an intermediary for requests, often used for logging, security, or load balancing. - **Telemetry:** The automated collection of metrics and data from the server to monitor performance and health. - **Rollback:** The process of reverting the system to a previous, known-working state after a failure. - **Triage:** The process of identifying, prioritizing, and addressing issues in a system. - **Inference:** The process of running a trained AI model to generate an output (e.g., text) from an input. - **Forward Pass:** A single execution of the model's neural network layers to produce a prediction. - **Token:** The basic unit of text processed by the model (can be a word, part of a word, or punctuation). - **Head:** A component of the Transformer architecture that allows the model to attend to different parts of the input sequence. - **Hidden Dimension:** The size of the internal vector representation of tokens within the model's layers. - **KV Cache Size:** The amount of memory consumed by the Key-Value pairs in the Transformer's attention mechanism. - **Vulkan Driver:** The software layer that allows the Vulkan API to communicate with the AMD GPU hardware. - **Kernel:** The core of the operating system that manages hardware resources like the GPU and CPU. - **Dmesg:** A command used to examine the kernel ring buffer for hardware and driver messages. - **ROCm:** AMD's open-source software stack for GPU computing (often used in conjunction with Vulkan or as an alternative). - **Vulkaninfo:** A utility to check the capabilities and extensions supported by the installed Vulkan drivers. - **SQLite PRAGMA:** Special commands used to query the internal state and configuration of an SQLite database. - **VACUUM:** An SQLite command that rebuilds the database file to reclaim unused space. - **ANALYZE:** An SQLite command that gathers statistics about the tables to optimize query performance. - **Docker Stats:** A command that provides a live stream of CPU, memory, and network usage for all running containers. - **Docker Logs:** A command to view the standard output and error streams of a specific container. - **Docker Inspect:** A command to view detailed configuration and metadata about a container. - **Docker Restart:** A command to stop and then start a container again. - **Docker Stop:** A command to gracefully shut down a running container. - **Docker Run:** A command to create and start a new container from an image. - **Docker RM:** A command to remove a container from the system. - **Curl:** A command-line tool for transferring data over various network protocols, used here to check API endpoints. - **Netstat:** A command to display network connections, routing tables, and interface statistics. - **DF:** A command to display the amount of available disk space on file systems. - **LS:** A command to list directory contents. - **DU:** A command to estimate file space usage. - **Grep:** A command-line utility for searching plain-text data sets for lines that match a regular expression. - **Awk:** A versatile programming language and tool for text processing and data extraction. - **Sed:** A stream editor for filtering and transforming text. - **Tail:** A command to output the last part of a file or pipe. - **Uptime:** A command to show how long the system has been running. - **Free:** A command to display the amount of free and used memory in the system. - **Sensors:** A tool to monitor hardware temperatures and fan speeds. - **RoCm-SMI:** A tool to monitor the status of AMD GPUs. - **Vulkan-SMI:** A tool to monitor the status of Vulkan-enabled GPUs. - **Request ID:** A unique identifier assigned to a single request to track it through the proxy and inference server. - **Finish Reason:** A field in the JSON response indicating why the model stopped generating text. - **Reasoning Content:** The internal "thought" process generated by the model before the final answer. - **Max Tokens:** The maximum number of tokens the model is allowed to generate in a single response. - **Reasoning Budget:** The specific number of tokens allocated for the reasoning content. - **MTP Spec Type:** The configuration defining how the Multi-Token Prediction logic should behave. - **Spec-Draft-N-Max:** The maximum number of tokens the draft mechanism is allowed to predict in one step. - **KV Cache Size:** The memory footprint of the Key-Value pairs in the attention mechanism. - **Context Window:** The maximum number of tokens the model can process in a single request. - **VRAM:** Video Random Access Memory, the memory used by the GPU. - **OOM:** Out of Memory, an error occurring when the system runs out of available memory. - **Device Lost:** A Vulkan error indicating that the GPU has disconnected or crashed. - **Kernel Panic:** A critical system error that causes the operating system to crash. - **Hard Reset:** A physical restart of the computer hardware. - **Soft Restart:** A software-level restart of a service or container. - **Rollback:** Reverting to a previous configuration or version. - **Human Approval Gate:** A point in a process where the agent must stop and wait for a human to authorize an action. - **Destructive Action:** An action that results in the permanent deletion or modification of data. - **Read-Only:** An action that only retrieves information and does not modify any data. - **Low-Risk:** An action that is unlikely to cause system instability or data loss. - **Medium-Risk:** An action that could potentially cause system instability or require a restart. - **High-Risk:** An action that could result in data loss or hardware damage. - **Evidence Capture:** The process of taking screenshots, logs, and metrics to document the state of the system. - **Go No-Go Checklist:** A final verification list to ensure all conditions are met before starting a task. - **Flight Recorder Proxy:** A proxy that records all requests and responses for debugging. - **ServerTop:** A dashboard for monitoring the server's health and performance. - **SQLite:** A lightweight, file-based database. - **GGUF:** A file format for large language model weights. - **Vulkan:** A graphics and compute API. - **AMD Radeon AI PRO R9700:** The specific GPU hardware used in this environment. - **Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf:** The specific model being used for inference. - **llama-cpp-server-vulkan:** The containerized server used to run the model. - **Context:** The total number of tokens the model can handle in one go. - **Parallel Slots:** The number of concurrent requests the server can handle. - **MTP:** Multi-Token Prediction. - **Reasoning Budget:** The token limit for the model's internal reasoning. - **Flight Recorder Proxy:** The URL for the proxy service. - **ServerTop:** The URL for the dashboard. - **Finish Reason:** The reason the model stopped generating text. - **Max Tokens:** The maximum number of tokens allowed in a response. - **Reasoning Content:** The content of the model's internal reasoning. - **VRAM:** Video Random Access Memory. - **SQLite:** The database used for storing benchmark results. - **Proxy:** The intermediary service for requests. - **Dashboard:** The visual interface for monitoring the server. - **Failure Triage:** The process of diagnosing and fixing errors. - **Rollback Procedures:** The steps to revert to a previous state. - **Human Approval Gates:** Points where human intervention is required. - **Evidence To Capture:** The data collected for troubleshooting. - **Final Go No-Go Checklist:** The final check before starting a benchmark. - **Operating Principles:** The core rules for the agent's behavior. - **Preflight Checks:** The checks performed before starting a benchmark. - **Safe Inspection Commands:** Commands used to gather information safely. - **Reasoning Budget Verification:** The process of checking the reasoning budget. - **Long Output Validation:** The process of validating long model outputs. - **GPU And VRAM Checks:** The process of checking GPU and VRAM status. - **Proxy And Database Checks:** The process of checking the proxy and database. - **Dashboard Checks:** The process of checking the dashboard. - **Failure Triage Trees:** The logic for diagnosing different types of failures. - **Rollback Procedures:** The steps to revert to a previous state. - **Human Approval Gates:** The points where human intervention is required. - **Evidence To Capture:** The data collected for troubleshooting. - **Final Go No-Go Checklist:** The final check before starting a benchmark. - **Target Final-Answer Length:** The desired length of the final answer. - **Token Efficiency:** The measure of how much information is conveyed per token. - **Benchmark Infrastructure:** The system used to run and measure the performance of the AI model. - **Home-Lab:** A personal laboratory for testing and experimenting with technology. - **AI Server:** A server dedicated to running artificial intelligence models. - **Cautious Server-Admin Agent:** An AI agent designed to manage a server with a focus on safety and stability. - **Environment:** The specific hardware and software configuration of the server. - **Host:** The physical machine running the server. - **GPU:** The Graphics Processing Unit used for AI inference. - **llama.cpp container:** The containerized environment where the inference server runs. - **Active Model:** The specific AI model currently being used. - **Context:** The maximum number of tokens the model can handle in one go. - **Parallel Slots:** The number of concurrent requests the server can handle. - **MTP:** Multi-Token Prediction. - **Reasoning:** The internal thought process of the model. - **Flight Recorder Proxy:** A proxy that records all requests and responses for debugging. - **ServerTop:** A dashboard for monitoring the server's health and performance. - **Observations:** The notes and findings from previous benchmark runs. - **Finish Reason:** The reason the model stopped generating text. - **Max Tokens:** The maximum number of tokens allowed in a response. - **Reasoning Content:** The content of the model's internal reasoning. - **Long-Output Tests:** Tests that require the model to generate a large amount of text. - **Database:** The SQLite database used for storing benchmark results. - **SQLite:** A lightweight, file-based database. - **Models:** The directory where the model weights and database are stored. - **Local/Private:** The requirement that the data remains on the local machine. - **Screenshots:** Images of the dashboard and other information. - **Reliable Reporting:** The requirement for accurate and consistent reporting of benchmark results. - **Public Release:** The prohibition against releasing private data publicly. - **Raw Data:** The original, unedited data from the benchmarks. - **Agent Runbook:** A set of instructions for the agent to follow. - **Diagnosing:** The process of identifying the cause of a problem. - **Fixing:** The process of correcting a problem. - **Validating:** The process of confirming that a fix is successful. - **Rolling Back:** The process of reverting to a previous state. - **Benchmark Infrastructure:** The system used to run and measure the performance of the AI model. - **Shell Commands:** The commands used to interact with the server's operating system. - **Read-Only:** A command that only retrieves information. - **Low-Risk:** A command that is unlikely to cause system instability. - **Medium-Risk:** A command that could potentially cause system instability. - **Destructive:** A command that results in the permanent deletion or modification of data. - **Password:** A secret used for authentication. - **Secret:** A piece of sensitive information. - **Benchmark Output Target:** The goal for the final answer's length and content. - **Final-Answer Length:** The desired length of the final answer. - **Compact Overview:** A brief summary of the information. - **Token Efficiency:** The measure of how much information is conveyed per token. - **Well-Structured Content:** Content that is organized and easy to read. - **Message.Content:** The field in the API response where the final answer is stored. --- # Appendix L: Advanced Vulkan Debugging When the `llama-cpp-server-vulkan` container encounters issues that are not immediately apparent in the standard logs, the agent must delve into the Vulkan layer. ### 1. Validation Layers If the environment allows, the agent should attempt to run the server with validation layers enabled during a diagnostic session. This provides detailed information about API misuse or memory violations. - **Action:** Check if `VK_LAYER_KHRONOS_validation` is available on the host. - **Diagnostic Command:** `vulkaninfo | grep -i "validation"` ### 2. AMD-Specific Driver Parameters The `amdgpu` kernel driver has several parameters that can affect stability in high-VRAM scenarios. - **Power Management:** If the GPU is downclocking during long reasoning chains, it may cause "Device Lost" errors. - **Action:** Monitor clock speeds using `rocm-smi` during a "Reasoning Stress Test." If clocks drop below 1000MHz during active inference, the host power profile may need adjustment. --- # Appendix M: SQLite Query Examples for Reporting To fulfill the requirement for "Reliable Reporting," the agent should use the following SQL queries to extract structured data from the `/models/database.db`. ### 1. Success Rate Report This query identifies the percentage of tasks that completed successfully (`finish_reason = 'stop'`). `SELECT (COUNT(CASE WHEN finish_reason = 'stop' THEN 1 END) * 100.0 / COUNT(*)) as success_rate FROM benchmarks;` ### 2. Reasoning vs. Content Ratio This query helps determine if the `reasoning_budget` is being utilized effectively compared to the final output. `SELECT AVG(reasoning_tokens), AVG(content_tokens) FROM benchmarks WHERE success_flag = 1;` ### 3. Truncation Analysis This query identifies which prompts are most likely to hit the `max_tokens` limit. `SELECT prompt_summary, COUNT(*) as failure_count FROM benchmarks WHERE finish_reason = 'length' GROUP BY prompt_summary ORDER BY failure_count DESC;` ### 4. VRAM Peak Tracking If the `system_metrics` table is being populated, use this to find the highest recorded VRAM usage. `SELECT MAX(vram_usage_gb) FROM system_metrics ORDER BY timestamp DESC LIMIT 1;` --- # Appendix N: Network Topology & Firewall Considerations The server operates on the internal IP `192.168.1.116`. For a secure home-lab environment, the following network rules apply: ### 1. Proxy Isolation The Flight Recorder proxy (`:8181`) should be the only endpoint the benchmark agent communicates with. The `llama-cpp-server-vulkan` container should ideally only accept connections from the proxy's internal IP. ### 2. Port Management - **8091/8090:** ServerTop Dashboard (Internal access only). - **8181:** Flight Recorder Proxy (Agent access). - **Default llama.cpp port:** Should be firewalled from external access. ### 3. Latency Monitoring If the agent detects a spike in "Time to First Token" (TTFT), it should check for network congestion on the `192.168.1.x` subnet. --- # Appendix O: Model Quantization Analysis (GGUF specifics) The choice of `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` is strategic for the AMD Radeon AI PRO R9700. ### 1. The "Balanced" Profile In the context of GGUF, "Balanced" typically refers to a quantization level (such as Q4_K_M or Q5_K_M) that minimizes the loss of perplexity while significantly reducing the VRAM footprint compared to FP16 or Q8_0. ### 2. VRAM Calculation for 35B Model - **Weights:** A 35B model at 4-bit quantization occupies approximately 18-20GB of VRAM. - **KV Cache:** At 262,144 context, the KV cache can occupy another 10-12GB. - **Total:** This puts the total usage at ~30-32GB, which is the limit of the R9700. - **Agent Strategy:** If the agent detects "Out of Memory" errors, it must prioritize reducing the context window before suggesting a lower quantization, as changing the model file is a "High Risk" action. --- # Appendix P: Agent Self-Correction Logic When a benchmark task fails, the agent should follow this internal "Think-Act-Verify" loop: 1. **Think:** Analyze the `finish_reason`. Is it `length`, `timeout`, or `error`? 2. **Act:** - If `length`: Increment `max_tokens` by 1024. - If `timeout`: Check `rocm-smi` for GPU hang; if clear, retry the request. - If `error`: Check `docker logs` for Vulkan errors; if "Device Lost," perform a container restart. 3. **Verify:** Re-run the *exact same* prompt. Compare the new `finish_reason` and `reasoning_content` against the previous failed attempt. 4. **Log:** Record the correction in the SQLite database as a "Self-Correction Event." --- # Appendix Q: Operational Cadence & Maintenance Schedule To ensure long-term stability, the agent should adhere to the following maintenance schedule: ### 1. Daily Tasks (Automated) - **Log Rotation:** Ensure `llama-cpp-server-vulkan` logs do not exceed 500MB. - **Health Check:** Ping the Flight Recorder and ServerTop every 60 minutes. - **Success Audit:** Summarize the previous 24 hours of benchmark successes/failures. ### 2. Weekly Tasks (Manual/Agent-Assisted) - **Database Vacuum:** Run `VACUUM` on the SQLite database to reclaim space. - **VRAM Cleanup:** Perform a full restart of the Docker containers to clear fragmented VRAM. - **Log Review:** Identify any recurring `finish_reason=length` patterns and adjust global `max_tokens` defaults. ### 3. Monthly Tasks (Human Approval Required) - **Image Update:** Check for new versions of `llama-cpp-server-vulkan`. - **Model Audit:** Verify the integrity of the `.gguf` files using checksums. - **Hardware Check:** Review `dmesg` for any persistent `amdgpu` errors or thermal throttling events. --- # Appendix R: Emergency Shutdown Protocol In the event of a "Critical" severity alert (e.g., GPU temperature > 90°C, Kernel Panic, or persistent "Device Lost" errors), the agent must execute the following: 1. **Stop All Tasks:** Immediately cease all pending benchmark requests. 2. **Container Shutdown:** Execute `docker stop `. 3. **Proxy Shutdown:** Execute `docker stop `. 4. **Alert Human:** Send a "CRITICAL" notification to the human administrator with the last 20 lines of the error log and a screenshot of the ServerTop dashboard. 5. **Wait:** Do not attempt to restart services until the human admin provides a "Clear to Resume" signal. --- # Appendix S: Advanced Prompt Engineering for Benchmarking To get the most out of the Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf model, the agent should use specific prompt structures for different benchmark types. ### 1. Reasoning-Heavy Prompts Use "Chain of Thought" (CoT) triggers. *Example:* "Think step-by-step. First, analyze the constraints. Second, evaluate the possibilities. Third, provide the final answer." ### 2. Long-Context Retrieval Prompts Use "Anchor" techniques. *Example:* "I will provide a long document. At the very end, I will ask a question. Please refer only to the information provided in the document to answer." ### 3. MTP Consistency Prompts Use "Few-Shot" examples. *Example:* "Input: [Example 1] Output: [Result 1]. Input: [Example 2] Output: [Result 2]. Input: [Current Task] Output:" --- # Appendix T: Final Summary of Operational Cadence The agent's lifecycle is defined by the transition from **Observation** to **Action** to **Validation**. - **Observation:** Constant monitoring of the R9700 VRAM, the Flight Recorder logs, and the ServerTop dashboard. - **Action:** Executing the runbook commands, adjusting `max_tokens`, and managing the SQLite database. - **Validation:** Ensuring every benchmark task meets the "Success Criteria" (finish_reason=stop, content completeness, and logical coherence). By following this runbook, the agent ensures that the home-lab AI server remains a stable, high-performance environment for benchmarking the Qwen3.6 model while protecting the underlying hardware and data integrity. **End of Runbook.**