# Home-Lab AI Server Administration Runbook: Benchmark Infrastructure & Stability

**Version:** 1.0.0-STABLE
**Role:** Cautious Server-Admin Agent
**Environment:** 192.168.1.116 | AMD Radeon AI PRO R9700 (32GB) | llama.cpp (Vulkan)
**Target Model:** Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf

---

# Operating Principles

The primary objective of this runbook is to maintain a stable, private, and high-performance benchmarking environment while preventing data leakage and avoiding catastrophic hardware or software failure. As a "Cautious Admin," the following principles govern every action:

1.  **Principle of Least Privilege (PoLP):** Never run commands as root unless absolutely necessary. Use the most restrictive permission set possible for database access and container management.
2.  **Read-Only First:** Before attempting any "Fix," a "Diagnosis" phase must be completed using read-only commands. No configuration changes are permitted until the root cause is identified and logged.
3.  **The "Measure Twice, Cut Once" Rule:** Every change to the `llama.cpp` startup flags (especially context size or MTP settings) must be documented in a change log before execution.
4.  **Privacy Preservation:** The SQLite database located on `/models` contains proprietary benchmark data. No raw database dumps or logs containing prompt/response pairs are to be transmitted outside the local network (192.168.1.0/24).
5.  **Incremental Validation:** When fixing `finish_reason=length` issues, changes to `max_tokens` or `reasoning_budget` must be tested with a single, controlled prompt before being applied to a bulk benchmark suite.
6.  **Hardware Conservatism:** The AMD Radeon AI PRO R9700 is a high-performance asset. Avoid overclocking or pushing VRAM to 99% saturation for extended periods to prevent thermal throttling or driver crashes (Vulkan timeouts).
7.  **Evidence-Based Reporting:** Every failure must be accompanied by a "Evidence Bundle" (Logs + Screenshot + Request/Response JSON) before a "Go" decision is made for a new benchmark run.

---

# Preflight Checks

Before initiating any benchmark or diagnostic session, the following preflight checks must be executed to ensure the environment is in a known-good state.

### 1. Network Connectivity Matrix
Verify that all internal components can communicate.

| Target | Port | Purpose | Expected Result |
| :--- | :--- | :--- | :--- |
| 192.168.1.116 | 8080 | llama.cpp Server | HTTP 200 OK |
| 192.168.1.116 | 8181 | Flight Recorder Proxy | HTTP 200 OK |
| 192.168.1.116 | 8090 | ServerTop Dashboard | HTTP 200 OK |

### 2. Container Health Check
Ensure the specific image `llama-cpp-server-vulkan:working-20260613` is running and not in a restart loop.

**Command:** `docker ps --filter "name=llama-cpp-server"`
**Risk Level:** Read-only
**Verification:** Check that the status is `Up` and the image ID matches the working version.

### 3. Model Load Verification
Confirm the correct GGUF is loaded into VRAM.

**Command:** `curl -s http://192.168.1.116:8080/v1/models`
**Risk Level:** Read-only
**Verification:** Ensure `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` is listed as the active model.

### 4. Storage Availability
Verify that the `/models` mount has sufficient space for the SQLite database and temporary logs.

**Command:** `df -h /models`
**Risk Level:** Read-only
**Verification:** Ensure available space is $> 10\%$ of total capacity to prevent SQLite write-locks.

---

# Safe Inspection Commands

These commands are used for diagnosis. They do not alter the state of the system.

### System & Container Inspection
| Command | Risk Level | Purpose |
| :--- | :--- | :--- |
| `docker stats llama-cpp-server` | Read-only | Monitor real-time CPU/RAM/Network usage of the container. |
| `docker logs --tail 100 llama-cpp-server` | Read-only | Inspect the last 100 lines of server output for Vulkan errors. |
| `uptime` | Read-only | Check system load and stability duration. |
| `free -m` | Read-only | Check system RAM availability (outside of VRAM). |

### API & Proxy Inspection
| Command | Risk Level | Purpose |
| :--- | :--- | :--- |
| `curl -v http://192.168.1.116:8181/v1/health` | Read-only | Verify Flight Recorder proxy is routing requests. |
| `curl -X GET http://192.168.1.116:8080/health` | Read-only | Check llama.cpp internal health status. |
| `netstat -tulpn \| grep -E '8080\|8181\|8090'` | Read-only | Confirm all required ports are listening. |

### Database Inspection (SQLite)
*Note: Use these commands only to verify structure or record counts, never to modify data.*

| Command | Risk Level | Purpose |
| :--- | :--- | :--- |
| `sqlite3 /models/benchmarks.db "PRAGMA integrity_check;"` | Read-only | Verify the database is not corrupted. |
| `sqlite3 /models/benchmarks.db "SELECT count(*) FROM results;"` | Read-only | Check the number of recorded benchmark entries. |
| `sqlite3 /models/benchmarks.db ".tables"` | Read-only | List all tables to ensure schema consistency. |

---

# Reasoning Budget Verification

The `Qwen3.6` model utilizes a reasoning budget (`--reasoning-budget 8192`). A common failure mode is the `finish_reason=length` error, where the reasoning content consumes the total token limit, leaving no room for the final answer.

### The "Budget vs. Limit" Logic
- **Reasoning Budget:** The maximum tokens the model can spend "thinking" (internal monologue).
- **Max Tokens:** The hard limit on the total response (Reasoning + Final Answer).
- **The Conflict:** If `max_tokens` is set to 8192 and the model uses 8190 tokens for reasoning, the final answer will be truncated after 2 tokens, resulting in `finish_reason=length`.

### Verification Protocol
To verify if the reasoning budget is causing truncation, execute the following test:

1.  **The "Deep Thought" Prompt:** Send a prompt that explicitly requires complex reasoning (e.g., "Provide a step-by-step mathematical proof for the Riemann Hypothesis, thinking deeply about each step").
2.  **The Request Configuration:**
    - `max_tokens`: 4096 (Intentionally low)
    - `reasoning_budget`: 8192
3.  **Observation:** If the response ends abruptly during the reasoning phase with `finish_reason=length`, the budget is functioning, but the `max_tokens` limit is too restrictive.

### Fix Implementation
If truncation is detected, the `max_tokens` parameter in the benchmark script must be increased to:
`max_tokens = reasoning_budget + expected_final_answer_length` (e.g., $8192 + 2048 = 10240$).

**Command to update benchmark config (Example):**
`sed -i 's/"max_tokens": [0-9]*/"max_tokens": 12288/' /models/config/benchmark_settings.json`
**Risk Level:** Medium-risk (Modifies configuration)

---

# Long Output Validation

Long-output tests are prone to failure due to context window saturation or VRAM fragmentation. Bulk runs are forbidden until a single-task validation is successful.

### Validation Workflow
1.  **Single-Task Isolation:** Select one prompt known to generate $> 2000$ tokens of output.
2.  **Parameter Set:**
    - `max_tokens`: 16384
    - `reasoning_budget`: 8192
3.  **Execution:** Run the prompt through the Flight Recorder proxy.
4.  **Verification Criteria:**
    - `finish_reason` must be `stop`.
    - The final answer must contain a concluding sentence/closing bracket.
    - The `reasoning_content` must be fully present in the logs.

### Bulk Run Safety Gate
Bulk runs may only proceed if:
- $\text{Success Rate of Long Output Validation} = 100\%$ over 5 consecutive trials.
- VRAM usage remains stable (no "sawtooth" pattern in ServerTop).
- The SQLite database is not experiencing `SQLITE_BUSY` errors.

---

# GPU And VRAM Checks

The AMD Radeon AI PRO R9700 (32GB) is the core of the system. Vulkan acceleration is used via `llama-cpp-server-vulkan`.

### VRAM Monitoring
VRAM must be monitored during the "Reasoning" phase, as MTP (Multi-Token Prediction) and large context windows (262k) can cause sudden spikes.

**Command:** `rocm-smi`
**Risk Level:** Read-only
**Key Metrics to Watch:**
- **VRAM Usage:** Should not exceed 30GB (leave 2GB for OS/Driver overhead).
- **GPU Temperature:** Should remain below $85^\circ\text{C}$.
- **Power Draw:** Ensure it is within the TDP limits of the PRO R9700.

### Vulkan Driver Diagnostics
If the server crashes with a "Vulkan Device Lost" or "Segmentation Fault," check the kernel logs.

**Command:** `dmesg \| grep -iE 'amdgpu\|vulkan\|drm'`
**Risk Level:** Read-only
**Warning Signs:**
- `amdgpu: [drm] ERROR`
- `ring gfx_0.0.0 timeout`
- `Vulkan Error: VK_ERROR_DEVICE_LOST`

### VRAM Recovery Procedure
If VRAM is fragmented or "leaking" (not releasing after a request):
1.  **Stop Container:** `docker stop llama-cpp-server` (Low-risk)
2.  **Clear Cache:** `sync; echo 3 > /proc/sys/vm/drop_caches` (Medium-risk, requires sudo)
3.  **Restart Container:** `docker start llama-cpp-server` (Low-risk)

---

# Proxy And Database Checks

The Flight Recorder proxy (8181) and the SQLite database on `/models` are the primary data ingestion points.

### Proxy Latency Analysis
If benchmarks are slow, determine if the bottleneck is the LLM or the Proxy.

**Test:**
1.  `curl -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total: %{time_total}\n" -o /dev/null -s http://192.168.1.116:8080/v1/models` (Direct)
2.  `curl -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total: %{time_total}\n" -o /dev/null -s http://192.168.1.116:8181/v1/models` (Via Proxy)

**Analysis:** If the "Total" time for the proxy is $> 100\text{ms}$ higher than the direct call, the proxy is the bottleneck.

### SQLite Database Maintenance
Since the DB is on `/models`, it is persistent. Large benchmark runs can bloat the DB, slowing down reporting.

**Maintenance Commands:**
| Command | Risk Level | Purpose |
| :--- | :--- | :--- |
| `sqlite3 /models/benchmarks.db "VACUUM;"` | Medium-risk | Rebuilds the DB file to reclaim space and defragment. |
| `sqlite3 /models/benchmarks.db "ANALYZE;"` | Low-risk | Updates statistics for the query planner. |
| `cp /models/benchmarks.db /models/backups/benchmarks_$(date +%F).db` | Low-risk | Create a manual backup before any schema change. |

---

# Dashboard Checks

ServerTop (8090) provides the visual telemetry required for "Cautious" administration.

### Critical Dashboard Metrics
The admin must monitor the following in ServerTop during a benchmark run:
1.  **KV Cache Usage:** With a 262k context, the KV cache can consume massive VRAM. If the "KV Cache %" hits 100%, the server may crash or begin swapping to system RAM (causing a massive performance drop).
2.  **Tokens Per Second (TPS):**
    - **Prompt Processing (PP):** Should be high (Vulkan optimized).
    - **Generation (Gen):** Should be stable. A sudden drop in Gen TPS indicates thermal throttling.
3.  **MTP Efficiency:** Check if the draft tokens are being accepted. If the "Acceptance Rate" is $< 10\%$, the `--spec-draft-n-max 2` setting is providing no benefit and may be adding overhead.

---

# Failure Triage Trees

When a problem occurs, follow the corresponding tree.

### Tree A: "Response is Truncated / finish_reason=length"
1.  **Check `max_tokens` in request.**
    - Is `max_tokens` $\le$ `reasoning_budget`? $\rightarrow$ **FIX:** Increase `max_tokens` to `reasoning_budget + 2048`.
    - Is `max_tokens` $> 12000$? $\rightarrow$ **GO TO STEP 2**.
2.  **Check Server Logs.**
    - Is there a "Context Window Exceeded" error? $\rightarrow$ **FIX:** Reduce prompt length or clear KV cache.
    - No error? $\rightarrow$ **FIX:** Check if the model is hitting a hard stop sequence.

### Tree B: "Server Unresponsive / Connection Refused"
1.  **Check Container Status.**
    - `docker ps` $\rightarrow$ Container is `Exited`? $\rightarrow$ **GO TO STEP 2**.
    - Container is `Up`? $\rightarrow$ **GO TO STEP 3**.
2.  **Inspect Logs.**
    - `docker logs` $\rightarrow$ "Out of Memory" or "Vulkan Device Lost"? $\rightarrow$ **FIX:** Restart container and reduce context size.
3.  **Check Proxy.**
    - `curl http://192.168.1.116:8181/v1/health` $\rightarrow$ Fails? $\rightarrow$ **FIX:** Restart Flight Recorder proxy.

### Tree C: "Performance Degradation (Slow TPS)"
1.  **Check ServerTop.**
    - GPU Temp $> 90^\circ\text{C}$? $\rightarrow$ **FIX:** Check fan curves/cooling.
    - VRAM Usage $> 31\text{GB}$? $\rightarrow$ **FIX:** Reduce parallel slots or context window.
2.  **Check System Load.**
    - `uptime` $\rightarrow$ Load average $> \text{CPU Core Count}$? $\rightarrow$ **FIX:** Kill background processes.

---

# Rollback Procedures

If a configuration change or a container update causes instability, the following rollback steps must be followed in order.

### Level 1: Configuration Rollback (Low Risk)
Revert the `llama.cpp` startup flags to the last known stable set.
1.  Stop container: `docker stop llama-cpp-server`
2.  Restore config: `cp /models/config/stable_flags.txt /models/config/current_flags.txt`
3.  Start container: `docker start llama-cpp-server`

### Level 2: Model Rollback (Medium Risk)
If the `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` is corrupted or performing poorly.
1.  Rename current model: `mv /models/Qwen3.6...gguf /models/Qwen3.6...gguf.bak`
2.  Restore backup: `cp /models/backups/Qwen3.6...gguf /models/Qwen3.6...gguf`
3.  Restart server.

### Level 3: Image Rollback (Medium Risk)
If the `working-20260613` image is unstable.
1.  Stop container: `docker stop llama-cpp-server`
2.  Remove container: `docker rm llama-cpp-server`
3.  Run previous version: `docker run ... llama-cpp-server-vulkan:stable-20260501`

---

# Human Approval Gates

The Agent is **forbidden** from performing the following actions without explicit human approval via the chat interface:

1.  **Destructive Database Actions:** Any command involving `DROP TABLE`, `DELETE FROM`, or `rm /models/*.db`.
2.  **Hardware Modifications:** Any change to GPU clock speeds, power limits, or fan profiles.
3.  **Public Data Export:** Any attempt to `curl` or `scp` data from the `/models` directory to an external IP address.
4.  **Major Version Upgrades:** Updating the `llama.cpp` image to a version not explicitly marked as `working` or `stable`.
5.  **Context Window Expansion:** Increasing context beyond 262,144, as this may cause immediate VRAM OOM.

**Approval Request Format:**
> "REQUEST: [Action]
> REASON: [Diagnosis]
> RISK: [Low/Medium/High]
> IMPACT: [Expected Result]
> Please confirm 'GO' or 'NO-GO'."

---

# Evidence To Capture

For every benchmark failure or system crash, the following "Evidence Bundle" must be compiled before the issue is marked as resolved.

### 1. The Log Bundle
- **Container Logs:** `docker logs llama-cpp-server > failure_logs.txt`
- **Kernel Logs:** `dmesg > kernel_logs.txt`
- **Proxy Logs:** `tail -n 500 /var/log/flight-recorder.log > proxy_logs.txt`

### 2. The Telemetry Bundle
- **VRAM Snapshot:** Output of `rocm-smi` at the moment of failure.
- **ServerTop Screenshot:** A capture of the TPS and KV Cache graphs.
- **System State:** Output of `top -b -n 1` and `free -m`.

### 3. The Request Bundle
- **Input JSON:** The exact prompt and parameters (`max_tokens`, `temperature`, etc.) sent to the proxy.
- **Output JSON:** The raw response from the server, including the `finish_reason` and `usage` statistics.

---

# Final Go No-Go Checklist

Before declaring the benchmark infrastructure "Ready for Production," the admin must check every box.

- [ ] **Connectivity:** All ports (8080, 8181, 8090) are reachable.
- [ ] **Model:** `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` is loaded and responding.
- [ ] **VRAM:** `rocm-smi` shows usage $< 30\text{GB}$ at idle.
- [ ] **Reasoning:** A "Deep Thought" prompt returns `finish_reason=stop` with `max_tokens` set to 12288.
- [ ] **Long Output:** A $> 2000$ token response is generated without truncation.
- [ ] **Database:** `PRAGMA integrity_check` returns `ok`.
- [ ] **Privacy:** No external network routes are active to the `/models` directory.
- [ ] **MTP:** ServerTop confirms draft tokens are being utilized and accepted.
- [ ] **Stability:** System has been up for $> 1$ hour without a Vulkan driver reset.

**Final Status:** `[ GO / NO-GO ]`
**Admin Signature:** `Cautious-Server-Admin-Agent`
**Timestamp:** `YYYY-MM-DD HH:MM:SS`