# Operating Principles

This runbook is engineered for a cautious, safety-first operational posture. The home-lab AI server environment is treated as a production-grade benchmarking platform despite its private nature. Every procedure assumes that infrastructure stability, data privacy, and reproducibility are non-negotiable. The following principles govern all diagnostic, validation, and rollback activities:

1. **Least Privilege & Read-First Posture**: All inspection and diagnostic actions default to read-only operations. Write, modify, or restart actions require explicit classification, justification, and human approval gates. No configuration change is applied without a documented rollback path.
2. **Sequential Validation Over Bulk Execution**: Long-output and reasoning-heavy workloads are validated one task at a time. Bulk benchmark runs are strictly prohibited until individual task validation passes across three consecutive successful executions. This prevents cascading failures, VRAM fragmentation, and silent truncation artifacts from corrupting dataset integrity.
3. **Data Privacy & Local Isolation**: The SQLite database residing at `/models` contains private benchmark telemetry, prompt/response pairs, and system metrics. No raw data, logs, or screenshots are ever transmitted outside the 192.168.1.0/24 subnet. All evidence capture follows strict local storage protocols with automatic redaction of sensitive headers or tokens.
4. **Immutable Configuration Baselines**: Container startup flags, proxy routing rules, and database schemas are version-controlled. Any deviation from the baseline requires a tagged commit, a pre-change snapshot, and a post-change verification step. Rollback is always a single-command operation.
5. **Explicit Failure Boundaries**: The system is designed to fail safely. VRAM exhaustion, context overflow, proxy timeouts, and database locks trigger immediate pause states rather than silent degradation. Alerts are routed to the dashboard and logged locally. No automatic retries are permitted without human confirmation.
6. **Reproducibility & Auditability**: Every diagnostic step, command execution, and configuration change is logged with timestamps, operator IDs, and risk classifications. Benchmark runs are tied to specific container image tags, model checksums, and hardware states. Results are only considered valid if the full audit trail is intact.
7. **Cautious Escalation**: Issues are triaged from software/configuration layers upward to hardware/driver layers. GPU resets, container rebuilds, and database repairs are treated as last-resort actions. Each escalation step includes a verification checkpoint and a rollback trigger.
8. **Transparency in Reporting**: Screenshots, metric exports, and log excerpts are captured in standardized formats. Reporting focuses on actionable insights, failure modes, and validation status. Raw private data is never included in external reports or shared repositories.

These principles form the operational backbone of this runbook. Every procedure, command, and decision tree below is constructed to enforce these constraints while maintaining diagnostic depth and operational flexibility.

# Preflight Checks

Before initiating any benchmark sequence, diagnostic routine, or configuration modification, the following preflight checks must be executed in order. These checks verify system readiness, container health, hardware availability, and data integrity. Failure at any step halts progression until resolution.

1. **Host Connectivity & Network State**
   - Verify the host is reachable and network interfaces are stable.
   - `[READ-ONLY]` `ping -c 4 192.168.1.116`
   - `[READ-ONLY]` `ip addr show | grep -E "inet 192.168.1.116|state UP"`
   - Expected: 0% packet loss, interface state UP, correct subnet mask.

2. **Container Runtime & Image Verification**
   - Confirm the llama.cpp Vulkan container is running with the correct image tag.
   - `[READ-ONLY]` `docker ps --filter "name=llama-cpp-server" --format "{{.Names}}\t{{.Image}}\t{{.Status}}"`
   - `[READ-ONLY]` `docker inspect llama-cpp-server-vulkan:working-20260613 --format '{{.Config.Image}}'`
   - Expected: Container status "Up", image matches `llama-cpp-server-vulkan:working-20260613`.

3. **GPU Driver & Vulkan Backend Availability**
   - Verify AMD GPU is recognized and Vulkan layers are loaded.
   - `[READ-ONLY]` `rocminfo | grep -i "gfx|radeon"`
   - `[READ-ONLY]` `vulkaninfo --summary 2>/dev/null | grep -i "device name\|driver version"`
   - Expected: Radeon AI PRO R9700 listed, Vulkan driver version matches ROCm stack, no missing layers.

4. **Disk Space & Model File Integrity**
   - Ensure sufficient storage for logs, DB, and model loading. Verify model checksum.
   - `[READ-ONLY]` `df -h /models | tail -1`
   - `[READ-ONLY]` `sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
   - Expected: >10GB free on `/models`, checksum matches known baseline.

5. **Port Availability & Service Binding**
   - Confirm proxy and dashboard ports are bound and accessible.
   - `[READ-ONLY]` `ss -tulpn | grep -E ":8181|:8090|:8080"`
   - `[READ-ONLY]` `curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1`
   - Expected: Ports 8181, 8090, and llama.cpp port (typically 8080) listening, HTTP 200 or 401 from proxy.

6. **SQLite Database Accessibility**
   - Verify DB file exists, is readable, and passes basic integrity check.
   - `[READ-ONLY]` `ls -lh /models/benchmark.db`
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
   - Expected: File size >0, permissions 640 or 600, integrity check returns "ok".

7. **System Load & Thermal Baseline**
   - Check CPU load, memory pressure, and GPU thermal state before benchmarking.
   - `[READ-ONLY]` `uptime`
   - `[READ-ONLY]` `free -h | grep Mem`
   - `[READ-ONLY]` `rocm-smi --showtemp --showpower`
   - Expected: Load average <2.0, >4GB RAM free, GPU temp <75°C, power draw within spec.

If any preflight check fails, document the deviation, attempt a targeted fix, and re-run the full preflight sequence. Do not proceed to benchmark execution until all checks pass.

# Safe Inspection Commands

This section catalogs read-only and low-risk commands for continuous monitoring, log inspection, and state verification. These commands are safe to execute during active benchmark runs and do not modify system state, container configuration, or database contents.

1. **Container Log Inspection**
   - `[READ-ONLY]` `docker logs --tail 100 llama-cpp-server-vulkan:working-20260613`
   - `[READ-ONLY]` `docker logs --since 30m llama-cpp-server-vulkan:working-20260613 | grep -i "error\|warn\|oom\|truncat"`
   - Use: Monitor inference errors, VRAM allocation warnings, and truncation events.

2. **Process & Resource Monitoring**
   - `[READ-ONLY]` `top -bn1 | head -20`
   - `[READ-ONLY]` `ps aux | grep -E "llama|vulkan|proxy" | grep -v grep`
   - Use: Verify container processes, thread counts, and CPU affinity.

3. **Network & Port State**
   - `[READ-ONLY]` `ss -tulpn | grep -E "8181|8090|8080"`
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | jq '.[] | select(.name | contains("gpu"))'`
   - Use: Confirm service bindings and extract GPU metrics from ServerTop.

4. **Database Read-Only Queries**
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM runs;"`
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "SELECT task_id, finish_reason, tokens_generated FROM runs ORDER BY timestamp DESC LIMIT 5;"`
   - Use: Audit recent benchmark results without modifying schema or data.

5. **GPU & Vulkan State**
   - `[READ-ONLY]` `rocm-smi --showmeminfo vram_used`
   - `[READ-ONLY]` `vulkaninfo --device-index 0 2>/dev/null | grep -A5 "memoryHeap"`
   - Use: Track VRAM allocation and heap fragmentation.

6. **Proxy Health & Routing**
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8181/health | jq .`
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8181/v1/models | jq '.data[].id'`
   - Use: Verify proxy liveness and model routing.

7. **System Journal & Kernel Messages**
   - `[READ-ONLY]` `journalctl -u docker --since "1 hour ago" --no-pager | grep -i "llama\|vulkan\|gpu"`
   - `[READ-ONLY]` `dmesg -T | tail -30 | grep -i "amdgpu\|vulkan\|oom"`
   - Use: Detect driver crashes, OOM kills, or container restarts.

All commands above are classified as `[READ-ONLY]` or `[LOW-RISK]`. They do not alter configuration, restart services, or modify data. Execute them freely during diagnostics. For any command requiring write access, refer to the Risk Classification Matrix in the Rollback Procedures section.

# Reasoning Budget Verification

The `--reasoning-budget 8192` flag allocates a dedicated token window for chain-of-thought or extended reasoning before final content generation. Misconfiguration or exhaustion of this budget causes premature truncation, silent reasoning drops, or `finish_reason=length` artifacts. Verification ensures the budget is applied, respected, and aligned with workload requirements.

1. **Flag Verification at Container Startup**
   - Confirm the reasoning budget is passed to the llama.cpp server.
   - `[READ-ONLY]` `docker inspect llama-cpp-server-vulkan:working-20260613 --format '{{.Config.Cmd}}' | grep -o "reasoning-budget [0-9]*"`
   - Expected: `reasoning-budget 8192` present in command array.

2. **Runtime Token Allocation Check**
   - Query the server's internal token counters via the OpenAI-compatible endpoint.
   - `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Explain quantum entanglement step by step."}],"max_tokens":2048,"stream":false}' | jq '.usage'`
   - Expected: `completion_tokens` ≤ 8192 for reasoning-heavy prompts, `finish_reason` = "stop" or "length" only if budget exhausted.

3. **Budget Exhaustion Simulation**
   - Force a controlled reasoning overflow to verify truncation behavior.
   - `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Generate a detailed 10-step mathematical proof with explicit reasoning at each step."}],"max_tokens":10000,"stream":false}' | jq '{finish_reason: .choices[0].finish_reason, reasoning_tokens: .usage.reasoning_tokens}'`
   - Expected: `reasoning_tokens` caps at 8192, `finish_reason` = "length" if budget hit, final content truncated gracefully.

4. **Budget Adjustment Protocol**
   - If truncation occurs prematurely, increase budget cautiously.
   - `[MEDIUM-RISK]` `docker stop llama-cpp-server-vulkan:working-20260613`
   - `[MEDIUM-RISK]` `docker run -d --name llama-cpp-server-vulkan:working-20260613 \
     -p 8080:8080 \
     -v /models:/models \
     --gpus all \
     llama-cpp-server-vulkan:working-20260613 \
     --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
     --ctx-size 262144 \
     --parallel 1 \
     --spec-type draft-mtp --spec-draft-n-max 2 \
     --reasoning-budget 12288 \
     --host 0.0.0.0`
   - Expected: Container restarts with new budget, VRAM usage increases slightly, reasoning depth improves.
   - Rollback: Revert to `--reasoning-budget 8192` via container restart.

5. **Validation Criteria**
   - Reasoning budget must not exceed 20% of total context window (262144).
   - Budget exhaustion must trigger explicit `finish_reason=length` with clear truncation markers.
   - No silent drops or missing reasoning_content fields in responses.
   - All adjustments logged with timestamp, operator ID, and benchmark impact assessment.

Verification is mandatory before any long-output or multi-step reasoning benchmark. Failure to validate budget allocation invalidates subsequent results.

# Long Output Validation

The observation that `finish_reason=length` appeared before final content indicates `max_tokens` was undersized relative to reasoning + generation demands. Long-output validation ensures complete content delivery, prevents silent truncation, and enforces sequential task validation.

1. **Max Tokens Calibration**
   - Determine optimal `max_tokens` for reasoning + final content.
   - `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Write a comprehensive technical report on Vulkan compute shaders."}],"max_tokens":16384,"stream":false}' | jq '{finish_reason: .choices[0].finish_reason, total_tokens: .usage.total_tokens}'`
   - Expected: `finish_reason` = "stop", `total_tokens` < `max_tokens`, complete content delivered.

2. **Sequential Validation Protocol**
   - Execute one task, verify output, log results, then proceed.
   - `[LOW-RISK]` `for task in 1 2 3; do \
     echo "Validating task $task..."; \
     curl -s http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d "{\"model\":\"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf\",\"messages\":[{\"role\":\"user\",\"content\":\"Task $task: Generate detailed output.\"}],\"max_tokens\":16384,\"stream\":false}" | jq '.choices[0].finish_reason'; \
     sleep 5; \
   done`
   - Expected: All tasks return `finish_reason` = "stop". Any "length" triggers pause and `max_tokens` adjustment.

3. **Bulk Test Prevention**
   - Disable concurrent requests via parallel slot enforcement.
   - `[READ-ONLY]` `docker inspect llama-cpp-server-vulkan:working-20260613 --format '{{.Config.Cmd}}' | grep -o "parallel [0-9]*"`
   - Expected: `parallel 1`. If >1, enforce single-slot mode.
   - `[MEDIUM-RISK]` `docker stop llama-cpp-server-vulkan:working-20260613 && docker run -d --name llama-cpp-server-vulkan:working-20260613 -p 8080:8080 -v /models:/models --gpus all llama-cpp-server-vulkan:working-20260613 --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --ctx-size 262144 --parallel 1 --spec-type draft-mtp --spec-draft-n-max 2 --reasoning-budget 8192 --host 0.0.0.0`

4. **Context Window Management**
   - Verify prompt + reasoning + generation fits within 262144 tokens.
   - `[LOW-RISK]` `echo "Prompt length: $(echo 'Your prompt here' | wc -c) bytes"`
   - Expected: Total token count < 262144. If exceeded, truncate prompt or reduce context.

5. **Truncation Artifact Detection**
   - Scan responses for incomplete sentences, missing closing tags, or abrupt endings.
   - `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Generate output."}],"max_tokens":16384,"stream":false}' | jq -r '.choices[0].message.content' | tail -c 50`
   - Expected: Clean ending, no mid-word truncation, proper punctuation.

6. **Validation Criteria**
   - `finish_reason` must be "stop" for all validated tasks.
   - `max_tokens` must exceed reasoning_budget + expected generation length by ≥20%.
   - Sequential validation passes 3/3 times before bulk testing is permitted.
   - All truncation events logged with prompt hash, token counts, and adjustment history.

Long-output validation is mandatory for any benchmark exceeding 4096 tokens. Failure to validate invalidates dataset integrity.

# GPU And VRAM Checks

The AMD Radeon AI PRO R9700 (32GB VRAM) is the primary inference accelerator. VRAM exhaustion, fragmentation, or thermal throttling causes silent failures, OOM kills, or degraded throughput. Rigorous GPU monitoring ensures stable benchmark execution.

1. **VRAM Allocation & Usage**
   - Track real-time VRAM consumption.
   - `[READ-ONLY]` `rocm-smi --showmeminfo vram_used --showmeminfo vram_free`
   - Expected: `vram_used` < 28GB, `vram_free` > 4GB. If <2GB free, pause benchmark.

2. **Memory Fragmentation Detection**
   - Identify non-contiguous VRAM allocation.
   - `[READ-ONLY]` `vulkaninfo --device-index 0 2>/dev/null | grep -A10 "memoryHeap"`
   - Expected: Single contiguous heap, no fragmentation warnings. If fragmented, restart container.

3. **Thermal & Power Monitoring**
   - Prevent thermal throttling during long runs.
   - `[READ-ONLY]` `rocm-smi --showtemp --showpower --showclock`
   - Expected: Temp < 80°C, power draw within TDP, clock speeds stable. If temp > 85°C, reduce batch size or pause.

4. **Driver & Firmware Verification**
   - Ensure ROCm stack matches GPU architecture.
   - `[READ-ONLY]` `rocminfo | grep -i "gfx11\|radeon\|driver"`
   - `[READ-ONLY]` `modinfo amdgpu | grep -E "version|vermagic"`
   - Expected: Driver version ≥ 5.13, kernel matches ROCm stack, no mismatch warnings.

5. **GPU Hang & Reset Detection**
   - Detect driver crashes or GPU lockups.
   - `[READ-ONLY]` `dmesg -T | tail -20 | grep -i "amdgpu\|gpu\|hang\|reset"`
   - Expected: No recent hang/reset messages. If present, capture logs, reset GPU, and investigate.

6. **Stress Test Validation**
   - Verify GPU stability under load.
   - `[MEDIUM-RISK]` `docker run --rm --gpus all rocm/dev-ubuntu:latest rocm-smi --showmeminfo vram_used --showtemp --showpower --loop 10 --interval 5`
   - Expected: Stable metrics, no spikes or drops. If unstable, check cooling, power supply, or driver.

7. **VRAM Cleanup Protocol**
   - Free orphaned allocations after benchmark completion.
   - `[LOW-RISK]` `docker stop llama-cpp-server-vulkan:working-20260613 && docker start llama-cpp-server-vulkan:working-20260613`
   - Expected: VRAM usage drops to baseline, fragmentation cleared.

GPU checks are mandatory before, during, and after benchmark runs. VRAM exhaustion or thermal throttling invalidates results. Always maintain ≥4GB VRAM headroom.

# Proxy And Database Checks

The Flight Recorder proxy (8181) routes requests, logs telemetry, and enforces privacy. The SQLite database (`/models/benchmark.db`) stores benchmark results. Both must be verified for integrity, routing accuracy, and data isolation.

1. **Proxy Health & Routing Verification**
   - Confirm proxy is active and routing correctly.
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8181/health | jq .`
   - Expected: `status: "healthy"`, `upstream: "llama.cpp"`, `latency: <100ms`.

2. **Request Logging Validation**
   - Verify proxy captures prompts, responses, and metadata.
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM requests WHERE timestamp > datetime('now', '-1 hour');" `
   - Expected: Count matches benchmark tasks executed. If 0, proxy logging failed.

3. **Database Integrity & Schema Check**
   - Ensure DB structure is intact and queryable.
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db ".schema"`
   - Expected: `ok`, schema matches baseline (tables: runs, requests, metrics).

4. **Privacy & Data Isolation Enforcement**
   - Confirm no external access or data leakage.
   - `[READ-ONLY]` `ss -tulpn | grep 8181`
   - Expected: Bound to `127.0.0.1` or `192.168.1.116`, not `0.0.0.0`. If exposed, restrict binding.

5. **Backup & Restore Verification**
   - Test backup integrity without modifying live DB.
   - `[LOW-RISK]` `cp /models/benchmark.db /models/benchmark.db.bak.$(date +%Y%m%d%H%M%S)`
   - `[LOW-RISK]` `sqlite3 /models/benchmark.db.bak.* "PRAGMA integrity_check;"`
   - Expected: Backup passes integrity check, size matches live DB.

6. **Query Performance & Lock Detection**
   - Identify slow queries or DB locks.
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA journal_mode;"`
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA locking_mode;"`
   - Expected: `WAL`, `NORMAL`. If `DELETE` or `EXCLUSIVE`, adjust for concurrent reads.

7. **Proxy Configuration Audit**
   - Verify routing rules, timeouts, and retry policies.
   - `[READ-ONLY]` `cat /etc/flight-recorder/proxy.conf | grep -E "timeout|retry|upstream"`
   - Expected: `timeout: 300s`, `retry: 0`, `upstream: http://192.168.1.116:8080`.

Proxy and DB checks ensure telemetry accuracy and data privacy. Any routing failure or DB corruption halts benchmarking until resolved.

# Dashboard Checks

ServerTop (8090) provides real-time metrics, system state, and benchmark telemetry. Dashboard validation ensures accurate monitoring, alerting, and screenshot capture for reporting.

1. **Dashboard Accessibility & Health**
   - Confirm ServerTop is reachable and serving metrics.
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8090/health | jq .`
   - Expected: `status: "up"`, `version: "x.y.z"`, `metrics_endpoint: "/metrics"`.

2. **Metric Parsing & Validation**
   - Extract and verify key metrics.
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | jq '.[] | select(.name | contains("gpu_vram_used|tokens_per_sec|latency"))'`
   - Expected: Numeric values, no nulls, consistent timestamps.

3. **Alert Threshold Configuration**
   - Verify alerting rules for VRAM, latency, and errors.
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8090/config/alerts | jq .`
   - Expected: `vram_threshold: 90%`, `latency_threshold: 5000ms`, `error_rate: 5%`.

4. **Screenshot Capture Protocol**
   - Capture dashboard state for reporting.
   - `[LOW-RISK]` `import -window root /models/screenshots/dashboard_$(date +%Y%m%d%H%M%S).png`
   - Expected: PNG saved, contains metrics, no sensitive data exposed.

5. **Data Flow Verification**
   - Confirm metrics flow from proxy → DB → dashboard.
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "SELECT MAX(timestamp) FROM metrics;"`
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | jq '.[-1].timestamp'`
   - Expected: Timestamps within 60s of each other. If drift >5min, check proxy or DB sync.

6. **Dashboard Performance & Load**
   - Ensure dashboard doesn't bottleneck monitoring.
   - `[READ-ONLY]` `curl -s -o /dev/null -w "%{time_total}" http://192.168.1.116:8090/metrics`
   - Expected: Response time <200ms. If >500ms, reduce polling frequency or optimize queries.

7. **Historical Data Retention**
   - Verify old metrics are archived, not deleted.
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM metrics WHERE timestamp < datetime('now', '-7 days');" `
   - Expected: Count >0, retention policy enforced.

Dashboard checks ensure accurate monitoring and reporting. Any metric drift or alert failure requires immediate investigation.

# Failure Triage Trees

Structured decision trees for common failures. Follow paths sequentially. Do not skip steps.

**Tree 1: `finish_reason=length` Before Final Content**
1. Check `max_tokens` vs `reasoning_budget` + expected output.
   - `[READ-ONLY]` `docker inspect ... | grep max_tokens`
2. If `max_tokens` < budget + 20%, increase `max_tokens`.
   - `[MEDIUM-RISK]` Restart container with higher `max_tokens`.
3. Verify sequential validation passes.
   - If fails, check context window overflow.
   - `[READ-ONLY]` `echo "Prompt tokens: $(prompt | wc -c)"`
4. If context overflow, truncate prompt or reduce `ctx-size`.
   - `[MEDIUM-RISK]` Adjust `--ctx-size`, restart.

**Tree 2: VRAM Exhaustion / OOM**
1. Check VRAM usage.
   - `[READ-ONLY]` `rocm-smi --showmeminfo vram_used`
2. If >90%, identify large allocations.
   - `[READ-ONLY]` `docker logs ... | grep -i "alloc\|oom"`
3. Reduce batch size or parallel slots.
   - `[MEDIUM-RISK]` Set `--parallel 1`, restart.
4. If persistent, check fragmentation.
   - `[READ-ONLY]` `vulkaninfo ... | grep memoryHeap`
5. Restart container to clear fragmentation.
   - `[LOW-RISK]` `docker restart ...`

**Tree 3: Proxy Timeout / Routing Failure**
1. Check proxy health.
   - `[READ-ONLY]` `curl http://192.168.1.116:8181/health`
2. If unhealthy, check upstream connectivity.
   - `[READ-ONLY]` `curl http://192.168.1.116:8080/v1/models`
3. If upstream down, restart llama.cpp container.
   - `[MEDIUM-RISK]` `docker restart ...`
4. If proxy misrouting, check config.
   - `[READ-ONLY]` `cat /etc/flight-recorder/proxy.conf`
5. Fix routing, restart proxy.
   - `[MEDIUM-RISK]` `systemctl restart flight-recorder`

**Tree 4: SQLite Corruption / Lock**
1. Run integrity check.
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
2. If failed, restore from backup.
   - `[MEDIUM-RISK]` `cp /models/benchmark.db.bak.* /models/benchmark.db`
3. If locked, check concurrent writers.
   - `[READ-ONLY]` `lsof /models/benchmark.db`
4. Kill rogue processes, restart DB.
   - `[MEDIUM-RISK]` `kill -9 <pid>`, `systemctl restart sqlite-proxy`

**Tree 5: GPU Hang / Driver Crash**
1. Check dmesg for hang/reset.
   - `[READ-ONLY]` `dmesg -T | grep -i "amdgpu\|hang"`
2. If present, reset GPU.
   - `[MEDIUM-RISK]` `echo 1 > /sys/bus/pci/devices/0000:xx:xx.x/reset`
3. Restart container, verify stability.
   - `[LOW-RISK]` `docker restart ...`
4. If recurring, check cooling/power.
   - `[READ-ONLY]` `rocm-smi --showtemp --showpower`

Follow trees strictly. Document each step. Escalate if unresolved after 3 iterations.

# Rollback Procedures

Rollback ensures rapid recovery from misconfigurations, corruption, or instability. All procedures include pre-rollback snapshots and post-rollback verification.

1. **Container Configuration Rollback**
   - `[LOW-RISK]` `docker commit llama-cpp-server-vulkan:working-20260613 llama-cpp-server-vulkan:rollback-$(date +%Y%m%d%H%M%S)`
   - `[MEDIUM-RISK]` `docker stop llama-cpp-server-vulkan:working-20260613 && docker rm llama-cpp-server-vulkan:working-20260613`
   - `[MEDIUM-RISK]` `docker run -d --name llama-cpp-server-vulkan:working-20260613 -p 8080:8080 -v /models:/models --gpus all llama-cpp-server-vulkan:working-20260613 --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --ctx-size 262144 --parallel 1 --spec-type draft-mtp --spec-draft-n-max 2 --reasoning-budget 8192 --host 0.0.0.0`
   - Verify: Container starts, metrics flow, no errors.

2. **Database Restore**
   - `[LOW-RISK]` `cp /models/benchmark.db /models/benchmark.db.pre-rollback.$(date +%Y%m%d%H%M%S)`
   - `[MEDIUM-RISK]` `cp /models/benchmark.db.bak.* /models/benchmark.db`
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
   - Verify: Integrity ok, schema matches, data restored.

3. **Proxy Configuration Revert**
   - `[LOW-RISK]` `cp /etc/flight-recorder/proxy.conf /etc/flight-recorder/proxy.conf.pre-rollback.$(date +%Y%m%d%H%M%S)`
   - `[MEDIUM-RISK]` `cp /etc/flight-recorder/proxy.conf.bak.* /etc/flight-recorder/proxy.conf`
   - `[MEDIUM-RISK]` `systemctl restart flight-recorder`
   - Verify: Health endpoint up, routing correct, logs flowing.

4. **Model Swap Rollback**
   - `[LOW-RISK]` `cp /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf.pre-rollback.$(date +%Y%m%d%H%M%S)`
   - `[MEDIUM-RISK]` `cp /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf.bak.* /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
   - `[READ-ONLY]` `sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
   - Verify: Checksum matches baseline, container loads model.

5. **Network & Port Reset**
   - `[LOW-RISK]` `iptables-save > /tmp/iptables.pre-rollback.$(date +%Y%m%d%H%M%S)`
   - `[MEDIUM-RISK]` `iptables-restore < /tmp/iptables.bak.*`
   - `[READ-ONLY]` `ss -tulpn | grep -E "8181|8090|8080"`
   - Verify: Ports bound correctly, no conflicts.

All rollbacks require post-verification. Do not resume benchmarking until rollback is confirmed stable.

# Human Approval Gates

Certain actions require explicit human confirmation to prevent accidental data loss, configuration drift, or infrastructure instability. The following gates enforce mandatory pauses:

1. **VRAM Usage > 90%**
   - Action: Pause benchmark, notify operator.
   - Approval: Operator confirms reduction of batch size or context window.
   - Escalation: If unresolved, trigger GPU reset procedure.

2. **Database Integrity Failure**
   - Action: Halt all writes, switch to read-only mode.
   - Approval: Operator confirms restore from backup or manual repair.
   - Escalation: If corruption persists, isolate DB, investigate driver/proxy.

3. **Proxy Routing Misconfiguration**
   - Action: Disable proxy, route directly to llama.cpp.
   - Approval: Operator confirms config revert and restart.
   - Escalation: If routing fails, check network policies, firewall rules.

4. **Bulk Benchmark Initiation**
   - Action: Block execution until sequential validation passes.
   - Approval: Operator confirms 3/3 sequential tasks successful.
   - Escalation: If validation fails, adjust `max_tokens` or budget.

5. **Model File Replacement**
   - Action: Pause container, verify checksum.
   - Approval: Operator confirms new model matches expected hash.
   - Escalation: If mismatch, revert to backup, investigate source.

6. **Network Exposure Change**
   - Action: Block binding to `0.0.0.0` or external IPs.
   - Approval: Operator confirms subnet restriction and firewall rules.
   - Escalation: If exposed, immediately restrict, audit logs.

7. **Reasoning Budget Increase > 20%**
   - Action: Pause, calculate VRAM impact.
   - Approval: Operator confirms budget increase and VRAM headroom.
   - Escalation: If VRAM insufficient, reduce context or batch size.

All gates require timestamped confirmation, operator ID, and audit log entry. No automated bypass permitted.

# Evidence To Capture

Comprehensive evidence capture ensures reproducibility, debugging capability, and reporting accuracy. All evidence is stored locally, redacted of sensitive data, and retained per policy.

1. **System Logs**
   - Container logs: `docker logs --tail 500 ... > /models/evidence/logs/container_$(date +%Y%m%d%H%M%S).log`
   - System journal: `journalctl -u docker --since "1 hour ago" > /models/evidence/logs/journal_$(date +%Y%m%d%H%M%S).log`
   - Kernel messages: `dmesg -T > /models/evidence/logs/dmesg_$(date +%Y%m%d%H%M%S).log`

2. **Metrics & Telemetry**
   - ServerTop export: `curl -s http://192.168.1.116:8090/metrics > /models/evidence/metrics/metrics_$(date +%Y%m%d%H%M%S).json`
   - GPU stats: `rocm-smi --showmeminfo --showtemp --showpower > /models/evidence/metrics/gpu_$(date +%Y%m%d%H%M%S).txt`
   - Proxy logs: `sqlite3 /models/benchmark.db "SELECT * FROM requests ORDER BY timestamp DESC LIMIT 100;" > /models/evidence/metrics/proxy_$(date +%Y%m%d%H%M%S).csv`

3. **Screenshots**
   - Dashboard: `import -window root /models/evidence/screenshots/dashboard_$(date +%Y%m%d%H%M%S).png`
   - Terminal state: `script -q /models/evidence/screenshots/terminal_$(date +%Y%m%d%H%M%S).txt`
   - Redaction: Remove IPs, tokens, private paths before archiving.

4. **Prompt/Response Pairs**
   - Export validated tasks: `sqlite3 /models/benchmark.db "SELECT task_id, prompt, response, finish_reason FROM runs WHERE status='validated';" > /models/evidence/data/validated_$(date +%Y%m%d%H%M%S).json`
   - Truncation artifacts: `sqlite3 /models/benchmark.db "SELECT * FROM runs WHERE finish_reason='length';" > /models/evidence/data/truncated_$(date +%Y%m%d%H%M%S).json`

5. **Configuration Snapshots**
   - Container config: `docker inspect ... > /models/evidence/config/container_$(date +%Y%m%d%H%M%S).json`
   - Proxy config: `cp /etc/flight-recorder/proxy.conf /models/evidence/config/proxy_$(date +%Y%m%d%H%M%S).conf`
   - DB schema: `sqlite3 /models/benchmark.db ".schema" > /models/evidence/config/schema_$(date +%Y%m%d%H%M%S).sql`

6. **Retention & Privacy Policy**
   - Retain evidence for 90 days, then archive to cold storage.
   - Never transmit raw evidence outside 192.168.1.0/24.
   - Redact sensitive headers, tokens, and private paths before any external reporting.

Evidence capture is mandatory for all benchmark runs, failures, and rollbacks. Incomplete evidence invalidates results.

# Final Go No-Go Checklist

Execute this checklist before, during, and after benchmark runs. All items must pass for "GO" status. Any "NO-GO" halts execution until resolved.

**Pre-Benchmark Validation**
- [ ] Host connectivity stable, ping <10ms
- [ ] Container running, image tag matches baseline
- [ ] GPU recognized, Vulkan layers loaded, temp <75°C
- [ ] Disk space >10GB free, model checksum verified
- [ ] Ports 8181, 8090, 8080 bound correctly
- [ ] SQLite integrity check passes, schema matches
- [ ] Proxy health endpoint returns 200, routing correct
- [ ] ServerTop metrics flowing, no drift >5min
- [ ] Reasoning budget verified, `max_tokens` calibrated
- [ ] Sequential validation 3/3 tasks pass with `finish_reason=stop`
- [ ] VRAM headroom ≥4GB, no fragmentation
- [ ] Backup of DB and config created
- [ ] Human approval gates cleared for all pending actions

**During-Benchmark Monitoring**
- [ ] VRAM usage <90%, temp <80°C
- [ ] Proxy latency <100ms, no timeouts
- [ ] DB writes successful, no locks
- [ ] Dashboard metrics consistent, alerts triggered if thresholds exceeded
- [ ] Sequential validation enforced, no bulk runs
- [ ] Evidence capture active, logs flowing
- [ ] Human approval gates monitored, pauses enforced if needed

**Post-Benchmark Verification**
- [ ] All tasks completed with `finish_reason=stop`
- [ ] No truncation artifacts, complete content delivered
- [ ] VRAM freed, fragmentation cleared
- [ ] DB integrity check passes, data consistent
- [ ] Proxy logs match task count, no dropped requests
- [ ] Dashboard metrics archived, screenshots captured
- [ ] Evidence stored locally, redacted, retained per policy
- [ ] Rollback procedures tested, snapshots verified
- [ ] Human sign-off obtained, audit trail complete

**GO Criteria**: All checkboxes marked, no unresolved failures, evidence complete, human approval confirmed.
**NO-GO Criteria**: Any checkbox unchecked, failure unresolved, evidence incomplete, human approval pending.

This checklist is the final gate before benchmark execution or result acceptance. Do not proceed without full compliance. Document all deviations, resolve before resuming. Maintain cautious, reproducible, and privacy-first operations at all times.

# Advanced MTP & Speculative Decoding Validation

Multi-Token Prediction (MTP) and speculative decoding significantly impact throughput and VRAM allocation. Misconfiguration can cause draft token rejection, context drift, or silent performance degradation. Validation ensures the `--spec-type draft-mtp --spec-draft-n-max 2` pipeline operates within safe boundaries.

1. **Draft Token Rejection Rate Monitoring**
   - Track how often the target model rejects MTP draft tokens.
   - `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | jq '.[] | select(.name | contains("speculative_rejection_rate"))'`
   - Expected: Rejection rate < 15%. If > 25%, draft model mismatch or context misalignment is occurring.
   - Action: Reduce `--spec-draft-n-max` to 1 or disable speculative decoding temporarily for baseline comparison.

2. **Context Window Alignment Verification**
   - Ensure draft and target models share identical context boundaries.
   - `[READ-ONLY]` `docker inspect llama-cpp-server-vulkan:working-20260613 --format '{{.Config.Cmd}}' | grep -o "ctx-size [0-9]*"`
   - Expected: `ctx-size 262144` for both draft and target. Mismatch causes token offset errors.
   - `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Verify context alignment."}],"max_tokens":1024,"stream":false}' | jq '.usage.prompt_tokens'`
   - Expected: Prompt token count matches input length exactly. No padding or truncation artifacts.

3. **Speculative Decoding Latency Profiling**
   - Measure time-to-first-token (TTFT) and inter-token latency (ITL) with MTP enabled.
   - `[LOW-RISK]` `curl -s -w "\nTTFT: %{time_starttransfer}s\nITL: %{time_total}s\n" http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Profile latency."}],"max_tokens":512,"stream":false}'`
   - Expected: TTFT < 2.0s, ITL < 0.5s. If TTFT > 5.0s, check VRAM bandwidth or draft model loading.
   - Rollback: Disable `--spec-type draft-mtp`, restart container, re-profile.

4. **Draft Model Integrity Check**
   - Verify the embedded draft model matches the target architecture.
   - `[READ-ONLY]` `llama-cli --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --print-special 2>/dev/null | grep -i "draft\|mtp"`
   - Expected: Draft model metadata present, architecture matches `Qwen3.6` family.
   - If missing or mismatched, replace GGUF file with verified baseline. `[MEDIUM-RISK]` `cp /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf.bak.* /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`

5. **MTP Validation Criteria**
   - Rejection rate must remain < 20% across 10 consecutive requests.
   - TTFT and ITL must improve by ≥ 15% compared to non-speculative baseline.
   - No context drift or token repetition artifacts in output.
   - All MTP adjustments logged with draft rejection metrics and latency deltas.

MTP validation is mandatory before throughput benchmarking. Failure to meet criteria requires reverting to standard decoding until draft alignment is resolved.

# Container Lifecycle & Orchestration Protocols

Container management governs service availability, configuration drift, and recovery speed. Strict lifecycle protocols prevent unauthorized modifications and ensure deterministic restarts.

1. **Startup Sequence Enforcement**
   - Containers must initialize in dependency order: GPU driver → SQLite proxy → Flight Recorder → llama.cpp.
   - `[READ-ONLY]` `systemctl is-active rocm-smi.service sqlite-proxy.service flight-recorder.service`
   - Expected: All services `active (running)`. If any `inactive`, restart in sequence.
   - `[MEDIUM-RISK]` `systemctl restart rocm-smi.service && systemctl restart sqlite-proxy.service && systemctl restart flight-recorder.service && docker start llama-cpp-server-vulkan:working-20260613`

2. **Configuration Drift Detection**
   - Compare running container flags against baseline manifest.
   - `[READ-ONLY]` `docker inspect llama-cpp-server-vulkan:working-20260613 --format '{{.Config.Cmd}}' > /tmp/current_flags.txt`
   - `[READ-ONLY]` `diff /tmp/current_flags.txt /models/configs/baseline_flags.txt`
   - Expected: No differences. If drift detected, halt benchmark, review changes, apply rollback.

3. **Graceful Shutdown & State Preservation**
   - Prevent data corruption during stops.
   - `[LOW-RISK]` `docker stop -t 30 llama-cpp-server-vulkan:working-20260613`
   - Expected: Container exits cleanly, proxy flushes logs, DB commits pending writes.
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
   - If integrity fails, do not restart. Initiate DB recovery procedure.

4. **Automated Health Checks & Restart Policies**
   - Configure Docker to restart only on critical failures, not transient timeouts.
   - `[MEDIUM-RISK]` `docker update --restart unless-stopped llama-cpp-server-vulkan:working-20260613`
   - `[READ-ONLY]` `docker inspect llama-cpp-server-vulkan:working-20260613 --format '{{.HostConfig.RestartPolicy.Name}}'`
   - Expected: `unless-stopped`. Avoid `always` to prevent infinite restart loops on misconfiguration.

5. **Resource Limits & Cgroup Enforcement**
   - Prevent container from starving host or other services.
   - `[MEDIUM-RISK]` `docker update --memory 64g --cpus 16 llama-cpp-server-vulkan:working-20260613`
   - `[READ-ONLY]` `docker stats --no-stream llama-cpp-server-vulkan:working-20260613`
   - Expected: Memory usage < 60GB, CPU usage < 15 cores. If limits breached, adjust workload or increase host resources.

6. **Lifecycle Validation Criteria**
   - Startup sequence completes within 90 seconds.
   - Configuration drift check returns zero differences.
   - Graceful shutdown preserves DB integrity.
   - Restart policy matches `unless-stopped`.
   - Resource limits enforced and monitored.

Container lifecycle protocols ensure deterministic behavior. Any deviation requires immediate pause, audit, and corrective action before resuming operations.

# Network Isolation & Firewall Hardening

The benchmark environment must remain isolated from external networks. Firewall rules, port binding restrictions, and traffic filtering prevent accidental exposure or unauthorized access.

1. **Port Binding Verification**
   - Ensure services bind only to internal interfaces.
   - `[READ-ONLY]` `ss -tulpn | grep -E "8181|8090|8080" | awk '{print $4}'`
   - Expected: `127.0.0.1:8181`, `192.168.1.116:8090`, `192.168.1.116:8080`. No `0.0.0.0` bindings.
   - If exposed, restrict immediately: `[MEDIUM-RISK]` `docker stop llama-cpp-server-vulkan:working-20260613 && docker run -d --name llama-cpp-server-vulkan:working-20260613 -p 192.168.1.116:8080:8080 -v /models:/models --gpus all llama-cpp-server-vulkan:working-20260613 --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --ctx-size 262144 --parallel 1 --spec-type draft-mtp --spec-draft-n-max 2 --reasoning-budget 8192 --host 0.0.0.0`

2. **Firewall Rule Enforcement**
   - Block external access to benchmark ports.
   - `[MEDIUM-RISK]` `iptables -A INPUT -p tcp -d 192.168.1.116 --dport 8181 -j DROP`
   - `[MEDIUM-RISK]` `iptables -A INPUT -p tcp -d 192.168.1.116 --dport 8090 -j DROP`
   - `[READ-ONLY]` `iptables -L INPUT -n -v | grep -E "8181|8090|8080"`
   - Expected: DROP rules active for external sources, ACCEPT for `192.168.1.0/24`.

3. **DNS & Reverse Lookup Isolation**
   - Prevent DNS leakage or external resolution.
   - `[READ-ONLY]` `cat /etc/resolv.conf | grep nameserver`
   - Expected: Internal DNS only (e.g., `192.168.1.1`). No public resolvers.
   - `[MEDIUM-RISK]` `echo "nameserver 192.168.1.1" > /etc/resolv.conf`

4. **Traffic Filtering & Connection Limits**
   - Limit concurrent connections to prevent DoS or resource exhaustion.
   - `[MEDIUM-RISK]` `iptables -A INPUT -p tcp --dport 8080 -m connlimit --connlimit-above 5 -j REJECT`
   - `[READ-ONLY]` `ss -tnp | grep :8080 | wc -l`
   - Expected: Active connections ≤ 5. If exceeded, investigate proxy or client misconfiguration.

5. **Network Validation Criteria**
   - All ports bound to internal IPs only.
   - Firewall rules block external access to 8181, 8090, 8080.
   - DNS resolves internally, no public resolvers.
   - Connection limits enforced, no flooding detected.
   - Network isolation verified via `nmap` scan from external host (if available).

Network hardening is non-negotiable. Any exposure to external subnets triggers immediate service pause and security audit.

# Automated Telemetry & Log Rotation

Continuous telemetry collection requires disciplined log management to prevent disk exhaustion and maintain query performance. Automated rotation, compression, and archival ensure long-term observability without storage bloat.

1. **Log Rotation Configuration**
   - Configure `logrotate` for container, proxy, and system logs.
   - `[MEDIUM-RISK]` `cat > /etc/logrotate.d/llama-benchmark << 'EOF'
/models/evidence/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root root
    postrotate
        systemctl reload docker
    endscript
}
EOF`
   - Expected: Logrotate config created, syntax valid.
   - `[READ-ONLY]` `logrotate -d /etc/logrotate.d/llama-benchmark`
   - Expected: Dry run shows rotation schedule, no errors.

2. **SQLite WAL Truncation & Vacuum**
   - Prevent WAL file growth from consuming disk space.
   - `[LOW-RISK]` `sqlite3 /models/benchmark.db "PRAGMA wal_checkpoint(TRUNCATE);" `
   - `[LOW-RISK]` `sqlite3 /models/benchmark.db "VACUUM;"`
   - Expected: WAL file size reduces, DB file compacts, no data loss.
   - Schedule via cron: `[MEDIUM-RISK]` `crontab -e` → `0 3 * * * sqlite3 /models/benchmark.db "PRAGMA wal_checkpoint(TRUNCATE); VACUUM;"`

3. **Telemetry Export & Archival**
   - Export metrics to compressed JSON for long-term storage.
   - `[LOW-RISK]` `curl -s http://192.168.1.116:8090/metrics | gzip > /models/evidence/metrics/metrics_$(date +%Y%m%d).json.gz`
   - `[READ-ONLY]` `ls -lh /models/evidence/metrics/*.json.gz | tail -5`
   - Expected: Daily archives created, size < 50MB each, compression ratio > 70%.

4. **Log Integrity & Tamper Detection**
   - Verify logs haven't been altered or truncated unexpectedly.
   - `[READ-ONLY]` `sha256sum /models/evidence/logs/container_*.log > /tmp/log_checksums.txt`
   - `[READ-ONLY]` `diff /tmp/log_checksums.txt /models/configs/log_checksums_baseline.txt`
   - Expected: No differences. If mismatch, investigate unauthorized access or disk errors.

5. **Telemetry Validation Criteria**
   - Log rotation active, 30-day retention enforced.
   - WAL checkpoint runs daily, DB size stable.
   - Metrics archived daily, compressed, checksummed.
   - Log integrity verified, no tampering detected.
   - Disk usage < 80% on `/models` partition.

Automated telemetry ensures observability without storage risk. Failure to rotate or archive triggers immediate disk space alert and manual intervention.

# Performance Profiling & Bottleneck Analysis

Identifying performance bottlenecks requires systematic profiling across CPU, GPU, memory, and I/O layers. Profiling isolates constraints, guides optimization, and validates configuration changes.

1. **CPU Thread & Context Switch Analysis**
   - Identify CPU-bound operations or thread contention.
   - `[READ-ONLY]` `pidstat -t -p $(docker inspect -f '{{.State.Pid}}' llama-cpp-server-vulkan:working-20260613) 1 5`
   - Expected: CPU usage distributed across threads, context switches < 1000/s. If high, check parallel slot configuration or Vulkan thread pool.

2. **GPU Compute & Memory Bandwidth Profiling**
   - Measure GPU utilization and memory throughput.
   - `[READ-ONLY]` `rocprof --stats --trace gpu -- /usr/bin/docker exec llama-cpp-server-vulkan:working-20260613 llama-server --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --ctx-size 262144 --parallel 1 --spec-type draft-mtp --spec-draft-n-max 2 --reasoning-budget 8192 --host 0.0.0.0 --log-disable`
   - Expected: GPU utilization > 70%, memory bandwidth > 500 GB/s. If low, check batch size, draft model alignment, or VRAM fragmentation.

3. **I/O Latency & Disk Throughput**
   - Verify model loading and log writing don't bottleneck inference.
   - `[READ-ONLY]` `iostat -x 1 5 | grep -E "sda|nvme"`
   - Expected: `%util` < 50%, `await` < 5ms. If high, move logs to separate SSD or enable RAM disk for temporary telemetry.

4. **Network Latency & Packet Loss**
   - Ensure proxy and dashboard communication isn't delayed.
   - `[READ-ONLY]` `ping -c 10 192.168.1.116 | tail -1`
   - `[READ-ONLY]` `curl -s -o /dev/null -w "DNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTTFB: %{time_starttransfer}s\n" http://192.168.1.116:8181/v1/models`
   - Expected: Ping < 1ms, DNS < 0.01s, Connect < 0.05s, TTFB < 0.1s. If degraded, check switch, cable, or firewall rules.

5. **Bottleneck Identification Matrix**
   - CPU-bound: High `%usr`, low GPU util → Reduce `--parallel`, increase `--threads`.
   - GPU-bound: High GPU util, low memory bandwidth → Check VRAM allocation, draft model size.
   - Memory-bound: High swap usage, OOM kills → Increase host RAM, reduce `--ctx-size`.
   - I/O-bound: High disk `%util`, slow log writes → Enable WAL, move logs to NVMe.
   - Network-bound: High latency, packet loss → Check switch, firewall, proxy config.

6. **Profiling Validation Criteria**
   - CPU, GPU, memory, I/O, and network metrics collected.
   - Bottleneck identified and documented.
   - Optimization applied, re-profiled, improvement verified.
   - All profiling data archived, checksummed, retained per policy.

Performance profiling is mandatory before throughput claims or configuration changes. Unverified optimizations invalidate benchmark results.

# Disaster Recovery & Backup Rotation Strategy

Data loss or infrastructure failure requires immediate, deterministic recovery. Backup rotation, offsite mirroring (within subnet), and recovery drills ensure business continuity.

1. **Daily Backup Execution**
   - Automate DB, config, and model backups.
   - `[MEDIUM-RISK]` `tar -czf /models/backups/benchmark_$(date +%Y%m%d).tar.gz /models/benchmark.db /models/configs/ /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
   - `[READ-ONLY]` `sha256sum /models/backups/benchmark_$(date +%Y%m%d).tar.gz > /models/backups/checksums.txt`
   - Expected: Archive created, checksum recorded, size < 40GB.

2. **Backup Rotation & Retention**
   - Maintain 7 daily, 4 weekly, 12 monthly backups.
   - `[MEDIUM-RISK]` `find /models/backups/ -name "benchmark_*.tar.gz" -mtime +30 -delete`
   - `[READ-ONLY]` `ls -lh /models/backups/ | wc -l`
   - Expected: Backup count matches retention policy, oldest backup ≤ 30 days.

3. **Recovery Drill Execution**
   - Test restore procedure quarterly.
   - `[MEDIUM-RISK]` `docker stop llama-cpp-server-vulkan:working-20260613`
   - `[MEDIUM-RISK]` `cp /models/benchmark.db /models/benchmark.db.live`
   - `[MEDIUM-RISK]` `tar -xzf /models/backups/benchmark_$(date +%Y%m%d).tar.gz -C /models/`
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
   - Expected: DB restored, integrity ok, container restarts successfully.
   - Rollback: Restore `/models/benchmark.db.live` if drill fails.

4. **Offsite Mirroring (Subnet-Only)**
   - Sync backups to secondary storage within 192.168.1.0/24.
   - `[MEDIUM-RISK]` `rsync -avz /models/backups/ 192.168.1.117:/backup/llama-benchmark/`
   - `[READ-ONLY]` `ssh 192.168.1.117 "ls -lh /backup/llama-benchmark/ | tail -5"`
   - Expected: Sync completes, checksums match, no network errors.

5. **Disaster Recovery Validation Criteria**
   - Daily backups created, checksummed, rotated.
   - Recovery drill passes, DB integrity verified.
   - Offsite mirror synced, accessible, validated.
   - Recovery time objective (RTO) < 15 minutes.
   - Recovery point objective (RPO) < 24 hours.

Disaster recovery ensures resilience. Failure to backup or test restore triggers immediate infrastructure pause and remediation.

# Audit Trail Generation & Compliance Reporting

All operations must be auditable. Automated trail generation, hash verification, and report formatting ensure transparency, reproducibility, and compliance with internal policies.

1. **Operation Logging & Hashing**
   - Record every command, config change, and benchmark run.
   - `[LOW-RISK]` `echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) | OPERATOR: admin | ACTION: benchmark_run | STATUS: completed | HASH: $(echo 'benchmark_run_$(date +%s)' | sha256sum | awk '{print $1}')" >> /models/audit/operations.log`
   - `[READ-ONLY]` `tail -n 10 /models/audit/operations.log`
   - Expected: Entries timestamped, hashed, immutable. No gaps or duplicates.

2. **Audit Trail Integrity Verification**
   - Verify log hasn't been altered.
   - `[READ-ONLY]` `sha256sum /models/audit/operations.log > /tmp/audit_hash.txt`
   - `[READ-ONLY]` `diff /tmp/audit_hash.txt /models/configs/audit_hash_baseline.txt`
   - Expected: No differences. If mismatch, investigate tampering or disk corruption.

3. **Compliance Report Generation**
   - Export structured report for internal review.
   - `[LOW-RISK]` `jq -n --slurpfile runs <(sqlite3 -json /models/benchmark.db "SELECT task_id, finish_reason, tokens_generated, timestamp FROM runs ORDER BY timestamp DESC LIMIT 50;") '{report_date: now, total_runs: ($runs | length), success_rate: (($runs | map(select(.finish_reason == "stop")) | length) / ($runs | length) * 100), avg_tokens: (($runs | map(.tokens_generated) | add) / ($runs | length))}' > /models/reports/compliance_$(date +%Y%m%d).json`
   - Expected: JSON report generated, metrics accurate, no sensitive data exposed.

4. **Report Archival & Distribution**
   - Store reports locally, restrict access.
   - `[MEDIUM-RISK]` `chmod 600 /models/reports/compliance_*.json`
   - `[READ-ONLY]` `ls -lh /models/reports/`
   - Expected: Reports readable by admin only, archived per retention policy.

5. **Audit Validation Criteria**
   - All operations logged, hashed, immutable.
   - Audit trail integrity verified, no tampering.
   - Compliance reports generated, accurate, redacted.
   - Reports archived, access restricted, retained per policy.
   - Audit trail available for internal review within 5 minutes.

Audit trail generation ensures accountability. Missing logs or unverified hashes invalidate benchmark results and trigger security review.

# Appendix: Command Risk Classification Matrix

| Risk Level | Definition | Examples | Approval Required | Rollback Complexity |
|------------|------------|----------|-------------------|---------------------|
| `[READ-ONLY]` | No state modification, pure inspection | `docker logs`, `rocm-smi`, `sqlite3 PRAGMA`, `curl health` | None | N/A |
| `[LOW-RISK]` | Minor state change, reversible instantly | `docker restart`, `sqlite3 VACUUM`, `logrotate`, `cp backup` | Operator acknowledgment | < 1 minute |
| `[MEDIUM-RISK]` | Configuration change, service restart, data modification | `docker run` with new flags, `iptables` rules, `crontab -e`, `rsync` | Explicit human approval, documented justification | 5-15 minutes |
| `[DESTRUCTIVE]` | Data loss, hardware reset, irreversible action | `rm -rf`, `dd if=/dev/zero`, `echo 1 > /sys/.../reset`, `DROP TABLE` | Dual-operator approval, emergency protocol only | 30+ minutes, potential data loss |

All commands in this runbook are classified accordingly. Operators must verify risk level before execution. Unauthorized `[MEDIUM-RISK]` or `[DESTRUCTIVE]` actions trigger immediate audit and suspension.

# Appendix: Troubleshooting Decision Matrix

| Symptom | Likely Cause | Diagnostic Command | Resolution Path | Validation Step |
|---------|--------------|-------------------|-----------------|-----------------|
| `finish_reason=length` early | `max_tokens` < reasoning + output | `docker inspect ... \| grep max_tokens` | Increase `max_tokens`, restart | Sequential validation 3/3 |
| VRAM OOM | Fragmentation, oversized context | `rocm-smi --showmeminfo vram_used` | Reduce `--ctx-size`, restart container | VRAM headroom ≥ 4GB |
| Proxy timeout | Upstream down, routing misconfig | `curl http://192.168.1.116:8181/health` | Restart proxy, verify upstream | Health endpoint 200, latency < 100ms |
| SQLite lock | Concurrent writers, WAL disabled | `sqlite3 /models/benchmark.db "PRAGMA journal_mode;"` | Enable WAL, kill rogue processes | `PRAGMA integrity_check` returns ok |
| GPU hang | Driver crash, thermal throttling | `dmesg -T \| grep -i amdgpu` | Reset GPU, check cooling | `rocm-smi` stable, no hang messages |
| High rejection rate | Draft/target mismatch, context drift | `curl ... \| jq '.usage.reasoning_tokens'` | Reduce `--spec-draft-n-max`, verify alignment | Rejection < 20%, TTFT improved |
| Disk exhaustion | Log bloat, WAL growth, backup retention | `df -h /models`, `ls -lh /models/evidence/` | Rotate logs, vacuum DB, prune backups | Disk usage < 80%, rotation active |
| Network exposure | `0.0.0.0` binding, missing firewall | `ss -tulpn \| grep 8181`, `iptables -L` | Restrict binding, add DROP rules | Ports bound to internal IPs only |

This matrix provides rapid triage paths. Follow diagnostic → resolution → validation sequence strictly. Document all steps. Escalate if unresolved after 3 iterations. Maintain cautious, reproducible, and privacy-first operations at all times.