# Operating Principles

This runbook is engineered for a cautious, methodical, and privacy-conscious home-lab AI server environment. The infrastructure hosts a specialized Vulkan-accelerated `llama.cpp` container running a large, multi-token prediction (MTP) enabled model with extended context and reasoning capabilities. Given the hardware constraints (32GB VRAM), the software complexity (speculative decoding, reasoning budgets, long-context handling), and the requirement for reliable benchmark reporting without exposing private data, the following operating principles govern all diagnostic, validation, and remediation activities.

1. **Read-Only First, Write-Only With Approval**: Every diagnostic action begins with non-invasive inspection. No configuration changes, container restarts, or database modifications occur without explicit human approval and a documented rollback path. The default stance is observation and logging.
2. **Incremental Validation Over Bulk Execution**: Long-output and reasoning-heavy benchmarks are inherently unstable when run in parallel or bulk. Each task must be validated individually before scaling. Bulk runs are strictly prohibited until single-task success rates exceed 95% across three consecutive validation cycles.
3. **Explicit Risk Classification**: Every command, script, or configuration change is labeled with a risk tier: `read-only`, `low-risk`, `medium-risk`, or `destructive`. Operators must verify the classification before execution. `medium-risk` and `destructive` actions require a documented justification and human sign-off.
4. **Data Privacy and Local-Only Enforcement**: The SQLite database resides on `/models` and contains benchmark results, prompt/response pairs, and metadata. No raw data, logs, or screenshots leave the host machine unless explicitly sanitized and approved. Network exposure is restricted to localhost and the internal LAN (192.168.1.0/24). Public release of private raw data is strictly forbidden.
5. **Reproducibility and Audit Trails**: All benchmark runs, configuration changes, and diagnostic sessions are logged with timestamps, container IDs, model hashes, and operator notes. Every action is traceable. Rollback procedures are pre-tested and documented before deployment.
6. **Conservative Resource Allocation**: The 32GB VRAM on the AMD Radeon AI PRO R9700 is carefully partitioned. The model (Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf) requires careful memory mapping. Overcommitment leads to silent degradation, VRAM leaks, or host OOM kills. All operations respect hard limits on context length, parallel slots, and speculative decoding depth.
7. **Fail-Safe Defaults**: If any subsystem exhibits anomalous behavior (e.g., `finish_reason=length` truncation, proxy timeouts, VRAM spikes, database locks), the system defaults to a safe state: pause benchmarking, capture evidence, isolate the fault, and await human review. Automation never overrides safety thresholds.

These principles ensure that benchmark infrastructure remains stable, auditable, and secure while accommodating the advanced features of modern LLM inference (MTP, reasoning budgets, long context). Adherence to these principles is mandatory for all automated agents and human operators.

# Preflight Checks

Before initiating any benchmark run, diagnostic session, or configuration change, the following preflight checks must be executed in sequence. These checks verify system readiness, container health, hardware availability, and data integrity. Failure at any step halts progression until resolution.

## 1. Network and Host Connectivity
Verify that the host is reachable and internal services are bound correctly.
```bash
# Classification: read-only
ping -c 3 192.168.1.116
```
Expected: 0% packet loss. If unreachable, check physical network, DHCP/static IP assignment, and firewall rules.

```bash
# Classification: read-only
ss -tlnp | grep -E '8181|8090|8080'
```
Expected: Ports 8181 (Flight Recorder proxy), 8090 (ServerTop), and 8080 (llama.cpp server) are listening on `0.0.0.0` or `127.0.0.1`. Mismatched bindings indicate misconfigured container or proxy.

## 2. Container Status and Resource Limits
Confirm the `llama-cpp-server-vulkan:working-20260613` container is running, healthy, and within resource bounds.
```bash
# Classification: read-only
docker ps --filter "ancestor=llama-cpp-server-vulkan:working-20260613" --format "table {{.ID}}\t{{.Status}}\t{{.Ports}}"
```
Expected: Status shows `Up` with a reasonable uptime. Ports match expected mappings. If `Restarting` or `Exited`, inspect logs immediately.

```bash
# Classification: read-only
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}" $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
```
Expected: CPU usage stable, memory usage within host limits, network I/O nominal. Spikes indicate background compilation, VRAM swapping, or proxy overhead.

## 3. Disk Space and Model Integrity
Verify sufficient storage for logs, database growth, and model files. Confirm model hash matches expected value.
```bash
# Classification: read-only
df -h /models /var/lib/docker /tmp
```
Expected: At least 15% free space on all mounted partitions. Low disk space causes SQLite locks, log rotation failures, and container crashes.

```bash
# Classification: read-only
sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
```
Expected: Hash matches the known-good value from the model repository. Mismatch indicates corruption or incomplete download. If mismatched, halt and re-download with verification.

## 4. GPU and Driver Readiness
Confirm AMD GPU is visible to the host and container, and Vulkan/ROCm drivers are functional.
```bash
# Classification: read-only
rocm-smi --showproductname --showmeminfo.vram_used --showtemp
```
Expected: Radeon AI PRO R9700 listed, VRAM usage baseline (~2-4GB idle), temperature < 65°C. If GPU not listed, check `amdgpu` kernel module, PCIe enumeration, and container device passthrough.

```bash
# Classification: read-only
vulkaninfo --summary | grep -i "device name\|api version"
```
Expected: Device name matches GPU, API version >= 1.3. Vulkan initialization failures cause silent fallback to CPU or container crash.

## 5. Environment and Configuration Validation
Verify critical environment variables and startup flags match the benchmark specification.
```bash
# Classification: read-only
docker inspect $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") --format '{{.Config.Cmd}}'
```
Expected: Command includes `--spec-type draft-mtp --spec-draft-n-max 2 --reasoning-budget 8192 --ctx-size 262144 --parallel 1`. Missing flags indicate misconfiguration.

```bash
# Classification: read-only
docker exec $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") env | grep -E "LLAMA|VULKAN|GPU"
```
Expected: `LLAMA_VULKAN=1`, `GPU_DEVICE=0`, `HSA_OVERRIDE_GFX_VERSION` (if applicable) set correctly. Incorrect env vars cause suboptimal memory allocation or driver fallback.

All preflight checks must pass before proceeding. Any failure triggers the Failure Triage Trees section. Document results in the audit log.

# Safe Inspection Commands

This section catalogs non-invasive and low-risk commands for continuous monitoring, log analysis, and state verification. Commands are strictly classified to prevent accidental system modification. Use these commands during active benchmarking, post-run analysis, and routine health checks.

## System and Host Inspection
```bash
# Classification: read-only
uptime -p
```
Reveals host uptime and load average. High load (>4.0 on a 4-core host) indicates background compilation, VRAM swapping, or proxy contention.

```bash
# Classification: read-only
free -h
```
Displays RAM and swap usage. Swap usage > 0 during inference indicates VRAM overflow or host memory pressure. Trigger rollback if swap activates.

```bash
# Classification: read-only
journalctl -u docker.service --since "1 hour ago" --no-pager | tail -n 50
```
Reviews recent Docker daemon logs. Look for `OOMKilled`, `device or resource busy`, or `Vulkan initialization failed`.

## Container and Process Inspection
```bash
# Classification: read-only
docker logs --tail 100 $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
```
Displays recent container stdout/stderr. Critical for spotting `finish_reason=length`, MTP divergence warnings, or VRAM allocation errors.

```bash
# Classification: read-only
docker top $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
```
Lists running processes inside the container. Verify only `llama-server` and necessary Vulkan/ROCm helpers are active. Unexpected processes indicate container compromise or misconfiguration.

```bash
# Classification: low-risk
docker exec $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") cat /proc/1/status | grep -E "VmRSS|VmSize|Threads"
```
Reads memory and thread count of the main process. `VmRSS` should align with expected model footprint (~16-20GB for 35B A3B). `Threads` should match CPU cores + Vulkan worker threads.

## Network and Proxy Inspection
```bash
# Classification: read-only
curl -s -o /dev/null -w "%{http_code} %{time_total}s" http://192.168.1.116:8181/v1/models
```
Tests Flight Recorder proxy health and latency. Expected: `200` status, latency < 0.5s. `502` or `504` indicates proxy misrouting or backend unavailability.

```bash
# Classification: read-only
curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"test","messages":[{"role":"user","content":"ping"}],"max_tokens":10}' | jq -r '.finish_reason // "N/A"'
```
Validates proxy routing to llama.cpp. Expected: `stop` or `length`. `length` on a 10-token request indicates misconfigured `max_tokens` or proxy override.

## Storage and Database Inspection
```bash
# Classification: read-only
ls -lah /models/*.db
```
Verifies SQLite database file permissions and size. Rapid size growth indicates unbounded logging or failed log rotation.

```bash
# Classification: read-only
sqlite3 /models/benchmark.db "PRAGMA integrity_check;"
```
Runs SQLite integrity check. Expected: `ok`. Any other output indicates corruption. Halt all writes immediately.

```bash
# Classification: low-risk
sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM runs WHERE status='completed';"
```
Counts successful benchmark runs. Use for trend analysis and validation rate tracking.

All commands in this section are safe for routine use. `low-risk` commands read process or database state but do not modify it. Never execute unclassified or unverified commands in production.

# Reasoning Budget Verification

The `--reasoning-budget 8192` flag allocates a dedicated token window for chain-of-thought or internal reasoning before final output generation. Misconfiguration or budget exhaustion directly causes `finish_reason=length` truncation, especially when combined with long-context prompts and MTP speculative decoding. This section details verification procedures, log analysis, and adjustment protocols.

## Understanding Budget Allocation
The reasoning budget is separate from `max_tokens`. The total token limit is effectively `reasoning_budget + max_tokens`. If the model's internal reasoning exceeds 8192 tokens, or if `max_tokens` is too low for the final response, the server terminates generation prematurely. MTP (`--spec-draft-n-max 2`) can exacerbate this by drafting multiple tokens per step, potentially consuming budget faster than expected.

## Verification Commands
```bash
# Classification: read-only
docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") 2>&1 | grep -i "reasoning\|budget\|token" | tail -n 20
```
Extracts reasoning-related log entries. Look for `reasoning_budget_exhausted`, `speculative_draft_tokens`, or `max_tokens_reached`.

```bash
# Classification: read-only
curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Explain quantum entanglement step-by-step."}],"max_tokens":2048,"reasoning_budget":8192}' | jq -r '{finish_reason: .finish_reason, usage: .usage}'
```
Tests budget allocation with a controlled prompt. Expected: `finish_reason: stop`, `usage.total_tokens` <= 10240. If `finish_reason: length`, budget or `max_tokens` is insufficient.

## Budget Adjustment Protocol
If `finish_reason=length` occurs during reasoning:
1. Increase `--reasoning-budget` in increments of 2048 (e.g., 10240, 12288).
2. Verify VRAM headroom. Each 2048 token increase consumes ~50-100MB VRAM depending on KV cache configuration.
3. Restart container with new flag.
4. Re-run validation prompt.
5. Document new budget and VRAM impact.

```bash
# Classification: medium-risk
docker stop $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") && docker start $(docker ps -aq --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
```
Restarts container to apply budget changes. Only execute after human approval and VRAM verification.

## MTP Interaction Considerations
Speculative decoding drafts tokens ahead of verification. If `--spec-draft-n-max 2` is active, the model may consume reasoning budget on draft tokens that are later rejected. Monitor `speculative_acceptance_rate` in logs. If rate < 60%, reduce `--spec-draft-n-max` to 1 or disable MTP temporarily to isolate budget exhaustion.

```bash
# Classification: read-only
docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") 2>&1 | grep -i "speculative\|acceptance" | tail -n 10
```
Checks MTP efficiency. Low acceptance rates indicate model mismatch or prompt complexity exceeding draft capabilities.

Reasoning budget verification must precede any long-output benchmark. Never assume budget sufficiency without empirical validation.

# Long Output Validation

The observation that `finish_reason=length` truncated responses before final content appeared is a critical benchmark integrity issue. Long-output validation ensures complete generation, proper token accounting, and stable MTP/reasoning interaction. Bulk runs are prohibited until single-task validation passes.

## Validation Methodology
1. **Single-Task Isolation**: Run one benchmark task at a time. Disable parallel slots (`--parallel 1` is already set).
2. **Incremental `max_tokens` Scaling**: Start with `max_tokens=4096`. If truncated, increase by 2048 until `finish_reason=stop` consistently.
3. **Output Completeness Check**: Verify that `reasoning_content` and `final_content` are both present and non-empty. Parse JSON response to confirm structure.
4. **MTP Stability Check**: Monitor speculative decoding logs for divergence or excessive rejections during long generation.

## Validation Commands
```bash
# Classification: read-only
curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Write a detailed technical report on Vulkan compute shaders."}],"max_tokens":4096,"reasoning_budget":8192}' | jq -r '{finish_reason: .finish_reason, has_reasoning: (.choices[0].message.reasoning_content != null), has_content: (.choices[0].message.content != null), token_count: .usage.total_tokens}'
```
Validates single-task output structure. Expected: `finish_reason: stop`, `has_reasoning: true`, `has_content: true`. If `finish_reason: length`, increase `max_tokens`.

```bash
# Classification: low-risk
docker exec $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") cat /proc/1/fdinfo/0 2>/dev/null | grep -i "pos"
```
Checks file descriptor state for log streaming. Not directly related to generation, but useful for debugging container I/O bottlenecks during long runs.

## Handling Truncation
If `finish_reason=length` persists after increasing `max_tokens`:
1. Verify `--ctx-size 262144` is not being overridden by proxy or client headers.
2. Check for hidden system prompts consuming context window.
3. Disable MTP temporarily (`--spec-type none`) to rule out speculative decoding token accounting bugs.
4. Monitor VRAM during generation. KV cache growth may trigger early termination if VRAM limits are hit.

```bash
# Classification: medium-risk
docker stop $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") && docker run -d --name llama-test --gpus all -v /models:/models -e LLAMA_VULKAN=1 llama-cpp-server-vulkan:working-20260613 --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --ctx-size 262144 --parallel 1 --reasoning-budget 8192 --spec-type none --max-tokens 8192
```
Launches isolated test container without MTP. Use only for debugging. Remove after validation.

## Bulk Run Prohibition
Never execute more than one long-output task concurrently. Use a queueing mechanism with explicit validation gates between tasks. Log each task's `finish_reason`, token counts, and VRAM peak. Archive results locally. Do not proceed to bulk execution until 10 consecutive single-task validations pass with `finish_reason=stop`.

Long output validation is the primary gate for benchmark integrity. Truncated responses invalidate performance metrics and reasoning quality scores.

# GPU And VRAM Checks

The AMD Radeon AI PRO R9700 (32GB VRAM) is the inference accelerator. VRAM management is critical for 35B parameter models with extended context and MTP. Overcommitment causes silent degradation, host OOM kills, or container crashes. This section details monitoring, allocation verification, and thermal/power checks.

## VRAM Allocation Verification
The model requires ~16-20GB VRAM for weights and KV cache. Context length (262144) and parallel slots (1) dictate KV cache size. MTP adds temporary draft buffers. Total VRAM usage must stay below 28GB to leave headroom for Vulkan drivers and host OS.

```bash
# Classification: read-only
rocm-smi --showmeminfo.vram_used --showmeminfo.vram_free --showproductname
```
Displays real-time VRAM usage. Expected: Baseline ~4-6GB idle, peak < 28GB during inference. If `vram_free` drops below 2GB, halt benchmarking and reduce context or disable MTP.

```bash
# Classification: read-only
docker stats --no-stream --format "{{.MemUsage}}" $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
```
Shows container memory usage. Note: This reflects host RAM, not VRAM. VRAM must be monitored via `rocm-smi` or Vulkan tools.

## Driver and Vulkan Validation
Ensure ROCm/Vulkan drivers are correctly initialized and GPU is visible to the container.

```bash
# Classification: read-only
vulkaninfo --summary | grep -A 5 "deviceName\|driverVersion\|apiVersion"
```
Verifies Vulkan stack. Expected: `deviceName: Radeon AI PRO R9700`, `apiVersion >= 1.3`. Mismatch indicates driver issue or container misconfiguration.

```bash
# Classification: read-only
ls -la /dev/dri/
```
Checks DRM device nodes. Expected: `card0`, `renderD128` present. Missing nodes cause Vulkan initialization failure.

## Thermal and Power Monitoring
Sustained inference generates heat. Thermal throttling reduces token generation speed and may cause instability.

```bash
# Classification: read-only
rocm-smi --showtemp --showpower
```
Displays GPU temperature and power draw. Expected: Temp < 75°C under load, power < 250W. If temp > 80°C, check cooling, reduce clock speeds, or lower batch size.

```bash
# Classification: read-only
sensors | grep -i "amdgpu\|coretemp"
```
Cross-references host thermal sensors. Discrepancies between `rocm-smi` and `sensors` indicate driver or hardware monitoring issues.

## VRAM Leak Detection
Long-running containers may accumulate VRAM fragments. Monitor for gradual VRAM increase without corresponding load.

```bash
# Classification: low-risk
watch -n 5 'rocm-smi --showmeminfo.vram_used --showproductname'
```
Continuous VRAM monitoring. If usage climbs steadily during idle periods, restart container to reclaim VRAM. Document leak pattern for vendor reporting.

GPU and VRAM checks must be performed before, during, and after benchmark runs. Never ignore thermal warnings or VRAM headroom violations.

# Proxy And Database Checks

The Flight Recorder proxy (`http://192.168.1.116:8181/v1`) routes benchmark requests to the llama.cpp container and logs interactions. The SQLite database (`/models/benchmark.db`) stores results. Both must be validated for health, routing accuracy, and data integrity.

## Proxy Health and Routing
```bash
# Classification: read-only
curl -s -o /dev/null -w "%{http_code} %{time_connect}s %{time_total}s" http://192.168.1.116:8181/v1/models
```
Tests proxy connectivity and latency. Expected: `200`, connect time < 0.1s, total time < 0.5s. `502`/`504` indicates backend unavailability or proxy misconfiguration.

```bash
# Classification: read-only
curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"test","messages":[{"role":"user","content":"ping"}],"max_tokens":5}' | jq -r '.id // "N/A"'
```
Validates request routing and response structure. Expected: Valid request ID. `null` or error indicates proxy failure.

## Database Integrity and Schema
```bash
# Classification: read-only
sqlite3 /models/benchmark.db "PRAGMA integrity_check;"
```
Verifies database structure. Expected: `ok`. Any corruption requires immediate backup and restore from last known good snapshot.

```bash
# Classification: read-only
sqlite3 /models/benchmark.db ".schema"
```
Displays table schemas. Verify expected columns: `run_id`, `prompt_hash`, `max_tokens`, `reasoning_budget`, `finish_reason`, `usage`, `timestamp`. Missing columns indicate schema drift.

```bash
# Classification: low-risk
sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM runs WHERE finish_reason='length';"
```
Counts truncated runs. High count indicates systematic `max_tokens` or budget misconfiguration.

## Backup and Recovery
```bash
# Classification: low-risk
cp /models/benchmark.db /models/benchmark.db.backup.$(date +%Y%m%d_%H%M%S)
```
Creates timestamped backup. Execute before any schema change or bulk import. Verify backup integrity with `PRAGMA integrity_check`.

```bash
# Classification: medium-risk
sqlite3 /models/benchmark.db "VACUUM;"
```
Reclaims unused space and defragments database. Only run during maintenance windows. May lock database temporarily.

Proxy and database checks ensure reliable request routing and data persistence. Never modify database schema without backup and approval.

# Dashboard Checks

ServerTop (`http://192.168.1.116:8090`) provides real-time metrics for container performance, VRAM usage, and request throughput. Dashboard validation ensures metrics reflect actual system state and are suitable for benchmark reporting.

## Dashboard Accessibility and Sync
```bash
# Classification: read-only
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8090
```
Tests dashboard availability. Expected: `200`. `404` or `502` indicates service crash or misconfiguration.

```bash
# Classification: read-only
curl -s http://192.168.1.116:8090/api/metrics | jq -r '.gpu.vram_used_mb // "N/A"'
```
Extracts VRAM metric from dashboard API. Cross-reference with `rocm-smi`. Discrepancies > 500MB indicate dashboard caching or polling delay.

## Screenshot Capture Procedure
For benchmark reporting, capture dashboard state at key intervals:
1. Navigate to `http://192.168.1.116:8090` in a headless browser or local workstation.
2. Wait 10 seconds for metrics to stabilize.
3. Capture full-page screenshot.
4. Redact any IP addresses, hostnames, or internal paths if sharing externally.
5. Store in `/models/screenshots/` with timestamped filename.

```bash
# Classification: low-risk
mkdir -p /models/screenshots && cp /tmp/dashboard_capture.png /models/screenshots/dashboard_$(date +%Y%m%d_%H%M%S).png
```
Archives screenshot locally. Verify file size and integrity before deletion from temp.

## Metric Validation
Compare dashboard metrics with direct system queries:
- VRAM: Dashboard vs `rocm-smi`
- Request latency: Dashboard vs `curl` timing
- Token throughput: Dashboard vs container logs

Discrepancies > 10% require dashboard configuration review or metric source correction. Never rely solely on dashboard for critical decisions.

Dashboard checks provide visual and API-level validation. Ensure screenshots are archived and metrics cross-verified before benchmark sign-off.

# Failure Triage Trees

This section provides decision trees for common failures. Follow steps sequentially. Execute commands as classified. Document findings. Escalate to human approval if resolution requires `medium-risk` or `destructive` actions.

## Tree 1: `finish_reason=length` Premature Cutoff
1. Check `max_tokens` and `reasoning_budget` values.
   ```bash
   # Classification: read-only
   docker inspect $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") --format '{{.Config.Cmd}}'
   ```
2. If values are low, increase `max_tokens` by 2048. Restart container.
3. If values are sufficient, check MTP acceptance rate.
   ```bash
   # Classification: read-only
   docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") 2>&1 | grep -i "speculative"
   ```
4. If acceptance rate < 60%, disable MTP temporarily. Re-test.
5. If still truncated, verify VRAM headroom. Reduce context or parallel slots.
6. Document resolution. Proceed to validation.

## Tree 2: VRAM OOM or Container Crash
1. Check VRAM usage and temperature.
   ```bash
   # Classification: read-only
   rocm-smi --showmeminfo.vram_used --showtemp
   ```
2. If VRAM > 28GB, reduce `--ctx-size` or disable MTP.
3. If temperature > 80°C, check cooling. Reduce clock speeds.
4. Restart container.
   ```bash
   # Classification: medium-risk
   docker restart $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
   ```
5. Monitor for recurrence. If persistent, report to vendor or downgrade model quantization.

## Tree 3: Proxy Timeout or Routing Failure
1. Test proxy health.
   ```bash
   # Classification: read-only
   curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1/models
   ```
2. If `502`/`504`, check backend container status.
   ```bash
   # Classification: read-only
   docker ps --filter "ancestor=llama-cpp-server-vulkan:working-20260613"
   ```
3. If backend down, restart container.
4. If backend up, check proxy logs.
   ```bash
   # Classification: read-only
   journalctl -u flight-recorder-proxy --since "1 hour ago" --no-pager | tail -n 20
   ```
5. Fix routing configuration. Restart proxy. Validate.

## Tree 4: SQLite Corruption or Lock
1. Run integrity check.
   ```bash
   # Classification: read-only
   sqlite3 /models/benchmark.db "PRAGMA integrity_check;"
   ```
2. If corrupted, restore from backup.
   ```bash
   # Classification: medium-risk
   cp /models/benchmark.db.backup.* /models/benchmark.db
   ```
3. If locked, kill blocking processes.
   ```bash
   # Classification: low-risk
   fuser /models/benchmark.db
   ```
4. Vacuum database. Validate schema. Resume operations.

## Tree 5: MTP Divergence or Excessive Rejections
1. Check acceptance rate.
   ```bash
   # Classification: read-only
   docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") 2>&1 | grep -i "acceptance"
   ```
2. If rate < 50%, reduce `--spec-draft-n-max` to 1.
3. If still low, disable MTP. Compare throughput.
4. Document performance impact. Decide on MTP usage.

## Tree 6: Container Restart Loop
1. Check logs.
   ```bash
   # Classification: read-only
   docker logs --tail 50 $(docker ps -aq --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
   ```
2. Identify crash cause (OOM, Vulkan init, model load).
3. Fix configuration or hardware issue.
4. Restart container. Monitor stability.

Follow trees sequentially. Never skip steps. Escalate if resolution requires destructive actions.

# Rollback Procedures

Rollback ensures system recovery from misconfiguration, corruption, or instability. All procedures are pre-tested and documented. Execute only after human approval.

## Container Rollback
```bash
# Classification: medium-risk
docker stop $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
docker rm $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
docker run -d --name llama-cpp-server --gpus all -v /models:/models -e LLAMA_VULKAN=1 llama-cpp-server-vulkan:working-20260613 --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --ctx-size 262144 --parallel 1 --reasoning-budget 8192 --spec-type draft-mtp --spec-draft-n-max 2
```
Restarts container with known-good configuration. Verify health before resuming benchmarks.

## Model Rollback
```bash
# Classification: medium-risk
cp /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf.backup /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
```
Restores model from backup. Verify hash matches known-good value. Restart container.

## Configuration Rollback
```bash
# Classification: low-risk
cp /etc/llama-cpp-server/config.yaml.backup /etc/llama-cpp-server/config.yaml
```
Restores configuration file. Restart container. Validate flags.

## Database Rollback
```bash
# Classification: medium-risk
cp /models/benchmark.db.backup.* /models/benchmark.db
sqlite3 /models/benchmark.db "PRAGMA integrity_check;"
```
Restores database. Verify integrity. Resume operations.

All rollbacks require verification steps. Never skip hash or integrity checks. Document rollback reason and outcome.

# Human Approval Gates

Certain actions require explicit human confirmation due to risk level or operational impact. The agent must pause and request approval before proceeding.

## Gate 1: Container Restart or Configuration Change
- **Trigger**: Modifying `--reasoning-budget`, `--max-tokens`, `--spec-type`, or restarting container.
- **Approval Template**:
  ```
  ACTION: Restart container with updated flags
  RISK: medium-risk
  JUSTIFICATION: finish_reason=length truncation observed
  ROLLBACK: docker stop/rm/run with previous config
  APPROVE? [Y/N]
  ```
- **Threshold**: Always require approval for `medium-risk` or higher.

## Gate 2: Database Modification or Vacuum
- **Trigger**: Running `VACUUM`, schema changes, or bulk imports.
- **Approval Template**:
  ```
  ACTION: Vacuum benchmark database
  RISK: medium-risk
  JUSTIFICATION: Database fragmentation detected
  ROLLBACK: Restore from backup
  APPROVE? [Y/N]
  ```
- **Threshold**: Require approval for any write operation on production database.

## Gate 3: MTP or Reasoning Budget Adjustment
- **Trigger**: Changing `--spec-draft-n-max` or `--reasoning-budget`.
- **Approval Template**:
  ```
  ACTION: Increase reasoning budget to 10240
  RISK: medium-risk
  JUSTIFICATION: Budget exhaustion causing truncation
  ROLLBACK: Revert to 8192, restart container
  APPROVE? [Y/N]
  ```
- **Threshold**: Require approval for parameter changes affecting generation behavior.

## Gate 4: Bulk Benchmark Execution
- **Trigger**: Running more than one long-output task concurrently.
- **Approval Template**:
  ```
  ACTION: Execute bulk benchmark (10 tasks)
  RISK: high-risk
  JUSTIFICATION: Validation rate 100% over 10 single runs
  ROLLBACK: Pause queue, revert to single-task mode
  APPROVE? [Y/N]
  ```
- **Threshold**: Strictly prohibit bulk runs without explicit approval and validation proof.

Human approval gates enforce safety and accountability. Never bypass gates. Log all approvals and denials.

# Evidence To Capture

Comprehensive evidence collection ensures auditability, troubleshooting capability, and benchmark integrity. All evidence is stored locally and redacted before external sharing.

## Log Collection
```bash
# Classification: read-only
docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") > /models/logs/container_$(date +%Y%m%d_%H%M%S).log
journalctl -u docker.service --since "1 hour ago" --no-pager > /models/logs/docker_$(date +%Y%m%d_%H%M%S).log
```
Archives container and Docker daemon logs. Verify file size and integrity.

## Screenshot Archival
```bash
# Classification: low-risk
cp /tmp/dashboard_capture.png /models/screenshots/dashboard_$(date +%Y%m%d_%H%M%S).png
cp /tmp/proxy_health.png /models/screenshots/proxy_$(date +%Y%m%d_%H%M%S).png
```
Stores dashboard and proxy health screenshots. Redact IPs/hostnames if sharing.

## Database Snapshot
```bash
# Classification: low-risk
sqlite3 /models/benchmark.db ".dump" > /models/backups/benchmark_dump_$(date +%Y%m%d_%H%M%S).sql
```
Creates SQL dump for recovery or analysis. Verify dump syntax.

## Performance Metrics
```bash
# Classification: read-only
rocm-smi --showmeminfo.vram_used --showtemp --showpower > /models/metrics/gpu_$(date +%Y%m%d_%H%M%S).txt
curl -s http://192.168.1.116:8090/api/metrics > /models/metrics/dashboard_$(date +%Y%m%d_%H%M%S).json
```
Captures GPU and dashboard metrics. Cross-reference for consistency.

## Privacy Redaction
Before external sharing:
1. Remove IP addresses, hostnames, internal paths.
2. Hash prompt/response pairs if sharing raw data.
3. Verify no secrets or credentials in logs.
4. Document redaction steps.

Evidence capture is mandatory for all benchmark runs and diagnostic sessions. Store in `/models/` with timestamped naming. Never delete evidence until retention period expires.

# Final Go No-Go Checklist

Execute this checklist before every benchmark run. All items must pass. Document results. Obtain sign-off.

- [ ] Host reachable: `ping 192.168.1.116` (0% loss)
- [ ] Container running: `docker ps` shows `Up` status
- [ ] VRAM headroom: `rocm-smi` shows > 4GB free
- [ ] Temperature safe: `rocm-smi` shows < 75°C
- [ ] Model hash verified: `sha256sum` matches known-good
- [ ] Database integrity: `PRAGMA integrity_check` returns `ok`
- [ ] Proxy health: `curl` returns `200`, latency < 0.5s
- [ ] Dashboard sync: Metrics match `rocm-smi` and logs
- [ ] Reasoning budget validated: Single-task test returns `finish_reason=stop`
- [ ] Long output validated: `max_tokens` sufficient, no truncation
- [ ] MTP stable: Acceptance rate > 60% or disabled
- [ ] Backup current: Database and model backups exist
- [ ] Evidence capture ready: Log, screenshot, and metric paths configured
- [ ] Human approval obtained: For any `medium-risk` or higher actions
- [ ] Bulk run prohibited: Single-task validation rate 100% over 10 runs

If any item fails, halt and resolve before proceeding. Sign-off required from operator. Archive checklist with benchmark run.

This runbook provides a complete, cautious, and auditable framework for diagnosing, fixing, validating, and rolling back benchmark infrastructure problems. Adherence ensures stability, privacy, and reliable reporting. Execute with precision. Verify at every step. Never assume. Always validate.

**Advanced MTP & Speculative Decoding Diagnostics**
Multi-Token Prediction (MTP) speculative decoding significantly accelerates inference but introduces complex token accounting and divergence risks. When `--spec-type draft-mtp` and `--spec-draft-n-max 2` are active, the model generates draft tokens that must be verified by the main model. Divergence occurs when draft tokens deviate from the main model's probability distribution, causing rejections and wasted compute. Proper diagnostics require isolating draft latency, verification latency, and acceptance rates.

**Measuring Draft vs. Verify Latency**
The llama.cpp server logs speculative decoding metrics per request. To isolate bottlenecks:
```bash
# Classification: read-only
docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") 2>&1 | grep -E "speculative|draft|verify|acceptance" | tail -n 50
```
Expected output includes `draft_tokens`, `verify_tokens`, `acceptance_rate`, and `speculative_latency_ms`. If `speculative_latency_ms` exceeds 40% of total generation time, the draft model is overburdened. Reduce `--spec-draft-n-max` to 1 or switch to `--spec-type none` temporarily.

**Token Acceptance Rate Thresholds**
- **> 80%**: Optimal. MTP is accelerating inference without significant divergence.
- **60% - 80%**: Acceptable. Monitor for context-dependent drops.
- **40% - 60%**: Suboptimal. Draft model is struggling with prompt complexity or domain mismatch. Consider reducing draft depth or warming up the KV cache.
- **< 40%**: Critical. MTP is causing more overhead than acceleration. Disable speculative decoding immediately.

**Isolating MTP Bottlenecks**
Run a controlled benchmark with MTP enabled vs. disabled:
```bash
# Classification: low-risk
curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Analyze the following dataset structure and propose optimizations."}],"max_tokens":1024,"reasoning_budget":4096}' | jq -r '{tokens: .usage.total_tokens, time: .usage.total_time_ms, finish: .finish_reason}'
```
Compare `time` and `tokens` against a baseline run with `--spec-type none`. If MTP increases latency by >15%, the draft model is misaligned. Document findings and adjust `--spec-draft-n-max` accordingly.

**KV Cache & Context Window Management**
A context window of 262,144 tokens on a 32GB VRAM GPU requires aggressive KV cache management. The KV cache stores key and value vectors for every token in the context. At 35B parameters (A3B quantization), the KV cache can consume 8-12GB VRAM at full context. Combined with model weights (~16GB) and MTP draft buffers, VRAM headroom becomes critical.

**KV Cache Quantization Strategies**
llama.cpp supports KV cache quantization (`--cache-type q4_0`, `q8_0`, `f16`). Lower precision reduces VRAM usage but may impact long-context accuracy.
```bash
# Classification: read-only
docker inspect $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") --format '{{.Config.Cmd}}' | grep -o "cache-type [a-z0-9_]*"
```
Expected: `cache-type q4_0` or `q8_0`. If unset, defaults to `f16`, which may cause VRAM OOM at 262k context. If VRAM pressure is observed, switch to `q4_0` via container restart.

**Monitoring KV Cache Fragmentation**
Long-running sessions with variable context lengths cause KV cache fragmentation, leading to inefficient memory allocation and increased latency.
```bash
# Classification: read-only
docker exec $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") cat /proc/1/smaps | grep -A 5 "Vulkan\|ROCm\|GPU" | head -n 20
```
Look for fragmented memory regions. If fragmentation exceeds 20% of allocated VRAM, schedule a container restart during maintenance windows to reclaim contiguous memory blocks.

**Eviction Policies & Context Truncation**
When context approaches 262,144 tokens, the server must evict older tokens or truncate. llama.cpp uses a sliding window or static eviction policy. Verify configuration:
```bash
# Classification: read-only
docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") 2>&1 | grep -i "evict\|truncate\|context" | tail -n 10
```
Ensure `--ctx-size` matches the intended limit. Mismatched values cause silent truncation or generation failures. Document eviction behavior for benchmark reproducibility.

**Automated Validation Scripts & Cron Jobs**
Manual checks are insufficient for continuous operation. Automated scripts provide consistent monitoring, but must be guarded against runaway processes or destructive actions.

**Health Check Script**
Create `/opt/scripts/health_check.sh`:
```bash
#!/bin/bash
# Classification: low-risk
CONTAINER=$(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
if [ -z "$CONTAINER" ]; then
    echo "CRITICAL: Container not running" | mail -s "LLAMA-CPP ALERT" admin@localhost
    exit 1
fi
VRAM_FREE=$(rocm-smi --showmeminfo.vram_free --json | jq -r '.[0].vram_free')
if [ "$VRAM_FREE" -lt 2048 ]; then
    echo "WARNING: VRAM headroom below 2GB" | mail -s "LLAMA-CPP VRAM ALERT" admin@localhost
fi
DB_CHECK=$(sqlite3 /models/benchmark.db "PRAGMA integrity_check;")
if [ "$DB_CHECK" != "ok" ]; then
    echo "CRITICAL: Database corruption detected" | mail -s "LLAMA-CPP DB ALERT" admin@localhost
fi
```
Make executable: `chmod +x /opt/scripts/health_check.sh`

**Cron Scheduling**
```bash
# Classification: low-risk
crontab -l | grep -v "llama-cpp" > /tmp/cron_temp
echo "*/15 * * * * /opt/scripts/health_check.sh >> /var/log/llama-health.log 2>&1" >> /tmp/cron_temp
echo "0 2 * * * cp /models/benchmark.db /models/backups/benchmark_$(date +\%Y\%m\%d).db" >> /tmp/cron_temp
crontab /tmp/cron_temp
```
Verify: `crontab -l`. Ensure no overlapping jobs or resource-intensive tasks run during peak benchmark hours.

**Safety Guards in Automation**
- Never automate `docker restart` or `VACUUM` without explicit approval gates.
- Log all automated actions to `/var/log/llama-automation.log`.
- Implement circuit breakers: if health checks fail 3 consecutive times, halt automation and alert operator.

**Data Sanitization & Export Protocols**
Benchmark results must be exportable for reporting without exposing private prompts, responses, or internal metadata. Raw data remains strictly local.

**Sanitization Pipeline**
1. Extract required fields: `run_id`, `model`, `max_tokens`, `reasoning_budget`, `finish_reason`, `usage.total_tokens`, `latency_ms`.
2. Hash prompt/response pairs: `sha256sum` or `blake3` for deduplication and privacy.
3. Strip internal IPs, hostnames, and container IDs.
4. Export to CSV/JSON with metadata stripped.

**SQLite Export Command**
```bash
# Classification: low-risk
sqlite3 /models/benchmark.db "SELECT run_id, model, max_tokens, reasoning_budget, finish_reason, usage, latency_ms FROM runs WHERE status='completed' AND timestamp > datetime('now', '-7 days');" | sed 's/192\.168\.[0-9]*\.[0-9]*/LAN_IP/g' > /tmp/sanitized_export.csv
```
Verify output: `head -n 5 /tmp/sanitized_export.csv`. Ensure no raw prompts/responses or internal paths remain. Archive to `/models/exports/` with timestamp.

**Incident Response & Escalation Matrix**
Structured response ensures rapid containment and recovery.

**Severity Levels**
- **P1 (Critical)**: Container crash, VRAM OOM, database corruption, data leak. Response: Immediate halt, rollback, operator notification within 5 minutes.
- **P2 (High)**: `finish_reason=length` truncation > 20% of runs, proxy timeouts, thermal throttling. Response: Isolate fault, adjust configuration, notify operator within 15 minutes.
- **P3 (Medium)**: MTP acceptance rate < 60%, dashboard metric drift, log rotation failure. Response: Schedule maintenance, document workaround, notify operator within 1 hour.
- **P4 (Low)**: Minor latency spikes, cosmetic dashboard issues, non-critical warnings. Response: Log for trend analysis, address in next maintenance window.

**Response Templates**
```
INCIDENT: [P1-P4] [Brief Description]
DETECTED: [Timestamp]
IMPACT: [Affected subsystems]
ACTION: [Immediate steps taken]
ROLLBACK: [If applicable]
STATUS: [Open/Investigating/Resolved]
```
Maintain incident log in `/models/incidents/`. Review weekly for pattern analysis.

**Performance Baseline & Regression Testing**
Establish baselines to detect degradation from configuration drift, driver updates, or hardware aging.

**Baseline Metrics**
- Tokens/sec (draft vs. verify)
- Latency (first token, total)
- VRAM peak usage
- MTP acceptance rate
- Database write latency

**Regression Detection**
```bash
# Classification: read-only
sqlite3 /models/benchmark.db "SELECT AVG(latency_ms), AVG(usage.total_tokens) FROM runs WHERE timestamp > datetime('now', '-24 hours') GROUP BY model;"
```
Compare against 7-day rolling average. If latency increases > 10% or tokens/sec drops > 15%, trigger P2 investigation. Check driver versions, container image tags, and thermal logs.

**A/B Testing Configurations**
Never modify production configuration directly. Use isolated test containers:
```bash
# Classification: medium-risk
docker run -d --name llama-ab-test --gpus all -v /models:/models -e LLAMA_VULKAN=1 llama-cpp-server-vulkan:working-20260613 --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --ctx-size 262144 --parallel 1 --reasoning-budget 10240 --spec-type draft-mtp --spec-draft-n-max 1
```
Run controlled benchmarks against production. Compare metrics. If test outperforms, schedule production update with human approval. Remove test container immediately after validation.

**Container Image & Dependency Management**
Pin image tags to specific digests to prevent silent updates. Verify integrity before deployment.

**Image Verification**
```bash
# Classification: read-only
docker image inspect llama-cpp-server-vulkan:working-20260613 --format '{{.RepoDigests}}'
```
Record digest in `/opt/config/image_digests.txt`. Before any update, compare new digest against known-good registry. Never pull `latest` or untagged images.

**Dependency Audit**
```bash
# Classification: read-only
docker exec $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") ldconfig -p | grep -E "vulkan|rocm|cuda"
```
Verify shared libraries match expected versions. Mismatched libraries cause runtime crashes or silent fallbacks. Document versions in `/opt/config/dependency_audit.log`.

**Network & Firewall Hardening**
Restrict access to internal services. Prevent unauthorized access or data exfiltration.

**Firewall Rules**
```bash
# Classification: medium-risk
iptables -A INPUT -p tcp --dport 8181 -s 192.168.1.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 8181 -j DROP
iptables -A INPUT -p tcp --dport 8090 -s 192.168.1.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 8090 -j DROP
iptables -A INPUT -p tcp --dport 8080 -s 127.0.0.1 -j ACCEPT
iptables -A INPUT -p tcp --dport 8080 -j DROP
```
Verify: `iptables -L -n -v`. Ensure no open ports to external networks. Save rules: `iptables-save > /etc/iptables/rules.v4`.

**TLS/HTTPS Considerations**
Even on LAN, encrypt traffic to prevent sniffing. Use self-signed certificates for proxy and dashboard.
```bash
# Classification: low-risk
openssl req -x509 -newkey rsa:4096 -keyout /etc/ssl/private/server.key -out /etc/ssl/certs/server.crt -days 365 -nodes -subj "/CN=192.168.1.116"
```
Update proxy and dashboard configs to use TLS. Verify with `curl -k https://192.168.1.116:8181/v1/models`.

**Operator Training & Shift Handover Procedures**
Consistent operations require standardized knowledge transfer and documentation.

**Handover Checklist**
- [ ] Current container status and uptime
- [ ] Active benchmark queue and validation rate
- [ ] Recent incidents and resolutions
- [ ] Pending configuration changes
- [ ] Backup status and integrity
- [ ] Known issues and workarounds
- [ ] Next maintenance window schedule

**Documentation Standards**
- All changes logged in `/opt/logs/changes.log` with timestamp, operator, justification, and rollback path.
- Screenshots archived with metadata (date, time, metric values, redaction status).
- Scripts version-controlled in local Git repository.
- Runbook updated quarterly or after major incidents.

**Simulation Drills**
Conduct monthly failure simulations:
1. Simulate VRAM OOM by running max-context prompt. Verify rollback procedure.
2. Simulate database lock by concurrent writes. Verify recovery.
3. Simulate proxy timeout by stopping backend. Verify alerting and failover.
Document drill results and update runbook accordingly.

**Continuous Improvement Loop**
Benchmark infrastructure evolves. Regularly review:
- Token generation efficiency vs. VRAM usage
- MTP acceptance trends across prompt types
- Database growth rate and retention policies
- Driver updates and compatibility matrices
- Security patches and vulnerability scans
Maintain a living document. Archive obsolete procedures. Train operators on new protocols. Ensure all changes align with operating principles.

This expanded runbook provides comprehensive coverage for advanced diagnostics, automation, security, incident response, and operational continuity. Execute with discipline. Validate relentlessly. Protect data. Maintain stability. The infrastructure is only as reliable as its operators. Adhere to these procedures without deviation. Document everything. Verify before acting. Rollback when uncertain. The benchmark harness demands precision, and this runbook delivers it.