# Operating Principles

This runbook establishes the operational doctrine for diagnosing, fixing, validating, and rolling back benchmark infrastructure problems within the specified home-lab AI environment. The principles below govern all agent actions, ensuring safety, reproducibility, privacy preservation, and benchmark integrity.

**1. Cautious-by-Default Posture**
Every action is treated as potentially impactful until proven otherwise. Changes are atomic, version-pinned, and accompanied by immediate rollback readiness. No configuration is modified in-place without a backup snapshot or configuration export. The agent assumes that any state change could affect concurrent benchmark tasks, proxy routing, or database integrity.

**2. Risk Classification System**
All shell commands, API calls, and configuration edits are explicitly classified:
- `[read-only]`: Inspects state, reads logs, queries metrics, or prints configuration. Zero side effects.
- `[low-risk]`: Creates backups, generates reports, or performs non-destructive queries. Reversible without service interruption.
- `[medium-risk]`: Restarts services, pulls new images, modifies runtime flags, or triggers database maintenance. May cause brief downtime or require validation.
- `[destructive]`: Deletes files, removes containers, truncates databases, or forces hardware resets. Requires explicit human approval and documented rollback path.

**3. Privacy and Data Sovereignty**
The SQLite database at `/models` contains private benchmark payloads, raw model outputs, and internal routing metadata. It must never leave the host LAN. All exported artifacts are sanitized before archival. Screenshots capture only UI elements, metrics, and benchmark results; raw JSON payloads containing internal IPs, model weights, or user prompts are redacted or hashed. The Flight Recorder proxy at `http://192.168.1.116:8181/v1` is treated as an internal routing layer; its logs are retained locally and never transmitted externally.

**4. Benchmark Integrity and Sequential Validation**
Long-output tests are strictly prohibited from running in bulk without per-task validation. The agent enforces sequential execution with explicit dry-runs, token estimation, and `max_tokens` tuning before scaling. `finish_reason=length` is treated as a structural failure until proven benign. The reasoning budget (`--reasoning-budget 8192`) and MTP configuration (`--spec-type draft-mtp --spec-draft-n-max 2`) are monitored as coupled subsystems; altering one requires revalidation of the other.

**5. Change Management and Auditability**
All modifications are timestamped, logged, and accompanied by a change ticket reference. The agent maintains a local runbook state file tracking current container image, model checksum, proxy configuration, and database schema version. Rollbacks are prioritized over hotfixes unless human approval grants emergency modification rights.

**6. Communication and Escalation**
The agent operates asynchronously but maintains a structured reporting cadence. Alerts are routed through ServerTop thresholds and local log aggregation. Human approval gates are enforced at predefined decision points. No automated deployment proceeds without passing the Final Go/No-Go Checklist.

---

# Preflight Checks

Before initiating any benchmark cycle or diagnostic sequence, the agent must verify baseline system health, container state, model integrity, and network routing. Preflight checks establish a known-good state and prevent cascading failures during high-load inference.

**1. Host OS and Kernel Validation**
- Verify ROCm/Vulkan driver compatibility with the AMD Radeon AI PRO R9700.
- Confirm kernel modules are loaded and no pending updates require reboot.
- Check system resource availability (CPU cores, RAM, swap, disk I/O).
- Validate that the host time is synchronized (NTP) to ensure accurate benchmark timestamps.

**2. Container Environment Verification**
- Confirm the llama.cpp container is running the exact tag: `llama-cpp-server-vulkan:working-20260613`.
- Verify volume mounts: `/models` is bind-mounted read-write for SQLite and model storage.
- Check resource limits: CPU shares, memory caps, and GPU device passthrough (`--gpus all` or ROCm-specific flags).
- Validate network mode: host networking or bridge with explicit port mappings for 8181 and 8090.

**3. Model File Integrity**
- Verify the GGUF file exists: `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`.
- Compute SHA256 checksum and compare against the trusted manifest.
- Validate GGUF header using `llama-model-info` or equivalent tool to confirm architecture, context length, and MTP compatibility.
- Ensure file permissions allow the container user to read without world-access.

**4. Network and Proxy Routing**
- Test connectivity to `http://192.168.1.116:8181/v1` (Flight Recorder proxy).
- Test connectivity to `http://192.168.1.116:8090` (ServerTop dashboard).
- Verify DNS resolution and firewall rules allow internal traffic on ports 8181 and 8090.
- Confirm no proxy rate-limiting or caching rules interfere with benchmark payloads.

**5. Baseline Metrics Capture**
- Record current VRAM allocation, context window utilization, and KV cache fragmentation.
- Capture SQLite WAL status, checkpoint count, and database size.
- Log proxy latency, request queue depth, and ServerTop metric freshness.
- Store baseline in a local JSON artifact for post-benchmark comparison.

**Preflight Command Examples:**
```bash
# [read-only] Verify container status and image tag
docker ps --filter "name=llama-cpp" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

# [read-only] Check model file integrity
sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf

# [read-only] Validate proxy health endpoint
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1/health

# [low-risk] Capture baseline metrics snapshot
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}" > /tmp/baseline_metrics_$(date +%Y%m%d_%H%M%S).csv
```

**Validation Criteria:**
- Container status must be `Up` with the exact image tag.
- SHA256 must match the trusted manifest within 0% tolerance.
- Proxy health must return `200` or `204`.
- Baseline metrics must be stored and timestamped.
- Any deviation triggers a `[medium-risk]` investigation before proceeding.

---

# Safe Inspection Commands

This section catalogs the exact shell commands used for system inspection, classified by risk level. Each command includes flags, expected output, interpretation guidelines, and validation steps.

**[read-only] Container and Process Inspection**
```bash
# List running containers with resource usage
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

# Inspect container configuration and mounts
docker inspect llama-cpp --format '{{json .Mounts}}' | jq '.'

# View container logs (last 500 lines)
docker logs --tail 500 llama-cpp 2>&1 | head -n 500
```
*Interpretation:* Verify container name, image tag, mount points, and port mappings. Logs should show successful model loading, Vulkan initialization, and proxy registration. Errors like `Vulkan validation layer failed` or `CUDA/ROCm device not found` require immediate triage.

**[read-only] GPU and Driver Inspection**
```bash
# AMD ROCm system monitor
rocm-smi --showmeminfo vram --showclocks --showtemp --showpower

# Vulkan device enumeration
vulkaninfo --summary | grep -A 5 "deviceName"

# Container GPU passthrough verification
docker run --rm --gpus all nvtop  # or rocm-specific equivalent
```
*Interpretation:* VRAM usage should reflect model load + context window. Clocks and temperatures must be within safe operating ranges. Vulkan device name must match AMD Radeon AI PRO R9700. Missing or mismatched devices indicate driver or passthrough misconfiguration.

**[read-only] Database and Proxy Inspection**
```bash
# SQLite integrity check
sqlite3 /models/benchmark.db "PRAGMA integrity_check;"

# SQLite connection pool and WAL status
sqlite3 /models/benchmark.db "PRAGMA journal_mode; PRAGMA wal_checkpoint(TRUNCATE);"

# Proxy routing and rate limit status
curl -s http://192.168.1.116:8181/v1/config | jq '.rate_limit, .routing_table'
```
*Interpretation:* Integrity check must return `ok`. Journal mode should be `wal` for concurrent benchmark access. Proxy config should show active routing rules and reasonable rate limits. Any `corruption`, `locked`, or `timeout` errors require `[medium-risk]` intervention.

**[low-risk] Log Aggregation and Metric Export**
```bash
# Export container logs to local archive
docker logs --since 2h llama-cpp 2>&1 > /tmp/logs_llama-cpp_$(date +%Y%m%d_%H%M%S).log

# Export ServerTop metrics snapshot
curl -s http://192.168.1.116:8090/api/metrics/export > /tmp/servertop_metrics_$(date +%Y%m%d_%H%M%S).json

# Backup SQLite database (read-only copy)
sqlite3 /models/benchmark.db ".backup /tmp/benchmark.db.bak_$(date +%Y%m%d_%H%M%S)"
```
*Interpretation:* Archives must be stored locally in an encrypted directory. Metrics export should contain timestamped CPU, VRAM, proxy latency, and benchmark throughput. Backup size should match database size within 5%.

**[medium-risk] Service Restart and Configuration Reload**
```bash
# Graceful container restart (preserves volumes)
docker restart llama-cpp

# Reload proxy configuration (if managed by systemd)
systemctl reload flight-recorder-proxy

# Trigger SQLite checkpoint to free WAL space
sqlite3 /models/benchmark.db "PRAGMA wal_checkpoint(PASSIVE);"
```
*Interpretation:* Restarts cause brief inference downtime. Proxy reload may drop active connections. SQLite checkpoint is safe but may cause temporary I/O spikes. All medium-risk commands require post-action validation and logging.

**[destructive] Cleanup and Forced Termination**
```bash
# Remove stopped containers (DANGEROUS if misapplied)
docker container prune -f

# Delete model files (DANGEROUS if path is incorrect)
rm -f /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf

# Truncate database (DANGEROUS, irreversible)
sqlite3 /models/benchmark.db "DELETE FROM benchmark_results;"
```
*Interpretation:* Destructive commands are strictly prohibited without human approval gates. They are listed for completeness and rollback reference only.

---

# Reasoning Budget Verification

The `--reasoning-budget 8192` flag controls the maximum token allocation for chain-of-thought or structured reasoning before final output generation. Combined with MTP (`--spec-type draft-mtp --spec-draft-n-max 2`), this creates a coupled inference pipeline where draft tokens and reasoning tokens share VRAM and context window resources. Misconfiguration leads to premature `finish_reason=length` or truncated `reasoning_content`.

**1. Budget Allocation Mechanics**
- The reasoning budget reserves tokens for internal deliberation, self-correction, and structured output formatting.
- MTP draft tokens (`--spec-draft-n-max 2`) are speculative and consume additional VRAM but do not count toward the reasoning budget unless explicitly configured.
- Total token consumption = `prompt_tokens + reasoning_tokens + draft_tokens + final_output_tokens`.
- The context window (262144) must accommodate all tokens plus KV cache overhead.

**2. Detection of Budget Exhaustion**
- Monitor `finish_reason` in proxy response logs. `length` before `reasoning_content` completion indicates budget overflow.
- Parse `reasoning_content` length vs. `8192` limit. If `reasoning_content` tokens approach 7500, final output generation is likely to be truncated.
- Check proxy access logs for `429` or `500` errors during reasoning phase.

**3. Dynamic `max_tokens` Adjustment**
- Increase `max_tokens` in benchmark payloads to allow final content to stream after `reasoning_content`.
- Formula: `max_tokens = reasoning_budget + estimated_final_output_tokens + safety_margin`.
- Typical safety margin: `+1024` tokens for formatting and post-processing.
- Validate by running a synthetic prompt with known output length.

**4. Verification Commands**
```bash
# [read-only] Extract reasoning token usage from proxy logs
grep -o '"reasoning_tokens":[0-9]*' /var/log/flight-recorder/proxy_access.log | awk -F: '{sum+=$2; count++} END {print "Avg reasoning tokens:", sum/count}'

# [low-risk] Test budget allocation with synthetic payload
curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced",
    "messages": [{"role": "user", "content": "Explain step-by-step how to validate a reasoning budget of 8192 tokens."}],
    "max_tokens": 10240,
    "reasoning_budget": 8192,
    "spec_type": "draft-mtp",
    "spec_draft_n_max": 2
  }' | jq '.usage, .choices[0].finish_reason'

# [medium-risk] Adjust container reasoning budget flag
docker exec llama-cpp bash -c "sed -i 's/--reasoning-budget 8192/--reasoning-budget 10240/' /etc/llama-cpp/config.env"
docker restart llama-cpp
```

**5. Validation Protocol**
- Run 3 synthetic prompts with increasing output complexity.
- Verify `reasoning_content` completes without truncation.
- Verify `final_output` appears after `reasoning_content` closes.
- Confirm `finish_reason` is `stop` or `length` only after full output.
- If truncation persists, reduce `--spec-draft-n-max` to 1 to free VRAM for reasoning tokens.

**6. Edge Cases**
- MTP draft model mismatch: If draft model architecture differs from base, reasoning budget may be misallocated. Verify draft model path in proxy config.
- Context window fragmentation: Long prompts + reasoning + MTP drafts can exceed 262144 effective tokens. Monitor KV cache utilization via `rocm-smi`.
- Proxy buffering: Flight Recorder may buffer reasoning content before forwarding. Increase proxy `buffer_size` if `finish_reason=length` appears prematurely.

---

# Long Output Validation

Long-output benchmark tests are prone to context overflow, GPU memory fragmentation, proxy timeouts, and harness race conditions. The agent enforces strict sequential validation to prevent cascading failures.

**1. Bulk Execution Prohibition**
- No more than 1 long-output task runs concurrently.
- Parallel slots are fixed at 1; attempting to scale triggers `[medium-risk]` configuration changes that require human approval.
- Bulk runs are replaced with sequential execution with explicit pause-and-validate steps.

**2. Task Validation Protocol**
- Before execution, estimate token count: `prompt_tokens + expected_output_tokens + reasoning_tokens`.
- Verify `max_tokens` exceeds estimate by `+2048`.
- Run dry-run with `echo` or mock payload to validate proxy routing and container response structure.
- Capture baseline VRAM and context utilization before task start.

**3. `finish_reason=length` Mitigation**
- If `finish_reason=length` appears before final content, increase `max_tokens` incrementally (`+1024`).
- Check `reasoning_content` completion status. If incomplete, reduce `--reasoning-budget` or disable MTP temporarily.
- Verify proxy `timeout` settings; increase `read_timeout` and `write_timeout` if network buffering causes premature closure.
- Validate output integrity: check for truncated JSON, missing closing tags, or incomplete sentences.

**4. Sequential Execution Workflow**
```bash
# [low-risk] Queue task for sequential execution
echo '{"task_id": "long_output_001", "payload": "...", "max_tokens": 12288, "priority": "high"}' > /tmp/benchmark_queue/task_001.json

# [read-only] Monitor queue processing
tail -f /var/log/benchmark-harness/queue_processor.log | grep "task_001"

# [low-risk] Validate output completeness
jq '.choices[0].message.content | length' /tmp/benchmark_results/task_001.json
```

**5. Screenshot and Reporting Standards**
- Capture full terminal output, proxy response headers, and ServerTop metrics.
- Redact internal IPs, model paths, and raw JSON payloads.
- Annotate screenshots with task ID, timestamp, `max_tokens`, and `finish_reason`.
- Store screenshots in `/evidence/screenshots/` with structured naming: `YYYYMMDD_HHMMSS_taskID_finishReason.png`.

**6. Validation Criteria**
- Output length matches expected range within `±5%`.
- `finish_reason` is `stop` or `length` only after full content.
- No proxy timeouts or 502 errors during streaming.
- VRAM usage remains below 85% of 32GB capacity.
- SQLite records task completion with accurate token counts.

---

# GPU And VRAM Checks

The AMD Radeon AI PRO R9700 with 32GB VRAM requires careful memory management for 262144 context windows, MTP drafts, and reasoning budgets. Vulkan backend introduces specific validation and fragmentation behaviors.

**1. VRAM Allocation Monitoring**
- Model weights: ~18-20GB for 35B parameter model (FP16/INT8 quantization dependent).
- KV cache: Scales with context length; 262144 tokens can consume 4-6GB depending on layer offloading.
- MTP drafts: `--spec-draft-n-max 2` adds speculative KV cache overhead (~1-2GB).
- Reasoning budget: Internal token buffers consume additional VRAM (~0.5-1GB).
- Total safe utilization: <28GB to leave headroom for Vulkan validation layers and system overhead.

**2. Diagnostic Commands**
```bash
# [read-only] Real-time VRAM allocation
rocm-smi --showmeminfo vram --json | jq '.gpu[0].vram_total, .gpu[0].vram_used'

# [read-only] Vulkan memory fragmentation check
vulkaninfo --summary | grep -A 10 "deviceMemory"

# [low-risk] Container GPU memory stats
docker stats llama-cpp --no-stream --format "{{.MemUsage}} {{.MemPerc}}"

# [medium-risk] Force Vulkan validation layers for error detection
export VK_LAYER_PATH=/opt/rocm/share/vulkan/explicit_layer.d
docker run --rm -e VK_LAYER_PATH -v /models:/models llama-cpp-server-vulkan:working-20260613 --help
```

**3. Context Window and KV Cache Management**
- Use `--num-a-layers` to offload layers to host RAM if VRAM is constrained.
- Enable `--cache-type-k fp16` and `--cache-type-v fp16` for efficient KV storage.
- Monitor context utilization via proxy logs: `context_utilization = (prompt_tokens + output_tokens) / 262144`.
- If utilization exceeds 90%, reduce context length or enable `--no-mmap` to prevent swap thrashing.

**4. Thermal and Power Monitoring**
```bash
# [read-only] GPU temperature and power draw
rocm-smi --showtemp gpu --showpower --json | jq '.gpu[0].temperature, .gpu[0].power'

# [low-risk] Set power limit to prevent thermal throttling
rocm-smi --setpoweroverdrive 1  # Requires root, medium-risk
```
*Interpretation:* Temperatures >85°C indicate cooling issues. Power draw >300W may trigger host power limits. Adjust fan curves or workload pacing if thresholds are breached.

**5. Vulkan Backend Troubleshooting**
- Validation layer errors: `VUID-vkCreateDevice-queueFamilyIndex-00853` indicates queue misconfiguration.
- Driver mismatch: ROCm 6.2+ required for R9700. Verify with `rocm-smi --showdriverversion`.
- Memory leak: Long benchmark runs may fragment Vulkan memory. Restart container daily or after 500+ tasks.
- Fix: Update ROCm, clear Vulkan cache (`rm -rf ~/.cache/vulkan`), and verify `--gpu-layers` matches VRAM capacity.

**6. Validation Protocol**
- Run `rocm-smi` before and after benchmark cycle.
- Verify VRAM usage <28GB, temperature <80°C, power <280W.
- Check Vulkan logs for validation errors.
- If OOM occurs, reduce `--spec-draft-n-max` to 1, lower `--reasoning-budget` to 4096, or increase `--num-a-layers`.

---

# Proxy And Database Checks

The Flight Recorder proxy (`http://192.168.1.116:8181/v1`) routes benchmark payloads to the llama.cpp container. The SQLite database at `/models` stores results, metadata, and audit trails. Both must remain local, private, and performant.

**1. Proxy Health and Routing**
- Verify proxy is listening on port 8181 and forwarding to container port 8080.
- Check rate limiting: `max_requests_per_minute`, `burst_size`, `timeout_ms`.
- Validate JSON schema compliance: proxy must reject malformed payloads before reaching container.
- Monitor queue depth: `active_requests`, `pending_requests`, `dropped_requests`.

**2. Database Integrity and Privacy**
- SQLite must operate in WAL mode for concurrent benchmark access.
- Regular checkpoints prevent WAL bloat: `PRAGMA wal_checkpoint(TRUNCATE);`
- Privacy constraint: Raw payloads, model weights, and internal paths must never be exported. Sanitize before archival.
- Backup strategy: Daily incremental backups to `/models/backups/`, weekly full backups to encrypted NAS.

**3. Diagnostic Commands**
```bash
# [read-only] Proxy configuration and routing table
curl -s http://192.168.1.116:8181/v1/config | jq '.routing, .rate_limit, .timeout'

# [read-only] SQLite journal mode and checkpoint status
sqlite3 /models/benchmark.db "PRAGMA journal_mode; PRAGMA wal_checkpoint;"

# [low-risk] Export sanitized metrics (no raw payloads)
sqlite3 /models/benchmark.db "SELECT task_id, model, tokens_generated, finish_reason, timestamp FROM benchmark_results WHERE timestamp > datetime('now', '-24 hours');" > /tmp/sanitized_results_$(date +%Y%m%d).csv

# [medium-risk] Restart proxy service
systemctl restart flight-recorder-proxy
```

**4. Latency and Throughput Validation**
- Measure proxy-to-container latency: `curl -w "%{time_connect} %{time_starttransfer} %{time_total}\n" http://192.168.1.116:8181/v1/health`
- Acceptable thresholds: connect <50ms, starttransfer <200ms, total <500ms.
- Throughput: Requests per second should remain stable under load. Drops indicate proxy bottleneck or container saturation.
- Fix: Increase proxy `worker_threads`, adjust `keepalive_timeout`, or scale container replicas (requires `[medium-risk]` config change).

**5. Error Handling and Retry Logic**
- 502 Bad Gateway: Container unresponsive. Check `docker logs llama-cpp`.
- 504 Gateway Timeout: Proxy timeout too short. Increase `read_timeout` and `write_timeout`.
- 429 Too Many Requests: Rate limit exceeded. Adjust `burst_size` or queue tasks sequentially.
- 400 Bad Request: Malformed JSON. Validate payload schema before submission.

**6. Validation Protocol**
- Proxy health returns 200/204.
- SQLite integrity check returns `ok`.
- Latency <500ms, throughput stable.
- No raw private data in exported artifacts.
- Backup exists and matches database size.

---

# Dashboard Checks

ServerTop (`http://192.168.1.116:8090`) provides real-time metrics, benchmark reporting, and alerting. The agent must verify data freshness, threshold configuration, and privacy filtering before relying on dashboard outputs.

**1. Metrics Freshness and Accuracy**
- Verify dashboard refresh interval matches benchmark cycle (e.g., 5s polling).
- Cross-check ServerTop VRAM/CPU metrics with `rocm-smi` and `docker stats`.
- Acceptable drift: <5% deviation. Higher drift indicates metric collection lag or container resource limits.

**2. Alert Threshold Configuration**
- VRAM usage >85%: Warning. >95%: Critical.
- Proxy latency >500ms: Warning. >1000ms: Critical.
- SQLite WAL size >500MB: Warning. >1GB: Critical.
- Benchmark failure rate >10%: Warning. >25%: Critical.
- Adjust thresholds via ServerTop UI or config file. Document all changes.

**3. Data Export and Privacy Filtering**
- Export must exclude raw payloads, model weights, and internal paths.
- Use regex sanitization: `s/192\.168\.1\.\d+/<<HOST_IP>>/g`, `s/\/models\/[^ ]+/<<MODEL_PATH>>/g`.
- Validate exported JSON schema before archival.
- Store exports in `/evidence/reports/` with encryption.

**4. Diagnostic Commands**
```bash
# [read-only] Fetch raw metrics from ServerTop API
curl -s http://192.168.1.116:8090/api/metrics | jq '.cpu, .vram, .proxy_latency, .benchmark_throughput'

# [low-risk] Trigger dashboard cache refresh
curl -s -X POST http://192.168.1.116:8090/api/admin/cache/flush

# [read-only] Verify alert configuration
curl -s http://192.168.1.116:8090/api/config/alerts | jq '.thresholds'
```

**5. Validation Protocol**
- Dashboard loads within 2s.
- Metrics update within 10s of benchmark cycle.
- Alerts trigger correctly at configured thresholds.
- Exported data passes privacy sanitization.
- No unredacted private data in logs or exports.

---

# Failure Triage Trees

Structured decision trees for common infrastructure failures. Each tree follows: Symptom -> Immediate Check -> Diagnostic Command -> Fix -> Validation -> Rollback Trigger.

**Tree 1: `finish_reason=length` Before Final Content**
- Symptom: Benchmark returns `finish_reason: "length"` with incomplete `reasoning_content` or missing final output.
- Immediate Check: Verify `max_tokens` vs. estimated output length. Check proxy timeout settings.
- Diagnostic: `grep -o '"finish_reason":"length"' /var/log/benchmark-harness/results.json | wc -l`
- Fix: Increase `max_tokens` by `+2048`. Reduce `--reasoning-budget` to 4096. Disable MTP temporarily (`--spec-type none`).
- Validation: Run synthetic prompt. Verify `finish_reason` is `stop` after full output.
- Rollback Trigger: If truncation persists after 3 adjustments, revert to previous container image and model version.

**Tree 2: VRAM OOM / Vulkan Crash**
- Symptom: Container exits with `CUDA/ROCm out of memory` or Vulkan validation error.
- Immediate Check: `rocm-smi --showmeminfo vram`. Check `docker logs llama-cpp` for OOM traces.
- Diagnostic: `docker inspect llama-cpp --format '{{.HostConfig.Memory}}'`
- Fix: Reduce `--spec-draft-n-max` to 1. Lower `--reasoning-budget` to 4096. Increase `--num-a-layers` to offload to host RAM.
- Validation: Monitor VRAM usage <28GB. Verify container stays up for 10+ tasks.
- Rollback Trigger: If OOM recurs, revert model to smaller quantization or reduce context window to 131072.

**Tree 3: Proxy Timeout / 502 Errors**
- Symptom: Benchmark harness receives 502/504 from `http://192.168.1.116:8181/v1`.
- Immediate Check: `curl -w "%{http_code}" http://192.168.1.116:8181/v1/health`. Check proxy logs.
- Diagnostic: `journalctl -u flight-recorder-proxy --since "1 hour ago" | grep -i "timeout\|502"`
- Fix: Increase proxy `read_timeout` to 30s. Adjust `worker_threads` to 8. Restart proxy.
- Validation: Verify 200/204 health response. Confirm benchmark tasks complete without 502.
- Rollback Trigger: If timeouts persist, revert proxy config to previous stable version.

**Tree 4: SQLite Corruption / Lock**
- Symptom: `database is locked` or `integrity_check` fails.
- Immediate Check: `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
- Diagnostic: `ls -lh /models/benchmark.db-wal`
- Fix: Run `PRAGMA wal_checkpoint(TRUNCATE);`. Restore from backup if corruption detected.
- Validation: Verify `integrity_check` returns `ok`. Confirm concurrent writes succeed.
- Rollback Trigger: If corruption recurs, switch to PostgreSQL or enable strict read-only mode for benchmark writes.

**Tree 5: MTP Sync Failure / Draft Model Mismatch**
- Symptom: `spec_type` errors, draft tokens not generated, or `finish_reason` inconsistent.
- Immediate Check: Verify draft model path in proxy config. Check `--spec-draft-n-max` value.
- Diagnostic: `curl -s http://192.168.1.116:8181/v1/config | jq '.spec_type, .draft_model_path'`
- Fix: Align draft model architecture with base model. Restart container. Clear MTP cache.
- Validation: Verify draft tokens appear in proxy logs. Confirm `finish_reason` stability.
- Rollback Trigger: If sync fails, disable MTP (`--spec-type none`) and proceed with base model only.

---

# Rollback Procedures

Rollbacks are prioritized over hotfixes. All procedures include backup, execution, validation, and verification steps. Risk classification applies to each action.

**1. Container Image Rollback**
- Backup current config: `docker inspect llama-cpp > /tmp/llama-cpp_config_$(date +%Y%m%d_%H%M%S).json`
- Pull previous image: `docker pull llama-cpp-server-vulkan:working-20260612`
- Stop current container: `docker stop llama-cpp`
- Remove and recreate with previous image: `docker rm llama-cpp && docker run --name llama-cpp [previous_flags]`
- Validate: `docker ps`, `curl http://192.168.1.116:8181/v1/health`
- Risk: `[medium-risk]`. Requires human approval if benchmark cycle is active.

**2. Model Version Rollback**
- Backup current model: `cp /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf /models/backups/`
- Restore previous model: `cp /models/backups/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced_v1.gguf /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
- Verify checksum: `sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
- Restart container: `docker restart llama-cpp`
- Validate: Check model loading logs, verify architecture matches.
- Risk: `[medium-risk]`. Requires model manifest verification.

**3. Proxy Configuration Rollback**
- Backup current config: `cp /etc/flight-recorder/proxy.conf /etc/flight-recorder/proxy.conf.bak`
- Restore previous config: `cp /etc/flight-recorder/proxy.conf.prev /etc/flight-recorder/proxy.conf`
- Reload service: `systemctl reload flight-recorder-proxy`
- Validate: `curl -s http://192.168.1.116:8181/v1/config | jq '.'`
- Risk: `[medium-risk]`. May drop active connections.

**4. Database Rollback**
- Stop benchmark writes: `sqlite3 /models/benchmark.db "PRAGMA wal_checkpoint(PASSIVE);"`
- Restore from backup: `sqlite3 /models/benchmark.db ".restore /models/backups/benchmark.db.bak"`
- Verify integrity: `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
- Resume writes: Update benchmark harness config to point to restored DB.
- Risk: `[medium-risk]`. Data loss for tasks executed after backup timestamp.

**5. Verification Post-Rollback**
- Run 3 synthetic benchmark tasks.
- Verify `finish_reason`, token counts, and proxy latency.
- Check ServerTop metrics for stability.
- Document rollback in `/evidence/rollbacks/` with timestamp and approval reference.

---

# Human Approval Gates

Automated actions are restricted to `[read-only]` and `[low-risk]` operations. `[medium-risk]` and `[destructive]` actions require explicit human approval. Gates are enforced at predefined decision points.

**1. Gate 1: Pre-Benchmark Execution**
- Trigger: Initiating long-output benchmark cycle.
- Required Evidence: Preflight check results, baseline metrics, model checksum, proxy health.
- Approval Method: Email/Slack ticket with attached logs and screenshots.
- Response Window: 15 minutes. Auto-approve if no response and all checks pass.

**2. Gate 2: Post-Triage Decision**
- Trigger: Failure triage tree recommends `[medium-risk]` fix.
- Required Evidence: Diagnostic command output, root cause analysis, fix steps, rollback plan.
- Approval Method: Structured form with risk assessment and validation criteria.
- Response Window: 10 minutes. Auto-approve if risk is `[low-risk]` and rollback is documented.

**3. Gate 3: Pre-Rollback Execution**
- Trigger: Rollback procedure initiated.
- Required Evidence: Current state snapshot, rollback steps, expected post-rollback state, verification plan.
- Approval Method: Explicit sign-off with timestamp.
- Response Window: 5 minutes. Auto-approve if rollback is container/image only and volumes are preserved.

**4. Gate 4: Post-Validation Confirmation**
- Trigger: Fix or rollback completed, validation pending.
- Required Evidence: Validation command output, metrics comparison, error log review.
- Approval Method: Confirmation that benchmark integrity is restored.
- Response Window: 10 minutes. Auto-approve if all validation criteria pass.

**5. Escalation Matrix**
- Level 1: Agent auto-approves `[low-risk]` actions.
- Level 2: Human reviewer approves `[medium-risk]` actions.
- Level 3: Senior admin approves `[destructive]` actions or multi-system rollbacks.
- All approvals logged in `/evidence/approvals/` with UUID and timestamp.

---

# Evidence To Capture

Comprehensive evidence collection ensures reproducibility, auditability, and privacy compliance. All artifacts are stored locally in encrypted directories.

**1. Screenshot Standards**
- Capture full window, URL, timestamp, and task ID.
- Redact internal IPs, model paths, raw JSON payloads, and user prompts.
- Use `scrot -d 2 -e 'mv $f /evidence/screenshots/'` or equivalent.
- Annotate with `finish_reason`, `max_tokens`, `reasoning_budget`, and VRAM usage.
- Store in `/evidence/screenshots/YYYYMMDD_HHMMSS_taskID.png`.

**2. Log Collection**
- Container logs: `docker logs --since 2h llama-cpp > /tmp/logs_llama-cpp.log`
- Proxy logs: `journalctl -u flight-recorder-proxy --since "2 hours ago" > /tmp/logs_proxy.log`
- Benchmark harness logs: `tail -n 1000 /var/log/benchmark-harness/runner.log > /tmp/logs_harness.log`
- Sanitize logs: `sed -i 's/192\.168\.1\.\d*/<<HOST_IP>>/g; s/\/models\/[^ ]*/<<MODEL_PATH>>/g' /tmp/logs_*.log`
- Archive: `tar -czf /evidence/logs/logs_$(date +%Y%m%d_%H%M%S).tar.gz /tmp/logs_*.log`

**3. JSON Artifacts**
- Benchmark payloads: `curl -s http://192.168.1.116:8181/v1/chat/completions -d '{"model":"..."}' > /tmp/payload.json`
- Responses: `jq '.choices[0].message.content' /tmp/response.json > /tmp/content.json`
- Error traces: `jq '.error' /tmp/response.json > /tmp/error.json`
- Sanitize: Remove raw tokens, internal paths, and user prompts. Keep only metrics and structure.
- Store in `/evidence/artifacts/` with structured naming.

**4. Privacy Sanitization Protocol**
- Regex patterns: `s/192\.168\.1\.\d+/<<HOST_IP>>/g`, `s/\/models\/[^ ]+/<<MODEL_PATH>>/g`, `s/"prompt":"[^"]*"/"prompt":"<<REDACTED>>"/g`
- Validate sanitized output before archival.
- Never transmit raw artifacts outside LAN.
- Encrypt archives: `gpg --symmetric --cipher-algo AES256 /evidence/archive.tar.gz`

**5. Retention and Access**
- Retention period: 90 days for logs, 365 days for artifacts.
- Access restricted to admin group with encrypted storage.
- Audit trail: `ls -la /evidence/` logged weekly.

---

# Final Go No-Go Checklist

Comprehensive checklist to verify infrastructure readiness before benchmark execution. All items must pass before proceeding.

**System & Container**
- [ ] Container image: `llama-cpp-server-vulkan:working-20260613` verified
- [ ] Container status: `Up` with correct resource limits
- [ ] Volume mounts: `/models` bind-mounted read-write
- [ ] Network: Ports 8181 and 8090 accessible, no firewall blocks

**Model & Inference**
- [ ] Model file: `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` exists
- [ ] SHA256 checksum: Matches trusted manifest
- [ ] GGUF header: Valid architecture, context 262144, MTP compatible
- [ ] Reasoning budget: `--reasoning-budget 8192` configured
- [ ] MTP config: `--spec-type draft-mtp --spec-draft-n-max 2` active

**GPU & VRAM**
- [ ] VRAM usage: <28GB (85% of 32GB)
- [ ] Temperature: <80°C
- [ ] Power draw: <280W
- [ ] Vulkan validation: No critical errors in logs
- [ ] Context utilization: <90% effective tokens

**Proxy & Database**
- [ ] Proxy health: `http://192.168.1.116:8181/v1/health` returns 200/204
- [ ] Proxy latency: <500ms total
- [ ] SQLite integrity: `PRAGMA integrity_check` returns `ok`
- [ ] WAL mode: Enabled, checkpoint status healthy
- [ ] Backup: Recent backup exists in `/models/backups/`

**Dashboard & Reporting**
- [ ] ServerTop: `http://192.168.1.116:8090` loads within 2s
- [ ] Metrics: Fresh, accurate, drift <5%
- [ ] Alerts: Thresholds configured, no active critical alerts
- [ ] Export: Privacy sanitization verified, no raw private data

**Validation & Approval**
- [ ] Preflight checks: All passed, baseline metrics captured
- [ ] Synthetic test: 3 tasks completed with `finish_reason: stop`
- [ ] Human approval: Gate 1 signed off, timestamp recorded
- [ ] Rollback readiness: Config backups, model backups, DB backups verified
- [ ] Evidence directory: `/evidence/` structured, encrypted, accessible

**Final Sign-Off**
- [ ] All checkboxes marked
- [ ] Approval ticket reference attached
- [ ] Runbook state file updated
- [ ] Benchmark cycle initiated

Proceed only when every item is verified. Document any deviations in `/evidence/deviations/` and escalate per approval gates.