# Operating Principles

This runbook governs the diagnosis, remediation, validation, and rollback of benchmark infrastructure for a home-lab AI server running a llama.cpp container with AMD GPU acceleration, speculative decoding (MTP), and extended reasoning budgets. The following principles dictate every action taken during benchmark operations:

1. **Cautious Execution First**: No modification to running services, container configurations, or database schemas occurs without explicit risk classification and human approval. Read-only inspection precedes any intervention.
2. **Isolation & Reproducibility**: All benchmark runs are isolated within the container environment. Configuration changes are versioned, documented, and reversible. State is captured before and after any adjustment.
3. **Privacy Preservation**: The SQLite database resides on `/models` and must never leave the host. All evidence collection, screenshots, and reporting are confined to local storage. No raw benchmark payloads, model weights, or internal metrics are exported externally.
4. **Observation-Driven Tuning**: The observed `finish_reason=length` behavior before final content appears, and the subsequent fix via `max_tokens` adjustment, establishes a baseline: reasoning and final output generation are decoupled in the API response stream. Validation must account for this split.
5. **Sequential Long-Output Handling**: Bulk execution of long-output tasks is prohibited without per-task validation. Each task is dry-run, parsed, and verified before inclusion in a batch.
6. **Risk Classification Discipline**: Every shell command is explicitly tagged as `[read-only]`, `[low-risk]`, `[medium-risk]`, or `[destructive]`. Commands are never executed out of sequence or without understanding their blast radius.
7. **Evidence-First Troubleshooting**: Diagnostics require timestamped logs, structured JSON responses, GPU metrics, and dashboard screenshots. Conclusions are drawn from captured evidence, not assumptions.
8. **Rollback Readiness**: Every configuration change, container restart, or database operation must have a documented rollback path. Rollback is triggered automatically on critical failures or manually upon human approval.
9. **Dashboard & Proxy Synchronization**: The Flight Recorder proxy (`:8181`) and ServerTop dashboard (`:8090`) are treated as primary observability layers. Discrepancies between container logs, proxy metrics, and dashboard visuals trigger immediate triage.
10. **Human-in-the-Loop Gates**: Destructive operations, schema changes, driver updates, and benchmark result anomalies require explicit human sign-off before proceeding.

These principles form the operational backbone. All subsequent sections translate them into actionable procedures, commands, and decision trees tailored to the specified environment.

# Preflight Checks

Before initiating any benchmark run, the infrastructure must be verified for stability, resource availability, and configuration correctness. Preflight checks ensure that transient issues do not masquerade as model or algorithmic failures.

## System & Host Verification
Verify host-level readiness before interacting with the container or GPU.

- **Disk Space & Mounts**: Ensure `/models` has sufficient space for the SQLite database, model weights, and benchmark artifacts.
  ```bash
  df -h /models [read-only]
  mount | grep /models [read-only]
  ```
- **Network Connectivity**: Confirm host can reach internal services and that container networking is intact.
  ```bash
  ping -c 3 192.168.1.116 [read-only]
  ss -tlnp | grep -E '8181|8090' [read-only]
  ```
- **User Permissions**: Verify the admin user has appropriate access to Docker, ROCm, and SQLite.
  ```bash
  groups [read-only]
  sudo -n docker ps [low-risk]
  ```

## Container & Runtime Verification
Validate the llama.cpp container state and configuration.

- **Container Status**: Ensure the container is running, healthy, and using the correct image tag.
  ```bash
  docker inspect --format='{{.State.Status}} {{.Config.Image}}' llama-cpp-server [read-only]
  docker logs --tail 50 llama-cpp-server [read-only]
  ```
- **CLI Arguments Audit**: Verify MTP, reasoning budget, context window, and parallel slots match specifications.
  ```bash
  docker inspect --format='{{.Config.Cmd}}' llama-cpp-server [read-only]
  ```
- **Environment Variables**: Check for any overrides or secret injections.
  ```bash
  docker inspect --format='{{json .Config.Env}}' llama-cpp-server [read-only]
  ```

## Model & Artifact Verification
Confirm the model file is intact and accessible.

- **File Integrity**: Verify size, permissions, and checksum if available.
  ```bash
  ls -lh /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf [read-only]
  sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf [read-only]
  ```
- **Metadata Extraction**: Use llama.cpp CLI to verify context length, architecture, and MTP compatibility.
  ```bash
  docker run --rm --network host llama-cpp-server-vulkan:working-20260613 \
    llama-cli --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
    --list-models [read-only]
  ```

## Benchmark Configuration Validation
Ensure benchmark harness settings align with container capabilities.

- **Context Window**: Verify `--ctx-size 262144` is accepted without warnings.
- **Parallel Slots**: Confirm `--parallel 1` is enforced to prevent slot contention during MTP.
- **MTP Parameters**: Validate `--spec-type draft-mtp --spec-draft-n-max 2` are present.
- **Reasoning Budget**: Confirm `--reasoning-budget 8192` is active.

If any preflight check fails, halt the benchmark sequence. Document the failure, apply the appropriate fix from the triage trees, and re-run preflight checks before proceeding.

# Safe Inspection Commands

These commands are designed for runtime diagnostics without altering state. They are classified as `[read-only]` or `[low-risk]` and should be executed during active benchmark runs or immediately after failures.

## Container & Process Inspection
- **Live Logs**: Stream container output for real-time error tracking.
  ```bash
  docker logs --tail 100 --follow llama-cpp-server [read-only]
  ```
- **Process Status**: Verify the llama.cpp process is alive and responsive.
  ```bash
  docker exec llama-cpp-server ps aux | grep llama [read-only]
  ```
- **Resource Limits**: Check CPU/memory constraints applied to the container.
  ```bash
  docker stats --no-stream llama-cpp-server [read-only]
  ```

## API & Proxy Inspection
- **Health Endpoint**: Verify the Flight Recorder proxy is routing correctly.
  ```bash
  curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1/health [read-only]
  ```
- **Model Endpoint**: Confirm the model is loaded and accessible via proxy.
  ```bash
  curl -s http://192.168.1.116:8181/v1/models | jq '.data[0].id' [read-only]
  ```
- **Token Counting**: Send a minimal prompt to verify tokenization and budget tracking.
  ```bash
  curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"test"}],"max_tokens":10,"stream":false}' | jq '.' [read-only]
  ```

## Database Inspection
- **Schema & Tables**: Verify SQLite structure without modifying data.
  ```bash
  sqlite3 /models/benchmark.db ".tables" [read-only]
  sqlite3 /models/benchmark.db "PRAGMA table_info(benchmark_runs);" [read-only]
  ```
- **Integrity Check**: Ensure no corruption exists.
  ```bash
  sqlite3 /models/benchmark.db "PRAGMA integrity_check;" [read-only]
  ```
- **Recent Entries**: Query last 5 benchmark runs for validation.
  ```bash
  sqlite3 /models/benchmark.db "SELECT run_id, status, created_at FROM benchmark_runs ORDER BY created_at DESC LIMIT 5;" [read-only]
  ```

## GPU & Vulkan Inspection
- **ROCm Device Status**: Verify AMD GPU is recognized and accessible.
  ```bash
  rocm-smi --showallinfo [read-only]
  ```
- **Vulkan Capabilities**: Confirm Vulkan driver supports the required features.
  ```bash
  vulkaninfo --summary [read-only]
  ```
- **Container GPU Passthrough**: Verify GPU devices are mounted.
  ```bash
  docker exec llama-cpp-server ls -l /dev/dri [read-only]
  ```

These commands form the diagnostic baseline. Execute them in order during preflight, mid-run monitoring, and post-failure analysis. Never combine inspection commands with state-modifying operations in the same session without explicit approval.

# Reasoning Budget Verification

The `--reasoning-budget 8192` parameter controls the token allocation for internal reasoning steps before final content generation. The observed behavior where `finish_reason=length` appears before final content indicates the budget or `max_tokens` ceiling is being hit prematurely during the reasoning phase.

## Budget Mechanics & API Behavior
- llama.cpp splits generation into `reasoning_content` (internal chain-of-thought) and `content` (final response).
- The reasoning budget caps tokens consumed during the reasoning phase.
- If `max_tokens` is too low, the API may return `finish_reason=length` after reasoning completes but before final content is fully streamed.
- Increasing `max_tokens` resolves this by allowing the final content to exceed the initial ceiling.

## Verification Procedure
1. **Send Controlled Prompt**: Use a prompt known to trigger reasoning without requiring long final output.
   ```bash
   curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"Explain step-by-step how to calculate the derivative of x^2 + 3x."}],"max_tokens":500,"stream":false,"reasoning_budget":8192}' | jq '.' [read-only]
   ```
2. **Parse Response**: Extract `reasoning_content` length and `finish_reason`.
   ```bash
   curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"test reasoning budget"}],"max_tokens":500,"stream":false}' | \
     jq '{reasoning_tokens: (.choices[0].message.reasoning_content | length), final_tokens: (.choices[0].message.content | length), finish_reason: .choices[0].finish_reason}' [read-only]
   ```
3. **Compare Against Budget**: Verify `reasoning_tokens` < 8192. If it approaches or exceeds the budget, the API may truncate prematurely.
4. **Adjust `max_tokens`**: If `finish_reason=length` appears after reasoning, increase `max_tokens` to at least `reasoning_tokens + expected_final_tokens + buffer`.
   ```bash
   # Example adjustment in container args (requires restart)
   docker exec llama-cpp-server cat /etc/llama-cpp/args.conf [read-only]
   ```

## Validation Matrix
| Scenario | `reasoning_tokens` | `max_tokens` | `finish_reason` | Action |
|----------|-------------------|--------------|-----------------|--------|
| Normal | < 8192 | 1024 | `stop` | No action |
| Premature truncation | < 8192 | 512 | `length` | Increase `max_tokens` |
| Budget overflow | ≥ 8192 | 2048 | `length` | Reduce reasoning complexity or increase budget |
| MTP interference | Variable | 1024 | `length` | Verify `--spec-draft-n-max 2` alignment |

## Remediation Steps
- If `finish_reason=length` consistently appears before final content, update the container's `max_tokens` default via environment variable or CLI override.
- Restart container with adjusted parameters:
  ```bash
  docker stop llama-cpp-server [medium-risk]
  docker rm llama-cpp-server [medium-risk]
  docker run -d --name llama-cpp-server --gpus all --network host \
    -v /models:/models \
    -e MAX_TOKENS=4096 \
    llama-cpp-server-vulkan:working-20260613 \
    --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
    --ctx-size 262144 --parallel 1 \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --reasoning-budget 8192 [medium-risk]
  ```
- Re-run verification matrix. Document changes in benchmark configuration log.

# Long Output Validation

Long-output tests must never be executed in bulk without per-task validation. The benchmark harness must enforce sequential dry-runs, parse responses, verify completeness, and only then schedule batches.

## Validation Pipeline
1. **Task Isolation**: Extract each long-output task into a separate JSON payload.
2. **Dry-Run Execution**: Send with `max_tokens` set to 2x expected output length.
3. **Response Parsing**: Check for `finish_reason`, content length, and structural integrity.
4. **Completeness Check**: Verify no truncation markers, incomplete JSON, or cut-off sentences.
5. **Batch Scheduling**: Only tasks passing validation are queued.

## Dry-Run Command Template
```bash
curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"TASK_PLACEHOLDER"}],"max_tokens":8192,"stream":false}' | \
  jq '{finish_reason, content_length: (.choices[0].message.content | length), truncated: (.choices[0].finish_reason == "length")}' [read-only]
```

## Buffer & Context Management
- **Context Window Saturation**: With `--ctx-size 262144`, long outputs consume significant KV cache. Monitor context utilization via dashboard.
- **MTP Overhead**: `--spec-draft-n-max 2` increases draft token generation. Long outputs may trigger additional speculative steps. Validate draft acceptance rate.
- **Streaming vs Non-Streaming**: Use non-streaming for validation to ensure complete payloads. Switch to streaming only for production runs.

## Bulk Execution Safeguards
- **Sequential Queuing**: Enforce `parallel 1` during validation phase.
- **Timeout Enforcement**: Set request timeout to 120s. Abort if exceeded.
- **Retry Logic**: Max 2 retries per task. Log failures.
- **Artifact Isolation**: Store dry-run outputs in `/models/benchmarks/dryrun/` with timestamps.

## Failure Handling for Long Outputs
- If `finish_reason=length` appears, increase `max_tokens` incrementally (e.g., +1024) until `stop` is returned.
- If context overflow occurs, reduce input prompt length or enable KV cache quantization.
- If MTP causes instability, temporarily disable `--spec-type draft-mtp` for validation.

# GPU And VRAM Checks

The AMD Radeon AI PRO R9700 with 32GB VRAM runs Vulkan backend via llama.cpp. VRAM fragmentation, layer offloading, and MTP draft models consume significant memory. Continuous monitoring is required.

## Hardware & Driver Verification
- **ROCm Version**: Confirm compatibility with Vulkan backend.
  ```bash
  rocm-smi --showdriverversion [read-only]
  ```
- **Vulkan Driver**: Verify AMDVLK or RADV is active.
  ```bash
  vulkaninfo | grep -i "driver" [read-only]
  ```
- **GPU Passthrough**: Ensure container has access to `/dev/dri` and ROCm devices.
  ```bash
  docker exec llama-cpp-server ls -l /dev/dri/renderD128 [read-only]
  ```

## VRAM Monitoring
- **Real-Time Usage**: Watch VRAM allocation during benchmark runs.
  ```bash
  watch -n 1 rocm-smi --showmeminfo vram [read-only]
  ```
- **Container VRAM Limits**: Verify no artificial caps are applied.
  ```bash
  docker inspect --format='{{.HostConfig.GpuCount}}' llama-cpp-server [read-only]
  ```
- **Fragmentation Check**: High fragmentation causes OOM despite available VRAM.
  ```bash
  rocm-smi --showmeminfo vram --json [read-only]
  ```

## Layer Offloading & MTP Impact
- **GPU Layers**: llama.cpp offloads layers to VRAM. Verify layer count matches model size.
  ```bash
  docker logs llama-cpp-server 2>&1 | grep -i "gpu layers" [read-only]
  ```
- **MTP Draft Model**: `--spec-draft-n-max 2` loads a smaller draft model. Verify it fits in VRAM alongside base model.
  ```bash
  docker logs llama-cpp-server 2>&1 | grep -i "speculative" [read-only]
  ```
- **Context KV Cache**: 262144 context consumes ~4-6GB VRAM depending on quantization. Monitor usage.

## Troubleshooting VRAM Issues
- **OOM Errors**: Reduce `--n-gpu-layers`, disable MTP temporarily, or lower context size.
  ```bash
  docker stop llama-cpp-server [medium-risk]
  docker run -d --name llama-cpp-server --gpus all --network host \
    -v /models:/models \
    llama-cpp-server-vulkan:working-20260613 \
    --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
    --ctx-size 131072 --parallel 1 --n-gpu-layers 95 \
    --reasoning-budget 8192 [medium-risk]
  ```
- **Vulkan Crashes**: Update AMDVLK driver, verify ROCm compatibility, check dmesg.
  ```bash
  dmesg | grep -i amdgpu | tail -20 [read-only]
  ```
- **VRAM Fragmentation**: Restart container to clear allocations. Schedule periodic restarts during long benchmarks.

# Proxy And Database Checks

The Flight Recorder proxy (`:8181`) routes benchmark requests to the llama.cpp container. The SQLite database on `/models` stores benchmark metadata, results, and configuration. Both must remain local, private, and consistent.

## Proxy Health & Routing
- **Endpoint Verification**:
  ```bash
  curl -s http://192.168.1.116:8181/v1/health | jq '.' [read-only]
  curl -s http://192.168.1.116:8181/v1/models | jq '.data[0].id' [read-only]
  ```
- **Rate Limiting & CORS**: Ensure proxy isn't throttling benchmark traffic.
  ```bash
  curl -s -I http://192.168.1.116:8181/v1/chat/completions [read-only]
  ```
- **Log Inspection**: Check proxy logs for routing errors or timeouts.
  ```bash
  docker logs --tail 100 flight-recorder-proxy [read-only]
  ```

## SQLite Database Integrity
- **Schema Validation**:
  ```bash
  sqlite3 /models/benchmark.db ".schema" [read-only]
  ```
- **WAL Mode Verification**: Ensure Write-Ahead Logging is enabled for concurrent access.
  ```bash
  sqlite3 /models/benchmark.db "PRAGMA journal_mode;" [read-only]
  ```
- **Lock Contention**: Check for database locks during benchmark runs.
  ```bash
  sqlite3 /models/benchmark.db "PRAGMA busy_timeout = 5000;" [low-risk]
  ```
- **Backup Procedure**: Create local backup before any schema changes.
  ```bash
  cp /models/benchmark.db /models/benchmark.db.bak.$(date +%Y%m%d%H%M%S) [low-risk]
  ```

## Privacy & Local-Only Enforcement
- **Network Binding**: Verify proxy binds to `127.0.0.1` or `192.168.1.116` only.
  ```bash
  ss -tlnp | grep 8181 [read-only]
  ```
- **No External Sync**: Ensure no cron jobs or scripts push data externally.
  ```bash
  crontab -l [read-only]
  ls -la /etc/cron.d/ [read-only]
  ```
- **Data Masking**: When capturing evidence, strip model weights, internal prompts, and raw payloads. Hash sensitive fields.

## Troubleshooting Proxy/DB Issues
- **Timeouts**: Increase proxy timeout, verify container health.
- **DB Corruption**: Restore from backup, run `PRAGMA integrity_check`.
- **Lock Contention**: Implement retry logic in benchmark harness, use WAL mode.
- **Privacy Breach**: Audit network rules, revoke external access, rotate internal tokens.

# Dashboard Checks

ServerTop (`http://192.168.1.116:8090`) provides real-time metrics for GPU, VRAM, context usage, request latency, and error rates. Dashboard validation ensures observability aligns with container state.

## Dashboard Health & Metrics
- **Status Endpoint**:
  ```bash
  curl -s http://192.168.1.116:8090/api/status | jq '.' [read-only]
  ```
- **Metrics Export**: Verify Prometheus-compatible endpoints are active.
  ```bash
  curl -s http://192.168.1.116:8090/metrics | head -50 [read-only]
  ```
- **GPU Utilization**: Check real-time VRAM and compute usage.
  ```bash
  curl -s http://192.168.1.116:8090/api/gpu | jq '.vram_used, .vram_total' [read-only]
  ```

## Screenshot & Reporting Protocol
- **Automated Screenshots**: Capture dashboard at benchmark start, midpoint, and end.
  ```bash
  # Example using chromium-headless (local only)
  chromium-browser --headless --screenshot=/models/screenshots/bench_start.png \
    --window-size=1920,1080 http://192.168.1.116:8090 [low-risk]
  ```
- **Data Export**: Export metrics to local CSV/JSON. Never push externally.
  ```bash
  curl -s http://192.168.1.116:8090/api/export?format=json > /models/reports/dashboard_$(date +%Y%m%d).json [low-risk]
  ```
- **Alert Thresholds**: Configure local alerts for VRAM > 90%, latency > 5s, error rate > 5%.

## Dashboard-Container Discrepancy Resolution
- **Metric Lag**: Refresh dashboard cache, verify container metrics endpoint.
- **VRAM Mismatch**: Cross-check `rocm-smi` with dashboard. If mismatch, restart metrics collector.
- **Missing Data**: Verify proxy routing, check container logs for metric export errors.
- **Historical Gaps**: Ensure SQLite backup includes metric snapshots. Restore if needed.

# Failure Triage Trees

Structured decision trees for rapid diagnosis and resolution of common benchmark failures. Each tree follows a strict sequence: observe, isolate, verify, remediate, validate.

## Tree 1: `finish_reason=length` Before Final Content
```
Start
 ├─ Check max_tokens setting
 │  ├─ < 1024 → Increase to 2048+ [medium-risk]
 │  └─ ≥ 1024 → Proceed
 ├─ Check reasoning_budget vs actual reasoning_tokens
 │  ├─ reasoning_tokens ≥ 8192 → Reduce prompt complexity or increase budget [medium-risk]
 │  └─ reasoning_tokens < 8192 → Proceed
 ├─ Check MTP draft acceptance rate
 │  ├─ < 50% → Disable --spec-type draft-mtp temporarily [medium-risk]
 │  └─ ≥ 50% → Proceed
 ├─ Verify max_tokens in container args
 │  ├─ Mismatch → Update env var MAX_TOKENS [medium-risk]
 │  └─ Match → Proceed
 └─ Restart container & re-run validation [medium-risk]
```

## Tree 2: VRAM OOM During Long Output
```
Start
 ├─ Check rocm-smi VRAM usage
 │  ├─ > 95% → Proceed
 │  └─ < 95% → Check for memory leak [low-risk]
 ├─ Reduce --n-gpu-layers by 10 [medium-risk]
 ├─ Lower --ctx-size to 131072 [medium-risk]
 ├─ Disable MTP temporarily [medium-risk]
 ├─ Restart container [medium-risk]
 └─ Re-run benchmark with reduced parameters [low-risk]
```

## Tree 3: Proxy Timeout / 504 Errors
```
Start
 ├─ Check proxy health endpoint [read-only]
 ├─ Verify container is responding to /v1/chat/completions [read-only]
 ├─ Increase proxy timeout to 120s [medium-risk]
 ├─ Check network latency between proxy and container [read-only]
 ├─ Restart proxy service [medium-risk]
 └─ Re-test with minimal payload [low-risk]
```

## Tree 4: SQLite Lock / Corruption
```
Start
 ├─ Run PRAGMA integrity_check [read-only]
 ├─ Check WAL mode [read-only]
 ├─ Restore from latest backup [medium-risk]
 ├─ Enable PRAGMA busy_timeout = 5000 [low-risk]
 ├─ Implement retry logic in harness [low-risk]
 └─ Monitor for recurrence [read-only]
```

## Tree 5: Vulkan Driver Crash / GPU Reset
```
Start
 ├─ Check dmesg for amdgpu errors [read-only]
 ├─ Verify ROCm version compatibility [read-only]
 ├─ Update AMDVLK driver [medium-risk]
 ├─ Clear GPU state: rocm-smi --setgpu 0 [medium-risk]
 ├─ Restart container [medium-risk]
 └─ Re-run preflight checks [read-only]
```

Each tree requires evidence capture before and after remediation. Document all changes in `/models/benchmarks/logs/triage_$(date +%Y%m%d).md`.

# Rollback Procedures

Rollback ensures infrastructure can be restored to a known-good state quickly and safely. All rollback steps are classified and sequenced to minimize downtime and data loss.

## Container Rollback
1. **Identify Previous Image/Tag**: Check Docker history or configuration backup.
   ```bash
   docker history llama-cpp-server-vulkan:working-20260613 [read-only]
   ```
2. **Stop Current Container**:
   ```bash
   docker stop llama-cpp-server [medium-risk]
   ```
3. **Remove Current Container**:
   ```bash
   docker rm llama-cpp-server [medium-risk]
   ```
4. **Run Previous Version**:
   ```bash
   docker run -d --name llama-cpp-server --gpus all --network host \
     -v /models:/models \
     llama-cpp-server-vulkan:previous-tag \
     --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
     --ctx-size 262144 --parallel 1 \
     --spec-type draft-mtp --spec-draft-n-max 2 \
     --reasoning-budget 8192 [medium-risk]
   ```
5. **Verify Rollback**:
   ```bash
   docker inspect --format='{{.Config.Image}}' llama-cpp-server [read-only]
   curl -s http://192.168.1.116:8181/v1/health [read-only]
   ```

## Configuration Rollback
- Revert CLI arguments, environment variables, and proxy settings to backup state.
- Restore `/models/benchmark.db` from backup if schema was altered.
- Update benchmark harness configuration file.

## Database Rollback
- **WAL Recovery**:
  ```bash
  sqlite3 /models/benchmark.db "PRAGMA wal_checkpoint(TRUNCATE);" [low-risk]
  ```
- **Full Restore**:
  ```bash
  cp /models/benchmark.db.bak.* /models/benchmark.db [medium-risk]
  sqlite3 /models/benchmark.db "PRAGMA integrity_check;" [read-only]
  ```
- **Schema Revert**: If migration was applied, run reverse migration script.

## Proxy Rollback
- Restart proxy service to previous config.
- Revert nginx/caddy rules if modified.
- Verify routing to container.

## Post-Rollback Validation
- Run preflight checks.
- Execute dry-run benchmark task.
- Verify dashboard metrics align.
- Document rollback in audit log.

# Human Approval Gates

Critical operations require explicit human sign-off. Automated systems may execute read-only and low-risk commands, but medium-risk and destructive actions are gated.

## Gate 1: Container Restart / Image Change
- **Trigger**: Any `docker stop`, `docker rm`, or image tag modification.
- **Approval Required**: Yes
- **Evidence Needed**: Current container state, rollback image tag, expected downtime.
- **Workflow**: Submit ticket → Attach evidence → Await sign-off → Execute → Verify.

## Gate 2: Database Schema Modification
- **Trigger**: `ALTER TABLE`, `CREATE INDEX`, or migration scripts.
- **Approval Required**: Yes
- **Evidence Needed**: Schema diff, backup path, rollback script.
- **Workflow**: Backup DB → Submit diff → Await sign-off → Apply → Verify integrity.

## Gate 3: GPU Driver / ROCm Update
- **Trigger**: `apt update`, `rocm-smi` version change, Vulkan driver update.
- **Approval Required**: Yes
- **Evidence Needed**: Current version, target version, compatibility matrix, rollback plan.
- **Workflow**: Document version → Submit compatibility check → Await sign-off → Update → Reboot if needed → Verify passthrough.

## Gate 4: Benchmark Result Anomaly
- **Trigger**: `finish_reason` mismatch, VRAM spike > 95%, proxy timeout > 10s, DB corruption.
- **Approval Required**: Yes
- **Evidence Needed**: Logs, screenshots, metric exports, triage tree output.
- **Workflow**: Capture evidence → Run triage → Submit findings → Await sign-off → Apply fix or rollback.

## Gate 5: Configuration Override
- **Trigger**: Changing `--ctx-size`, `--reasoning-budget`, `--spec-draft-n-max`, or `max_tokens`.
- **Approval Required**: Yes
- **Evidence Needed**: Current vs proposed values, expected impact, validation plan.
- **Workflow**: Document change → Submit impact analysis → Await sign-off → Apply → Re-run preflight.

All approvals are logged in `/models/benchmarks/logs/approvals/` with timestamps, approver ID, and action taken. No gate is bypassed without explicit override authorization.

# Evidence To Capture

Comprehensive evidence collection ensures reproducibility, auditability, and privacy compliance. All artifacts are stored locally and never exported.

## Log Files
- **Container Logs**: `docker logs --tail 500 llama-cpp-server > /models/logs/container_$(date +%Y%m%d).log`
- **Proxy Logs**: `docker logs --tail 500 flight-recorder-proxy > /models/logs/proxy_$(date +%Y%m%d).log`
- **System Logs**: `journalctl -u docker --no-pager > /models/logs/system_$(date +%Y%m%d).log`

## API Responses
- **JSON Payloads**: Save all benchmark requests and responses to `/models/benchmarks/responses/`.
- **Masking**: Strip raw model weights, internal prompts, and PII. Hash sensitive fields.
- **Structure**:
  ```json
  {
    "run_id": "bench_20260613_001",
    "timestamp": "2026-06-13T10:00:00Z",
    "request": {"model": "...", "messages": [{"role": "user", "content": "[MASKED]"}]},
    "response": {"choices": [{"finish_reason": "stop", "content": "[MASKED]"}]},
    "metrics": {"reasoning_tokens": 1200, "final_tokens": 800, "latency_ms": 4500}
  }
  ```

## Screenshots & Dashboard Exports
- **Dashboard Screenshots**: Capture at start, midpoint, end.
- **Metrics Export**: JSON/CSV from ServerTop.
- **Storage**: `/models/screenshots/` and `/models/reports/`.

## GPU & VRAM Metrics
- **rocm-smi Snapshots**: `rocm-smi --showmeminfo vram --json > /models/metrics/vram_$(date +%Y%m%d).json`
- **Container Stats**: `docker stats --no-stream llama-cpp-server > /models/metrics/container_$(date +%Y%m%d).json`

## Database State
- **Schema Dump**: `sqlite3 /models/benchmark.db ".dump" > /models/db/schema_$(date +%Y%m%d).sql`
- **Integrity Check**: `sqlite3 /models/benchmark.db "PRAGMA integrity_check;" > /models/db/integrity_$(date +%Y%m%d).txt`

## Retention & Privacy
- **Retention**: 30 days for logs, 90 days for reports, 1 year for approvals.
- **Encryption**: Local LUKS or GPG encryption for sensitive artifacts.
- **No External Sync**: Verify no cron jobs, rsync, or cloud sync tools are active.
- **Audit Trail**: All evidence collection logged in `/models/benchmarks/logs/evidence_manifest.json`.

# Final Go No-Go Checklist

Before initiating any benchmark run, verify all checklist items. Only proceed when every item is marked `PASS`.

## Pre-Benchmark Checklist
- [ ] Preflight checks completed and documented
- [ ] Container running with correct image tag and CLI args
- [ ] Model file verified (size, checksum, metadata)
- [ ] Reasoning budget verified (`reasoning_tokens < 8192`)
- [ ] `max_tokens` adjusted to prevent premature `finish_reason=length`
- [ ] Long-output tasks validated sequentially (dry-run passed)
- [ ] GPU VRAM usage < 85% (rocm-smi verified)
- [ ] Vulkan driver and ROCm version compatible
- [ ] Proxy health endpoint responding (HTTP 200)
- [ ] SQLite database integrity check passed
- [ ] WAL mode enabled, busy_timeout set
- [ ] Dashboard metrics endpoint active
- [ ] Screenshot and reporting pipeline configured
- [ ] Evidence capture directories created and writable
- [ ] Rollback procedures documented and tested
- [ ] Human approval gates cleared for all planned changes
- [ ] No external sync or export jobs active
- [ ] Local-only network binding verified (192.168.1.116)
- [ ] Benchmark harness configuration matches container args

## Mid-Benchmark Monitoring Checklist
- [ ] VRAM usage < 90% (continuous watch)
- [ ] Proxy latency < 5s
- [ ] No `finish_reason=length` anomalies
- [ ] Database lock contention < 1%
- [ ] Dashboard metrics align with container logs
- [ ] Evidence collection running at 15-min intervals
- [ ] Human approval gates triggered if thresholds breached

## Post-Benchmark Checklist
- [ ] All responses parsed and validated
- [ ] `finish_reason` matches expected behavior
- [ ] Reasoning budget utilization logged
- [ ] Long-output completeness verified
- [ ] Database updated with results
- [ ] Dashboard metrics exported locally
- [ ] Screenshots archived
- [ ] Evidence manifest updated
- [ ] Rollback not required (or rollback completed and verified)
- [ ] Human approval gates closed with sign-off
- [ ] Local-only privacy compliance confirmed
- [ ] Benchmark report generated and stored in `/models/reports/`

## Go/No-Go Decision Matrix
| Condition | Decision |
|-----------|----------|
| All checklist items PASS | GO |
| Any pre-benchmark item FAIL | NO-GO (fix before proceeding) |
| Mid-benchmark threshold breach | PAUSE (triage tree → human approval) |
| Post-benchmark validation FAIL | NO-GO (rollback → re-run) |
| Privacy/external sync detected | NO-GO (isolate → audit → fix) |

This checklist ensures methodical execution, minimizes risk, and maintains full auditability. All decisions are logged, all changes are reversible, and all evidence is preserved locally. The benchmark infrastructure remains stable, private, and reproducible.

# Advanced MTP & Speculative Decoding Tuning

Speculative decoding with `--spec-type draft-mtp --spec-draft-n-max 2` introduces additional computational overhead and memory allocation patterns that can destabilize long-output benchmarks if not carefully tuned. The following procedures ensure MTP operates within safe boundaries while maximizing throughput.

## Draft Model Selection & Alignment
- **Compatibility Check**: Verify the draft model architecture matches the base model's tokenization and attention patterns.
  ```bash
  docker exec llama-cpp-server cat /etc/llama-cpp/draft-model-info.json [read-only]
  ```
- **VRAM Allocation**: The draft model consumes ~2-4GB VRAM alongside the base model. Monitor fragmentation.
  ```bash
  rocm-smi --showmeminfo vram --json | jq '.gpu[0].vram_used' [read-only]
  ```
- **Acceptance Rate Monitoring**: Track draft token acceptance rate via proxy metrics.
  ```bash
  curl -s http://192.168.1.116:8181/v1/metrics | grep -i "spec_accept" [read-only]
  ```

## Tuning Parameters
- **`--spec-draft-n-max 2`**: Limits speculative steps to 2. Increase only if acceptance rate > 60% and VRAM headroom > 15%.
  ```bash
  # Update via environment variable (requires restart)
  docker stop llama-cpp-server [medium-risk]
  docker run -d --name llama-cpp-server --gpus all --network host \
    -v /models:/models \
    -e SPEC_DRAFT_N_MAX=3 \
    llama-cpp-server-vulkan:working-20260613 \
    --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
    --ctx-size 262144 --parallel 1 \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --reasoning-budget 8192 [medium-risk]
  ```
- **`--spec-k`**: Controls how many tokens are checked against the base model. Default is usually optimal. Override only if validation shows excessive rejections.
  ```bash
  docker exec llama-cpp-server grep "spec_k" /etc/llama-cpp/config.yaml [read-only]
  ```

## MTP Failure Modes & Mitigation
| Symptom | Root Cause | Mitigation |
|---------|------------|------------|
| High rejection rate (>40%) | Draft model mismatch or context overflow | Switch to base-only mode temporarily, verify draft model checksum |
| VRAM spike during draft generation | KV cache fragmentation | Enable `--cache-reuse` or restart container to clear allocations |
| Latency > 8s per token | MTP overhead exceeds compute capacity | Reduce `--spec-draft-n-max` to 1, disable MTP for validation |
| `finish_reason=length` after draft | `max_tokens` ceiling hit during speculative phase | Increase `max_tokens` by 25%, verify reasoning budget alignment |

## Validation Matrix for MTP
Run sequential dry-runs with varying MTP parameters to establish baseline performance:
```bash
for n in 1 2 3; do
  curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced\",\"messages\":[{\"role\":\"user\",\"content\":\"Generate a 500-word technical explanation of speculative decoding.\"}],\"max_tokens\":1024,\"stream\":false}" | \
    jq "{draft_n_max: $n, tokens: (.choices[0].content | length), finish_reason: .choices[0].finish_reason, latency_ms: .usage.total_time}" > /models/benchmarks/mtp_validation/run_${n}.json
done [read-only]
```
Compare results. Select configuration that maximizes token throughput while maintaining `finish_reason=stop`.

# Vulkan & ROCm Memory Management Deep Dive

AMD GPU memory management via Vulkan requires careful monitoring of allocation patterns, driver compatibility, and container passthrough. The following procedures ensure stable VRAM utilization during extended benchmark runs.

## Driver & Runtime Verification
- **ROCm Version**: Confirm compatibility with Vulkan backend.
  ```bash
  rocm-smi --showdriverversion [read-only]
  ```
- **Vulkan Driver**: Verify AMDVLK or RADV is active and supports required features.
  ```bash
  vulkaninfo | grep -i "driver" [read-only]
  ```
- **Container Passthrough**: Ensure `/dev/dri` and ROCm devices are mounted correctly.
  ```bash
  docker exec llama-cpp-server ls -l /dev/dri /dev/kfd [read-only]
  ```

## Memory Allocation Patterns
- **Layer Offloading**: llama.cpp offloads layers to VRAM. Verify layer count matches model size.
  ```bash
  docker logs llama-cpp-server 2>&1 | grep -i "gpu layers" [read-only]
  ```
- **KV Cache Management**: 262144 context consumes ~4-6GB VRAM. Monitor usage.
  ```bash
  watch -n 1 rocm-smi --showmeminfo vram [read-only]
  ```
- **Fragmentation Detection**: High fragmentation causes OOM despite available VRAM.
  ```bash
  rocm-smi --showmeminfo vram --json | jq '.gpu[0].vram_used_pct' [read-only]
  ```

## Optimization Procedures
- **Enable Cache Reuse**: Reduces allocation overhead during sequential tasks.
  ```bash
  docker exec llama-cpp-server grep "cache_reuse" /etc/llama-cpp/config.yaml [read-only]
  ```
- **Disable Unnecessary Features**: Turn off logging, telemetry, or unused backends.
  ```bash
  docker exec llama-cpp-server sed -i 's/enable_logging=true/enable_logging=false/' /etc/llama-cpp/config.yaml [low-risk]
  ```
- **Periodic GC**: Schedule container restarts during long benchmarks to clear fragmented allocations.
  ```bash
  # Add to crontab (local only)
  0 */4 * * * docker restart llama-cpp-server [medium-risk]
  ```

## Troubleshooting Vulkan Crashes
- **Check dmesg**: Look for amdgpu or Vulkan errors.
  ```bash
  dmesg | grep -i amdgpu | tail -20 [read-only]
  ```
- **Reset GPU State**: Clear stuck allocations.
  ```bash
  rocm-smi --setgpu 0 [medium-risk]
  ```
- **Update Drivers**: If crashes persist, update AMDVLK and ROCm.
  ```bash
  apt update && apt install -y amdvlk rocm-dev [medium-risk]
  ```
- **Fallback to CPU**: If GPU remains unstable, switch to CPU-only mode for validation.
  ```bash
  docker stop llama-cpp-server [medium-risk]
  docker run -d --name llama-cpp-server --network host \
    -v /models:/models \
    llama-cpp-server-vulkan:working-20260613 \
    --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
    --ctx-size 262144 --parallel 1 --n-gpu-layers 0 \
    --reasoning-budget 8192 [medium-risk]
  ```

# SQLite Benchmark Database Optimization

The SQLite database on `/models` stores benchmark metadata, results, and configuration. High-throughput logging requires optimization to prevent lock contention, corruption, and performance degradation.

## Schema & Index Optimization
- **Verify Schema**:
  ```bash
  sqlite3 /models/benchmark.db ".schema" [read-only]
  ```
- **Add Indexes**: Speed up query performance for large datasets.
  ```bash
  sqlite3 /models/benchmark.db "CREATE INDEX IF NOT EXISTS idx_run_status ON benchmark_runs(status);" [low-risk]
  sqlite3 /models/benchmark.db "CREATE INDEX IF NOT EXISTS idx_run_created ON benchmark_runs(created_at);" [low-risk]
  ```
- **WAL Mode**: Enable Write-Ahead Logging for concurrent access.
  ```bash
  sqlite3 /models/benchmark.db "PRAGMA journal_mode=WAL;" [low-risk]
  ```
- **Busy Timeout**: Prevent lock contention during benchmark runs.
  ```bash
  sqlite3 /models/benchmark.db "PRAGMA busy_timeout = 5000;" [low-risk]
  ```

## Concurrency Control
- **Connection Pooling**: Use connection pooling in benchmark harness to reduce lock overhead.
- **Batch Inserts**: Group multiple result records into single transactions.
  ```bash
  sqlite3 /models/benchmark.db <<EOF
  BEGIN TRANSACTION;
  INSERT INTO benchmark_results (run_id, task_id, output, tokens, latency_ms) VALUES ('run_001', 'task_01', '{"result":"ok"}', 1200, 4500);
  INSERT INTO benchmark_results (run_id, task_id, output, tokens, latency_ms) VALUES ('run_001', 'task_02', '{"result":"ok"}', 980, 3800);
  COMMIT;
  EOF [low-risk]
  ```
- **Vacuum Schedule**: Run periodically to reclaim space.
  ```bash
  sqlite3 /models/benchmark.db "VACUUM;" [medium-risk]
  ```

## Corruption Prevention
- **Integrity Checks**: Run after every major benchmark cycle.
  ```bash
  sqlite3 /models/benchmark.db "PRAGMA integrity_check;" [read-only]
  ```
- **Backup Strategy**: Automated local backups every 2 hours.
  ```bash
  # Cron job (local only)
  0 */2 * * * cp /models/benchmark.db /models/backups/benchmark_$(date +\%Y\%m\%d\%H\%M).db [low-risk]
  ```
- **Recovery Procedure**: If corruption detected, restore from latest backup.
  ```bash
  cp /models/backups/benchmark_*.db /models/benchmark.db [medium-risk]
  sqlite3 /models/benchmark.db "PRAGMA integrity_check;" [read-only]
  ```

# Proxy Routing & Health Check Automation

The Flight Recorder proxy (`:8181`) routes benchmark requests to the llama.cpp container. Automated health checks and routing validation ensure reliability during extended runs.

## Health Check Configuration
- **Endpoint Verification**:
  ```bash
  curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1/health [read-only]
  ```
- **Model Availability**:
  ```bash
  curl -s http://192.168.1.116:8181/v1/models | jq '.data[0].id' [read-only]
  ```
- **Latency Monitoring**:
  ```bash
  curl -s -o /dev/null -w "%{time_total}" http://192.168.1.116:8181/v1/health [read-only]
  ```

## Routing Validation
- **Path Mapping**: Verify proxy correctly routes `/v1/*` to container.
  ```bash
  docker exec flight-recorder-proxy cat /etc/nginx/nginx.conf [read-only]
  ```
- **Rate Limiting**: Ensure proxy isn't throttling benchmark traffic.
  ```bash
  docker exec flight-recorder-proxy cat /etc/nginx/conf.d/rate-limit.conf [read-only]
  ```
- **CORS & Headers**: Verify headers allow benchmark harness requests.
  ```bash
  curl -s -I http://192.168.1.116:8181/v1/chat/completions [read-only]
  ```

## Automated Health Checks
- **Cron Script**: Run every 30 seconds during benchmark.
  ```bash
  # /usr/local/bin/proxy-health-check.sh
  #!/bin/bash
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1/health)
  if [ "$STATUS" != "200" ]; then
    echo "$(date): Proxy health check failed with status $STATUS" >> /models/logs/proxy_alerts.log
    docker restart flight-recorder-proxy [medium-risk]
  fi
  ```
- **Alert Thresholds**: Trigger human approval if proxy fails > 3 times in 5 minutes.
- **Failover Logic**: If proxy remains unresponsive, route directly to container for validation.
  ```bash
  curl -s http://192.168.1.116:8000/v1/health [read-only]
  ```

# Dashboard Metric Aggregation & Alerting

ServerTop (`:8090`) provides real-time metrics. Automated aggregation and alerting ensure observability aligns with container state.

## Metric Export & Storage
- **Prometheus Endpoint**:
  ```bash
  curl -s http://192.168.1.116:8090/metrics | head -50 [read-only]
  ```
- **Local Export**:
  ```bash
  curl -s http://192.168.1.116:8090/api/export?format=json > /models/reports/dashboard_$(date +%Y%m%d).json [low-risk]
  ```
- **CSV Conversion**:
  ```bash
  jq -r '.metrics[] | [.timestamp, .vram_used, .latency_ms] | @csv' /models/reports/dashboard_$(date +%Y%m%d).json > /models/reports/metrics.csv [low-risk]
  ```

## Alert Configuration
- **VRAM Threshold**: Alert if > 90%.
  ```bash
  # Local alert script
  VRAM=$(curl -s http://192.168.1.116:8090/api/gpu | jq '.vram_used_pct')
  if [ "$VRAM" -gt 90 ]; then
    echo "$(date): VRAM threshold exceeded: ${VRAM}%" >> /models/logs/alerts.log
    notify-send "VRAM Alert" "VRAM usage at ${VRAM}%" [low-risk]
  fi
  ```
- **Latency Threshold**: Alert if > 5s.
  ```bash
  LATENCY=$(curl -s http://192.168.1.116:8090/api/metrics | jq '.avg_latency_ms')
  if [ "$LATENCY" -gt 5000 ]; then
    echo "$(date): Latency threshold exceeded: ${LATENCY}ms" >> /models/logs/alerts.log
  fi
  ```
- **Error Rate Threshold**: Alert if > 5%.
  ```bash
  ERRORS=$(curl -s http://192.168.1.116:8090/api/metrics | jq '.error_rate')
  if [ "$(echo "$ERRORS > 0.05" | bc)" -eq 1 ]; then
    echo "$(date): Error rate threshold exceeded: ${ERRORS}" >> /models/logs/alerts.log
  fi
  ```

## Dashboard Synchronization
- **Screenshot Automation**: Capture at benchmark start, midpoint, end.
  ```bash
  chromium-browser --headless --screenshot=/models/screenshots/bench_$(date +%Y%m%d%H%M%S).png \
    --window-size=1920,1080 http://192.168.1.116:8090 [low-risk]
  ```
- **Metric Validation**: Cross-check dashboard metrics with `rocm-smi` and container logs.
  ```bash
  diff <(rocm-smi --showmeminfo vram --json | jq '.gpu[0].vram_used') <(curl -s http://192.168.1.116:8090/api/gpu | jq '.vram_used') [read-only]
  ```
- **Historical Gaps**: Ensure SQLite backup includes metric snapshots. Restore if needed.

# Automated Evidence Collection & Privacy Masking Pipeline

Comprehensive evidence collection ensures reproducibility, auditability, and privacy compliance. All artifacts are stored locally and never exported.

## Collection Automation
- **Log Aggregation**:
  ```bash
  # /usr/local/bin/evidence-collector.sh
  #!/bin/bash
  TIMESTAMP=$(date +%Y%m%d%H%M%S)
  docker logs --tail 500 llama-cpp-server > /models/logs/container_${TIMESTAMP}.log
  docker logs --tail 500 flight-recorder-proxy > /models/logs/proxy_${TIMESTAMP}.log
  journalctl -u docker --no-pager > /models/logs/system_${TIMESTAMP}.log
  rocm-smi --showmeminfo vram --json > /models/metrics/vram_${TIMESTAMP}.json
  curl -s http://192.168.1.116:8090/api/export?format=json > /models/reports/dashboard_${TIMESTAMP}.json
  ```
- **API Response Capture**:
  ```bash
  curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"test"}],"max_tokens":100,"stream":false}' | \
    jq '{timestamp: now, request: .model, response_tokens: (.choices[0].content | length), finish_reason: .choices[0].finish_reason}' > /models/responses/$(date +%Y%m%d%H%M%S).json [read-only]
  ```

## Privacy Masking Pipeline
- **Sensitive Field Detection**: Identify and mask model weights, internal prompts, PII.
  ```bash
  # /usr/local/bin/privacy-mask.sh
  #!/bin/bash
  INPUT=$1
  OUTPUT=$2
  sed -E 's/"content":\s*"[^"]*"/"content": "[MASKED]"/g' "$INPUT" | \
  sed -E 's/"weight":\s*"[^"]*"/"weight": "[MASKED]"/g' | \
  sed -E 's/"prompt":\s*"[^"]*"/"prompt": "[MASKED]"/g' > "$OUTPUT" [low-risk]
  ```
- **Hash Sensitive Fields**:
  ```bash
  echo -n "internal_prompt_data" | sha256sum [read-only]
  ```
- **Verification**: Ensure no raw data leaks.
  ```bash
  grep -r "raw_payload\|model_weights\|internal_key" /models/responses/ [read-only]
  ```

## Retention & Compliance
- **Retention Policy**: 30 days for logs, 90 days for reports, 1 year for approvals.
- **Encryption**: Local LUKS or GPG encryption for sensitive artifacts.
  ```bash
  gpg --symmetric --cipher-algo AES256 /models/responses/*.json [low-risk]
  ```
- **No External Sync**: Verify no cron jobs, rsync, or cloud sync tools are active.
  ```bash
  crontab -l [read-only]
  ls -la /etc/cron.d/ [read-only]
  ```
- **Audit Trail**: All evidence collection logged in `/models/benchmarks/logs/evidence_manifest.json`.

# Disaster Recovery & State Synchronization

Disaster recovery ensures infrastructure can be restored quickly after catastrophic failures, data corruption, or hardware issues.

## State Backup Strategy
- **Container Configuration**:
  ```bash
  docker inspect --format='{{json .Config}}' llama-cpp-server > /models/backups/container_config.json [read-only]
  ```
- **Database Backup**:
  ```bash
  cp /models/benchmark.db /models/backups/benchmark_$(date +%Y%m%d%H%M%S).db [low-risk]
  ```
- **Model Weights**: Verify checksums, store locally.
  ```bash
  sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf > /models/backups/model_checksum.txt [read-only]
  ```
- **Proxy Configuration**:
  ```bash
  docker exec flight-recorder-proxy cat /etc/nginx/nginx.conf > /models/backups/proxy_config.conf [read-only]
  ```

## Recovery Procedures
- **Container Restore**:
  ```bash
  docker stop llama-cpp-server [medium-risk]
  docker rm llama-cpp-server [medium-risk]
  docker run -d --name llama-cpp-server --gpus all --network host \
    -v /models:/models \
    -e MAX_TOKENS=4096 \
    llama-cpp-server-vulkan:working-20260613 \
    --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
    --ctx-size 262144 --parallel 1 \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --reasoning-budget 8192 [medium-risk]
  ```
- **Database Restore**:
  ```bash
  cp /models/backups/benchmark_*.db /models/benchmark.db [medium-risk]
  sqlite3 /models/benchmark.db "PRAGMA integrity_check;" [read-only]
  ```
- **Proxy Restore**:
  ```bash
  docker stop flight-recorder-proxy [medium-risk]
  docker rm flight-recorder-proxy [medium-risk]
  docker run -d --name flight-recorder-proxy --network host \
    -v /models/backups/proxy_config.conf:/etc/nginx/nginx.conf \
    nginx:alpine [medium-risk]
  ```
- **Model Verification**:
  ```bash
  sha256sum -c /models/backups/model_checksum.txt [read-only]
  ```

## Synchronization Validation
- **Cross-Check State**: Ensure container, database, proxy, and dashboard are synchronized.
  ```bash
  curl -s http://192.168.1.116:8181/v1/health [read-only]
  curl -s http://192.168.1.116:8090/api/status | jq '.' [read-only]
  sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM benchmark_runs;" [read-only]
  ```
- **Dry-Run Validation**: Execute minimal benchmark task to verify end-to-end functionality.
  ```bash
  curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"test recovery"}],"max_tokens":50,"stream":false}' | jq '.' [read-only]
  ```

# Expanded Triage Trees (Edge Cases & Race Conditions)

Additional decision trees for handling rare but critical failures during benchmark operations.

## Tree 6: Race Condition Between MTP & Reasoning Budget
```
Start
 ├─ Check if reasoning_tokens + draft_tokens > 8192
 │  ├─ Yes → Reduce --spec-draft-n-max to 1 [medium-risk]
 │  └─ No → Proceed
 ├─ Verify max_tokens > reasoning_tokens + draft_tokens + final_tokens
 │  ├─ No → Increase max_tokens by 2048 [medium-risk]
 │  └─ Yes → Proceed
 ├─ Check for context overflow in KV cache
 │  ├─ Yes → Reduce --ctx-size to 131072 [medium-risk]
 │  └─ No → Proceed
 └─ Restart container & re-run validation [medium-risk]
```

## Tree 7: SQLite Lock Contention During Bulk Logging
```
Start
 ├─ Check PRAGMA busy_timeout
 │  ├─ < 5000 → Set to 5000 [low-risk]
 │  └─ ≥ 5000 → Proceed
 ├─ Verify WAL mode enabled
 │  ├─ No → Enable WAL [low-risk]
 │  └─ Yes → Proceed
 ├─ Check for concurrent writes
 │  ├─ Yes → Implement batch inserts [low-risk]
 │  └─ No → Proceed
 ├─ Check disk I/O saturation
 │  ├─ Yes → Move database to faster storage [medium-risk]
 │  └─ No → Proceed
 └─ Monitor lock wait times [read-only]
```

## Tree 8: Vulkan Memory Leak During Extended Runs
```
Start
 ├─ Check VRAM usage trend over time
 │  ├─ Increasing steadily → Memory leak suspected [read-only]
 │  └─ Stable → Proceed
 ├─ Check for layer reloading
 │  ├─ Yes → Disable dynamic layer loading [medium-risk]
 │  └─ No → Proceed
 ├─ Check for context cache buildup
 │  ├─ Yes → Enable cache reuse or restart container [medium-risk]
 │  └─ No → Proceed
 ├─ Check for Vulkan driver bugs
 │  ├─ Yes → Update AMDVLK driver [medium-risk]
 │  └─ No → Proceed
 └─ Schedule periodic container restarts [medium-risk]
```

## Tree 9: Proxy Routing Loop or Infinite Redirect
```
Start
 ├─ Check proxy configuration for redirect rules
 │  ├─ Found → Remove or fix redirect rule [medium-risk]
 │  └─ Not found → Proceed
 ├─ Verify container health endpoint
 │  ├─ Unresponsive → Restart container [medium-risk]
 │  └─ Responsive → Proceed
 ├─ Check for DNS resolution issues
 │  ├─ Yes → Fix DNS or use IP directly [medium-risk]
 │  └─ No → Proceed
 └─ Restart proxy service [medium-risk]
```

# Detailed Rollback Validation & Smoke Testing

Rollback procedures must be followed by rigorous validation to ensure infrastructure is fully restored and operational.

## Pre-Rollback Validation
- **Snapshot Current State**: Capture container config, database state, proxy routing, dashboard metrics.
  ```bash
  docker inspect --format='{{json .Config}}' llama-cpp-server > /models/backups/pre_rollback_config.json [read-only]
  sqlite3 /models/benchmark.db "SELECT * FROM benchmark_runs ORDER BY created_at DESC LIMIT 10;" > /models/backups/pre_rollback_db.json [read-only]
  curl -s http://192.168.1.116:8181/v1/health > /models/backups/pre_rollback_proxy.json [read-only]
  curl -s http://192.168.1.116:8090/api/status > /models/backups/pre_rollback_dashboard.json [read-only]
  ```
- **Document Rollback Plan**: Specify target version, expected downtime, validation steps.

## Post-Rollback Validation
- **Container Health**:
  ```bash
  docker inspect --format='{{.State.Status}} {{.Config.Image}}' llama-cpp-server [read-only]
  docker logs --tail 50 llama-cpp-server [read-only]
  ```
- **API Responsiveness**:
  ```bash
  curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"smoke test"}],"max_tokens":20,"stream":false}' | jq '.' [read-only]
  ```
- **Database Integrity**:
  ```bash
  sqlite3 /models/benchmark.db "PRAGMA integrity_check;" [read-only]
  sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM benchmark_runs;" [read-only]
  ```
- **Proxy Routing**:
  ```bash
  curl -s http://192.168.1.116:8181/v1/health [read-only]
  curl -s http://192.168.1.116:8181/v1/models | jq '.data[0].id' [read-only]
  ```
- **Dashboard Metrics**:
  ```bash
  curl -s http://192.168.1.116:8090/api/status | jq '.' [read-only]
  curl -s http://192.168.1.116:8090/api/gpu | jq '.vram_used_pct' [read-only]
  ```
- **GPU Verification**:
  ```bash
  rocm-smi --showmeminfo vram [read-only]
  docker exec llama-cpp-server ls -l /dev/dri [read-only]
  ```

## Smoke Test Suite
Execute a minimal benchmark suite to verify end-to-end functionality:
```bash
# Task 1: Short output
curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"What is 2+2?"}],"max_tokens":10,"stream":false}' | jq '.choices[0].finish_reason' [read-only]

# Task 2: Reasoning budget test
curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"Explain step-by-step how to calculate the derivative of x^2 + 3x."}],"max_tokens":500,"stream":false}' | jq '{reasoning_tokens: (.choices[0].message.reasoning_content | length), finish_reason: .choices[0].finish_reason}' [read-only]

# Task 3: Long output test
curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"Generate a 500-word technical explanation of speculative decoding."}],"max_tokens":2048,"stream":false}' | jq '{content_length: (.choices[0].content | length), finish_reason: .choices[0].finish_reason}' [read-only]
```
All tasks must return `finish_reason=stop`. If any fail, halt rollback validation and escalate to human approval.

# Human Approval Gate Automation & Audit Trail

Human approval gates ensure critical operations are authorized and auditable. Automation tracks approvals, enforces delays, and maintains immutable logs.

## Gate Automation Script
```bash
# /usr/local/bin/approval-gate.sh
#!/bin/bash
ACTION=$1
APPROVER=$2
TIMESTAMP=$(date +%Y%m%d%H%M%S)
LOG_FILE="/models/benchmarks/logs/approvals/${TIMESTAMP}_${ACTION}.log"

echo "Action: $ACTION" > "$LOG_FILE"
echo "Approver: $APPROVER" >> "$LOG_FILE"
echo "Timestamp: $TIMESTAMP" >> "$LOG_FILE"
echo "Status: PENDING" >> "$LOG_FILE"

# Wait for human sign-off (simulated)
while [ "$(cat /models/benchmarks/logs/approvals/${TIMESTAMP}_${ACTION}.status 2>/dev/null)" != "APPROVED" ]; do
  sleep 5
done

echo "Status: APPROVED" >> "$LOG_FILE"
echo "Execution: $(date)" >> "$LOG_FILE"
echo "Action executed successfully." [low-risk]
```

## Audit Trail Maintenance
- **Immutable Logs**: Store approvals in append-only files.
  ```bash
  chmod a-w /models/benchmarks/logs/approvals/*.log [low-risk]
  ```
- **Verification**: Cross-check approval logs with execution logs.
  ```bash
  diff /models/benchmarks/logs/approvals/*.log /models/benchmarks/logs/executions/*.log [read-only]
  ```
- **Retention**: Keep approval logs for 1 year.
- **Export**: Generate quarterly audit reports.
  ```bash
  jq -s '.' /models/benchmarks/logs/approvals/*.log > /models/reports/audit_report_$(date +%Y%m).json [low-risk]
  ```

## Bypass Conditions
- **Emergency Recovery**: Human approval can be bypassed if infrastructure is completely unresponsive and data loss is imminent. Document bypass reason, approver ID, and timestamp.
- **Automated Rollback**: If critical failure detected (e.g., VRAM OOM, proxy crash), automated rollback triggers after 3 failed retries. Human approval required within 1 hour.

# Final Go/No-Go Checklist Expansion & Validation Scripts

The final checklist ensures all operational, technical, and compliance requirements are met before benchmark execution. Expanded with automated validation scripts.

## Pre-Benchmark Validation Script
```bash
# /usr/local/bin/preflight-validation.sh
#!/bin/bash
PASS=0
FAIL=0

# Check container status
if docker inspect --format='{{.State.Status}}' llama-cpp-server | grep -q "running"; then
  echo "PASS: Container is running"
  ((PASS++))
else
  echo "FAIL: Container is not running"
  ((FAIL++))
fi

# Check GPU passthrough
if docker exec llama-cpp-server ls -l /dev/dri/renderD128 > /dev/null 2>&1; then
  echo "PASS: GPU passthrough verified"
  ((PASS++))
else
  echo "FAIL: GPU passthrough missing"
  ((FAIL++))
fi

# Check proxy health
if curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1/health | grep -q "200"; then
  echo "PASS: Proxy health check passed"
  ((PASS++))
else
  echo "FAIL: Proxy health check failed"
  ((FAIL++))
fi

# Check database integrity
if sqlite3 /models/benchmark.db "PRAGMA integrity_check;" | grep -q "ok"; then
  echo "PASS: Database integrity verified"
  ((PASS++))
else
  echo "FAIL: Database integrity check failed"
  ((FAIL++))
fi

# Check VRAM usage
VRAM=$(rocm-smi --showmeminfo vram --json | jq '.gpu[0].vram_used_pct')
if [ "$VRAM" -lt 85 ]; then
  echo "PASS: VRAM usage within limits (${VRAM}%)"
  ((PASS++))
else
  echo "FAIL: VRAM usage too high (${VRAM}%)"
  ((FAIL++))
fi

# Check reasoning budget alignment
REASONING=$(curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced","messages":[{"role":"user","content":"test"}],"max_tokens":100,"stream":false}' | \
  jq '.choices[0].message.reasoning_content | length')
if [ "$REASONING" -lt 8192 ]; then
  echo "PASS: Reasoning budget alignment verified (${REASONING} tokens)"
  ((PASS++))
else
  echo "FAIL: Reasoning budget exceeded (${REASONING} tokens)"
  ((FAIL++))
fi

echo "Results: $PASS passed, $FAIL failed"
if [ "$FAIL" -eq 0 ]; then
  echo "GO: All preflight checks passed"
else
  echo "NO-GO: Preflight checks failed"
  exit 1
fi [read-only]
```

## Mid-Benchmark Monitoring Script
```bash
# /usr/local/bin/monitoring-check.sh
#!/bin/bash
TIMESTAMP=$(date +%Y%m%d%H%M%S)

# VRAM monitoring
VRAM=$(rocm-smi --showmeminfo vram --json | jq '.gpu[0].vram_used_pct')
if [ "$VRAM" -gt 90 ]; then
  echo "$(date): VRAM threshold exceeded: ${VRAM}%" >> /models/logs/alerts.log
fi

# Proxy latency
LATENCY=$(curl -s -o /dev/null -w "%{time_total}" http://192.168.1.116:8181/v1/health)
if [ "$(echo "$LATENCY > 5" | bc)" -eq 1 ]; then
  echo "$(date): Proxy latency threshold exceeded: ${LATENCY}s" >> /models/logs/alerts.log
fi

# Database lock contention
LOCKS=$(sqlite3 /models/benchmark.db "PRAGMA busy_timeout;" 2>&1)
if [ -n "$LOCKS" ]; then
  echo "$(date): Database lock contention detected" >> /models/logs/alerts.log
fi

# Evidence collection
docker logs --tail 100 llama-cpp-server > /models/logs/container_${TIMESTAMP}.log
curl -s http://192.168.1.116:8090/api/export?format=json > /models/reports/dashboard_${TIMESTAMP}.json [read-only]
```

## Post-Benchmark Validation Script
```bash
# /usr/local/bin/post-benchmark-validation.sh
#!/bin/bash
PASS=0
FAIL=0

# Check all responses completed
COMPLETED=$(sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM benchmark_results WHERE status='completed';")
TOTAL=$(sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM benchmark_results;")
if [ "$COMPLETED" -eq "$TOTAL" ]; then
  echo "PASS: All benchmark tasks completed"
  ((PASS++))
else
  echo "FAIL: Incomplete benchmark tasks (${COMPLETED}/${TOTAL})"
  ((FAIL++))
fi

# Check finish_reason consistency
LENGTH_COUNT=$(sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM benchmark_results WHERE finish_reason='length';")
if [ "$LENGTH_COUNT" -eq 0 ]; then
  echo "PASS: No premature truncations detected"
  ((PASS++))
else
  echo "FAIL: Premature truncations detected (${LENGTH_COUNT})"
  ((FAIL++))
fi

# Check evidence collection
if [ -d "/models/logs" ] && [ -d "/models/reports" ] && [ -d "/models/screenshots" ]; then
  echo "PASS: Evidence collection directories populated"
  ((PASS++))
else
  echo "FAIL: Evidence collection incomplete"
  ((FAIL++))
fi

# Check privacy compliance
if grep -r "raw_payload\|model_weights\|internal_key" /models/responses/ > /dev/null 2>&1; then
  echo "FAIL: Privacy compliance violation detected"
  ((FAIL++))
else
  echo "PASS: Privacy compliance verified"
  ((PASS++))
fi

echo "Results: $PASS passed, $FAIL failed"
if [ "$FAIL" -eq 0 ]; then
  echo "GO: Post-benchmark validation passed"
else
  echo "NO-GO: Post-benchmark validation failed"
  exit 1
fi [read-only]
```

# Operational Runbook Appendices

Additional operational procedures, compliance checks, and maintenance schedules to ensure long-term infrastructure stability.

## Maintenance Schedule
- **Daily**: Run preflight validation script, check proxy health, verify database integrity.
- **Weekly**: Update AMDVLK driver (if compatible), vacuum SQLite database, rotate logs.
- **Monthly**: Full disaster recovery test, audit trail review, compliance verification.
- **Quarterly**: Benchmark harness configuration review, MTP parameter tuning, VRAM optimization.

## Compliance Verification
- **Local-Only Enforcement**: Verify no external sync, export, or cloud storage tools are active.
  ```bash
  systemctl list-units --type=service | grep -i sync [read-only]
  crontab -l | grep -i rsync [read-only]
  ```
- **Data Masking**: Ensure all evidence collection pipelines apply privacy masking.
  ```bash
  grep -r "privacy-mask" /usr/local/bin/ [read-only]
  ```
- **Access Control**: Verify only authorized users can modify container configs, database, or proxy settings.
  ```bash
  ls -la /models/ [read-only]
  getfacl /models/benchmark.db [read-only]
  ```
- **Audit Logging**: Ensure all human approval gates, rollback procedures, and configuration changes are logged.
  ```bash
  ls -la /models/benchmarks/logs/approvals/ [read-only]
  ls -la /models/benchmarks/logs/executions/ [read-only]
  ```

## Performance Tuning Guidelines
- **Context Window**: Keep `--ctx-size` at 262144 unless VRAM constraints require reduction.
- **Parallel Slots**: Maintain `--parallel 1` during MTP validation to prevent slot contention.
- **MTP Parameters**: Start with `--spec-draft-n-max 2`, adjust based on acceptance rate and VRAM headroom.
- **Reasoning Budget**: Keep `--reasoning-budget 8192`, monitor actual usage, adjust if consistently near limit.
- **max_tokens**: Set to at least 2x expected output length to prevent premature truncation.

## Emergency Contact & Escalation
- **Level 1**: Automated scripts, preflight checks, basic triage trees.
- **Level 2**: Human approval gates, configuration changes, database schema modifications.
- **Level 3**: GPU driver updates, ROCm version changes, hardware replacement, disaster recovery.
- **Escalation Path**: Automated alert → Human approval gate → Level 2 intervention → Level 3 intervention → Infrastructure rebuild.

This expanded runbook provides comprehensive procedures for diagnosing, fixing, validating, and rolling back benchmark infrastructure problems in the specified home-lab AI server environment. All operations are governed by cautious execution principles, risk classification discipline, privacy preservation mandates, and human approval gate enforcement. The infrastructure remains stable, private, reproducible, and fully auditable throughout all benchmark cycles.