# Operating Principles

This runbook governs the diagnosis, remediation, validation, and rollback of benchmark infrastructure problems within a constrained, privacy-sensitive home-lab environment. The core philosophy rests on five immutable principles:

1. **Safety-First Execution:** Every command and configuration change is evaluated for blast radius. Destructive operations are strictly isolated, version-controlled, and require explicit human approval. No benchmark run proceeds until the baseline state is verified and documented.
2. **Incremental Validation:** Long-output and high-complexity tasks are never executed in bulk. Each benchmark task is validated individually before scaling. This prevents cascading failures, VRAM exhaustion, and silent truncation that corrupts aggregate metrics.
3. **Data Sovereignty & Privacy:** The SQLite database residing on `/models` is treated as a closed, air-gapped artifact. All diagnostic output is sanitized before external review. Screenshots and aggregated reports are generated locally; raw traces, model weights, and internal routing tables never leave the host.
4. **Deterministic Reproducibility:** Every benchmark run is anchored to a specific container image, model hash, configuration dump, and environment snapshot. Rollbacks restore exact states, not approximations. MTP draft models, reasoning budgets, and context windows are pinned to known-good configurations.
5. **Observability-Driven Triage:** Failures are diagnosed through layered telemetry: container logs, GPU metrics, proxy routing traces, SQLite integrity checks, and dashboard snapshots. Root cause analysis follows a strict hierarchy: infrastructure → runtime → model → benchmark harness → network.

The agent operates as a cautious intermediary between automated benchmark execution and human oversight. All actions are logged, timestamped, and tagged with risk classifications. When uncertainty exceeds a predefined threshold, execution halts pending human approval. This runbook ensures that benchmark infrastructure remains stable, reproducible, and privacy-compliant throughout the lifecycle of any diagnostic or remediation cycle.

# Preflight Checks

Before initiating any benchmark diagnostic or remediation cycle, the environment must be verified against a strict preflight baseline. These checks ensure that the host, container, model, and supporting services are in a known-good state.

**1. Host & Network Baseline**
Verify host connectivity, DNS resolution, and port availability. Ensure the host IP `192.168.1.116` is stable and not subject to DHCP lease changes.
```bash
# [read-only] Verify host IP and routing
ip addr show | grep "192.168.1.116"
ip route show default

# [read-only] Test local port reachability for proxy and dashboard
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1/health
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8090/health
```

**2. Container State Verification**
Confirm the llama-cpp-server container is running, healthy, and using the correct image tag. Check for resource limits and restart policies.
```bash
# [read-only] List container status and image
docker ps --filter "name=llama-cpp-server" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

# [read-only] Inspect container configuration for MTP and reasoning flags
docker inspect llama-cpp-server --format '{{.Config.Cmd}}'
```

**3. Model Integrity & Path Validation**
Verify the model file exists, matches expected hash, and is accessible to the container. Ensure the SQLite database path is correctly mounted and not exposed externally.
```bash
# [read-only] Verify model file and hash
ls -lh /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf

# [read-only] Verify SQLite database presence and permissions
ls -lh /models/benchmark.db
stat /models/benchmark.db
```

**4. Dependency & Runtime Checks**
Ensure Vulkan drivers, AMDGPU firmware, and container runtime dependencies are loaded. Verify that no conflicting processes are binding to ports 8181 or 8090.
```bash
# [read-only] Check AMDGPU kernel modules
lsmod | grep -E "amdgpu|kfd"

# [read-only] Verify port binding conflicts
ss -tlnp | grep -E ":(8181|8090)\s"
```

**5. Baseline Metric Capture**
Record initial GPU memory usage, container CPU/RAM limits, and proxy latency. These values serve as the reference point for detecting drift during benchmark execution.
```bash
# [read-only] Capture initial GPU memory state
rocm-smi --showmeminfo vram

# [read-only] Capture initial container resource usage
docker stats --no-stream llama-cpp-server
```

Preflight checks must complete successfully before any diagnostic or benchmark run begins. Any deviation triggers a halt and requires human review. All preflight outputs are archived in `/models/logs/preflight/YYYYMMDD-HHMMSS/` for auditability.

# Safe Inspection Commands

All inspection commands are classified by risk level. Only read-only and low-risk commands are permitted during active benchmark execution. Medium-risk and destructive commands require explicit human approval and are isolated to maintenance windows.

**[read-only] System & Kernel Inspection**
```bash
# [read-only] Check kernel version and AMDGPU firmware status
uname -r
dmesg | grep -i amdgpu | tail -n 20

# [read-only] Verify system memory and swap configuration
free -h
swapon --show
```

**[read-only] Container & Runtime Inspection**
```bash
# [read-only] List all containers and resource limits
docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Size}}"

# [read-only] Inspect container environment variables for sensitive flags
docker inspect llama-cpp-server --format '{{range .Config.Env}}{{println .}}{{end}}'

# [read-only] Check container log tail for recent errors
docker logs --tail 100 llama-cpp-server 2>&1 | grep -iE "error|warn|fatal|oom"
```

**[low-risk] Model & Storage Inspection**
```bash
# [low-risk] Verify model file permissions and ownership
ls -l /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
chown -R 1000:1000 /models 2>/dev/null || true

# [low-risk] Compact SQLite database to reclaim space and verify integrity
sqlite3 /models/benchmark.db "PRAGMA integrity_check;"
sqlite3 /models/benchmark.db "VACUUM;"
```

**[medium-risk] Configuration & Service Inspection**
```bash
# [medium-risk] Restart container with dry-run validation (no state change)
docker run --rm --entrypoint /bin/sh llama-cpp-server-vulkan:working-20260613 -c "echo 'Config validation passed'"

# [medium-risk] Flush container network cache and verify proxy routing
docker network prune --filter "label=benchmark" --force
curl -s http://192.168.1.116:8181/v1/models | jq '.data[0].id'
```

**[destructive] Cleanup & Reset Inspection**
```bash
# [destructive] Remove stopped containers and dangling images (requires approval)
docker container prune --force
docker image prune --force

# [destructive] Reset SQLite database to clean state (requires approval)
cp /models/benchmark.db /models/benchmark.db.backup.$(date +%s)
rm -f /models/benchmark.db
sqlite3 /models/benchmark.db < /models/schema.sql
```

All inspection commands must be logged with timestamps, exit codes, and output hashes. Medium-risk and destructive commands are never executed without a documented rollback plan and human sign-off. Output is sanitized to remove internal IPs, container IDs, and model metadata before archival.

# Reasoning Budget Verification

The interaction between MTP (Multi-Token Prediction) and the reasoning budget is a critical failure vector. The `--reasoning-budget 8192` flag allocates tokens for internal chain-of-thought processing, while `--spec-draft-n-max 2` limits draft token generation. Misalignment between these parameters causes premature `finish_reason=length` truncation, especially when long-output tasks consume the budget before final content generation.

**1. Budget Allocation Analysis**
Verify that the reasoning budget is correctly parsed by the container and that draft tokens do not consume the primary generation budget.
```bash
# [read-only] Extract reasoning budget from container config
docker inspect llama-cpp-server --format '{{.Config.Cmd}}' | grep -oP 'reasoning-budget \K\d+'

# [read-only] Verify MTP draft configuration
docker inspect llama-cpp-server --format '{{.Config.Cmd}}' | grep -oP 'spec-draft-n-max \K\d+'
```

**2. Truncation Detection & Validation**
Monitor benchmark responses for `finish_reason=length` appearing before `reasoning_content` completion. This indicates the budget is exhausted during draft generation or internal reasoning steps.
```bash
# [read-only] Parse benchmark logs for truncation patterns
grep -r "finish_reason.*length" /models/logs/benchmark/ | wc -l
grep -r "reasoning_content" /models/logs/benchmark/ | wc -l
```

**3. Budget Adjustment Protocol**
If truncation is detected, increase `max_tokens` to accommodate both reasoning and final content. Validate that the new budget does not exceed VRAM capacity.
```bash
# [medium-risk] Update container with adjusted max_tokens (requires approval)
docker stop llama-cpp-server
docker run -d --name llama-cpp-server \
  --gpus all \
  -v /models:/models \
  -p 8181:8080 \
  llama-cpp-server-vulkan:working-20260613 \
  --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
  --ctx-size 262144 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --reasoning-budget 8192 \
  --max-tokens 16384 \
  --host 0.0.0.0 --port 8080
```

**4. Reasoning Content Validation**
Ensure that `reasoning_content` is fully captured before final output. Validate that the benchmark harness correctly separates internal reasoning from user-facing responses.
```bash
# [read-only] Verify reasoning content completeness in logs
grep -A 5 "reasoning_content" /models/logs/benchmark/latest_run.json | grep -c "finish_reason"
```

**5. Budget Drift Monitoring**
Track reasoning budget consumption across benchmark runs. If drift exceeds 5%, investigate draft token efficiency or context window fragmentation.
```bash
# [read-only] Calculate average reasoning token usage
jq '[.[] | select(.reasoning_tokens != null) | .reasoning_tokens] | add / length' /models/logs/benchmark/aggregated.json
```

Reasoning budget verification must be performed before each benchmark cycle. Adjustments are logged, validated, and rolled back if they introduce instability. The agent prioritizes deterministic reasoning output over aggressive draft token generation.

# Long Output Validation

Long-output tests are highly susceptible to truncation, VRAM exhaustion, and MTP drift. The observation that `finish_reason=length` appears before final content confirms that the generation pipeline is hitting token limits during internal processing. Bulk execution of long-output tasks must be prohibited until each task is individually validated.

**1. Task Isolation & Sequential Validation**
Never run long-output tasks in parallel or bulk. Each task must be executed sequentially with explicit validation gates.
```bash
# [low-risk] Extract individual long-output tasks from benchmark suite
jq -r '.tasks[] | select(.type == "long_output") | .id' /models/benchmark/tasks.json > /models/logs/long_output_tasks.txt
```

**2. Chunked Generation Strategy**
Implement chunked generation with explicit boundary markers. Validate that each chunk completes before proceeding to the next.
```bash
# [medium-risk] Run single long-output task with chunk validation (requires approval)
curl -s http://192.168.1.116:8181/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced",
    "prompt": "TASK_ID_001",
    "max_tokens": 8192,
    "stop": ["[CHUNK_END]"],
    "stream": false
  }' | jq '.choices[0].text' > /models/logs/validated_chunks/task_001.json
```

**3. Truncation Recovery Protocol**
If `finish_reason=length` occurs, append continuation tokens and validate that final content appears after `reasoning_content`.
```bash
# [medium-risk] Resume truncated generation with continuation flag (requires approval)
curl -s http://192.168.1.116:8181/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced",
    "prompt": "TASK_ID_001",
    "max_tokens": 16384,
    "resume_from": "last_valid_token",
    "stream": false
  }' | jq '.choices[0].text' > /models/logs/validated_chunks/task_001_resumed.json
```

**4. Validation Gate Checks**
Each long-output task must pass three validation gates before proceeding:
- Gate 1: `reasoning_content` is fully present and non-empty.
- Gate 2: `finish_reason` is `stop` or `length` with valid continuation.
- Gate 3: Final content matches expected schema and length thresholds.
```bash
# [read-only] Validate long-output task completion
jq 'select(.reasoning_content != null and (.finish_reason == "stop" or .finish_reason == "length")) | .id' /models/logs/validated_chunks/task_001_resumed.json
```

**5. Bulk Execution Prohibition**
Document and enforce the policy that long-output tasks are never executed in bulk. Any automated pipeline attempting bulk execution must be halted and audited.
```bash
# [medium-risk] Disable bulk execution flag in benchmark harness (requires approval)
sed -i 's/"bulk_execution": true/"bulk_execution": false/' /models/benchmark/config.json
```

Long-output validation ensures that benchmark results are not corrupted by silent truncation. Each task is treated as an independent experiment with explicit validation boundaries.

# GPU And VRAM Checks

The AMD Radeon AI PRO R9700 with 32GB VRAM operates under Vulkan backend constraints. VRAM fragmentation, OOM conditions, and draft token allocation are primary failure vectors. Continuous monitoring and proactive management are required.

**1. VRAM Allocation Monitoring**
Track VRAM usage across model loading, context window expansion, and MTP draft generation.
```bash
# [read-only] Monitor VRAM usage in real-time
watch -n 2 rocm-smi --showmeminfo vram

# [read-only] Check VRAM fragmentation status
rocm-smi --showmeminfo vram | grep -E "Total|Used|Free"
```

**2. OOM Detection & Mitigation**
Detect out-of-memory conditions and trigger graceful degradation or container restart.
```bash
# [read-only] Check kernel OOM logs for GPU processes
dmesg | grep -i "out of memory" | tail -n 10

# [medium-risk] Trigger graceful container restart on OOM (requires approval)
docker restart llama-cpp-server
```

**3. Vulkan Backend Validation**
Ensure Vulkan drivers are correctly loaded and compatible with the container runtime.
```bash
# [read-only] Verify Vulkan driver status
vulkaninfo --summary | grep -E "GPU|Driver|Vulkan"

# [read-only] Check container Vulkan device access
docker exec llama-cpp-server ls -l /dev/dri/
```

**4. Context Window & VRAM Correlation**
Validate that the 262144 context window does not exceed VRAM capacity when combined with MTP draft tokens.
```bash
# [read-only] Calculate theoretical VRAM usage for context window
echo "Context VRAM estimate: $((262144 * 4 / 1024)) MB"
```

**5. GPU Memory Profiling**
Profile memory usage during benchmark execution to identify leaks or fragmentation.
```bash
# [medium-risk] Run GPU memory profiler for 60 seconds (requires approval)
docker exec llama-cpp-server /opt/rocm/bin/rocm-smi --showmeminfo vram --csv > /models/logs/gpu_profile.csv
```

GPU and VRAM checks must be performed before and after each benchmark cycle. Any VRAM usage exceeding 85% triggers a halt and requires human review. The agent prioritizes stability over performance, ensuring that benchmark results are not corrupted by memory pressure.

# Proxy And Database Checks

The Flight Recorder proxy (`http://192.168.1.116:8181/v1`) and SQLite database (`/models/benchmark.db`) form the critical data pipeline. Proxy routing failures and database corruption can silently invalidate benchmark results.

**1. Proxy Health & Routing Validation**
Verify proxy availability, routing correctness, and response latency.
```bash
# [read-only] Check proxy health endpoint
curl -s http://192.168.1.116:8181/v1/health | jq '.status'

# [read-only] Verify proxy routing to container
curl -s http://192.168.1.116:8181/v1/models | jq '.data[0].id'
```

**2. SQLite Integrity & Privacy Enforcement**
Ensure the database remains local, private, and structurally sound.
```bash
# [low-risk] Verify SQLite integrity
sqlite3 /models/benchmark.db "PRAGMA integrity_check;"

# [low-risk] Check database size and table structure
sqlite3 /models/benchmark.db ".tables"
sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM benchmark_runs;"
```

**3. Connection Pool & Timeout Validation**
Verify that proxy connection pools are not exhausted and timeouts are appropriately configured.
```bash
# [read-only] Check proxy connection pool status
curl -s http://192.168.1.116:8181/v1/status | jq '.connections.active'
```

**4. Database Backup & Restore Protocol**
Maintain local backups and validate restore procedures.
```bash
# [low-risk] Create timestamped database backup
cp /models/benchmark.db /models/benchmark.db.backup.$(date +%s)

# [medium-risk] Restore database from backup (requires approval)
cp /models/benchmark.db.backup.1690000000 /models/benchmark.db
```

**5. Privacy Sanitization**
Ensure all exported data is sanitized to remove internal IPs, container IDs, and model metadata.
```bash
# [low-risk] Sanitize benchmark export for reporting
jq 'del(.internal_routing, .container_metadata)' /models/logs/benchmark/export.json > /models/logs/benchmark/export_sanitized.json
```

Proxy and database checks must be performed before each benchmark cycle. Any routing failure or integrity violation triggers a halt and requires human review. The agent enforces strict data privacy boundaries, ensuring that raw data never leaves the host.

# Dashboard Checks

ServerTop (`http://192.168.1.116:8090`) provides real-time metrics on latency, throughput, error rates, and resource utilization. Dashboard checks ensure that benchmark infrastructure is operating within acceptable thresholds.

**1. Metric Threshold Validation**
Verify that latency, throughput, and error rates are within predefined bounds.
```bash
# [read-only] Check dashboard latency metrics
curl -s http://192.168.1.116:8090/metrics/latency | jq '.p99'

# [read-only] Check dashboard error rate
curl -s http://192.168.1.116:8090/metrics/errors | jq '.rate'
```

**2. Screenshot Capture Protocol**
Capture dashboard screenshots for reporting while preserving privacy.
```bash
# [low-risk] Capture dashboard screenshot
curl -s http://192.168.1.116:8090/screenshot > /models/logs/dashboard_screenshot.png
```

**3. Alert Threshold Configuration**
Verify that alert thresholds are correctly configured and not triggering false positives.
```bash
# [medium-risk] Update alert thresholds (requires approval)
curl -X PUT http://192.168.1.116:8090/config/alerts -H "Content-Type: application/json" -d '{"latency_threshold": 5000, "error_threshold": 0.05}'
```

**4. Historical Data Validation**
Verify that historical metrics are correctly stored and accessible.
```bash
# [read-only] Check historical metric retention
curl -s http://192.168.1.116:8090/metrics/history?days=7 | jq '.count'
```

Dashboard checks must be performed before and after each benchmark cycle. Any threshold violation triggers a halt and requires human review. The agent prioritizes accurate reporting over real-time updates, ensuring that dashboard data reflects true infrastructure state.

# Failure Triage Trees

Failure triage follows a strict decision tree to ensure consistent diagnosis and remediation. Each tree addresses a common failure mode with explicit validation steps and rollback triggers.

**1. Truncation Failure Tree**
- Symptom: `finish_reason=length` before final content
- Step 1: Verify `max_tokens` and `reasoning-budget` alignment
- Step 2: Check VRAM usage for fragmentation
- Step 3: Validate MTP draft token efficiency
- Step 4: If unresolved, increase `max_tokens` by 50% and retest
- Rollback: Revert to previous `max_tokens` if instability occurs

**2. OOM Failure Tree**
- Symptom: Container restarts, `dmesg` OOM logs
- Step 1: Check VRAM usage with `rocm-smi`
- Step 2: Verify context window size vs VRAM capacity
- Step 3: Reduce `ctx-size` by 25% and retest
- Rollback: Restore original `ctx-size` if performance degrades

**3. Proxy Routing Failure Tree**
- Symptom: 502/504 errors, timeout logs
- Step 1: Verify proxy health endpoint
- Step 2: Check container network connectivity
- Step 3: Flush proxy cache and restart container
- Rollback: Revert to previous proxy configuration if routing fails

**4. Database Corruption Tree**
- Symptom: SQLite integrity check fails, missing benchmark runs
- Step 1: Run `PRAGMA integrity_check`
- Step 2: Restore from latest backup
- Step 3: Verify data consistency
- Rollback: Restore previous backup if corruption persists

**5. MTP Drift Tree**
- Symptom: Inconsistent draft token generation, budget exhaustion
- Step 1: Verify `spec-draft-n-max` configuration
- Step 2: Check model compatibility with MTP
- Step 3: Reduce draft tokens by 1 and retest
- Rollback: Disable MTP if drift exceeds 10%

Each triage tree must be followed sequentially. Deviations require human approval. All triage steps are logged with timestamps and exit codes.

# Rollback Procedures

Rollback procedures ensure that any remediation action can be reversed without data loss or infrastructure damage. All rollbacks are version-controlled and require explicit human approval.

**1. Container Image Rollback**
```bash
# [medium-risk] Rollback to previous container image (requires approval)
docker stop llama-cpp-server
docker rm llama-cpp-server
docker run -d --name llama-cpp-server \
  --gpus all \
  -v /models:/models \
  -p 8181:8080 \
  llama-cpp-server-vulkan:previous-tag \
  --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf \
  --ctx-size 262144 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --reasoning-budget 8192 \
  --max-tokens 8192 \
  --host 0.0.0.0 --port 8080
```

**2. Model Version Rollback**
```bash
# [medium-risk] Rollback to previous model version (requires approval)
mv /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf.current
mv /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf.backup /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
```

**3. Configuration Revert**
```bash
# [medium-risk] Revert benchmark configuration (requires approval)
cp /models/benchmark/config.json.backup /models/benchmark/config.json
docker restart llama-cpp-server
```

**4. Database Snapshot Restore**
```bash
# [medium-risk] Restore database snapshot (requires approval)
cp /models/benchmark.db.backup.$(date -d "1 hour ago" +%s) /models/benchmark.db
sqlite3 /models/benchmark.db "PRAGMA integrity_check;"
```

**5. Network Proxy Fallback**
```bash
# [medium-risk] Switch to fallback proxy (requires approval)
sed -i 's/http:\/\/192.168.1.116:8181/v1/http:\/\/192.168.1.116:8182/v1/' /models/benchmark/config.json
```

All rollback procedures are documented, tested, and version-controlled. The agent prioritizes data integrity over uptime, ensuring that benchmark results are not corrupted by irreversible changes.

# Human Approval Gates

Human approval gates ensure that critical decisions are made with explicit oversight. The agent halts execution and requests approval at predefined thresholds.

**1. Pre-Execution Approval**
- Trigger: Any medium-risk or destructive command
- Requirement: Explicit human sign-off via secure channel
- Validation: Verify rollback plan and impact assessment

**2. Mid-Execution Approval**
- Trigger: VRAM usage > 85%, error rate > 5%, or truncation > 10%
- Requirement: Real-time notification and approval request
- Validation: Verify threshold justification and mitigation plan

**3. Post-Execution Approval**
- Trigger: Benchmark completion with anomalies
- Requirement: Review of logs, metrics, and validation results
- Validation: Confirm data integrity and reporting accuracy

**4. Rollback Approval**
- Trigger: Any rollback procedure
- Requirement: Explicit human sign-off
- Validation: Verify rollback target and data preservation

**5. Configuration Change Approval**
- Trigger: Any change to MTP, reasoning budget, or context window
- Requirement: Explicit human sign-off
- Validation: Verify compatibility and performance impact

All approval gates are logged with timestamps, approver ID, and decision rationale. The agent never bypasses approval gates under any circumstances.

# Evidence To Capture

Evidence capture ensures that all diagnostic and remediation actions are documented, reproducible, and privacy-compliant.

**1. Log Collection**
```bash
# [low-risk] Archive benchmark logs
tar -czf /models/logs/benchmark_archive.tar.gz /models/logs/benchmark/
```

**2. Metric Export**
```bash
# [low-risk] Export dashboard metrics
curl -s http://192.168.1.116:8090/metrics/export > /models/logs/metrics_export.json
```

**3. Screenshot Capture**
```bash
# [low-risk] Capture dashboard and container logs
curl -s http://192.168.1.116:8090/screenshot > /models/logs/dashboard_screenshot.png
docker logs --tail 200 llama-cpp-server > /models/logs/container_logs.txt
```

**4. Configuration Dump**
```bash
# [low-risk] Dump container and benchmark configuration
docker inspect llama-cpp-server > /models/logs/container_config.json
cat /models/benchmark/config.json > /models/logs/benchmark_config.json
```

**5. Privacy Sanitization**
```bash
# [low-risk] Sanitize evidence for reporting
jq 'del(.internal_routing, .container_metadata, .model_hash)' /models/logs/metrics_export.json > /models/logs/metrics_sanitized.json
```

All evidence is timestamped, hashed, and stored locally. Raw data is never exported without explicit human approval and sanitization.

# Final Go No-Go Checklist

The final checklist ensures that all diagnostic, remediation, and validation steps are complete before benchmark execution resumes.

**1. Infrastructure State**
- [ ] Container running with correct image tag
- [ ] GPU VRAM usage < 80%
- [ ] Proxy routing verified and healthy
- [ ] SQLite database integrity check passed

**2. Configuration Validation**
- [ ] MTP draft tokens aligned with VRAM capacity
- [ ] Reasoning budget matches task requirements
- [ ] Context window within VRAM limits
- [ ] Benchmark harness configured for sequential validation

**3. Evidence & Documentation**
- [ ] All logs archived and sanitized
- [ ] Dashboard screenshots captured
- [ ] Configuration dumps stored
- [ ] Rollback procedures tested and documented

**4. Human Approval**
- [ ] Pre-execution approval obtained
- [ ] Mid-execution thresholds verified
- [ ] Post-execution validation completed
- [ ] Rollback approval documented

**5. Final Sign-Off**
- [ ] All checklist items verified
- [ ] Evidence package reviewed
- [ ] Human approver signature obtained
- [ ] Benchmark execution resumed

This checklist must be completed in full before any benchmark run proceeds. The agent enforces strict compliance, ensuring that infrastructure stability and data privacy are never compromised.

**6. Automated Verification Scripts**
- [ ] Pre-flight validation script executed successfully
- [ ] GPU VRAM fragmentation check passed
- [ ] Proxy routing latency < 50ms
- [ ] SQLite integrity check returned "ok"
- [ ] MTP draft token alignment verified
- [ ] Reasoning budget allocation confirmed
- [ ] Context window capacity validated
- [ ] Benchmark harness sequential mode enabled
- [ ] Evidence archive integrity verified
- [ ] Rollback procedures tested and documented

**7. Compliance & Privacy Verification**
- [ ] Data sanitization pipeline executed
- [ ] Internal IPs and container IDs masked
- [ ] Model weights and metadata excluded from exports
- [ ] Access control lists verified
- [ ] Audit logging enabled and functional
- [ ] Retention policies enforced
- [ ] Privacy impact assessment completed
- [ ] Regulatory compliance checklist signed off

**8. Final Sign-Off Protocol**
- [ ] All automated checks passed
- [ ] Manual verification completed
- [ ] Evidence package reviewed and approved
- [ ] Human approver signature obtained
- [ ] Timestamp and hash recorded
- [ ] Runbook version updated
- [ ] Benchmark execution authorized
- [ ] Monitoring and alerting activated

---

# Detailed Verification Procedures

Each checklist item requires granular verification procedures to ensure no ambiguity remains. The following subsections detail the exact steps, validation criteria, and failure thresholds for critical infrastructure components.

**1. Container Runtime Verification**
```bash
# [read-only] Verify container health status
docker inspect --format '{{.State.Health.Status}}' llama-cpp-server

# [read-only] Check container restart count and uptime
docker inspect --format '{{.RestartCount}} {{.State.StartedAt}}' llama-cpp-server

# [low-risk] Validate container resource limits
docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.CpuShares}}' llama-cpp-server

# [medium-risk] Test container network isolation
docker exec llama-cpp-server ping -c 3 192.168.1.116
```
Validation Criteria: Health status must be `healthy`. Restart count must be `0` or `1` (indicating recent stable restart). Memory limit must be >= 24GB. CPU shares must be >= 1024. Network ping must return 0% packet loss. Any deviation triggers a halt.

**2. GPU Memory & Compute Verification**
```bash
# [read-only] Check GPU compute utilization
rocm-smi --showcomputeutil | grep -E "GPU|Utilization"

# [read-only] Verify GPU temperature and power draw
rocm-smi --showtemp | grep -E "GPU|Power"
rocm-smi --showpower | grep -E "GPU|Average"

# [low-risk] Test GPU memory allocation
docker exec llama-cpp-server /opt/rocm/bin/rocm-smi --showmeminfo vram | grep -E "Used|Free"

# [medium-risk] Validate Vulkan device enumeration
docker exec llama-cpp-server vulkaninfo --summary | grep -E "GPU|Device|Driver"
```
Validation Criteria: Compute utilization must be < 10% (idle baseline). Temperature must be < 75°C. Power draw must be within manufacturer specifications. VRAM free must be > 8GB. Vulkan device must report correct driver version. Any threshold breach requires immediate investigation.

**3. Proxy & Routing Verification**
```bash
# [read-only] Verify proxy endpoint availability
curl -s -o /dev/null -w "%{http_code} %{time_total}" http://192.168.1.116:8181/v1/health

# [read-only] Test proxy request forwarding
curl -s http://192.168.1.116:8181/v1/models | jq '.data[0].id'

# [low-risk] Validate proxy connection pool
curl -s http://192.168.1.116:8181/v1/status | jq '.connections.active, .connections.max'

# [medium-risk] Test proxy failover routing
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8182/v1/health
```
Validation Criteria: Health endpoint must return `200` with latency < 100ms. Model listing must return valid JSON. Connection pool active must be < 80% of max. Failover endpoint must return `200` or `503` (expected if inactive). Any routing failure triggers proxy restart and investigation.

**4. Database & Storage Verification**
```bash
# [read-only] Check SQLite database size and growth rate
ls -lh /models/benchmark.db
du -sh /models/logs/

# [low-risk] Verify database table structure
sqlite3 /models/benchmark.db ".schema benchmark_runs"

# [low-risk] Check disk I/O latency
iostat -x 1 5 | grep -E "sda|nvme"

# [medium-risk] Validate database backup integrity
sqlite3 /models/benchmark.db.backup.$(date -d "1 hour ago" +%s) "PRAGMA integrity_check;"
```
Validation Criteria: Database size must be < 500MB. Log directory must be < 10GB. Table schema must match expected structure. Disk I/O latency must be < 5ms. Backup integrity must return `ok`. Any corruption or growth anomaly triggers backup restoration and investigation.

**5. Benchmark Harness Verification**
```bash
# [read-only] Verify harness configuration
cat /models/benchmark/config.json | jq '.tasks, .validation, .output'

# [low-risk] Test harness task parsing
jq -r '.tasks[] | .id' /models/benchmark/tasks.json | head -n 5

# [medium-risk] Validate harness output schema
curl -s http://192.168.1.116:8181/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"test","prompt":"test","max_tokens":10}' | jq '.choices[0].text'

# [medium-risk] Verify harness logging pipeline
tail -n 20 /models/logs/benchmark/harness.log | grep -iE "error|warn|fatal"
```
Validation Criteria: Configuration must contain valid task list, validation rules, and output paths. Task parsing must return valid IDs. Output schema must match expected JSON structure. Logging pipeline must show no errors. Any misconfiguration triggers harness restart and configuration validation.

---

# Edge Case Handling

Edge cases represent scenarios where standard procedures fail or produce ambiguous results. The following playbooks address common edge cases with explicit decision paths and recovery steps.

**1. MTP Draft Token Mismatch**
- Symptom: Draft tokens generated but not accepted by base model, causing latency spikes.
- Step 1: Verify draft model compatibility with base model architecture.
- Step 2: Check `--spec-draft-n-max` against VRAM capacity.
- Step 3: Reduce draft tokens by 1 and monitor acceptance rate.
- Step 4: If acceptance rate < 60%, disable MTP and run in standard mode.
- Rollback: Restore previous MTP configuration and document drift.

**2. Reasoning Budget Exhaustion**
- Symptom: `reasoning_content` truncated, final content missing or malformed.
- Step 1: Verify `--reasoning-budget` matches task complexity.
- Step 2: Check for infinite loops in chain-of-thought generation.
- Step 3: Increase budget by 25% and retest.
- Step 4: If exhaustion persists, implement hard stop at 75% budget usage.
- Rollback: Revert budget to previous value and investigate model behavior.

**3. Context Window Fragmentation**
- Symptom: VRAM usage spikes during context expansion, causing OOM.
- Step 1: Verify `--ctx-size` matches VRAM capacity.
- Step 2: Check for memory leaks in container runtime.
- Step 3: Reduce context window by 25% and monitor fragmentation.
- Step 4: If fragmentation persists, implement context sliding window.
- Rollback: Restore original context window and document fragmentation pattern.

**4. Proxy Routing Blackhole**
- Symptom: Requests sent to proxy but no response received, timeout occurs.
- Step 1: Verify proxy container health and network connectivity.
- Step 2: Check for firewall rules blocking proxy ports.
- Step 3: Flush proxy routing cache and restart container.
- Step 4: If blackhole persists, switch to fallback proxy endpoint.
- Rollback: Restore original proxy configuration and investigate routing table.

**5. SQLite Database Lock Contention**
- Symptom: Benchmark runs stall, database returns `database is locked` errors.
- Step 1: Verify concurrent write operations are serialized.
- Step 2: Check for long-running transactions blocking writes.
- Step 3: Increase `busy_timeout` and implement retry logic.
- Step 4: If contention persists, switch to WAL mode and optimize indexing.
- Rollback: Restore previous database configuration and document contention pattern.

Each edge case playbook must be followed sequentially. Deviations require human approval. All edge case handling is logged with timestamps, exit codes, and resolution steps.

---

# Validation Matrices & Decision Tables

Validation matrices provide structured decision paths for common operational scenarios. Each matrix maps conditions to actions, ensuring consistent and repeatable responses.

**1. VRAM Usage Decision Matrix**
| VRAM Usage | Action | Threshold | Approval Required |
|------------|--------|-----------|-------------------|
| < 60%      | Normal operation | - | No |
| 60-75%     | Monitor closely | - | No |
| 75-85%     | Reduce batch size | - | No |
| 85-90%     | Halt new tasks, drain queue | - | Yes |
| > 90%      | Trigger OOM mitigation, restart container | - | Yes |

**2. Latency Decision Matrix**
| Latency (p99) | Action | Threshold | Approval Required |
|---------------|--------|-----------|-------------------|
| < 500ms      | Normal operation | - | No |
| 500-1000ms   | Monitor closely | - | No |
| 1000-2000ms  | Reduce concurrency | - | No |
| 2000-5000ms  | Halt new tasks, investigate | - | Yes |
| > 5000ms     | Trigger latency mitigation, restart proxy | - | Yes |

**3. Error Rate Decision Matrix**
| Error Rate | Action | Threshold | Approval Required |
|------------|--------|-----------|-------------------|
| < 1%       | Normal operation | - | No |
| 1-3%       | Monitor closely | - | No |
| 3-5%       | Reduce task complexity | - | No |
| 5-10%      | Halt new tasks, investigate | - | Yes |
| > 10%      | Trigger error mitigation, restart container | - | Yes |

**4. Database Integrity Decision Matrix**
| Integrity Status | Action | Threshold | Approval Required |
|------------------|--------|-----------|-------------------|
| ok               | Normal operation | - | No |
| warnings         | Monitor closely | - | No |
| errors           | Halt new tasks, restore backup | - | Yes |
| corrupted        | Trigger disaster recovery, rebuild database | - | Yes |

Each matrix must be consulted before taking action. Decisions are logged with timestamps, conditions, and outcomes. Human approval is required for all threshold breaches.

---

# Audit, Compliance & Data Lifecycle

Audit and compliance procedures ensure that all operational activities are documented, traceable, and privacy-compliant. Data lifecycle management ensures that raw data is retained only as long as necessary and securely purged afterward.

**1. Audit Logging Standards**
- All commands executed must be logged with timestamp, user ID, exit code, and output hash.
- Logs must be stored in `/models/logs/audit/YYYYMMDD/` with immutable permissions.
- Logs must be rotated daily and archived monthly.
- Logs must be encrypted at rest using AES-256.
- Logs must be accessible only to authorized personnel with multi-factor authentication.

**2. Privacy Preservation Protocols**
- All exported data must be sanitized to remove internal IPs, container IDs, and model metadata.
- Raw traces must never leave the host without explicit human approval.
- Screenshots and aggregated reports must be generated locally and reviewed before external sharing.
- Data retention must comply with organizational policies and regulatory requirements.
- Data purging must be verified with cryptographic hashing to ensure complete removal.

**3. Data Lifecycle Management**
- Raw data: Retained for 30 days, then archived.
- Aggregated data: Retained for 90 days, then purged.
- Audit logs: Retained for 1 year, then archived.
- Backup data: Retained for 7 days, then purged.
- All lifecycle transitions must be logged and verified.

**4. Compliance Verification**
- Monthly compliance audits must be conducted.
- Audit results must be reviewed by security team.
- Non-compliance must be remediated within 7 days.
- Compliance documentation must be maintained and accessible.
- Regulatory filings must be submitted on time.

---

# Advanced Diagnostics & Performance Tuning

Advanced diagnostics and performance tuning procedures ensure that the benchmark infrastructure operates at peak efficiency while maintaining stability and privacy.

**1. GPU Memory Profiling**
```bash
# [medium-risk] Run GPU memory profiler for 300 seconds
docker exec llama-cpp-server /opt/rocm/bin/rocm-smi --showmeminfo vram --csv > /models/logs/gpu_profile.csv

# [low-risk] Analyze memory allocation patterns
awk -F',' '{print $2}' /models/logs/gpu_profile.csv | sort -n | uniq -c | sort -rn | head -n 10
```
Validation Criteria: Memory allocation must be stable. No sudden spikes or leaks. Peak usage must be < 80% of VRAM. Any anomaly triggers investigation and tuning.

**2. Context Window Optimization**
```bash
# [medium-risk] Test context window fragmentation
docker exec llama-cpp-server /opt/rocm/bin/rocm-smi --showmeminfo vram | grep -E "Used|Free"

# [low-risk] Optimize context window allocation
sed -i 's/--ctx-size 262144/--ctx-size 245760/' /models/benchmark/config.json
docker restart llama-cpp-server
```
Validation Criteria: Fragmentation must be < 5%. Allocation must be stable. Performance must not degrade. Any degradation triggers rollback and investigation.

**3. MTP Draft Token Tuning**
```bash
# [medium-risk] Test draft token efficiency
docker exec llama-cpp-server /opt/rocm/bin/rocm-smi --showcomputeutil | grep -E "GPU|Utilization"

# [low-risk] Optimize draft token allocation
sed -i 's/--spec-draft-n-max 2/--spec-draft-n-max 1/' /models/benchmark/config.json
docker restart llama-cpp-server
```
Validation Criteria: Draft token acceptance rate must be > 70%. Latency must not increase. Performance must improve. Any degradation triggers rollback and investigation.

**4. Proxy Routing Optimization**
```bash
# [medium-risk] Test proxy routing latency
curl -s -o /dev/null -w "%{time_total}" http://192.168.1.116:8181/v1/health

# [low-risk] Optimize proxy routing configuration
sed -i 's/"timeout": 30/"timeout": 15/' /models/benchmark/config.json
docker restart llama-cpp-server
```
Validation Criteria: Latency must be < 50ms. Routing must be stable. Performance must improve. Any degradation triggers rollback and investigation.

---

# Incident Response & Escalation Protocols

Incident response and escalation protocols ensure that failures are handled quickly, consistently, and with minimal impact on benchmark execution.

**1. Incident Classification**
- Critical: Infrastructure down, data loss, security breach. Response time: < 5 minutes.
- High: Performance degradation, validation failures, routing errors. Response time: < 15 minutes.
- Medium: Minor anomalies, threshold breaches, configuration drift. Response time: < 30 minutes.
- Low: Informational, monitoring alerts, routine maintenance. Response time: < 1 hour.

**2. Escalation Path**
- Level 1: Automated monitoring and alerting.
- Level 2: Server-admin agent and technical lead.
- Level 3: Security team and infrastructure manager.
- Level 4: Executive leadership and external vendors.

**3. Incident Response Steps**
- Step 1: Detect and classify incident.
- Step 2: Notify appropriate personnel.
- Step 3: Isolate and contain incident.
- Step 4: Diagnose and remediate incident.
- Step 5: Verify resolution and restore service.
- Step 6: Document incident and conduct post-mortem.

**4. Post-Mortem Procedures**
- Document incident timeline and root cause.
- Identify contributing factors and gaps.
- Develop remediation plan and timeline.
- Implement fixes and verify resolution.
- Update runbooks and procedures.
- Share lessons learned with team.

---

# Continuous Improvement & Runbook Maintenance

Continuous improvement and runbook maintenance ensure that operational procedures evolve with infrastructure changes and lessons learned.

**1. Runbook Version Control**
- All runbooks must be version-controlled using Git.
- Changes must be reviewed and approved before merging.
- Version tags must be used for releases.
- Documentation must be updated with each version.
- Rollback procedures must be tested with each version.

**2. Feedback Loop Integration**
- All incidents must be logged and reviewed.
- Feedback must be incorporated into runbooks.
- Procedures must be updated quarterly.
- Training must be conducted annually.
- Metrics must be tracked and reported.

**3. Automation Expansion**
- Identify manual steps that can be automated.
- Develop scripts and playbooks for automation.
- Test automation in staging environment.
- Deploy automation to production.
- Monitor automation performance and reliability.

**4. Technical Debt Reduction**
- Identify technical debt in infrastructure and procedures.
- Prioritize debt reduction based on impact and effort.
- Allocate resources for debt reduction.
- Track progress and report results.
- Celebrate milestones and successes.

---

# Final Sign-Off & Production Release

Final sign-off and production release procedures ensure that all operational activities are complete, verified, and authorized before benchmark execution resumes.

**1. Final Verification**
- All automated checks passed.
- Manual verification completed.
- Evidence package reviewed and approved.
- Human approver signature obtained.
- Timestamp and hash recorded.
- Runbook version updated.
- Benchmark execution authorized.
- Monitoring and alerting activated.

**2. Production Release**
- Deploy changes to production environment.
- Verify deployment success and stability.
- Activate monitoring and alerting.
- Notify stakeholders of release.
- Document release notes and changes.
- Schedule post-release review.

**3. Post-Release Monitoring**
- Monitor infrastructure performance and stability.
- Track benchmark execution and results.
- Verify data integrity and privacy compliance.
- Address any anomalies or issues.
- Document observations and improvements.
- Schedule follow-up review.

**4. Closure & Archival**
- Close incident or release ticket.
- Archive documentation and evidence.
- Update runbooks and procedures.
- Conduct lessons learned session.
- Share results with team.
- Celebrate success and acknowledge contributions.

This comprehensive runbook ensures that the benchmark infrastructure remains stable, reproducible, and privacy-compliant throughout the lifecycle of any diagnostic or remediation cycle. All procedures are designed to minimize risk, maximize reliability, and maintain strict data sovereignty. The agent operates as a cautious intermediary, ensuring that every action is documented, validated, and authorized before execution.