# Operating Principles

## Core Philosophy
As a cautious server-admin agent operating within a private home-lab benchmark harness, every action is governed by the principle of **preservation before modification**. The infrastructure is designed for high-fidelity AI model evaluation, not production traffic, but the data integrity, model state, and hardware stability must be treated with enterprise-grade discipline. The following principles dictate all diagnostic, remediation, and validation workflows:

1. **Isolate Variables First**: Never change multiple configuration parameters simultaneously. Benchmark anomalies (e.g., `finish_reason=length` truncating final content) often stem from the interaction between context window limits, reasoning budgets, MTP draft token limits, and container resource constraints. Each variable must be tested in isolation.
2. **State Before Change**: Capture the current system state, configuration files, container environment, and database snapshots before applying any fix. This enables deterministic rollback and precise post-remediation comparison.
3. **Log Everything, Verify Everything**: Every command execution, configuration adjustment, and benchmark run must be logged with timestamps, exit codes, and output hashes. Verification steps must confirm that the fix resolves the symptom without introducing regression.
4. **Privacy-First Data Handling**: The SQLite database at `/models` contains private benchmark raw data. All inspection, backup, and reporting workflows must operate locally. No data leaves the host unless explicitly masked and approved. Screenshots and reports are generated locally and stored in isolated, permission-restricted directories.
5. **Incremental Validation**: Long-output tests and MTP-heavy workloads must never be executed in bulk until individual task validation passes. Bulk execution amplifies failures and obscures root causes.
6. **Risk-Scoped Command Execution**: Every shell command is classified by risk level. Destructive commands are strictly prohibited without explicit human approval gates. Medium-risk commands require state verification before and after execution. Low-risk and read-only commands form the baseline diagnostic toolkit.

## Risk Classification Framework
Commands in this runbook are explicitly tagged to enforce operational discipline:
- `[read-only]`: Inspects state, reads logs, queries metrics, or parses files. Zero side effects.
- `[low-risk]`: Creates backups, adjusts non-critical config values, restarts non-essential services, or captures screenshots. Reversible within seconds.
- `[medium-risk]`: Modifies container runtime parameters, adjusts GPU memory limits, changes proxy routing, or alters benchmark parameters. Requires state verification and rollback readiness.
- `[destructive]`: Deletes files, resets databases, forces container recreation, or reboots the host. Strictly prohibited without human approval gates.

## Communication & Escalation Protocol
- **Automated Logging**: All diagnostic steps are logged to `/var/log/benchmark-agent/runbook.log` with structured JSON entries containing `timestamp`, `action`, `risk_level`, `command`, `exit_code`, and `status`.
- **Human Approval Gates**: Triggered automatically when medium-risk or destructive actions are queued, when benchmark results deviate >15% from baseline, or when GPU VRAM utilization exceeds 90% sustained.
- **Evidence Archiving**: Screenshots, logs, and database exports are compressed into timestamped archives stored at `/models/.evidence/`. Raw data is never transmitted externally.

---

# Preflight Checks

Before initiating any benchmark run or diagnostic sequence, the host environment must be verified against a strict baseline. Preflight checks ensure that the container, model, GPU, network, and storage subsystems are in a known-good state.

## 1. Container & Runtime Verification
The `llama-cpp-server-vulkan:working-20260613` container must be running with the correct image tag, resource limits, and environment variables.

```bash
# Verify container status and image tag
docker inspect llama-cpp-server --format='{{.State.Status}} {{.Config.Image}}' [read-only]

# Check container resource limits (CPU, memory, GPU)
docker inspect llama-cpp-server --format='{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}' [read-only]

# Verify GPU passthrough configuration
docker inspect llama-cpp-server --format='{{range .HostConfig.DeviceRequests}}{{.Driver}}{{end}}' [read-only]
```

**Validation Criteria**:
- Status must be `running`.
- Image must exactly match `llama-cpp-server-vulkan:working-20260613`.
- Memory limit should be ≥16GB to accommodate 32GB-class VRAM offloading and context caching.
- GPU device requests must reference the AMD GPU correctly.

## 2. Model & Configuration Verification
The active model `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` must be present, intact, and paired with the correct server arguments.

```bash
# Verify model file integrity and size
ls -lh /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf [read-only]

# Verify model file hash (compare against known baseline)
sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf [read-only]

# Check server configuration file (if mounted)
cat /etc/llama-cpp/server.conf 2>/dev/null || echo "Config not mounted" [read-only]
```

**Validation Criteria**:
- File size should match expected dimensions (~20-22GB for 35B Q4_K_M/Q5_K_M variants).
- SHA256 hash must match the baseline recorded during initial deployment.
- Configuration file (if used) must contain `--ctx-size 262144`, `--spec-type draft-mtp`, `--spec-draft-n-max 2`, and `--reasoning-budget 8192`.

## 3. Host Resource & Storage Verification
The host must have sufficient CPU, RAM, disk I/O capacity, and network connectivity to support the container and benchmark harness.

```bash
# Check available RAM and swap
free -h [read-only]

# Check disk space and mount options
df -h /models [read-only]

# Verify disk I/O scheduler and queue depth
cat /sys/block/sda/queue/scheduler [read-only]

# Check network connectivity to proxy and dashboard
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1 [read-only]
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8090 [read-only]
```

**Validation Criteria**:
- Available RAM ≥24GB (system + container).
- `/models` partition ≥50GB free.
- HTTP status codes for proxy and dashboard must be `200` or `301/302`.
- Disk I/O scheduler should be `none` or `mq-deadline` for low-latency benchmarking.

## 4. Benchmark Harness & Task Queue Verification
Ensure the benchmark runner is configured to respect the long-output validation rule and does not execute bulk tasks prematurely.

```bash
# Verify benchmark runner configuration
cat /opt/benchmark/config.yaml [read-only]

# Check task queue status
curl -s http://192.168.1.116:8181/v1/benchmark/queue/status [read-only]
```

**Validation Criteria**:
- `validate_each_task: true` must be set.
- `bulk_execution: false` must be enforced.
- Queue status should show `idle` or `paused` before preflight completion.

---

# Safe Inspection Commands

Safe inspection commands form the diagnostic foundation. They gather state without modifying system configuration, container runtime, or benchmark data. All commands in this section are classified as `[read-only]`.

## 1. Container & Process Inspection
Monitor the llama.cpp server process, its arguments, and internal logs for anomalies.

```bash
# View container logs (last 500 lines)
docker logs llama-cpp-server --tail 500 [read-only]

# Inspect running processes inside container
docker exec llama-cpp-server ps aux [read-only]

# Check environment variables passed to container
docker exec llama-cpp-server env [read-only]

# Verify Vulkan driver availability inside container
docker exec llama-cpp-server ls /dev/dri/ [read-only]
```

**Validation Criteria**:
- Logs should show successful model loading, context initialization, and MTP draft token allocation.
- No repeated `CUDA/Vulkan error` or `out of memory` messages.
- Environment variables must include `LLAMA_VULKAN=1`, `ROCR_VISIBLE_DEVICES=0`, and correct proxy URLs.

## 2. Log Parsing & Pattern Matching
Extract specific diagnostic patterns from logs to identify recurring issues.

```bash
# Search for finish_reason anomalies
docker logs llama-cpp-server 2>&1 | grep -i "finish_reason" [read-only]

# Search for reasoning budget exhaustion
docker logs llama-cpp-server 2>&1 | grep -i "reasoning_budget" [read-only]

# Search for MTP draft token warnings
docker logs llama-cpp-server 2>&1 | grep -i "draft-mtp\|spec-draft" [read-only]

# Extract token usage statistics
docker logs llama-cpp-server 2>&1 | grep -i "tokens\|prompt\|completion" [read-only]
```

**Validation Criteria**:
- `finish_reason=length` should only appear when `max_tokens` is explicitly exceeded.
- Reasoning budget warnings should not trigger prematurely.
- MTP draft token allocation should match `--spec-draft-n-max 2`.

## 3. Network & Proxy Inspection
Verify that the Flight Recorder proxy is correctly routing requests and logging benchmark data.

```bash
# Check proxy listening ports
ss -tlnp | grep 8181 [read-only]

# Verify proxy configuration file
cat /etc/flight-recorder/proxy.conf [read-only]

# Test proxy health endpoint
curl -s http://192.168.1.116:8181/v1/health [read-only]

# Check proxy log rotation status
ls -lh /var/log/flight-recorder/ [read-only]
```

**Validation Criteria**:
- Proxy must be listening on `0.0.0.0:8181`.
- Health endpoint must return `{"status":"ok"}`.
- Log files must be rotating correctly to prevent disk exhaustion.

## 4. Database Integrity Inspection
Verify the SQLite database at `/models` without modifying it.

```bash
# Check database file size and permissions
ls -lh /models/benchmark_data.db [read-only]

# Verify SQLite integrity
sqlite3 /models/benchmark_data.db "PRAGMA integrity_check;" [read-only]

# Check database table schema
sqlite3 /models/benchmark_data.db ".schema" [read-only]

# Count recent benchmark entries
sqlite3 /models/benchmark_data.db "SELECT COUNT(*) FROM benchmark_runs WHERE timestamp > datetime('now', '-1 hour');" [read-only]
```

**Validation Criteria**:
- Integrity check must return `ok`.
- Schema must match expected benchmark schema (no unexpected columns or missing tables).
- Recent entry count should align with expected benchmark frequency.

---

# Reasoning Budget Verification

The `--reasoning-budget 8192` parameter controls how many tokens the model allocates to internal reasoning before generating final output. Observations indicate that `finish_reason=length` truncates final content when the budget is exhausted or misaligned with `max_tokens`. This section provides a systematic verification and adjustment protocol.

## 1. Budget Consumption Analysis
Determine how the reasoning budget is being consumed during benchmark runs.

```bash
# Extract reasoning token counts from logs
docker logs llama-cpp-server 2>&1 | grep -i "reasoning_tokens" [read-only]

# Extract completion token counts
docker logs llama-cpp-server 2>&1 | grep -i "completion_tokens" [read-only]

# Calculate budget utilization ratio
docker logs llama-cpp-server 2>&1 | awk '/reasoning_tokens/{r=$NF} /completion_tokens/{c=$NF} END{print "Budget Utilization:", r/c*100 "%"}' [read-only]
```

**Validation Criteria**:
- Reasoning tokens should not exceed 8192.
- Completion tokens should remain within `max_tokens` limits.
- Budget utilization ratio should be <95% to allow margin for final content generation.

## 2. Dynamic `max_tokens` Adjustment
If `finish_reason=length` appears before final content, increase `max_tokens` to accommodate reasoning + completion tokens.

```bash
# Backup current server configuration
cp /etc/llama-cpp/server.conf /etc/llama-cpp/server.conf.bak [low-risk]

# Update max_tokens in configuration
sed -i 's/--max-tokens [0-9]*/--max-tokens 16384/' /etc/llama-cpp/server.conf [medium-risk]

# Restart container to apply changes
docker restart llama-cpp-server [medium-risk]

# Verify new configuration
docker exec llama-cpp-server cat /proc/1/cmdline | tr '\0' ' ' [read-only]
```

**Validation Criteria**:
- Container must restart successfully.
- New `max_tokens` value must be reflected in process arguments.
- Benchmark runs should complete without premature truncation.

## 3. Reasoning vs. Completion Token Separation
Ensure the benchmark harness correctly parses `reasoning_content` and `content` fields.

```bash
# Inspect benchmark response structure
curl -s http://192.168.1.116:8181/v1/benchmark/response/latest | jq '.reasoning_content, .content' [read-only]

# Verify field separation in database
sqlite3 /models/benchmark_data.db "SELECT reasoning_content, content FROM benchmark_runs ORDER BY timestamp DESC LIMIT 1;" [read-only]
```

**Validation Criteria**:
- `reasoning_content` and `content` must be non-empty and correctly separated.
- No overlapping or truncated JSON fields.
- Database storage must preserve both fields intact.

## 4. Budget Exhaustion Fallback
If reasoning budget is consistently exhausted, implement a graceful fallback mechanism.

```bash
# Create fallback script
cat > /opt/benchmark/fallback_reasoning.sh << 'EOF'
#!/bin/bash
# Gracefully reduce reasoning budget if utilization exceeds 90%
BUDGET=8192
UTIL=$(docker logs llama-cpp-server 2>&1 | grep -i "reasoning_tokens" | awk '{print $NF}')
if [ "$UTIL" -gt $((BUDGET * 90 / 100)) ]; then
  sed -i "s/--reasoning-budget [0-9]*/--reasoning-budget $((BUDGET * 80 / 100))/" /etc/llama-cpp/server.conf
  docker restart llama-cpp-server
  echo "Reasoning budget reduced to $((BUDGET * 80 / 100))"
fi
EOF
chmod +x /opt/benchmark/fallback_reasoning.sh [low-risk]
```

**Validation Criteria**:
- Script must execute without errors.
- Budget reduction must be logged and applied.
- Benchmark performance must remain within acceptable thresholds.

---

# Long Output Validation

Long-output tests are highly sensitive to context window limits, MTP draft token allocation, and reasoning budget exhaustion. Bulk execution without validation risks cascading failures and corrupted benchmark data. This section establishes a strict validation protocol.

## 1. Individual Task Validation Workflow
Each long-output task must be validated in isolation before bulk execution is permitted.

```bash
# Extract task list from benchmark config
jq -r '.tasks[] | select(.type == "long_output") | .id' /opt/benchmark/config.yaml [read-only]

# Run single task validation
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/run \
  -H "Content-Type: application/json" \
  -d '{"task_id": "long_output_001", "validate_each": true}' [read-only]

# Check validation result
curl -s http://192.168.1.116:8181/v1/benchmark/validation/long_output_001 [read-only]
```

**Validation Criteria**:
- Task must complete with `finish_reason: stop` (not `length`).
- Output length must match expected token range.
- Validation result must return `{"status": "passed"}`.

## 2. Context Window & MTP Overlap Analysis
Long outputs often exceed the effective context window when combined with MTP draft tokens and reasoning content.

```bash
# Calculate effective context usage
docker logs llama-cpp-server 2>&1 | grep -i "ctx_used\|ctx_size" [read-only]

# Verify MTP draft token allocation
docker logs llama-cpp-server 2>&1 | grep -i "draft_tokens\|spec-draft-n-max" [read-only]

# Check context fragmentation warnings
docker logs llama-cpp-server 2>&1 | grep -i "fragmentation\|kv_cache" [read-only]
```

**Validation Criteria**:
- `ctx_used` must be < `ctx_size` (262144).
- MTP draft tokens must not exceed `--spec-draft-n-max 2`.
- No fragmentation warnings should appear.

## 3. Pagination & Streaming Validation
For tasks exceeding 16K tokens, implement pagination or streaming validation to prevent memory exhaustion.

```bash
# Enable streaming mode for long outputs
sed -i 's/--stream false/--stream true/' /etc/llama-cpp/server.conf [medium-risk]

# Restart container
docker restart llama-cpp-server [medium-risk]

# Verify streaming configuration
docker exec llama-cpp-server cat /proc/1/cmdline | tr '\0' ' ' | grep -o "stream [a-z]*" [read-only]
```

**Validation Criteria**:
- Streaming must be enabled.
- Benchmark harness must handle chunked responses correctly.
- No memory exhaustion or OOM errors during streaming.

## 4. Bulk Execution Gate
Bulk execution is only permitted after 100% individual task validation passes.

```bash
# Check validation gate status
curl -s http://192.168.1.116:8181/v1/benchmark/validation/gate [read-only]

# If gate passes, enable bulk execution
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/config \
  -H "Content-Type: application/json" \
  -d '{"bulk_execution": true}' [medium-risk]
```

**Validation Criteria**:
- Gate must return `{"status": "open"}`.
- Bulk execution must be enabled in configuration.
- Benchmark harness must log bulk run start/end timestamps.

---

# GPU And VRAM Checks

The AMD Radeon AI PRO R9700 (32GB class VRAM) requires careful monitoring to prevent VRAM fragmentation, OOM errors, and MTP draft token allocation failures. ROCm-based diagnostics are essential.

## 1. GPU Hardware & Driver Verification
Ensure the AMD GPU is correctly detected and ROCm drivers are functioning.

```bash
# Verify GPU detection
rocm-smi --showallinfo [read-only]

# Check ROCm driver version
cat /opt/rocm/version [read-only]

# Verify Vulkan support for AMD GPU
vulkaninfo --summary [read-only]

# Check GPU memory allocation
rocm-smi --showmeminfo vram [read-only]
```

**Validation Criteria**:
- GPU must be listed in `rocm-smi` output.
- ROCm version must match container expectations.
- Vulkan support must be enabled.
- VRAM allocation must show available memory ≥20GB.

## 2. Container GPU Passthrough Verification
Ensure the container correctly accesses the GPU and VRAM.

```bash
# Verify GPU device mapping in container
docker exec llama-cpp-server ls -l /dev/dri/ [read-only]

# Check GPU memory usage inside container
docker exec llama-cpp-server rocm-smi --showmeminfo vram [read-only]

# Verify Vulkan device selection
docker exec llama-cpp-server cat /proc/1/cmdline | tr '\0' ' ' | grep -o "vulkan_device [0-9]*" [read-only]
```

**Validation Criteria**:
- `/dev/dri/card0` and `/dev/dri/renderD128` must be present.
- Container must report correct VRAM usage.
- Vulkan device must match host GPU index.

## 3. VRAM Fragmentation & MTP Overhead
MTP draft tokens and long outputs can fragment VRAM, causing allocation failures.

```bash
# Monitor VRAM allocation patterns
rocm-smi --showmeminfo vram --json [read-only]

# Check for VRAM fragmentation warnings
dmesg | grep -i "amdgpu\|vram\|fragmentation" [read-only]

# Verify MTP draft token VRAM allocation
docker logs llama-cpp-server 2>&1 | grep -i "draft_vram\|mtp_memory" [read-only]
```

**Validation Criteria**:
- VRAM allocation must be contiguous.
- No fragmentation warnings in `dmesg`.
- MTP draft token VRAM allocation must match `--spec-draft-n-max 2`.

## 4. GPU Utilization & Thermal Monitoring
Sustained high utilization can trigger thermal throttling, affecting benchmark performance.

```bash
# Check GPU utilization
rocm-smi --showgpuutil [read-only]

# Check GPU temperature
rocm-smi --showtemp [read-only]

# Monitor GPU power draw
rocm-smi --showpower [read-only]
```

**Validation Criteria**:
- Utilization should be >70% during active benchmarking.
- Temperature must be <85°C.
- Power draw must be within TDP limits.

---

# Proxy And Database Checks

The Flight Recorder proxy (8181) and SQLite database (/models) are critical for benchmark data collection and reporting. Proxy routing, rate limiting, and database integrity must be verified regularly.

## 1. Proxy Health & Routing Verification
Ensure the proxy correctly routes requests to the llama.cpp server and logs benchmark data.

```bash
# Check proxy service status
systemctl status flight-recorder-proxy [read-only]

# Verify proxy routing configuration
cat /etc/flight-recorder/routing.conf [read-only]

# Test proxy request forwarding
curl -s -X POST http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced", "prompt": "test", "max_tokens": 10}' [read-only]

# Check proxy log for forwarding errors
journalctl -u flight-recorder-proxy --since "1 hour ago" | grep -i "error\|timeout\|502" [read-only]
```

**Validation Criteria**:
- Proxy service must be active.
- Routing configuration must point to `192.168.1.116:8080` (llama.cpp).
- Test request must return valid JSON response.
- No forwarding errors in logs.

## 2. Rate Limiting & Concurrency Control
Prevent benchmark harness from overwhelming the proxy or GPU.

```bash
# Check rate limit configuration
cat /etc/flight-recorder/rate-limit.conf [read-only]

# Verify current rate limit status
curl -s http://192.168.1.116:8181/v1/limits/status [read-only]

# Adjust rate limit if necessary
sed -i 's/max_requests_per_minute [0-9]*/max_requests_per_minute 60/' /etc/flight-recorder/rate-limit.conf [medium-risk]
systemctl restart flight-recorder-proxy [medium-risk]
```

**Validation Criteria**:
- Rate limit must be ≤60 requests/minute for long-output tasks.
- Status endpoint must reflect current limits.
- Proxy must restart successfully.

## 3. Database Integrity & Privacy
SQLite database must remain local, private, and intact.

```bash
# Verify database file permissions
ls -l /models/benchmark_data.db [read-only]

# Check database size growth
du -sh /models/benchmark_data.db [read-only]

# Verify database encryption (if enabled)
sqlite3 /models/benchmark_data.db "PRAGMA cipher_provider_version;" [read-only]

# Backup database before any modification
cp /models/benchmark_data.db /models/benchmark_data.db.bak.$(date +%Y%m%d%H%M%S) [low-risk]
```

**Validation Criteria**:
- Permissions must be `600` (owner read/write only).
- Database size must grow predictably.
- Encryption must be enabled (if configured).
- Backup must be created successfully.

## 4. Data Masking & Reporting Preparation
Prepare data for reporting without exposing private raw content.

```bash
# Create masked export for reporting
sqlite3 /models/benchmark_data.db ".mode csv" ".headers on" ".output /models/.evidence/masked_report.csv" "SELECT id, timestamp, model, task_type, finish_reason, tokens_generated FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours');" [read-only]

# Verify masked export
head -n 5 /models/.evidence/masked_report.csv [read-only]
```

**Validation Criteria**:
- Export must exclude `reasoning_content` and `content` fields.
- CSV must contain only metadata and aggregated metrics.
- File must be stored in `/models/.evidence/`.

---

# Dashboard Checks

ServerTop (8090) provides real-time metrics for CPU, RAM, GPU, and network utilization. Dashboard checks ensure metrics are accurate, historical data is preserved, and alert thresholds are appropriate.

## 1. Dashboard Service Verification
Ensure ServerTop is running and accessible.

```bash
# Check ServerTop service status
systemctl status servertop [read-only]

# Verify dashboard listening port
ss -tlnp | grep 8090 [read-only]

# Test dashboard health endpoint
curl -s http://192.168.1.116:8090/health [read-only]

# Check dashboard configuration
cat /etc/servertop/config.yaml [read-only]
```

**Validation Criteria**:
- Service must be active.
- Port 8090 must be listening.
- Health endpoint must return `{"status":"ok"}`.
- Configuration must match expected metrics sources.

## 2. Metrics Collection & Accuracy
Verify that metrics are being collected accurately from the host and container.

```bash
# Check metrics collection logs
journalctl -u servertop --since "1 hour ago" | grep -i "collect\|metric\|error" [read-only]

# Verify GPU metrics collection
curl -s http://192.168.1.116:8090/api/metrics/gpu | jq '.vram_used, .vram_total' [read-only]

# Verify container metrics collection
curl -s http://192.168.1.116:8090/api/metrics/container/llama-cpp-server | jq '.cpu_percent, .memory_percent' [read-only]
```

**Validation Criteria**:
- No collection errors in logs.
- GPU metrics must match `rocm-smi` output.
- Container metrics must reflect actual resource usage.

## 3. Alert Thresholds & Historical Data
Ensure alert thresholds are appropriate for benchmark workloads and historical data is preserved.

```bash
# Check alert configuration
cat /etc/servertop/alerts.yaml [read-only]

# Verify historical data retention
ls -lh /var/lib/servertop/data/ [read-only]

# Adjust alert thresholds if necessary
sed -i 's/vram_utilization_threshold: 85/vram_utilization_threshold: 90/' /etc/servertop/alerts.yaml [medium-risk]
systemctl restart servertop [medium-risk]
```

**Validation Criteria**:
- Alert thresholds must be appropriate for 32GB VRAM.
- Historical data must be retained for ≥7 days.
- Threshold adjustments must be logged and applied.

## 4. Dashboard Screenshot & Reporting
Capture dashboard state for evidence and reporting.

```bash
# Capture dashboard screenshot
curl -s http://192.168.1.116:8090/screenshot > /models/.evidence/dashboard_$(date +%Y%m%d%H%M%S).png [low-risk]

# Verify screenshot
file /models/.evidence/dashboard_*.png [read-only]
```

**Validation Criteria**:
- Screenshot must be saved successfully.
- File must be a valid PNG image.
- Screenshot must be stored in `/models/.evidence/`.

---

# Failure Triage Trees

When benchmark failures occur, systematic triage trees isolate the root cause. Each tree follows a decision path from symptom to resolution, with command classifications and validation steps.

## Tree 1: `finish_reason=length` Truncation
```
Symptom: Benchmark responses return finish_reason=length before final content appears.
├─ Check 1: Is max_tokens too low?
│  ├─ Command: docker logs llama-cpp-server 2>&1 | grep -i "max_tokens" [read-only]
│  ├─ If yes: Increase max_tokens to 16384 [medium-risk]
│  └─ Validate: Run single task, verify finish_reason: stop [read-only]
├─ Check 2: Is reasoning budget exhausted?
│  ├─ Command: docker logs llama-cpp-server 2>&1 | grep -i "reasoning_tokens" [read-only]
│  ├─ If yes: Reduce reasoning budget to 6553 [medium-risk]
│  └─ Validate: Verify reasoning_content + content length < max_tokens [read-only]
├─ Check 3: Is context window exceeded?
│  ├─ Command: docker logs llama-cpp-server 2>&1 | grep -i "ctx_used" [read-only]
│  ├─ If yes: Reduce context size to 131072 [medium-risk]
│  └─ Validate: Verify ctx_used < ctx_size [read-only]
└─ Resolution: Apply fix, restart container, re-run benchmark [medium-risk]
```

## Tree 2: GPU VRAM Exhaustion / OOM
```
Symptom: Benchmark fails with OOM or GPU allocation errors.
├─ Check 1: Is VRAM fragmented?
│  ├─ Command: rocm-smi --showmeminfo vram --json [read-only]
│  ├─ If yes: Restart container to clear fragmentation [medium-risk]
│  └─ Validate: Verify VRAM allocation is contiguous [read-only]
├─ Check 2: Is MTP draft token allocation too high?
│  ├─ Command: docker logs llama-cpp-server 2>&1 | grep -i "draft_tokens" [read-only]
│  ├─ If yes: Reduce --spec-draft-n-max to 1 [medium-risk]
│  └─ Validate: Verify draft tokens < 2 [read-only]
├─ Check 3: Is container memory limit too low?
│  ├─ Command: docker inspect llama-cpp-server --format='{{.HostConfig.Memory}}' [read-only]
│  ├─ If yes: Increase memory limit to 20GB [medium-risk]
│  └─ Validate: Verify container restarts successfully [read-only]
└─ Resolution: Apply fix, monitor VRAM, re-run benchmark [medium-risk]
```

## Tree 3: Proxy Routing / Database Corruption
```
Symptom: Benchmark data not recorded or proxy returns 502.
├─ Check 1: Is proxy service running?
│  ├─ Command: systemctl status flight-recorder-proxy [read-only]
│  ├─ If no: systemctl start flight-recorder-proxy [low-risk]
│  └─ Validate: Verify proxy health endpoint [read-only]
├─ Check 2: Is database integrity compromised?
│  ├─ Command: sqlite3 /models/benchmark_data.db "PRAGMA integrity_check;" [read-only]
│  ├─ If failed: Restore from backup [low-risk]
│  └─ Validate: Verify integrity check returns ok [read-only]
├─ Check 3: Is rate limiting too aggressive?
│  ├─ Command: curl -s http://192.168.1.116:8181/v1/limits/status [read-only]
│  ├─ If yes: Increase rate limit [medium-risk]
│  └─ Validate: Verify benchmark runs complete [read-only]
└─ Resolution: Apply fix, verify data recording, re-run benchmark [medium-risk]
```

## Tree 4: Dashboard Metrics Inaccuracy
```
Symptom: ServerTop shows incorrect GPU/CPU metrics.
├─ Check 1: Is metrics collection service running?
│  ├─ Command: systemctl status servertop [read-only]
│  ├─ If no: systemctl start servertop [low-risk]
│  └─ Validate: Verify health endpoint [read-only]
├─ Check 2: Are ROCm drivers functioning?
│  ├─ Command: rocm-smi --showallinfo [read-only]
│  ├─ If no: Check dmesg for driver errors [read-only]
│  └─ Validate: Verify GPU detection [read-only]
├─ Check 3: Is configuration misaligned?
│  ├─ Command: cat /etc/servertop/config.yaml [read-only]
│  ├─ If yes: Update config to match host [medium-risk]
│  └─ Validate: Verify metrics accuracy [read-only]
└─ Resolution: Apply fix, restart service, verify metrics [medium-risk]
```

---

# Rollback Procedures

Rollback procedures ensure that any fix or configuration change can be reversed quickly and deterministically. All rollbacks preserve state and minimize downtime.

## 1. Container Rollback
Revert container image, configuration, or environment variables to a known-good state.

```bash
# Backup current container state
docker commit llama-cpp-server llama-cpp-server:pre-fix-$(date +%Y%m%d%H%M%S) [low-risk]

# Revert configuration file
cp /etc/llama-cpp/server.conf.bak /etc/llama-cpp/server.conf [low-risk]

# Restart container with original image
docker stop llama-cpp-server [low-risk]
docker rm llama-cpp-server [low-risk]
docker run -d --name llama-cpp-server --gpus all -v /models:/models -p 8080:8080 llama-cpp-server-vulkan:working-20260613 [medium-risk]

# Verify rollback
docker inspect llama-cpp-server --format='{{.State.Status}} {{.Config.Image}}' [read-only]
```

**Validation Criteria**:
- Container must restart successfully.
- Original image tag must be restored.
- Configuration must match backup.

## 2. Database Rollback
Restore SQLite database from backup if corruption or data loss occurs.

```bash
# Stop proxy to prevent writes
systemctl stop flight-recorder-proxy [low-risk]

# Restore database from backup
cp /models/benchmark_data.db.bak.$(ls -t /models/benchmark_data.db.bak.* | head -1) /models/benchmark_data.db [low-risk]

# Verify database integrity
sqlite3 /models/benchmark_data.db "PRAGMA integrity_check;" [read-only]

# Restart proxy
systemctl start flight-recorder-proxy [low-risk]
```

**Validation Criteria**:
- Database must pass integrity check.
- Proxy must restart successfully.
- Benchmark data recording must resume.

## 3. Configuration Rollback
Revert any configuration changes made during triage.

```bash
# List configuration backups
ls -lh /etc/llama-cpp/*.bak /etc/flight-recorder/*.bak /etc/servertop/*.bak [read-only]

# Restore specific configuration
cp /etc/llama-cpp/server.conf.bak /etc/llama-cpp/server.conf [low-risk]

# Restart affected services
docker restart llama-cpp-server [medium-risk]
systemctl restart flight-recorder-proxy [low-risk]
systemctl restart servertop [low-risk]
```

**Validation Criteria**:
- All services must restart successfully.
- Configuration must match backup.
- Benchmark harness must resume normal operation.

## 4. State Preservation & Audit
Ensure all rollback actions are logged and auditable.

```bash
# Log rollback actions
echo "$(date +%Y-%m-%dT%H:%M:%S) ROLLBACK: $(cat /var/log/benchmark-agent/runbook.log | tail -n 10)" >> /var/log/benchmark-agent/rollback_audit.log [low-risk]

# Verify audit log
cat /var/log/benchmark-agent/rollback_audit.log [read-only]
```

**Validation Criteria**:
- Audit log must contain rollback timestamps and actions.
- Log must be accessible for review.

---

# Human Approval Gates

Human approval gates prevent automated systems from executing high-risk actions without explicit authorization. Gates are triggered automatically based on predefined conditions.

## 1. Trigger Conditions
- Medium-risk or destructive commands are queued.
- Benchmark results deviate >15% from baseline.
- GPU VRAM utilization exceeds 90% sustained.
- Database integrity check fails.
- Proxy routing errors exceed 5% of requests.

## 2. Approval Workflow
```bash
# Check approval gate status
curl -s http://192.168.1.116:8181/v1/benchmark/approval/gate [read-only]

# If gate is closed, request approval
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/approval/request \
  -H "Content-Type: application/json" \
  -d '{"action": "medium-risk", "reason": "Increase max_tokens to 16384", "risk_level": "medium"}' [read-only]

# Wait for approval
sleep 30
curl -s http://192.168.1.116:8181/v1/benchmark/approval/status [read-only]
```

**Validation Criteria**:
- Gate must be open before proceeding.
- Approval request must be logged.
- Approval status must be `approved` before execution.

## 3. Escalation Protocol
If approval is not granted within 15 minutes, escalate to human administrator.

```bash
# Send escalation notification
echo "Benchmark agent requires human approval for medium-risk action. Please review /models/.evidence/escalation_$(date +%Y%m%d%H%M%S).log" [low-risk]

# Log escalation
echo "$(date +%Y-%m-%dT%H:%M:%S) ESCALATION: Awaiting human approval" >> /var/log/benchmark-agent/escalation.log [low-risk]
```

**Validation Criteria**:
- Escalation log must be updated.
- Notification must be delivered to administrator.
- Agent must pause execution until approval is granted.

---

# Evidence To Capture

Comprehensive evidence capture ensures that all diagnostic steps, fixes, and benchmark results are documented for review, reporting, and future troubleshooting.

## 1. Log & Output Capture
```bash
# Capture container logs
docker logs llama-cpp-server > /models/.evidence/container_logs_$(date +%Y%m%d%H%M%S).log [low-risk]

# Capture benchmark responses
curl -s http://192.168.1.116:8181/v1/benchmark/response/latest > /models/.evidence/benchmark_response_$(date +%Y%m%d%H%M%S).json [low-risk]

# Capture proxy logs
journalctl -u flight-recorder-proxy > /models/.evidence/proxy_logs_$(date +%Y%m%d%H%M%S).log [low-risk]
```

**Validation Criteria**:
- Logs must be saved to `/models/.evidence/`.
- Files must be named with timestamps.
- Logs must be complete and uncorrupted.

## 2. Screenshot & Dashboard Capture
```bash
# Capture dashboard screenshot
curl -s http://192.168.1.116:8090/screenshot > /models/.evidence/dashboard_$(date +%Y%m%d%H%M%S).png [low-risk]

# Capture GPU metrics screenshot
rocm-smi --showallinfo --json > /models/.evidence/gpu_metrics_$(date +%Y%m%d%H%M%S).json [low-risk]
```

**Validation Criteria**:
- Screenshots must be valid PNG/JSON files.
- Files must be stored in `/models/.evidence/`.
- Screenshots must capture relevant metrics.

## 3. Database Export & Masking
```bash
# Export masked benchmark data
sqlite3 /models/benchmark_data.db ".mode csv" ".headers on" ".output /models/.evidence/masked_benchmark_$(date +%Y%m%d%H%M%S).csv" "SELECT id, timestamp, model, task_type, finish_reason, tokens_generated FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours');" [read-only]

# Verify export
head -n 5 /models/.evidence/masked_benchmark_*.csv [read-only]
```

**Validation Criteria**:
- Export must exclude private content fields.
- CSV must contain only metadata and aggregated metrics.
- File must be stored in `/models/.evidence/`.

## 4. Evidence Archive & Privacy
```bash
# Compress evidence archive
tar -czf /models/.evidence/archive_$(date +%Y%m%d%H%M%S).tar.gz /models/.evidence/ [low-risk]

# Set restrictive permissions
chmod 600 /models/.evidence/archive_*.tar.gz [low-risk]

# Verify archive
tar -tzf /models/.evidence/archive_*.tar.gz | head -n 10 [read-only]
```

**Validation Criteria**:
- Archive must be compressed successfully.
- Permissions must be `600`.
- Archive must contain all evidence files.

---

# Final Go No-Go Checklist

Before initiating any benchmark run or diagnostic sequence, complete this checklist. All items must pass before proceeding.

## Pre-Flight Checklist
- [ ] Container status: `running` with correct image tag `[read-only]`
- [ ] Model file integrity: SHA256 matches baseline `[read-only]`
- [ ] Host resources: RAM ≥24GB, disk ≥50GB free `[read-only]`
- [ ] Network connectivity: Proxy and dashboard return `200` `[read-only]`
- [ ] Benchmark harness: `validate_each_task: true`, `bulk_execution: false` `[read-only]`
- [ ] GPU detection: ROCm and Vulkan drivers functioning `[read-only]`
- [ ] Database integrity: `PRAGMA integrity_check` returns `ok` `[read-only]`
- [ ] Evidence directory: `/models/.evidence/` exists and is writable `[low-risk]`

## Post-Fix Validation Checklist
- [ ] Container restarts successfully with new configuration `[medium-risk]`
- [ ] Benchmark runs complete with `finish_reason: stop` `[read-only]`
- [ ] Reasoning budget utilization <95% `[read-only]`
- [ ] VRAM utilization <90% sustained `[read-only]`
- [ ] Proxy routing errors <5% `[read-only]`
- [ ] Database records benchmark data correctly `[read-only]`
- [ ] Dashboard metrics match host/container values `[read-only]`
- [ ] Evidence archive created and permissions set `[low-risk]`

## Human Approval Gate Checklist
- [ ] All medium-risk/destructive actions approved `[read-only]`
- [ ] Escalation protocol documented `[low-risk]`
- [ ] Rollback procedures verified `[low-risk]`
- [ ] Final sign-off recorded `[low-risk]`

**Final Status**: `GO` if all checks pass. `NO-GO` if any check fails. Document failures in `/models/.evidence/failure_report_$(date +%Y%m%d%H%M%S).md` and initiate triage trees.

### Advanced MTP & Speculative Decoding Tuning

The `--spec-type draft-mtp --spec-draft-n-max 2` configuration is highly sensitive to context window fragmentation and VRAM allocation. Mismatched draft token limits can cause speculative decoding failures, leading to `finish_reason=length` or severe latency spikes. Precise tuning ensures optimal throughput without compromising output integrity.

#### 1. Draft Token Allocation Verification
Verify that the draft model is correctly loaded and allocated within the VRAM budget. MTP draft tokens consume significant VRAM, and improper allocation can cause OOM errors or silent decoding failures.

```bash
# Check draft model VRAM allocation
docker logs llama-cpp-server 2>&1 | grep -i "draft_model\|speculative" [read-only]

# Verify draft token acceptance rate
docker logs llama-cpp-server 2>&1 | grep -i "accepted\|rejected" [read-only]

# Calculate speculative decoding efficiency
docker logs llama-cpp-server 2>&1 | awk '/accepted/{a=$NF} /rejected/{r=$NF} END{print "Speculative Efficiency:", a/(a+r)*100 "%"}' [read-only]
```

**Validation Criteria**:
- Draft model must load without VRAM errors.
- Acceptance rate should be >60% for optimal performance.
- Efficiency calculation must not return NaN or zero.
- No `speculative decoding failed` warnings in logs.

#### 2. Dynamic Draft Token Adjustment
If speculative efficiency drops below 50%, reduce `--spec-draft-n-max` to minimize VRAM pressure and improve stability. This is critical for long-output tasks where draft token accumulation can fragment the KV cache.

```bash
# Backup current configuration
cp /etc/llama-cpp/server.conf /etc/llama-cpp/server.conf.pre-mtp-tune [low-risk]

# Reduce draft token limit
sed -i 's/--spec-draft-n-max [0-9]*/--spec-draft-n-max 1/' /etc/llama-cpp/server.conf [medium-risk]

# Restart container
docker restart llama-cpp-server [medium-risk]

# Verify new draft token limit
docker exec llama-cpp-server cat /proc/1/cmdline | tr '\0' ' ' | grep -o "spec-draft-n-max [0-9]*" [read-only]
```

**Validation Criteria**:
- Container must restart successfully.
- Draft token limit must reflect the new value.
- Speculative efficiency should improve or stabilize.
- Benchmark runs should complete without `finish_reason=length` truncation.

#### 3. Context Window Fragmentation Mitigation
Long outputs and MTP draft tokens can fragment the KV cache, causing allocation failures. Enabling memory locking (`--mlock`) and disabling KV offload can mitigate fragmentation on systems with sufficient RAM.

```bash
# Check KV cache fragmentation status
docker logs llama-cpp-server 2>&1 | grep -i "kv_cache\|fragmentation" [read-only]

# Enable KV cache defragmentation (if supported)
sed -i 's/--no-kv-offload/--no-kv-offload --mlock/' /etc/llama-cpp/server.conf [medium-risk]

# Restart container
docker restart llama-cpp-server [medium-risk]

# Verify KV cache defragmentation
docker logs llama-cpp-server 2>&1 | grep -i "mlock\|kv_cache" [read-only]
```

**Validation Criteria**:
- KV cache defragmentation must be enabled.
- No fragmentation warnings in logs.
- Benchmark performance must remain stable.
- System RAM usage must not exceed physical limits.

### Vulkan & ROCm Driver Deep Dive

The AMD Radeon AI PRO R9700 relies on ROCm and Vulkan for optimal performance. Driver misconfigurations, version mismatches, or missing extensions can cause severe benchmark degradation, silent data corruption, or crashes. Deep driver verification ensures hardware-level stability.

#### 1. ROCm Environment Verification
Ensure ROCm environment variables are correctly set inside the container. Incorrect environment variables can cause the container to fall back to CPU execution or fail to initialize the GPU.

```bash
# Check ROCm environment variables
docker exec llama-cpp-server env | grep -i "rocm\|hip\|vulkan" [read-only]

# Verify ROCm device enumeration
docker exec llama-cpp-server rocm-smi --showdevicename [read-only]

# Check HIP runtime version
docker exec llama-cpp-server cat /opt/rocm/hip/version [read-only]
```

**Validation Criteria**:
- ROCm environment variables must be present and correct.
- Device enumeration must match host GPU.
- HIP runtime version must match container expectations.
- No `HIP error` or `ROCm error` messages in logs.

#### 2. Vulkan Driver & Extension Verification
Vulkan extensions are critical for llama.cpp performance on AMD hardware. Missing extensions can cause fallback to slower execution paths or complete failure.

```bash
# Check Vulkan extensions
docker exec llama-cpp-server vulkaninfo --summary | grep -i "extensions" [read-only]

# Verify Vulkan driver version
vulkaninfo --summary | grep -i "driver" [read-only]

# Check for Vulkan memory allocation errors
dmesg | grep -i "vulkan\|amdgpu" | tail -n 20 [read-only]
```

**Validation Criteria**:
- Required Vulkan extensions must be present.
- Driver version must be up-to-date.
- No Vulkan memory allocation errors in `dmesg`.
- `vulkaninfo` must report no errors.

#### 3. ROCm & Vulkan Synchronization
Ensure ROCm and Vulkan are synchronized to prevent race conditions. Synchronization issues can cause benchmark hangs or incorrect output.

```bash
# Check ROCm & Vulkan synchronization status
cat /sys/kernel/debug/dri/0/amdgpu_fence_info [read-only]

# Verify synchronization logs
docker logs llama-cpp-server 2>&1 | grep -i "sync\|fence" [read-only]
```

**Validation Criteria**:
- Synchronization status must be stable.
- No synchronization errors in logs.
- Fence info must show normal operation.

### Benchmark Harness & Task Queue Orchestration

The benchmark harness must be orchestrated to prevent task collisions, ensure proper sequencing, and handle long-output tasks correctly. Improper orchestration can lead to resource contention, data loss, and inaccurate benchmark results.

#### 1. Task Queue Status & Health
Monitor the benchmark task queue to prevent bottlenecks and ensure smooth execution.

```bash
# Check task queue status
curl -s http://192.168.1.116:8181/v1/benchmark/queue/status [read-only]

# Verify task queue depth
curl -s http://192.168.1.116:8181/v1/benchmark/queue/depth [read-only]

# Check task queue health
curl -s http://192.168.1.116:8181/v1/benchmark/queue/health [read-only]
```

**Validation Criteria**:
- Queue status must be `idle` or `processing`.
- Queue depth must be within expected limits.
- Queue health must be `healthy`.
- No `queue overflow` or `task timeout` warnings.

#### 2. Task Sequencing & Dependency Management
Ensure tasks are sequenced correctly to avoid resource contention. Long-output tasks should be sequenced after short-output tasks to prevent VRAM fragmentation.

```bash
# Verify task sequencing configuration
cat /opt/benchmark/sequencing.yaml [read-only]

# Check task dependencies
curl -s http://192.168.1.116:8181/v1/benchmark/dependencies/list [read-only]

# Resolve task dependencies
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/dependencies/resolve [read-only]
```

**Validation Criteria**:
- Sequencing configuration must be correct.
- Task dependencies must be resolved.
- No resource contention warnings.
- Long-output tasks must be sequenced appropriately.

#### 3. Long-Output Task Isolation
Isolate long-output tasks to prevent interference with short-output tasks. Isolation ensures that VRAM fragmentation and context window limits do not affect other benchmark runs.

```bash
# Create isolated task group for long outputs
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/task-group/create \
  -H "Content-Type: application/json" \
  -d '{"group_name": "long_output_isolated", "task_types": ["long_output"], "max_concurrent": 1}' [medium-risk]

# Verify isolated task group
curl -s http://192.168.1.116:8181/v1/benchmark/task-group/list [read-only]
```

**Validation Criteria**:
- Isolated task group must be created successfully.
- Task group must only contain long-output tasks.
- Max concurrent tasks must be 1.
- No interference with other task groups.

### Automated Remediation Scripts

Automated scripts can handle common benchmark failures quickly and deterministically. They reduce manual intervention, ensure consistency, and prevent human error during critical operations.

#### 1. Automated VRAM Clear Script
Clear VRAM fragmentation automatically when utilization exceeds 90%. This script monitors VRAM usage and restarts the container if fragmentation is detected.

```bash
cat > /opt/benchmark/remediate_vram.sh << 'EOF'
#!/bin/bash
VRAM_THRESHOLD=90
VRAM_USED=$(rocm-smi --showmeminfo vram --json | jq '.[0].vram_used_pct')
if [ "$VRAM_USED" -gt "$VRAM_THRESHOLD" ]; then
  docker restart llama-cpp-server
  echo "VRAM cleared automatically"
fi
EOF
chmod +x /opt/benchmark/remediate_vram.sh [low-risk]
```

**Validation Criteria**:
- Script must execute without errors.
- VRAM must be cleared automatically when threshold is exceeded.
- Container must restart successfully.
- No data loss during restart.

#### 2. Automated Log Rotation Script
Prevent disk exhaustion by rotating logs automatically. Log rotation ensures that disk space is preserved and logs remain accessible for diagnosis.

```bash
cat > /opt/benchmark/rotate_logs.sh << 'EOF'
#!/bin/bash
LOG_DIR="/var/log/benchmark-agent"
MAX_SIZE=100M
for log in "$LOG_DIR"/*.log; do
  if [ "$(du -b "$log" | cut -f1)" -gt "$MAX_SIZE" ]; then
    mv "$log" "$log.$(date +%Y%m%d%H%M%S)"
    touch "$log"
    echo "Log rotated: $log"
  fi
done
EOF
chmod +x /opt/benchmark/rotate_logs.sh [low-risk]
```

**Validation Criteria**:
- Script must rotate logs automatically.
- Logs must be compressed and archived.
- Disk space must be preserved.
- No log data loss.

### Post-Benchmark Data Sanitization & Reporting

After benchmark runs, data must be sanitized, aggregated, and reported without exposing private raw content. Sanitization ensures data privacy, while aggregation provides actionable insights for performance tuning.

#### 1. Data Sanitization
Remove private content fields from benchmark data before reporting. Sanitization is critical for maintaining privacy and compliance.

```bash
# Sanitize benchmark data
sqlite3 /models/benchmark_data.db "UPDATE benchmark_runs SET reasoning_content = NULL, content = NULL WHERE timestamp > datetime('now', '-24 hours');" [medium-risk]

# Verify sanitization
sqlite3 /models/benchmark_data.db "SELECT reasoning_content, content FROM benchmark_runs ORDER BY timestamp DESC LIMIT 1;" [read-only]
```

**Validation Criteria**:
- Private content fields must be nullified.
- Sanitization must be logged.
- Data must remain intact for aggregated metrics.
- No accidental data deletion.

#### 2. Aggregated Metrics Generation
Generate aggregated metrics for reporting. Aggregated metrics provide a high-level view of benchmark performance without exposing raw data.

```bash
# Generate aggregated metrics
sqlite3 /models/benchmark_data.db ".mode csv" ".headers on" ".output /models/.evidence/aggregated_metrics_$(date +%Y%m%d%H%M%S).csv" "SELECT model, task_type, AVG(tokens_generated) as avg_tokens, AVG(completion_time) as avg_time, COUNT(*) as total_runs FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours') GROUP BY model, task_type;" [read-only]

# Verify aggregated metrics
head -n 5 /models/.evidence/aggregated_metrics_*.csv [read-only]
```

**Validation Criteria**:
- Aggregated metrics must be generated correctly.
- CSV must contain only aggregated data.
- File must be stored in `/models/.evidence/`.
- No raw content fields in output.

#### 3. Reporting & Archival
Create a comprehensive report and archive it securely. Archival ensures that benchmark results are preserved for future reference and analysis.

```bash
# Create report
cat > /models/.evidence/report_$(date +%Y%m%d%H%M%S).md << EOF
# Benchmark Report $(date +%Y-%m-%d)
## Summary
- Total Runs: $(sqlite3 /models/benchmark_data.db "SELECT COUNT(*) FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours');")
- Average Tokens: $(sqlite3 /models/benchmark_data.db "SELECT AVG(tokens_generated) FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours');")
- Average Time: $(sqlite3 /models/benchmark_data.db "SELECT AVG(completion_time) FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours');")
## Metrics
$(cat /models/.evidence/aggregated_metrics_*.csv)
## Evidence
$(ls /models/.evidence/)
EOF

# Archive report
tar -czf /models/.evidence/final_archive_$(date +%Y%m%d%H%M%S).tar.gz /models/.evidence/ [low-risk]

# Set restrictive permissions
chmod 600 /models/.evidence/final_archive_*.tar.gz [low-risk]
```

**Validation Criteria**:
- Report must be generated correctly.
- Archive must be compressed successfully.
- Permissions must be `600`.
- Archive must contain all evidence files.

### Incident Response & Root Cause Analysis (RCA) Framework

When benchmark failures occur, a structured RCA framework ensures systematic diagnosis and resolution. The framework prevents ad-hoc troubleshooting, ensures consistency, and facilitates knowledge sharing.

#### 1. Incident Classification
Classify incidents by severity and impact. Classification determines the response priority and escalation path.
- **Critical**: Benchmark completely halted, data loss, hardware damage.
- **High**: Benchmark severely degraded, significant data corruption.
- **Medium**: Benchmark partially functional, minor data loss.
- **Low**: Benchmark operational, minor anomalies.

#### 2. RCA Process
1. **Identify**: Determine the exact symptom and affected components.
2. **Isolate**: Narrow down the root cause using triage trees.
3. **Resolve**: Apply fixes and validate results.
4. **Document**: Record the incident, root cause, and resolution in `/models/.evidence/rca_$(date +%Y%m%d%H%M%S).md`.
5. **Prevent**: Implement measures to prevent recurrence.

#### 3. RCA Documentation Template
```markdown
# Root Cause Analysis
## Incident Summary
- Timestamp: [YYYY-MM-DD HH:MM:SS]
- Severity: [Critical/High/Medium/Low]
- Affected Components: [GPU/Proxy/Database/Benchmark Harness]

## Root Cause
- [Detailed description of the root cause]

## Resolution
- [Steps taken to resolve the issue]

## Prevention
- [Measures to prevent recurrence]

## Evidence
- [Links to logs, screenshots, and database exports]
```

### Hardware & Power Management Optimization

Optimize hardware and power management to ensure stable benchmark performance. Power management settings can significantly impact GPU utilization, thermal throttling, and overall system stability.

#### 1. CPU & Memory Frequency Scaling
Ensure CPU and memory are operating at optimal frequencies. Frequency scaling affects benchmark latency and throughput.

```bash
# Check CPU frequency scaling
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor [read-only]

# Set CPU frequency scaling to performance
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor [medium-risk]

# Check memory frequency
dmidecode -t memory | grep -i "speed" [read-only]
```

**Validation Criteria**:
- CPU frequency scaling must be set to `performance`.
- Memory frequency must be optimal.
- No thermal throttling warnings.
- Benchmark latency must be minimized.

#### 2. Power Supply & Thermal Management
Ensure power supply and thermal management are optimized. Thermal throttling can cause severe benchmark degradation.

```bash
# Check power supply status
cat /sys/class/power_supply/AC/online [read-only]

# Check thermal zones
cat /sys/class/thermal/thermal_zone*/type [read-only]
cat /sys/class/thermal/thermal_zone*/temp [read-only]

# Adjust fan curves if necessary
cat /sys/class/hwmon/hwmon*/pwm1 [read-only]
```

**Validation Criteria**:
- Power supply must be online.
- Thermal zones must be within safe limits.
- Fan curves must be optimized.
- No thermal throttling during benchmark runs.

### Container Orchestration & Lifecycle Management

Manage container lifecycle to ensure stability and reproducibility. Container orchestration includes health monitoring, backup, and restore procedures.

#### 1. Container Health Monitoring
Monitor container health to detect anomalies early. Health monitoring ensures that issues are identified before they impact benchmark results.

```bash
# Check container health status
docker inspect --format='{{.State.Health.Status}}' llama-cpp-server [read-only]

# Check container resource usage
docker stats llama-cpp-server --no-stream [read-only]

# Check container logs for errors
docker logs llama-cpp-server --since 1h | grep -i "error\|fatal" [read-only]
```

**Validation Criteria**:
- Container health status must be `healthy`.
- Resource usage must be within limits.
- No errors in logs.
- No `OOM` or `crash` warnings.

#### 2. Container Backup & Restore
Backup and restore container state for disaster recovery. Backup ensures that container configuration and state are preserved.

```bash
# Backup container image
docker save llama-cpp-server-vulkan:working-20260613 > /models/.evidence/container_image_$(date +%Y%m%d%H%M%S).tar [low-risk]

# Restore container image
docker load < /models/.evidence/container_image_$(date +%Y%m%d%H%M%S).tar [low-risk]

# Verify container image
docker images | grep llama-cpp-server-vulkan [read-only]
```

**Validation Criteria**:
- Container image must be backed up successfully.
- Container image must be restored successfully.
- Image must match original tag.
- No data loss during backup/restore.

### Network & Latency Optimization

Optimize network and latency to ensure fast and reliable benchmark execution. Network optimization reduces request latency and improves overall benchmark throughput.

#### 1. Network Latency Monitoring
Monitor network latency to detect bottlenecks. Latency monitoring ensures that network issues are identified and resolved quickly.

```bash
# Check network latency to proxy
ping -c 10 192.168.1.116 | grep -i "rtt" [read-only]

# Check network latency to dashboard
ping -c 10 192.168.1.116 | grep -i "rtt" [read-only]

# Check network interface statistics
ip -s link [read-only]
```

**Validation Criteria**:
- Network latency must be <1ms.
- No packet loss.
- Network interface statistics must be normal.
- No `dropped` or `error` packets.

#### 2. Network Configuration Optimization
Optimize network configuration for low-latency benchmarking. Network optimization includes MTU adjustment and traffic shaping.

```bash
# Check network MTU
ip link show | grep -i "mtu" [read-only]

# Set network MTU to 9000 (jumbo frames) if supported
ip link set dev eth0 mtu 9000 [medium-risk]

# Verify network MTU
ip link show | grep -i "mtu" [read-only]
```

**Validation Criteria**:
- Network MTU must be set to 9000 if supported.
- No network errors.
- Latency must be optimized.
- Throughput must be maximized.

### Long-Term Maintenance & Model Updates

Maintain and update models and infrastructure for long-term stability. Maintenance includes model integrity verification, driver updates, and configuration reviews.

#### 1. Model Integrity Verification
Verify model integrity before and after updates. Integrity verification ensures that models are not corrupted and perform as expected.

```bash
# Verify model integrity
sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf [read-only]

# Compare with baseline
diff <(sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf) /models/.baseline/sha256sum.txt [read-only]
```

**Validation Criteria**:
- Model integrity must match baseline.
- No discrepancies.
- Model must load successfully.
- No `model load error` warnings.

#### 2. Infrastructure Updates
Update infrastructure components for security and performance. Updates include ROCm, Vulkan, and container runtime updates.

```bash
# Update ROCm drivers
apt update && apt upgrade rocm-hip-runtime -y [medium-risk]

# Update Vulkan drivers
apt update && apt upgrade vulkan-utils -y [medium-risk]

# Restart container
docker restart llama-cpp-server [medium-risk]

# Verify updates
rocm-smi --version [read-only]
vulkaninfo --version [read-only]
```

**Validation Criteria**:
- ROCm and Vulkan drivers must be updated.
- Container must restart successfully.
- Updates must be verified.
- No `driver mismatch` errors.

### Emergency Shutdown & Safe State Preservation

In case of critical failure, preserve state and shut down safely. Emergency shutdown procedures prevent data loss, hardware damage, and system instability.

#### 1. Emergency Shutdown Procedure
```bash
# Stop benchmark harness
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/shutdown [low-risk]

# Stop proxy
systemctl stop flight-recorder-proxy [low-risk]

# Stop container
docker stop llama-cpp-server [low-risk]

# Unmount /models if necessary
umount /models [medium-risk]

# Verify safe state
systemctl is-active flight-recorder-proxy [read-only]
docker ps -a | grep llama-cpp-server [read-only]
```

**Validation Criteria**:
- Benchmark harness must be stopped.
- Proxy must be stopped.
- Container must be stopped.
- `/models` must be unmounted if necessary.
- Safe state must be verified.

#### 2. State Preservation
Preserve state for post-mortem analysis. State preservation ensures that diagnostic data is available for troubleshooting.

```bash
# Capture final logs
docker logs llama-cpp-server > /models/.evidence/final_container_logs_$(date +%Y%m%d%H%M%S).log [low-risk]

# Capture final database state
cp /models/benchmark_data.db /models/.evidence/final_benchmark_data_$(date +%Y%m%d%H%M%S).db [low-risk]

# Archive state
tar -czf /models/.evidence/emergency_archive_$(date +%Y%m%d%H%M%S).tar.gz /models/.evidence/ [low-risk]

# Set restrictive permissions
chmod 600 /models/.evidence/emergency_archive_*.tar.gz [low-risk]
```

**Validation Criteria**:
- Logs must be captured.
- Database state must be preserved.
- Archive must be created successfully.
- Permissions must be `600`.

### Final Verification & Sign-Off

Complete final verification and sign-off before resuming normal operations. Verification ensures that all systems are operational and benchmark results are accurate.

#### 1. Final Verification
```bash
# Verify all services are running
systemctl is-active flight-recorder-proxy [read-only]
docker ps -a | grep llama-cpp-server [read-only]
curl -s http://192.168.1.116:8090/health [read-only]

# Verify benchmark harness is operational
curl -s http://192.168.1.116:8181/v1/benchmark/queue/status [read-only]

# Verify database integrity
sqlite3 /models/benchmark_data.db "PRAGMA integrity_check;" [read-only]
```

**Validation Criteria**:
- All services must be running.
- Benchmark harness must be operational.
- Database integrity must be verified.
- No `service down` or `integrity failed` warnings.

#### 2. Sign-Off
```bash
# Record sign-off
echo "$(date +%Y-%m-%dT%H:%M:%S) SIGN-OFF: All checks passed. Resuming normal operations." >> /var/log/benchmark-agent/signoff.log [low-risk]

# Verify sign-off
cat /var/log/benchmark-agent/signoff.log [read-only]
```

**Validation Criteria**:
- Sign-off must be recorded.
- Sign-off must be verified.
- Normal operations must resume.
- No `sign-off failed` errors.

### Conclusion

This runbook provides a comprehensive framework for diagnosing, fixing, validating, and rolling back benchmark infrastructure problems in a private home-lab AI server environment. By adhering to these principles, executing precise commands, and maintaining strict operational discipline, the agent ensures benchmark integrity, data privacy, and hardware stability. Continuous monitoring, automated remediation, and structured incident response guarantee long-term reliability and performance. The combination of proactive checks, reactive triage, and systematic rollback procedures creates a resilient infrastructure capable of handling complex benchmark workloads without compromising data security or system stability.