# Operating Principles

This runbook governs all diagnostic, validation, configuration, and rollback activities for the home-lab AI benchmark infrastructure. The environment operates under strict constraints: a single parallel slot, a fixed 32 GB VRAM ceiling, speculative decoding via MTP, a bounded reasoning budget, and a local-only SQLite database. Every procedure is designed to preserve system stability, protect private data, and ensure reproducible benchmark results.

1. **Caution-First Execution**: No configuration change, container restart, or database modification occurs without explicit pre-validation, documented rollback paths, and human approval gates. The system is treated as a production-grade benchmark harness despite its home-lab classification.
2. **Single-Slot Isolation**: With `parallel_slots=1`, the server cannot multiplex requests. All benchmark tasks must be serialized. Bulk execution is prohibited until per-task validation confirms stable token generation, reasoning budget compliance, and correct `finish_reason` behavior.
3. **Speculative Decoding Awareness**: MTP (`--spec-type draft-mtp --spec-draft-n-max 2`) accelerates generation but introduces draft-token interference. If the draft model diverges or the acceptance rate drops, the server may prematurely emit `finish_reason=length`. This runbook treats MTP as a performance multiplier, not a correctness guarantee.
4. **Reasoning Budget Enforcement**: `--reasoning-budget 8192` caps internal chain-of-thought tokens. Exceeding this budget without adjusting `max_tokens` causes truncation. The runbook enforces explicit budget verification before any long-output test.
5. **Data Privacy & Local-Only Storage**: The SQLite database resides at `/models`. No raw benchmark outputs, logs, or database exports leave the host. Screenshots and aggregated metrics may be captured for internal reporting, but raw data remains strictly local.
6. **Risk Classification Schema**:
   - `[READ-ONLY]`: Inspects state without modifying files, memory, or running processes.
   - `[LOW-RISK]`: Modifies volatile state (e.g., container restart, config reload) with immediate rollback capability.
   - `[MEDIUM-RISK]`: Alters persistent state (e.g., database schema, model cache, server args) requiring explicit backup before execution.
   - `[DESTRUCTIVE]`: Irreversible or high-impact operations (e.g., volume deletion, model overwrite, driver reinstall). Requires dual human approval and full system snapshot.
7. **Observability-First Diagnostics**: Every diagnostic step begins with log inspection, metric cross-referencing, and non-intrusive probing. Invasive commands are only executed after read-only checks confirm the failure domain.
8. **Incremental Validation**: Configuration changes are applied in isolation. Each change is validated against a controlled test prompt before proceeding to the next. Bulk benchmark runs are only authorized after all incremental validations pass.
9. **Rollback Readiness**: Every procedure includes a defined rollback trigger, a verified backup location, and a step-by-step restoration sequence. Rollback is preferred over extended troubleshooting when stability is compromised.
10. **Documentation & Evidence**: All actions, observations, and outcomes are logged locally. Screenshots, token counts, timing metrics, and log excerpts are captured per the Evidence To Capture section. No data is transmitted externally.

---

# Preflight Checks

Before initiating any benchmark sequence or configuration modification, the following preflight checks must be completed. These checks verify host reachability, container health, model integrity, GPU readiness, proxy routing, database accessibility, and dashboard responsiveness.

1. **Host & Network Reachability**
   - Verify the host is responsive and the local network interface is stable.
   - `[READ-ONLY]` `ping -c 4 192.168.1.116`
   - `[READ-ONLY]` `ip addr show | grep 192.168.1.116`
   - Confirm no firewall rules block ports 8090 (ServerTop) or 8181 (Flight Recorder proxy).

2. **Container Status & Version Verification**
   - Ensure the `llama-cpp-server-vulkan:working-20260613` container is running and matches the expected image tag.
   - `[READ-ONLY]` `docker ps --filter "name=llama-cpp-server" --format "{{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"`
   - `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{.Config.Image}}'`
   - Verify the container has not drifted to a newer or older tag. If drift is detected, halt and initiate rollback.

3. **Model File Integrity**
   - Confirm `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` exists, is readable, and matches expected size/hash.
   - `[READ-ONLY]` `ls -lh /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
   - `[READ-ONLY]` `sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
   - Compare against the known-good hash stored in `/models/.model_hashes`. Mismatch triggers immediate halt.

4. **VRAM & Vulkan Driver Readiness**
   - Verify the AMD Radeon AI PRO R9700 is recognized, Vulkan layers are loaded, and VRAM is available.
   - `[READ-ONLY]` `vulkaninfo --summary | grep -i "device name\|driver info\|memory heap"`
   - `[READ-ONLY]` `rocm-smi --showmeminfo vram` (or equivalent AMD monitoring tool)
   - Confirm VRAM free ≥ 12 GB before model load. Lower values indicate stale allocations or background processes.

5. **Proxy & Dashboard Connectivity**
   - Validate Flight Recorder proxy and ServerTop dashboard are responding.
   - `[READ-ONLY]` `curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8181/v1/health`
   - `[READ-ONLY]` `curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8090/status`
   - Both must return `200`. Non-200 responses trigger proxy/dashboard triage.

6. **SQLite Database Accessibility**
   - Ensure `/models/benchmark.db` (or equivalent) is readable and not locked.
   - `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
   - `[READ-ONLY]` `fuser /models/benchmark.db`
   - `integrity_check` must return `ok`. `fuser` must show no active writers.

7. **Disk Space & Temp Directory**
   - Verify sufficient space for logs, temporary outputs, and database journaling.
   - `[READ-ONLY]` `df -h /models /tmp /var/log`
   - Require ≥ 15 GB free on `/models` and ≥ 5 GB on `/tmp`.

8. **Environment Variable & Config Consistency**
   - Confirm server arguments match expected baseline.
   - `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{range .Config.Cmd}}{{.}} {{end}}'`
   - Verify `--context-size 262144`, `--parallel 1`, `--spec-type draft-mtp`, `--spec-draft-n-max 2`, `--reasoning-budget 8192` are present.

If any preflight check fails, halt execution, capture evidence, and proceed to Failure Triage Trees. Do not proceed to benchmark execution until all checks pass.

---

# Safe Inspection Commands

This section catalogs non-intrusive and low-impact commands for continuous monitoring and diagnostic probing. Commands are classified by risk and grouped by subsystem.

## Container & Process Inspection
- `[READ-ONLY]` `docker stats --no-stream llama-cpp-server`
- `[READ-ONLY]` `docker logs --tail 200 llama-cpp-server 2>&1 | grep -i "error\|warn\|oom\|vulkan\|mtp\|reasoning"`
- `[READ-ONLY]` `ps aux | grep llama.cpp`
- `[LOW-RISK]` `docker exec llama-cpp-server cat /proc/1/status`

## GPU & Vulkan Diagnostics
- `[READ-ONLY]` `vulkaninfo --device-index 0 --summary`
- `[READ-ONLY]` `rocm-smi --showtemp --showpower --showclocks`
- `[READ-ONLY]` `dmesg | grep -i "amdgpu\|vulkan\|gpu\|error\|reset"`
- `[LOW-RISK]` `vulkaninfo --device-index 0 --layer-properties`

## Proxy & Network Inspection
- `[READ-ONLY]` `curl -v http://192.168.1.116:8181/v1/models 2>&1 | head -20`
- `[READ-ONLY]` `ss -tulnp | grep -E "8090|8181"`
- `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | grep -E "llama\|vulkan\|mtp\|reasoning"`

## Database & Filesystem Inspection
- `[READ-ONLY]` `sqlite3 /models/benchmark.db ".tables"`
- `[READ-ONLY]` `sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM runs;"`
- `[READ-ONLY]` `ls -lR /models/ | head -50`
- `[LOW-RISK]` `sqlite3 /models/benchmark.db "PRAGMA journal_mode;"`

## Log & Metric Aggregation
- `[READ-ONLY]` `journalctl -u llama-cpp-server --since "1 hour ago" --no-pager | tail -100`
- `[READ-ONLY]` `grep -c "finish_reason=length" /var/log/llama-server.log`
- `[READ-ONLY]` `awk '/reasoning_budget/ {print}' /var/log/llama-server.log | tail -10`

All read-only commands may be executed continuously. Low-risk commands require a 5-second cooldown between executions to avoid log flooding or metric skew. Medium-risk and destructive commands are excluded from this section and governed by explicit approval gates.

---

# Reasoning Budget Verification

The `--reasoning-budget 8192` parameter caps internal chain-of-thought tokens. When combined with MTP speculative decoding and large context windows, budget exhaustion can cause premature truncation or `finish_reason=length` before final content is emitted. Verification ensures the budget is respected, correctly partitioned, and does not interfere with output completeness.

## 1. Baseline Configuration Audit
- Confirm the server is running with the exact reasoning budget flag.
- `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{range .Config.Cmd}}{{.}} {{end}}' | grep -oP '(?<=--reasoning-budget )\d+'`
- Expected output: `8192`. Any deviation requires immediate correction.

## 2. Controlled Prompt Testing
- Execute a deterministic prompt designed to trigger reasoning tokens without exceeding the budget.
- `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Explain the concept of speculative decoding in three steps. Keep it concise."}],"max_tokens":2048}' | jq '.choices[0].message.reasoning_content | length'`
- Measure `reasoning_content` length. Must be ≤ 8192. If closer to 8000, increase `max_tokens` or reduce prompt complexity.

## 3. Budget Exhaustion Simulation
- Intentionally push the reasoning budget to verify graceful handling.
- `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Perform a detailed step-by-step mathematical derivation of the quadratic formula, showing every intermediate algebraic manipulation."}],"max_tokens":4096}' | jq '{reasoning_len: (.choices[0].message.reasoning_content | length), content_len: (.choices[0].message.content | length), finish_reason: .choices[0].finish_reason}'`
- Expected behavior: `finish_reason` should be `stop` or `length` only after both reasoning and content are complete. If `length` appears early, the budget is misaligned with `max_tokens`.

## 4. MTP Interaction Validation
- MTP draft tokens can consume reasoning budget if not properly isolated. Verify draft acceptance rate and budget partitioning.
- `[READ-ONLY]` `docker logs llama-cpp-server 2>&1 | grep -i "mtp\|speculative\|draft\|acceptance" | tail -20`
- Look for `draft_tokens_accepted`, `draft_tokens_rejected`, and `reasoning_budget_remaining`. If `reasoning_budget_remaining` hits 0 before content generation, increase `--reasoning-budget` or reduce `--spec-draft-n-max`.

## 5. Dynamic Budget Adjustment Procedure
- If budget exhaustion is confirmed, adjust safely.
- `[MEDIUM-RISK]` `docker exec llama-cpp-server sed -i 's/--reasoning-budget 8192/--reasoning-budget 12288/' /app/server_args.conf`
- `[LOW-RISK]` `docker restart llama-cpp-server`
- Re-run controlled prompt test. Verify new budget is respected. Rollback if VRAM spikes or generation stalls.

## 6. Verification Thresholds
- Reasoning content length ≤ 80% of budget during normal operation.
- `finish_reason=length` must only occur after `content` field is populated.
- MTP acceptance rate ≥ 60%. Below 50% indicates draft interference; reduce `--spec-draft-n-max` to 1.
- VRAM usage must not exceed 28 GB during reasoning-heavy prompts.

All budget verification steps must be logged. Any deviation from thresholds triggers a halt and proceeds to Failure Triage Trees.

---

# Long Output Validation

The observation that `finish_reason=length` appeared before final content indicates a misalignment between `max_tokens`, reasoning budget, and MTP draft consumption. Long-output tests require strict per-task validation to prevent bulk truncation and ensure complete generation.

## 1. Incremental `max_tokens` Calibration
- Never assume a fixed `max_tokens` value works across all prompts. Calibrate per task.
- `[LOW-RISK]` `for tokens in 2048 4096 8192 16384; do echo "Testing max_tokens=$tokens"; curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf\",\"messages\":[{\"role\":\"user\",\"content\":\"Write a detailed technical explanation of Vulkan memory management.\"}],\"max_tokens\":$tokens}" | jq -r '.choices[0].finish_reason'; done`
- Record the smallest `max_tokens` that yields `finish_reason=stop` with complete content. Use this as the baseline for the task.

## 2. Per-Task Validation Protocol
- Before adding a task to the benchmark queue, validate it individually.
- Step 1: Submit prompt with calibrated `max_tokens`.
- Step 2: Verify `reasoning_content` and `content` fields are non-empty.
- Step 3: Confirm `finish_reason=stop`.
- Step 4: Check token counts against budget limits.
- Step 5: Log timing, VRAM peak, and MTP acceptance rate.
- Only after all steps pass is the task marked `VALIDATED` and eligible for queue inclusion.

## 3. Truncation Detection Script
- Automated detection of premature `finish_reason=length`.
- `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Generate a comprehensive system architecture document for a distributed AI inference cluster."}],"max_tokens":16384}' | jq -r 'if .choices[0].finish_reason == "length" and (.choices[0].message.content | length) < 500 then "TRUNCATION_DETECTED" else "COMPLETE" end'`
- If `TRUNCATION_DETECTED`, increase `max_tokens` by 25% and revalidate. Repeat until `COMPLETE`.

## 4. MTP Draft Interference Mitigation
- MTP can cause early termination if draft tokens consume the token budget. Validate draft isolation.
- `[READ-ONLY]` `docker logs llama-cpp-server 2>&1 | grep -E "draft_tokens|speculative|budget_exhausted" | tail -30`
- If `budget_exhausted` appears before content generation, reduce `--spec-draft-n-max` to 1 or disable MTP temporarily for validation.
- `[MEDIUM-RISK]` `docker exec llama-cpp-server sed -i 's/--spec-draft-n-max 2/--spec-draft-n-max 1/' /app/server_args.conf && docker restart llama-cpp-server`
- Re-run validation. Restore to 2 after confirmation.

## 5. Chunked Output Verification
- For extremely long outputs, verify streaming completeness.
- `[LOW-RISK]` `curl -N http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Write a 2000-word essay on the history of neural network architectures."}],"max_tokens":16384,"stream":true}' | grep -c '"finish_reason"'`
- Must return exactly 1. Multiple `finish_reason` events indicate stream fragmentation or server instability.

## 6. Validation Thresholds & Rollback Triggers
- `finish_reason=stop` required for all validated tasks.
- Content length ≥ 80% of expected output for the prompt category.
- VRAM peak ≤ 28 GB.
- MTP acceptance rate ≥ 55%.
- If any threshold is breached, halt queue, capture evidence, and proceed to rollback or configuration adjustment.

Long-output validation is mandatory. Bulk execution without per-task validation is strictly prohibited.

---

# GPU And VRAM Checks

The AMD Radeon AI PRO R9700 provides 32 GB VRAM. With a 35B parameter model, 262K context, MTP draft buffers, and reasoning budget allocations, VRAM management is critical. OOM conditions, Vulkan driver faults, or thermal throttling can silently corrupt benchmark results.

## 1. VRAM Allocation Baseline
- Calculate expected VRAM usage:
  - Model weights (Q4_K_M/Q5_K_M quantization): ~14-18 GB
  - KV Cache (262144 context, 1 slot): ~6-8 GB
  - MTP draft buffers: ~1-2 GB
  - Reasoning budget overhead: ~0.5-1 GB
  - Vulkan driver & OS overhead: ~2-3 GB
  - Total expected: ~24-29 GB. Leave ≥ 3 GB headroom.

## 2. Real-Time VRAM Monitoring
- `[READ-ONLY]` `rocm-smi --showmeminfo vram --json`
- `[READ-ONLY]` `vulkaninfo --device-index 0 --memory-properties`
- Cross-reference with ServerTop dashboard VRAM graph. Discrepancies > 10% indicate metric drift or driver caching issues.

## 3. Thermal & Power Validation
- `[READ-ONLY]` `rocm-smi --showtemp --showpower --showclocks`
- Thresholds:
  - Temperature: ≤ 85°C sustained. > 90°C triggers thermal throttling.
  - Power: ≤ 250W sustained. Spikes > 280W indicate unstable power delivery.
  - Clocks: Stable at boost frequencies. Droop > 15% indicates throttling.

## 4. Vulkan Driver & Layer Verification
- `[READ-ONLY]` `vulkaninfo --summary | grep -i "driver\|layer\|validation"`
- Ensure no validation layers are enabled during benchmarking (they add overhead).
- `[LOW-RISK]` `export VK_LOADER_DEBUG=error && docker restart llama-cpp-server`
- Verify clean startup without Vulkan warnings.

## 5. OOM & Allocation Failure Detection
- `[READ-ONLY]` `dmesg | grep -i "amdgpu\|vulkan\|oom\|allocation\|failed" | tail -20`
- `[READ-ONLY]` `docker logs llama-cpp-server 2>&1 | grep -i "vram\|oom\|allocation\|cuda\|vulkan error"`
- If OOM detected, reduce context size or disable MTP temporarily.
- `[MEDIUM-RISK]` `docker exec llama-cpp-server sed -i 's/--context-size 262144/--context-size 131072/' /app/server_args.conf && docker restart llama-cpp-server`
- Validate stability, then restore context size after VRAM headroom is confirmed.

## 6. VRAM Leak Prevention
- Long-running servers can accumulate Vulkan allocations. Periodically verify clean state.
- `[LOW-RISK]` `docker restart llama-cpp-server`
- `[READ-ONLY]` `rocm-smi --showmeminfo vram`
- Confirm VRAM drops to baseline (~2-3 GB) after restart. If not, investigate background processes or driver leaks.

## 7. GPU Reset Procedure (Last Resort)
- If GPU becomes unresponsive or Vulkan context corrupts:
- `[DESTRUCTIVE]` `sudo rmmod amdgpu && sudo modprobe amdgpu` (requires root, may drop container)
- `[LOW-RISK]` `docker restart llama-cpp-server`
- Verify GPU reinitialization and VRAM availability before resuming.

All GPU checks must be logged. Thermal, power, or VRAM anomalies trigger immediate halt and proceed to Failure Triage Trees.

---

# Proxy And Database Checks

The Flight Recorder proxy (`http://192.168.1.116:8181/v1`) routes requests to the llama.cpp server. The SQLite database at `/models` stores benchmark metadata. Both must be healthy, consistent, and isolated.

## 1. Proxy Health & Routing Validation
- `[READ-ONLY]` `curl -s -o /dev/null -w "%{http_code} %{time_total}s" http://192.168.1.116:8181/v1/health`
- Expected: `200` and latency < 50ms. Higher latency indicates proxy overload or network congestion.
- `[READ-ONLY]` `curl -s http://192.168.1.116:8181/v1/models | jq '.data[].id'`
- Verify model ID matches `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`.

## 2. Proxy Timeout & Retry Configuration
- Check proxy timeout settings to prevent premature request drops.
- `[READ-ONLY]` `docker inspect flight-recorder-proxy --format '{{range .Config.Env}}{{.}} {{end}}' | grep -i "timeout\|retry"`
- Ensure `REQUEST_TIMEOUT ≥ 300s` for long-output tasks. Adjust if needed.
- `[MEDIUM-RISK]` `docker exec flight-recorder-proxy sed -i 's/REQUEST_TIMEOUT=120/REQUEST_TIMEOUT=300/' /etc/proxy.conf && docker restart flight-recorder-proxy`

## 3. SQLite Database Integrity
- `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
- Must return `ok`. Any corruption triggers immediate halt.
- `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA journal_mode;"`
- Should be `WAL` or `DELETE`. `MEMORY` mode risks data loss on crash.

## 4. Database Backup & Restore Readiness
- `[MEDIUM-RISK]` `cp /models/benchmark.db /models/benchmark.db.bak.$(date +%Y%m%d_%H%M%S)`
- Verify backup size matches original.
- `[LOW-RISK]` `sqlite3 /models/benchmark.db.bak.* ".tables"`
- Confirm schema matches production.

## 5. Query Validation & Isolation
- `[READ-ONLY]` `sqlite3 /models/benchmark.db "SELECT task_id, status, finish_reason FROM runs WHERE status='FAILED' LIMIT 5;"`
- Investigate failed runs. Cross-reference with server logs.
- Ensure no external connections to `/models`. Verify permissions.
- `[READ-ONLY]` `ls -l /models/benchmark.db`
- Permissions should be `640` or `600`, owned by `llama:llama` or equivalent.

## 6. Proxy-Server Sync Verification
- Ensure proxy forwards requests without modification.
- `[LOW-RISK]` `curl -v http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"test","messages":[],"max_tokens":1}' 2>&1 | grep -i "error\|forward\|upstream"`
- Verify upstream points to `192.168.1.116:8080` (or internal container port). Misrouting causes silent failures.

All proxy and database checks must pass before benchmark execution. Corruption, misrouting, or timeout misconfiguration triggers rollback.

---

# Dashboard Checks

ServerTop (`http://192.168.1.116:8090`) provides real-time metrics. Dashboard accuracy must be validated against CLI and log data to prevent false confidence.

## 1. Dashboard Responsiveness & Load
- `[READ-ONLY]` `curl -s -o /dev/null -w "%{http_code} %{time_total}s" http://192.168.1.116:8090/status`
- Expected: `200`, latency < 100ms. Higher indicates dashboard backend overload.

## 2. Metric Cross-Validation
- Compare dashboard VRAM, token counts, and MTP stats with CLI.
- `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | grep -E "vram_used\|tokens_generated\|mtp_acceptance"`
- `[READ-ONLY]` `rocm-smi --showmeminfo vram --json | jq '.[0].vram_used'`
- Discrepancies > 5% indicate metric drift. Restart dashboard backend if persistent.

## 3. Token Counter Accuracy
- `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Count to 10."}],"max_tokens":512}' | jq '.usage.total_tokens'`
- Verify dashboard token counter increments by same amount. Mismatch indicates counter leak or reset.

## 4. MTP & Reasoning Budget Visualization
- Confirm dashboard displays MTP acceptance rate and reasoning budget usage.
- `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | grep -i "mtp\|reasoning\|budget"`
- If missing, verify server exposes metrics endpoint. Dashboard may need config update.

## 5. Dashboard Rollback
- If dashboard becomes unresponsive or displays corrupted data:
- `[LOW-RISK]` `docker restart servertop-dashboard`
- `[READ-ONLY]` `curl -s http://192.168.1.116:8090/status`
- Verify recovery. If persistent, clear dashboard cache.
- `[MEDIUM-RISK]` `docker exec servertop-dashboard rm -rf /tmp/dashboard_cache/*`

Dashboard checks are non-blocking but critical for accurate reporting. Anomalies trigger investigation before proceeding.

---

# Failure Triage Trees

Structured decision paths for common failures. Each tree begins with read-only checks, escalates to low/medium-risk interventions, and defines rollback triggers.

## Tree 1: `finish_reason=length` Before Final Content
1. `[READ-ONLY]` Check `max_tokens` vs expected output length.
2. `[READ-ONLY]` Verify `reasoning_budget` not exhausted early.
3. `[READ-ONLY]` Check MTP acceptance rate in logs.
4. If budget exhausted → Increase `--reasoning-budget` or reduce prompt complexity.
5. If MTP acceptance < 50% → Reduce `--spec-draft-n-max` to 1.
6. If `max_tokens` insufficient → Increase incrementally, revalidate.
7. Rollback trigger: VRAM > 28 GB or generation stalls > 60s.

## Tree 2: VRAM OOM / Vulkan Allocation Failure
1. `[READ-ONLY]` `rocm-smi --showmeminfo vram`
2. `[READ-ONLY]` `dmesg | grep -i "amdgpu\|vulkan\|oom"`
3. If VRAM > 30 GB → Reduce context size or disable MTP.
4. If Vulkan error → Restart container, verify driver.
5. If persistent → Clear Vulkan cache, restart GPU driver (destructive gate).
6. Rollback trigger: GPU unresponsive or container crash loop.

## Tree 3: Proxy Timeout / Request Drop
1. `[READ-ONLY]` `curl -v http://192.168.1.116:8181/v1/health`
2. `[READ-ONLY]` Check proxy logs for `upstream timeout`.
3. If timeout < 300s → Increase `REQUEST_TIMEOUT`.
4. If upstream unreachable → Verify container network, restart proxy.
5. Rollback trigger: Proxy returns 502/504 consistently.

## Tree 4: SQLite Corruption / Lock Contention
1. `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
2. `[READ-ONLY]` `fuser /models/benchmark.db`
3. If corruption → Restore from latest backup.
4. If locked → Kill writer process, verify WAL mode.
5. Rollback trigger: Database returns `database disk image is malformed`.

## Tree 5: Dashboard Metric Drift
1. `[READ-ONLY]` Cross-validate CLI vs dashboard metrics.
2. `[READ-ONLY]` Check dashboard backend logs.
3. If drift > 5% → Restart dashboard, clear cache.
4. If persistent → Verify server metrics endpoint exposure.
5. Rollback trigger: Dashboard unresponsive > 5 min.

Each tree requires evidence capture before escalation. Rollback is preferred over extended troubleshooting when stability is compromised.

---

# Rollback Procedures

Defined restoration paths for configuration, container, database, and GPU state. All procedures include verification steps.

## 1. Container Configuration Rollback
- `[MEDIUM-RISK]` `docker exec llama-cpp-server cp /app/server_args.conf.bak /app/server_args.conf`
- `[LOW-RISK]` `docker restart llama-cpp-server`
- Verify args match baseline. Run controlled prompt test.

## 2. Image Tag Rollback
- `[MEDIUM-RISK]` `docker stop llama-cpp-server && docker rm llama-cpp-server`
- `[LOW-RISK]` `docker run -d --name llama-cpp-server llama-cpp-server-vulkan:working-20260613 ...`
- Verify container starts, model loads, VRAM baseline restored.

## 3. Database Rollback
- `[MEDIUM-RISK]` `cp /models/benchmark.db.bak.$(date -d '1 hour ago' +%Y%m%d_%H%M%S) /models/benchmark.db`
- `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
- Verify schema, recent runs preserved, no corruption.

## 4. Proxy Configuration Rollback
- `[MEDIUM-RISK]` `docker exec flight-recorder-proxy cp /etc/proxy.conf.bak /etc/proxy.conf`
- `[LOW-RISK]` `docker restart flight-recorder-proxy`
- Verify routing, timeout, health endpoint.

## 5. GPU Driver Reset (Destructive Gate)
- `[DESTRUCTIVE]` `sudo rmmod amdgpu && sudo modprobe amdgpu`
- `[LOW-RISK]` `docker restart llama-cpp-server`
- Verify GPU recognized, VRAM available, Vulkan context clean.
- Requires dual human approval. Full system snapshot recommended before execution.

All rollbacks require post-verification. If rollback fails, halt and escalate to human approval gate.

---

# Human Approval Gates

Explicit checkpoints requiring operator sign-off before proceeding.

1. **Pre-Benchmark Configuration Change**: Any modification to `max_tokens`, `reasoning_budget`, `context-size`, or MTP parameters requires approval.
2. **Destructive GPU/Driver Operations**: Module reload, driver reinstall, or volume deletion requires dual approval.
3. **Database Schema Modification**: Adding tables, altering columns, or running migrations requires approval.
4. **Bulk Benchmark Execution**: Queueing > 5 tasks without per-task validation requires approval.
5. **Data Export/Transfer**: Moving logs, DB exports, or screenshots outside `/models` requires approval.

Approval workflow:
- Operator reviews evidence, thresholds, and rollback readiness.
- Signs off in `/models/approval_log.txt` with timestamp, action, and justification.
- Agent proceeds only after explicit `APPROVED` entry.

---

# Evidence To Capture

All diagnostic and validation activities must generate local evidence. No data leaves the host.

1. **Screenshots**: ServerTop dashboard, VRAM graphs, MTP stats, token counters.
2. **Log Excerpts**: Server logs, proxy logs, dmesg GPU errors, SQLite integrity checks.
3. **Metric Snapshots**: JSON exports of `/8090/metrics`, `rocm-smi` JSON, `docker stats`.
4. **Token Counts & Timing**: `max_tokens`, `reasoning_content` length, `content` length, generation time, MTP acceptance rate.
5. **Database Snapshots**: `.bak` files, schema dumps, run status summaries.
6. **Storage Protocol**: `/models/evidence/YYYYMMDD_HHMMSS/` directory structure. Encrypted if sensitive. Retention: 90 days.
7. **Verification**: Hash each evidence file. Log hash to `/models/evidence_manifest.txt`.

---

# Final Go No-Go Checklist

Binary validation before benchmark execution. All must pass.

- [ ] Host reachable, network stable
- [ ] Container running, image tag verified
- [ ] Model file integrity confirmed
- [ ] VRAM free ≥ 12 GB, GPU recognized
- [ ] Proxy health 200, routing correct
- [ ] SQLite integrity `ok`, no locks
- [ ] Dashboard responsive, metrics aligned
- [ ] Reasoning budget ≤ 80% utilization in test
- [ ] `finish_reason=stop` in controlled test
- [ ] MTP acceptance ≥ 55%
- [ ] Thermal ≤ 85°C, power ≤ 250W
- [ ] Rollback backups verified
- [ ] Human approval logged
- [ ] Evidence capture directory ready

If any item fails, halt. Proceed to triage. Only when all items pass is the system `GO` for benchmark execution.

# Advanced MTP & Speculative Decoding Diagnostics

Speculative decoding via MTP (`--spec-type draft-mtp --spec-draft-n-max 2`) introduces draft-token generation that runs in parallel with the primary model. While this accelerates throughput, it can cause token budget misalignment, draft rejection cascades, and premature `finish_reason=length` emissions if not carefully monitored. The following procedures isolate MTP-specific failures and calibrate draft behavior without compromising generation quality.

## 1. Draft Acceptance Rate Monitoring
- Track acceptance rate continuously during benchmark execution. Low acceptance indicates draft-model divergence or context misalignment.
- `[READ-ONLY]` `docker logs llama-cpp-server 2>&1 | grep -oP 'draft_tokens_accepted=\K\d+' | tail -50 | awk '{sum+=$1; count++} END {print "Avg Acceptance:", sum/count}'`
- Threshold: ≥ 60% sustained. Below 50% triggers draft reduction or temporary MTP disable.
- `[READ-ONLY]` `docker logs llama-cpp-server 2>&1 | grep -oP 'draft_tokens_rejected=\K\d+' | tail -50 | awk '{sum+=$1; count++} END {print "Avg Rejection:", sum/count}'`
- High rejection correlates with VRAM fragmentation and KV cache thrashing.

## 2. Draft Token Budget Partitioning
- MTP draft tokens consume from the same `max_tokens` pool unless explicitly isolated. Verify partitioning behavior.
- `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"List five distinct architectural patterns for distributed inference."}],"max_tokens":1024}' | jq '{prompt_tokens: .usage.prompt_tokens, completion_tokens: .usage.completion_tokens, total_tokens: .usage.total_tokens, finish_reason: .choices[0].finish_reason}'`
- If `completion_tokens` approaches `max_tokens` before content completion, draft tokens are consuming budget. Reduce `--spec-draft-n-max` or increase `max_tokens` by 15%.
- `[MEDIUM-RISK]` `docker exec llama-cpp-server sed -i 's/--spec-draft-n-max 2/--spec-draft-n-max 1/' /app/server_args.conf && docker restart llama-cpp-server`
- Validate stability, then restore to 2 after confirmation.

## 3. Draft Cache Eviction & Fragmentation
- MTP maintains a separate draft KV cache. Fragmentation causes allocation failures and silent truncation.
- `[READ-ONLY]` `docker stats --no-stream llama-cpp-server | grep llama-cpp-server | awk '{print $5, $6}'`
- `[READ-ONLY]` `rocm-smi --showmeminfo vram --json | jq '.[0].vram_used'`
- If VRAM usage spikes > 29 GB during draft generation, force cache defragmentation.
- `[LOW-RISK]` `docker exec llama-cpp-server kill -USR1 1` (triggers internal KV cache compaction in llama.cpp)
- Verify VRAM drops to baseline within 10 seconds. If not, proceed to GPU reset procedure.

## 4. MTP Timeout & Fallback Configuration
- Configure graceful fallback to non-speculative decoding if MTP stalls.
- `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{range .Config.Cmd}}{{.}} {{end}}' | grep -i "spec\|timeout\|fallback"`
- Ensure `--spec-decay 0.5` and `--spec-decay-interval 10` are present. Adjust if draft divergence persists.
- `[MEDIUM-RISK]` `docker exec llama-cpp-server sed -i 's/--spec-decay 0.5/--spec-decay 0.3/' /app/server_args.conf && docker restart llama-cpp-server`
- Lower decay reduces draft aggressiveness, improving stability at the cost of throughput.

## 5. MTP Validation Thresholds
- Acceptance rate ≥ 55%
- Draft rejection < 20% per generation cycle
- VRAM peak ≤ 28 GB during speculative phase
- `finish_reason=length` only after content completion
- If any threshold breached, halt queue, capture evidence, and proceed to rollback or configuration adjustment.

---

# KV Cache & Context Window Management

With a 262,144 context window and single parallel slot, KV cache management is critical. Fragmentation, eviction misconfiguration, or attention window misalignment can cause silent truncation or OOM conditions.

## 1. KV Cache Allocation Verification
- Confirm KV cache size matches context window and slot configuration.
- `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{range .Config.Cmd}}{{.}} {{end}}' | grep -oP '(?<=--context-size )\d+'`
- Expected: `262144`. Mismatch triggers immediate halt.
- `[READ-ONLY]` `docker logs llama-cpp-server 2>&1 | grep -i "kv cache\|context\|allocation" | tail -20`
- Verify no `kv cache full` or `context overflow` warnings.

## 2. Sliding Window vs. Full Attention
- Qwen3.6 supports sliding window attention. Verify configuration matches model capabilities.
- `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{range .Config.Cmd}}{{.}} {{end}}' | grep -i "attention\|window\|rope"`
- If `--attention-type sliding` is missing, add it to reduce VRAM pressure.
- `[MEDIUM-RISK]` `docker exec llama-cpp-server sed -i '/--context-size/a --attention-type sliding' /app/server_args.conf && docker restart llama-cpp-server`
- Validate VRAM reduction and generation stability.

## 3. KV Cache Defragmentation & Eviction
- Long-running sessions accumulate fragmented KV blocks. Schedule periodic defragmentation.
- `[LOW-RISK]` `docker exec llama-cpp-server kill -USR2 1` (triggers KV cache defrag in llama.cpp)
- `[READ-ONLY]` `rocm-smi --showmeminfo vram --json | jq '.[0].vram_used'`
- Confirm VRAM drops by ≥ 500 MB. If not, force container restart.
- `[LOW-RISK]` `docker restart llama-cpp-server`

## 4. Context Overflow Prevention
- Prevent prompt + reasoning + content from exceeding context window.
- `[LOW-RISK]` `curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Analyze the following 5000-token technical document and summarize key architectural decisions."}],"max_tokens":8192}' | jq '{prompt_tokens: .usage.prompt_tokens, completion_tokens: .usage.completion_tokens, total_tokens: .usage.total_tokens}'`
- If `total_tokens` approaches 262144, reduce prompt size or increase context window (requires model reload).
- `[MEDIUM-RISK]` `docker exec llama-cpp-server sed -i 's/--context-size 262144/--context-size 131072/' /app/server_args.conf && docker restart llama-cpp-server`
- Validate stability, then restore after VRAM headroom confirmed.

## 5. KV Cache Validation Thresholds
- VRAM usage ≤ 28 GB during full context load
- No `kv cache full` warnings in logs
- Defragmentation reduces VRAM by ≥ 500 MB
- Context overflow prevented via prompt/token budgeting
- Breach triggers halt, evidence capture, and rollback.

---

# Automated Validation & Benchmark Orchestration

Bulk execution is prohibited without per-task validation. The following orchestration framework ensures serialized execution, automatic failure isolation, and metric aggregation.

## 1. Task Queue Serialization
- Implement a FIFO queue with single-slot enforcement.
- `[LOW-RISK]` `mkdir -p /models/benchmark_queue && echo '{"task_id":"T001","prompt":"Explain Vulkan memory barriers.","max_tokens":2048}' > /models/benchmark_queue/T001.json`
- `[READ-ONLY]` `ls -1 /models/benchmark_queue/ | wc -l`
- Queue size must be 1 during active validation. Additional tasks wait in `pending` state.

## 2. Per-Task Validation Script
- Automated validation loop with threshold enforcement.
- `[LOW-RISK]` `cat << 'EOF' > /models/scripts/validate_task.sh
#!/bin/bash
TASK_FILE=$1
TASK_ID=$(basename $TASK_FILE .json)
PROMPT=$(jq -r '.prompt' $TASK_FILE)
MAX_TOKENS=$(jq -r '.max_tokens' $TASK_FILE)
RESPONSE=$(curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf\",\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}],\"max_tokens\":$MAX_TOKENS}")
FINISH=$(echo $RESPONSE | jq -r '.choices[0].finish_reason')
CONTENT_LEN=$(echo $RESPONSE | jq -r '.choices[0].message.content | length')
if [ "$FINISH" == "stop" ] && [ "$CONTENT_LEN" -gt 100 ]; then
  echo "VALIDATED:$TASK_ID"
  mv $TASK_FILE /models/benchmark_queue/validated/
else
  echo "FAILED:$TASK_ID:finish=$FINISH,len=$CONTENT_LEN"
  mv $TASK_FILE /models/benchmark_queue/failed/
fi
EOF
chmod +x /models/scripts/validate_task.sh`
- Execute per task. Only `VALIDATED` tasks proceed to bulk queue.

## 3. Rate Limiting & Backpressure
- Prevent server overload during validation.
- `[LOW-RISK]` `sleep 5` between task submissions.
- `[READ-ONLY]` `docker stats --no-stream llama-cpp-server | awk '{print $4}' | cut -d'%' -f1 | awk '{if($1>80) print "CPU_HIGH"}'`
- If CPU > 80% or VRAM > 27 GB, pause queue for 30 seconds.

## 4. Failure Isolation & Retry Logic
- Automatic retry on transient failures.
- `[LOW-RISK]` `for i in 1 2 3; do echo "Retry $i"; /models/scripts/validate_task.sh /models/benchmark_queue/T001.json; [ $? -eq 0 ] && break; sleep 10; done`
- Max 3 retries. Persistent failure moves task to `quarantine` for manual review.

## 5. Orchestration Thresholds
- Queue size ≤ 1 during validation
- CPU ≤ 80%, VRAM ≤ 27 GB
- Retry limit ≤ 3
- Failure rate < 5% per batch
- Breach triggers halt, evidence capture, and rollback.

---

# Data Privacy & Evidence Handling Protocols

All benchmark data, logs, and screenshots remain strictly local. The following protocols ensure compliance with privacy requirements while maintaining auditability.

## 1. Local-Only Storage Enforcement
- Verify no external mounts or network shares.
- `[READ-ONLY]` `mount | grep -i "models\|tmp\|var"`
- `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{range .Mounts}}{{.Destination}} {{end}}'`
- Ensure `/models` is local filesystem. No NFS, SMB, or cloud sync.

## 2. Log Sanitization & Rotation
- Prevent sensitive data leakage in logs.
- `[LOW-RISK]` `logrotate -f /etc/logrotate.d/llama-server`
- `[READ-ONLY]` `grep -i "password\|secret\|key\|token" /var/log/llama-server.log`
- If matches found, sanitize immediately.
- `[MEDIUM-RISK]` `sed -i 's/\(password\|secret\|key\|token\)=\([^ ]*\)/\1=REDACTED/g' /var/log/llama-server.log`

## 3. Screenshot Capture & Storage
- Capture dashboard and metric states without exposing raw data.
- `[LOW-RISK]` `import -window root /models/evidence/screenshots/dashboard_$(date +%Y%m%d_%H%M%S).png`
- `[READ-ONLY]` `ls -lh /models/evidence/screenshots/`
- Verify files are local, encrypted if sensitive, and hashed.
- `[LOW-RISK]` `sha256sum /models/evidence/screenshots/*.png >> /models/evidence_manifest.txt`

## 4. Database Export Restrictions
- Prevent raw data export.
- `[READ-ONLY]` `sqlite3 /models/benchmark.db ".dump" | grep -i "content\|prompt\|response"`
- If raw content detected, restrict export to metadata only.
- `[MEDIUM-RISK]` `sqlite3 /models/benchmark.db "SELECT task_id, status, finish_reason, timestamp FROM runs;" > /models/evidence/metadata_export.csv`

## 5. Retention & Purge Policy
- 90-day retention. Automatic purge after expiration.
- `[LOW-RISK]` `find /models/evidence/ -type f -mtime +90 -delete`
- `[READ-ONLY]` `du -sh /models/evidence/`
- Verify storage compliance. Breach triggers manual review.

---

# Extended Failure Triage Trees

Additional decision paths for network, container, SQLite, Vulkan, and proxy failures.

## Tree 6: Network Partition / Proxy Unreachable
1. `[READ-ONLY]` `ping -c 4 192.168.1.116`
2. `[READ-ONLY]` `ss -tulnp | grep -E "8090|8181"`
3. If ports closed → Restart proxy/dashboard containers.
4. If network unreachable → Verify switch/router, check ARP table.
5. Rollback trigger: Persistent 504/502 > 5 min.

## Tree 7: Container Resource Limits / OOM Kill
1. `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{.HostConfig.Memory}}'`
2. `[READ-ONLY]` `dmesg | grep -i "oom\|kill\|memory"`
3. If OOM kill → Increase container memory limit or reduce context.
4. `[MEDIUM-RISK]` `docker update --memory 32g llama-cpp-server`
5. Rollback trigger: Container crash loop.

## Tree 8: SQLite WAL Journal Bloat
1. `[READ-ONLY]` `ls -lh /models/benchmark.db-wal /models/benchmark.db-shm`
2. If WAL > 1 GB → Force checkpoint.
3. `[LOW-RISK]` `sqlite3 /models/benchmark.db "PRAGMA wal_checkpoint(TRUNCATE);" `
4. Verify journal shrinks. If not, restore from backup.
5. Rollback trigger: Database locked > 2 min.

## Tree 9: Vulkan Driver Timeout / Context Loss
1. `[READ-ONLY]` `dmesg | grep -i "vulkan\|amdgpu\|timeout\|reset"`
2. `[READ-ONLY]` `rocm-smi --showtemp --showpower`
3. If timeout → Restart container, clear Vulkan cache.
4. `[MEDIUM-RISK]` `docker exec llama-cpp-server rm -rf /tmp/vulkan_cache/* && docker restart llama-cpp-server`
5. Rollback trigger: GPU unresponsive.

## Tree 10: Proxy Header Mismatch / Routing Failure
1. `[READ-ONLY]` `curl -v http://192.168.1.116:8181/v1/models 2>&1 | grep -i "upstream\|host\|forward"`
2. If mismatch → Verify proxy config, restart.
3. `[MEDIUM-RISK]` `docker exec flight-recorder-proxy sed -i 's/upstream_host=wrong/upstream_host=192.168.1.116/' /etc/proxy.conf && docker restart flight-recorder-proxy`
4. Rollback trigger: 404/502 persistent.

---

# Rollback & Recovery Automation

Automated restoration procedures for configuration, container, database, and GPU state.

## 1. Configuration Versioning
- Maintain versioned backups of all config files.
- `[MEDIUM-RISK]` `cp /app/server_args.conf /app/server_args.conf.v$(date +%Y%m%d_%H%M%S)`
- `[READ-ONLY]` `ls -lt /app/server_args.conf.v* | head -5`
- Verify latest backup matches baseline.

## 2. Automated Rollback Trigger
- Script to detect failure and restore previous state.
- `[LOW-RISK]` `cat << 'EOF' > /models/scripts/auto_rollback.sh
#!/bin/bash
if docker logs llama-cpp-server 2>&1 | grep -q "oom\|vulkan error\|kv cache full"; then
  echo "ROLLBACK_TRIGGERED"
  cp /app/server_args.conf.bak /app/server_args.conf
  docker restart llama-cpp-server
  echo "ROLLBACK_COMPLETE"
fi
EOF
chmod +x /models/scripts/auto_rollback.sh`
- Execute periodically via cron or manual invocation.

## 3. Database State Restoration
- Restore from verified backup on corruption.
- `[MEDIUM-RISK]` `cp /models/benchmark.db.bak.latest /models/benchmark.db`
- `[READ-ONLY]` `sqlite3 /models/benchmark.db "PRAGMA integrity_check;"`
- Verify schema, recent runs, no corruption.

## 4. GPU Driver Recovery
- Automated module reload on driver fault.
- `[DESTRUCTIVE]` `sudo rmmod amdgpu && sudo modprobe amdgpu`
- `[LOW-RISK]` `docker restart llama-cpp-server`
- Verify GPU recognized, VRAM available, Vulkan context clean.
- Requires dual human approval. Full system snapshot recommended.

## 5. Rollback Verification Protocol
- Post-rollback validation checklist:
  - Container running, image tag verified
  - VRAM baseline restored
  - Proxy/dashboard responsive
  - SQLite integrity `ok`
  - Controlled prompt test passes
- If any fails, halt and escalate.

---

# Human Approval & Change Management Workflows

Formal change request templates, approval workflows, audit logging, and rollback authorization matrices.

## 1. Change Request Template
- Standardized format for all modifications.
- Fields: Request ID, Operator, Timestamp, Change Description, Risk Level, Rollback Plan, Approval Status.
- `[LOW-RISK]` `cat << 'EOF' > /models/approval_log.txt
CHANGE_REQUEST: CR-20260613-001
OPERATOR: admin
TIMESTAMP: $(date -u +%Y-%m-%dT%H:%M:%SZ)
CHANGE: Increase --reasoning-budget from 8192 to 12288
RISK: MEDIUM
ROLLBACK: Restore server_args.conf.bak, restart container
APPROVAL: PENDING
EOF`

## 2. Approval Workflow
- Dual approval required for medium/destructive changes.
- Operator 1 reviews evidence, thresholds, rollback readiness.
- Operator 2 verifies compliance, signs off.
- `[LOW-RISK]` `echo "APPROVED_BY: operator2" >> /models/approval_log.txt`
- Agent proceeds only after explicit `APPROVED` entry.

## 3. Audit Trail Maintenance
- Log all actions, outcomes, and rollbacks.
- `[READ-ONLY]` `tail -50 /models/approval_log.txt`
- Verify chronological integrity, no gaps.
- `[LOW-RISK]` `sha256sum /models/approval_log.txt >> /models/audit_hash.log`

## 4. Rollback Authorization Matrix
- Read-only/low-risk: Single operator approval.
- Medium-risk: Dual approval, verified backup.
- Destructive: Dual approval, full system snapshot, maintenance window.
- Breach triggers immediate halt.

## 5. Change Management Thresholds
- Approval latency ≤ 15 min
- Rollback readiness verified before execution
- Audit trail complete, no gaps
- Breach triggers manual review.

---

# Continuous Monitoring & Alerting Thresholds

Threshold definitions, metric aggregation, log rotation, and dashboard calibration procedures.

## 1. Metric Aggregation & Baseline
- Aggregate metrics over 5-minute windows.
- `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | jq '{vram: .vram_used, tokens: .tokens_generated, mtp: .mtp_acceptance, cpu: .cpu_usage}'`
- Compare against baseline. Deviation > 10% triggers alert.

## 2. Log Rotation & Archival
- Prevent log bloat.
- `[LOW-RISK]` `logrotate -f /etc/logrotate.d/llama-server`
- `[READ-ONLY]` `du -sh /var/log/llama-server.log*`
- Verify rotation success, archival to `/models/logs/archive/`.

## 3. Dashboard Calibration
- Cross-validate dashboard vs CLI metrics.
- `[READ-ONLY]` `curl -s http://192.168.1.116:8090/metrics | jq '.vram_used'`
- `[READ-ONLY]` `rocm-smi --showmeminfo vram --json | jq '.[0].vram_used'`
- Discrepancy > 5% → Restart dashboard, clear cache.

## 4. Alerting Thresholds
- VRAM > 28 GB → Warning
- VRAM > 30 GB → Critical, halt
- MTP acceptance < 50% → Warning
- MTP acceptance < 40% → Critical, reduce draft
- CPU > 85% → Warning
- CPU > 95% → Critical, pause queue
- Breach triggers automated response or manual intervention.

## 5. Monitoring Validation
- Verify alerting pipeline functional.
- `[LOW-RISK]` `echo "TEST_ALERT" >> /models/alerts.log`
- `[READ-ONLY]` `tail -5 /models/alerts.log`
- Confirm logging, notification, and response mechanisms active.

---

# Security & Access Control Hardening

Network isolation, container hardening, database permissions, and secret management.

## 1. Network Isolation
- Restrict access to local subnet.
- `[READ-ONLY]` `iptables -L -n | grep -E "8090|8181"`
- Verify no external exposure. Block if needed.
- `[MEDIUM-RISK]` `iptables -A INPUT -p tcp --dport 8090 -s 192.168.1.0/24 -j ACCEPT && iptables -A INPUT -p tcp --dport 8090 -j DROP`

## 2. Container Hardening
- Run as non-root, read-only filesystem where possible.
- `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{.Config.User}}'`
- If root, change to `llama:llama`.
- `[MEDIUM-RISK]` `docker update --user llama:llama llama-cpp-server`

## 3. Database Permissions
- Restrict access to `/models`.
- `[READ-ONLY]` `ls -l /models/benchmark.db`
- Verify `640` or `600`, owned by `llama:llama`.
- `[LOW-RISK]` `chmod 640 /models/benchmark.db && chown llama:llama /models/benchmark.db`

## 4. Secret Management
- No real secrets in config. Use environment variables.
- `[READ-ONLY]` `docker inspect llama-cpp-server --format '{{range .Config.Env}}{{.}} {{end}}' | grep -i "password\|secret\|key"`
- If found, migrate to Docker secrets or vault.
- `[MEDIUM-RISK]` `docker secret create db_password /dev/null` (placeholder for real secret)

## 5. Security Validation
- Verify isolation, permissions, secret handling.
- `[READ-ONLY]` `nmap -sV 192.168.1.116 -p 8090,8181`
- Confirm only expected services exposed. Breach triggers immediate lockdown.

---

# Post-Benchmark Analysis & Reporting Generation

Aggregation methods, metric normalization, result validation, and report generation protocols.

## 1. Metric Aggregation
- Collect per-task metrics.
- `[READ-ONLY]` `sqlite3 /models/benchmark.db "SELECT task_id, status, finish_reason, tokens_generated, generation_time FROM runs WHERE status='VALIDATED';"`
- Aggregate into summary CSV.
- `[LOW-RISK]` `sqlite3 /models/benchmark.db -csv "SELECT task_id, status, finish_reason, tokens_generated, generation_time FROM runs WHERE status='VALIDATED';" > /models/reports/summary.csv`

## 2. Metric Normalization
- Normalize against baseline.
- `[LOW-RISK]` `awk -F',' 'NR>1 {print $4/$5}' /models/reports/summary.csv | awk '{sum+=$1; count++} END {print "Avg Tokens/sec:", sum/count}'`
- Compare against expected throughput. Deviation > 15% triggers review.

## 3. Result Validation
- Cross-reference with logs, dashboard, CLI.
- `[READ-ONLY]` `grep -c "finish_reason=stop" /var/log/llama-server.log`
- Verify consistency. Mismatch indicates logging drift.

## 4. Report Generation
- Generate local-only report.
- `[LOW-RISK]` `cat << 'EOF' > /models/reports/benchmark_report.md
# Benchmark Report
Date: $(date -u +%Y-%m-%d)
Tasks Validated: $(sqlite3 /models/benchmark.db "SELECT COUNT(*) FROM runs WHERE status='VALIDATED';")
Avg Throughput: $(awk -F',' 'NR>1 {print $4/$5}' /models/reports/summary.csv | awk '{sum+=$1; count++} END {print sum/count}') tokens/sec
MTP Acceptance: $(docker logs llama-cpp-server 2>&1 | grep -oP 'draft_tokens_accepted=\K\d+' | tail -50 | awk '{sum+=$1; count++} END {print sum/count}')%
VRAM Peak: $(rocm-smi --showmeminfo vram --json | jq '.[0].vram_used') MB
Status: COMPLETE
EOF`
- Verify report accuracy, store locally.

## 5. Reporting Thresholds
- Validation rate ≥ 95%
- Throughput within 15% of baseline
- MTP acceptance ≥ 55%
- VRAM peak ≤ 28 GB
- Breach triggers manual review, potential re-run.

All post-benchmark procedures must be logged. Evidence captured, reports stored locally, no external transmission. System returns to idle state, ready for next cycle.