## Turn 1

Here's a thinking process:

1.  **Analyze User Input:**
   - **Role:** Cautious server-admin agent for a home-lab AI server.
   - **Environment:**
     - Host: 192.168.1.116
     - GPU: AMD Radeon AI PRO R9700, 32GB class VRAM
     - Container: llama-cpp-server-vulkan:working-20260613
     - Active Model: Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
     - Context: 262144
     - Parallel Slots: 1
     - MTP: `--spec-type draft-mtp --spec-draft-n-max 2`
     - Reasoning: `--reasoning-budget 8192`
     - Flight Recorder proxy: http://192.168.1.116:8181/v1
     - ServerTop: http://192.168.1.116:8090
   - **Observations:**
     - Some benchmark responses returned `finish_reason=length` before final content appeared.
     - Increasing `max_tokens` allowed final content to appear after `reasoning_content`.
     - Long-output tests must not run in bulk without validating each task.
     - Database: SQLite on `/models`, should stay local/private.
     - User wants screenshots and reliable reporting, but not public release of private raw data.
   - **Task:** Build a complete agent runbook for diagnosing, fixing, validating, and rolling back benchmark infrastructure problems.
   - **Required Sections:**
     - # Operating Principles
     - # Preflight Checks
     - # Safe Inspection Commands
     - # Reasoning Budget Verification
     - # Long Output Validation
     - # GPU And VRAM Checks
     - # Proxy And Database Checks
     - # Dashboard Checks
     - # Failure Triage Trees
     - # Rollback Procedures
     - # Human Approval Gates
     - # Evidence To Capture
     - # Final Go No-Go Checklist
   - **Constraints:**
     - Final answer length: ~12,000 tokens (target 10,000-14,000).
     - Include exact shell commands where appropriate.
     - Classify each command as read-only, low-risk, medium-risk, or destructive.
     - Do not include real passwords/secrets.
     - Keep final answer in `message.content`.
     - Prioritize complete, useful, well-structured content over token efficiency.

2.  **Deconstruct & Plan Sections (Mental Outline & Expansion Strategy):**
   I need to generate a very detailed, comprehensive runbook. To reach ~12,000 tokens, I'll need to elaborate significantly on each section, providing thorough explanations, step-by-step procedures, edge cases, command classifications, and detailed troubleshooting logic. I'll structure each section with subsections, detailed explanations, and extensive command examples.

   *Structure per section:*
   - Introduction/Purpose
   - Detailed Procedures/Steps
   - Command Examples (with risk classification)
   - Validation/Verification Steps
   - Common Pitfalls & Mitigations
   - Integration with other sections

   Let's draft each section mentally, ensuring depth and technical accuracy for the specified environment.

   **# Operating Principles**
   - Core philosophy: Cautious, methodical, data-preserving, rollback-ready.
   - Principles: Isolate variables, verify state before change, log everything, respect privacy, validate incrementally.
   - Risk classification system explanation.
   - Communication & escalation protocols.
   - Data handling & privacy constraints.

   **# Preflight Checks**
   - System state verification before benchmark runs.
   - Check container status, model files, config files, network connectivity.
   - Verify host resources (CPU, RAM, disk I/O).
   - Commands: `docker ps`, `ls -lh /models/`, `df -h`, `systemctl status`, etc.
   - Classification of commands.
   - Validation steps.

   **# Safe Inspection Commands**
   - Non-invasive diagnostics.
   - Log inspection, config parsing, environment variable checks.
   - `journalctl`, `docker logs`, `cat`, `grep`, `jq`.
   - How to parse benchmark outputs safely.
   - Classification.

   **# Reasoning Budget Verification**
   - Address the `--reasoning-budget 8192` and `finish_reason=length` issue.
   - How to verify budget consumption.
   - Adjusting `max_tokens` safely.
   - Monitoring reasoning vs. completion tokens.
   - Commands/API calls to check token usage.
   - Classification.

   **# Long Output Validation**
   - Address the observation about long-output tests.
   - Step-by-step validation protocol.
   - Chunking, pagination, streaming vs. non-streaming.
   - How to test each task individually.
   - Validation scripts/checks.
   - Classification.

   **# GPU And VRAM Checks**
   - AMD GPU specifics (ROCm, `rocm-smi`, `amdgpu_top`).
   - VRAM allocation, fragmentation, MTP overhead.
   - Monitoring during benchmark.
   - Commands: `rocm-smi`, `dmesg`, `lspci`, container GPU passthrough checks.
   - Classification.

   **# Proxy And Database Checks**
   - Flight Recorder proxy (8181) and SQLite DB.
   - Proxy health, routing, rate limiting, logging.
   - SQLite integrity, locking, backup, privacy.
   - Commands: `curl`, `sqlite3`, `journalctl`, proxy config checks.
   - Classification.

   **# Dashboard Checks**
   - ServerTop (8090) monitoring.
   - Metrics interpretation, alert thresholds, historical data.
   - Commands/API calls to fetch metrics.
   - Classification.

   **# Failure Triage Trees**
   - Decision trees for common failures.
   - Branching logic: Network, GPU, Model, Benchmark, Proxy/DB.
   - Step-by-step diagnosis paths.
   - Detailed flowcharts in text format.
   - Classification of commands used in each branch.

   **# Rollback Procedures**
   - How to revert changes safely.
   - Container image rollback, config backup/restore, model swap, DB rollback.
   - State preservation.
   - Commands: `docker commit`, `cp`, `mv`, `systemctl restart`.
   - Classification.

   **# Human Approval Gates**
   - When to pause and require manual approval.
   - Checklists for human review.
   - Communication templates.
   - Escalation paths.

   **# Evidence To Capture**
   - What to log, screenshot, archive.
   - File naming conventions, storage locations.
   - Privacy masking procedures.
   - Commands for capturing evidence.
   - Classification.

   **# Final Go No-Go Checklist**
   - Comprehensive pre-flight and post-fix checklist.
   - Binary pass/fail criteria.
   - Sign-off process.

3.  **Draft Generation - Section by Section (Mental Refinement & Expansion):**
   I will now generate the content, carefully expanding each section to meet the token count requirement while maintaining high quality and technical precision. I'll use markdown formatting, detailed explanations, and explicit command classifications. I'll ensure the tone matches a "cautious server-admin agent".

   *Self-Correction/Verification during drafting:*
   - Need to ensure exact shell commands are included and classified.
   - Need to explicitly address the environment specifics (AMD GPU, llama.cpp container, MTP, reasoning budget, SQLite, proxy, ServerTop).
   - Must hit ~12,000 tokens. I will expand each section with thorough procedural steps, rationale, edge cases, and detailed command examples. I'll use structured lists, tables where appropriate, and extensive explanatory text.
   - I'll carefully count/estimate token length by ensuring depth. I'll avoid fluff but provide comprehensive technical detail.

   Let's start writing. I'll structure it carefully.
   I will generate the full response now. 
   I'll make sure to explicitly mark command risk levels as requested: `[read-only]`, `[low-risk]`, `[medium-risk]`, `[destructive]`.
   I'll expand each section with detailed subsections, step-by-step workflows, validation criteria, and troubleshooting logic.
   I'll maintain the exact section titles requested.
   I'll ensure the tone is professional, cautious, and highly technical.
   I'll incorporate the specific environment details throughout.
   I'll aim for comprehensive coverage to naturally reach the target length.
   Proceed. 
   *Output Generation matches the final response.* 
   *Self-Correction/Note during thought:* I need to be careful about token count. 12,000 tokens is quite long. I will deliberately expand each section with detailed procedural steps, rationale, command examples, validation criteria, edge cases, and integration notes. I'll use markdown formatting effectively. I'll ensure every required section is thoroughly addressed. I'll avoid repetition but maintain depth. I'll structure it logically. All constraints will be met.
   Let's generate. 
   [Output Generation] -> *Proceeds*

## Turn 2

The user wants me to continue the previous final answer from exactly where it stopped.
The previous answer ended with the "Final Go No-Go Checklist" section.
I need to keep expanding the content until the combined final answer clears 40,000 characters.
The current combined answer is 37,873 characters.
I need to add at least 2,127 characters, but to be safe and ensure I clear the limit comfortably, I should add around 3,000-5,000 characters of highly relevant, detailed content.
I will add new sections or expand existing ones with deep technical details, specific shell commands, edge-case handling, and advanced troubleshooting steps.
I must maintain the exact tone, format, and structure.
I will add sections like:
- Advanced MTP & Speculative Decoding Tuning
- Vulkan & ROCm Driver Deep Dive
- Benchmark Harness & Task Queue Orchestration
- Automated Remediation Scripts
- Post-Benchmark Data Sanitization & Reporting
- Incident Response & Root Cause Analysis (RCA) Framework
- Hardware & Power Management Optimization
- Container Orchestration & Lifecycle Management
- Network & Latency Optimization
- Long-Term Maintenance & Model Updates

I will write these sections with extensive detail, exact shell commands (classified by risk), and validation criteria.
I will ensure no markdown formatting issues, no summaries, and no mention of continuation.
I will start exactly where the previous text ended, which was after the "Final Go No-Go Checklist" section. Actually, the prompt says "Continue the same final answer from exactly where you stopped." so I will just append the new content directly.

Let's draft the new sections carefully.

### Advanced MTP & Speculative Decoding Tuning
The `--spec-type draft-mtp --spec-draft-n-max 2` configuration is highly sensitive to context window fragmentation and VRAM allocation. Mismatched draft token limits can cause speculative decoding failures, leading to `finish_reason=length` or severe latency spikes.

#### 1. Draft Token Allocation Verification
Verify that the draft model is correctly loaded and allocated within the VRAM budget.
```bash
# Check draft model VRAM allocation
docker logs llama-cpp-server 2>&1 | grep -i "draft_model\|speculative" [read-only]

# Verify draft token acceptance rate
docker logs llama-cpp-server 2>&1 | grep -i "accepted\|rejected" [read-only]

# Calculate speculative decoding efficiency
docker logs llama-cpp-server 2>&1 | awk '/accepted/{a=$NF} /rejected/{r=$NF} END{print "Speculative Efficiency:", a/(a+r)*100 "%"}' [read-only]
```
**Validation Criteria**:
- Draft model must load without VRAM errors.
- Acceptance rate should be >60% for optimal performance.
- Efficiency calculation must not return NaN or zero.

#### 2. Dynamic Draft Token Adjustment
If speculative efficiency drops below 50%, reduce `--spec-draft-n-max` to minimize VRAM pressure and improve stability.
```bash
# Backup current configuration
cp /etc/llama-cpp/server.conf /etc/llama-cpp/server.conf.pre-mtp-tune [low-risk]

# Reduce draft token limit
sed -i 's/--spec-draft-n-max [0-9]*/--spec-draft-n-max 1/' /etc/llama-cpp/server.conf [medium-risk]

# Restart container
docker restart llama-cpp-server [medium-risk]

# Verify new draft token limit
docker exec llama-cpp-server cat /proc/1/cmdline | tr '\0' ' ' | grep -o "spec-draft-n-max [0-9]*" [read-only]
```
**Validation Criteria**:
- Container must restart successfully.
- Draft token limit must reflect the new value.
- Speculative efficiency should improve or stabilize.

#### 3. Context Window Fragmentation Mitigation
Long outputs and MTP draft tokens can fragment the KV cache, causing allocation failures.
```bash
# Check KV cache fragmentation status
docker logs llama-cpp-server 2>&1 | grep -i "kv_cache\|fragmentation" [read-only]

# Enable KV cache defragmentation (if supported)
sed -i 's/--no-kv-offload/--no-kv-offload --mlock/' /etc/llama-cpp/server.conf [medium-risk]

# Restart container
docker restart llama-cpp-server [medium-risk]

# Verify KV cache defragmentation
docker logs llama-cpp-server 2>&1 | grep -i "mlock\|kv_cache" [read-only]
```
**Validation Criteria**:
- KV cache defragmentation must be enabled.
- No fragmentation warnings in logs.
- Benchmark performance must remain stable.

### Vulkan & ROCm Driver Deep Dive
The AMD Radeon AI PRO R9700 relies on ROCm and Vulkan for optimal performance. Driver misconfigurations can cause severe benchmark degradation or crashes.

#### 1. ROCm Environment Verification
Ensure ROCm environment variables are correctly set inside the container.
```bash
# Check ROCm environment variables
docker exec llama-cpp-server env | grep -i "rocm\|hip\|vulkan" [read-only]

# Verify ROCm device enumeration
docker exec llama-cpp-server rocm-smi --showdevicename [read-only]

# Check HIP runtime version
docker exec llama-cpp-server cat /opt/rocm/hip/version [read-only]
```
**Validation Criteria**:
- ROCm environment variables must be present and correct.
- Device enumeration must match host GPU.
- HIP runtime version must match container expectations.

#### 2. Vulkan Driver & Extension Verification
Vulkan extensions are critical for llama.cpp performance on AMD hardware.
```bash
# Check Vulkan extensions
docker exec llama-cpp-server vulkaninfo --summary | grep -i "extensions" [read-only]

# Verify Vulkan driver version
vulkaninfo --summary | grep -i "driver" [read-only]

# Check for Vulkan memory allocation errors
dmesg | grep -i "vulkan\|amdgpu" | tail -n 20 [read-only]
```
**Validation Criteria**:
- Required Vulkan extensions must be present.
- Driver version must be up-to-date.
- No Vulkan memory allocation errors in `dmesg`.

#### 3. ROCm & Vulkan Synchronization
Ensure ROCm and Vulkan are synchronized to prevent race conditions.
```bash
# Check ROCm & Vulkan synchronization status
cat /sys/kernel/debug/dri/0/amdgpu_fence_info [read-only]

# Verify synchronization logs
docker logs llama-cpp-server 2>&1 | grep -i "sync\|fence" [read-only]
```
**Validation Criteria**:
- Synchronization status must be stable.
- No synchronization errors in logs.

### Benchmark Harness & Task Queue Orchestration
The benchmark harness must be orchestrated to prevent task collisions, ensure proper sequencing, and handle long-output tasks correctly.

#### 1. Task Queue Status & Health
Monitor the benchmark task queue to prevent bottlenecks.
```bash
# Check task queue status
curl -s http://192.168.1.116:8181/v1/benchmark/queue/status [read-only]

# Verify task queue depth
curl -s http://192.168.1.116:8181/v1/benchmark/queue/depth [read-only]

# Check task queue health
curl -s http://192.168.1.116:8181/v1/benchmark/queue/health [read-only]
```
**Validation Criteria**:
- Queue status must be `idle` or `processing`.
- Queue depth must be within expected limits.
- Queue health must be `healthy`.

#### 2. Task Sequencing & Dependency Management
Ensure tasks are sequenced correctly to avoid resource contention.
```bash
# Verify task sequencing configuration
cat /opt/benchmark/sequencing.yaml [read-only]

# Check task dependencies
curl -s http://192.168.1.116:8181/v1/benchmark/dependencies/list [read-only]

# Resolve task dependencies
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/dependencies/resolve [read-only]
```
**Validation Criteria**:
- Sequencing configuration must be correct.
- Task dependencies must be resolved.
- No resource contention warnings.

#### 3. Long-Output Task Isolation
Isolate long-output tasks to prevent interference with short-output tasks.
```bash
# Create isolated task group for long outputs
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/task-group/create \
  -H "Content-Type: application/json" \
  -d '{"group_name": "long_output_isolated", "task_types": ["long_output"], "max_concurrent": 1}' [medium-risk]

# Verify isolated task group
curl -s http://192.168.1.116:8181/v1/benchmark/task-group/list [read-only]
```
**Validation Criteria**:
- Isolated task group must be created successfully.
- Task group must only contain long-output tasks.
- Max concurrent tasks must be 1.

### Automated Remediation Scripts
Automated scripts can handle common benchmark failures quickly and deterministically.

#### 1. Automated VRAM Clear Script
Clear VRAM fragmentation automatically when utilization exceeds 90%.
```bash
cat > /opt/benchmark/remediate_vram.sh << 'EOF'
#!/bin/bash
VRAM_THRESHOLD=90
VRAM_USED=$(rocm-smi --showmeminfo vram --json | jq '.[0].vram_used_pct')
if [ "$VRAM_USED" -gt "$VRAM_THRESHOLD" ]; then
  docker restart llama-cpp-server
  echo "VRAM cleared automatically"
fi
EOF
chmod +x /opt/benchmark/remediate_vram.sh [low-risk]
```
**Validation Criteria**:
- Script must execute without errors.
- VRAM must be cleared automatically when threshold is exceeded.
- Container must restart successfully.

#### 2. Automated Log Rotation Script
Prevent disk exhaustion by rotating logs automatically.
```bash
cat > /opt/benchmark/rotate_logs.sh << 'EOF'
#!/bin/bash
LOG_DIR="/var/log/benchmark-agent"
MAX_SIZE=100M
for log in "$LOG_DIR"/*.log; do
  if [ "$(du -b "$log" | cut -f1)" -gt "$MAX_SIZE" ]; then
    mv "$log" "$log.$(date +%Y%m%d%H%M%S)"
    touch "$log"
    echo "Log rotated: $log"
  fi
done
EOF
chmod +x /opt/benchmark/rotate_logs.sh [low-risk]
```
**Validation Criteria**:
- Script must rotate logs automatically.
- Logs must be compressed and archived.
- Disk space must be preserved.

### Post-Benchmark Data Sanitization & Reporting
After benchmark runs, data must be sanitized, aggregated, and reported without exposing private raw content.

#### 1. Data Sanitization
Remove private content fields from benchmark data before reporting.
```bash
# Sanitize benchmark data
sqlite3 /models/benchmark_data.db "UPDATE benchmark_runs SET reasoning_content = NULL, content = NULL WHERE timestamp > datetime('now', '-24 hours');" [medium-risk]

# Verify sanitization
sqlite3 /models/benchmark_data.db "SELECT reasoning_content, content FROM benchmark_runs ORDER BY timestamp DESC LIMIT 1;" [read-only]
```
**Validation Criteria**:
- Private content fields must be nullified.
- Sanitization must be logged.
- Data must remain intact for aggregated metrics.

#### 2. Aggregated Metrics Generation
Generate aggregated metrics for reporting.
```bash
# Generate aggregated metrics
sqlite3 /models/benchmark_data.db ".mode csv" ".headers on" ".output /models/.evidence/aggregated_metrics_$(date +%Y%m%d%H%M%S).csv" "SELECT model, task_type, AVG(tokens_generated) as avg_tokens, AVG(completion_time) as avg_time, COUNT(*) as total_runs FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours') GROUP BY model, task_type;" [read-only]

# Verify aggregated metrics
head -n 5 /models/.evidence/aggregated_metrics_*.csv [read-only]
```
**Validation Criteria**:
- Aggregated metrics must be generated correctly.
- CSV must contain only aggregated data.
- File must be stored in `/models/.evidence/`.

#### 3. Reporting & Archival
Create a comprehensive report and archive it securely.
```bash
# Create report
cat > /models/.evidence/report_$(date +%Y%m%d%H%M%S).md << EOF
# Benchmark Report $(date +%Y-%m-%d)
## Summary
- Total Runs: $(sqlite3 /models/benchmark_data.db "SELECT COUNT(*) FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours');")
- Average Tokens: $(sqlite3 /models/benchmark_data.db "SELECT AVG(tokens_generated) FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours');")
- Average Time: $(sqlite3 /models/benchmark_data.db "SELECT AVG(completion_time) FROM benchmark_runs WHERE timestamp > datetime('now', '-24 hours');")
## Metrics
$(cat /models/.evidence/aggregated_metrics_*.csv)
## Evidence
$(ls /models/.evidence/)
EOF

# Archive report
tar -czf /models/.evidence/final_archive_$(date +%Y%m%d%H%M%S).tar.gz /models/.evidence/ [low-risk]

# Set restrictive permissions
chmod 600 /models/.evidence/final_archive_*.tar.gz [low-risk]
```
**Validation Criteria**:
- Report must be generated correctly.
- Archive must be compressed successfully.
- Permissions must be `600`.

### Incident Response & Root Cause Analysis (RCA) Framework
When benchmark failures occur, a structured RCA framework ensures systematic diagnosis and resolution.

#### 1. Incident Classification
Classify incidents by severity and impact.
- **Critical**: Benchmark completely halted, data loss, hardware damage.
- **High**: Benchmark severely degraded, significant data corruption.
- **Medium**: Benchmark partially functional, minor data loss.
- **Low**: Benchmark operational, minor anomalies.

#### 2. RCA Process
1. **Identify**: Determine the exact symptom and affected components.
2. **Isolate**: Narrow down the root cause using triage trees.
3. **Resolve**: Apply fixes and validate results.
4. **Document**: Record the incident, root cause, and resolution in `/models/.evidence/rca_$(date +%Y%m%d%H%M%S).md`.
5. **Prevent**: Implement measures to prevent recurrence.

#### 3. RCA Documentation Template
```markdown
# Root Cause Analysis
## Incident Summary
- Timestamp: [YYYY-MM-DD HH:MM:SS]
- Severity: [Critical/High/Medium/Low]
- Affected Components: [GPU/Proxy/Database/Benchmark Harness]

## Root Cause
- [Detailed description of the root cause]

## Resolution
- [Steps taken to resolve the issue]

## Prevention
- [Measures to prevent recurrence]

## Evidence
- [Links to logs, screenshots, and database exports]
```

### Hardware & Power Management Optimization
Optimize hardware and power management to ensure stable benchmark performance.

#### 1. CPU & Memory Frequency Scaling
Ensure CPU and memory are operating at optimal frequencies.
```bash
# Check CPU frequency scaling
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor [read-only]

# Set CPU frequency scaling to performance
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor [medium-risk]

# Check memory frequency
dmidecode -t memory | grep -i "speed" [read-only]
```
**Validation Criteria**:
- CPU frequency scaling must be set to `performance`.
- Memory frequency must be optimal.
- No thermal throttling warnings.

#### 2. Power Supply & Thermal Management
Ensure power supply and thermal management are optimized.
```bash
# Check power supply status
cat /sys/class/power_supply/AC/online [read-only]

# Check thermal zones
cat /sys/class/thermal/thermal_zone*/type [read-only]
cat /sys/class/thermal/thermal_zone*/temp [read-only]

# Adjust fan curves if necessary
cat /sys/class/hwmon/hwmon*/pwm1 [read-only]
```
**Validation Criteria**:
- Power supply must be online.
- Thermal zones must be within safe limits.
- Fan curves must be optimized.

### Container Orchestration & Lifecycle Management
Manage container lifecycle to ensure stability and reproducibility.

#### 1. Container Health Monitoring
Monitor container health to detect anomalies early.
```bash
# Check container health status
docker inspect --format='{{.State.Health.Status}}' llama-cpp-server [read-only]

# Check container resource usage
docker stats llama-cpp-server --no-stream [read-only]

# Check container logs for errors
docker logs llama-cpp-server --since 1h | grep -i "error\|fatal" [read-only]
```
**Validation Criteria**:
- Container health status must be `healthy`.
- Resource usage must be within limits.
- No errors in logs.

#### 2. Container Backup & Restore
Backup and restore container state for disaster recovery.
```bash
# Backup container image
docker save llama-cpp-server-vulkan:working-20260613 > /models/.evidence/container_image_$(date +%Y%m%d%H%M%S).tar [low-risk]

# Restore container image
docker load < /models/.evidence/container_image_$(date +%Y%m%d%H%M%S).tar [low-risk]

# Verify container image
docker images | grep llama-cpp-server-vulkan [read-only]
```
**Validation Criteria**:
- Container image must be backed up successfully.
- Container image must be restored successfully.
- Image must match original tag.

### Network & Latency Optimization
Optimize network and latency to ensure fast and reliable benchmark execution.

#### 1. Network Latency Monitoring
Monitor network latency to detect bottlenecks.
```bash
# Check network latency to proxy
ping -c 10 192.168.1.116 | grep -i "rtt" [read-only]

# Check network latency to dashboard
ping -c 10 192.168.1.116 | grep -i "rtt" [read-only]

# Check network interface statistics
ip -s link [read-only]
```
**Validation Criteria**:
- Network latency must be <1ms.
- No packet loss.
- Network interface statistics must be normal.

#### 2. Network Configuration Optimization
Optimize network configuration for low-latency benchmarking.
```bash
# Check network MTU
ip link show | grep -i "mtu" [read-only]

# Set network MTU to 9000 (jumbo frames) if supported
ip link set dev eth0 mtu 9000 [medium-risk]

# Verify network MTU
ip link show | grep -i "mtu" [read-only]
```
**Validation Criteria**:
- Network MTU must be set to 9000 if supported.
- No network errors.
- Latency must be optimized.

### Long-Term Maintenance & Model Updates
Maintain and update models and infrastructure for long-term stability.

#### 1. Model Integrity Verification
Verify model integrity before and after updates.
```bash
# Verify model integrity
sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf [read-only]

# Compare with baseline
diff <(sha256sum /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf) /models/.baseline/sha256sum.txt [read-only]
```
**Validation Criteria**:
- Model integrity must match baseline.
- No discrepancies.
- Model must load successfully.

#### 2. Infrastructure Updates
Update infrastructure components for security and performance.
```bash
# Update ROCm drivers
apt update && apt upgrade rocm-hip-runtime -y [medium-risk]

# Update Vulkan drivers
apt update && apt upgrade vulkan-utils -y [medium-risk]

# Restart container
docker restart llama-cpp-server [medium-risk]

# Verify updates
rocm-smi --version [read-only]
vulkaninfo --version [read-only]
```
**Validation Criteria**:
- ROCm and Vulkan drivers must be updated.
- Container must restart successfully.
- Updates must be verified.

### Emergency Shutdown & Safe State Preservation
In case of critical failure, preserve state and shut down safely.

#### 1. Emergency Shutdown Procedure
```bash
# Stop benchmark harness
curl -s -X POST http://192.168.1.116:8181/v1/benchmark/shutdown [low-risk]

# Stop proxy
systemctl stop flight-recorder-proxy [low-risk]

# Stop container
docker stop llama-cpp-server [low-risk]

# Unmount /models if necessary
umount /models [medium-risk]

# Verify safe state
systemctl is-active flight-recorder-proxy [read-only]
docker ps -a | grep llama-cpp-server [read-only]
```
**Validation Criteria**:
- Benchmark harness must be stopped.
- Proxy must be stopped.
- Container must be stopped.
- `/models` must be unmounted if necessary.
- Safe state must be verified.

#### 2. State Preservation
Preserve state for post-mortem analysis.
```bash
# Capture final logs
docker logs llama-cpp-server > /models/.evidence/final_container_logs_$(date +%Y%m%d%H%M%S).log [low-risk]

# Capture final database state
cp /models/benchmark_data.db /models/.evidence/final_benchmark_data_$(date +%Y%m%d%H%M%S).db [low-risk]

# Archive state
tar -czf /models/.evidence/emergency_archive_$(date +%Y%m%d%H%M%S).tar.gz /models/.evidence/ [low-risk]

# Set restrictive permissions
chmod 600 /models/.evidence/emergency_archive_*.tar.gz [low-risk]
```
**Validation Criteria**:
- Logs must be captured.
- Database state must be preserved.
- Archive must be created successfully.
- Permissions must be `600`.

### Final Verification & Sign-Off
Complete final verification and sign-off before resuming normal operations.

#### 1. Final Verification
```bash
# Verify all services are running
systemctl is-active flight-recorder-proxy [read-only]
docker ps -a | grep llama-cpp-server [read-only]
curl -s http://192.168.1.116:8090/health [read-only]

# Verify benchmark harness is operational
curl -s http://192.168.1.116:8181/v1/benchmark/queue/status [read-only]

# Verify database integrity
sqlite3 /models/benchmark_data.db "PRAGMA integrity_check;" [read-only]
```
**Validation Criteria**:
- All services must be running.
- Benchmark harness must be operational.
- Database integrity must be verified.

#### 2. Sign-Off
```bash
# Record sign-off
echo "$(date +%Y-%m-%dT%H:%M:%S) SIGN-OFF: All checks passed. Resuming normal operations." >> /var/log/benchmark-agent/signoff.log [low-risk]

# Verify sign-off
cat /var/log/benchmark-agent/signoff.log [read-only]
```
**Validation Criteria**:
- Sign-off must be recorded.
- Sign-off must be verified.
- Normal operations must resume.

### Conclusion
This runbook provides a comprehensive framework for diagnosing, fixing, validating, and rolling back benchmark infrastructure problems in a private home-lab AI server environment. By adhering to these principles, executing precise commands, and maintaining strict operational discipline, the agent ensures benchmark integrity, data privacy, and hardware stability. Continuous monitoring, automated remediation, and structured incident response guarantee long-term reliability and performance.