## Turn 1

Here's a thinking process:

1.  **Analyze User Input:**
   - **Role:** Cautious server-admin agent for a home-lab AI server.
   - **Environment:**
     - Host: 192.168.1.116
     - GPU: AMD Radeon AI PRO R9700, 32GB VRAM
     - Container: `llama-cpp-server-vulkan:working-20260613`
     - Model: `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`
     - Context: 262144
     - Parallel slots: 1
     - MTP: `--spec-type draft-mtp --spec-draft-n-max 2`
     - Reasoning: `--reasoning-budget 8192`
     - Flight Recorder proxy: `http://192.168.1.116:8181/v1`
     - ServerTop: `http://192.168.1.116:8090`
   - **Observations:**
     - `finish_reason=length` before final content appeared.
     - Increasing `max_tokens` fixed it.
     - Long-output tests must not run in bulk without validation.
     - SQLite DB on `/models`, local/private.
     - User wants screenshots/reliable reporting, no public release of raw data.
   - **Task:** Build a complete agent runbook for diagnosing, fixing, validating, and rolling back benchmark infrastructure problems.
   - **Length Requirement:** Roughly 10,000 to 14,000 tokens. Target ~12,000.
   - **Required Sections:**
     1. # Operating Principles
     2. # Preflight Checks
     3. # Safe Inspection Commands
     4. # Reasoning Budget Verification
     5. # Long Output Validation
     6. # GPU And VRAM Checks
     7. # Proxy And Database Checks
     8. # Dashboard Checks
     9. # Failure Triage Trees
     10. # Rollback Procedures
     11. # Human Approval Gates
     12. # Evidence To Capture
     13. # Final Go No-Go Checklist
   - **Constraints:**
     - Include exact shell commands, classified as read-only, low-risk, medium-risk, or destructive.
     - No real passwords/secrets.
     - Do not stop after a compact overview; expand thoroughly.
     - Prioritize complete, useful, well-structured content.
     - Keep final answer in `message.content`.

2.  **Deconstruct & Plan Content (Mental Outline & Token Estimation):**
   To reach ~12,000 tokens, I need to be extremely detailed, providing comprehensive explanations, step-by-step procedures, rationale, edge cases, troubleshooting paths, and explicit command classifications for every section. I'll structure each section with subsections, detailed explanations, command blocks with classifications, and operational notes.

   *Section 1: Operating Principles* (~800 tokens)
   - Core philosophy: caution, reproducibility, data privacy, incremental validation.
   - Rules of engagement: read-only first, explicit approval for changes, no bulk runs without validation, local-only data handling.
   - Documentation & audit trail requirements.

   *Section 2: Preflight Checks* (~1000 tokens)
   - Network connectivity, container status, disk space, GPU availability, model file integrity.
   - Step-by-step verification before any benchmark run.
   - Commands with classifications.

   *Section 3: Safe Inspection Commands* (~1200 tokens)
   - Comprehensive list of read-only/low-risk commands for system, container, network, and process inspection.
   - Detailed explanations of what each command reveals and how to interpret output.
   - Classification tags for every command.

   *Section 4: Reasoning Budget Verification* (~1000 tokens)
   - Understanding `--reasoning-budget 8192` and `finish_reason=length`.
   - How to test reasoning budget allocation vs. generation tokens.
   - Commands to verify budget consumption, logs to check, and adjustments.
   - Classification tags.

   *Section 5: Long Output Validation* (~1200 tokens)
   - Addressing the `finish_reason=length` issue.
   - Strategy for incremental `max_tokens` testing.
   - Validation scripts/approaches for single-task runs.
   - Handling MTP/speculative decoding interactions with long outputs.
   - Classification tags.

   *Section 6: GPU And VRAM Checks* (~1000 tokens)
   - AMD Radeon AI PRO R9700 specifics (ROCm/Vulkan).
   - VRAM monitoring, thermal checks, driver validation.
   - Commands for `rocm-smi`, `vulkaninfo`, container GPU passthrough verification.
   - Classification tags.

   *Section 7: Proxy And Database Checks* (~1000 tokens)
   - Flight Recorder proxy (`:8181`) health, routing, latency.
   - SQLite DB on `/models` integrity, backup, schema validation.
   - Commands for `sqlite3`, `curl`, `journalctl`.
   - Classification tags.

   *Section 8: Dashboard Checks* (~800 tokens)
   - ServerTop (`:8090`) validation, metrics extraction, screenshot capture procedures.
   - Ensuring dashboard reflects actual container state.
   - Classification tags.

   *Section 9: Failure Triage Trees* (~1200 tokens)
   - Decision trees for common failures: OOM, VRAM leak, proxy timeout, DB corruption, `finish_reason=length`, MTP divergence.
   - Step-by-step diagnostic paths with commands and expected outcomes.
   - Classification tags.

   *Section 10: Rollback Procedures* (~1000 tokens)
   - Container rollback, model rollback, config rollback, DB rollback.
   - Step-by-step safe rollback with verification.
   - Classification tags.

   *Section 11: Human Approval Gates* (~800 tokens)
   - When to pause and ask for human confirmation.
   - Template for approval requests.
   - Risk thresholds for each gate.

   *Section 12: Evidence To Capture* (~800 tokens)
   - What to log, screenshot, archive.
   - Naming conventions, storage paths, privacy redaction steps.
   - Classification tags.

   *Section 13: Final Go No-Go Checklist* (~600 tokens)
   - Comprehensive pre-run checklist.
   - Sign-off procedure.

   Total estimated: ~12,600 tokens. I will expand each section with thorough technical detail, operational context, explicit command classifications, and procedural rigor to meet the length and quality requirements.

3.  **Draft Generation - Section by Section (Mental Refinement & Expansion):**
   *I will write out the full content carefully, ensuring each section is detailed, commands are classified, and the tone matches a cautious server-admin agent. I'll monitor token count implicitly by being comprehensive.*

   **Operating Principles:** Focus on safety, reproducibility, privacy, incremental validation, explicit risk classification, audit trails, and conservative defaults. Explain why each principle matters in this specific home-lab context.

   **Preflight Checks:** Network, container, disk, GPU, model hash, environment variables. Provide exact commands. Classify each. Explain expected outputs.

   **Safe Inspection Commands:** Group by category (System, Container, Network, Process, Storage). Provide commands like `df -h`, `docker ps`, `curl -s`, `ls -la`, `cat /proc/...`. Classify each. Explain interpretation.

   **Reasoning Budget Verification:** Explain how `--reasoning-budget 8192` interacts with `max_tokens` and `finish_reason=length`. Provide commands to check logs, test budget allocation, verify token accounting. Classify commands. Detail troubleshooting steps.

   **Long Output Validation:** Address the specific observation. Provide a methodology for single-task validation, incremental `max_tokens` scaling, MTP interaction checks, and output completeness verification. Classify commands. Provide validation scripts/logic.

   **GPU And VRAM Checks:** AMD-specific checks. `rocm-smi`, `vulkaninfo`, container device mapping, VRAM allocation vs. model size (35B A3B ~ 16-20GB VRAM depending on quantization, leaving headroom). Classify commands. Explain thermal/power monitoring.

   **Proxy And Database Checks:** Flight recorder proxy health, routing verification, SQLite integrity (`PRAGMA integrity_check`), backup procedures, local-only enforcement. Classify commands. Explain log analysis.

   **Dashboard Checks:** ServerTop validation, metric extraction, screenshot capture workflow, ensuring dashboard sync with container state. Classify commands. Explain interpretation of metrics.

   **Failure Triage Trees:** Decision trees for: 1. `finish_reason=length` premature cutoff, 2. VRAM OOM, 3. Proxy timeout/routing failure, 4. SQLite corruption, 5. MTP speculative decoding divergence, 6. Container crash/restart loop. Each tree has steps, commands, classifications, and resolution paths.

   **Rollback Procedures:** Container image rollback, model file rollback, configuration rollback, database rollback. Step-by-step with verification. Classify commands. Emphasize safety.

   **Human Approval Gates:** Define thresholds for intervention. Provide templates. Explain risk assessment matrix. Classify actions requiring approval.

   **Evidence To Capture:** Logging standards, screenshot procedures, archive naming, privacy redaction, storage paths. Classify commands. Explain retention policy.

   **Final Go No-Go Checklist:** Comprehensive pre-run checklist. Sign-off procedure. Classification of final validation steps.

   *Self-Correction/Verification during drafting:*
   - Check token count: I need to ensure it's substantial. I'll expand explanations, add edge cases, provide detailed command outputs interpretation, and include operational notes for each section.
   - Check constraints: All commands must be classified. No secrets. Required sections present. Tone: cautious server-admin. Environment details integrated.
   - I will systematically generate the content, ensuring depth and adherence to the prompt.

   Let's write it out carefully. I'll structure it with markdown headings as requested. I'll make sure each command block has a clear classification tag. I'll expand on the technical rationale for each step to meet the length requirement naturally without fluff. I'll focus on precision, safety, and operational rigor.
   *Proceeds to generate response.* 
   *Self-Correction/Note during thought:* I need to be careful to actually hit the ~12k token range. I will deliberately expand each section with comprehensive operational procedures, detailed command explanations, expected outputs, troubleshooting paths, and safety rationale. I will avoid repetition but ensure thoroughness. I'll structure it meticulously.
   Let's begin. 
   *Output Generation matches the final response.* 
   *Structure check:*
   # Operating Principles
   # Preflight Checks
   # Safe Inspection Commands
   # Reasoning Budget Verification
   # Long Output Validation
   # GPU And VRAM Checks
   # Proxy And Database Checks
   # Dashboard Checks
   # Failure Triage Trees
   # Rollback Procedures
   # Human Approval Gates
   # Evidence To Capture
   # Final Go No-Go Checklist
   All present. Commands classified. Environment integrated. Length targeted. Tone cautious. Ready. 
   *Output Generation* (proceeds)

## Turn 2

Here's a thinking process:

1.  **Analyze User Input:**
   - **Context:** The user provided a detailed prompt for a "cautious server-admin agent" runbook for a home-lab AI server.
   - **Requirements:** 
     - Environment details provided (Host, GPU, container, model, context, parallel slots, MTP, reasoning budget, proxy, dashboard).
     - Observations provided (finish_reason=length truncation, max_tokens fix, no bulk runs, SQLite local/private, screenshots/reliable reporting, no public raw data).
     - Required sections: Operating Principles, Preflight Checks, Safe Inspection Commands, Reasoning Budget Verification, Long Output Validation, GPU And VRAM Checks, Proxy And Database Checks, Dashboard Checks, Failure Triage Trees, Rollback Procedures, Human Approval Gates, Evidence To Capture, Final Go No-Go Checklist.
     - Command classification required (read-only, low-risk, medium-risk, destructive).
     - No real passwords/secrets.
     - Target length: ~12,000 tokens (roughly 40,000-50,000 characters). The prompt says "The combined final answer is currently 35108 characters; keep expanding until it clears 40000 characters."
     - Must continue exactly where I stopped, without restarting, summarizing, or mentioning continuation mechanics.
     - Keep the final answer in `message.content`.

2.  **Assess Current State:**
   - I generated a comprehensive runbook covering all required sections.
   - The user says it's currently 35,108 characters and wants it expanded to >40,000 characters.
   - I need to continue seamlessly from the end of my previous output ("This runbook provides a complete, cautious, and auditable framework for diagnosing, fixing, validating, and rolling back benchmark infrastructure problems. Adherence ensures stability, privacy, and reliable reporting. Execute with precision. Verify at every step. Never assume. Always validate.")
   - I must add substantial, relevant content that fits the runbook's theme, expands on existing sections, or adds new operational procedures, troubleshooting deep-dives, compliance/privacy protocols, automation scripts, and advanced validation methodologies, all while maintaining the exact tone and structure.
   - I will add sections like:
     - Advanced MTP & Speculative Decoding Diagnostics
     - KV Cache & Context Window Management
     - Automated Validation Scripts & Cron Jobs
     - Data Sanitization & Export Protocols
     - Incident Response & Escalation Matrix
     - Performance Baseline & Regression Testing
     - Container Image & Dependency Management
     - Network & Firewall Hardening
     - Operator Training & Shift Handover Procedures
   - I will ensure all commands are classified, no secrets are included, and the tone remains cautious/administrative.
   - I will carefully count/estimate characters to ensure it pushes past 40,000.

3.  **Draft - Section by Section (Mental Refinement & Expansion):**
   *I'll continue directly from the last sentence, adding new detailed sections that logically follow the runbook structure.*

   **Advanced MTP & Speculative Decoding Diagnostics**
   - Deep dive into `--spec-type draft-mtp` and `--spec-draft-n-max 2`
   - How to measure draft vs verify latency
   - Token acceptance rate thresholds
   - Commands to isolate MTP bottlenecks
   - Risk classifications

   **KV Cache & Context Window Management**
   - 262144 context implications on 32GB VRAM
   - KV cache quantization strategies
   - Eviction policies
   - Monitoring KV cache fragmentation
   - Commands for cache inspection

   **Automated Validation Scripts & Cron Jobs**
   - Bash scripts for continuous health checks
   - Cron scheduling for backups, integrity checks, log rotation
   - Safety guards in automation
   - Commands to deploy/verify cron jobs

   **Data Sanitization & Export Protocols**
   - How to safely export benchmark results without leaking prompts/responses
   - Hashing, truncation, metadata stripping
   - SQLite export commands with sanitization
   - Risk classifications

   **Incident Response & Escalation Matrix**
   - Severity levels (P1-P4)
   - Response timelines
   - Communication templates
   - Rollback triggers per severity

   **Performance Baseline & Regression Testing**
   - Establishing baselines for tokens/sec, latency, VRAM usage
   - Regression detection thresholds
   - A/B testing configurations safely
   - Commands for benchmarking baselines

   **Container Image & Dependency Management**
   - Pinning image tags, verifying digests
   - Updating safely without breaking Vulkan/ROCm
   - Layer inspection commands
   - Rollback to previous image

   **Network & Firewall Hardening**
   - Restricting 8181, 8090, 8080 to LAN/localhost
   - iptables/nftables rules
   - TLS/HTTPS considerations (even if internal)
   - Commands to verify firewall state

   **Operator Training & Shift Handover Procedures**
   - Documentation standards
   - Handover checklist
   - Knowledge transfer protocols
   - Simulation drills

   I will write these out in detail, ensuring technical depth, exact command classifications, and adherence to the cautious admin persona. I'll monitor length to ensure it exceeds 40k characters total.

4.  **Writing Process (Mental to Text Generation):**
   *(I will generate the continuation directly, maintaining formatting and tone. I'll be thorough and expansive.)*

   [Start of continuation]
   **Advanced MTP & Speculative Decoding Diagnostics**
   Multi-Token Prediction (MTP) speculative decoding significantly accelerates inference but introduces complex token accounting and divergence risks. When `--spec-type draft-mtp` and `--spec-draft-n-max 2` are active, the model generates draft tokens that must be verified by the main model. Divergence occurs when draft tokens deviate from the main model's probability distribution, causing rejections and wasted compute. Proper diagnostics require isolating draft latency, verification latency, and acceptance rates.

   **Measuring Draft vs. Verify Latency**
   The llama.cpp server logs speculative decoding metrics per request. To isolate bottlenecks:
   ```bash
   # Classification: read-only
   docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") 2>&1 | grep -E "speculative|draft|verify|acceptance" | tail -n 50
   ```
   Expected output includes `draft_tokens`, `verify_tokens`, `acceptance_rate`, and `speculative_latency_ms`. If `speculative_latency_ms` exceeds 40% of total generation time, the draft model is overburdened. Reduce `--spec-draft-n-max` to 1 or switch to `--spec-type none` temporarily.

   **Token Acceptance Rate Thresholds**
   - **> 80%**: Optimal. MTP is accelerating inference without significant divergence.
   - **60% - 80%**: Acceptable. Monitor for context-dependent drops.
   - **40% - 60%**: Suboptimal. Draft model is struggling with prompt complexity or domain mismatch. Consider reducing draft depth or warming up the KV cache.
   - **< 40%**: Critical. MTP is causing more overhead than acceleration. Disable speculative decoding immediately.

   **Isolating MTP Bottlenecks**
   Run a controlled benchmark with MTP enabled vs. disabled:
   ```bash
   # Classification: low-risk
   curl -s http://192.168.1.116:8181/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf","messages":[{"role":"user","content":"Analyze the following dataset structure and propose optimizations."}],"max_tokens":1024,"reasoning_budget":4096}' | jq -r '{tokens: .usage.total_tokens, time: .usage.total_time_ms, finish: .finish_reason}'
   ```
   Compare `time` and `tokens` against a baseline run with `--spec-type none`. If MTP increases latency by >15%, the draft model is misaligned. Document findings and adjust `--spec-draft-n-max` accordingly.

   **KV Cache & Context Window Management**
   A context window of 262,144 tokens on a 32GB VRAM GPU requires aggressive KV cache management. The KV cache stores key and value vectors for every token in the context. At 35B parameters (A3B quantization), the KV cache can consume 8-12GB VRAM at full context. Combined with model weights (~16GB) and MTP draft buffers, VRAM headroom becomes critical.

   **KV Cache Quantization Strategies**
   llama.cpp supports KV cache quantization (`--cache-type q4_0`, `q8_0`, `f16`). Lower precision reduces VRAM usage but may impact long-context accuracy.
   ```bash
   # Classification: read-only
   docker inspect $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") --format '{{.Config.Cmd}}' | grep -o "cache-type [a-z0-9_]*"
   ```
   Expected: `cache-type q4_0` or `q8_0`. If unset, defaults to `f16`, which may cause VRAM OOM at 262k context. If VRAM pressure is observed, switch to `q4_0` via container restart.

   **Monitoring KV Cache Fragmentation**
   Long-running sessions with variable context lengths cause KV cache fragmentation, leading to inefficient memory allocation and increased latency.
   ```bash
   # Classification: read-only
   docker exec $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") cat /proc/1/smaps | grep -A 5 "Vulkan\|ROCm\|GPU" | head -n 20
   ```
   Look for fragmented memory regions. If fragmentation exceeds 20% of allocated VRAM, schedule a container restart during maintenance windows to reclaim contiguous memory blocks.

   **Eviction Policies & Context Truncation**
   When context approaches 262,144 tokens, the server must evict older tokens or truncate. llama.cpp uses a sliding window or static eviction policy. Verify configuration:
   ```bash
   # Classification: read-only
   docker logs $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") 2>&1 | grep -i "evict\|truncate\|context" | tail -n 10
   ```
   Ensure `--ctx-size` matches the intended limit. Mismatched values cause silent truncation or generation failures. Document eviction behavior for benchmark reproducibility.

   **Automated Validation Scripts & Cron Jobs**
   Manual checks are insufficient for continuous operation. Automated scripts provide consistent monitoring, but must be guarded against runaway processes or destructive actions.

   **Health Check Script**
   Create `/opt/scripts/health_check.sh`:
   ```bash
   #!/bin/bash
   # Classification: low-risk
   CONTAINER=$(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613")
   if [ -z "$CONTAINER" ]; then
       echo "CRITICAL: Container not running" | mail -s "LLAMA-CPP ALERT" admin@localhost
       exit 1
   fi
   VRAM_FREE=$(rocm-smi --showmeminfo.vram_free --json | jq -r '.[0].vram_free')
   if [ "$VRAM_FREE" -lt 2048 ]; then
       echo "WARNING: VRAM headroom below 2GB" | mail -s "LLAMA-CPP VRAM ALERT" admin@localhost
   fi
   DB_CHECK=$(sqlite3 /models/benchmark.db "PRAGMA integrity_check;")
   if [ "$DB_CHECK" != "ok" ]; then
       echo "CRITICAL: Database corruption detected" | mail -s "LLAMA-CPP DB ALERT" admin@localhost
   fi
   ```
   Make executable: `chmod +x /opt/scripts/health_check.sh`

   **Cron Scheduling**
   ```bash
   # Classification: low-risk
   crontab -l | grep -v "llama-cpp" > /tmp/cron_temp
   echo "*/15 * * * * /opt/scripts/health_check.sh >> /var/log/llama-health.log 2>&1" >> /tmp/cron_temp
   echo "0 2 * * * cp /models/benchmark.db /models/backups/benchmark_$(date +\%Y\%m\%d).db" >> /tmp/cron_temp
   crontab /tmp/cron_temp
   ```
   Verify: `crontab -l`. Ensure no overlapping jobs or resource-intensive tasks run during peak benchmark hours.

   **Safety Guards in Automation**
   - Never automate `docker restart` or `VACUUM` without explicit approval gates.
   - Log all automated actions to `/var/log/llama-automation.log`.
   - Implement circuit breakers: if health checks fail 3 consecutive times, halt automation and alert operator.

   **Data Sanitization & Export Protocols**
   Benchmark results must be exportable for reporting without exposing private prompts, responses, or internal metadata. Raw data remains strictly local.

   **Sanitization Pipeline**
   1. Extract required fields: `run_id`, `model`, `max_tokens`, `reasoning_budget`, `finish_reason`, `usage.total_tokens`, `latency_ms`.
   2. Hash prompt/response pairs: `sha256sum` or `blake3` for deduplication and privacy.
   3. Strip internal IPs, hostnames, and container IDs.
   4. Export to CSV/JSON with metadata stripped.

   **SQLite Export Command**
   ```bash
   # Classification: low-risk
   sqlite3 /models/benchmark.db "SELECT run_id, model, max_tokens, reasoning_budget, finish_reason, usage, latency_ms FROM runs WHERE status='completed' AND timestamp > datetime('now', '-7 days');" | sed 's/192\.168\.[0-9]*\.[0-9]*/LAN_IP/g' > /tmp/sanitized_export.csv
   ```
   Verify output: `head -n 5 /tmp/sanitized_export.csv`. Ensure no raw prompts/responses or internal paths remain. Archive to `/models/exports/` with timestamp.

   **Incident Response & Escalation Matrix**
   Structured response ensures rapid containment and recovery.

   **Severity Levels**
   - **P1 (Critical)**: Container crash, VRAM OOM, database corruption, data leak. Response: Immediate halt, rollback, operator notification within 5 minutes.
   - **P2 (High)**: `finish_reason=length` truncation > 20% of runs, proxy timeouts, thermal throttling. Response: Isolate fault, adjust configuration, notify operator within 15 minutes.
   - **P3 (Medium)**: MTP acceptance rate < 60%, dashboard metric drift, log rotation failure. Response: Schedule maintenance, document workaround, notify operator within 1 hour.
   - **P4 (Low)**: Minor latency spikes, cosmetic dashboard issues, non-critical warnings. Response: Log for trend analysis, address in next maintenance window.

   **Response Templates**
   ```
   INCIDENT: [P1-P4] [Brief Description]
   DETECTED: [Timestamp]
   IMPACT: [Affected subsystems]
   ACTION: [Immediate steps taken]
   ROLLBACK: [If applicable]
   STATUS: [Open/Investigating/Resolved]
   ```
   Maintain incident log in `/models/incidents/`. Review weekly for pattern analysis.

   **Performance Baseline & Regression Testing**
   Establish baselines to detect degradation from configuration drift, driver updates, or hardware aging.

   **Baseline Metrics**
   - Tokens/sec (draft vs. verify)
   - Latency (first token, total)
   - VRAM peak usage
   - MTP acceptance rate
   - Database write latency

   **Regression Detection**
   ```bash
   # Classification: read-only
   sqlite3 /models/benchmark.db "SELECT AVG(latency_ms), AVG(usage.total_tokens) FROM runs WHERE timestamp > datetime('now', '-24 hours') GROUP BY model;"
   ```
   Compare against 7-day rolling average. If latency increases > 10% or tokens/sec drops > 15%, trigger P2 investigation. Check driver versions, container image tags, and thermal logs.

   **A/B Testing Configurations**
   Never modify production configuration directly. Use isolated test containers:
   ```bash
   # Classification: medium-risk
   docker run -d --name llama-ab-test --gpus all -v /models:/models -e LLAMA_VULKAN=1 llama-cpp-server-vulkan:working-20260613 --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --ctx-size 262144 --parallel 1 --reasoning-budget 10240 --spec-type draft-mtp --spec-draft-n-max 1
   ```
   Run controlled benchmarks against production. Compare metrics. If test outperforms, schedule production update with human approval. Remove test container immediately after validation.

   **Container Image & Dependency Management**
   Pin image tags to specific digests to prevent silent updates. Verify integrity before deployment.

   **Image Verification**
   ```bash
   # Classification: read-only
   docker image inspect llama-cpp-server-vulkan:working-20260613 --format '{{.RepoDigests}}'
   ```
   Record digest in `/opt/config/image_digests.txt`. Before any update, compare new digest against known-good registry. Never pull `latest` or untagged images.

   **Dependency Audit**
   ```bash
   # Classification: read-only
   docker exec $(docker ps -q --filter "ancestor=llama-cpp-server-vulkan:working-20260613") ldconfig -p | grep -E "vulkan|rocm|cuda"
   ```
   Verify shared libraries match expected versions. Mismatched libraries cause runtime crashes or silent fallbacks. Document versions in `/opt/config/dependency_audit.log`.

   **Network & Firewall Hardening**
   Restrict access to internal services. Prevent unauthorized access or data exfiltration.

   **Firewall Rules**
   ```bash
   # Classification: medium-risk
   iptables -A INPUT -p tcp --dport 8181 -s 192.168.1.0/24 -j ACCEPT
   iptables -A INPUT -p tcp --dport 8181 -j DROP
   iptables -A INPUT -p tcp --dport 8090 -s 192.168.1.0/24 -j ACCEPT
   iptables -A INPUT -p tcp --dport 8090 -j DROP
   iptables -A INPUT -p tcp --dport 8080 -s 127.0.0.1 -j ACCEPT
   iptables -A INPUT -p tcp --dport 8080 -j DROP
   ```
   Verify: `iptables -L -n -v`. Ensure no open ports to external networks. Save rules: `iptables-save > /etc/iptables/rules.v4`.

   **TLS/HTTPS Considerations**
   Even on LAN, encrypt traffic to prevent sniffing. Use self-signed certificates for proxy and dashboard.
   ```bash
   # Classification: low-risk
   openssl req -x509 -newkey rsa:4096 -keyout /etc/ssl/private/server.key -out /etc/ssl/certs/server.crt -days 365 -nodes -subj "/CN=192.168.1.116"
   ```
   Update proxy and dashboard configs to use TLS. Verify with `curl -k https://192.168.1.116:8181/v1/models`.

   **Operator Training & Shift Handover Procedures**
   Consistent operations require standardized knowledge transfer and documentation.

   **Handover Checklist**
   - [ ] Current container status and uptime
   - [ ] Active benchmark queue and validation rate
   - [ ] Recent incidents and resolutions
   - [ ] Pending configuration changes
   - [ ] Backup status and integrity
   - [ ] Known issues and workarounds
   - [ ] Next maintenance window schedule

   **Documentation Standards**
   - All changes logged in `/opt/logs/changes.log` with timestamp, operator, justification, and rollback path.
   - Screenshots archived with metadata (date, time, metric values, redaction status).
   - Scripts version-controlled in local Git repository.
   - Runbook updated quarterly or after major incidents.

   **Simulation Drills**
   Conduct monthly failure simulations:
   1. Simulate VRAM OOM by running max-context prompt. Verify rollback procedure.
   2. Simulate database lock by concurrent writes. Verify recovery.
   3. Simulate proxy timeout by stopping backend. Verify alerting and failover.
   Document drill results and update runbook accordingly.

   **Continuous Improvement Loop**
   Benchmark infrastructure evolves. Regularly review:
   - Token generation efficiency vs. VRAM usage
   - MTP acceptance trends across prompt types
   - Database growth rate and retention policies
   - Driver updates and compatibility matrices
   - Security patches and vulnerability scans
   Maintain a living document. Archive obsolete procedures. Train operators on new protocols. Ensure all changes align with operating principles.

   This expanded runbook provides comprehensive coverage for advanced diagnostics, automation, security, incident response, and operational continuity. Execute with discipline. Validate relentlessly. Protect data. Maintain stability. The infrastructure is only as reliable as its operators. Adhere to these procedures without deviation. Document everything. Verify before acting. Rollback when uncertain. The benchmark harness demands precision, and this runbook delivers it.
   [End of continuation]

5.  **Character Count Check:**
   I need to ensure the total output (original + continuation) exceeds 40,000 characters. The original was ~35,108. The continuation I just drafted is substantial. Let's estimate:
   - Advanced MTP: ~1,500 chars
   - KV Cache: ~1,200 chars
   - Automated Scripts: ~1,800 chars
   - Data Sanitization: ~1,000 chars
   - Incident Response: ~1,200 chars
   - Performance Baseline: ~1,500 chars
   - Container/Image: ~1,000 chars
   - Network/Firewall: ~1,200 chars
   - Operator Training: ~1,000 chars
   - Continuous Improvement: ~800 chars
   Total addition: ~12,200 characters.
   35,108 + 12,200 = ~47,300 characters. This comfortably clears the 40,000 threshold.
   I will now output the continuation exactly as drafted, ensuring seamless integration with the previous text. I will maintain the exact formatting and tone. No extra commentary. Just the continuation. All commands classified. No secrets. Matches requirements. Proceeds. 
   One minor adjustment: I'll make sure the formatting matches the original exactly (markdown headers, code blocks, classifications). I'll output it now. 
   Self-Correction/Verification during thought: 
   The prompt says "Continue the same final answer from exactly where you stopped." I will paste the continuation directly. I'll ensure the markdown headers align with the existing structure. I'll double-check that all commands have the classification comment. I'll verify no passwords/secrets. I'll ensure the tone is consistent. The length requirement is met. All good. Output matches the drafted text.✅
   Proceeds. 
   [Output Generation] -> *Proceeds to generate the exact continuation text.* 
   *(Note: I will carefully format it to match the prompt's request exactly.)*
   All steps verified. Output matches response. 
   [Done.] 
   *Self-Correction/Refinement during generation prep*: I notice the prompt says "The combined final answer is currently 35108 characters; keep expanding until it clears 40000 characters." I will make sure the continuation is dense and technical, adding necessary operational depth without fluff. I'll structure it with clear headers matching the runbook style. I'll ensure every command has the classification tag. I'll output it now. 
   *Output Generation* (matches the drafted continuation exactly). 
   [Proceeds]