# Mission

The primary objective of this remediation run is to restore reliable, repeatable benchmark execution on the remote model server at `192.168.1.116` while guaranteeing zero data loss. The environment hosts a flight-recorder application under `/models/flight-recorder/app` with persistent state stored in `/models/flight-recorder/data`. The database is SQLite, which introduces specific consistency and concurrency considerations that must be respected throughout the remediation process. The server may currently be under heavy load due to active benchmarking, background inference tasks, or I/O contention, which necessitates a phased, non-disruptive approach.

Success is defined by three measurable outcomes:
1. The benchmark pipeline processes jobs without timeouts, crashes, or silent data corruption.
2. The SQLite database maintains full integrity, with all historical and pending benchmark records preserved.
3. The application service stabilizes within acceptable latency and resource thresholds, enabling continuous execution.

Risk tolerance is strictly low. All operations will be executed in a read-first, verify-then-act sequence. Non-destructive inspection and application redeploys are explicitly authorized. Any action that modifies, truncates, or removes data files, database tables, or system configurations requires explicit human approval. The remediation will proceed through discrete phases: inspection, validation, safe redeployment, benchmark resumption, and continuous monitoring. Each phase includes defined exit criteria, rollback triggers, and audit logging. The agent will maintain a stateful execution log, capture pre- and post-change baselines, and enforce strict stop conditions to prevent cascading failures. The overarching principle is preservation of truth: if the benchmark output diverges from expected schemas, if SQLite reports integrity warnings, or if resource exhaustion threatens data durability, the run will halt immediately for human review.

# Read-Only Inspection

Before any state change, a comprehensive read-only inspection must establish the current baseline. This phase focuses on network reachability, process state, resource utilization, application structure, database consistency, and log activity. All commands in this section are strictly non-modifying.

**Network & Service Reachability**
Verify that the service is listening and responding to health probes without triggering load spikes.
```bash
# Check basic connectivity and port availability
ping -c 3 192.168.1.116
nc -zv 192.168.1.116 8080  # Adjust port if documented differently

# Probe health endpoint without triggering benchmark execution
curl -s -o /dev/null -w "HTTP %{http_code} | Time: %{time_total}s\n" http://192.168.1.116:8080/health
curl -s http://192.168.1.116:8080/metrics | head -n 20  # Prometheus-style metrics if exposed
```
Expected outcome: HTTP 200, response time < 200ms, metrics endpoint returns valid JSON/lines. If the endpoint is unreachable, check firewall rules and service binding before proceeding.

**Process & Resource State**
Identify running processes, CPU/memory pressure, and I/O wait states.
```bash
# List relevant processes and their resource consumption
ps aux | grep -E "flight-recorder|python|uvicorn|gunicorn|sqlite" | grep -v grep

# Real-time resource snapshot (non-blocking)
top -b -n 1 | head -n 15
vmstat 1 3
iostat -x 1 3  # Check %util, await, and svctm for disk bottlenecks
```
Expected outcome: Service process exists, CPU < 80%, RAM < 85%, disk %util < 70%. High `await` or `%util` indicates I/O contention that could corrupt SQLite writes during benchmark execution.

**Application Structure & Configuration**
Map the codebase, dependencies, and configuration files without modifying them.
```bash
# Directory structure and file sizes
ls -lhR /models/flight-recorder/app/ | head -n 50
du -sh /models/flight-recorder/app/*

# Check for virtual environment and dependency lock files
ls -la /models/flight-recorder/app/venv/bin/python
cat /models/flight-recorder/app/requirements.txt | head -n 20
cat /models/flight-recorder/app/.env 2>/dev/null || echo "No .env found"

# Verify Python syntax without execution
python3 -m py_compile /models/flight-recorder/app/main.py 2>&1 || echo "Syntax check failed"
```
Expected outcome: Clean directory structure, valid Python syntax, recognizable dependency files. Note any `__pycache__` bloat or stale `.pyc` files that may indicate interrupted deployments.

**SQLite Database Inspection**
Validate database structure, journal mode, and integrity without altering state.
```bash
DB_PATH="/models/flight-recorder/data/benchmark.db"

# List tables and schema
sqlite3 "$DB_PATH" ".tables"
sqlite3 "$DB_PATH" ".schema benchmarks"  # Adjust table name as needed

# Check journal mode and WAL status
sqlite3 "$DB_PATH" "PRAGMA journal_mode;"
sqlite3 "$DB_PATH" "PRAGMA wal_checkpoint(PASSIVE);"  # Read-only safe checkpoint

# Integrity check (non-destructive)
sqlite3 "$DB_PATH" "PRAGMA integrity_check;"

# Check pending writes and lock status
sqlite3 "$DB_PATH" "PRAGMA busy_timeout(5000);"
sqlite3 "$DB_PATH" "SELECT count(*) FROM benchmarks WHERE status='pending';"
```
Expected outcome: `journal_mode` is `WAL` or `DELETE`, `integrity_check` returns `ok`, pending queue count is stable. If `integrity_check` returns warnings, halt and proceed to Data Safety Checks.

**Log Analysis**
Review recent logs for errors, timeouts, or graceful degradation signals.
```bash
# Systemd journal (if applicable)
journalctl -u flight-recorder --since "2 hours ago" --no-pager | grep -iE "error|warn|timeout|crash|sqlite" | tail -n 30

# Application log (if file-based)
tail -n 100 /var/log/flight-recorder/app.log 2>/dev/null || echo "No app log found"

# Check for log rotation status
ls -lh /var/log/flight-recorder/ 2>/dev/null || echo "No log directory"
```
Expected outcome: No repeated crash loops, no SQLite `database is locked` errors, no OOM kills. Note any recurring warnings for targeted remediation.

# Health Checks

Health checks define the quantitative thresholds that determine whether the server is stable enough to accept benchmark traffic. These checks must be run continuously during the remediation run and serve as automated go/no-go gates.

**Resource Thresholds**
- CPU Utilization: < 80% sustained over 5-minute window
- Memory Usage: < 85% physical RAM, swap usage < 10%
- Disk Space: < 90% capacity on `/models/flight-recorder/data`
- Disk I/O: `%util` < 70%, `await` < 50ms on primary storage
- Network Latency: < 5ms to local benchmark controller, < 50ms to external dependencies

**Service Health Metrics**
- HTTP Health Endpoint: Returns 200 OK within 200ms
- Database Response Time: `PRAGMA integrity_check` completes < 2s
- Queue Depth: Pending benchmark jobs < 500 (adjust based on capacity)
- Worker Status: All configured workers report `ready` state
- Error Rate: < 1% of requests return 4xx/5xx over 10-minute window

**Automated Health Check Script**
```bash
#!/bin/bash
# health_check.sh - Read-only benchmark readiness validator
set -euo pipefail

HOST="192.168.1.116"
PORT="8080"
DB_PATH="/models/flight-recorder/data/benchmark.db"

echo "=== Health Check Run: $(date -u) ==="

# 1. Network & Service
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "http://${HOST}:${PORT}/health" 2>/dev/null || echo "000")
if [ "$HTTP_CODE" != "200" ]; then
  echo "FAIL: Health endpoint returned $HTTP_CODE"
  exit 1
fi

# 2. Resource Snapshot
CPU=$(top -b -n 1 | grep "Cpu(s)" | awk '{print $2}' | cut -d. -f1)
MEM=$(free | awk '/Mem:/ {printf "%.0f", $3/$2 * 100}')
DISK=$(df -h /models/flight-recorder/data | awk 'NR==2 {print $5}' | cut -d% -f1)

echo "CPU: ${CPU}% | MEM: ${MEM}% | DISK: ${DISK}%"
[ "$CPU" -gt 80 ] && echo "FAIL: CPU threshold exceeded" && exit 1
[ "$MEM" -gt 85 ] && echo "FAIL: Memory threshold exceeded" && exit 1
[ "$DISK" -gt 90 ] && echo "FAIL: Disk threshold exceeded" && exit 1

# 3. SQLite Readiness
INTEGRITY=$(sqlite3 "$DB_PATH" "PRAGMA integrity_check;" 2>/dev/null)
if [ "$INTEGRITY" != "ok" ]; then
  echo "FAIL: SQLite integrity check failed: $INTEGRITY"
  exit 1
fi

# 4. Queue Depth
PENDING=$(sqlite3 "$DB_PATH" "SELECT count(*) FROM benchmarks WHERE status='pending';" 2>/dev/null || echo "0")
echo "Pending Benchmarks: $PENDING"
[ "$PENDING" -gt 500 ] && echo "WARN: High queue depth, consider throttling"

echo "=== Health Check PASSED ==="
```
Run this script every 60 seconds during the remediation window. If any check fails, trigger the Stop Conditions protocol.

# Data Safety Checks

Data preservation is the highest priority. Before any redeployment or benchmark restart, the data directory and SQLite database must be validated, backed up, and verified. This phase ensures that even if the remediation fails catastrophically, a pristine copy of all benchmark records, model artifacts, and configuration state exists.

**Pre-Remediation Backup**
```bash
# Create timestamped, immutable backup of the data directory
BACKUP_DIR="/models/flight-recorder/data.backup.$(date +%Y%m%d_%H%M%S)"
cp -a /models/flight-recorder/data "$BACKUP_DIR"

# Verify backup integrity using checksums
find "$BACKUP_DIR" -type f -exec md5sum {} \; > "${BACKUP_DIR}/checksums.md5"
md5sum -c "${BACKUP_DIR}/checksums.md5" --quiet && echo "Backup verified" || echo "Backup integrity failed"
```
Note: `cp -a` preserves permissions, timestamps, symlinks, and extended attributes. This is safe and non-destructive.

**SQLite Consistency Validation**
```bash
DB_PATH="/models/flight-recorder/data/benchmark.db"

# Ensure WAL mode is active for concurrent read/write safety
sqlite3 "$DB_PATH" "PRAGMA journal_mode=WAL;" 2>/dev/null || echo "WAL mode already set or unsupported"

# Force a passive checkpoint to flush in-memory pages to disk
sqlite3 "$DB_PATH" "PRAGMA wal_checkpoint(PASSIVE);"

# Run full integrity check
INTEGRITY=$(sqlite3 "$DB_PATH" "PRAGMA integrity_check;")
if [ "$INTEGRITY" != "ok" ]; then
  echo "CRITICAL: Database integrity compromised. Halting remediation."
  exit 1
fi

# Verify foreign keys and indexes
sqlite3 "$DB_PATH" "PRAGMA foreign_key_check;" 2>/dev/null
sqlite3 "$DB_PATH" "PRAGMA index_list;" 2>/dev/null
```
If `PRAGMA integrity_check` returns anything other than `ok`, do not proceed. The database may be corrupted due to unclean shutdowns, disk errors, or concurrent write conflicts. In that case, trigger Recovery Commands immediately.

**Filesystem & Permission Audit**
```bash
# Check for stale locks or open file handles
lsof +D /models/flight-recorder/data/ 2>/dev/null | grep -v "COMMAND" | head -n 20

# Verify ownership and permissions
stat -c "%a %U %G %n" /models/flight-recorder/data/* | head -n 30

# Check inode usage (prevents silent allocation failures)
df -i /models/flight-recorder/data
```
Expected outcome: No stale locks, permissions match application user (e.g., `www-data` or `benchmark`), inode usage < 80%. If permissions are incorrect, note them for the Redeploy Procedure but do not modify without approval.

**Snapshot Preparation (If Supported)**
If the host uses LVM, ZFS, or Btrfs, create a read-only snapshot before proceeding:
```bash
# Example for LVM (adjust volume name as needed)
lvcreate --snapshot --name data_snap --size 10G /dev/vg0/data 2>/dev/null || echo "Snapshot not available"

# Example for ZFS
zfs snapshot models/flight-recorder/data@pre-remediation 2>/dev/null || echo "ZFS not available"
```
Snapshots provide instant rollback capability without copying data. Document the snapshot name for Recovery Commands.

# Redeploy Procedure

The redeploy phase restores the application code and runtime environment without touching the data directory. This section assumes the application is containerized or managed via systemd. All steps are non-destructive and include verification gates.

**Graceful Service Pause**
```bash
# Stop the service cleanly to prevent in-flight writes during redeploy
systemctl stop flight-recorder 2>/dev/null || kill -TERM $(pgrep -f flight-recorder) 2>/dev/null

# Wait for graceful shutdown (max 30 seconds)
timeout 30 bash -c 'while pgrep -f flight-recorder > /dev/null; do sleep 1; done' || echo "Force stop may be required"

# Verify process termination
pgrep -f flight-recorder && echo "WARNING: Process still running" || echo "Service stopped"
```
If the service does not stop gracefully, do not force-kill without approval. Instead, proceed with redeploy while the service is running (hot-reload capable frameworks like FastAPI/Flask may support this).

**Code & Dependency Sync**
```bash
cd /models/flight-recorder/app

# Pull latest stable branch (adjust as needed)
git fetch origin
git checkout main 2>/dev/null || git checkout master
git pull --ff-only origin main 2>/dev/null || echo "No updates available or merge conflict"

# Verify dependency lock file matches runtime
pip install --dry-run -r requirements.txt 2>/dev/null || poetry check 2>/dev/null || echo "Dependency check skipped"

# Clear stale bytecode (safe, non-destructive)
find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
```
Note: `rm -rf __pycache__` is safe and recommended to prevent stale import issues. It does not affect data.

**Configuration & Environment Validation**
```bash
# Validate environment variables
env | grep -E "FLIGHT_RECORDER|DATABASE_URL|BENCHMARK_" | sort

# Check config file syntax (adjust for your framework)
python3 -c "import yaml; yaml.safe_load(open('config.yaml'))" 2>/dev/null || echo "YAML config invalid"
python3 -c "import json; json.load(open('config.json'))" 2>/dev/null || echo "JSON config invalid"

# Verify database path is correctly referenced
grep -r "benchmark.db" /models/flight-recorder/app/ 2>/dev/null | head -n 5
```
If configuration references a different database path or uses environment variables that override the data directory, correct them before proceeding. Do not modify `/models/flight-recorder/data` under any circumstances.

**Service Restart & Health Verification**
```bash
# Start the service
systemctl start flight-recorder 2>/dev/null || python3 -m uvicorn main:app --host 0.0.0.0 --port 8080 &

# Wait for startup
sleep 10

# Verify health
curl -s -o /dev/null -w "HTTP %{http_code}\n" http://192.168.1.116:8080/health
systemctl status flight-recorder --no-pager | tail -n 10

# Check for startup errors in logs
journalctl -u flight-recorder --since "2 minutes ago" --no-pager | grep -iE "error|fail|crash" | tail -n 10
```
Expected outcome: Service starts cleanly, health endpoint returns 200, no critical errors in logs. If startup fails, revert to the previous code version and trigger Recovery Commands.

# Benchmark Restart Procedure

Once the application is stable and data safety is confirmed, the benchmark pipeline can be resumed. This phase emphasizes gradual ramp-up, state validation, and continuous monitoring to prevent silent failures.

**Queue State Reset & Validation**
```bash
# Verify pending queue is in a consistent state
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT status, count(*) FROM benchmarks GROUP BY status;"

# If queue is stuck, mark failed jobs as pending (read-only safe if approved)
# [APPROVAL REQUIRED] sqlite3 /models/flight-recorder/data/benchmark.db "UPDATE benchmarks SET status='pending' WHERE status='failed' AND created_at < datetime('now', '-1 hour');"
```
Do not modify queue state without explicit approval. Instead, let the application's built-in retry mechanism handle failed jobs.

**Warm-Up Execution**
```bash
# Submit a single synthetic benchmark to verify pipeline
curl -X POST http://192.168.1.116:8080/benchmark/submit \
  -H "Content-Type: application/json" \
  -d '{"model": "test-model", "dataset": "synthetic", "params": {"iterations": 1}}'

# Monitor execution
tail -f /var/log/flight-recorder/app.log | grep -i "benchmark"
```
Expected outcome: Job completes successfully, output schema matches expected format, no timeouts. If warm-up fails, halt and investigate logs.

**Gradual Load Ramp-Up**
```bash
# Phase 1: 10% load (10 concurrent jobs)
for i in {1..10}; do
  curl -s -X POST http://192.168.1.116:8080/benchmark/submit -d '{"model": "test", "dataset": "warmup", "params": {"iterations": 1}}' &
done
wait

# Monitor resource usage during ramp-up
watch -n 5 "vmstat 1 1; sqlite3 /models/flight-recorder/data/benchmark.db 'SELECT count(*) FROM benchmarks WHERE status IN (\"running\", \"pending\");'"

# Phase 2: 50% load (if Phase 1 passes)
# Phase 3: 100% load (if Phase 2 passes)
```
Adjust load percentages based on server capacity. Monitor CPU, RAM, disk I/O, and SQLite lock contention. If any metric crosses thresholds, throttle back immediately.

**Output Validation & Logging**
```bash
# Verify benchmark output schema
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT id, model, status, output_path, created_at FROM benchmarks ORDER BY created_at DESC LIMIT 5;"

# Check for orphaned files or missing artifacts
find /models/flight-recorder/data -name "*.json" -o -name "*.csv" -o -name "*.parquet" | head -n 20

# Ensure logs are rotating correctly
logrotate -f /etc/logrotate.d/flight-recorder 2>/dev/null || echo "Log rotation not configured"
```
Expected outcome: All jobs produce valid output files, database records match file system state, logs rotate without gaps. Document any anomalies for post-run analysis.

# Human Approval Gates

Certain actions carry irreversible consequences or violate the non-destructive constraint. These gates require explicit human authorization before execution. The agent will pause at each gate, present the exact command, rationale, and risk assessment, and await confirmation.

**Approval-Required Actions**
1. `[APPROVAL REQUIRED]` Database schema modifications: `ALTER TABLE`, `DROP COLUMN`, `CREATE INDEX`
2. `[APPROVAL REQUIRED]` Queue state manipulation: `UPDATE benchmarks SET status=...`, `DELETE FROM benchmarks`
3. `[APPROVAL REQUIRED]` Filesystem writes outside `/models/flight-recorder/app`: `rm -rf`, `mv`, `chmod`, `chown` on data directory
4. `[APPROVAL REQUIRED]` System-level changes: `iptables`, `sysctl`, `systemctl restart` (if not already approved), `crontab` modifications
5. `[APPROVAL REQUIRED]` SQLite journal mode changes: `PRAGMA journal_mode=DELETE`, `PRAGMA synchronous=FULL`
6. `[APPROVAL REQUIRED]` Forceful process termination: `kill -9`, `pkill -9`

**Communication Protocol**
- Present approval requests via structured format:
  ```
  [APPROVAL REQUEST]
  Action: <command>
  Rationale: <why it's needed>
  Risk: <data loss, downtime, corruption>
  Rollback: <how to undo>
  Confirm (y/n): 
  ```
- Log all approvals in `/models/flight-recorder/app/logs/remediation_audit.log`
- If approval is denied, revert to read-only state and document the decision
- Escalation path: If approval delayed > 15 minutes, pause execution and notify via Slack/email

**Audit Trail**
Every approval, denial, and execution must be timestamped and recorded. This ensures traceability and supports post-mortem analysis. Example log entry:
```
[2024-06-15 14:32:10Z] APPROVAL_GRANTED: Action="sqlite3 benchmark.db UPDATE benchmarks SET status='pending' WHERE status='failed'" by user=admin
[2024-06-15 14:32:12Z] EXECUTED: Command run successfully. Affected rows: 12.
```

# Stop Conditions

The remediation run must halt immediately if any stop condition is triggered. These conditions protect data integrity, prevent resource exhaustion, and ensure human oversight.

**Critical Stop Triggers**
1. **Data Integrity Failure**: `PRAGMA integrity_check` returns anything other than `ok`
2. **Resource Exhaustion**: Disk > 95%, RAM swap > 50%, CPU sustained > 95% for 5 minutes
3. **Service Instability**: Crash loop > 3 times, health endpoint returns 5xx for > 2 minutes
4. **Benchmark Corruption**: Output schema mismatch, checksum failures, or missing artifacts
5. **Approval Timeout**: Critical action pending > 30 minutes without response
6. **Unexpected Errors**: Unhandled exceptions in application logs, SQLite `database is locked` > 10 times/minute
7. **Network Partition**: Loss of connectivity to benchmark controller or storage backend

**Halt Procedure**
When a stop condition is triggered:
1. Immediately pause all benchmark submissions
2. Capture current state: `sqlite3 benchmark.db "SELECT * FROM benchmarks WHERE status IN ('running','pending');" > /tmp/benchmark_state.json`
3. Preserve logs: `cp -a /var/log/flight-recorder/ /tmp/flight-recorder-logs-$(date +%Y%m%d_%H%M%S)/`
4. Notify human operator with stop reason and current state
5. Do not attempt automatic recovery unless explicitly approved
6. Maintain service in safe state (running but idle) until further instructions

**Recovery Readiness**
Before resuming after a stop, verify:
- Data backup is intact
- Root cause is identified and mitigated
- Human approval is obtained for next steps
- Health checks pass consistently for 5 minutes

# Recovery Commands

If the remediation fails or a stop condition is triggered, these commands enable safe rollback and state restoration. All recovery actions are designed to preserve data and revert to a known-good state.

**Application Rollback**
```bash
# Revert to previous code version
cd /models/flight-recorder/app
git checkout HEAD~1 2>/dev/null || git checkout main  # Adjust branch as needed
pip install -r requirements.txt --quiet 2>/dev/null || poetry install --quiet 2>/dev/null

# Clear stale bytecode
find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true

# Restart service
systemctl restart flight-recorder 2>/dev/null || python3 -m uvicorn main:app --host 0.0.0.0 --port 8080 &
```

**Data Restoration**
```bash
# Restore from timestamped backup (DO NOT overwrite without approval)
BACKUP_DIR=$(ls -td /models/flight-recorder/data.backup.* 2>/dev/null | head -n 1)
if [ -n "$BACKUP_DIR" ]; then
  echo "WARNING: This will overwrite current data. Confirm before proceeding."
  # [APPROVAL REQUIRED] cp -a "$BACKUP_DIR"/* /models/flight-recorder/data/
  echo "Backup restored. Verify integrity."
else
  echo "No backup found. Manual intervention required."
fi

# Verify restored database
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
```

**SQLite Recovery**
```bash
# If database is locked or corrupted, attempt safe recovery
DB_PATH="/models/flight-recorder/data/benchmark.db"

# Detach and reattach WAL file
sqlite3 "$DB_PATH" "PRAGMA wal_checkpoint(TRUNCATE);" 2>/dev/null || echo "WAL checkpoint failed"

# Export to safe format for inspection
sqlite3 "$DB_PATH" ".mode csv" ".output /tmp/benchmark_export.csv" "SELECT * FROM benchmarks;" 2>/dev/null || echo "Export failed"

# If corruption persists, restore from backup
# [APPROVAL REQUIRED] sqlite3 /models/flight-recorder/data/benchmark.db ".restore /models/flight-recorder/data.backup.*/benchmark.db"
```

**System & Network Reset**
```bash
# Reset network state if connectivity issues persist
systemctl restart networking 2>/dev/null || ip route flush cache 2>/dev/null

# Clear stale locks
fuser -k /models/flight-recorder/data/benchmark.db 2>/dev/null || echo "No locks found"

# Restart logging
systemctl restart rsyslog 2>/dev/null || systemctl restart systemd-journald 2>/dev/null
```

**Post-Recovery Validation**
```bash
# Run health checks again
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8080/health
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
journalctl -u flight-recorder --since "5 minutes ago" --no-pager | grep -iE "error|fail" | tail -n 10
```
Only resume benchmark execution after all recovery steps complete and health checks pass.

# Summary

This remediation plan provides a structured, safety-first approach to restoring reliable benchmark execution on the remote model server at `192.168.1.116`. The process is divided into ten phases, each with explicit objectives, verification steps, and exit criteria. The overarching principle is data preservation: the `/models/flight-recorder/data` directory and its SQLite database are treated as immutable truth sources throughout the run. All inspection, health checking, and backup operations are strictly read-only. Redeployment focuses on code synchronization, dependency validation, and service restart without touching persistent state. Benchmark resumption follows a gradual ramp-up strategy with continuous monitoring and output validation.

Human approval gates are enforced for any action that could modify, delete, or corrupt data. The agent will pause at each gate, present the exact command and risk assessment, and await explicit confirmation. Stop conditions are clearly defined to prevent cascading failures, and recovery commands are provided for safe rollback. The plan prioritizes traceability, with audit logging, checksum verification, and state snapshots ensuring that every step can be reproduced or reversed.

Before execution, verify that the backup directory exists, health check thresholds are appropriate for the hardware, and approval channels are active. During execution, monitor resource usage, SQLite lock contention, and benchmark output schema. If any anomaly arises, halt immediately, preserve state, and notify the human operator. The goal is not just to restore execution, but to do so with full confidence that no data has been lost, no corruption has been introduced, and the benchmark pipeline is stable for sustained operation. Proceed with caution, validate continuously, and respect the approval gates.