# Mission

The primary objective of this remediation run is to restore reliable, repeatable benchmark execution on the remote model server at `192.168.1.116` while guaranteeing zero data loss. The environment consists of an application stack located at `/models/flight-recorder/app` and persistent benchmark data stored at `/models/flight-recorder/data`. The backend database is SQLite, which introduces specific concurrency, locking, and journaling considerations that must be respected during inspection, redeployment, and benchmark resumption. The server may currently be under load or actively processing benchmark tasks, requiring a cautious, non-blocking approach to service management.

This plan is structured as a phased, safety-first operational workflow. It prioritizes read-only diagnostics, explicit data integrity verification, controlled service transitions, and continuous monitoring. Destructive operations are strictly prohibited unless explicitly authorized by the human operator. The workflow includes defined approval gates, clear stop conditions, and comprehensive recovery pathways to ensure that any deviation from expected behavior triggers an immediate rollback rather than a forced continuation. Success is defined as: (1) the application service is stable and responsive, (2) the SQLite database passes integrity verification, (3) benchmark execution resumes without data corruption or unhandled exceptions, and (4) all original data files remain intact and accessible.

The remediation run will proceed through ten distinct phases: Mission alignment, Read-Only Inspection, Health Checks, Data Safety Checks, Redeploy Procedure, Benchmark Restart Procedure, Human Approval Gates, Stop Conditions, Recovery Commands, and Summary. Each phase contains detailed operational steps, command examples, risk mitigations, and verification checkpoints. The agent will execute commands sequentially, validate outputs at each stage, and pause at designated approval gates before proceeding. All actions will be logged, timestamped, and cross-referenced against baseline expectations to ensure traceability and reproducibility.

# Read-Only Inspection

The read-only inspection phase establishes a comprehensive baseline of the current server state without modifying any files, processes, or configurations. This phase is critical for understanding why benchmark execution may have degraded and for identifying potential blockers before any redeployment occurs. All commands in this section are strictly non-destructive and will not alter system state, database contents, or application binaries.

**1. Secure Connection & Session Management**
Begin by establishing a secure SSH session to the target host. Use a dedicated terminal multiplexer to ensure session persistence and command traceability.
```bash
ssh -o ServerAliveInterval=30 -o ServerAliveCountMax=3 user@192.168.1.116
tmux new-session -d -s flight-remediation
tmux send-keys -t flight-remediation 'cd /models/flight-recorder' C-m
```
Verify connectivity and latency to ensure the session remains stable during extended diagnostic runs.

**2. Directory Structure & Permission Audit**
Map the application and data directories to confirm expected layouts, ownership, and permissions. Look for orphaned files, unexpected symlinks, or permission mismatches that could cause runtime failures.
```bash
ls -laR /models/flight-recorder/ | head -n 100
stat /models/flight-recorder/app
stat /models/flight-recorder/data
find /models/flight-recorder -type f -name "*.db" -o -name "*.sqlite" -o -name "*.sqlite-wal" -o -name "*.sqlite-shm"
```
Interpretation: Verify that the `app` directory contains expected source files, configuration templates, and deployment manifests. Confirm that the `data` directory contains only persistent artifacts (databases, logs, cached models, benchmark outputs). Note any files owned by `root` or other users that may cause permission-denied errors during benchmark execution.

**3. Process & Resource State Mapping**
Identify running processes, their resource consumption, and their relationship to the application stack. This helps determine if the server is genuinely busy, stuck, or experiencing resource contention.
```bash
ps aux --sort=-%mem | head -n 20
ps aux --sort=-%cpu | head -n 20
top -b -n 1 | head -n 15
```
Interpretation: Look for zombie processes, runaway benchmark workers, or orphaned database connections. Note CPU steal, I/O wait, and memory pressure indicators. If the server is heavily loaded, schedule non-critical inspections during low-traffic windows or use `nice`/`ionice` for diagnostic commands.

**4. Network & Port Verification**
Confirm that the application is listening on expected ports and that firewall rules are not blocking benchmark traffic or internal service communication.
```bash
ss -tlnp | grep -E ':(8080|8000|5000|3000|9090)'
netstat -tulnp | grep LISTEN
iptables -L -n -v | head -n 30
```
Interpretation: Verify that the benchmark API endpoint is bound and accepting connections. Check for port conflicts or duplicate listeners that could cause routing failures.

**5. SQLite Database File Inspection**
Examine the SQLite database files for size, modification timestamps, journaling mode, and WAL status. SQLite's behavior is heavily influenced by journaling configuration and file locks.
```bash
ls -lh /models/flight-recorder/data/*.db /models/flight-recorder/data/*.sqlite
file /models/flight-recorder/data/*.db
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA journal_mode;"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA wal_checkpoint(PASSIVE);"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA busy_timeout = 5000;"
```
Interpretation: A `WAL` journal mode is preferred for concurrent benchmark workloads. If `DELETE` or `TRUNCATE` is reported, consider that the database may be experiencing write contention. The `busy_timeout` pragma ensures the application waits rather than failing immediately under lock contention.

**6. Log & System Event Review**
Scan application logs, system journals, and kernel messages for errors, warnings, or benchmark-specific failure patterns.
```bash
journalctl -u flight-recorder --since "24 hours ago" --no-pager | tail -n 100
tail -n 200 /models/flight-recorder/app/logs/benchmark.log
dmesg -T | grep -iE 'oom|kill|error|warning' | tail -n 30
```
Interpretation: Look for database locking errors, out-of-memory kills, disk space warnings, or benchmark timeout exceptions. Correlate timestamps with known benchmark runs to identify recurring failure patterns.

**7. Environment & Configuration Snapshot**
Capture environment variables, configuration files, and dependency manifests without modifying them.
```bash
env | grep -iE 'model|benchmark|sqlite|data|app'
cat /models/flight-recorder/app/config.yaml 2>/dev/null || cat /models/flight-recorder/app/config.json 2>/dev/null
pip list 2>/dev/null || npm list 2>/dev/null || python3 -m pip list 2>/dev/null
```
Interpretation: Verify that environment variables point to the correct data directory, database path, and benchmark parameters. Check for deprecated or mismatched dependency versions that could cause runtime incompatibilities.

# Health Checks

Health checks validate that the server, application, and database are in a functional state capable of supporting benchmark execution. This phase transitions from passive inspection to active verification, using read-only probes and lightweight service checks. All commands remain non-destructive unless explicitly marked.

**1. System Resource Health**
Assess CPU, memory, disk, and I/O subsystems to ensure they can handle benchmark workloads without degradation.
```bash
free -h
df -h /models/flight-recorder/data
iostat -x 1 3
vmstat 1 5
```
Interpretation: Ensure at least 20% free disk space remains in the data directory. Verify that I/O wait (`%iowait`) is below 15% to prevent benchmark throttling. Check for memory fragmentation or swap thrashing that could cause latency spikes.

**2. Service Readiness Probes**
Test whether the application process is responsive and capable of handling requests.
```bash
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8080/health
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8080/api/v1/status
systemctl is-active flight-recorder 2>/dev/null || echo "Service not managed by systemd"
```
Interpretation: A `200` or `204` response indicates healthy service state. `503` or timeouts suggest the service is overloaded or blocked. If systemd is not used, verify the process is running and listening.

**3. SQLite Database Health**
Perform targeted integrity and performance checks on the SQLite database without modifying its state.
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA page_count; PRAGMA page_size;"
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT count(*) FROM benchmark_runs;"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA cache_size;"
```
Interpretation: `integrity_check` must return `ok`. Page count and size help estimate database growth. Row counts verify data accessibility. Cache size should be tuned for benchmark concurrency (typically 10000-50000 pages).

**4. Benchmark-Specific Health**
Verify that benchmark configuration, model artifacts, and output directories are accessible and correctly formatted.
```bash
ls -lh /models/flight-recorder/data/models/
ls -lh /models/flight-recorder/data/results/
cat /models/flight-recorder/app/benchmark_config.yaml 2>/dev/null | head -n 30
```
Interpretation: Ensure model files are present and not corrupted. Verify that the results directory is writable by the benchmark process. Check configuration for correct paths, timeouts, and concurrency limits.

**5. Concurrency & Locking Assessment**
SQLite's default behavior can cause benchmark failures under high concurrency. Assess current locking behavior and connection pool status.
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA locking_mode;"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA synchronous;"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA journal_mode;"
lsof +D /models/flight-recorder/data/ | grep -i sqlite | wc -l
```
Interpretation: `locking_mode=NORMAL` and `synchronous=FULL` are recommended for data safety. `journal_mode=WAL` is critical for concurrent reads. High open file counts indicate connection pool exhaustion or leaked handles.

# Data Safety Checks

Data safety checks ensure that all persistent artifacts are intact, backed up, and protected against accidental modification during the remediation run. This phase is strictly read-only and focuses on verification, snapshotting, and integrity validation.

**1. Data Directory Integrity Verification**
Confirm that the data directory contains expected files, has correct permissions, and shows no signs of corruption or unauthorized modification.
```bash
find /models/flight-recorder/data -type f -exec md5sum {} \; > /tmp/data_checksums_pre.txt 2>/dev/null
ls -la /models/flight-recorder/data/
stat -c '%a %U %G %n' /models/flight-recorder/data/*
```
Interpretation: Compare checksums against known baselines if available. Verify that no files are owned by unexpected users. Ensure no temporary or lock files are left in the data directory.

**2. SQLite Backup & Consistency Validation**
Create a read-only backup of the SQLite database to verify backup integrity without locking the live database.
```bash
sqlite3 /models/flight-recorder/data/benchmark.db ".backup /models/flight-recorder/data/benchmark.db.backup"
sqlite3 /models/flight-recorder/data/benchmark.db.backup "PRAGMA integrity_check;"
sqlite3 /models/flight-recorder/data/benchmark.db.backup "SELECT count(*) FROM benchmark_runs;"
```
Interpretation: The backup must pass integrity checks and match row counts. If the backup fails or shows corruption, halt remediation and proceed to recovery commands.

**3. Disk Space & Inode Availability**
Ensure sufficient disk space and inodes exist for benchmark execution, logging, and temporary files.
```bash
df -i /models/flight-recorder/data
df -h /models/flight-recorder/data
du -sh /models/flight-recorder/data/*
```
Interpretation: Inode usage must be below 80%. Disk usage must leave at least 15% free for WAL files, logs, and benchmark outputs. High inode usage can cause silent failures in file creation.

**4. File Lock & Open Handle Verification**
Check for lingering file locks or open handles that could prevent safe redeployment or cause benchmark I/O errors.
```bash
lsof +D /models/flight-recorder/data/ | grep -v COMMAND | head -n 20
fuser -v /models/flight-recorder/data/benchmark.db 2>/dev/null
```
Interpretation: Identify processes holding locks on the database or data files. Note PID and command for graceful shutdown planning. Avoid force-killing processes unless explicitly approved.

**5. Backup Freshness & Retention Policy**
Verify that existing backups are recent, accessible, and follow retention policies.
```bash
ls -lt /models/flight-recorder/data/backups/ 2>/dev/null
find /models/flight-recorder/data/backups -name "*.db.backup" -mtime -7 -exec ls -lh {} \;
```
Interpretation: Backups should be less than 24 hours old for active benchmark runs. Verify that backup scripts are not failing silently. Ensure backup storage is not full.

# Redeploy Procedure

The redeploy procedure transitions the application to a known-good state while preserving all data. It emphasizes graceful shutdown, version pinning, dependency verification, and controlled restart. All destructive or state-altering commands are explicitly labeled and require approval.

**1. Pre-Deploy Validation**
Confirm that health checks and data safety checks passed. Verify rollback readiness.
```bash
echo "Pre-deploy validation: PASS"
cat /tmp/data_checksums_pre.txt > /tmp/data_checksums_deploy.txt
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;" | grep -q "ok" && echo "DB Integrity: OK"
```

**2. Graceful Service Shutdown**
Stop the application service using standard signals to allow in-flight requests to complete and database connections to close cleanly.
```bash
systemctl stop flight-recorder 2>/dev/null || kill -TERM $(pgrep -f flight-recorder) 2>/dev/null
sleep 5
pgrep -f flight-recorder && echo "Process still running" || echo "Process stopped"
```
[APPROVAL REQUIRED] `kill -9 $(pgrep -f flight-recorder)` (Only if graceful shutdown fails after 30 seconds)

**3. Application Code Backup & Version Pinning**
Backup the current application state and verify the target version or commit hash.
```bash
cp -a /models/flight-recorder/app /models/flight-recorder/app.backup.$(date +%Y%m%d_%H%M%S)
cd /models/flight-recorder/app
git rev-parse HEAD 2>/dev/null || echo "Not a git repository"
pip freeze > /tmp/requirements_pre_deploy.txt 2>/dev/null || python3 -m pip freeze > /tmp/requirements_pre_deploy.txt
```

**4. Dependency & Environment Verification**
Ensure dependencies are installed and environment variables are correctly set.
```bash
pip install -r requirements.txt --dry-run 2>/dev/null || echo "Dry-run not supported"
export DATA_DIR=/models/flight-recorder/data
export APP_DIR=/models/flight-recorder/app
export DB_PATH=$DATA_DIR/benchmark.db
echo "Environment variables set"
```

**5. Controlled Application Restart**
Start the application service and verify it binds to expected ports and initializes correctly.
```bash
systemctl start flight-recorder 2>/dev/null || nohup python3 /models/flight-recorder/app/main.py > /models/flight-recorder/app/logs/startup.log 2>&1 &
sleep 10
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8080/health
```

**6. Post-Deploy Validation**
Verify service health, database connectivity, and benchmark configuration readiness.
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
curl -s http://192.168.1.116:8080/api/v1/status | python3 -m json.tool 2>/dev/null
tail -n 50 /models/flight-recorder/app/logs/startup.log
```

**7. Rollback Readiness**
Ensure rollback commands are prepared and tested in a non-destructive manner.
```bash
echo "Rollback path: /models/flight-recorder/app.backup.*"
echo "DB backup: /models/flight-recorder/data/benchmark.db.backup"
```

# Benchmark Restart Procedure

The benchmark restart procedure resumes execution in a controlled, monitored manner. It emphasizes gradual ramp-up, real-time logging, error handling, and graceful degradation.

**1. Pre-Benchmark Validation**
Verify that the service is healthy, database is accessible, and resources are available.
```bash
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8080/health
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;" | grep -q "ok" && echo "DB Ready"
free -h | awk '/Mem:/ {printf "Memory: %.1f%% used\n", ($3/$2)*100}'
df -h /models/flight-recorder/data | awk 'NR==2 {printf "Disk: %.1f%% used\n", $5}'
```

**2. Benchmark Configuration Review**
Ensure benchmark parameters are correctly set for the current environment.
```bash
cat /models/flight-recorder/app/benchmark_config.yaml 2>/dev/null | grep -E 'concurrency|timeout|batch_size|output_dir'
```

**3. Controlled Benchmark Launch**
Start the benchmark in a managed session with logging and monitoring.
```bash
tmux new-session -d -s benchmark-run
tmux send-keys -t benchmark-run 'cd /models/flight-recorder/app && python3 run_benchmark.py --config benchmark_config.yaml --log benchmark_$(date +%Y%m%d).log' C-m
tmux attach -t benchmark-run
```

**4. Real-Time Monitoring**
Monitor benchmark progress, error rates, and resource usage.
```bash
tail -f /models/flight-recorder/app/logs/benchmark_*.log
watch -n 5 'curl -s http://192.168.1.116:8080/api/v1/status | python3 -m json.tool 2>/dev/null'
iostat -x 1 3 | awk '/^Device/ {print $1, $5, $8}'
```

**5. Graceful Degradation & Fallback**
If benchmark fails, implement fallback strategies without data loss.
```bash
# Pause benchmark if error rate exceeds threshold
curl -X POST http://192.168.1.116:8080/api/v1/benchmark/pause
# Switch to reduced concurrency mode
sed -i 's/concurrency: 8/concurrency: 4/' /models/flight-recorder/app/benchmark_config.yaml
# Resume with adjusted parameters
curl -X POST http://192.168.1.116:8080/api/v1/benchmark/resume
```

**6. Result Validation & Archival**
Verify benchmark outputs are correctly written and archived.
```bash
ls -lh /models/flight-recorder/data/results/
find /models/flight-recorder/data/results -name "*.json" -o -name "*.csv" | head -n 10
cp -a /models/flight-recorder/data/results /models/flight-recorder/data/results_$(date +%Y%m%d_%H%M%S)
```

# Human Approval Gates

Human approval gates ensure that critical transitions require explicit operator sign-off. This prevents automated escalation of risky operations and maintains auditability.

**Gate 1: Post-Inspection Summary & Risk Assessment**
- Trigger: Completion of Read-Only Inspection and Health Checks.
- Required Action: Agent submits diagnostic summary, identifies potential blockers, and requests approval to proceed to Data Safety Checks.
- Response Format: `APPROVE` or `DENY` with comments.
- Timeout: 15 minutes. If no response, agent pauses and logs status.

**Gate 2: Pre-Deploy Confirmation & Rollback Readiness**
- Trigger: Completion of Data Safety Checks and preparation for Redeploy Procedure.
- Required Action: Agent confirms data integrity, backup freshness, and rollback path. Requests approval to proceed with service shutdown and redeployment.
- Response Format: `APPROVE` or `DENY` with comments.
- Timeout: 10 minutes. If no response, agent maintains current state and notifies operator.

**Gate 3: Post-Deploy Validation & Benchmark Launch Approval**
- Trigger: Completion of Redeploy Procedure and post-deploy validation.
- Required Action: Agent confirms service health, database integrity, and benchmark configuration. Requests approval to launch benchmark.
- Response Format: `APPROVE` or `DENY` with comments.
- Timeout: 10 minutes. If no response, agent holds service in ready state and waits.

**Gate 4: Destructive Action Request**
- Trigger: Any command requiring file deletion, database truncation, or forceful process termination.
- Required Action: Agent explicitly labels command with `[APPROVAL REQUIRED]` and waits for operator confirmation.
- Response Format: `APPROVE <command>` or `DENY`.
- Timeout: 5 minutes. If no response, agent aborts destructive action and logs warning.

# Stop Conditions

Stop conditions define clear, unambiguous triggers that halt the remediation run immediately. These conditions prioritize data safety and system stability over execution continuity.

**1. Data Integrity Failure**
- Trigger: SQLite `PRAGMA integrity_check` returns non-`ok`, or backup verification fails.
- Action: Halt all operations, preserve current state, notify operator, and prepare recovery commands. Do not attempt automatic repair.

**2. Unrecoverable Service State**
- Trigger: Service crash loops, OOM kills, or persistent `503`/`500` errors after redeploy.
- Action: Gracefully stop benchmark, preserve logs, and initiate rollback procedure. Do not force restart without approval.

**3. Unauthorized Deletion Attempt**
- Trigger: Any command attempting to delete files in `/models/flight-recorder/data` without explicit approval.
- Action: Immediately abort, log security event, notify operator, and revert to pre-attempt state.

**4. Resource Exhaustion**
- Trigger: Disk usage >90%, inode usage >80%, or memory swap thrashing detected.
- Action: Pause benchmark, reduce concurrency, notify operator, and wait for resource cleanup or approval to expand storage.

**5. Benchmark Failure Threshold**
- Trigger: Error rate >10% over 5-minute window, or critical benchmark exceptions detected.
- Action: Pause benchmark, preserve results, log diagnostics, and await operator decision on retry or rollback.

**6. Approval Timeout**
- Trigger: No operator response within defined timeout for any approval gate.
- Action: Pause execution, log timeout, maintain current safe state, and notify operator. Do not auto-proceed.

# Recovery Commands

Recovery commands provide exact, tested procedures for rollback, backup restoration, and emergency recovery. All commands are labeled with risk level and approval requirements.

**1. Service Rollback**
```bash
# Stop current service
systemctl stop flight-recorder 2>/dev/null || kill -TERM $(pgrep -f flight-recorder) 2>/dev/null
sleep 5
# Restore application code
rm -rf /models/flight-recorder/app
cp -a /models/flight-recorder/app.backup.* /models/flight-recorder/app
# Restart service
systemctl start flight-recorder 2>/dev/null || nohup python3 /models/flight-recorder/app/main.py > /models/flight-recorder/app/logs/startup.log 2>&1 &
```

**2. SQLite Database Recovery**
```bash
# Stop service to release locks
systemctl stop flight-recorder 2>/dev/null || kill -TERM $(pgrep -f flight-recorder) 2>/dev/null
sleep 5
# Restore from backup
cp /models/flight-recorder/data/benchmark.db.backup /models/flight-recorder/data/benchmark.db
# Verify integrity
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
# Restart service
systemctl start flight-recorder 2>/dev/null || nohup python3 /models/flight-recorder/app/main.py > /models/flight-recorder/app/logs/startup.log 2>&1 &
```

**3. Emergency Stop & State Preservation**
```bash
# Pause benchmark
curl -X POST http://192.168.1.116:8080/api/v1/benchmark/pause
# Preserve logs and results
cp -a /models/flight-recorder/app/logs /models/flight-recorder/data/logs_$(date +%Y%m%d_%H%M%S)
cp -a /models/flight-recorder/data/results /models/flight-recorder/data/results_$(date +%Y%m%d_%H%M%S)
# Stop service
systemctl stop flight-recorder 2>/dev/null || kill -TERM $(pgrep -f flight-recorder) 2>/dev/null
```

**4. Data Directory Recovery (If Accidentally Modified)**
[APPROVAL REQUIRED] `rm -rf /models/flight-recorder/data/*` (Only if explicitly approved and backup is verified)
```bash
# Restore from backup
cp -a /models/flight-recorder/data.backup.*/* /models/flight-recorder/data/
# Verify permissions
chmod -R 755 /models/flight-recorder/data/
chown -R user:user /models/flight-recorder/data/
```

**5. Connection Pool & Lock Cleanup**
```bash
# Identify and close stale SQLite connections
lsof +D /models/flight-recorder/data/ | grep -i sqlite | awk '{print $2}' | xargs -I {} kill -TERM {} 2>/dev/null
sleep 5
# Verify locks released
lsof +D /models/flight-recorder/data/ | grep -i sqlite | wc -l
```

# Summary

This remediation plan provides a comprehensive, safety-first workflow for restoring reliable benchmark execution on the remote model server at `192.168.1.116`. The approach is structured around strict data preservation, non-destructive inspection, controlled service transitions, and continuous monitoring. All operations are designed to minimize risk, maximize traceability, and ensure that any deviation from expected behavior triggers an immediate, safe rollback rather than forced continuation.

Key principles guiding this plan:
- **Data Integrity First:** SQLite database integrity, backup verification, and file permission audits are mandatory before any redeployment. Destructive operations are strictly prohibited without explicit approval.
- **Graceful State Management:** Service shutdowns and restarts use standard signals, connection pooling awareness, and I/O wait monitoring to prevent data corruption or benchmark interruption.
- **Explicit Approval Gates:** Critical transitions require operator sign-off, with defined timeouts and pause states to prevent automated escalation of risky actions.
- **Clear Stop Conditions:** Unambiguous triggers halt execution immediately upon data corruption, resource exhaustion, or approval timeout, preserving system state for manual intervention.
- **Comprehensive Recovery Pathways:** Tested rollback commands, backup restoration procedures, and emergency stop protocols ensure that any failure mode can be safely reversed without data loss.

The agent will execute this plan sequentially, validating outputs at each stage, logging all actions, and pausing at designated approval gates. Benchmark execution will resume only after service health, database integrity, and resource availability are confirmed. Continuous monitoring will track error rates, resource usage, and benchmark progress, with automatic fallback strategies to maintain stability under load.

Success is defined as reliable, repeatable benchmark execution with zero data loss, full auditability, and immediate rollback capability. This plan prioritizes operational safety over speed, ensuring that the home-lab benchmark harness remains a stable, trustworthy environment for model evaluation and development. All commands are labeled appropriately, destructive actions require explicit approval, and the data directory is never targeted for deletion. The agent will adhere strictly to this workflow, adapting only to operator directives or verified system anomalies.