# Mission

The primary objective of this remediation run is to restore reliable benchmark execution on the remote model server located at `192.168.1.116` while guaranteeing zero data loss. The environment consists of an application stack rooted at `/models/flight-recorder/app`, persistent data stored under `/models/flight-recorder/data`, and a SQLite-backed database. The server may currently be under load or experiencing transient instability, which necessitates a cautious, phased approach.

This plan is structured to prioritize data integrity, service stability, and benchmark reproducibility. The remediation process is divided into distinct phases: read-only inspection, comprehensive health validation, data safety verification, controlled redeployment, and benchmark restart. Each phase includes explicit checkpoints, automated and manual verification steps, and clearly defined stop conditions. Destructive operations are strictly prohibited unless explicitly authorized by the human operator. The agent will operate in a non-destructive inspection and safe redeployment mode, leveraging graceful service management, SQLite integrity safeguards, and progressive benchmark validation.

Success criteria for this run include:
1. All critical services are running and responding to health checks.
2. The SQLite database passes integrity validation with zero warnings.
3. The application code is updated and deployed without data corruption or configuration drift.
4. The benchmark harness executes successfully, producing reproducible results.
5. No data files under `/models/flight-recorder/data` are modified, moved, or deleted without explicit human approval.

The agent will maintain strict operational discipline, logging all actions, verifying state transitions, and halting immediately if any stop condition is triggered. This plan serves as the authoritative runbook for the remediation cycle.

# Read-Only Inspection

Before any state changes are introduced, a thorough read-only inspection must be conducted to establish a baseline understanding of the server's current condition. This phase is strictly non-destructive and focuses on gathering telemetry, verifying file structures, and identifying potential bottlenecks or anomalies.

**1. Secure Access & Environment Verification**
Establish a secure SSH session to the target host. Verify connectivity, latency, and authentication status.
```bash
ssh -o ConnectTimeout=10 -o ServerAliveInterval=30 user@192.168.1.116
```
Once connected, verify the working directory structure and permissions:
```bash
ls -la /models/flight-recorder/
ls -la /models/flight-recorder/app/
ls -la /models/flight-recorder/data/
```
Check for symbolic links, mount points, or unusual permissions that could affect I/O or execution:
```bash
df -hT /models/flight-recorder/
mount | grep flight-recorder
stat /models/flight-recorder/data
```

**2. Process & Service State**
Identify running processes associated with the application and benchmark harness. Look for zombie processes, orphaned workers, or resource-hogging threads.
```bash
ps aux | grep -E '(flight-recorder|benchmark|python|node|gunicorn|uvicorn|celery)'
systemctl list-units --type=service --state=running | grep -E '(flight|benchmark|app)'
ss -tlnp | grep -E '(8000|8080|5000|9000)'  # Adjust ports based on known service configuration
```
Check for stale lock files or PID files that might indicate a previous crash:
```bash
find /models/flight-recorder/ -name "*.pid" -o -name "*.lock" 2>/dev/null
cat /var/run/flight-recorder.pid 2>/dev/null || echo "No PID file found"
```

**3. Network & Resource Telemetry**
Assess network connectivity, port binding, and resource utilization. The server may be busy, so understanding current load is critical.
```bash
top -b -n 1 | head -20
vmstat 1 5
iostat -x 1 3
ss -s
netstat -an | grep ESTABLISHED | wc -l
```
Verify DNS resolution and outbound connectivity if the benchmark requires external model endpoints or telemetry services:
```bash
dig +short model-server.internal
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8080/health
```

**4. Application & Configuration Inspection**
Review configuration files, environment variables, and dependency manifests without modifying them.
```bash
cat /models/flight-recorder/app/config.yaml
cat /models/flight-recorder/app/.env
pip list 2>/dev/null || npm list 2>/dev/null
git -C /models/flight-recorder/app log --oneline -5
```
Check for pending migrations, uncommitted changes, or dependency conflicts:
```bash
git -C /models/flight-recorder/app status
git -C /models/flight-recorder/app diff --stat
```

**5. Log & Telemetry Review**
Examine recent logs for errors, warnings, or anomalies. Focus on startup failures, database connection issues, and benchmark runner crashes.
```bash
journalctl -u flight-recorder-app --since "1 hour ago" --no-pager | tail -50
tail -n 100 /models/flight-recorder/app/logs/error.log
tail -n 100 /models/flight-recorder/app/logs/benchmark.log
```
Check for disk I/O bottlenecks or filesystem errors:
```bash
dmesg | tail -20
cat /proc/sys/fs/file-nr
```

**Inspection Output Synthesis**
Compile the findings into a structured report. Note any discrepancies between expected and actual state. Verify that the SQLite database file exists, is accessible, and shows no signs of truncation or corruption. Confirm that the benchmark harness configuration aligns with the current application version. This baseline will be referenced throughout subsequent phases to ensure state transitions are predictable and reversible.

# Health Checks

Following the read-only inspection, a systematic health check protocol must be executed to validate the operational readiness of each subsystem. These checks are designed to be non-intrusive, leveraging existing endpoints, database pragmas, and system utilities to assess stability.

**1. Application Service Health**
Verify that the application server is listening on the expected ports and responding to health probes.
```bash
curl -s -f http://192.168.1.116:8080/health || echo "Health endpoint unreachable"
curl -s -f http://192.168.1.116:8080/metrics | head -20
```
Check worker process count and memory consumption:
```bash
ps aux --sort=-%mem | grep -E '(gunicorn|uvicorn|celery)' | head -10
free -m
```

**2. Database Integrity & Accessibility**
SQLite databases require specific validation to ensure they are not corrupted, locked, or experiencing journaling issues.
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA journal_mode;"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA wal_checkpoint(TRUNCATE);" 2>/dev/null || echo "WAL checkpoint not applicable or failed"
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT count(*) FROM benchmark_results;" 2>/dev/null || echo "Table query failed"
```
Verify database file permissions and ownership:
```bash
ls -la /models/flight-recorder/data/benchmark.db*
stat -c '%a %U %G' /models/flight-recorder/data/benchmark.db
```

**3. Benchmark Harness Status**
Assess the state of the benchmark runner. Check for active jobs, queued tasks, and historical result consistency.
```bash
cat /models/flight-recorder/app/benchmark_status.json 2>/dev/null || echo "Status file missing"
ls -la /models/flight-recorder/data/benchmark_runs/ 2>/dev/null
tail -n 50 /models/flight-recorder/app/logs/benchmark_runner.log
```
Verify that the harness configuration matches the expected schema and that required model artifacts are present:
```bash
python3 -c "import json; json.load(open('/models/flight-recorder/app/config/benchmark_config.json'))" 2>/dev/null || echo "Config validation failed"
ls -la /models/flight-recorder/data/models/ 2>/dev/null
```

**4. Resource & I/O Health**
Monitor system resources to ensure the server can handle benchmark workloads without throttling or OOM kills.
```bash
cat /proc/loadavg
cat /proc/cpuinfo | grep "model name" | head -1
nproc
cat /proc/meminfo | grep -E '(MemTotal|MemFree|Buffers|Cached)'
df -h /models/flight-recorder/data
```
Check for filesystem errors or disk latency spikes:
```bash
iostat -x 1 3 | grep -E '(sda|nvme|vda)'
cat /proc/diskstats | tail -5
```

**5. Network & External Dependencies**
Validate connectivity to external services, model registries, or telemetry endpoints required by the benchmark.
```bash
ping -c 3 -W 2 192.168.1.116
curl -s -o /dev/null -w "%{time_total}" https://api.model-provider.internal/v1/health
traceroute -n 192.168.1.116 2>/dev/null | head -10
```

**Health Check Pass Criteria**
- Application health endpoint returns HTTP 200
- SQLite integrity check returns "ok"
- Database file permissions are correct (typically 640 or 644)
- Benchmark harness reports idle or completed state
- System load is within acceptable thresholds (< 0.8 per core)
- No critical errors in recent logs
- External dependencies are reachable

If any health check fails, the agent will pause state changes, document the failure, and proceed to the Data Safety Checks phase to ensure no further degradation occurs.

# Data Safety Checks

Data preservation is the highest priority. This section outlines rigorous verification steps to ensure the SQLite database and associated data files remain intact, consistent, and recoverable. No destructive operations will be performed. All checks are read-only or involve creating explicit backups.

**1. Database File Verification**
Confirm the physical integrity of the SQLite database file.
```bash
file /models/flight-recorder/data/benchmark.db
ls -la /models/flight-recorder/data/benchmark.db*
stat /models/flight-recorder/data/benchmark.db
```
Check for SQLite magic header and version consistency:
```bash
hexdump -C /models/flight-recorder/data/benchmark.db | head -1
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT sqlite_version();"
```

**2. Journal & WAL Mode Analysis**
SQLite uses journal files or Write-Ahead Logging (WAL) to manage transactions. Verify the mode and ensure no orphaned journal files exist.
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA journal_mode;"
ls -la /models/flight-recorder/data/benchmark.db-wal 2>/dev/null
ls -la /models/flight-recorder/data/benchmark.db-shm 2>/dev/null
ls -la /models/flight-recorder/data/benchmark.db-journal 2>/dev/null
```
If WAL mode is active, verify checkpoint status:
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA wal_checkpoint(PASSIVE);"
```

**3. Backup Creation**
Create a verified backup before any state changes. Use both file-level copy and SQLite-native backup for redundancy.
```bash
cp /models/flight-recorder/data/benchmark.db /models/flight-recorder/data/benchmark.db.pre-remediation.bak
sqlite3 /models/flight-recorder/data/benchmark.db ".backup /models/flight-recorder/data/benchmark.db.pre-remediation.sqlite3.bak"
md5sum /models/flight-recorder/data/benchmark.db /models/flight-recorder/data/benchmark.db.pre-remediation.bak
md5sum /models/flight-recorder/data/benchmark.db /models/flight-recorder/data/benchmark.db.pre-remediation.sqlite3.bak
```
Verify backup integrity:
```bash
sqlite3 /models/flight-recorder/data/benchmark.db.pre-remediation.sqlite3.bak "PRAGMA integrity_check;"
```

**4. Schema & Data Consistency**
Validate table structures, indexes, and foreign key constraints without modifying data.
```bash
sqlite3 /models/flight-recorder/data/benchmark.db ".schema"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA foreign_key_list(benchmark_results);"
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT count(*) FROM benchmark_results;"
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT count(*) FROM model_artifacts;"
```
Check for orphaned records or constraint violations:
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA foreign_keys = ON; PRAGMA integrity_check;"
```

**5. Directory & File Integrity**
Ensure the data directory structure is intact and no files are corrupted or truncated.
```bash
find /models/flight-recorder/data -type f -exec file {} \; | grep -v "SQLite" | grep -v "JSON" | grep -v "YAML" | grep -v "CSV" | grep -v "PNG" | grep -v "JPEG" | grep -v "PDF" | grep -v "TXT" | grep -v "LOG" | grep -v "PID" | grep -v "LOCK"
ls -laR /models/flight-recorder/data/ | head -50
```
Verify disk space and inode availability:
```bash
df -i /models/flight-recorder/data
df -h /models/flight-recorder/data
```

**6. Permission & Ownership Validation**
Ensure the application user has appropriate read/write access without excessive privileges.
```bash
id user
stat -c '%U:%G %a' /models/flight-recorder/data
getfacl /models/flight-recorder/data 2>/dev/null || echo "ACL not configured"
```

**Data Safety Pass Criteria**
- SQLite integrity check returns "ok"
- Backup files match source checksums
- Schema and foreign key constraints are valid
- No orphaned journal or WAL files
- Disk space > 20% free
- Permissions align with application user
- No files under `/models/flight-recorder/data` are deleted, moved, or overwritten without explicit approval

If any data safety check fails, the agent will halt immediately, alert the human operator, and await instructions. No redeployment will proceed until data integrity is confirmed.

# Redeploy Procedure

With health and data safety verified, the agent will proceed with a controlled redeployment of the application code. This phase emphasizes graceful service management, dependency verification, and stateless configuration updates. The goal is to minimize downtime and ensure the new codebase integrates seamlessly with the existing data layer.

**1. Graceful Service Termination**
Stop the application service using the appropriate init system or process manager. Send SIGTERM to allow graceful shutdown, then verify termination.
```bash
systemctl stop flight-recorder-app
# OR
kill -TERM $(cat /var/run/flight-recorder.pid 2>/dev/null)
sleep 5
ps aux | grep flight-recorder | grep -v grep
```
If the service does not terminate within 10 seconds, escalate to SIGINT, then SIGKILL only if necessary.
```bash
kill -INT $(pgrep -f flight-recorder-app) 2>/dev/null
sleep 3
kill -KILL $(pgrep -f flight-recorder-app) 2>/dev/null
```

**2. Code & Dependency Update**
Pull the latest application code and verify dependency consistency.
```bash
cd /models/flight-recorder/app
git pull origin main
pip install -r requirements.txt --quiet
# OR
npm ci --production
```
Verify that the codebase matches the expected commit hash and that no untracked files interfere with execution:
```bash
git rev-parse HEAD
git status --porcelain
```

**3. Configuration Validation**
Ensure configuration files are syntactically valid and align with the new codebase version.
```bash
python3 -c "import yaml; yaml.safe_load(open('config.yaml'))"
python3 -c "import json; json.load(open('.env'))" 2>/dev/null || echo "Env file not JSON"
```
Check for environment variable overrides that might conflict with the new deployment:
```bash
env | grep -E '(FLIGHT|BENCHMARK|DB_|APP_)'
```

**4. Database Migration & Schema Check**
If the application uses migrations, run them in a read-only or dry-run mode first.
```bash
python3 manage.py showmigrations 2>/dev/null || echo "No migration framework detected"
python3 manage.py migrate --check 2>/dev/null || echo "Migration check skipped"
```
Verify that the SQLite database schema is compatible with the new code:
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT name FROM sqlite_master WHERE type='table';"
```

**5. Service Initialization & Startup**
Start the application service and monitor startup logs for errors.
```bash
systemctl start flight-recorder-app
# OR
nohup python3 app.py > /models/flight-recorder/app/logs/startup.log 2>&1 &
```
Verify service readiness:
```bash
systemctl status flight-recorder-app
curl -s -f http://192.168.1.116:8080/health
tail -n 50 /models/flight-recorder/app/logs/startup.log
```

**6. Post-Deployment Verification**
Run a quick smoke test to ensure the application responds correctly and can access the database.
```bash
curl -s http://192.168.1.116:8080/api/v1/status
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT count(*) FROM benchmark_results;"
```
Monitor resource usage during startup to ensure no memory leaks or CPU spikes:
```bash
top -b -n 1 | grep -E '(python|node|gunicorn)'
```

**Redeploy Pass Criteria**
- Service stops gracefully within 10 seconds
- Code pulls successfully with no conflicts
- Dependencies install without errors
- Configuration validates successfully
- Service starts and health endpoint responds
- Database access works without errors
- No unexpected log warnings or errors

If the redeployment fails at any step, the agent will roll back to the pre-remediation state using the recovery commands outlined later in this plan.

# Benchmark Restart Procedure

Once the application is redeployed and verified, the benchmark harness can be safely restarted. This phase focuses on progressive validation, environment isolation, and result reproducibility. The agent will avoid sudden high-load injections until baseline stability is confirmed.

**1. Environment Preparation**
Ensure the benchmark runner has access to required resources, model artifacts, and configuration files.
```bash
ls -la /models/flight-recorder/data/models/
cat /models/flight-recorder/app/config/benchmark_config.json
python3 -c "import json; c=json.load(open('/models/flight-recorder/app/config/benchmark_config.json')); print('Config valid')"
```
Verify that the benchmark user/context has appropriate permissions:
```bash
id benchmark-runner
ls -la /models/flight-recorder/data/benchmark_runs/
```

**2. Dry Run & Configuration Validation**
Execute a dry run to validate configuration, model loading, and data pipeline connectivity without executing full benchmarks.
```bash
python3 /models/flight-recorder/app/benchmark_runner.py --dry-run --config /models/flight-recorder/app/config/benchmark_config.json
```
Check for missing dependencies, model files, or database connection issues:
```bash
python3 -c "
import sys
sys.path.append('/models/flight-recorder/app')
from benchmark_engine import load_model, validate_config
try:
    validate_config('/models/flight-recorder/app/config/benchmark_config.json')
    print('Config validation passed')
except Exception as e:
    print(f'Config validation failed: {e}')
"
```

**3. Progressive Load Testing**
Start the benchmark with a reduced workload to verify stability before scaling to full load.
```bash
python3 /models/flight-recorder/app/benchmark_runner.py --config /models/flight-recorder/app/config/benchmark_config.json --scale 0.2 --output /models/flight-recorder/data/benchmark_runs/dry_run_$(date +%Y%m%d_%H%M%S).json
```
Monitor execution logs and resource usage:
```bash
tail -f /models/flight-recorder/app/logs/benchmark_runner.log
top -b -n 1 | grep -E '(python|benchmark)'
```

**4. Full Benchmark Execution**
Once dry run passes, execute the full benchmark suite.
```bash
python3 /models/flight-recorder/app/benchmark_runner.py --config /models/flight-recorder/app/config/benchmark_config.json --scale 1.0 --output /models/flight-recorder/data/benchmark_runs/full_$(date +%Y%m%d_%H%M%S).json
```
Track progress and handle failures gracefully:
```bash
watch -n 5 'cat /models/flight-recorder/data/benchmark_runs/full_*.json 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get(\"status\",\"unknown\"))"'
```

**5. Result Validation & Reporting**
Verify that benchmark results are written correctly and match expected schemas.
```bash
python3 -c "
import json, os
files = [f for f in os.listdir('/models/flight-recorder/data/benchmark_runs/') if f.endswith('.json')]
for f in files:
    with open(f'/models/flight-recorder/data/benchmark_runs/{f}') as fh:
        data = json.load(fh)
        print(f'{f}: status={data.get(\"status\",\"missing\")}, metrics={list(data.get(\"metrics\",{}).keys())}')
"
```
Archive results and update benchmark registry:
```bash
cp /models/flight-recorder/data/benchmark_runs/full_*.json /models/flight-recorder/data/benchmark_runs/archive/
```

**Benchmark Restart Pass Criteria**
- Dry run completes without errors
- Progressive load test shows stable resource usage
- Full benchmark executes and produces valid JSON results
- Result schema matches expected structure
- No database corruption or lock contention during execution

If the benchmark fails, the agent will capture logs, isolate the failure mode, and proceed to the Stop Conditions or Recovery Commands sections as appropriate.

# Human Approval Gates

Certain operations carry inherent risks to data integrity, service stability, or system state. These gates require explicit human approval before execution. The agent will pause and await confirmation before proceeding.

**[APPROVAL REQUIRED] Destructive File Operations**
```bash
rm -rf /models/flight-recorder/data/benchmark.db
rm -rf /models/flight-recorder/data/benchmark.db-wal
rm -rf /models/flight-recorder/data/benchmark.db-shm
rm -rf /models/flight-recorder/data/benchmark.db-journal
```
*Rationale:* Deleting database files or journal/WAL files can cause irreversible data loss. Only permitted if explicitly authorized for corruption recovery.

**[APPROVAL REQUIRED] Forceful Process Termination**
```bash
kill -9 $(pgrep -f flight-recorder-app)
kill -9 $(pgrep -f benchmark_runner)
```
*Rationale:* SIGKILL bypasses graceful shutdown, potentially leaving database locks or partial writes. Only permitted if graceful termination fails repeatedly.

**[APPROVAL REQUIRED] Database Rebuild or Schema Reset**
```bash
sqlite3 /models/flight-recorder/data/benchmark.db "DROP TABLE IF EXISTS benchmark_results;"
sqlite3 /models/flight-recorder/data/benchmark.db "DROP TABLE IF EXISTS model_artifacts;"
```
*Rationale:* Dropping tables destroys historical benchmark data. Only permitted if explicitly authorized for schema migration or corruption recovery.

**[APPROVAL REQUIRED] Disk Cleanup or Cache Flushing**
```bash
sync; echo 3 > /proc/sys/vm/drop_caches
rm -rf /tmp/benchmark_cache/*
truncate -s 0 /models/flight-recorder/app/logs/*.log
```
*Rationale:* Clearing caches or truncating logs can impact performance or remove critical diagnostic data. Only permitted if explicitly authorized for space recovery or log rotation.

**[APPROVAL REQUIRED] Network Reset or Firewall Modification**
```bash
iptables -F
systemctl restart firewalld
ip link set eth0 down && ip link set eth0 up
```
*Rationale:* Network resets can disrupt active connections, benchmark traffic, or external dependencies. Only permitted if explicitly authorized for network troubleshooting.

**[APPROVAL REQUIRED] System Reboot or Service Stack Restart**
```bash
reboot
systemctl restart nginx
systemctl restart postgresql
```
*Rationale:* Reboots or stack restarts cause downtime and may interrupt ongoing benchmarks. Only permitted if explicitly authorized for system-level recovery.

The agent will log all approval requests, wait for explicit confirmation, and proceed only after authorization. If approval is denied, the agent will halt the corresponding action and document the decision.

# Stop Conditions

The remediation run includes explicit stop conditions that trigger immediate halting of all state changes. These conditions are designed to prevent data loss, service degradation, or benchmark corruption.

**1. Data Integrity Failure**
- SQLite integrity check returns anything other than "ok"
- Backup verification fails checksum mismatch
- Database file is truncated, corrupted, or locked by another process
- Schema validation fails with constraint violations

**2. Resource Exhaustion**
- Disk space falls below 10% on `/models/flight-recorder/data`
- Memory usage exceeds 95% with OOM killer active
- CPU load average exceeds 3x core count for > 5 minutes
- File descriptor limit reached (`ulimit -n` exceeded)

**3. Service Instability**
- Application health endpoint returns HTTP 5xx or times out
- Service crashes or restarts > 3 times within 10 minutes
- Worker processes fail to spawn or terminate unexpectedly
- Database connection pool exhausted or connection refused

**4. Benchmark Execution Failure**
- Dry run fails with configuration or dependency errors
- Progressive load test shows > 50% failure rate
- Full benchmark produces invalid or empty result files
- Result schema validation fails or metrics are missing

**5. Unexpected State Changes**
- Files under `/models/flight-recorder/data` are modified without authorization
- Configuration files are altered outside of controlled redeployment
- Network state changes unexpectedly (ports closed, routes lost)
- Log files show critical errors or security alerts

**6. Human Abort Signal**
- Operator sends explicit abort command
- Approval gate remains unresponded for > 15 minutes
- Manual intervention requested via monitoring dashboard

Upon triggering any stop condition, the agent will:
1. Halt all ongoing operations
2. Capture current state logs and telemetry
3. Notify the human operator with detailed failure context
4. Await instructions before proceeding or initiating recovery

# Recovery Commands

If the remediation run encounters critical failures or requires rollback, the following recovery commands provide a structured path to restore stability without data loss. All recovery steps prioritize data preservation and service restoration.

**1. Service Rollback**
```bash
systemctl stop flight-recorder-app
cd /models/flight-recorder/app
git checkout <pre-remediation-commit-hash>
pip install -r requirements.txt --quiet
systemctl start flight-recorder-app
curl -s -f http://192.168.1.116:8080/health
```

**2. Database Recovery**
```bash
cp /models/flight-recorder/data/benchmark.db.pre-remediation.bak /models/flight-recorder/data/benchmark.db
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA journal_mode=DELETE;"
chmod 640 /models/flight-recorder/data/benchmark.db
chown user:user /models/flight-recorder/data/benchmark.db
```

**3. Log & State Dump**
```bash
journalctl -u flight-recorder-app --since "1 hour ago" > /tmp/flight-recorder-recovery.log
cp /models/flight-recorder/app/logs/*.log /tmp/benchmark-recovery-logs/
tar -czf /tmp/recovery-state-$(date +%Y%m%d_%H%M%S).tar.gz /models/flight-recorder/data/benchmark.db.pre-remediation.bak /tmp/flight-recorder-recovery.log /tmp/benchmark-recovery-logs/
```

**4. Network & Port Recovery**
```bash
ss -tlnp | grep -E '(8000|8080|5000|9000)'
systemctl restart nginx 2>/dev/null || echo "Nginx not configured"
iptables -L -n | grep -E '(8080|5000)'
```

**5. Benchmark Harness Reset**
```bash
rm -f /models/flight-recorder/data/benchmark_runs/active.lock
python3 /models/flight-recorder/app/benchmark_runner.py --reset --config /models/flight-recorder/app/config/benchmark_config.json
ls -la /models/flight-recorder/data/benchmark_runs/
```

**6. System-Level Recovery**
```bash
sync
echo 1 > /proc/sys/vm/drop_caches 2>/dev/null || echo "Drop caches requires root"
systemctl daemon-reload
systemctl restart flight-recorder-app
```

Recovery commands are executed only after stop conditions are met or human approval is granted. The agent will verify each step's success before proceeding to the next.

# Summary

This remediation plan provides a comprehensive, safety-first framework for restoring reliable benchmark execution on the remote model server at `192.168.1.116`. The process is structured to prioritize data integrity, service stability, and benchmark reproducibility through a phased approach: read-only inspection, health validation, data safety verification, controlled redeployment, and progressive benchmark restart.

Key operational principles include:
- **Non-Destructive Baseline:** All initial inspections and health checks are strictly read-only. No files are modified, moved, or deleted without explicit human approval.
- **Data Preservation:** SQLite database integrity is verified through multiple validation layers, including integrity checks, WAL/journal analysis, and redundant backup creation. The data directory is never deleted or overwritten.
- **Controlled Redeployment:** Service termination and startup follow graceful protocols. Code updates are validated against dependencies and configuration schemas before activation.
- **Progressive Benchmark Validation:** The benchmark harness is restarted with dry runs and scaled load testing to ensure stability before full execution.
- **Explicit Approval Gates:** High-risk operations are clearly labeled and require human authorization. The agent will pause and await confirmation before proceeding.
- **Clear Stop Conditions:** Automated and manual triggers halt the run immediately if data integrity, resource availability, or service stability is compromised.
- **Structured Recovery:** Rollback and recovery commands are pre-defined to restore state quickly and safely without data loss.

The agent will execute this plan methodically, logging all actions, verifying state transitions, and maintaining strict operational discipline. Success is defined by stable service operation, intact data, and reproducible benchmark results. Any deviations from expected state will trigger immediate halting and operator notification. This plan ensures that the remediation run is predictable, safe, and aligned with the goal of restoring reliable benchmark execution without compromising data integrity.