# Mission
The primary objective of this remediation run is to restore reliable benchmark execution on the remote model server located at `192.168.1.116` while guaranteeing absolute data preservation. The operational scope encompasses the application codebase residing at `/models/flight-recorder/app`, the persistent data store located at `/models/flight-recorder/data`, and the underlying SQLite database engine. Given that the model server may be actively processing inference requests or benchmark workloads, all actions must be executed with a non-disruptive, read-first philosophy. The agent will operate under a strict zero-data-loss policy: no files, directories, or database pages will be removed, truncated, or overwritten without explicit, documented human approval. Even then, deletion of the `/models/flight-recorder/data` directory or its contents is strictly prohibited under this operational directive.

Success is defined by the following measurable outcomes:
1. The application service is running in a stable, healthy state with clean startup logs.
2. The SQLite database passes integrity checks and maintains full read/write capability without corruption or locking deadlocks.
3. The benchmark runner can submit workloads, receive responses, and record results without timeout errors, connection drops, or data loss.
4. All pre- and post-remediation states are auditable via logs, checksums, and backup artifacts.
5. The system remains responsive to human oversight throughout the entire lifecycle of the run.

This plan is structured to be executed sequentially, with mandatory verification checkpoints between phases. Each phase includes explicit commands, expected outputs, interpretation guidelines, and fallback procedures. The agent will pause at designated approval gates, await explicit authorization, and log all decisions for post-run audit. The operational posture is defensive, verification-heavy, and rollback-ready at every stage.

# Read-Only Inspection
Before any modification or service manipulation occurs, the agent must establish a comprehensive baseline of the server's current state. This phase is strictly non-destructive and focuses on gathering telemetry, verifying file system integrity, and understanding the runtime environment.

**1. Secure Shell Access & Environment Verification**
Establish a persistent SSH session with verbose logging enabled for audit purposes:
```bash
ssh -o LogLevel=VERBOSE -o ServerAliveInterval=30 -o ServerAliveCountMax=3 user@192.168.1.116
```
Once connected, verify the operating environment and user privileges:
```bash
uname -a
whoami
id
df -h /models/flight-recorder
```
Expected output should confirm the kernel version, user identity, and available disk space on the mount containing the application and data directories. If disk usage exceeds 85%, flag it for immediate attention before proceeding.

**2. Process & Service Discovery**
Identify all running processes related to the flight-recorder application, model server, and benchmark runner:
```bash
ps aux | grep -E "flight-recorder|benchmark|model-server|python|node|gunicorn|uvicorn" | grep -v grep
systemctl list-units --type=service --state=running | grep -E "flight|benchmark|model"
```
Cross-reference process IDs (PIDs) with listening network ports to map service topology:
```bash
ss -tlnp | grep -E "80|443|8000|8080|5000|9090"
lsof -i -P -n | grep LISTEN
```
Document which PIDs correspond to which services. Note any zombie processes, orphaned threads, or unexpected port bindings.

**3. Application Codebase Inspection**
Verify the integrity and structure of the application directory without modifying any files:
```bash
ls -laR /models/flight-recorder/app | head -100
find /models/flight-recorder/app -type f \( -name "*.py" -o -name "*.js" -o -name "*.toml" -o -name "*.yaml" -o -name "*.env" \) | sort
stat /models/flight-recorder/app
```
Check for recent modifications that might indicate a failed deployment or manual patch:
```bash
find /models/flight-recorder/app -type f -mtime -1 -ls
```
If version control is present, inspect the repository state:
```bash
cd /models/flight-recorder/app && git status && git log --oneline -5
```
Record the current commit hash, branch, and any uncommitted changes. This establishes the baseline for potential rollback.

**4. Data Directory & SQLite Database Inspection**
Examine the data directory structure and file permissions:
```bash
ls -laR /models/flight-recorder/data | head -100
find /models/flight-recorder/data -type f -name "*.db" -o -name "*.sqlite" -o -name "*.wal" -o -name "*.shm"
```
Verify file ownership and permissions to ensure the application user has read/write access:
```bash
namei -l /models/flight-recorder/data
```
Perform a read-only SQLite integrity check on each database file:
```bash
for db in /models/flight-recorder/data/*.db; do
  echo "Checking $db..."
  sqlite3 "$db" "PRAGMA integrity_check;"
  sqlite3 "$db" "PRAGMA journal_mode;"
  sqlite3 "$db" "PRAGMA page_count;"
done
```
Expected output: `ok` for integrity checks, `wal` or `delete` for journal mode, and a positive integer for page count. Any deviation (e.g., `corrupt`, `empty`, or permission denied) must be logged and escalated.

**5. Log & Telemetry Review**
Collect recent application and system logs to identify error patterns, startup failures, or resource exhaustion:
```bash
journalctl -u flight-recorder --no-pager -n 200 --since "1 hour ago"
tail -n 100 /var/log/flight-recorder/*.log 2>/dev/null || echo "No log directory found"
dmesg -T | tail -50
```
Search for critical keywords:
```bash
grep -iE "error|fatal|panic|timeout|connection refused|disk full|permission denied" /var/log/flight-recorder/*.log 2>/dev/null | tail -50
```
Document recurring errors, stack traces, or warning patterns. This information will guide the redeployment and benchmark restart phases.

**6. Resource & Network Baseline**
Capture system resource utilization to establish a performance baseline:
```bash
top -bn1 | head -30
free -m
vmstat 1 5
iostat -x 1 3
ping -c 5 192.168.1.116
curl -s -o /dev/null -w "%{http_code} %{time_total}\n" http://192.168.1.116:8000/health 2>/dev/null || echo "Health endpoint unreachable"
```
Record CPU load averages, memory pressure, swap usage, disk I/O wait times, network latency, and HTTP response codes. This data will be compared against post-remediation metrics to verify stability.

All inspection outputs must be saved to a timestamped audit file:
```bash
mkdir -p /tmp/flight-recorder-audit/$(date +%Y%m%d_%H%M%S)
tar -czf /tmp/flight-recorder-audit/$(date +%Y%m%d_%H%M%S)/inspection_baseline.tar.gz -C /tmp/flight-recorder-audit/$(date +%Y%m%d_%H%M%S)/ .
```
The agent will not proceed until the baseline is fully documented and verified.

# Health Checks
Health checks serve as continuous validation gates throughout the remediation run. They ensure that the system remains within operational bounds and that no latent issues are introduced during transitions.

**1. System-Level Health Validation**
Monitor core system resources at regular intervals:
```bash
watch -n 10 'echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk "{print \$2+\$4}%") | MEM: $(free -m | awk "/Mem:/ {printf \"%.1f%%\", \$3/\$2*100}) | DISK: $(df -h /models | awk "NR==2 {print \$5}") | LOAD: $(uptime | awk "{print \$NF}")"'
```
Thresholds for healthy operation:
- CPU utilization: < 80% sustained
- Memory usage: < 85% (excluding cache/buffers)
- Disk usage: < 80% on `/models` partition
- Load average: < number of CPU cores
- Swap usage: < 10% (preferably 0%)

If any threshold is breached, the agent will pause and trigger a diagnostic routine before proceeding.

**2. Application Service Health**
Verify that the flight-recorder service is responsive and processing requests correctly:
```bash
curl -s -w "\nHTTP_CODE:%{http_code}\nTIME:%{time_total}\n" http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/metrics | grep -E "up|error|latency|queue"
```
Expected output: HTTP 200, response time < 200ms, `up 1`, error rate < 0.1%, queue depth < 10. If the health endpoint returns 5xx or times out, the service is considered unhealthy.

**3. Database Health & Lock Monitoring**
SQLite databases can suffer from locking contention, especially under concurrent benchmark loads. Monitor lock status and journal state:
```bash
sqlite3 /models/flight-recorder/data/main.db "PRAGMA locking_mode;"
sqlite3 /models/flight-recorder/data/main.db "PRAGMA wal_checkpoint(PASSIVE);"
ls -lh /models/flight-recorder/data/*.db-wal /models/flight-recorder/data/*.db-shm 2>/dev/null
```
If the WAL file exceeds 1GB or the checkpoint returns `BUSY`, the database may be experiencing write contention. In such cases, the agent will recommend reducing concurrent benchmark threads or implementing connection pooling.

**4. Model Server & Inference Queue Health**
If the model server exposes a status endpoint, query it for queue depth and GPU/CPU utilization:
```bash
curl -s http://127.0.0.1:8000/model/status | jq .
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader,nounits 2>/dev/null || echo "GPU metrics unavailable"
```
Expected output: queue depth < 20, GPU utilization < 90%, memory within allocated limits. If the model server is saturated, the benchmark restart procedure will implement backoff and rate limiting.

**5. Network & Connectivity Health**
Verify that internal and external network paths are stable:
```bash
ping -c 3 8.8.8.8
curl -s -o /dev/null -w "%{time_connect} %{time_starttransfer} %{time_total}\n" http://192.168.1.116:8000/health
ss -s
```
Monitor connection states for TIME_WAIT accumulation or SYN_RECV floods, which may indicate network congestion or misconfigured keep-alive settings.

**6. Continuous Health Monitoring Loop**
During active phases, run a background health monitor:
```bash
nohup bash -c 'while true; do curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/health >> /tmp/health_check.log 2>&1; sleep 5; done' &
```
This log will be analyzed post-run to detect intermittent failures or latency spikes.

All health checks are non-intrusive and read-only. The agent will log results at each checkpoint and halt progression if critical thresholds are violated.

# Data Safety Checks
Data preservation is the highest priority of this remediation run. The following procedures ensure that all persistent state is backed up, verified, and protected against accidental modification or corruption.

**1. Pre-Operation Backup Creation**
Before any service manipulation or code deployment, create a complete, timestamped backup of the data directory:
```bash
BACKUP_DIR="/models/flight-recorder/data.backup.$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"
cp -a /models/flight-recorder/data/* "$BACKUP_DIR/"
```
The `-a` flag preserves permissions, timestamps, and symbolic links. This operation is read-only from the source perspective and creates a duplicate copy.

**2. Backup Verification & Integrity Validation**
Verify that the backup matches the original directory exactly:
```bash
diff -r /models/flight-recorder/data "$BACKUP_DIR"
echo "Diff exit code: $?"
```
An exit code of `0` indicates identical content. Any differences must be investigated before proceeding.

**3. SQLite Database Backup & Checksumming**
Create a dedicated SQLite backup using the built-in `.backup` command, which ensures a consistent snapshot even under concurrent reads:
```bash
sqlite3 /models/flight-recorder/data/main.db ".backup '/models/flight-recorder/data/main.db.backup'"
sha256sum /models/flight-recorder/data/main.db > /tmp/db_checksums_original.txt
sha256sum /models/flight-recorder/data/main.db.backup > /tmp/db_checksums_backup.txt
diff /tmp/db_checksums_original.txt /tmp/db_checksums_backup.txt
```
Record the checksums for post-remediation comparison. If the backup fails or checksums mismatch, halt immediately and notify the operator.

**4. Read-Only Mount Simulation (Optional Safety Layer)**
To prevent accidental writes during inspection, temporarily remount the data directory as read-only:
```bash
mount -o remount,ro /models/flight-recorder/data
```
Verify that write operations are blocked:
```bash
touch /models/flight-recorder/data/test_write 2>&1 || echo "Write blocked successfully"
```
Remount as read-write before proceeding to deployment:
```bash
mount -o remount,rw /models/flight-recorder/data
```
This step is optional but recommended for high-integrity environments.

**5. Data Directory Access Control Verification**
Ensure that only authorized users and processes can access the data directory:
```bash
ls -ld /models/flight-recorder/data
getfacl /models/flight-recorder/data
```
Verify that permissions are set to `750` or `700`, owned by the application user and group. Restrictive permissions prevent unauthorized modifications and reduce the attack surface.

**6. Continuous Data Integrity Monitoring**
During the run, periodically verify that the database remains intact:
```bash
sqlite3 /models/flight-recorder/data/main.db "PRAGMA integrity_check;" | tail -1
```
If `ok` is not returned, trigger an immediate stop condition and initiate recovery procedures.

**7. Explicit Prohibition of Data Deletion**
Under no circumstances will the agent execute commands that remove, truncate, or overwrite files within `/models/flight-recorder/data`. Commands such as `rm`, `truncate`, `>`, or `dd` targeting this directory are strictly forbidden. Any request to delete data must be explicitly approved in writing by the human operator, and even then, the data directory itself will never be removed.

All data safety checks are logged, timestamped, and stored in the audit directory. The agent will not proceed until backup verification passes and integrity checks return `ok`.

# Redeploy Procedure
The redeployment phase replaces or updates the application code while preserving all persistent data. This procedure is designed to be atomic, reversible, and minimally disruptive.

**1. Pre-Deployment Verification**
Confirm that backups exist and are verified:
```bash
ls -lh /models/flight-recorder/data.backup.*
diff -r /models/flight-recorder/data /models/flight-recorder/data.backup.* | head -5
```
Verify that the current application state is documented:
```bash
cd /models/flight-recorder/app && git rev-parse HEAD
```
Ensure that the system is in a low-traffic window or that the benchmark runner is paused.

**2. Graceful Service Stop**
Stop the flight-recorder service gracefully to allow pending requests to complete:
```bash
[APPROVAL REQUIRED] systemctl stop flight-recorder
```
Wait for the process to terminate:
```bash
sleep 5
ps aux | grep flight-recorder | grep -v grep
```
If the process remains active, send a SIGTERM:
```bash
[APPROVAL REQUIRED] kill -SIGTERM $(pgrep -f flight-recorder)
```
Verify termination:
```bash
systemctl status flight-recorder
```

**3. Application Code Backup**
Create a backup of the current application directory before overwriting:
```bash
tar -czf /tmp/app_backup_$(date +%Y%m%d_%H%M%S).tar.gz -C /models/flight-recorder app
sha256sum /tmp/app_backup_*.tar.gz > /tmp/app_checksums.txt
```

**4. Code Deployment**
Transfer the updated application code to the server. Use `rsync` for efficient, incremental synchronization:
```bash
[APPROVAL REQUIRED] rsync -avz --exclude=".git" --exclude="__pycache__" /local/path/to/flight-recorder/app/ /models/flight-recorder/app/
```
Verify deployment integrity:
```bash
ls -la /models/flight-recorder/app
find /models/flight-recorder/app -type f -name "*.py" -exec python -m py_compile {} \; 2>&1 | grep -i error || echo "Syntax check passed"
```
If version control is used, verify the deployed commit:
```bash
cd /models/flight-recorder/app && git status && git log --oneline -1
```

**5. Dependency & Configuration Validation**
Ensure that all required dependencies are installed and configuration files are valid:
```bash
cd /models/flight-recorder/app && pip install -r requirements.txt --dry-run 2>&1 | tail -10
python -c "import yaml; yaml.safe_load(open('config.yaml'))" 2>&1 || echo "Config validation failed"
```
Verify environment variables and secrets:
```bash
env | grep -E "FLIGHT|RECORD|DB|MODEL" | sed 's/=.*/=***/'
```

**6. Service Start & Health Verification**
Start the service and monitor startup logs:
```bash
[APPROVAL REQUIRED] systemctl start flight-recorder
journalctl -u flight-recorder -f --no-pager
```
Wait for the health endpoint to respond:
```bash
for i in {1..30}; do
  curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/health | grep -q 200 && break
  sleep 2
done
```
If the service fails to start after 3 retries, trigger rollback:
```bash
[APPROVAL REQUIRED] systemctl stop flight-recorder
tar -xzf /tmp/app_backup_*.tar.gz -C /models/flight-recorder
[APPROVAL REQUIRED] systemctl start flight-recorder
```

**7. Post-Deployment Validation**
Run a lightweight smoke test to verify core functionality:
```bash
curl -s -X POST http://127.0.0.1:8000/api/test -H "Content-Type: application/json" -d '{"test": true}' | jq .
```
Verify that database connections are established and queries execute without errors:
```bash
sqlite3 /models/flight-recorder/data/main.db "SELECT COUNT(*) FROM benchmarks;" 2>&1
```
Log all deployment artifacts, checksums, and health check results to the audit directory.

The redeployment procedure is complete only when the service is running, health checks pass, and smoke tests succeed. The agent will pause for human approval before proceeding to benchmark restart.

# Benchmark Restart Procedure
Restarting the benchmark runner requires careful coordination with the model server to avoid overwhelming the system, causing timeouts, or corrupting results. This procedure implements gradual load introduction, queue management, and continuous monitoring.

**1. Pre-Restart System Readiness Check**
Verify that the application service is healthy and the database is responsive:
```bash
curl -s http://127.0.0.1:8000/health | jq .
sqlite3 /models/flight-recorder/data/main.db "PRAGMA integrity_check;" | tail -1
```
Ensure that the benchmark runner service is stopped or idle:
```bash
systemctl status benchmark-runner
```

**2. Clear Stale Sessions & Queues**
Remove any leftover benchmark sessions or queued tasks that may cause conflicts:
```bash
sqlite3 /models/flight-recorder/data/main.db "DELETE FROM benchmark_queue WHERE status='pending' AND created_at < datetime('now', '-1 hour');"
sqlite3 /models/flight-recorder/data/main.db "DELETE FROM benchmark_sessions WHERE status='running' AND updated_at < datetime('now', '-30 minutes');"
```
Verify cleanup:
```bash
sqlite3 /models/flight-recorder/data/main.db "SELECT COUNT(*) FROM benchmark_queue;"
```

**3. Gradual Benchmark Restart**
Start the benchmark runner with minimal concurrency to verify stability:
```bash
[APPROVAL REQUIRED] systemctl start benchmark-runner
```
Monitor initial requests:
```bash
watch -n 5 'curl -s http://127.0.0.1:8000/metrics | grep -E "benchmark|error|latency"'
```
If error rate < 1% and p95 latency < threshold, gradually increase concurrency:
```bash
# Example: Update benchmark config to increase workers
sed -i 's/workers: 1/workers: 4/' /models/flight-recorder/app/config/benchmark.yaml
systemctl reload benchmark-runner
```
Repeat scaling steps until target throughput is reached, monitoring health at each stage.

**4. Connection Pool & Timeout Management**
Ensure that the benchmark runner uses connection pooling and appropriate timeouts to prevent resource exhaustion:
```bash
grep -E "pool_size|timeout|retry" /models/flight-recorder/app/config/benchmark.yaml
```
Recommended settings:
- `pool_size: 10`
- `request_timeout: 30s`
- `retry_attempts: 3`
- `backoff_strategy: exponential`

**5. Model Server Load Balancing & Backoff**
If the model server is busy, implement client-side backoff and queue draining:
```bash
curl -s http://127.0.0.1:8000/model/status | jq '.queue_depth'
```
If queue depth > 50, pause benchmark submission and wait for drain:
```bash
sleep 10
curl -s http://127.0.0.1:8000/model/status | jq '.queue_depth'
```
Resume only when queue depth < 20.

**6. Continuous Benchmark Monitoring**
Run a monitoring loop to track benchmark health:
```bash
nohup bash -c 'while true; do curl -s http://127.0.0.1:8000/metrics | grep -E "benchmark_success|benchmark_error|benchmark_latency" >> /tmp/benchmark_metrics.log 2>&1; sleep 10; done' &
```
Set up alerting for error rate spikes:
```bash
awk '/benchmark_error/ {if ($2 > 0.05) print "ALERT: Error rate exceeded 5%"}' /tmp/benchmark_metrics.log
```

**7. Post-Restart Validation**
Verify that benchmark results are being recorded correctly:
```bash
sqlite3 /models/flight-recorder/data/main.db "SELECT COUNT(*) FROM benchmark_results WHERE created_at > datetime('now', '-5 minutes');"
```
Check for data consistency:
```bash
sqlite3 /models/flight-recorder/data/main.db "PRAGMA integrity_check;" | tail -1
```
If all metrics are within acceptable bounds, the benchmark restart is considered successful.

The agent will log all scaling steps, metric trends, and validation results. Human approval is required before increasing concurrency beyond initial safe thresholds.

# Human Approval Gates
This remediation run includes explicit approval gates to ensure human oversight at critical junctures. The agent will pause execution, display the pending action, and await explicit authorization before proceeding. All approvals are logged with timestamps and operator identifiers.

**Gate 1: Pre-Deployment Service Stop**
Before stopping the flight-recorder service, the agent will pause and request approval:
```
[APPROVAL REQUIRED] systemctl stop flight-recorder
Reason: Graceful shutdown required for code deployment.
Impact: Service will be unavailable for ~30 seconds.
Proceed? (APPROVE/DENY)
```

**Gate 2: Application Code Overwrite**
Before deploying new code, the agent will verify backup existence and request approval:
```
[APPROVAL REQUIRED] rsync -avz --exclude=".git" --exclude="__pycache__" /local/path/to/flight-recorder/app/ /models/flight-recorder/app/
Reason: Deploy updated application code.
Backup verified: /tmp/app_backup_*.tar.gz
Proceed? (APPROVE/DENY)
```

**Gate 3: Benchmark Runner Start & Scaling**
Before starting or scaling the benchmark runner, the agent will request approval:
```
[APPROVAL REQUIRED] systemctl start benchmark-runner
Reason: Restart benchmark execution after remediation.
Initial concurrency: 1 worker. Scaling pending validation.
Proceed? (APPROVE/DENY)
```

**Gate 4: Database Maintenance Operations**
Any SQLite maintenance commands (e.g., VACUUM, WAL checkpoint) require approval:
```
[APPROVAL REQUIRED] sqlite3 /models/flight-recorder/data/main.db "VACUUM;"
Reason: Optimize database file size and reclaim unused pages.
Impact: Brief write lock during execution.
Proceed? (APPROVE/DENY)
```

**Gate 5: Rollback Execution**
If deployment fails, the agent will request approval before rolling back:
```
[APPROVAL REQUIRED] tar -xzf /tmp/app_backup_*.tar.gz -C /models/flight-recorder
Reason: Rollback to previous application version due to startup failure.
Proceed? (APPROVE/DENY)
```

**Approval Workflow:**
1. Agent prints approval prompt with command, reason, and impact.
2. Agent pauses execution and waits for explicit `APPROVE` or `DENY` input.
3. If `APPROVE`, agent logs decision, executes command, and proceeds.
4. If `DENY`, agent logs decision, skips step, and evaluates alternative path or halts run.
5. All approvals are recorded in `/tmp/flight-recorder-audit/$(date +%Y%m%d_%H%M%S)/approvals.log`.

Destructive deletes are strictly prohibited. Even with approval, the agent will never execute commands that remove `/models/flight-recorder/data` or its contents.

# Stop Conditions
The remediation run will be halted immediately if any of the following stop conditions are met. These conditions are designed to prevent data loss, system instability, or irreversible damage.

**Hard Stop Conditions (Immediate Halt):**
1. **Data Corruption Detected:** `PRAGMA integrity_check` returns anything other than `ok`.
2. **Backup Verification Failure:** `diff -r` between original and backup directories shows mismatches.
3. **Service Startup Failure:** Flight-recorder fails to start after 3 consecutive retries.
4. **Disk Space Exhaustion:** Available disk space on `/models` drops below 5%.
5. **Database Lock Deadlock:** SQLite reports `database is locked` for > 30 seconds under normal load.
6. **Health Endpoint Unresponsive:** HTTP 5xx or timeout for > 60 seconds consecutively.
7. **Unauthorized Modification Attempt:** Any command targeting `/models/flight-recorder/data` with write/delete intent without explicit approval.

**Soft Stop Conditions (Pause & Evaluate):**
1. **High Error Rate:** Benchmark error rate exceeds 5% for > 2 minutes.
2. **Latency Spike:** p95 latency exceeds 2x baseline threshold.
3. **Resource Saturation:** CPU > 90%, Memory > 90%, or Swap > 20% sustained.
4. **Model Server Queue Saturation:** Queue depth > 100 for > 5 minutes.
5. **Network Instability:** Packet loss > 2% or latency > 200ms to internal endpoints.

**Stop Condition Response Protocol:**
1. Halt all active operations immediately.
2. Preserve current state: snapshot logs, metrics, and process lists.
3. Trigger recovery commands if applicable.
4. Notify human operator with detailed diagnostic report.
5. Await explicit instructions before resuming or terminating the run.

All stop conditions are monitored continuously via background scripts and health check loops. The agent will log the exact condition, timestamp, and system state when a stop is triggered.

# Recovery Commands
If the remediation run encounters a failure or triggers a stop condition, the following recovery commands will restore the system to a known-good state. These commands are designed to be safe, reversible, and non-destructive.

**1. Application Rollback**
Restore the previous application version from backup:
```bash
[APPROVAL REQUIRED] systemctl stop flight-recorder
tar -xzf /tmp/app_backup_*.tar.gz -C /models/flight-recorder
[APPROVAL REQUIRED] systemctl start flight-recorder
journalctl -u flight-recorder -n 50 --no-pager
```
Verify rollback success:
```bash
curl -s http://127.0.0.1:8000/health | jq .
```

**2. Database Restore**
Restore the SQLite database from the verified backup:
```bash
[APPROVAL REQUIRED] sqlite3 /models/flight-recorder/data/main.db ".restore '/models/flight-recorder/data/main.db.backup'"
sqlite3 /models/flight-recorder/data/main.db "PRAGMA integrity_check;" | tail -1
```
Verify data consistency:
```bash
sqlite3 /models/flight-recorder/data/main.db "SELECT COUNT(*) FROM benchmarks;"
```

**3. Service Restart & State Reset**
Restart the flight-recorder service and clear stale connections:
```bash
[APPROVAL REQUIRED] systemctl restart flight-recorder
sleep 5
curl -s http://127.0.0.1:8000/health | jq .
```
Clear application cache if applicable:
```bash
find /models/flight-recorder/app -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
```

**4. Network & Connection Reset**
Flush DNS cache and reset network connections if experiencing connectivity issues:
```bash
ip route flush cache
systemctl restart networking 2>/dev/null || true
ss -s
```

**5. Log Collection & Post-Mortem Archive**
Gather all relevant logs for analysis:
```bash
mkdir -p /tmp/flight-recorder-recovery/$(date +%Y%m%d_%H%M%S)
tar -czf /tmp/flight-recorder-recovery/$(date +%Y%m%d_%H%M%S)/logs.tar.gz /var/log/flight-recorder/ /models/flight-recorder/app/logs/ /tmp/health_check.log /tmp/benchmark_metrics.log 2>/dev/null
journalctl -u flight-recorder --no-pager > /tmp/flight-recorder-recovery/$(date +%Y%m%d_%H%M%S)/journal.log
```
Archive audit and approval logs:
```bash
cp -r /tmp/flight-recorder-audit/* /tmp/flight-recorder-recovery/$(date +%Y%m%d_%H%M%S)/
```

**6. Verification & Resumption**
After recovery, run a full health check suite:
```bash
curl -s http://127.0.0.1:8000/health | jq .
sqlite3 /models/flight-recorder/data/main.db "PRAGMA integrity_check;" | tail -1
df -h /models
free -m
```
If all checks pass, await human approval to resume the remediation run or restart the benchmark.

All recovery commands are logged, timestamped, and require explicit approval where they modify system state. The agent will not proceed until recovery is verified and approved.

# Summary
This remediation plan provides a comprehensive, safety-first approach to restoring reliable benchmark execution on the remote model server at `192.168.1.116`. The operational philosophy centers on non-destructive inspection, rigorous verification, explicit human approval gates, and absolute data preservation. The `/models/flight-recorder/data` directory and its SQLite database are treated as immutable assets; no deletion, truncation, or unauthorized modification is permitted under any circumstances.

The plan is structured into sequential phases: read-only inspection establishes a complete system baseline; health checks ensure continuous validation; data safety checks guarantee backup integrity and corruption detection; the redeploy procedure updates application code with atomic, reversible steps; the benchmark restart procedure implements gradual load introduction and queue management; human approval gates enforce oversight at critical junctures; stop conditions define clear halt criteria; and recovery commands provide safe rollback and restoration paths.

Throughout the run, the agent will maintain detailed audit logs, checksum records, and metric histories. All commands are labeled with approval requirements where they modify system state, and destructive operations are explicitly prohibited. The expected outcome is a stable, healthy service capable of running benchmarks reliably, with full data integrity preserved and complete operational transparency maintained.

This plan is designed to be executed methodically, with verification at every step. The agent will pause for human approval at designated gates, halt immediately if stop conditions are met, and utilize recovery commands to restore stability if failures occur. By adhering to this structured, cautious approach, the remediation run will achieve its goal of restoring reliable benchmark execution without compromising data safety or system integrity.