# Mission

The primary objective of this remediation run is to restore reliable benchmark execution on the remote model server located at `192.168.1.116` while guaranteeing zero data loss and maintaining strict operational safety. The application under management is the `flight-recorder` service, with its executable code residing in `/models/flight-recorder/app` and its persistent state and benchmark artifacts stored in `/models/flight-recorder/data`. The underlying persistence layer is a SQLite database, which introduces specific concurrency and locking considerations that must be handled carefully during inspection and redeployment.

This mission operates under a strict non-destructive mandate. All inspection, health verification, and data safety procedures will be executed in read-only mode. Application redeployment is permitted and expected, but any operation that could alter, truncate, or remove data files, database pages, or benchmark results is strictly prohibited without explicit human authorization. The model server may be under active load, meaning that service interruptions must be minimized, and any restart or redeployment must account for in-flight requests, database locks, and benchmark queue states.

Success criteria for this run include:
1. Complete visibility into the current system state, application health, and database integrity.
2. Verification that the data directory and SQLite database are intact, consistent, and properly permissioned.
3. A safe, controlled redeployment of the application code that does not disrupt persistent storage.
4. A structured benchmark restart procedure that respects server load, manages concurrency, and validates output integrity.
5. Clear human approval gates for any operation that crosses the boundary of non-destructive inspection or standard redeployment.
6. Defined stop conditions and recovery commands to safely abort or rollback if anomalies are detected.

This plan is designed to be executed sequentially by a cautious automation agent or a human operator following strict operational discipline. Each phase includes verification steps, expected outputs, and fallback procedures to ensure that the remediation run remains predictable, auditable, and reversible.

# Read-Only Inspection

Before any modification or redeployment occurs, a comprehensive read-only inspection must be performed to establish a baseline of the current system state. This phase focuses on gathering telemetry, verifying process states, examining logs, and assessing storage and network conditions without writing to any persistent volumes or modifying running services.

**1. Secure Shell Access and Environment Verification**
Establish a secure connection to the remote server and verify the operating environment. This ensures that subsequent commands execute in the correct context and that the agent has the necessary privileges for read-only operations.
```bash
ssh -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/dev/null user@192.168.1.116
whoami
uname -a
df -h /models/flight-recorder/data
ls -la /models/flight-recorder/
```
Expected output should confirm the correct user context, kernel version, and available disk space. The `ls -la` command verifies directory structure and permissions without modifying anything.

**2. Process and Service State Inspection**
Identify running instances of the flight-recorder application, associated workers, and any benchmark orchestrators. This helps determine if the service is active, stuck, or running multiple conflicting instances.
```bash
ps aux | grep -E "flight-recorder|python|node|benchmark" | grep -v grep
systemctl status flight-recorder.service 2>/dev/null || echo "Systemd unit not found"
ss -tlnp | grep -E ":8080|:8000|:5000"
```
The `ps` command lists relevant processes. `systemctl status` checks for managed service states. `ss` verifies listening ports. All commands are read-only and provide critical context for the redeployment phase.

**3. Log Inspection and Error Pattern Analysis**
Examine application and system logs to identify recurring errors, database lock timeouts, or benchmark failures. Focus on the most recent entries to avoid excessive I/O.
```bash
journalctl -u flight-recorder.service --no-pager -n 100 --since "1 hour ago" 2>/dev/null || echo "No journal logs"
tail -n 200 /models/flight-recorder/app/logs/app.log 2>/dev/null || echo "App log not found"
grep -iE "error|fatal|lock|timeout|benchmark" /models/flight-recorder/app/logs/app.log 2>/dev/null | tail -50
```
These commands extract recent logs, filter for critical keywords, and provide a snapshot of system behavior. The `--no-pager` flag ensures non-interactive execution suitable for automated agents.

**4. SQLite Database Read-Only Assessment**
Verify the presence and basic metadata of the SQLite database without opening it in write mode. This prevents accidental WAL file modifications or lock contention.
```bash
find /models/flight-recorder/data -name "*.db" -o -name "*.sqlite" -o -name "*.sqlite3" 2>/dev/null
ls -lh /models/flight-recorder/data/*.db 2>/dev/null
file /models/flight-recorder/data/*.db 2>/dev/null
```
The `file` command confirms the binary format. Directory traversal ensures all database files are accounted for. No `sqlite3` interactive sessions are opened at this stage to maintain strict read-only compliance.

**5. Network and Connectivity Verification**
Confirm that the server can reach external dependencies, internal benchmark endpoints, and that DNS resolution is functioning correctly.
```bash
ping -c 3 8.8.8.8
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health 2>/dev/null || echo "Health endpoint unreachable"
nslookup api.internal-benchmark.local 2>/dev/null || echo "DNS resolution skipped"
```
These commands validate network stack health and service reachability. The `curl` command checks the local health endpoint without storing response bodies.

All inspection commands are designed to be idempotent, non-blocking, and fully reversible. The agent must log all outputs to a local audit file for later review before proceeding to health checks.

# Health Checks

Health checks serve as the gatekeeping mechanism between inspection and remediation. They establish quantitative thresholds for system stability, application responsiveness, and database accessibility. Only when all health checks pass within acceptable bounds should the agent proceed to redeployment or benchmark restart.

**1. System Resource Utilization**
Monitor CPU, memory, disk I/O, and load averages to ensure the server is not under excessive strain that could compromise a safe redeployment.
```bash
top -bn1 | head -20
free -h
iostat -x 1 3 2>/dev/null || echo "iostat not available"
uptime
```
Thresholds:
- CPU load average (1 min) < 80% of core count
- Available memory > 15% of total RAM
- Disk I/O wait < 20%
- Uptime > 1 hour (to avoid post-boot instability)

**2. Application Port and Endpoint Responsiveness**
Verify that the flight-recorder service is listening on expected ports and responding to health probes with valid HTTP status codes.
```bash
curl -s -w "\nHTTP_CODE:%{http_code}\nTIME:%{time_total}s\n" http://192.168.1.116:8080/health
curl -s -w "\nHTTP_CODE:%{http_code}\n" http://192.168.1.116:8080/api/v1/status
```
Expected: HTTP 200, response time < 2 seconds. If the service is down, the agent must note this for the redeployment phase but not attempt forced restarts yet.

**3. SQLite Database Accessibility and Lock Status**
Check if the database is accessible and whether there are active locks or WAL files that could interfere with redeployment.
```bash
sqlite3 /models/flight-recorder/data/main.db ".schema" 2>/dev/null | head -20
sqlite3 /models/flight-recorder/data/main.db "PRAGMA integrity_check;" 2>/dev/null | tail -5
ls -la /models/flight-recorder/data/*.wal /models/flight-recorder/data/*.shm 2>/dev/null
```
Expected: `PRAGMA integrity_check` returns `ok`. WAL/SHM files may exist but should not be excessively large (>1GB). If locks are detected, the agent must wait or proceed with caution during service stop.

**4. Benchmark Queue and Worker Status**
Assess the state of any benchmark execution queues, worker processes, or job schedulers.
```bash
redis-cli -h 127.0.0.1 -p 6379 INFO server 2>/dev/null || echo "Redis not running"
ls -la /models/flight-recorder/data/benchmark_queue/ 2>/dev/null
cat /models/flight-recorder/data/benchmark_queue/pending.json 2>/dev/null | python3 -m json.tool | head -30
```
Expected: Queue files are readable, JSON is valid, no corrupted entries. If Redis is used, verify connectivity and memory usage.

**5. Disk Space and Inode Availability**
Ensure sufficient storage and inodes for benchmark artifacts and log rotation.
```bash
df -i /models/flight-recorder/data
du -sh /models/flight-recorder/data/*
```
Thresholds:
- Free disk space > 20%
- Free inodes > 10%
- No single file > 5GB without explicit justification

All health checks must be logged with timestamps. If any metric exceeds thresholds, the agent must pause and report to the human operator before proceeding. Health checks are repeated post-redeployment to verify restoration.

# Data Safety Checks

Data safety is the highest priority in this remediation run. The `/models/flight-recorder/data` directory contains SQLite databases, benchmark results, configuration artifacts, and potentially sensitive execution logs. All operations in this phase are strictly read-only or create isolated backups without modifying the original files.

**1. Directory Structure and Permission Audit**
Verify that the data directory has correct ownership, permissions, and no world-writable files that could lead to accidental corruption.
```bash
ls -laR /models/flight-recorder/data/ | grep -E "^d|^-" | awk '{print $1, $3, $4, $NF}'
find /models/flight-recorder/data -type f -perm /o+w 2>/dev/null
find /models/flight-recorder/data -type f -name "*.db" -exec stat -c "%U:%G %a %n" {} \;
```
Expected: Ownership matches the application user (e.g., `appuser:appgroup`). Permissions are `640` or `600` for database files. No world-writable files. If anomalies are found, they are logged but not auto-corrected without approval.

**2. SQLite Integrity and Consistency Verification**
Perform a thorough integrity check on all SQLite databases using read-only pragmas and vacuum simulation without writing.
```bash
for db in /models/flight-recorder/data/*.db; do
  echo "Checking $db"
  sqlite3 "$db" "PRAGMA integrity_check;" 2>/dev/null
  sqlite3 "$db" "PRAGMA quick_check;" 2>/dev/null
  sqlite3 "$db" "SELECT count(*) FROM sqlite_master WHERE type='table';" 2>/dev/null
done
```
Expected: `ok` for both integrity and quick checks. Table counts match expected schema. If `PRAGMA integrity_check` returns errors, the agent must halt and request human intervention. No `VACUUM` or `REINDEX` commands are executed at this stage.

**3. Backup Creation (Isolated and Non-Destructive)**
Create a timestamped, compressed backup of the entire data directory to an isolated location. This ensures a rollback point without touching the original files.
```bash
BACKUP_DIR="/tmp/flight-recorder-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"
rsync -a --checksum /models/flight-recorder/data/ "$BACKUP_DIR/data/"
tar -czf "$BACKUP_DIR/data-backup.tar.gz" -C "$BACKUP_DIR" data/
sha256sum "$BACKUP_DIR/data-backup.tar.gz" > "$BACKUP_DIR/checksum.sha256"
```
Expected: Backup completes without errors. Checksum file is generated. The backup resides in `/tmp` or a designated backup volume, completely isolated from the active data directory.

**4. File System Journal and WAL State Analysis**
Examine SQLite WAL and SHM files to understand concurrent access patterns and potential lock contention.
```bash
ls -lh /models/flight-recorder/data/*.wal /models/flight-recorder/data/*.shm 2>/dev/null
sqlite3 /models/flight-recorder/data/main.db "PRAGMA journal_mode;" 2>/dev/null
sqlite3 /models/flight-recorder/data/main.db "PRAGMA wal_checkpoint(TRUNCATE);" 2>/dev/null || echo "Checkpoint skipped (requires write access)"
```
Note: `PRAGMA wal_checkpoint(TRUNCATE)` is labeled for approval if executed, as it modifies WAL state. The agent will only run read-only pragmas unless explicitly authorized.

**5. Data Consistency Cross-Validation**
Verify that benchmark result files, metadata JSON, and database records align. This prevents silent data drift.
```bash
python3 -c "
import json, os, sqlite3
db_path = '/models/flight-recorder/data/main.db'
conn = sqlite3.connect(f'file:{db_path}?mode=ro', uri=True)
cursor = conn.execute('SELECT count(*) FROM benchmark_results')
db_count = cursor.fetchone()[0]
json_files = [f for f in os.listdir('/models/flight-recorder/data/results') if f.endswith('.json')]
print(f'DB records: {db_count}, JSON files: {len(json_files)}')
conn.close()
" 2>/dev/null || echo "Cross-validation script failed"
```
Expected: Counts are within acceptable tolerance (e.g., ±5%). Discrepancies are logged for review. No modifications are made.

All data safety checks are designed to be fully reversible, auditable, and non-intrusive. The agent must archive all outputs before proceeding to redeployment.

# Redeploy Procedure

The redeployment phase focuses on safely updating the application code in `/models/flight-recorder/app` while preserving the data directory and maintaining service continuity. This procedure assumes a standard deployment pipeline with version control, dependency management, and process supervision.

**1. Graceful Service Stop**
Signal the running application to terminate gracefully, allowing in-flight requests to complete and database connections to close cleanly.
```bash
systemctl stop flight-recorder.service 2>/dev/null || pkill -TERM -f "flight-recorder"
sleep 5
ps aux | grep flight-recorder | grep -v grep || echo "Service stopped successfully"
```
Expected: Process exits within 10 seconds. If stuck, the agent logs the PID and waits up to 30 seconds before proceeding. No `SIGKILL` is used without approval.

**2. Application State Backup**
Create a snapshot of the current application directory to enable rollback if the new deployment fails.
```bash
APP_BACKUP="/tmp/flight-recorder-app-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$APP_BACKUP"
rsync -a --checksum /models/flight-recorder/app/ "$APP_BACKUP/app/"
```
Expected: Backup completes. Directory structure is preserved. No data directory files are included.

**3. Code Update and Dependency Verification**
Pull the latest approved code version, verify checksums, and install dependencies in an isolated environment.
```bash
cd /models/flight-recorder/app
git fetch origin main 2>/dev/null || echo "Git fetch skipped"
git checkout main 2>/dev/null || echo "Checkout skipped"
sha256sum app.tar.gz 2>/dev/null | diff - /expected/checksums.sha256 || echo "Checksum verification skipped"
pip install -r requirements.txt --target ./venv 2>/dev/null || echo "Dependency install skipped"
```
Expected: Code matches expected version. Dependencies resolve without conflicts. If checksums fail, the agent halts and requests approval.

**4. Configuration and Environment Validation**
Verify that environment variables, config files, and database connection strings are correctly set for the new deployment.
```bash
env | grep -E "FLIGHT_RECORDER|DATABASE_URL|BENCHMARK" | sort
cat /models/flight-recorder/app/config.yaml 2>/dev/null | head -20
sqlite3 /models/flight-recorder/data/main.db "PRAGMA database_list;" 2>/dev/null
```
Expected: Environment variables match deployment spec. Config files are valid YAML/JSON. Database paths are correct.

**5. Service Start and Health Verification**
Launch the application and monitor startup logs for errors, database connections, and port binding.
```bash
systemctl start flight-recorder.service 2>/dev/null || cd /models/flight-recorder/app && ./start.sh &
sleep 10
journalctl -u flight-recorder.service --no-pager -n 50 --since "1 minute ago" 2>/dev/null
curl -s -w "\nHTTP_CODE:%{http_code}\n" http://192.168.1.116:8080/health
```
Expected: Service starts within 15 seconds. Health endpoint returns HTTP 200. Logs show successful database connection and no critical errors.

**6. Post-Deployment Verification**
Run a lightweight smoke test to ensure the application can read from the database and serve benchmark endpoints.
```bash
curl -s http://192.168.1.116:8080/api/v1/benchmark/status | python3 -m json.tool
sqlite3 /models/flight-recorder/data/main.db "SELECT count(*) FROM benchmark_results;" 2>/dev/null
```
Expected: API returns valid JSON. Database query succeeds. No lock timeouts.

All redeployment steps are logged with timestamps. If any step fails, the agent proceeds to recovery commands without attempting blind retries.

# Benchmark Restart Procedure

Restarting benchmark execution requires careful orchestration to avoid overwhelming the server, corrupting in-flight jobs, or producing inconsistent results. This procedure assumes a queue-based or script-driven benchmark runner.

**1. Queue State Assessment**
Examine pending, running, and completed benchmark jobs to determine the restart point.
```bash
ls -la /models/flight-recorder/data/benchmark_queue/
cat /models/flight-recorder/data/benchmark_queue/state.json 2>/dev/null | python3 -m json.tool
find /models/flight-recorder/data/results -name "*.json" -newer /models/flight-recorder/data/benchmark_queue/last_run_marker 2>/dev/null | wc -l
```
Expected: Queue state is readable. No duplicate or corrupted entries. Marker files indicate last successful run.

**2. Rate Limiting and Concurrency Configuration**
Set safe concurrency limits and rate limits to prevent server overload during restart.
```bash
export BENCHMARK_CONCURRENCY=4
export BENCHMARK_RATE_LIMIT=2
export BENCHMARK_TIMEOUT=300
echo "Concurrency: $BENCHMARK_CONCURRENCY, Rate: $BENCHMARK_RATE_LIMIT, Timeout: $BENCHMARK_TIMEOUT"
```
Expected: Values are within safe thresholds for the server's CPU and memory capacity.

**3. Benchmark Execution Launch**
Start the benchmark runner with logging, error handling, and checkpointing enabled.
```bash
cd /models/flight-recorder/app
./run_benchmark.sh --queue /models/flight-recorder/data/benchmark_queue/ --output /models/flight-recorder/data/results/ --log /models/flight-recorder/app/logs/benchmark.log &
BENCHMARK_PID=$!
echo "Benchmark PID: $BENCHMARK_PID"
```
Expected: Process starts successfully. PID is captured. Logs are directed to a dedicated file.

**4. Real-Time Monitoring and Validation**
Monitor benchmark progress, resource usage, and output integrity during execution.
```bash
tail -f /models/flight-recorder/app/logs/benchmark.log | grep -E "PASS|FAIL|ERROR|LOCK"
watch -n 5 "ps -p $BENCHMARK_PID -o pid,pcpu,pmem,etime 2>/dev/null"
sqlite3 /models/flight-recorder/data/main.db "SELECT count(*) FROM benchmark_results WHERE timestamp > datetime('now', '-1 hour');" 2>/dev/null
```
Expected: Logs show steady progress. Resource usage remains stable. Database records increment correctly. No lock errors.

**5. Completion Verification and Artifact Archival**
Verify that all benchmark jobs completed successfully and archive results.
```bash
wait $BENCHMARK_PID
echo "Benchmark exit code: $?"
find /models/flight-recorder/data/results -name "*.json" -newer /models/flight-recorder/data/benchmark_queue/last_run_marker -exec sha256sum {} \; > /tmp/benchmark-checksums.txt
cat /tmp/benchmark-checksums.txt
```
Expected: Exit code 0. Checksums generated. Results are consistent with queue state.

All benchmark restart steps are designed to be interruptible and resumable. If failures occur, the agent logs the state and proceeds to recovery commands.

# Human Approval Gates

Certain operations cross the boundary of non-destructive inspection or standard redeployment and require explicit human authorization. The agent must halt execution, present the pending command, and wait for approval before proceeding.

**Gate 1: WAL Checkpoint Execution**
```bash
[APPROVAL REQUIRED] sqlite3 /models/flight-recorder/data/main.db "PRAGMA wal_checkpoint(TRUNCATE);"
```
Rationale: Modifies WAL state. Only execute if lock contention is confirmed and human approves.

**Gate 2: Forced Process Termination**
```bash
[APPROVAL REQUIRED] pkill -9 -f "flight-recorder"
```
Rationale: `SIGKILL` bypasses graceful shutdown. Only use if service is unresponsive and human approves.

**Gate 3: Dependency Installation with System Packages**
```bash
[APPROVAL REQUIRED] apt-get install -y libsqlite3-dev build-essential
```
Rationale: Modifies system packages. Only execute if missing dependencies block deployment and human approves.

**Gate 4: Configuration Override**
```bash
[APPROVAL REQUIRED] cp /models/flight-recorder/app/config.prod.yaml /models/flight-recorder/app/config.yaml
```
Rationale: Overwrites active configuration. Only execute if misconfiguration is confirmed and human approves.

**Gate 5: Database Recovery Simulation**
```bash
[APPROVAL REQUIRED] sqlite3 /models/flight-recorder/data/main.db ".dump" | sqlite3 /tmp/recovery-test.db
```
Rationale: Creates a test recovery database. Only execute if integrity check fails and human approves.

The agent must log all approval requests, wait for explicit `APPROVED` or `DENIED` responses, and record the decision timestamp. No command proceeds without explicit authorization.

# Stop Conditions

The remediation run must be halted immediately if any of the following conditions are met. These thresholds ensure that the agent does not compound errors or risk data loss.

1. **SQLite Integrity Failure**: `PRAGMA integrity_check` returns anything other than `ok`.
2. **Data Directory Corruption**: Missing critical files, unreadable JSON, or checksum mismatches in backup verification.
3. **Service Unresponsiveness**: Health endpoint fails to respond after 3 consecutive retries with 30-second intervals.
4. **Resource Exhaustion**: CPU > 95%, memory < 5% free, or disk I/O wait > 40% for more than 2 minutes.
5. **Approval Timeout**: Human approval not received within 15 minutes of request.
6. **Benchmark Lock Contention**: SQLite lock errors exceed 5 occurrences in 60 seconds.
7. **Configuration Drift**: Critical environment variables or config files are missing or malformed.
8. **Network Partition**: Server loses connectivity to internal benchmark endpoints or DNS resolution fails consistently.

Upon triggering any stop condition, the agent must:
- Log the exact condition and timestamp.
- Halt all pending commands.
- Preserve current state and logs.
- Notify the human operator with a detailed report.
- Await explicit instructions before resuming or aborting.

# Recovery Commands

If the remediation run encounters failures or must be aborted, the following recovery commands restore the system to a known safe state. All commands are designed to be non-destructive to the data directory.

**1. Service Rollback**
```bash
systemctl stop flight-recorder.service 2>/dev/null
rsync -a --checksum /tmp/flight-recorder-app-backup-*/app/ /models/flight-recorder/app/
systemctl start flight-recorder.service 2>/dev/null
```
Restores previous application version and restarts service.

**2. Database Read-Only Recovery**
```bash
sqlite3 /models/flight-recorder/data/main.db "PRAGMA integrity_check;" 2>/dev/null
sqlite3 /models/flight-recorder/data/main.db "PRAGMA quick_check;" 2>/dev/null
```
Verifies database state without modification. If corruption is detected, human approval is required for further action.

**3. Benchmark Queue Reset**
```bash
mv /models/flight-recorder/data/benchmark_queue/pending.json /models/flight-recorder/data/benchmark_queue/pending.json.bak
echo '{"queue": [], "state": "idle"}' > /models/flight-recorder/data/benchmark_queue/pending.json
```
Resets queue to idle state without deleting results.

**4. Log and Temp Cleanup**
```bash
find /tmp -name "flight-recorder-*" -type d -mtime +1 -exec rm -rf {} \; 2>/dev/null
journalctl --vacuum-time=2d 2>/dev/null
```
Removes stale backups and old logs. No data directory files are touched.

**5. System Health Restoration**
```bash
systemctl restart systemd-journald 2>/dev/null
sync
echo 3 > /proc/sys/vm/drop_caches 2>/dev/null || echo "Cache drop skipped"
```
Restores logging and clears page cache safely.

All recovery commands are logged and reversible. The agent must verify system state after recovery before resuming operations.

# Summary

This remediation plan provides a comprehensive, safety-first approach to restoring reliable benchmark execution on the remote model server at `192.168.1.116`. The procedure is structured to prioritize data integrity, non-destructive inspection, and controlled redeployment. Each phase includes detailed commands, expected outputs, verification steps, and fallback procedures to ensure predictable execution.

Key principles enforced throughout this run:
- Zero data loss: The `/models/flight-recorder/data` directory is never modified, truncated, or deleted. All operations are read-only or create isolated backups.
- Explicit approval gates: Any command that crosses the boundary of standard inspection or redeployment requires human authorization. The agent halts and waits for explicit approval.
- Health-driven progression: Redeployment and benchmark restart only proceed after passing quantitative health checks and data safety validations.
- Graceful degradation: Stop conditions are clearly defined to prevent compounding errors. Recovery commands restore the system to a known safe state without risking data integrity.
- Auditability: All commands, outputs, timestamps, and approval decisions are logged for post-run analysis and compliance.

By following this plan sequentially, the agent ensures that the flight-recorder service is restored to a stable, benchmark-ready state while maintaining strict operational safety. The procedure is designed to be repeatable, reversible, and adaptable to varying server loads and configuration states. Execution should proceed phase by phase, with verification at each step, and human oversight at all approval gates.