# Mission
The primary objective of this remediation run is to restore reliable, repeatable benchmark execution on the remote model server located at `192.168.1.116` while guaranteeing absolute data preservation. The operational scope encompasses the application codebase residing at `/models/flight-recorder/app`, the persistent data store at `/models/flight-recorder/data`, and the underlying SQLite database that tracks benchmark state, model weights, telemetry, and execution logs. Given that the model server may be under active load or experiencing transient contention, all actions must be non-disruptive, reversible, and strictly bounded by safety guardrails. Success is defined as a stable service state, passing health and integrity checks, a successfully redeployed application layer (if required), and a benchmark harness that completes its execution cycle without data loss, corruption, or unhandled exceptions. The mission explicitly prohibits any destructive operations against the data directory; instead, it relies on defensive copying, atomic backups, and state verification. The plan is structured to proceed incrementally, validating each phase before advancing, and incorporates explicit human approval gates for any operation that crosses into service interruption or state mutation.

# Read-Only Inspection
Before initiating any remediation, the agent must establish a comprehensive baseline of the server's current state using strictly non-destructive commands. This phase ensures that subsequent actions are informed by accurate telemetry and that no hidden corruption or misconfiguration is overlooked.

1. **Secure Remote Access & Session Hardening**
   Establish an SSH session with verbose logging and connection multiplexing to prevent accidental disconnections during long-running checks.
   ```bash
   ssh -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ControlMaster=yes -o ControlPath=/tmp/ssh-%r@%h:%p user@192.168.1.116
   ```
   Verify the remote environment, Python/runtime versions, and available disk space without modifying any files.
   ```bash
   uname -a
   python3 --version
   df -h /models/flight-recorder
   ```

2. **Filesystem & Codebase Audit**
   Inspect the application directory structure, file permissions, and recent modification timestamps to identify stale artifacts or permission drift.
   ```bash
   ls -laR /models/flight-recorder/app | head -100
   find /models/flight-recorder/app -type f -name "*.py" -o -name "*.js" -o -name "*.yaml" | wc -l
   stat /models/flight-recorder/app/main.py
   ```
   Cross-reference the deployed code against the expected repository state using read-only git commands (if version control is present).
   ```bash
   cd /models/flight-recorder/app && git status --porcelain 2>/dev/null || echo "No git repo detected"
   git log --oneline -5 2>/dev/null
   ```

3. **Process & Service Enumeration**
   Identify all running processes related to the model server, web framework, or background workers. Capture PIDs, memory footprints, and CPU utilization.
   ```bash
   ps aux | grep -E "python|node|gunicorn|uvicorn|celery|model-server" | grep -v grep
   systemctl list-units --type=service --state=running | grep -i flight
   ```
   Inspect systemd or supervisor configurations to understand restart policies, resource limits, and logging targets.
   ```bash
   systemctl show flight-recorder.service 2>/dev/null | grep -E "Restart=|MemoryLimit=|ExecStart="
   cat /etc/supervisor/conf.d/flight-recorder.conf 2>/dev/null
   ```

4. **Log & Telemetry Review**
   Extract recent logs to identify error patterns, timeout events, or database lock contention. Use pagination to avoid memory exhaustion.
   ```bash
   journalctl -u flight-recorder --no-pager -n 200 --since "1 hour ago"
   tail -n 100 /var/log/flight-recorder/app.log 2>/dev/null || echo "Log path not found"
   grep -iE "error|warning|timeout|deadlock|sqlite" /var/log/flight-recorder/app.log 2>/dev/null | tail -50
   ```

5. **Network & Port Verification**
   Confirm that the service is listening on expected ports and that network paths to the benchmark harness are stable.
   ```bash
   ss -tlnp | grep -E "8080|8000|9090|5000"
   curl -s -o /dev/null -w "HTTP Status: %{http_code}\nResponse Time: %{time_total}s\n" http://localhost:8080/health 2>/dev/null || echo "Health endpoint unreachable"
   ping -c 5 192.168.1.116
   ```

All commands in this phase are strictly read-only. No files are modified, no processes are signaled, and no network state is altered. The output is parsed to build a diagnostic profile that informs the health check and data safety phases.

# Health Checks
Health checks validate that the server, application, and database are operating within acceptable parameters before any remediation or redeployment is attempted. Each check includes pass/fail thresholds and fallback verification steps.

1. **Service Availability & Endpoint Validation**
   Verify that the model server responds to health probes and returns expected status codes.
   ```bash
   curl -f http://localhost:8080/health || echo "FAIL: Health endpoint unreachable"
   curl -s http://localhost:8080/metrics | grep -E "up|status|ready" || echo "WARN: Metrics endpoint malformed"
   ```
   If the service is managed by systemd, verify its active state and recent restart history.
   ```bash
   systemctl is-active flight-recorder.service
   systemctl status flight-recorder.service --no-pager -l | head -20
   ```

2. **SQLite Database Integrity**
   Run built-in SQLite integrity checks to detect page corruption, index mismatches, or journal inconsistencies. This is critical before any benchmark state mutation.
   ```bash
   sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
   sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA quick_check;"
   sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA journal_mode;"
   sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA wal_checkpoint(PASSIVE);"
   ```
   Expected output: `ok` for integrity checks. Any deviation triggers an immediate pause and escalation to the data safety phase.

3. **Resource Utilization Thresholds**
   Monitor CPU, memory, disk I/O, and swap usage to ensure the server can handle benchmark loads without thrashing.
   ```bash
   vmstat 1 5
   free -m
   iostat -x 1 3
   ```
   Pass criteria: CPU idle > 15%, available RAM > 20%, disk utilization < 85%, swap usage < 10%. If thresholds are breached, the agent must wait for resource reclamation or request human intervention.

4. **Dependency & Runtime Verification**
   Confirm that required Python packages, system libraries, and GPU drivers (if applicable) are present and compatible.
   ```bash
   pip list --format=columns | grep -E "torch|tensorflow|numpy|pandas|fastapi|uvicorn"
   ldconfig -p | grep -E "libsqlite|libssl|libffi"
   nvidia-smi 2>/dev/null || echo "GPU not detected or driver unavailable"
   ```

5. **Network Latency & Packet Loss**
   Ensure stable connectivity between the model server and the benchmark orchestrator.
   ```bash
   mtr -r -c 10 192.168.1.116 2>/dev/null || ping -c 10 192.168.1.116
   ```
   Pass criteria: latency < 50ms, packet loss = 0%. Intermittent drops trigger a network retry policy before proceeding.

All health checks are non-intrusive. If any check fails, the agent halts, logs the failure, and requests human approval before attempting corrective actions.

# Data Safety Checks
Data preservation is the highest priority. This phase establishes defensive copies, verifies filesystem consistency, and ensures that the SQLite database can be safely restored if corruption is detected. No files are moved, truncated, or deleted.

1. **Atomic Directory Backup**
   Create a timestamped, read-only backup of the entire data directory using `rsync` with hardlink preservation and checksum verification.
   ```bash
   BACKUP_DIR="/tmp/flight-recorder-data-backup-$(date +%Y%m%d_%H%M%S)"
   mkdir -p "$BACKUP_DIR"
   rsync -av --checksum --no-compress /models/flight-recorder/data/ "$BACKUP_DIR/"
   chmod -R a-w "$BACKUP_DIR"
   ```
   Verify backup completeness by comparing file counts and total size.
   ```bash
   echo "Source files: $(find /models/flight-recorder/data -type f | wc -l)"
   echo "Backup files: $(find "$BACKUP_DIR" -type f | wc -l)"
   du -sh /models/flight-recorder/data "$BACKUP_DIR"
   ```

2. **SQLite Database Backup & Verification**
   Use SQLite's built-in `.backup` command to create a consistent snapshot, independent of WAL or journal states.
   ```bash
   sqlite3 /models/flight-recorder/data/benchmark.db ".backup '/tmp/benchmark.db.bak'"
   sqlite3 /tmp/benchmark.db.bak "PRAGMA integrity_check;"
   sha256sum /models/flight-recorder/data/benchmark.db /tmp/benchmark.db.bak
   ```
   If the backup integrity check returns `ok`, the snapshot is considered safe for recovery. The original database remains untouched.

3. **Filesystem Consistency & Permission Audit**
   Verify that file permissions, ownership, and extended attributes match expected baselines. Misconfigured permissions can cause silent benchmark failures.
   ```bash
   find /models/flight-recorder/data -type f -perm /o+w -exec ls -la {} \;
   getfacl /models/flight-recorder/data/benchmark.db 2>/dev/null
   lsattr /models/flight-recorder/data/ 2>/dev/null
   ```
   Ensure that the application user has read/write access to the data directory but that world-writable flags are absent.

4. **Checksum Manifest Generation**
   Generate a cryptographic manifest of critical data files to detect silent corruption during benchmark execution.
   ```bash
   find /models/flight-recorder/data -type f -name "*.db" -o -name "*.json" -o -name "*.parquet" | xargs sha256sum > /tmp/data-manifest-$(date +%Y%m%d_%H%M%S).sha256
   ```
   This manifest is stored in `/tmp` and referenced during post-benchmark validation.

5. **Snapshot/Clone Fallback (Optional)**
   If the filesystem supports LVM or Btrfs, create a read-only snapshot for instant rollback capability.
   ```bash
   # Example for Btrfs (if applicable)
   btrfs subvolume snapshot -r /models/flight-recorder/data /tmp/data-snapshot-$(date +%Y%m%d_%H%M%S) 2>/dev/null || echo "Snapshot not supported or skipped"
   ```
   All backup operations are strictly additive. The original data directory is never modified, truncated, or deleted.

# Redeploy Procedure
If health checks or inspection reveal application-level degradation, a controlled redeploy is initiated. The procedure emphasizes graceful transitions, configuration validation, and atomic replacement.

1. **Pre-Deploy Validation**
   Verify that the new codebase passes syntax checks, dependency resolution, and configuration linting before touching the running service.
   ```bash
   cd /models/flight-recorder/app
   python3 -m py_compile main.py 2>/dev/null && echo "Syntax OK" || echo "FAIL: Syntax error"
   pip install -r requirements.txt --dry-run --no-cache-dir
   python3 -c "import yaml; yaml.safe_load(open('config.yaml'))" 2>/dev/null && echo "Config OK" || echo "FAIL: Config malformed"
   ```

2. **Graceful Service Drain**
   Signal the running service to stop accepting new requests and complete in-flight benchmark jobs. This prevents data inconsistency.
   ```bash
   [APPROVAL REQUIRED] systemctl stop flight-recorder.service
   [APPROVAL REQUIRED] kill -SIGTERM $(pgrep -f "flight-recorder") 2>/dev/null
   sleep 10
   ps aux | grep flight-recorder | grep -v grep || echo "Service drained successfully"
   ```
   The `[APPROVAL REQUIRED]` tag indicates that human confirmation is mandatory before interrupting service continuity.

3. **Atomic Code Replacement**
   Deploy the updated application layer using a staging directory to avoid partial writes.
   ```bash
   STAGING_DIR="/tmp/flight-recorder-app-staging-$(date +%Y%m%d_%H%M%S)"
   mkdir -p "$STAGING_DIR"
   rsync -av --checksum /models/flight-recorder/app/ "$STAGING_DIR/"
   diff -rq /models/flight-recorder/app/ "$STAGING_DIR/" || echo "Differences detected (expected)"
   [APPROVAL REQUIRED] mv /models/flight-recorder/app /models/flight-recorder/app.bak
   [APPROVAL REQUIRED] mv "$STAGING_DIR" /models/flight-recorder/app
   ```
   The original directory is preserved as `.bak` for instant rollback. No files are deleted.

4. **Environment & Cache Warming**
   Activate the virtual environment, install dependencies, and pre-warm model caches if applicable.
   ```bash
   source /models/flight-recorder/venv/bin/activate
   pip install -r /models/flight-recorder/app/requirements.txt --no-cache-dir
   python3 /models/flight-recorder/app/scripts/warm_cache.py 2>/dev/null || echo "Cache warming skipped"
   ```

5. **Service Restart & Verification**
   Start the service and monitor startup logs for errors.
   ```bash
   [APPROVAL REQUIRED] systemctl start flight-recorder.service
   journalctl -u flight-recorder --no-pager -f &
   sleep 15
   curl -f http://localhost:8080/health && echo "Service healthy" || echo "FAIL: Service unhealthy"
   ```
   Post-deploy smoke tests validate that benchmark endpoints respond correctly and that database connections are established.

# Benchmark Restart Procedure
Once the service is stable and data is secured, the benchmark harness is restarted with controlled concurrency, monitoring, and failure isolation.

1. **Queue & State Reset**
   Clear pending benchmark jobs safely without corrupting completed results.
   ```bash
   sqlite3 /models/flight-recorder/data/benchmark.db "UPDATE jobs SET status='reset' WHERE status='pending' AND created_at < datetime('now', '-1 hour');"
   sqlite3 /models/flight-recorder/data/benchmark.db "SELECT COUNT(*) FROM jobs WHERE status='reset';"
   ```
   Only stale or timed-out jobs are reset. Active or completed jobs remain untouched.

2. **Load Calibration & Ramp-Up**
   Begin with minimal concurrency to verify stability, then gradually increase load.
   ```bash
   # Phase 1: Single-threaded validation
   python3 /models/flight-recorder/app/benchmark_runner.py --concurrency 1 --dry-run
   # Phase 2: Low concurrency
   python3 /models/flight-recorder/app/benchmark_runner.py --concurrency 4 --timeout 300
   # Phase 3: Target concurrency
   python3 /models/flight-recorder/app/benchmark_runner.py --concurrency 16 --timeout 600
   ```
   Each phase includes automatic pause/resume logic if error rates exceed 5%.

3. **Real-Time Monitoring & Telemetry**
   Track resource utilization, request latency, and database lock contention during execution.
   ```bash
   watch -n 2 "ps aux | grep flight-recorder | awk '{print \$3, \$4, \$11}'"
   sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA wal_checkpoint(PASSIVE);"
   ```
   If CPU exceeds 85% or SQLite returns `database is locked`, the agent automatically throttles concurrency.

4. **Failure Isolation & Retry Logic**
   Implement circuit breakers for transient failures. Failed jobs are retried up to 3 times with exponential backoff.
   ```bash
   python3 /models/flight-recorder/app/benchmark_runner.py --retry 3 --backoff 2 --max-failures 10
   ```
   Persistent failures trigger a pause and log archival for human review.

5. **Result Validation & Checksum Verification**
   Compare benchmark outputs against the pre-run manifest to ensure data consistency.
   ```bash
   sha256sum -c /tmp/data-manifest-*.sha256
   python3 /models/flight-recorder/app/validate_results.py --strict
   ```
   Only after successful validation is the benchmark considered complete.

# Human Approval Gates
To prevent unintended service disruption or data mutation, explicit human approval is required at critical junctures. The agent will pause execution and await confirmation before proceeding.

1. **Service Interruption Gate**
   Before stopping or draining the model server, the agent requests approval to ensure no active benchmark jobs are at risk.
   ```bash
   [APPROVAL REQUIRED] systemctl stop flight-recorder.service
   ```
   Approval is granted via interactive prompt, ticket system, or explicit confirmation string.

2. **Code Replacement Gate**
   Before moving the application directory, the agent verifies backup integrity and requests approval for atomic replacement.
   ```bash
   [APPROVAL REQUIRED] mv /models/flight-recorder/app /models/flight-recorder/app.bak
   [APPROVAL REQUIRED] mv "$STAGING_DIR" /models/flight-recorder/app
   ```

3. **Database State Mutation Gate**
   Any SQL command that modifies job status or clears queues requires approval to prevent accidental data loss.
   ```bash
   [APPROVAL REQUIRED] sqlite3 /models/flight-recorder/data/benchmark.db "UPDATE jobs SET status='reset' WHERE status='pending';"
   ```

4. **Benchmark Restart Gate**
   Before initiating the full benchmark run, the agent confirms that health checks, backups, and redeployment are complete.
   ```bash
   [APPROVAL REQUIRED] python3 /models/flight-recorder/app/benchmark_runner.py --concurrency 16 --timeout 600
   ```

5. **Escalation & Rollback Gate**
   If any phase fails stop conditions, the agent pauses and requests human direction before executing recovery commands.
   ```bash
   [APPROVAL REQUIRED] bash /models/flight-recorder/scripts/rollback.sh
   ```

All approval gates are logged with timestamps, operator IDs, and decision rationale. The agent never proceeds past a gate without explicit confirmation.

# Stop Conditions
The remediation run will automatically halt if any of the following conditions are met. These thresholds protect data integrity and prevent cascading failures.

1. **Resource Exhaustion**
   - CPU utilization > 90% sustained for 5 minutes
   - Available RAM < 15% or swap usage > 20%
   - Disk utilization > 90% on `/models/flight-recorder` or `/tmp`
   - Action: Throttle concurrency, pause benchmark, request human review.

2. **Data Corruption Indicators**
   - SQLite `PRAGMA integrity_check` returns anything other than `ok`
   - Checksum mismatch between source data and backup
   - File permission drift detected on critical database files
   - Action: Halt immediately, preserve state, trigger recovery protocol.

3. **Network & Connectivity Failure**
   - > 3 consecutive health endpoint timeouts
   - Packet loss > 5% to benchmark orchestrator
   - SSH session drops without graceful disconnect
   - Action: Pause execution, archive logs, wait for network stabilization.

4. **Service Instability**
   - > 3 service restarts within 10 minutes
   - Segfaults, OOM kills, or unhandled exceptions in logs
   - Database lock contention > 30 seconds
   - Action: Drain requests, snapshot state, request human intervention.

5. **Timeout Policies**
   - Maximum wait time for service drain: 120 seconds
   - Maximum wait time for health check recovery: 300 seconds
   - Maximum benchmark execution time per phase: 1800 seconds
   - Action: Automatic pause, log archival, escalation to approval gate.

All stop conditions are monitored continuously. If triggered, the agent preserves the current state, archives telemetry, and awaits human direction before proceeding.

# Recovery Commands
If remediation fails or data/service integrity is compromised, the following recovery procedures restore the system to a known-good state. All commands are non-destructive and preserve the original data directory.

1. **Application Rollback**
   Restore the previous codebase from the `.bak` directory.
   ```bash
   [APPROVAL REQUIRED] mv /models/flight-recorder/app /models/flight-recorder/app.failed
   [APPROVAL REQUIRED] mv /models/flight-recorder/app.bak /models/flight-recorder/app
   systemctl restart flight-recorder.service
   ```

2. **Database Recovery**
   Restore the SQLite database from the verified backup snapshot.
   ```bash
   [APPROVAL REQUIRED] cp /tmp/benchmark.db.bak /models/flight-recorder/data/benchmark.db
   sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
   systemctl restart flight-recorder.service
   ```
   If corruption is severe, use SQLite's `.recover` command to extract intact pages.
   ```bash
   sqlite3 /models/flight-recorder/data/benchmark.db ".recover" > /tmp/recovered.sql
   sqlite3 /tmp/benchmark.db.recovered < /tmp/recovered.sql
   ```

3. **Service Recovery & Lock Clearing**
   Clear stale PID files, reset systemd state, and restart cleanly.
   ```bash
   rm -f /run/flight-recorder.pid
   systemctl reset-failed flight-recorder.service
   [APPROVAL REQUIRED] systemctl start flight-recorder.service
   ```

4. **Log Archival & Disk Reclamation**
   Compress and move logs to prevent disk exhaustion without deleting active data.
   ```bash
   tar -czf /tmp/logs-$(date +%Y%m%d_%H%M%S).tar.gz /var/log/flight-recorder/
   find /var/log/flight-recorder/ -type f -name "*.log" -exec truncate -s 0 {} \;
   ```

5. **Emergency Shutdown**
   Gracefully drain all requests, snapshot state, and disconnect safely.
   ```bash
   [APPROVAL REQUIRED] systemctl stop flight-recorder.service
   rsync -av /models/flight-recorder/data/ /tmp/emergency-snapshot-$(date +%Y%m%d_%H%M%S)/
   sync
   ```
   All recovery commands are labeled `[APPROVAL REQUIRED]` where they modify running state or replace files. The data directory is never deleted; backups and snapshots ensure full reversibility.

# Summary
This remediation plan provides a methodical, safety-first approach to restoring reliable benchmark execution on the remote model server at `192.168.1.116`. The strategy prioritizes data preservation above all else, explicitly prohibiting any deletion or truncation of the `/models/flight-recorder/data` directory. Instead, it relies on atomic backups, SQLite integrity verification, cryptographic manifests, and read-only inspection to establish a secure baseline before any intervention. Health checks validate service availability, resource utilization, database consistency, and network stability, ensuring that the server can safely handle benchmark loads. If application-level degradation is detected, a controlled redeploy procedure gracefully drains active jobs, validates new code, performs atomic replacement, and verifies post-deploy health. The benchmark restart phase introduces controlled concurrency, real-time monitoring, failure isolation, and result validation to guarantee repeatable execution. Human approval gates are embedded at every critical juncture, preventing unintended service interruption or state mutation. Stop conditions automatically halt operations if resource exhaustion, data corruption, network failure, or service instability is detected, preserving system state for safe recovery. Recovery commands provide reversible rollback paths for the application, database, and service layer, ensuring that any failure can be mitigated without data loss. By adhering to this structured, cautious workflow, the agent ensures that benchmark reliability is restored while maintaining absolute data integrity, operational transparency, and human oversight. The plan is designed to be executed incrementally, with each phase validated before advancing, guaranteeing a stable, reproducible benchmark environment.