# Mission

The primary objective of this remediation run is to restore reliable, repeatable benchmark execution on the remote model server at `192.168.1.116` while guaranteeing zero data loss. The environment consists of an application stack located at `/models/flight-recorder/app` and persistent state stored in `/models/flight-recorder/data`, utilizing a SQLite database for runtime and benchmark metadata. The server may currently be under load, which necessitates a cautious, non-blocking approach to inspection and remediation. 

This plan is structured around a phased, defensive methodology:
1. **Non-destructive state acquisition** to understand current resource utilization, process states, and database integrity.
2. **Health validation** to identify bottlenecks, stale locks, or configuration drift.
3. **Data preservation** through verified SQLite backups and filesystem snapshots.
4. **Controlled redeployment** of the application code with graceful service transitions.
5. **Safe benchmark restoration** with dry-run validation and incremental execution.
6. **Explicit human oversight** at critical decision points, with clear stop conditions and rollback pathways.

All operations will prioritize read-only access, graceful process management, and SQLite best practices. Destructive actions are strictly prohibited unless explicitly authorized by the human operator. The plan assumes standard Linux server tooling, SSH access, and Python-based benchmark runners, but remains adaptable to containerized or systemd-managed deployments.

---

# Read-Only Inspection

Before any modification, a comprehensive read-only inspection must be conducted to map the current state of the server, application, and database. This phase avoids blocking I/O, prevents lock contention, and gathers diagnostic data without altering runtime behavior.

### 1. Secure Session Establishment
Establish a persistent, resilient SSH session to handle potential network interruptions during long-running inspections.
```bash
ssh -o ServerAliveInterval=30 -o ServerAliveCountMax=5 user@192.168.1.116
```
Once connected, create a dedicated inspection directory to store diagnostic outputs without polluting the working environment:
```bash
mkdir -p /tmp/flight-recorder-inspection
```

### 2. Resource & Load Assessment
Since the server may be busy, we must identify high-CPU, high-I/O, or long-running processes that could interfere with benchmark execution or redeployment.
```bash
# Capture top resource consumers (non-blocking)
top -b -n 1 -o %CPU > /tmp/flight-recorder-inspection/cpu_load.txt
top -b -n 1 -o %MEM > /tmp/flight-recorder-inspection/mem_load.txt

# Check disk utilization and I/O wait
df -h /models/flight-recorder/data > /tmp/flight-recorder-inspection/disk_usage.txt
iostat -x 1 3 > /tmp/flight-recorder-inspection/io_stats.txt

# Identify long-running or blocked processes
ps aux --sort=-%cpu | head -20 > /tmp/flight-recorder-inspection/process_list.txt
```
If the server is heavily loaded, defer heavy operations (like full database dumps) until a low-traffic window or use `ionice -c3 nice -n19` to deprioritize inspection commands.

### 3. Application & Directory Mapping
Verify the structure and permissions of the app and data directories without traversing or modifying them.
```bash
# List directory structure (depth 2, no execution)
ls -laR /models/flight-recorder/ > /tmp/flight-recorder-inspection/dir_structure.txt

# Check file sizes and modification times
find /models/flight-recorder/data -type f -exec ls -lh {} \; > /tmp/flight-recorder-inspection/data_files.txt
find /models/flight-recorder/app -type f -exec ls -lh {} \; > /tmp/flight-recorder-inspection/app_files.txt

# Verify ownership and permissions
stat /models/flight-recorder/data /models/flight-recorder/app > /tmp/flight-recorder-inspection/permissions.txt
```

### 4. Network & Port Verification
Ensure the benchmark runner and model server are bound to expected interfaces and ports.
```bash
# List listening TCP/UDP sockets
ss -tlnp > /tmp/flight-recorder-inspection/listening_ports.txt

# Check for stale connections or TIME_WAIT states
ss -s > /tmp/flight-recorder-inspection/socket_summary.txt

# Verify DNS/resolution if applicable
getent hosts 192.168.1.116 > /tmp/flight-recorder-inspection/resolution.txt
```

### 5. Service & Process State
Determine how the application is managed (systemd, supervisor, docker, or direct Python process).
```bash
# Check systemd status if applicable
systemctl status flight-recorder.service > /tmp/flight-recorder-inspection/service_status.txt 2>&1

# Check for running Python processes related to the app
ps aux | grep -E "python.*flight|benchmark|model_server" > /tmp/flight-recorder-inspection/app_processes.txt

# Review recent logs for errors or warnings
journalctl -u flight-recorder.service --since "1 hour ago" --no-pager > /tmp/flight-recorder-inspection/recent_logs.txt 2>/dev/null || tail -n 500 /var/log/flight-recorder/app.log > /tmp/flight-recorder-inspection/recent_logs.txt
```

All inspection outputs are stored in `/tmp/flight-recorder-inspection/` for later reference. No files in `/models/flight-recorder/` are modified during this phase.

---

# Health Checks

With the baseline state captured, we proceed to targeted health checks that validate the operational readiness of the application, database, and benchmark infrastructure. These checks are designed to be idempotent and non-intrusive.

### 1. SQLite Database Integrity & Configuration
SQLite requires careful handling to avoid corruption during concurrent reads/writes. We will verify integrity, journal mode, and WAL status without acquiring exclusive locks.
```bash
# Open database in read-only mode to prevent accidental writes
sqlite3 -readonly /models/flight-recorder/data/benchmark.db <<'EOF'
PRAGMA integrity_check;
PRAGMA journal_mode;
PRAGMA wal_checkpoint(PASSIVE);
PRAGMA busy_timeout(5000);
SELECT name, type, sql FROM sqlite_master WHERE type IN ('table','index','trigger');
EOF
```
**Interpretation Guidelines:**
- `integrity_check` must return `ok`. Any other result indicates corruption requiring recovery.
- `journal_mode` should ideally be `WAL` for concurrent read/write performance. If `DELETE` or `TRUNCATE`, consider migrating to WAL during redeployment.
- `wal_checkpoint(PASSIVE)` attempts a safe checkpoint without blocking writers. If it hangs, the database may be under heavy write load; defer checkpointing until after redeployment.

### 2. Application Dependency & Environment Validation
Verify that Python dependencies, environment variables, and configuration files are intact.
```bash
# Check Python version and virtual environment
python3 --version
which python3
pip list > /tmp/flight-recorder-inspection/installed_packages.txt

# Validate configuration file syntax (YAML/JSON)
python3 -c "import yaml; yaml.safe_load(open('/models/flight-recorder/app/config.yaml'))" 2>&1 > /tmp/flight-recorder-inspection/config_syntax.txt
python3 -c "import json; json.load(open('/models/flight-recorder/app/config.json'))" 2>&1 > /tmp/flight-recorder-inspection/config_syntax.txt

# Check for missing environment variables
env | grep -E "FLIGHT|BENCHMARK|MODEL|DB" > /tmp/flight-recorder-inspection/env_vars.txt
```

### 3. Benchmark Runner State & Lock Files
Benchmark runners often use lock files or state directories to prevent concurrent executions. We must verify these are not stale.
```bash
# Search for lock files or PID files
find /models/flight-recorder -name "*.lock" -o -name "*.pid" -o -name "benchmark_state.json" 2>/dev/null > /tmp/flight-recorder-inspection/lock_files.txt

# Check for stale PID files (verify if process exists)
while read -r file; do
  if [ -f "$file" ]; then
    pid=$(cat "$file" 2>/dev/null)
    if kill -0 "$pid" 2>/dev/null; then
      echo "ACTIVE: $file (PID: $pid)"
    else
      echo "STALE: $file (PID: $pid)"
    fi
  fi
done < /tmp/flight-recorder-inspection/lock_files.txt > /tmp/flight-recorder-inspection/lock_analysis.txt
```

### 4. Network & API Health (If Applicable)
If the model server exposes an HTTP/gRPC endpoint for benchmark queries, verify responsiveness.
```bash
# HTTP health check (adjust port/path as needed)
curl -s -o /dev/null -w "%{http_code}" http://192.168.1.116:8080/health > /tmp/flight-recorder-inspection/http_health.txt

# gRPC health check (if grpcurl is available)
grpcurl -plaintext 192.168.1.116:50051 grpc.health.v1.Health/Check > /tmp/flight-recorder-inspection/grpc_health.txt 2>&1
```

### 5. Resource Contention Simulation
Before redeployment, simulate a lightweight load to ensure the system can handle benchmark traffic without degradation.
```bash
# Run a quick I/O and CPU stress test (non-destructive, short duration)
stress-ng --cpu 2 --io 2 --timeout 10s > /tmp/flight-recorder-inspection/stress_test.txt 2>&1
```
If the system becomes unresponsive or benchmarks fail under this minimal load, the underlying infrastructure may require scaling or maintenance before proceeding.

---

# Data Safety Checks

Data preservation is the highest priority. SQLite databases are vulnerable to corruption if interrupted during writes, and benchmark metadata must survive redeployment. This section outlines a rigorous backup and verification protocol.

### 1. Pre-Redeployment SQLite Backup
We will create a consistent, point-in-time backup using SQLite's native `.backup` command, which handles locking and journaling safely.
```bash
# Create a timestamped backup directory
mkdir -p /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)

# Perform safe backup
sqlite3 /models/flight-recorder/data/benchmark.db ".backup /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)/benchmark.db.backup"

# Verify backup integrity
sqlite3 /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)/benchmark.db.backup "PRAGMA integrity_check;"
```
**Safety Notes:**
- `.backup` copies the database file while maintaining transactional consistency.
- If the database is large (>10GB), consider using `pg_dump`-style streaming or `sqlite3 .dump` to a file, though `.backup` is preferred for speed and consistency.
- Never use `cp` or `rsync` on a live SQLite database without ensuring it's in WAL mode and idle, as file-level copies may capture partial writes.

### 2. Filesystem & Metadata Snapshot
Capture the exact state of the data directory for rollback purposes.
```bash
# Create a tar archive of the data directory (preserving permissions and timestamps)
tar -czvf /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)/data_directory.tar.gz -C /models/flight-recorder data/

# Generate a checksum for verification
sha256sum /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)/data_directory.tar.gz > /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)/checksums.sha256
```

### 3. Database Schema & Seed Data Verification
Ensure the schema matches expected benchmark requirements and that seed data is intact.
```bash
# Extract schema for comparison
sqlite3 /models/flight-recorder/data/benchmark.db ".schema" > /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)/schema.sql

# Count records in critical tables
sqlite3 /models/flight-recorder/data/benchmark.db <<'EOF'
SELECT 'runs' as tbl, COUNT(*) as cnt FROM runs
UNION ALL
SELECT 'metrics' as tbl, COUNT(*) as cnt FROM metrics
UNION ALL
SELECT 'configs' as tbl, COUNT(*) as cnt FROM configs;
EOF
```

### 4. Lock & Journal File Audit
Verify that no stale journal or WAL files are interfering with the database.
```bash
ls -la /models/flight-recorder/data/benchmark.db*
```
Expected files: `benchmark.db`, `benchmark.db-wal`, `benchmark.db-shm` (if WAL mode). If `benchmark.db-journal` exists, the database was using DELETE journal mode. Note this for potential migration during redeployment.

### 5. Backup Verification & Retention Policy
Confirm backups are readable and complete.
```bash
# Mount backup read-only to verify accessibility
sqlite3 /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)/benchmark.db.backup ".tables"
tar -tzf /tmp/flight-recorder-backups/$(date +%Y%m%d_%H%M%S)/data_directory.tar.gz | head -20
```
Backups are stored in `/tmp/flight-recorder-backups/` and will be retained until post-remediation verification is complete. They will not be deleted automatically.

---

# Redeploy Procedure

The redeployment phase focuses on updating the application code, refreshing dependencies, and restarting the service with minimal disruption. We assume a Python-based application managed by systemd or a direct process supervisor.

### 1. Pre-Redeployment Graceful Pause
If the server is actively running benchmarks, we must pause new executions without killing active processes.
```bash
# Check if a benchmark controller or API exists
curl -s -X POST http://192.168.1.116:8080/api/benchmark/pause 2>/dev/null || echo "No pause endpoint found"
```
If no API exists, we will rely on graceful process termination during restart.

### 2. Code & Dependency Update
Pull the latest stable code and verify dependency integrity.
```bash
cd /models/flight-recorder/app

# Pull latest changes (read-only check first)
git fetch origin
git status

# Install/verify dependencies in a dry-run mode
pip install --dry-run -r requirements.txt > /tmp/flight-recorder-inspection/dep_check.txt

# Apply updates
pip install -r requirements.txt --quiet
```

### 3. Configuration Validation
Ensure configuration files are syntactically correct and reference valid paths.
```bash
# Validate config against schema (if pydantic/jsonschema is used)
python3 -c "
import yaml, json
with open('config.yaml') as f:
    cfg = yaml.safe_load(f)
assert 'benchmark' in cfg, 'Missing benchmark section'
assert 'db_path' in cfg, 'Missing db_path'
print('Config validation passed')
"
```

### 4. Service Restart Strategy
Choose the appropriate restart method based on the observed service manager.

**Option A: Systemd Service**
```bash
# Reload daemon to pick up any unit file changes
systemctl daemon-reload

# Graceful restart (waits for current requests to finish)
systemctl restart flight-recorder.service

# Monitor startup logs
journalctl -u flight-recorder.service -f --no-pager &
```

**Option B: Direct Python Process**
```bash
# Identify current process
PID=$(pgrep -f "flight-recorder.*app")

# Send SIGTERM for graceful shutdown
if [ -n "$PID" ]; then
  kill -TERM "$PID"
  # Wait up to 30 seconds for graceful exit
  for i in {1..30}; do
    kill -0 "$PID" 2>/dev/null || break
    sleep 1
  done
  # If still running, escalate to SIGKILL [APPROVAL REQUIRED]
  if kill -0 "$PID" 2>/dev/null; then
    echo "Process did not exit gracefully. Awaiting approval for SIGKILL."
  fi
fi

# Start new process in background
nohup python3 /models/flight-recorder/app/main.py --config /models/flight-recorder/app/config.yaml > /var/log/flight-recorder/app.log 2>&1 &
echo $! > /tmp/flight-recorder-app.pid
```

### 5. Post-Restart Verification
Confirm the service is running and healthy.
```bash
# Check process status
ps aux | grep -E "flight-recorder|benchmark" | grep -v grep

# Verify port binding
ss -tlnp | grep 8080

# Check application logs for startup success
tail -n 50 /var/log/flight-recorder/app.log
```

### 6. Database Migration (If Required)
If the new code requires schema changes, apply them safely.
```bash
# Run migrations in a transaction
sqlite3 /models/flight-recorder/data/benchmark.db <<'EOF'
BEGIN TRANSACTION;
-- Example migration
-- ALTER TABLE runs ADD COLUMN benchmark_version TEXT;
COMMIT;
PRAGMA integrity_check;
EOF
```
Migrations are only applied if explicitly documented in the redeployment checklist.

---

# Benchmark Restart Procedure

With the application redeployed and data secured, we proceed to restore benchmark execution. The focus is on idempotency, state isolation, and observable progress.

### 1. Benchmark Runner State Reset
Clear stale execution artifacts without deleting historical data.
```bash
# Remove temporary lock files and PID files (non-destructive to data)
find /models/flight-recorder -name "*.lock" -delete 2>/dev/null
find /models/flight-recorder -name "*.pid" -delete 2>/dev/null

# Reset benchmark state tracker if it exists
python3 /models/flight-recorder/app/benchmark_runner.py --reset-state --dry-run
```

### 2. Configuration & Environment Validation
Ensure the benchmark runner can locate the database and configuration.
```bash
# Verify database connectivity
python3 -c "
import sqlite3
conn = sqlite3.connect('/models/flight-recorder/data/benchmark.db')
cursor = conn.cursor()
cursor.execute('SELECT COUNT(*) FROM runs')
print(f'Runs table accessible: {cursor.fetchone()[0]} rows')
conn.close()
"
```

### 3. Dry-Run Execution
Validate the benchmark pipeline without generating actual results.
```bash
# Run benchmark in dry-run mode (simulates steps, validates config, checks dependencies)
python3 /models/flight-recorder/app/benchmark_runner.py --config /models/flight-recorder/app/config.yaml --mode dry-run --output /tmp/flight-recorder-inspection/dry_run_report.json
```
Review the dry-run report for missing dependencies, invalid paths, or configuration errors.

### 4. Incremental Benchmark Execution
Start the benchmark runner with logging and progress tracking.
```bash
# Launch benchmark runner with verbose logging
nohup python3 /models/flight-recorder/app/benchmark_runner.py \
  --config /models/flight-recorder/app/config.yaml \
  --mode full \
  --log-level DEBUG \
  --output /models/flight-recorder/data/benchmark_results/ \
  > /var/log/flight-recorder/benchmark.log 2>&1 &

echo $! > /tmp/flight-recorder-benchmark.pid
```

### 5. Progress Monitoring & Validation
Monitor execution in real-time and verify output integrity.
```bash
# Tail logs for progress
tail -f /var/log/flight-recorder/benchmark.log

# Check for output files
ls -lh /models/flight-recorder/data/benchmark_results/

# Verify database writes
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT * FROM runs ORDER BY created_at DESC LIMIT 5;"
```

### 6. Result Consistency Check
Ensure benchmark outputs are deterministic and match expected formats.
```bash
# Run a quick validation script
python3 /models/flight-recorder/app/validate_benchmark_output.py --path /models/flight-recorder/data/benchmark_results/
```

---

# Human Approval Gates

Certain actions carry irreversible consequences or require architectural decisions. These gates must be explicitly approved before proceeding.

### Gate 1: Forceful Process Termination
If the application process does not exit gracefully after SIGTERM, escalation to SIGKILL requires approval.
```bash
[APPROVAL REQUIRED] kill -9 <PID>
```
**Rationale:** SIGKILL bypasses cleanup handlers, potentially leaving temporary files or database locks in an inconsistent state.

### Gate 2: Database Schema Migration
Applying schema changes to the production database requires explicit confirmation.
```bash
[APPROVAL REQUIRED] sqlite3 /models/flight-recorder/data/benchmark.db "ALTER TABLE runs ADD COLUMN benchmark_version TEXT;"
```
**Rationale:** Schema changes are difficult to reverse and may break older benchmark runners or data pipelines.

### Gate 3: Stale Lock File Removal
Deleting lock files that may indicate an active but unresponsive process requires confirmation.
```bash
[APPROVAL REQUIRED] rm -f /models/flight-recorder/data/benchmark.lock
```
**Rationale:** Removing a lock file while a process is still running may cause concurrent execution conflicts.

### Gate 4: Network/Port Reconfiguration
Changing listening ports or firewall rules requires approval.
```bash
[APPROVAL REQUIRED] iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
[APPROVAL REQUIRED] firewall-cmd --add-port=8080/tcp --permanent
```
**Rationale:** Network changes can disrupt external benchmark clients or monitoring systems.

### Gate 5: Backup Deletion
Removing old backups to free disk space requires explicit authorization.
```bash
[APPROVAL REQUIRED] rm -rf /tmp/flight-recorder-backups/old_timestamp/
```
**Rationale:** Backups are the primary recovery mechanism. Premature deletion eliminates rollback capability.

Each gate will present a clear prompt with the exact command, rationale, and risk assessment. The agent will halt execution until explicit approval is received.

---

# Stop Conditions

The remediation run must be immediately halted if any of the following conditions are detected:

1. **Database Integrity Failure:** `PRAGMA integrity_check` returns anything other than `ok`. Continuing could propagate corruption.
2. **Unresponsive Server:** SSH session drops repeatedly, or system load remains at 100% for >5 minutes despite idle state, indicating kernel panic or hardware failure.
3. **Benchmark Runner Crash:** The benchmark process exits with a non-zero code and core dump, indicating a fatal application error.
4. **Data Directory Modification:** Any unexpected file creation, deletion, or permission change in `/models/flight-recorder/data/` outside of controlled operations.
5. **Manual Abort Signal:** Receipt of `SIGINT` or `SIGTERM` from the operator, or explicit command to halt.
6. **Stale Lock Persistence:** Lock files remain after process termination, indicating zombie processes or filesystem corruption.
7. **Backup Verification Failure:** The restored backup fails integrity checks or cannot be opened by SQLite.
8. **Resource Exhaustion:** Disk usage exceeds 95%, or memory swap usage spikes dramatically, indicating insufficient capacity for benchmark execution.

Upon hitting a stop condition, the agent will:
- Log the exact state and error context.
- Preserve all diagnostic outputs in `/tmp/flight-recorder-inspection/`.
- Notify the operator with a clear summary of the failure point.
- Await further instructions before proceeding.

---

# Recovery Commands

If the remediation run fails or produces an unstable state, the following recovery commands will restore the environment to its pre-remediation baseline.

### 1. Database Restoration
```bash
# Stop any running benchmark processes
pkill -f "benchmark_runner" || true

# Restore database from verified backup
sqlite3 /models/flight-recorder/data/benchmark.db ".restore /tmp/flight-recorder-backups/<timestamp>/benchmark.db.backup"

# Verify restored database
sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;"
```

### 2. Filesystem Rollback
```bash
# Remove potentially corrupted or modified files
rm -f /models/flight-recorder/data/benchmark_results/*

# Restore data directory from tar backup
tar -xzf /tmp/flight-recorder-backups/<timestamp>/data_directory.tar.gz -C /models/flight-recorder/

# Verify permissions
chmod -R 755 /models/flight-recorder/data
chown -R user:user /models/flight-recorder/data
```

### 3. Application Rollback
```bash
# Revert code to previous stable commit
cd /models/flight-recorder/app
git checkout <previous-stable-commit>

# Restore dependencies
pip install -r requirements.txt --quiet

# Restart service
systemctl restart flight-recorder.service || nohup python3 main.py > /var/log/flight-recorder/app.log 2>&1 &
```

### 4. State & Lock Cleanup
```bash
# Remove stale artifacts
rm -f /tmp/flight-recorder-app.pid /tmp/flight-recorder-benchmark.pid
find /models/flight-recorder -name "*.lock" -delete 2>/dev/null

# Clear temporary inspection files
rm -rf /tmp/flight-recorder-inspection/
```

### 5. Verification Post-Recovery
```bash
# Confirm service is healthy
systemctl status flight-recorder.service
curl -s http://192.168.1.116:8080/health

# Verify database accessibility
sqlite3 /models/flight-recorder/data/benchmark.db "SELECT COUNT(*) FROM runs;"

# Run dry-run benchmark to confirm pipeline
python3 /models/flight-recorder/app/benchmark_runner.py --mode dry-run
```

Recovery commands are designed to be idempotent and safe to run multiple times. All operations preserve the original backup until explicit deletion is approved.

---

# Summary

This remediation plan provides a comprehensive, safety-first pathway to restore reliable benchmark execution on the remote model server at `192.168.1.116`. The approach is structured around five core principles:

1. **Non-Destructive Inspection:** All initial diagnostics use read-only commands, deprioritized I/O, and safe process enumeration to avoid interfering with a busy server.
2. **Data Preservation:** SQLite backups are created using `.backup` for transactional consistency, verified for integrity, and archived with checksums. The data directory is never deleted or modified without explicit approval.
3. **Controlled Redeployment:** Application updates are validated through dry-runs, dependency checks, and graceful service restarts. Schema changes and forceful process termination are gated behind explicit human approval.
4. **Safe Benchmark Restoration:** The benchmark runner is restarted with state isolation, dry-run validation, and incremental execution. Progress is monitored via logs and database writes, with consistency checks ensuring deterministic output.
5. **Explicit Oversight & Rollback:** Human approval gates prevent irreversible actions. Stop conditions halt execution on integrity failures, resource exhaustion, or manual abort. Recovery commands provide a verified rollback path to the pre-remediation state.

By adhering to this plan, the agent ensures that benchmark execution is restored reliably while maintaining strict data safety, minimizing downtime, and preserving full auditability. The workflow balances operational efficiency with defensive engineering practices, making it suitable for production-grade model server environments where data integrity and execution reliability are paramount.