# Mission
The primary objective of this remediation run is to restore the reliability of the benchmark execution pipeline on the remote model server (192.168.1.116) while ensuring 100% data integrity. The agent must transition the service from its current potentially unstable or "busy" state to a verified, stable state where benchmarks can be executed predictably. 

Key constraints for this mission:
1. **Data Preservation:** The `/models/flight-recorder/data` directory is the source of truth. No files within this directory shall be deleted or modified without explicit human authorization.
2. **Non-Destructive Approach:** All initial actions must be read-only.
3. **Service Stability:** The goal is to achieve a "Green" status where the application is responsive, the database is healthy, and the benchmark script can complete a full cycle without crashing.
4. **Auditability:** Every step taken must be logged, and any action that modifies the state of the system (restarts, redeploys, etc.) must be preceded by a human approval gate.

# Read-Only Inspection
Before any changes are made, the agent must perform a comprehensive audit of the current environment to understand the failure state.

### 1. File System Structure Audit
Verify the existence and permissions of the application and data directories.
- **Command:** `ls -laR /models/flight-recorder/app`
- **Purpose:** Ensure all source files are present and have correct ownership.
- **Command:** `ls -laR /models/flight-recorder/data`
- **Purpose:** Verify that the SQLite database files exist and are accessible.

### 2. Environment and Process Audit
Identify what is currently running and how the environment is configured.
- **Command:** `ps aux | grep flight-recorder`
- **Purpose:** Identify the Process IDs (PIDs) of the running application. Check if multiple instances are running (which could cause SQLite locking issues).
- **Command:** `env`
- **Purpose:** Check environment variables for database paths, API keys, or configuration flags.
- **Command:** `netstat -tulpn | grep :<port>` (Replace `<port>` with the known service port)
- **Purpose:** Confirm the service is listening on the expected port and identify the process owning that port.

### 3. Log Analysis
Review recent logs to identify the specific cause of the "busy" or "unreliable" state.
- **Command:** `tail -n 200 /models/flight-recorder/app/logs/*.log` (Adjust path if logs are elsewhere)
- **Purpose:** Look for "Database is locked," "Out of Memory," or "Timeout" errors.
- **Command:** `grep -iE "error|critical|exception" /models/flight-recorder/app/logs/*.log`
- **Purpose:** Filter for high-priority issues across the entire log history.

### 4. SQLite Connection Audit
Check if any processes are currently holding a lock on the database.
- **Command:** `lsof /models/flight-recorder/data/*.db`
- **Purpose:** Identify if a zombie process or a hung benchmark is holding a write lock on the SQLite file.

# Health Checks
These checks determine the "fitness" of the server to host the benchmark.

### 1. Resource Utilization
- **Command:** `uptime`
- **Purpose:** Check the load average. If load average is significantly higher than the CPU core count, the server is over-saturated.
- **Command:** `free -m`
- **Purpose:** Check available RAM. If "available" memory is < 500MB, the model server may be swapping, causing extreme latency.
- **Command:** `df -h /models/flight-recorder/data`
- **Purpose:** Ensure there is at least 20% free space on the data partition to prevent "Disk Full" errors during benchmark writes.

### 2. Network Connectivity
- **Command:** `ping -c 4 192.168.1.116` (Run from a jump host)
- **Purpose:** Verify basic reachability.
- **Command:** `curl -I http://192.168.1.116:<port>/health` (If a health endpoint exists)
- **Purpose:** Verify the application layer is responding to requests.

### 3. Database Integrity Check
This is a critical safety step.
- **Command:** `sqlite3 /models/flight-recorder/data/database.db "PRAGMA integrity_check;"`
- **Purpose:** Verify that the SQLite file is not corrupted.
- **Expected Output:** `ok`
- **Warning Sign:** Any output other than `ok` indicates corruption and requires an immediate halt and human intervention.

# Data Safety Checks
Before proceeding to any redeployment, the agent must ensure the data is backed up and protected.

### 1. Snapshot Creation
Even though we are not deleting data, we must create a point-in-time copy before any redeploy.
- **Command:** `cp -r /models/flight-recorder/data /models/flight-recorder/data_backup_$(date +%Y%m%d_%H%M%S)`
- **Purpose:** Create a safety net. If the redeploy fails or corrupts the database, we can revert to this snapshot.

### 2. Write Permission Verification
- **Command:** `test -w /models/flight-recorder/data && echo "Writable" || echo "Read-Only"`
- **Purpose:** Ensure the application user has the necessary permissions to write to the data directory.

### 3. SQLite Lock Check
- **Command:** `sqlite3 /models/flight-recorder/data/database.db "PRAGMA journal_mode;"`
- **Purpose:** Check the journaling mode (e.g., WAL, DELETE). WAL mode is generally preferred for concurrent access.

# Redeploy Procedure
If the inspection reveals configuration errors, outdated code, or hung processes, a redeploy is necessary.

### 1. Stop Current Service
- **[APPROVAL REQUIRED]**
- **Command:** `sudo systemctl stop flight-recorder.service` (or the relevant process manager command)
- **Purpose:** Gracefully shut down the application to release file locks.

### 2. Clean Environment
- **Command:** `cd /models/flight-recorder/app && git fetch --all && git reset --hard origin/main`
- **Purpose:** Ensure the local code matches the desired state exactly.
- **Command:** `pip install -r requirements.txt --no-cache-dir`
- **Purpose:** Reinstall dependencies to ensure no corrupted packages are present.

### 3. Service Restart
- **[APPROVAL REQUIRED]**
- **Command:** `sudo systemctl start flight-recorder.service`
- **Purpose:** Bring the application back online.

### 4. Post-Redeploy Verification
- **Command:** `tail -f /models/flight-recorder/app/logs/app.log`
- **Purpose:** Monitor the startup sequence for 60 seconds. Look for "Connected to database" and "Server started" messages.

# Benchmark Restart Procedure
Once the service is confirmed stable, the benchmark can be re-initiated.

### 1. Pre-Benchmark Warmup
- **Command:** `python3 /models/flight-recorder/app/scripts/warmup.py` (If available)
- **Purpose:** Populate caches and initialize connections without recording benchmark metrics.

### 2. Execute Benchmark
- **Command:** `python3 /models/flight-recorder/app/scripts/run_benchmark.py --mode=standard --output=/models/flight-recorder/data/results/`
- **Purpose:** Execute the actual benchmark.
- **Monitoring:** The agent should monitor the output in real-time.

### 3. Result Verification
- **Command:** `ls -l /models/flight-recorder/data/results/`
- **Purpose:** Confirm that a new result file was created and has a non-zero file size.
- **Command:** `sqlite3 /models/flight-recorder/data/database.db "SELECT * FROM benchmark_results ORDER BY timestamp DESC LIMIT 1;"`
- **Purpose:** Verify the result was successfully committed to the database.

# Human Approval Gates
The following actions require explicit "GO" from the human operator before execution:

1. **[APPROVAL REQUIRED] Stop Service:** Required before any code changes or database migrations.
2. **[APPROVAL REQUIRED] Code Redeploy:** Required before `git reset --hard` or `pip install`.
3. **[APPROVAL REQUIRED] Database Migration:** If the inspection reveals a schema mismatch, any migration script must be approved.
4. **[APPROVAL REQUIRED] Cache Clearing:** If the agent identifies a corrupted cache directory, clearing it requires approval.
5. **[APPROVAL REQUIRED] Service Restart:** Required to bring the system back to an active state.

# Stop Conditions
The agent must immediately cease all operations and alert the human operator if any of the following occur:

1. **Data Corruption:** If `PRAGMA integrity_check;` returns anything other than `ok`.
2. **Disk Exhaustion:** If the disk space on the data partition drops below 5% during the run.
3. **Persistent Lock:** If the service fails to start because the SQLite database is locked by a process that refuses to die.
4. **Unexpected Deletion:** If any command results in the removal of files in `/models/flight-recorder/data`.
5. **Resource Spike:** If CPU usage stays at 100% for more than 5 minutes without any benchmark progress (potential infinite loop).
6. **Network Isolation:** If the server loses connectivity to the internal network during the benchmark run.

# Recovery Commands
In the event of a failure during the remediation run, use these commands to return to a known state:

- **Emergency Stop:** `sudo systemctl stop flight-recorder.service`
- **Kill Zombie Processes:** `pkill -9 -f flight-recorder`
- **Rollback Data:** `cp -r /models/flight-recorder/data_backup_XXXX /models/flight-recorder/data` (Replace XXXX with the actual timestamp)
- **Clear Logs:** `truncate -s 0 /models/flight-recorder/app/logs/*.log` (Use only if logs are filling up disk space)
- **Reset Network:** `sudo systemctl restart networking` (If network issues are detected)

# Summary
This plan follows a "Safety-First" methodology. By prioritizing read-only inspection and data integrity checks (specifically SQLite integrity and disk space), we minimize the risk of exacerbating the current issue. The use of a snapshot backup before any redeployment ensures that even in a worst-case scenario, the data can be restored. The agent will proceed through the Read-Only Inspection phase first, providing a detailed report of the current state before requesting approval for any state-changing actions. The ultimate goal is a verified, stable environment where the benchmark can run to completion and record accurate data.

## Detailed Troubleshooting and Optimization
To ensure the long-term reliability of the benchmark execution, the agent must go beyond simple "fix and restart" actions and address the underlying systemic issues that lead to "busy" or "unreliable" states. This section provides deep-dive troubleshooting for common failure modes encountered in model server environments.

### 1. SQLite Concurrency and Locking Optimization
SQLite is highly efficient but can become a bottleneck or enter a "locked" state if multiple processes attempt to write simultaneously or if a transaction remains open for too long.
- **Symptom:** `sqlite3.OperationalError: database is locked`
- **Analysis:** This occurs when the database is currently being written to by another process, or a previous write operation failed to close its transaction properly.
- **Optimization Actions:**
    - **Busy Timeout:** Ensure the application is configured with a `busy_timeout`. This tells SQLite to wait for a specified duration (e.g., 30 seconds) before throwing a "locked" error.
    - **WAL Mode:** Verify that Write-Ahead Logging (WAL) is enabled. WAL allows multiple readers and one writer to operate concurrently without blocking each other.
    - **Command:** `sqlite3 /models/flight-recorder/data/database.db "PRAGMA journal_mode=WAL;"`
    - **Command:** `sqlite3 /models/flight-recorder/data/database.db "PRAGMA synchronous=NORMAL;"` (Balances safety and performance).
    - **Command:** `sqlite3 /models/flight-recorder/data/database.db "PRAGMA cache_size=-64000;"` (Allocates ~64MB of memory for the cache).

### 2. Memory Management and OOM Prevention
Model servers often experience "busy" states due to memory exhaustion, which triggers the Linux Out-Of-Memory (OOM) Killer or causes extreme swap usage.
- **Symptom:** Process disappears from `ps aux` or logs show "Killed" messages.
- **Analysis:** Check the kernel ring buffer for OOM events.
- **Command:** `dmesg | grep -i "out of memory"`
- **Action:** If OOM is detected, the agent must identify memory-intensive processes.
- **Optimization Actions:**
    - **Swap Check:** Check if the system is swapping heavily using `vmstat 1 5`. High swap activity indicates the physical RAM is insufficient for the current model load.
    - **Garbage Collection:** If the application is Python-based, ensure that large objects (like model weights or large dataframes) are being explicitly deleted or cleared from memory after use.
    - **Ulimit Check:** Verify the maximum number of open files allowed for the user.
    - **Command:** `ulimit -n` (If the value is low, e.g., 1024, it may cause "Too many open files" errors during high-concurrency benchmarks).

### 3. Dependency Integrity and Environment Isolation
A "busy" state can sometimes be caused by a corrupted Python environment or a version mismatch in a critical library (e.g., `torch`, `numpy`, or `sqlite3`).
- **Symptom:** `ImportError`, `ModuleNotFoundError`, or unexpected behavior in mathematical operations.
- **Analysis:** Compare the installed packages against the `requirements.txt` file.
- **Command:** `pip freeze > current_env.txt && diff current_env.txt requirements.txt`
- **Action:** If discrepancies are found, the agent should perform a clean reinstall of the dependencies.
- **Optimization Actions:**
    - **Virtual Environment:** Ensure the application is running inside a dedicated `venv` or `conda` environment to prevent conflicts with system-level packages.
    - **Pip Check:** Run `pip check` to identify any broken dependencies or version conflicts in the current environment.

## Dependency Integrity and Environment Isolation
To ensure the application code behaves predictably, the agent must verify that the execution environment is pristine and matches the intended specification.

### 1. Requirements Verification
The agent will perform a deep audit of the installed Python packages.
- **Command:** `pip list --format=json > installed_packages.json`
- **Purpose:** Export the current state of all installed libraries for comparison.
- **Command:** `pip check`
- **Purpose:** Automatically check for broken dependencies or incompatible versions.
- **Action:** If `pip check` returns errors, the agent must request approval to run `pip install -r requirements.txt --force-reinstall`.

### 2. Hash Verification (Optional but Recommended)
If the environment is highly sensitive, the agent can verify the integrity of the installed wheels.
- **Command:** `pip install --no-deps --require-hashes -r requirements.txt` (Requires a requirements file with hashes).
- **Purpose:** Ensures that the code being executed has not been tampered with or corrupted during download.

### 3. Path Integrity
Ensure that the Python interpreter is using the correct site-packages.
- **Command:** `python3 -c "import sys; print(sys.path)"`
- **Purpose:** Verify that the application is not accidentally importing libraries from a global or incorrect directory that might contain incompatible versions.

## Network Path and Latency Analysis
If the model server is "busy" but CPU/RAM usage is low, the bottleneck may be network-related, especially if the benchmark involves remote data fetching or API calls.

### 1. Latency and Jitter Profiling
- **Command:** `ping -c 20 192.168.1.116`
- **Purpose:** Check for packet loss and high jitter. If packet loss exceeds 1%, the network path is unstable.
- **Command:** `mtr -rw 192.168.1.116` (If `mtr` is installed).
- **Purpose:** Identify which specific hop in the network path is introducing latency or dropping packets.

### 2. TCP Connection Audit
- **Command:** `ss -tan | grep :<port>`
- **Purpose:** Check the state of TCP connections. Look for a high number of connections in `TIME_WAIT` or `CLOSE_WAIT` states, which could indicate a connection leak in the application code.
- **Command:** `curl -o /dev/null -s -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total: %{time_total}\n" http://192.168.1.116:<port>/health`
- **Purpose:** Measure the specific time taken for the TCP handshake, the Time to First Byte (TTFB), and the total request time.

## Symptom-Action Matrix
The agent will use the following matrix to quickly diagnose and respond to specific errors encountered during the remediation run.

| Symptom | Likely Cause | Diagnostic Command | Remediation Action |
| :--- | :--- | :--- | :--- |
| **"Database is locked"** | Concurrent write or hung transaction | `lsof /models/flight-recorder/data/*.db` | Kill zombie PID; Enable WAL mode; Increase `busy_timeout`. |
| **"Connection Refused"** | Service is down or port is blocked | `netstat -tulpn \| grep :<port>` | Check `systemctl status`; Check firewall rules. |
| **"Out of Memory" (OOM)** | Memory leak or model too large | `dmesg \| grep -i "oom"` | Restart service; Optimize model loading; Check `free -m`. |
| **"Timeout" (Request)** | High load or network latency | `curl -w "%{time_total}" ...` | Check `uptime`; Check `mtr`; Check DB query complexity. |
| **"Permission Denied"** | Incorrect file ownership | `ls -la /models/flight-recorder/data` | `chown` or `chmod` (Requires Approval). |
| **"ModuleNotFoundError"** | Missing dependency | `pip list` | `pip install -r requirements.txt`. |
| **"Disk Full"** | Log bloat or large data files | `df -h` | Truncate logs; Delete old temporary files (Requires Approval). |
| **"Slow Execution"** | CPU throttling or I/O wait | `iostat -xz 1` | Check for high `%iowait`; Check for CPU thermal throttling. |

## Post-Remediation Validation Suite
Once the service is redeployed and the benchmark is restarted, the agent must perform a series of "Smoke Tests" to confirm the fix is permanent.

### 1. Connectivity Smoke Test
- **Command:** `curl -s http://192.168.1.116:<port>/health | grep "OK"`
- **Success Criteria:** The response must contain "OK" or a 200 status code.

### 2. Database Write Test
- **Command:** `sqlite3 /models/flight-recorder/data/database.db "INSERT INTO test_table (val) VALUES ('test'); SELECT count(*) FROM test_table;"` (Adjust table name as needed).
- **Success Criteria:** The count should increment by 1 without any "locked" errors.

### 3. Benchmark Iteration Test
- **Action:** Run a single, short iteration of the benchmark.
- **Success Criteria:** The benchmark script completes without crashing and produces a result file in `/models/flight-recorder/data/results/`.

### 4. Log Integrity Check
- **Command:** `tail -n 50 /models/flight-recorder/app/logs/app.log`
- **Success Criteria:** No "Error," "Critical," or "Exception" strings should appear in the last 50 lines of the log.

## Security and Filesystem Permission Audit
To ensure the environment remains stable and secure, the agent will verify the filesystem permissions.

### 1. Ownership Audit
- **Command:** `ls -ld /models/flight-recorder/app /models/flight-recorder/data`
- **Purpose:** Ensure that the directories are owned by the correct service user (e.g., `model_user`) and not by `root` or a generic user.

### 2. Permission Bit Audit
- **Command:** `stat -c "%a %n" /models/flight-recorder/data`
- **Purpose:** Verify that the data directory has the correct permissions (e.g., `755` or `775`).
- **Action:** If permissions are incorrect, the agent will report the discrepancy and request approval to run `chmod`.

### 3. Umask Verification
- **Command:** `umask`
- **Purpose:** Check the default file permissions for the current session. A restrictive umask might prevent the application from creating new files in the data directory.

## Extended Rollback and Disaster Recovery
In the event of a catastrophic failure during the remediation (e.g., the database becomes unreadable or the service fails to start after a redeploy), the agent will follow this multi-stage rollback procedure.

### Stage 1: Immediate Service Halt
- **Command:** `sudo systemctl stop flight-recorder.service`
- **Command:** `pkill -9 -f flight-recorder`
- **Purpose:** Stop all activity to prevent further data corruption.

### Stage 2: Data Integrity Assessment
- **Command:** `sqlite3 /models/flight-recorder/data/database.db "PRAGMA integrity_check;"`
- **Decision:**
    - If `ok`: Proceed to Stage 3.
    - If `not ok`: **STOP.** Alert human operator. Do not attempt to fix the database automatically.

### Stage 3: Snapshot Restoration
- **Command:** `rm -rf /models/flight-recorder/data/*` (Requires Approval)
- **Command:** `cp -r /models/flight-recorder/data_backup_XXXX /models/flight-recorder/data/*`
- **Purpose:** Restore the data to the state it was in before the remediation run began.

### Stage 4: Environment Rollback
- **Command:** `git reset --hard HEAD@{1}`
- **Command:** `pip install -r requirements.txt`
- **Purpose:** Revert the application code and dependencies to the previous known-working state.

### Stage 5: Service Re-activation
- **Command:** `sudo systemctl start flight-recorder.service`
- **Purpose:** Bring the system back online in its original state.

## Long-term Stability and Prevention Roadmap
To prevent the recurrence of the issues identified during this remediation, the agent recommends the following long-term improvements.

### 1. Automated Monitoring and Alerting
- **Recommendation:** Implement a Prometheus exporter for the model server to track:
    - SQLite lock duration.
    - Memory usage per process.
    - Request latency percentiles (P95, P99).
- **Action:** Set up alerts to trigger when memory usage exceeds 80% or when the "Database Locked" error appears in the logs more than 3 times in 5 minutes.

### 2. Automated Backup Schedule
- **Recommendation:** Implement a cron job to perform a daily `sqlite3 .backup` of the database.
- **Action:** Store backups in a separate directory or a remote storage bucket to ensure data is preserved even if the local disk fails.

### 3. CI/CD Integration
- **Recommendation:** Move from manual redeploys to a CI/CD pipeline (e.g., GitHub Actions, GitLab CI).
- **Action:** Automate the `git fetch`, `pip install`, and `systemctl restart` steps. This ensures that every deployment is tested in a staging environment before hitting the production model server.

### 4. Resource Quotas
- **Recommendation:** Use Linux `cgroups` or Docker resource limits to cap the memory and CPU usage of the model server.
- **Action:** This prevents a single runaway benchmark or a memory leak from crashing the entire host machine.

### 5. Database Connection Pooling
- **Recommendation:** If the application is not already using it, implement a connection pool for SQLite.
- **Action:** This manages the number of concurrent connections and ensures that connections are returned to the pool promptly, reducing the likelihood of "Database Locked" errors.

## Final Execution Checklist
Before the agent begins the first step of the plan, it must confirm the following:
1. [ ] **Backup Verified:** Has the snapshot of `/models/flight-recorder/data` been successfully created?
2. [ ] **Permissions Confirmed:** Does the agent have `sudo` or sufficient permissions to restart the service?
3. [ ] **Human Availability:** Is there a human operator available to provide "GO" for the approval gates?
4. [ ] **Log Access:** Can the agent read the logs in `/models/flight-recorder/app/logs/`?
5. [ ] **Network Path:** Is the agent's connection to `192.168.1.116` stable?

The agent will now proceed to the **Read-Only Inspection** phase.