# Mission
The primary objective of this remediation run is to restore reliable benchmark execution on the remote model server (192.168.1.116) while maintaining absolute data integrity. The "Flight Recorder" application serves as a critical telemetry and logging system; therefore, the remediation must prioritize the preservation of the SQLite database and any existing records within `/models/flight-recorder/data`.

The agent will follow a "Look-Before-You-Leap" methodology:
1.  **Assess:** Determine the current state of the application, the health of the underlying hardware/OS, and the integrity of the database.
2.  **Stabilize:** Identify and resolve any blocking processes or configuration errors without altering the data layer.
3.  **Restore:** Redeploy the application code to a known-good state and resume the benchmark suite.
4.  **Verify:** Confirm that the benchmark is producing valid, consistent results and that the service remains stable under load.

The agent is strictly prohibited from performing any destructive actions on the `/models/flight-recorder/data` directory. Any command that could result in data loss, even if intended to "clean up" temporary files, requires explicit human authorization.

# Read-Only Inspection
Before any changes are made, the agent must perform a comprehensive audit of the current environment to establish a baseline.

### 1.1 File System Audit
The agent will inspect the application structure to ensure the code is intact and identify the entry points.
*   **Action:** List all files in the application directory.
*   **Command:** `ls -R /models/flight-recorder/app`
*   **Goal:** Identify the main execution script (e.g., `main.py`, `app.py`, or `run_benchmark.py`) and check for recent modifications.
*   **Action:** Inspect configuration files.
*   **Command:** `cat /models/flight-recorder/app/config.yaml` (or `.env`, `settings.py`)
*   **Goal:** Verify database paths, port assignments, and environment variables.

### 1.2 Log Analysis
The agent will search for error patterns that explain the current failure of the benchmark.
*   **Action:** Check the last 200 lines of the application logs.
*   **Command:** `tail -n 200 /models/flight-recorder/app/logs/error.log` (Adjust path as necessary based on discovery).
*   **Goal:** Look for "Database is locked," "Connection refused," "Out of Memory (OOM)," or "Timeout" errors.
*   **Action:** Search for specific benchmark failure keywords.
*   **Command:** `grep -i "error" /models/flight-recorder/app/logs/benchmark.log`

### 1.3 Database Schema Inspection
The agent will verify the structure of the SQLite database without querying or modifying the records.
*   **Action:** List tables and verify schema.
*   **Command:** `sqlite3 /models/flight-recorder/data/flight_recorder.db ".tables"`
*   **Command:** `sqlite3 /models/flight-recorder/data/flight_recorder.db ".schema"`
*   **Goal:** Ensure the schema matches the expected version of the application code.

### 1.4 Process Inspection
The agent will identify what is currently running and how it is consuming resources.
*   **Action:** List all processes related to the flight recorder.
*   **Command:** `ps aux | grep -i "flight-recorder"`
*   **Goal:** Identify zombie processes, multiple instances of the app running simultaneously, or hung threads.

# Health Checks
The agent must ensure the server environment is capable of supporting the benchmark load.

### 2.1 Resource Utilization
*   **CPU/Memory:** Run `top -b -n 1 | head -n 20` to check for high load averages or memory exhaustion.
*   **Disk Space:** Run `df -h /models/flight-recorder/data` to ensure there is sufficient space for new benchmark entries.
*   **Inodes:** Run `df -i` to ensure the filesystem hasn't run out of inodes (common in high-frequency logging).

### 2.2 Network Connectivity
*   **Port Check:** Verify if the application port is listening.
*   **Command:** `ss -tulpn | grep :<PORT_NUMBER>`
*   **Goal:** Confirm the service is bound to the correct interface and port.

### 2.3 SQLite Lock Status
Since SQLite can suffer from "Database is locked" errors during concurrent writes:
*   **Action:** Check for active file locks.
*   **Command:** `lsof /models/flight-recorder/data/flight_recorder.db`
*   **Goal:** Identify if another process (e.g., a hung previous run) is holding a write lock on the database.

# Data Safety Checks
This is the most critical phase. The agent must guarantee that the data is safe before proceeding with a redeploy.

### 3.1 Integrity Check
The agent will run the built-in SQLite integrity check.
*   **Command:** `sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA integrity_check;"`
*   **Expected Result:** `ok`
*   **Action if Failed:** Stop immediately and alert the human operator. Do not attempt to "fix" the database.

### 3.2 Write-Ahead Log (WAL) Check
If the database uses WAL mode, the agent should check for the existence of `-wal` and `-shm` files.
*   **Action:** `ls -lh /models/flight-recorder/data/*.db*`
*   **Goal:** Ensure these files are present and not excessively large, which could indicate a stalled checkpoint.

### 3.3 Backup Verification (If applicable)
If a backup script exists, the agent will verify the timestamp of the last successful backup.
*   **Action:** `ls -lt /models/flight-recorder/backups/`
*   **Goal:** Confirm a recent recovery point exists before any redeployment occurs.

# Redeploy Procedure
Once the environment is inspected and data is confirmed safe, the agent will proceed with a clean redeployment.

### 4.1 Graceful Shutdown
The agent will attempt to stop the current service gracefully to release file handles and network ports.
*   **Command:** `systemctl stop flight-recorder` (or the relevant process kill command).
*   **Fallback:** If the service is hung, the agent will request permission to use `SIGKILL`.

### 4.2 Environment Preparation
*   **Dependency Check:** Verify that the Python environment (venv/conda) is active and dependencies are met.
*   **Command:** `pip freeze > requirements_current.txt` (for comparison).
*   **Action:** If a specific version mismatch is found, the agent will prepare a `pip install -r requirements.txt` command.

### 4.3 Application Restart
*   **Action:** Restart the service.
*   **Command:** `systemctl start flight-recorder`
*   **Verification:** Check the logs immediately after start to ensure the "Application Started" message appears.
*   **Command:** `tail -f /models/flight-recorder/app/logs/error.log`

# Benchmark Restart Procedure
After the service is confirmed stable, the agent will resume the benchmark.

### 5.1 State Synchronization
*   **Action:** Identify the last successful benchmark ID in the database.
*   **Command:** `sqlite3 /models/flight-recorder/data/flight_recorder.db "SELECT id FROM benchmarks ORDER BY timestamp DESC LIMIT 1;"`
*   **Goal:** Ensure the benchmark resumes from the correct point rather than restarting from zero (if the benchmark logic supports resuming).

### 5.2 Execution
*   **Action:** Trigger the benchmark script.
*   **Command:** `python3 /models/flight-recorder/app/run_benchmark.py --resume` (or appropriate flag).
*   **Monitoring:** The agent will monitor the output stream for 5 minutes to ensure no immediate crashes occur.

### 5.3 Validation
*   **Action:** Verify that new rows are being inserted into the database.
*   **Command:** `sqlite3 /models/flight-recorder/data/flight_recorder.db "SELECT count(*) FROM benchmarks WHERE timestamp > datetime('now', '-1 minute');"`

# Human Approval Gates
The agent must pause and request explicit human approval at the following points:

1.  **Pre-Shutdown:** Before stopping the active `flight-recorder` service (to ensure no critical data is currently being flushed).
2.  **Dependency Modification:** Before running any `pip install` or `git pull` commands that modify the application code.
3.  **Force Kill:** If a process is unresponsive and requires a `kill -9` command.
4.  **Benchmark Resume:** Before initiating the benchmark run to ensure the human operator is ready to monitor the output.
5.  **Data Anomaly:** If `PRAGMA integrity_check` returns anything other than `ok`.

# Stop Conditions
The agent will abort the remediation and notify the human operator if any of the following occur:

*   **Data Corruption:** The SQLite integrity check fails.
*   **Disk Full:** Available space on `/models/flight-recorder/data` falls below 5%.
*   **Persistent Failure:** The application fails to start 3 consecutive times after a redeploy.
*   **Unauthorized Action:** The agent detects a command that would delete files in the `/data` directory.
*   **Human Denial:** The human operator denies permission for a required step.
*   **Resource Exhaustion:** CPU load remains at 100% for more than 10 minutes with no progress in the logs.

# Recovery Commands
The following commands are categorized by their role in the remediation.

### Inspection Commands
```bash
# Check app structure
ls -R /models/flight-recorder/app

# Check logs for errors
tail -n 200 /models/flight-recorder/app/logs/error.log

# Check database schema
sqlite3 /models/flight-recorder/data/flight_recorder.db ".schema"

# Check for locked files
lsof /models/flight-recorder/data/flight_recorder.db
```

### Health & Safety Commands
```bash
# Check disk space
df -h /models/flight-recorder/data

# Check SQLite integrity
sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA integrity_check;"

# Check memory/CPU
top -b -n 1 | head -n 20
```

### Redeploy Commands
```bash
# Stop service (Requires Approval)
sudo systemctl stop flight-recorder

# Start service
sudo systemctl start flight-recorder

# Verify service status
systemctl status flight-recorder
```

### Benchmark Commands
```bash
# Check last benchmark ID
sqlite3 /models/flight-recorder/data/flight_recorder.db "SELECT id FROM benchmarks ORDER BY timestamp DESC LIMIT 1;"

# Run benchmark (Requires Approval)
python3 /models/flight-recorder/app/run_benchmark.py --resume
```

# Summary
The remediation plan follows a strict safety-first protocol. By prioritizing **Read-Only Inspection** and **Data Safety Checks** (specifically SQLite integrity), we ensure that the "Flight Recorder" data remains untouched. The **Redeploy Procedure** focuses on a clean restart of the application logic, while the **Benchmark Restart Procedure** ensures continuity of the benchmark suite. Human approval gates are strategically placed at every point of potential risk (service interruption, code modification, or benchmark execution). The agent will only proceed if the environment is healthy and the data is verified as intact.

# Detailed Contingency and Rollback Plan
A cautious agent must always have a "Plan B" and "Plan C" ready before initiating any changes to a production or benchmark environment. This section outlines the specific actions to take if the primary remediation steps fail or if unexpected issues arise during the process.

### 1.1 Rollback Strategy for Application Redeployment
If the application fails to start or exhibits critical errors (e.g., 500 Internal Server Errors, immediate crashes, or database connection failures) after the redeployment procedure:
*   **Action:** Revert to the previous known-good state of the application code.
*   **Command:** `git checkout <previous_stable_commit_hash> /models/flight-recorder/app`
*   **Action:** Restart the service.
*   **Command:** `sudo systemctl restart flight-recorder`
*   **Verification:** Check logs to ensure the application is running on the previous version.
*   **Human Approval Gate:** Required before any `git checkout` or rollback action.

### 1.2 Database Lock Recovery
If the application fails to start because the SQLite database is persistently locked by a "zombie" process that `systemctl stop` failed to terminate:
*   **Action:** Identify the PID holding the lock.
*   **Command:** `lsof /models/flight-recorder/data/flight_recorder.db`
*   **Action:** If the process is identified as a defunct or orphaned instance of the flight-recorder, request permission to terminate it.
*   **Command (Requires Approval):** `kill -9 <PID>`
*   **Action:** Verify the lock is released.
*   **Command:** `lsof /models/flight-recorder/data/flight_recorder.db` (Should return no output).

### 1.3 Data Integrity Failure Contingency
If `PRAGMA integrity_check` returns anything other than `ok`:
*   **Action:** **STOP ALL ACTIONS.** Do not attempt to restart the service or run the benchmark.
*   **Action:** Notify the human operator immediately with the specific error message from the integrity check.
*   **Action:** Provide the operator with the output of `sqlite3 /models/flight-recorder/data/flight_recorder.db ".dump" | head -n 100"` to help them assess the extent of the corruption.
*   **Action:** Wait for manual instructions. No automated recovery should be attempted on a corrupted database.

### 1.4 Resource Exhaustion Contingency
If the benchmark causes the server to hit 100% CPU or 100% Memory usage within the first 2 minutes of execution:
*   **Action:** Immediately stop the benchmark script.
*   **Command:** `pkill -f run_benchmark.py`
*   **Action:** Analyze the `top` and `free -m` output to identify the bottleneck.
*   **Action:** Request human approval to adjust the benchmark parameters (e.g., reducing concurrency, increasing sleep intervals, or decreasing batch sizes).

# SQLite Concurrency and Locking Deep Dive
Because the system uses SQLite, understanding its locking mechanisms is vital for a safe remediation. SQLite's locking behavior can vary significantly based on the journal mode and the environment.

### 2.1 Journal Mode Analysis
The agent will check the current journal mode of the database.
*   **Command:** `sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA journal_mode;"`
*   **Analysis:**
    *   **DELETE (Default):** The database is locked exclusively during writes. This is prone to "Database is locked" errors if multiple processes attempt to write.
    *   **WAL (Write-Ahead Logging):** Allows multiple readers and one writer to operate simultaneously. This is much more robust for a benchmark environment.
*   **Recommendation:** If the database is in `DELETE` mode and the benchmark is failing due to locks, the agent will request permission to switch to `WAL` mode.
*   **Command (Requires Approval):** `sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA journal_mode=WAL;"`

### 2.2 Busy Timeout Configuration
SQLite has a `busy_timeout` setting that tells the connection how long to wait for a lock to clear before throwing an error.
*   **Action:** Inspect the application code (e.g., `app.py` or `db_config.py`) to see if a `busy_timeout` is set.
*   **Goal:** Ensure the timeout is at least 5000ms (5 seconds). If it is set to 0 or a very low value, the agent will propose a configuration change to increase it.

### 2.3 WAL File Management
If the database is in WAL mode, the agent will monitor the size of the `-wal` file.
*   **Action:** `ls -lh /models/flight-recorder/data/*.db-wal`
*   **Analysis:** If the `-wal` file is several gigabytes in size, it indicates that a "checkpoint" is not occurring. This can happen if a process holds a long-running read transaction.
*   **Action:** The agent will report this to the human operator as a potential performance bottleneck.

# Network and Latency Profiling
Since the server is remote (192.168.1.116), network stability is a prerequisite for reliable benchmark execution.

### 3.1 Connectivity Baseline
*   **Action:** Perform a series of pings to check for packet loss.
*   **Command:** `ping -c 10 192.168.1.116`
*   **Goal:** Ensure < 1% packet loss.

### 3.2 Port Availability and Latency
*   **Action:** Use `nc` (netcat) to check the latency of the application port.
*   **Command:** `nc -zv 192.168.1.116 <PORT_NUMBER>`
*   **Action:** Use `curl` to check the response time of the health check endpoint (if available).
*   **Command:** `curl -o /dev/null -s -w "Connect: %{time_connect}s\nTotal: %{time_total}s\n" http://192.168.1.116:<PORT_NUMBER>/health`
*   **Goal:** Identify if network jitter is causing the benchmark to time out.

# Post-Remediation Validation Suite
Once the benchmark is restarted, the agent will perform a structured validation to ensure the remediation was successful and the system is stable.

### 4.1 Success Metrics
The agent will monitor the following metrics for a 10-minute window post-restart:
1.  **Database Growth:** Verify that the database file size is increasing steadily (indicating successful writes).
2.  **Error Rate:** Verify that `error.log` shows zero new entries for 5 minutes.
3.  **Throughput:** Compare the number of records inserted per minute against the expected benchmark baseline.
4.  **Resource Stability:** Ensure CPU and Memory usage do not show a linear upward trend (which would indicate a memory leak).

### 4.2 Validation Commands
*   **Check for new records:**
    `sqlite3 /models/flight-recorder/data/flight_recorder.db "SELECT count(*) FROM benchmarks WHERE timestamp > datetime('now', '-2 minutes');"`
*   **Check for memory leaks:**
    `vmstat 1 10` (Look for increasing 'swp' or 'free' memory depletion).
*   **Check for log errors:**
    `tail -n 50 /models/flight-recorder/app/logs/error.log`

# Detailed Execution Timeline
The agent will follow this chronological sequence of operations. Each step must be completed and verified before moving to the next.

| Time (Est.) | Phase | Action | Verification |
| :--- | :--- | :--- | :--- |
| T+0 | **Inspection** | Run `ls -R`, `tail` logs, and `sqlite3 .schema`. | Agent confirms current state. |
| T+5 | **Health Check** | Run `df -h`, `top`, and `lsof` on the DB. | Agent confirms resources are available. |
| T+10 | **Data Safety** | Run `PRAGMA integrity_check`. | Agent confirms "ok" status. |
| T+15 | **Approval** | Request permission to stop service and redeploy. | **Human Approval Required.** |
| T+20 | **Shutdown** | Execute `systemctl stop`. | Agent confirms process is dead. |
| T+25 | **Redeploy** | Restart service and check logs. | Agent confirms "Started" message. |
| T+30 | **Sync** | Query last benchmark ID. | Agent confirms resume point. |
| T+35 | **Approval** | Request permission to start benchmark. | **Human Approval Required.** |
| T+40 | **Execution** | Run `run_benchmark.py --resume`. | Agent monitors output stream. |
| T+50 | **Validation** | Run count queries and check logs. | Agent confirms success. |
| T+60 | **Final Report** | Summarize all actions and status. | Agent provides final summary. |

# Extended Log Analysis Patterns
The agent will use the following regex patterns to scan logs for specific failure modes during the inspection phase.

### 5.1 Database Errors
*   **Pattern:** `(SQLITE_BUSY|SQLITE_LOCKED)`
*   **Interpretation:** Indicates concurrent access issues.
*   **Action:** Check for multiple instances of the app running.

### 5.2 Connection Errors
*   **Pattern:** `(ConnectionRefusedError|TimeoutError)`
*   **Interpretation:** The application cannot reach the database or a downstream service.
*   **Action:** Check network connectivity and port bindings.

### 5.3 Memory/Resource Errors
*   **Pattern:** `(MemoryError|Out of Memory|OOM)`
*   **Interpretation:** The process is being killed by the OS or hitting Python's memory limits.
*   **Action:** Check `dmesg` for OOM killer activity and `free -m` for available RAM.

### 5.4 Permission Errors
*   **Pattern:** `(PermissionError|AccessDenied)`
*   **Interpretation:** The application user does not have write access to the `/data` directory or the `.db` file.
*   **Action:** Check `ls -l` permissions on the data directory.

# Risk Matrix
The agent will evaluate every action against this risk matrix before execution.

| Action | Risk Level | Potential Impact | Mitigation Strategy |
| :--- | :--- | :--- | :--- |
| `ls`, `cat`, `grep` | Low | None (Read-only) | None required. |
| `sqlite3 .schema` | Low | None (Read-only) | None required. |
| `systemctl stop` | Medium | Temporary service downtime | Ensure logs are flushed; request approval. |
| `pip install` | Medium | Dependency conflict | Use `requirements.txt`; request approval. |
| `kill -9` | High | Potential data corruption | Only use if `systemctl stop` fails; request approval. |
| `run_benchmark.py` | High | Resource exhaustion | Monitor `top` and `df` in real-time; request approval. |
| `PRAGMA journal_mode`| Medium | Database lock during change | Request approval; perform during maintenance window. |

# Final Verification Checklist
Before declaring the mission complete, the agent must check off the following:
- [ ] **Data Integrity:** `PRAGMA integrity_check` returned `ok`.
- [ ] **Service Status:** `systemctl status` shows `active (running)`.
- [ ] **Log Cleanliness:** No new errors in `error.log` for 5 minutes.
- [ ] **Benchmark Progress:** Database row count is increasing.
- [ ] **Resource Stability:** CPU/Memory usage is within normal bounds.
- [ ] **Human Confirmation:** Human operator has acknowledged the successful restart.

# Detailed Environment Audit
To ensure the environment is perfectly prepared for the benchmark, the agent will perform a deep audit of the environment variables and system limits.

### 6.1 Environment Variable Audit
The agent will check for the following variables which might affect the Flight Recorder's behavior:
*   **PYTHONPATH:** Ensure it points to the correct application directory.
*   **DB_PATH:** Verify it matches `/models/flight-recorder/data/flight_recorder.db`.
*   **LOG_LEVEL:** Ensure it is set to `INFO` or `DEBUG` for the duration of the remediation.
*   **MAX_CONCURRENT_THREADS:** Check if this is set too high for the available CPU cores.

### 6.2 System Limits (ulimit)
The agent will check the open file limits, as high-frequency logging can hit these limits quickly.
*   **Command:** `ulimit -n`
*   **Goal:** Ensure the limit is at least 4096. If it is lower (e.g., 1024), the agent will report this as a potential risk for the benchmark.

### 6.3 Timezone Consistency
The agent will verify that the system time and the database time are synchronized.
*   **Command:** `date`
*   **Action:** Compare with the timestamp of the last entry in the database.
*   **Goal:** Ensure there are no significant offsets that could mess up the benchmark's chronological ordering.

# Summary of Cautious Agent Behavior
The agent's behavior is governed by the principle of **Least Privilege** and **Maximum Observability**. By refusing to perform any action that could modify the `/data` directory without explicit human consent, the agent ensures that the primary goal—data preservation—is never compromised. The use of read-only inspection tools allows the agent to build a complete mental model of the failure before attempting any repairs. The structured redeployment and benchmark restart procedures ensure that the transition from a "failed" state to a "running" state is smooth, predictable, and easily reversible. Every step is logged, every risk is weighed, and every critical action is gated by human oversight, creating a robust framework for safe remote remediation.