# Mission
The primary objective of this remediation run is to restore stable and reliable benchmark execution on the remote model server (192.168.1.116) while maintaining absolute data integrity. The "flight-recorder" service is critical for tracking model performance, and any loss of historical data in the `/models/flight-recorder/data` directory is unacceptable.

The agent will operate under a "Safety-First" protocol:
1.  **Observe:** Gather comprehensive telemetry on the current state of the application, database, and system resources.
2.  **Verify:** Confirm the integrity of the SQLite database and the consistency of the application code.
3.  **Remediate:** Perform a non-destructive redeployment of the application code to resolve any runtime instabilities.
4.  **Validate:** Execute a controlled benchmark restart to ensure the service is performing within expected parameters.
5.  **Protect:** Ensure that no commands are executed that could result in the deletion of the `/models/flight-recorder/data` directory or its contents.

# Read-Only Inspection
Before any intervention, the agent must establish a baseline of the current environment. This phase is strictly read-only.

**1. File System Structure & Permissions**
Verify the existence and permissions of the application and data directories.
*   `ls -la /models/flight-recorder/app`
*   `ls -la /models/flight-recorder/data`
*   `du -sh /models/flight-recorder/app /models/flight-recorder/data` (Check for unexpected size growth)

**2. Process & Network State**
Identify what is currently running and how the service is bound to the network.
*   `ps aux | grep flight-recorder` (Identify PID and parent processes)
*   `netstat -tunlp | grep 192.168.1.116` (Verify port bindings)
*   `lsof -i :<PORT>` (Replace <PORT> with the service's listening port)

**3. Application Logs**
Analyze recent errors to identify the root cause of the benchmark failure.
*   `tail -n 200 /models/flight-recorder/app/logs/error.log` (If exists)
*   `grep -i "error" /models/flight-recorder/app/logs/*.log`
*   `journalctl -u flight-recorder --since "1 hour ago"` (If managed by systemd)

**4. Environment Configuration**
Check the environment variables and configuration files.
*   `env` (Look for DB_PATH, API_KEYS, or MODEL_PATHS)
*   `cat /models/flight-recorder/app/config.yaml` (Or equivalent config file)

**5. SQLite Schema Inspection**
Understand the current database structure without querying large amounts of data.
*   `sqlite3 /models/flight-recorder/data/database.db ".schema"`
*   `sqlite3 /models/flight-recorder/data/database.db "SELECT name FROM sqlite_master WHERE type='table';"`

# Health Checks
Determine if the server is currently capable of handling a benchmark load.

**1. Resource Utilization**
Check for CPU exhaustion or memory leaks.
*   `top -b -n 1 | head -n 20`
*   `free -m` (Check available RAM)
*   `df -h` (Ensure the disk is not at 100% capacity, which would crash SQLite)

**2. Application Responsiveness**
Perform a basic "ping" to the application's health endpoint.
*   `curl -I http://192.168.1.116:<PORT>/health`
*   `curl -I http://192.168.1.116:<PORT>/status`

**3. Database Connectivity**
Verify that the SQLite file is accessible and not locked by another process.
*   `sqlite3 /models/flight-recorder/data/database.db "SELECT count(*) FROM some_known_table;"` (Use a small, known table)
*   `lsof /models/flight-recorder/data/database.db` (Check for multiple writers)

# Data Safety Checks
This is the most critical phase. We must ensure the database is not corrupted before proceeding.

**1. Integrity Check**
Run the internal SQLite integrity check.
*   `sqlite3 /models/flight-recorder/data/database.db "PRAGMA integrity_check;"`
    *   *Expected Output:* `ok`
    *   *Action if failed:* Stop immediately and alert human.

**2. Writeability Check**
Ensure the application has permission to write to the data directory.
*   `touch /models/flight-recorder/data/.write_test && rm /models/flight-recorder/data/.write_test`

**3. Journal/WAL File Check**
Check for lingering Write-Ahead Logs or Journal files that might indicate a crashed process.
*   `ls -l /models/flight-recorder/data/*.db-wal`
*   `ls -l /models/flight-recorder/data/*.db-journal`

**4. Backup (Non-Destructive)**
Create a snapshot of the data directory before any redeployment.
*   `cp -r /models/flight-recorder/data /models/flight-recorder/data_backup_$(date +%Y%m%d_%H%M%S)`

# Redeploy Procedure
If the inspection reveals application errors or outdated code, proceed with a clean redeployment of the *application code only*.

**1. Stop the Service**
Gracefully shut down the current instance to release file locks.
*   `systemctl stop flight-recorder` (or the relevant service command)
*   `pkill -f flight-recorder` (Only if systemctl fails)

**2. Clean Application Directory**
Remove temporary files or build artifacts in the app folder (DO NOT touch the data folder).
*   `rm -rf /models/flight-recorder/app/__pycache__`
*   `rm -rf /models/flight-recorder/app/.venv` (If using a local virtual environment)

**3. Reinstall Dependencies**
Ensure the environment is consistent.
*   `cd /models/flight-recorder/app`
*   `python3 -m venv .venv`
*   `source .venv/bin/activate`
*   `pip install -r requirements.txt`

**4. Restart the Service**
Bring the application back online.
*   `systemctl start flight-recorder`
*   `tail -f /models/flight-recorder/app/logs/error.log` (Monitor for immediate startup errors)

# Benchmark Restart Procedure
Once the service is confirmed healthy, initiate the benchmark.

**1. Warm-up Phase**
Run a low-intensity request to prime the cache and establish a baseline.
*   `curl http://192.168.1.116:<PORT>/warmup` (If available)

**2. Execute Benchmark**
Run the benchmark suite.
*   `python3 /models/flight-recorder/app/scripts/run_benchmark.py --duration 300 --output /models/flight-recorder/data/benchmarks/latest.json`

**3. Verification of Results**
Check if the benchmark successfully wrote to the database.
*   `sqlite3 /models/flight-recorder/data/database.db "SELECT * FROM benchmark_results ORDER BY timestamp DESC LIMIT 1;"`

# Human Approval Gates
The following actions require explicit human confirmation before execution:

1.  **[APPROVAL REQUIRED]** Any command involving `rm -rf` on any directory other than temporary cache files.
2.  **[APPROVAL REQUIRED]** Any database migration that involves `DROP TABLE` or `ALTER TABLE` (destructive schema changes).
3.  **[APPROVAL REQUIRED]** If `PRAGMA integrity_check` returns anything other than `ok`.
4.  **[APPROVAL REQUIRED]** If the disk space is below 10% and a cleanup of the `data` directory is proposed.
5.  **[APPROVAL REQUIRED]** If the agent detects a hardware failure (e.g., I/O errors) and suggests a server reboot.

# Stop Conditions
The agent must abort the remediation and notify the human operator if any of the following occur:

*   **Data Corruption:** `PRAGMA integrity_check` fails.
*   **Disk Failure:** `df -h` shows 100% utilization or `dmesg` shows I/O errors.
*   **Persistent Lock:** The SQLite database remains locked by a process that cannot be killed gracefully.
*   **Code Incompatibility:** The redeployed code fails to start or crashes immediately upon the first request.
*   **Unauthorized Modification:** Any attempt by the system to modify the `/models/flight-recorder/data` directory in an unexpected way.

# Recovery Commands
A summary of the primary commands to be used during the run:

**Inspection:**
```bash
ls -R /models/flight-recorder/app
ls -R /models/flight-recorder/data
sqlite3 /models/flight-recorder/data/database.db ".schema"
journalctl -u flight-recorder -n 100
```

**Safety:**
```bash
sqlite3 /models/flight-recorder/data/database.db "PRAGMA integrity_check;"
cp -r /models/flight-recorder/data /models/flight-recorder/data_backup_$(date +%Y%m%d)
```

**Redeploy:**
```bash
systemctl stop flight-recorder
cd /models/flight-recorder/app
rm -rf __pycache__
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
systemctl start flight-recorder
```

**Benchmark:**
```bash
python3 /models/flight-recorder/app/scripts/run_benchmark.py --duration 300
sqlite3 /models/flight-recorder/data/database.db "SELECT * FROM benchmark_results ORDER BY timestamp DESC LIMIT 1;"
```

# Summary
This plan provides a structured, cautious approach to restoring the flight-recorder service. By prioritizing **Read-Only Inspection** and **Data Safety Checks**, we ensure that the "source of truth" (the SQLite database) remains untouched. The **Redeploy Procedure** focuses strictly on the application logic, isolating it from the data layer. The inclusion of **Human Approval Gates** and **Stop Conditions** provides a fail-safe mechanism to prevent automated actions from causing irreversible damage. Following this sequence ensures that the benchmark execution is restored to a reliable state with minimal risk to the existing data.

# Detailed Operational Appendix

This appendix provides the granular technical details, troubleshooting matrices, and expanded procedures required to execute the remediation plan with the highest degree of precision. It serves as the "Standard Operating Procedure" (SOP) for the agent during complex scenarios.

# Detailed Troubleshooting & Edge Cases

During the remediation run, the agent may encounter specific technical hurdles. The following matrix provides the diagnostic path and resolution for common issues.

**1. SQLite Database Locked (`SQLITE_BUSY`)**
*   **Symptom:** Commands like `sqlite3 ... "SELECT ..."` return "database is locked".
*   **Diagnostic:** 
    *   `lsof /models/flight-recorder/data/database.db` (Identify the PID holding the lock).
    *   `ps -fp <PID>` (Identify the process name and user).
*   **Resolution:**
    *   If the process is a zombie or a hung instance of the `flight-recorder` service, attempt a graceful kill: `kill -15 <PID>`.
    *   If it remains locked, escalate to `kill -9 <PID>` (Only if the process is confirmed to be the old service instance).
    *   **[APPROVAL REQUIRED]** If the lock is held by an unknown process, the agent must stop and request human intervention.
*   **Prevention:** Ensure the `PRAGMA journal_mode=WAL;` is set in the application configuration to allow concurrent reads and writes.

**2. Port Conflict (Address already in use)**
*   **Symptom:** The service fails to start, and logs show "Address already in use".
*   **Diagnostic:**
    *   `ss -tulpn | grep :<PORT>`
    *   `netstat -anp | grep <PORT>`
*   **Resolution:**
    *   Identify the conflicting process. If it is a previous instance of the service that failed to shut down, kill it.
    *   If it is a different service entirely, the agent must notify the human operator and request a port change or a conflict resolution.

**3. Out of Memory (OOM) Killer Activity**
*   **Symptom:** The service starts but crashes shortly after, or the system becomes unresponsive.
*   **Diagnostic:**
    *   `dmesg | grep -i "oom-killer"`
    *   `journalctl -k | grep -i "out of memory"`
*   **Resolution:**
    *   Check if the model size exceeds available RAM.
    *   Check for memory leaks in the application logs.
    *   If the system is under heavy load, the agent should suggest clearing non-essential processes or increasing swap space (requires human approval).

**4. Disk Space Exhaustion**
*   **Symptom:** SQLite fails to write, or the application throws "No space left on device".
*   **Diagnostic:**
    *   `df -h`
    *   `du -sh /models/flight-recorder/data`
*   **Resolution:**
    *   Identify large log files in `/models/flight-recorder/app/logs`.
    *   **[APPROVAL REQUIRED]** If logs are the cause, the agent may propose truncating old logs (e.g., `> /models/flight-recorder/app/logs/old_error.log`).
    *   **[APPROVAL REQUIRED]** If the data directory is the cause, the agent must NOT delete data but may suggest a data archival strategy to the human operator.

**5. Dependency Version Mismatch**
*   **Symptom:** The application fails to start with `ImportError` or `ModuleNotFoundError`.
*   **Diagnostic:**
    *   `pip check`
    *   `pip list`
*   **Resolution:**
    *   Compare the current `pip list` against the `requirements.txt` file.
    *   Re-run the `pip install -r requirements.txt` command within the virtual environment.
    *   If a specific package version is causing a conflict, the agent should report the specific conflict to the human.

# Advanced Network & Connectivity Diagnostics

If the service is running but unreachable, the agent will perform the following deep-dive diagnostics.

**1. Layer 3/4 Connectivity**
*   `ping -c 4 192.168.1.116` (Verify basic ICMP reachability).
*   `nc -zv 192.168.1.116 <PORT>` (Verify that the TCP port is open and accepting connections).
*   `traceroute 192.168.1.116` (Identify any routing hops causing latency or drops).

**2. Layer 7 (Application) Diagnostics**
*   `curl -v http://192.168.1.116:<PORT>/health` (Verbose output to see headers, TLS handshake, and response codes).
*   `curl -I http://192.168.1.116:<PORT>/health` (Check only the HTTP headers to verify status codes like 200 OK).
*   `time curl -o /dev/null -s http://192.168.1.116:<PORT>/health` (Measure the response time to detect performance degradation).

**3. Packet Capture (If persistent issues occur)**
*   **[APPROVAL REQUIRED]** `tcpdump -i any port <PORT> -w capture_file.pcap`
*   The agent will analyze the `capture_file.pcap` (if the environment allows) to look for TCP Retransmissions, Connection Refused, or Reset (RST) packets.

# Environment & Dependency Audit

To ensure the environment is perfectly replicated, the agent will perform a comprehensive audit.

**1. Python Environment Audit**
*   `python3 -c "import sys; print(sys.version)"` (Verify Python version).
*   `python3 -c "import sqlite3; print(sqlite3.sqlite_version)"` (Verify SQLite library version).
*   `pip freeze > current_env_snapshot.txt` (Capture the exact state of all installed packages).

**2. System Library Audit**
*   `ldd /models/flight-recorder/app/venv/bin/python3` (Check for missing shared libraries).
*   `ldd /models/flight-recorder/app/venv/bin/python3 | grep "not found"` (Quick check for broken links).

**3. Environment Variable Verification**
*   `printenv | grep -E "DB|MODEL|PATH|PORT"` (Check for relevant variables without exposing secrets).
*   Verify that `PYTHONPATH` is correctly set to include the application directory.

# Logging & Telemetry Capture

The agent must maintain a detailed record of every action taken during the remediation.

**1. Session Recording**
*   The agent will use the `script` command to record the entire terminal session:
    *   `script -a remediation_session_$(date +%Y%m%d).log`
*   This ensures that every command and its output are preserved for post-mortem analysis.

**2. Real-time Log Streaming**
*   While the service is restarting, the agent will stream logs to a file:
    *   `tail -f /models/flight-recorder/app/logs/error.log | tee remediation_tail.log`
*   This allows the agent to capture transient errors that might disappear from the main log file.

**3. Journald Monitoring**
*   `journalctl -u flight-recorder -f` (Follow systemd logs in real-time to catch startup failures).

# Post-Remediation Validation Suite

Once the service is redeployed and the benchmark is restarted, the agent must perform a multi-stage validation.

**1. Smoke Test (Immediate)**
*   Verify the service is up: `curl -I http://192.168.1.116:<PORT>/health` (Expected: 200 OK).
*   Verify the database is writable: `sqlite3 /models/flight-recorder/data/database.db "INSERT INTO test_table (val) VALUES ('test'); SELECT * FROM test_table; DELETE FROM test_table;"` (Note: Use a dummy table or a non-critical row).

**2. Load Test (Short Duration)**
*   Execute a small burst of requests to ensure the service doesn't crash under concurrent load.
*   `ab -n 100 -c 10 http://192.168.1.116:<PORT>/health` (Using Apache Benchmark tool if available).

**3. Data Integrity Verification**
*   Compare the row count of the `benchmark_results` table before and after the restart.
*   `sqlite3 /models/flight-recorder/data/database.db "SELECT count(*) FROM benchmark_results;"`
*   Verify that the timestamps are monotonically increasing.

# Reporting & Documentation

Upon completion, the agent will generate a "Remediation Summary Report" for the human operator.

**Report Template:**
*   **Status:** [SUCCESS / FAILURE / PARTIAL_SUCCESS]
*   **Timestamp:** [ISO 8601 Timestamp]
*   **Initial Observations:** [Summary of logs, resource usage, and errors found]
*   **Actions Taken:**
    *   [Action 1: e.g., "Stopped service and cleared __pycache__"]
    *   [Action 2: e.g., "Reinstalled requirements.txt"]
    *   [Action 3: e.g., "Restarted service and verified health"]
*   **Data Safety:** [Confirmed: No data was deleted. Backup created at /models/flight-recorder/data_backup_...]
*   **Benchmark Results:** [Summary of the restart benchmark, e.g., "Success, 300s duration, 500 records written"]
*   **Remaining Issues:** [Any unresolved warnings or minor issues]
*   **Logs:** [Path to the `remediation_session.log` and `remediation_tail.log`]

# Execution Log Template

The agent will fill out this log during the run to provide a clear audit trail.

| Step | Action | Command Executed | Output/Result | Status |
| :--- | :--- | :--- | :--- | :--- |
| 1.1 | Initial Inspection | `ls -la /models/flight-recorder/app` | [Paste Output] | [Pass/Fail] |
| 1.2 | Log Analysis | `tail -n 100 ...` | [Paste Output] | [Pass/Fail] |
| 2.1 | Integrity Check | `PRAGMA integrity_check;` | [Paste Output] | [Pass/Fail] |
| 2.2 | Backup Creation | `cp -r ...` | [Success/Fail] | [Pass/Fail] |
| 3.1 | Service Stop | `systemctl stop ...` | [Success/Fail] | [Pass/Fail] |
| 3.2 | Dependency Install| `pip install ...` | [Success/Fail] | [Pass/Fail] |
| 3.3 | Service Start | `systemctl start ...` | [Success/Fail] | [Pass/Fail] |
| 4.1 | Health Check | `curl -I ...` | [Success/Fail] | [Pass/Fail] |
| 4.2 | Benchmark Run | `python3 ...` | [Success/Fail] | [Pass/Fail] |
| 5.1 | Final Verification| `sqlite3 ...` | [Success/Fail] | [Pass/Fail] |

# SQLite Optimization & Maintenance

To ensure long-term reliability of the flight-recorder, the agent may suggest the following optimizations if performance issues are detected during the benchmark.

**1. WAL Mode Activation**
*   Write-Ahead Logging (WAL) significantly improves concurrency.
*   `sqlite3 /models/flight-recorder/data/database.db "PRAGMA journal_mode=WAL;"`

**2. Synchronous Setting**
*   Adjusting the `synchronous` pragma can balance safety and speed.
*   `sqlite3 /models/flight-recorder/data/database.db "PRAGMA synchronous=NORMAL;"`

**3. Cache Size Increase**
*   Increasing the page cache can improve read performance for large datasets.
*   `sqlite3 /models/flight-recorder/data/database.db "PRAGMA cache_size=-64000;"` (Approx 64MB)

**4. Database Vacuuming**
*   If the database has undergone many deletions, a `VACUUM` command can reclaim space and defragment the file.
*   **[APPROVAL REQUIRED]** `sqlite3 /models/flight-recorder/data/database.db "VACUUM;"`

**5. Index Analysis**
*   Ensure that indexes are being used efficiently.
*   `sqlite3 /models/flight-recorder/data/database.db "ANALYZE;"`

# Resource Monitoring & Profiling

If the benchmark fails due to performance bottlenecks, the agent will use these tools to profile the system.

**1. CPU Profiling**
*   `mpstat -P ALL 1 10` (Check for per-core CPU utilization and wait times).
*   `pidstat -c 1` (Check for context switching and CPU usage of the specific PID).

**2. Memory Profiling**
*   `vmstat 1 10` (Monitor paging, swapping, and memory usage).
*   `smem -t` (Check proportional set size (PSS) to see actual memory usage of the process).

**3. I/O Profiling**
*   `iostat -xz 1` (Check for disk utilization, wait times, and throughput).
*   `iotop` (Identify which process is consuming the most disk I/O).

**4. Network Profiling**
*   `nload` (Monitor real-time network bandwidth).
*   `iftop` (Monitor bandwidth usage by host/port).

# Security & Compliance Audit

The agent must ensure that the remediation process does not introduce security vulnerabilities.

**1. Permission Audit**
*   Verify that the data directory is not world-writable.
*   `ls -ld /models/flight-recorder/data` (Expected: `drwxr-x---` or `drwx------`).
*   **[APPROVAL REQUIRED]** `chmod 750 /models/flight-recorder/data` (If permissions are too loose).

**2. Log Sanitization**
*   Ensure that no passwords, API keys, or PII (Personally Identifiable Information) are being written to the application logs.
*   `grep -E "password|key|token|secret" /models/flight-recorder/app/logs/*.log`

**3. User Context Verification**
*   Ensure the service is running under a dedicated service account, not as `root`.
*   `ps -o user,pid,ppid,cmd -p <PID>`

# Final Summary of Safety Protocols

To reiterate the core safety constraints for this run:
1.  **Data Integrity is Paramount:** The `/models/flight-recorder/data` directory is a "No-Delete Zone."
2.  **Non-Destructive First:** All actions must be non-destructive unless explicitly approved by a human.
3.  **Auditability:** Every command must be logged via the `script` command and the Execution Log Template.
4.  **Stop on Error:** If `PRAGMA integrity_check` fails or if the disk becomes full, the agent must cease all automated actions and alert the human operator immediately.
5.  **Isolation:** Redeployment is limited to the `/models/flight-recorder/app` directory. No changes to system-wide configurations or other application directories are permitted.