# Mission
The primary objective of this remediation run is to restore the reliability of the benchmark execution on the remote model server (192.168.1.116) while ensuring the absolute integrity of the existing data. The "flight-recorder" service is critical; it serves as the primary telemetry and logging mechanism for model performance. Any loss of data within the `/models/flight-recorder/data` directory is considered a failure of the mission.

The agent must transition the system from its current "unreliable" or "stalled" state back to a "steady-state" where benchmarks can be executed, recorded, and retrieved without interruption. This involves a systematic progression from passive observation to active service restoration, governed by strict safety protocols.

**Key Objectives:**
1.  **Identify the Root Cause:** Determine if the failure is due to application logic errors, resource exhaustion (CPU/RAM/Disk), database locking, or network instability.
2.  **Verify Data Integrity:** Ensure the SQLite database is not corrupted and that the "flight-recorder" data remains consistent.
3.  **Restore Service:** Redeploy the application code from `/models/flight-recorder/app` to a known-good state.
4.  **Resume Benchmarking:** Verify that the benchmark suite can successfully communicate with the service and write new records.

---

# Read-Only Inspection
Before any changes are made, the agent must perform a comprehensive "look-but-don't-touch" audit of the environment. This phase is designed to gather intelligence without altering the state of the system.

**1. File System Audit:**
Examine the directory structure to ensure no unexpected files have appeared and that the application code matches the expected version.
- `ls -alR /models/flight-recorder/app`
- `ls -alR /models/flight-recorder/data`
- `du -sh /models/flight-recorder/data` (Check for unexpected growth in specific sub-directories)

**2. Log Analysis:**
Search for "Error", "Critical", "Timeout", or "SQLite" in the application logs.
- `tail -n 500 /models/flight-recorder/app/logs/*.log` (Adjust path as necessary based on discovery)
- `grep -iE "error|critical|timeout|sqlite|locked" /models/flight-recorder/app/logs/*`

**3. Database Schema Inspection:**
Use the SQLite3 CLI to inspect the table structures without performing any writes.
- `sqlite3 /models/flight-recorder/data/flight_recorder.db ".tables"`
- `sqlite3 /models/flight-recorder/data/flight_recorder.db ".schema"`
- `sqlite3 /models/flight-recorder/data/flight_recorder.db "SELECT count(*) FROM benchmarks;"` (Verify record counts)

**4. Network State:**
Check the current connections to the service.
- `netstat -tunlp | grep 192.168.1.116`
- `ss -ant | grep :<PORT_NUMBER>` (Replace with the actual service port)

**Risk Assessment:** This phase is low risk. No commands here should modify the database or the application files.

---

# Health Checks
Once the environment is understood, the agent must verify the "vital signs" of the server.

**1. Resource Utilization:**
Check if the server is being throttled or exhausted.
- `top -b -n 1 | head -n 20` (Check CPU/Memory)
- `df -h` (Check if the disk is full, which often causes SQLite write failures)
- `free -m` (Check available memory)

**2. Service Connectivity:**
Verify if the application is responding to basic requests.
- `curl -I http://192.168.1.116:<PORT>/health` (If a health endpoint exists)
- `ping -c 4 192.168.1.116`

**3. Process Status:**
Identify the PID of the flight-recorder application.
- `ps aux | grep flight-recorder`
- Check if the process is in a "Zombie" state or "Uninterruptible Sleep" (D state).

**Expected Outcome:** The agent should identify whether the service is "Down," "Slow," or "Zombie." If the disk is 100% full, the remediation must prioritize clearing non-essential logs before proceeding.

---

# Data Safety Checks
This is the most critical phase. Since the goal is "no data loss," we must verify the SQLite database's health.

**1. SQLite Integrity Check:**
Run the internal SQLite integrity check. This is a read-only operation that verifies the internal structure of the B-trees and pages.
- `sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA integrity_check;"`
- *Success Criteria:* The output must be "ok". If it returns errors, the database is corrupted.

**2. Lock Analysis:**
Check if the database is locked by a hung process.
- `lsof /models/flight-recorder/data/flight_recorder.db`
- If a process ID is found that is not the current active service, it may be holding a stale lock.

**3. WAL File Inspection:**
Check for Write-Ahead Logging (WAL) files.
- `ls -l /models/flight-recorder/data/*.db-wal`
- `ls -l /models/flight-recorder/data/*.db-shm`
- If these files are very large, it may indicate that a checkpoint hasn't occurred.

**4. Backup Verification (If applicable):**
If a backup directory exists, verify the timestamp of the last successful backup.
- `ls -lt /models/flight-recorder/backups/`

**Stop Condition:** If `PRAGMA integrity_check` returns anything other than "ok", the agent must **STOP** and request human intervention for a manual database recovery.

---

# Redeploy Procedure
If the health checks indicate a hung process or the logs show application-level crashes, a redeploy is required.

**1. Graceful Shutdown:**
Attempt to stop the service gracefully to allow the SQLite connection to close.
- `systemctl stop flight-recorder` (or the relevant service command)
- If `systemctl` is unavailable: `kill -15 <PID>`

**2. Verification of Shutdown:**
Ensure no processes are still holding the database file.
- `lsof /models/flight-recorder/data/flight_recorder.db` (Should return no results)

**3. Environment Cleanup:**
Clear temporary files or cache (do not touch the data directory).
- `rm -rf /models/flight-recorder/app/.cache/*` (Example only)

**4. Application Restart:**
Restart the service.
- `systemctl start flight-recorder`
- Or, if manual: `python3 /models/flight-recorder/app/main.py &` (Adjust based on actual entry point)

**5. Post-Redeploy Validation:**
Check the logs immediately after startup.
- `tail -f /models/flight-recorder/app/logs/startup.log`
- Verify the service is listening on the correct port.

**Human Approval Gate:**
- **[GATE 1]** Required before executing `kill -9` on any process.
- **[GATE 2]** Required before restarting the service if the service is currently processing a high-priority benchmark.

---

# Benchmark Restart Procedure
Once the service is confirmed healthy, the benchmark suite must be resumed.

**1. State Identification:**
Query the database to find the last successfully completed benchmark ID.
- `sqlite3 /models/flight-recorder/data/flight_recorder.db "SELECT id, status FROM benchmarks ORDER BY id DESC LIMIT 1;"`

**2. Resume Logic:**
Determine the starting point for the next benchmark.
- If the last status was "Completed", start the next sequential ID.
- If the last status was "In Progress" or "Failed", the agent must decide whether to retry the same ID or skip it.

**3. Execution:**
Trigger the benchmark script.
- `python3 /models/flight-recorder/benchmarks/run_suite.py --resume-from <ID>`

**4. Monitoring:**
Monitor the first 3 minutes of the benchmark for "Write Success" in the database.
- `watch -n 5 "sqlite3 /models/flight-recorder/data/flight_recorder.db 'SELECT count(*) FROM benchmarks WHERE status=\"In Progress\";'"`

---

# Human Approval Gates
The agent must pause and request explicit human confirmation at the following points:

1.  **Destructive Action Warning:** Before any command that uses `rm`, `mv`, or `truncate` on any file within the `/models/flight-recorder/` tree (except for temporary cache files).
2.  **Database Modification:** Before any `UPDATE`, `DELETE`, or `DROP` commands on the SQLite database.
3.  **Service Interruption:** Before stopping the service if the agent detects an active benchmark is currently running (to prevent partial data writes).
4.  **Data Directory Modification:** Any action that involves moving or renaming files inside `/models/flight-recorder/data`.

**Format for Requesting Approval:**
> "I am requesting approval to [ACTION] on [PATH/RESOURCE].
> Reason: [EXPLANATION].
> Risk: [LOW/MEDIUM/HIGH].
> Expected Result: [RESULT]."

---

# Stop Conditions
The agent must cease all automated actions and alert a human operator if any of the following occur:

1.  **Database Corruption:** `PRAGMA integrity_check` returns an error.
2.  **Disk Failure:** The filesystem becomes read-only or `df -h` shows 100% utilization and cannot be cleared.
3.  **Persistent Crash Loop:** The application fails to start more than 3 times consecutively after a redeploy.
4.  **Unauthorized Changes:** The agent detects a change in the file hash of the core application code that was not part of the planned redeploy.
5.  **Network Isolation:** The agent loses the ability to communicate with 192.168.1.116.

---

# Recovery Commands
*Note: These are grouped by phase. Only execute the command relevant to the current phase.*

**Inspection Phase:**
```bash
# List all files in data directory
ls -R /models/flight-recorder/data
# Check SQLite tables
sqlite3 /models/flight-recorder/data/flight_recorder.db ".tables"
# Check logs for errors
grep -i "error" /models/flight-recorder/app/logs/*
```

**Health Phase:**
```bash
# Check disk space
df -h
# Check memory
free -m
# Check process list
ps aux | grep flight-recorder
```

**Data Safety Phase:**
```bash
# Integrity check (CRITICAL)
sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA integrity_check;"
# Check for file locks
lsof /models/flight-recorder/data/flight_recorder.db
```

**Redeploy Phase:**
```bash
# Stop service (Graceful)
systemctl stop flight-recorder
# Force kill if hung (REQUIRES APPROVAL)
# kill -9 <PID>
# Start service
systemctl start flight-recorder
```

**Benchmark Phase:**
```bash
# Get last benchmark ID
sqlite3 /models/flight-recorder/data/flight_recorder.db "SELECT id FROM benchmarks ORDER BY id DESC LIMIT 1;"
# Run benchmark
python3 /models/flight-recorder/benchmarks/run_suite.py --resume-from <ID>
```

---

# Summary
This remediation plan follows a "Safety-First" methodology. By prioritizing read-only inspection and integrity checks, we ensure that the "flight-recorder" data remains the source of truth. The plan moves from observation to intervention only when necessary, with clear gates for human oversight.

**Workflow Summary:**
1.  **Observe:** Gather logs, check disk/memory, and inspect SQLite schema.
2.  **Verify:** Run `integrity_check` to ensure the database is healthy.
3.  **Remediate:** If the service is down or hung, perform a graceful restart.
4.  **Resume:** Identify the last successful benchmark and restart the suite.
5.  **Monitor:** Ensure new data is being written correctly.

**Strict Prohibition:** At no point during this run shall the agent delete the `/models/flight-recorder/data` directory or any files within it without explicit, multi-step human approval.

# Detailed Technical Appendix and Contingency Planning

This section provides the granular technical details required for the agent to handle complex edge cases that may arise during the remediation of the flight-recorder service. It serves as a deep-dive reference for the "Read-Only Inspection" and "Data Safety" phases.

## Advanced SQLite Diagnostics and Concurrency Management
Since the service uses SQLite, understanding its locking mechanisms is paramount to resolving "busy" or "locked" errors without risking data corruption.

**1. Journal Mode Analysis:**
The agent should identify the current journal mode. SQLite can operate in `DELETE`, `TRUNCATE`, `PERSIST`, `MEMORY`, or `WAL` (Write-Ahead Logging) modes.
- `sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA journal_mode;"`
- *Context:* If the mode is `WAL`, the agent should look for `-wal` and `-shm` files. If these files are present but the database is inaccessible, it may indicate a crash during a checkpoint.

**2. Synchronous Settings:**
Check the `synchronous` pragma to understand how the database handles disk writes.
- `sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA synchronous;"`
- *Context:* A value of `0` is fast but risky; `1` is standard; `2` is the safest (full ACID compliance).

**3. Handling `SQLITE_BUSY`:**
If the agent detects that the database is locked, it should check the `busy_timeout`.
- `sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA busy_timeout;"`
- *Action:* If the timeout is set to 0, the application will fail immediately upon encountering a lock. The agent should recommend increasing this to 5000ms (5 seconds) during the redeploy phase to allow for transient locks to clear.

**4. Vacuuming and Maintenance:**
If the database has undergone many deletions or updates, it may be fragmented.
- *Note:* `VACUUM` is a heavy operation and should **only** be performed after human approval, as it locks the entire database.
- `sqlite3 /models/flight-recorder/data/flight_recorder.db "VACUUM;"` (REQUIRES APPROVAL)

## Network Layer Analysis and Troubleshooting
If the service is reachable but performing poorly, the agent must investigate the network path between the benchmark runner and the model server (192.168.1.116).

**1. Path Latency and Packet Loss:**
Use `mtr` (My Traceroute) to identify where in the network path latency spikes occur.
- `mtr -rw 192.168.1.116` (Run for at least 50 packets)

**2. TCP Connection State:**
Analyze the state of the TCP stack to see if connections are hanging in `TIME_WAIT` or `CLOSE_WAIT`.
- `ss -ant | grep 192.168.1.116`
- *Analysis:* A high number of `CLOSE_WAIT` connections suggests the application is not properly closing its sockets.

**3. Packet Capture (Read-Only):**
If the application is dropping packets, a brief capture can identify the reason (e.g., TCP Retransmissions).
- `tcpdump -i any host 192.168.1.116 -w /tmp/capture.pcap` (Run for 30 seconds)
- *Note:* The agent must ensure the capture does not exceed disk limits.

## Application Environment and Dependency Audit
A "stalled" service is often caused by an inconsistent environment (e.g., a library update that broke a dependency).

**1. Python Environment Check:**
Verify the Python version and the path of the interpreter being used.
- `which python3`
- `python3 -c "import sys; print(sys.version); print(sys.path)"`

**2. Dependency Integrity:**
Check for missing or mismatched packages in the flight-recorder app.
- `pip list` (or `pip3 list`)
- Compare the output against a known-good `requirements.txt` or `environment.yml` file if available in `/models/flight-recorder/app`.

**3. Shared Library Links:**
If the application uses C-extensions (common in ML/Model servers), check for broken shared library links.
- `ldd /models/flight-recorder/app/main.py` (or the primary executable)

## Benchmark State Machine Logic
To ensure "no data loss" and "reliable execution," the agent must follow a strict state machine when resuming benchmarks.

**State Definitions:**
- **PENDING:** Benchmark is queued but has not started.
- **IN_PROGRESS:** Benchmark is currently being executed by the model server.
- **COMPLETED:** Benchmark finished successfully; data is committed to SQLite.
- **FAILED_RETRY:** Benchmark failed due to a transient error (e.g., Timeout, Network Glitch).
- **FAILED_FATAL:** Benchmark failed due to a non-recoverable error (e.g., Model Error, Out of Memory).

**Agent Logic for Resumption:**
1.  **Scan:** Query the database for all records where `status` is NOT `COMPLETED`.
2.  **Prioritize:** Sort by `id` ascending.
3.  **Evaluate `IN_PROGRESS`:**
    - If a record is `IN_PROGRESS` for more than 30 minutes (or a predefined threshold), mark it as `FAILED_RETRY` and log a warning.
4.  **Evaluate `FAILED_RETRY`:**
    - Attempt to restart the benchmark. If it fails 3 consecutive times, move to `FAILED_FATAL`.
5.  **Evaluate `FAILED_FATAL`:**
    - Do not automatically retry. Flag for human review.

## Rollback and Contingency Procedures
If the redeploy procedure fails or the service remains unstable, the agent must be able to revert to the previous state.

**1. Application Snapshotting:**
Before any redeploy, the agent should create a timestamped backup of the application directory.
- `cp -r /models/flight-recorder/app /models/flight-recorder/app_backup_$(date +%Y%m%d_%H%M%S)`

**2. Atomic Symlink Switching (If applicable):**
If the server uses a symlink for the app directory (e.g., `/models/flight-recorder/app` -> `/models/flight-recorder/versions/v1.2`), the agent should use this for instant rollbacks.
- `ln -sfn /models/flight-recorder/versions/v1.1 /models/flight-recorder/app`

**3. Database Point-in-Time Recovery (PITR):**
If the database is corrupted during a redeploy, the agent must stop and alert the human.
- *Action:* Do not attempt to "fix" a corrupted SQLite file with automated scripts.

## Log Management and Rotation Strategy
To prevent the "Disk Full" condition from recurring, the agent should inspect the current log rotation policy.

**1. Log Size Audit:**
- `du -sh /models/flight-recorder/app/logs/*`

**2. Rotation Configuration:**
Check if `logrotate` is configured for the flight-recorder logs.
- `ls -l /etc/logrotate.d/`
- If no configuration exists, the agent should propose a `logrotate` configuration that keeps 7 days of logs and rotates at 100MB.

**3. Log Level Adjustment:**
If logs are growing too fast, the agent may suggest changing the log level from `DEBUG` to `INFO` or `WARNING`.
- *Action:* This requires human approval as it may hide the root cause of the original failure.

## Post-Remediation Verification Checklist
After the service is restored and the benchmark is resumed, the agent must verify the following 10 points before declaring the mission "Success":

1.  [ ] **Service Status:** Is the process running and healthy?
2.  [ ] **Port Binding:** Is the service listening on the expected port?
3.  [ ] **Log Flow:** Are new logs appearing in the log files?
4.  [ ] **Disk Space:** Is there at least 20% free space on the partition?
5.  [ ] **SQLite Write:** Can a dummy record be written and read?
6.  [ ] **Benchmark Progress:** Is the benchmark suite moving through the queue?
7.  [ ] **Latency Check:** Is the response time within acceptable bounds?
8.  [ ] **Memory Stability:** Is the memory usage plateauing or leaking?
9.  [ ] **CPU Stability:** Is the CPU usage consistent with expected benchmark load?
10. [ ] **Data Integrity:** Does the count of `COMPLETED` benchmarks match the expected count?

## Incident Reporting Template
Upon completion of the remediation, the agent should generate a report using the following structure:

**Remediation Report: Flight-Recorder Service**
- **Timestamp:** [Current Time]
- **Initial State:** [e.g., Service Down, Database Locked]
- **Root Cause Identified:** [e.g., Disk Full, SQLite Lock, Python Dependency Mismatch]
- **Actions Taken:**
    - [Action 1: e.g., Cleared /tmp/ logs]
    - [Action 2: e.g., Restarted service]
    - [Action 3: e.g., Resumed benchmark from ID 450]
- **Data Integrity Status:** [Verified / Corrupted / Partial Loss]
- **Human Intervention Required:** [Yes/No - Describe]
- **Final Status:** [Success / Partial Success / Failed]

---

# Recovery Commands (Expanded)

**Advanced Inspection Commands:**
```bash
# Check SQLite journal mode and synchronous settings
sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA journal_mode;"
sqlite3 /models/flight-recorder/data/flight_recorder.db "PRAGMA synchronous;"

# Check for specific SQLite error codes in logs
grep -i "SQLITE_BUSY" /models/flight-recorder/app/logs/*
grep -i "SQLITE_CORRUPT" /models/flight-recorder/app/logs/*

# Check for open file descriptors (to see if the app is leaking handles)
ls /proc/<PID>/fd | wc -l
```

**Network Diagnostic Commands:**
```bash
# Check for TCP retransmissions (requires sudo/root)
# sudo netstat -s | grep retransmitted
# Check for specific port status
ss -ant | grep :<PORT>
```

**Redeploy & Rollback Commands:**
```bash
# Create a backup of the app directory before redeploy
cp -r /models/flight-recorder/app /models/flight-recorder/app_backup_$(date +%Y%m%d)

# Check for Python environment issues
python3 -m pip check
# Check for missing shared libraries
ldd /models/flight-recorder/app/main.py
```

**Benchmark State Management Commands:**
```bash
# Find all non-completed benchmarks
sqlite3 /models/flight-recorder/data/flight_recorder.db "SELECT id, status FROM benchmarks WHERE status != 'COMPLETED' ORDER BY id ASC;"

# Update a stuck benchmark to FAILED_RETRY
sqlite3 /models/flight-recorder/data/flight_recorder.db "UPDATE benchmarks SET status='FAILED_RETRY' WHERE id=<ID> AND status='IN_PROGRESS';"
```

# Final Safety Protocol
The agent is reminded that the primary directive is **Data Preservation**. 
- If at any point a command is requested that would result in the deletion of a `.db`, `.db-wal`, or `.db-shm` file, the agent must halt and request a "High-Risk Action" approval.
- If the agent detects that the `flight_recorder.db` file size is decreasing unexpectedly, it must immediately stop all write operations and alert the human operator.
- The agent shall not attempt to "repair" the database using external tools (like `sqlite3_analyzer` or `dump`) without explicit human oversight, as these tools can occasionally alter the file structure in ways that are incompatible with the application's specific driver version.