# Mission  
**Objective:** Restore the remote model server at **192.168.1.116** to a state where the benchmark can run reliably and repeatedly, without any loss of existing data or corruption of the SQLite database. The plan must be **non‑destructive** (no deletion of `/models/flight-recorder/data` or any other data directory) and must include explicit human approval gates for any action that could potentially alter the system state beyond read‑only inspection.

**Scope of Work:**
- Verify the health of the host machine and its services.
- Inspect the current application code and configuration.
- Validate the integrity of the SQLite database.
- Redeploy the application code if necessary, ensuring that the deployment process does not overwrite or delete any data.
- Restart the benchmark process in a controlled manner.
- Provide recovery commands in case of failure.
- Document all steps, decisions, and required approvals.

---

# Read‑Only Inspection  
The first phase is to gather information without making any changes. All commands below are **read‑only** and can be executed without approval.

| Step | Command | Purpose | Notes |
|------|---------|---------|-------|
| 1 | `ssh user@192.168.1.116` | Open a shell on the remote host | Replace `user` with the actual SSH user. |
| 2 | `cd /models/flight-recorder` | Navigate to the project root | |
| 3 | `ls -alh` | List all files and directories with details | |
| 4 | `cat app/README.md` | Inspect the README for deployment notes | |
| 5 | `cat app/requirements.txt` | Verify Python dependencies | |
| 6 | `cat app/config.yaml` | Inspect configuration file | |
| 7 | `cat app/main.py | head -n 20` | Peek at the main application code | |
| 8 | `cat data/metadata.json | head -n 20` | Inspect data metadata | |
| 9 | `ls -alh data/` | Confirm presence of data files | |
| 10 | `sqlite3 data/flight_recorder.db ".schema"` | Show database schema | |
| 11 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights;"` | Count rows in the main table | |
| 12 | `df -h` | Check disk usage | |
| 13 | `free -h` | Check memory usage | |
| 14 | `top -b -n 1 | head -n 20` | Quick CPU usage snapshot | |
| 15 | `systemctl status flight-recorder.service` | Check if the service is running | |
| 16 | `journalctl -u flight-recorder.service --since "2024-01-01" | head -n 20` | Inspect recent logs | |
| 17 | `ps aux | grep -i flight-recorder` | Verify running processes | |
| 18 | `netstat -tulnp | grep 8000` | Confirm the service port is listening | |
| 19 | `uptime` | Check system uptime | |
| 20 | `hostname` | Confirm host identity | |

**Human Approval Gates** – None of the above commands require approval. They are purely observational.

---

# Health Checks  
These checks confirm that the host machine and its services are in a healthy state. If any check fails, the plan will include remedial actions that require explicit approval.

| Check | Command | Expected Result | Approval Required? |
|-------|---------|------------------|--------------------|
| **CPU Load** | `uptime` | Load averages < 1.0 for 1‑5‑15 min | No |
| **Memory Usage** | `free -h` | Swap usage < 10% | No |
| **Disk Space** | `df -h` | Free space > 10% on `/` and `/models` | No |
| **Service Status** | `systemctl is-active flight-recorder.service` | `active` | No |
| **Service Logs** | `journalctl -u flight-recorder.service --since "2024-01-01" | grep -i error` | No errors | No |
| **Network Connectivity** | `ping -c 4 8.8.8.8` | All packets received | No |
| **Port Listening** | `ss -tuln | grep 8000` | `LISTEN` | No |
| **SQLite DB Integrity** | `sqlite3 data/flight_recorder.db "PRAGMA integrity_check;"` | `ok` | No |
| **SQLite DB Size** | `du -sh data/flight_recorder.db` | < 500 MB | No |
| **Python Version** | `python3 --version` | `Python 3.10.x` or newer | No |
| **Virtualenv Activation** | `source app/venv/bin/activate` | `app-venv` prompt | No |
| **Dependency Check** | `pip list` | All dependencies from `requirements.txt` present | No |

If any of the above checks fail, the plan will include remedial actions that require explicit human approval. For example, if the SQLite integrity check fails, we will need to restore from backup, which requires approval.

---

# Data Safety Checks  
The goal is to confirm that the data directory and the SQLite database are intact and that no data loss will occur during redeployment or benchmark restart.

| Step | Command | Purpose | Approval Required? |
|------|---------|---------|--------------------|
| 1 | `tar -cf - data/ | md5sum` | Compute MD5 of entire data directory | No |
| 2 | `sqlite3 data/flight_recorder.db "PRAGMA quick_check;"` | Quick integrity check | No |
| 3 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights;"` | Verify row count | No |
| 4 | `sqlite3 data/flight_recorder.db "SELECT MAX(id) FROM flights;"` | Verify highest ID | No |
| 5 | `sqlite3 data/flight_recorder.db "SELECT MIN(id) FROM flights;"` | Verify lowest ID | No |
| 6 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE status='completed';"` | Verify status distribution | No |
| 7 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE status='failed';"` | Verify status distribution | No |
| 8 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE status='pending';"` | Verify status distribution | No |
| 9 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE status IS NULL;"` | Verify no null status | No |
| 10 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id IS NULL;"` | Verify no null IDs | No |
| 11 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id NOT IN (SELECT id FROM flights);" ` | Verify no orphan rows | No |
| 12 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id > 0;"` | Verify positive IDs | No |
| 13 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id < 0;"` | Verify no negative IDs | No |
| 14 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id = 0;"` | Verify no zero IDs | No |
| 15 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id BETWEEN 1 AND 1000000;"` | Verify ID range | No |
| 16 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id > 1000000;"` | Verify no IDs beyond expected | No |
| 17 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id < 1;"` | Verify no IDs below 1 | No |
| 18 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id IS NOT NULL;"` | Verify no null IDs | No |
| 19 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id IS NULL;"` | Verify no null IDs | No |
| 20 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights WHERE id = 0;"` | Verify no zero IDs | No |

**Backup Plan** – If any of the above checks indicate corruption or missing data, we will create a backup of the database and data directory. The backup commands are labeled as requiring approval:

| Command | Approval Required? |
|---------|--------------------|
| `sudo cp -a data/flight_recorder.db /tmp/flight_recorder.db.bak` | **Yes** |
| `sudo tar -czf /tmp/flight_recorder_data.tar.gz -C /models/flight-recorder data/` | **Yes** |

These commands are **not** executed until explicit approval is obtained.

---

# Redeploy Procedure  
The redeployment must be performed in a way that **does not** overwrite or delete any existing data. The plan uses a **staged deployment** approach: first copy new code to a temporary directory, then switch the service to use the new code, and finally clean up the old code. All destructive actions (e.g., removing old code) are gated for approval.

## 1. Prepare Deployment Directory  
```bash
# Create a temporary deployment directory
mkdir -p /tmp/flight-recorder-deploy
```

## 2. Copy New Code  
```bash
# Copy the new application code to the temporary directory
rsync -avz --exclude='data/' app/ /tmp/flight-recorder-deploy/
```

**Note:** The `--exclude='data/'` flag ensures that the data directory is not touched.

## 3. Verify Code Integrity  
```bash
# Check that all required files are present
ls -alh /tmp/flight-recorder-deploy/app
```

## 4. Install Dependencies (in a virtual environment)  
```bash
# Create a new virtual environment in the temporary directory
python3 -m venv /tmp/flight-recorder-deploy/app/venv

# Activate the environment
source /tmp/flight-recorder-deploy/app/venv/bin/activate

# Install dependencies
pip install -r /tmp/flight-recorder-deploy/app/requirements.txt
```

## 5. Test the Application (Read‑Only)  
```bash
# Run a quick lint or syntax check
python3 -m py_compile /tmp/flight-recorder-deploy/app/main.py
```

## 6. Switch Service to New Code  
The service is assumed to be managed by `systemd`. We will create a new service file that points to the new code directory. This step requires approval because it modifies system configuration.

```bash
# Create a new service file
cat <<EOF | sudo tee /etc/systemd/system/flight-recorder-new.service
[Unit]
Description=Flight Recorder New Service
After=network.target

[Service]
Type=simple
WorkingDirectory=/tmp/flight-recorder-deploy/app
ExecStart=/tmp/flight-recorder-deploy/app/venv/bin/python3 /tmp/flight-recorder-deploy/app/main.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
```

**Approval Required:** The creation of `/etc/systemd/system/flight-recorder-new.service` modifies system configuration. **Human approval** is required before executing the above command.

## 7. Reload systemd and Enable New Service  
```bash
# Reload systemd daemon
sudo systemctl daemon-reload

# Disable old service
sudo systemctl stop flight-recorder.service
sudo systemctl disable flight-recorder.service

# Enable new service
sudo systemctl enable flight-recorder-new.service
sudo systemctl start flight-recorder-new.service
```

**Approval Required:** Stopping and disabling the old service, and enabling the new one, are system‑level changes. **Human approval** is required before executing these commands.

## 8. Verify New Service Running  
```bash
# Check status
systemctl status flight-recorder-new.service

# Check logs
journalctl -u flight-recorder-new.service --since "2024-01-01" | head -n 20
```

## 9. Clean Up Old Code (Optional)  
If the new service is confirmed to be running correctly, we can optionally remove the old code directory. This step is **destructive** and requires explicit approval.

```bash
# Remove old code directory
sudo rm -rf /models/flight-recorder/app
```

**Approval Required:** This command deletes the old code. **Human approval** is required before executing.

---

# Benchmark Restart Procedure  
Once the new service is running, we can restart the benchmark. The benchmark is assumed to be a separate script or process that interacts with the service. The restart procedure is split into **pre‑restart checks**, **restart**, and **post‑restart validation**.

## 1. Pre‑Restart Checks  
```bash
# Verify service is reachable
curl -s -o /dev/null http://192.168.1.116:8000/health | jq .
```

**Approval Required?** No – this is a read‑only check.

## 2. Stop Existing Benchmark (if any)  
```bash
# Find benchmark process
ps aux | grep -i benchmark | grep -v grep

# Stop it gracefully
sudo kill -TERM <pid>
```

**Approval Required?** The `kill` command modifies process state. **Human approval** is required before executing.

## 3. Start Benchmark  
Assuming the benchmark is a Python script `benchmark.py` located in `/models/flight-recorder/benchmark/`.

```bash
# Activate virtual environment
source app/venv/bin/activate

# Run benchmark
python3 /models/flight-recorder/benchmark/benchmark.py --config /models/flight-recorder/benchmark/config.yaml
```

**Approval Required?** No – this is a normal execution.

## 4. Monitor Benchmark Output  
```bash
# Tail the benchmark log
tail -f /models/flight-recorder/benchmark/benchmark.log
```

**Approval Required?** No.

## 5. Post‑Restart Validation  
```bash
# Verify benchmark produced expected output
grep "Benchmark completed" /models/flight-recorder/benchmark/benchmark.log

# Verify service still healthy
curl -s -o /dev/null http://192.168.1.116:8000/health | jq .
```

**Approval Required?** No.

---

# Human Approval Gates  
The following actions require explicit human approval before execution. Each command is labeled with **[APPROVAL]**.

| Action | Command | Approval Note |
|--------|---------|---------------|
| **Create new systemd service file** | `cat <<EOF | sudo tee /etc/systemd/system/flight-recorder-new.service ... EOF` | **[APPROVAL]** |
| **Stop old service** | `sudo systemctl stop flight-recorder.service` | **[APPROVAL]** |
| **Disable old service** | `sudo systemctl disable flight-recorder.service` | **[APPROVAL]** |
| **Enable new service** | `sudo systemctl enable flight-recorder-new.service` | **[APPROVAL]** |
| **Start new service** | `sudo systemctl start flight-recorder-new.service` | **[APPROVAL]** |
| **Delete old code directory** | `sudo rm -rf /models/flight-recorder/app` | **[APPROVAL]** |
| **Kill existing benchmark process** | `sudo kill -TERM <pid>` | **[APPROVAL]** |
| **Create database backup** | `sudo cp -a data/flight_recorder.db /tmp/flight_recorder.db.bak` | **[APPROVAL]** |
| **Create data backup** | `sudo tar -czf /tmp/flight_recorder_data.tar.gz -C /models/flight-recorder data/` | **[APPROVAL]** |

All other commands are read‑only or do not modify system state and do not require approval.

---

# Stop Conditions  
The remediation run should be halted immediately if any of the following conditions occur:

1. **SQLite Integrity Check Fails** – `PRAGMA integrity_check` returns anything other than `ok`.
2. **Disk Space < 10%** – `df -h` shows less than 10% free on `/` or `/models`.
3. **Service Crash** – `systemctl status flight-recorder.service` shows `failed` or `inactive`.
4. **Benchmark Crash** – `grep "ERROR" benchmark.log` shows any error.
5. **Unexpected Data Loss** – Row count in `flights` table drops by >5% compared to baseline.
6. **Network Failure** – `curl` to service health endpoint fails >3 times in a row.
7. **Unauthorized Process** – Any process not part of the known service or benchmark appears in `ps aux`.
8. **Manual Override** – Human operator explicitly requests to stop the run.

If any stop condition is met, the plan will execute the **Recovery Commands** section to restore the system to a safe state.

---

# Recovery Commands  
These commands are used to revert to the previous stable state if the run fails. They are labeled with **[RECOVERY]** and should be executed only after a thorough assessment.

| Step | Command | Purpose | Notes |
|------|---------|---------|-------|
| 1 | `sudo systemctl stop flight-recorder-new.service` | Stop new service | **[RECOVERY]** |
| 2 | `sudo systemctl disable flight-recorder-new.service` | Disable new service | **[RECOVERY]** |
| 3 | `sudo systemctl enable flight-recorder.service` | Re‑enable old service | **[RECOVERY]** |
| 4 | `sudo systemctl start flight-recorder.service` | Restart old service | **[RECOVERY]** |
| 5 | `sudo systemctl daemon-reload` | Reload systemd | **[RECOVERY]** |
| 6 | `sudo rm -rf /tmp/flight-recorder-deploy` | Clean temporary deployment | **[RECOVERY]** |
| 7 | `sudo rm -rf /etc/systemd/system/flight-recorder-new.service` | Remove new service file | **[RECOVERY]** |
| 8 | `sudo systemctl daemon-reload` | Reload systemd again | **[RECOVERY]** |
| 9 | `sudo systemctl restart flight-recorder.service` | Restart old service | **[RECOVERY]** |
| 10 | `sqlite3 data/flight_recorder.db "PRAGMA quick_check;"` | Quick check after recovery | **[RECOVERY]** |
| 11 | `sqlite3 data/flight_recorder.db "SELECT COUNT(*) FROM flights;"` | Verify row count after recovery | **[RECOVERY]** |
| 12 | `journalctl -u flight-recorder.service --since "2024-01-01" | grep -i error` | Inspect logs for errors | **[RECOVERY]** |
| 13 | `ps aux | grep -i flight-recorder` | Verify service process | **[RECOVERY]** |
| 14 | `curl -s -o /dev/null http://192.168.1.116:8000/health | jq .` | Verify service health | **[RECOVERY]** |

**Note:** All recovery commands are destructive to the new deployment state but preserve the data. They are executed only after the run is deemed failed.

---

# Summary  
1. **Mission** – Restore reliable benchmark execution on 192.168.1.116 without data loss.  
2. **Read‑Only Inspection** – Gather all system, code, and data information without modifying anything.  
3. **Health Checks** – Verify CPU, memory, disk, service status, logs, network, SQLite integrity, and Python environment.  
4. **Data Safety Checks** – Compute checksums, run SQLite integrity checks, and verify row counts.  
5. **Redeploy Procedure** – Use a staged deployment: copy new code to `/tmp`, install dependencies, create a new systemd service file, switch services, and optionally clean old code. All destructive actions are gated for approval.  
6. **Benchmark Restart Procedure** – Verify service health, stop any existing benchmark, start the new benchmark, monitor logs, and validate completion.  
7. **Human Approval Gates** – List all commands that require explicit approval.  
8. **Stop Conditions** – Define when to halt the run.  
9. **Recovery Commands** – Provide a step‑by‑step rollback to the previous stable state.  
10. **Summary** – Recap the entire plan, emphasizing safety, data integrity, and minimal disruption.

This plan ensures that the benchmark can be restarted reliably while preserving all existing data and requiring human oversight for any potentially destructive actions.

Finally, after the benchmark completes, a comprehensive audit should be performed. This audit includes verifying that all expected output files exist, cross‑checking the number of processed records against the database, and ensuring that the service logs contain no error entries. Once the audit passes, the system can be marked as fully restored, and the benchmark schedule can resume as normal.