# Mission  
**Objective:** Restore the flight‑recorder benchmark to a reliable, reproducible state on the remote model server at `192.168.1.116` without compromising any existing data.  
**Scope:**  
- Inspect the current deployment (app code, data, SQLite DB, logs).  
- Verify system health and resource availability.  
- Safely redeploy the application if necessary.  
- Restart the benchmark process and confirm successful execution.  
- Preserve all data; no destructive deletes are permitted.  

---

# Read‑Only Inspection  
The first step is to gather information about the current state of the server, the application, and the data. All commands below are **read‑only** and will not alter any files or processes.

| Step | Command | Purpose | Notes |
|------|---------|---------|-------|
| 1 | `ssh user@192.168.1.116 "uname -a"` | Identify OS and kernel | Replace `user` with the actual SSH user. |
| 2 | `ssh user@192.168.1.116 "ls -la /models/flight-recorder"` | Verify directory structure | |
| 3 | `ssh user@192.168.1.116 "ls -la /models/flight-recorder/app"` | List app files | |
| 4 | `ssh user@192.168.1.116 "ls -la /models/flight-recorder/data"` | List data files | |
| 5 | `ssh user@192.168.1.116 "du -sh /models/flight-recorder/data"` | Size of data directory | |
| 6 | `ssh user@192.168.1.116 "find /models/flight-recorder/data -type f -exec ls -l {} +"` | Detailed file list | |
| 7 | `ssh user@192.168.1.116 "sqlite3 /models/flight-recorder/data/flight.db \"PRAGMA integrity_check;\""` | Verify SQLite integrity | Expect `ok`. |
| 8 | `ssh user@192.168.1.116 "sqlite3 /models/flight-recorder/data/flight.db \"SELECT name FROM sqlite_master WHERE type='table';\""` | List tables | |
| 9 | `ssh user@192.168.1.116 "cat /models/flight-recorder/app/config.yaml"` | Inspect config | |
| 10 | `ssh user@192.168.1.116 "journalctl -u flight-recorder.service --no-pager -n 50"` | Recent service logs | |
| 11 | `ssh user@192.168.1.116 "ps aux | grep flight-recorder"` | Confirm running process | |
| 12 | `ssh user@192.168.1.116 "df -h"` | Disk usage | |
| 13 | `ssh user@192.168.1.116 "free -h"` | Memory usage | |
| 14 | `ssh user@192.168.1.116 "uptime"` | System load | |
| 15 | `ssh user@192.168.1.116 "netstat -tulnp | grep 8080"` | Verify listening port | |

**What to look for:**  
- No missing critical files.  
- SQLite integrity passes.  
- Service is running and listening on the expected port.  
- Disk and memory are within healthy limits.  

---

# Health Checks  
These checks confirm that the system can safely handle a redeploy and benchmark restart. They are also read‑only.

| Check | Command | Expected Result | Action if Failed |
|-------|---------|-----------------|------------------|
| CPU Load | `ssh user@192.168.1.116 "uptime"` | Load < 1.0 (or < 0.5 per core) | Wait or ask for approval to proceed |
| Memory | `ssh user@192.168.1.116 "free -h"` | Free > 30% | Wait or ask for approval |
| Disk | `ssh user@192.168.1.116 "df -h"` | Available > 10% | Wait or ask for approval |
| Service Status | `ssh user@192.168.1.116 "systemctl status flight-recorder.service"` | Active (running) | If inactive, attempt restart |
| Port Listening | `ssh user@192.168.1.116 "ss -tuln | grep :8080"` | LISTEN | If not listening, investigate |
| Log Errors | `ssh user@192.168.1.116 "journalctl -u flight-recorder.service --no-pager -n 200 | grep -i error"` | None | Investigate errors |

**If any of these checks fail, the agent should pause and request human approval before proceeding.**

---

# Data Safety Checks  
Ensuring data integrity and creating a backup before any destructive operation.

1. **Checksum Verification**  
   ```bash
   ssh user@192.168.1.116 "cd /models/flight-recorder/data && find . -type f -exec sha256sum {} + > checksums.txt"
   ```
   - Store `checksums.txt` locally for audit.

2. **SQLite Integrity**  
   ```bash
   ssh user@192.168.1.116 "sqlite3 /models/flight-recorder/data/flight.db \"PRAGMA integrity_check;\""
   ```
   - Expect `ok`. If not, abort and request human intervention.

3. **Backup Database**  
   ```bash
   ssh user@192.168.1.116 "cp /models/flight-recorder/data/flight.db /models/flight-recorder/data/flight.db.bak_$(date +%Y%m%d_%H%M%S)"
   ```
   - This creates a timestamped backup.  
   - **[APPROVAL REQUIRED]** before executing the backup to ensure no concurrent writes.

4. **Backup Data Directory**  
   ```bash
   ssh user@192.168.1.116 "tar -czf /tmp/flight-recorder-data_$(date +%Y%m%d_%H%M%S).tar.gz -C /models/flight-recorder data"
   ```
   - Store the archive in `/tmp` or transfer to a safe location.  
   - **[APPROVAL REQUIRED]** before executing.

5. **Verify Backup Integrity**  
   ```bash
   ssh user@192.168.1.116 "tar -tzf /tmp/flight-recorder-data_$(date +%Y%m%d_%H%M%S).tar.gz | wc -l"
   ```
   - Compare the file count with the original.  

**All backups are read‑only from the perspective of the agent; they are only copied, not modified.**

---

# Redeploy Procedure  
The redeploy is performed in a controlled, non‑destructive manner. Each step is annotated with whether human approval is required.

1. **Stop the Service**  
   ```bash
   ssh user@192.168.1.116 "sudo systemctl stop flight-recorder.service"
   ```
   - **[APPROVAL REQUIRED]** before stopping the service.

2. **Verify Service Stopped**  
   ```bash
   ssh user@192.168.1.116 "systemctl status flight-recorder.service"
   ```
   - Expect `inactive (dead)`.

3. **Backup Current App Code**  
   ```bash
   ssh user@192.168.1.116 "cp -r /models/flight-recorder/app /models/flight-recorder/app_backup_$(date +%Y%m%d_%H%M%S)"
   ```
   - **[APPROVAL REQUIRED]** before copying.

4. **Pull Latest Code (if using Git)**  
   ```bash
   ssh user@192.168.1.116 "cd /models/flight-recorder/app && git fetch && git reset --hard origin/main"
   ```
   - If not using Git, replace with appropriate copy/rsync commands.

5. **Install Dependencies**  
   ```bash
   ssh user@192.168.1.116 "cd /models/flight-recorder/app && pip install -r requirements.txt"
   ```
   - **[APPROVAL REQUIRED]** before installing.

6. **Run Migrations (if any)**  
   ```bash
   ssh user@192.168.1.116 "cd /models/flight-recorder/app && python manage.py migrate"
   ```
   - **[APPROVAL REQUIRED]** before running migrations.

7. **Start the Service**  
   ```bash
   ssh user@192.168.1.116 "sudo systemctl start flight-recorder.service"
   ```
   - **[APPROVAL REQUIRED]** before starting.

8. **Verify Service Running**  
   ```bash
   ssh user@192.168.1.116 "systemctl status flight-recorder.service"
   ```
   - Expect `active (running)`.

9. **Check Health Endpoint**  
   ```bash
   curl -s http://192.168.1.116:8080/health | jq .
   ```
   - Expect JSON with `"status":"ok"`.

10. **Log Review**  
    ```bash
    ssh user@192.168.1.116 "journalctl -u flight-recorder.service --no-pager -n 50"
    ```
    - Look for any startup errors.

---

# Benchmark Restart Procedure  
Once the application is confirmed healthy, the benchmark can be restarted.

1. **Stop Existing Benchmark Process**  
   ```bash
   ssh user@192.168.1.116 "pkill -f benchmark_script.py"
   ```
   - **[APPROVAL REQUIRED]** before killing.

2. **Verify No Benchmark Processes**  
   ```bash
   ssh user@192.168.1.116 "ps aux | grep benchmark_script.py | grep -v grep"
   ```
   - Expect no output.

3. **Start Benchmark**  
   ```bash
   ssh user@192.168.1.116 "nohup python /models/flight-recorder/benchmark/benchmark_script.py > /models/flight-recorder/benchmark/benchmark.log 2>&1 & echo $!"
   ```
   - **[APPROVAL REQUIRED]** before starting.

4. **Capture Benchmark PID**  
   ```bash
   ssh user@192.168.1.116 "pgrep -f benchmark_script.py"
   ```
   - Store PID for monitoring.

5. **Monitor Benchmark Output**  
   ```bash
   ssh user@192.168.1.116 "tail -f /models/flight-recorder/benchmark/benchmark.log"
   ```
   - Watch for completion or errors.

6. **Verify Benchmark Completion**  
   ```bash
   ssh user@192.168.1.116 "grep -i 'benchmark completed' /models/flight-recorder/benchmark/benchmark.log"
   ```
   - Expect a success line.

7. **Post‑Benchmark Health Check**  
   ```bash
   curl -s http://192.168.1.116:8080/metrics | jq .
   ```
   - Ensure metrics are updated.

---

# Human Approval Gates  
The following actions require explicit human approval before execution:

| Action | Command | Approval Note |
|--------|---------|---------------|
| Stop flight‑recorder service | `sudo systemctl stop flight-recorder.service` | Service downtime may affect users |
| Backup app code | `cp -r /models/flight-recorder/app /models/flight-recorder/app_backup_*` | Preserve current state |
| Pull latest code | `git fetch && git reset --hard origin/main` | Ensure code is correct |
| Install dependencies | `pip install -r requirements.txt` | May alter environment |
| Run migrations | `python manage.py migrate` | Database schema changes |
| Start flight‑recorder service | `sudo systemctl start flight-recorder.service` | Service availability |
| Kill benchmark process | `pkill -f benchmark_script.py` | Interrupts current run |
| Start benchmark | `nohup python benchmark_script.py &` | Initiates new run |

**If any of these steps fail or produce unexpected output, the agent must pause and request human intervention.**

---

# Stop Conditions  
The agent will abort the plan if any of the following conditions are met:

1. **High CPU Load** – `uptime` shows load > 1.5 (or > 0.75 per core).  
2. **Low Memory** – `free -h` shows free memory < 30%.  
3. **Disk Full** – `df -h` shows usage > 90%.  
4. **Service Unresponsive** – `systemctl status` shows `failed` or `inactive` after a restart attempt.  
5. **SQLite Integrity Failure** – `PRAGMA integrity_check` returns anything other than `ok`.  
6. **Backup Failure** – Any backup command returns non‑zero exit status.  
7. **Unexpected Errors in Logs** – `journalctl` shows repeated critical errors.  
8. **Human Denial** – Human explicitly denies an approval request.  

Upon hitting any stop condition, the agent will:

- Log the reason.  
- Notify the human operator.  
- Revert any partial changes if possible (e.g., restore backup).  

---

# Recovery Commands  
Prepared commands to recover from common failure scenarios.

| Scenario | Recovery Action | Command |
|----------|-----------------|---------|
| Service failed to start | Restart service | `sudo systemctl restart flight-recorder.service` |
| Service crashed mid‑run | Stop and restart | `sudo systemctl stop flight-recorder.service && sudo systemctl start flight-recorder.service` |
| Benchmark hung | Kill process | `pkill -f benchmark_script.py` |
| Data corruption detected | Restore backup | `cp /models/flight-recorder/data/flight.db.bak_* /models/flight-recorder/data/flight.db` |
| Disk full | Free space | `sudo rm -rf /tmp/*` (only after confirming no critical temp files) |
| High CPU | Kill heavy process | `pkill -f heavy_process_name` |
| Network unreachable | Re‑establish network | `sudo systemctl restart networking` |
| Unexpected log errors | View full logs | `journalctl -u flight-recorder.service -b` |
| Backup failed | Re‑attempt backup | `cp /models/flight-recorder/data/flight.db /models/flight-recorder/data/flight.db.bak_$(date +%Y%m%d_%H%M%S)` |

**All recovery commands are to be executed only after human approval if they affect running services.**

---

# Summary  
1. **Read‑Only Inspection** confirms the current state of the server, app, data, and database.  
2. **Health Checks** ensure the system can safely handle redeploy and benchmark restarts.  
3. **Data Safety Checks** create timestamped backups and verify integrity, protecting against data loss.  
4. **Redeploy Procedure** safely stops the service, backs up code, pulls the latest changes, installs dependencies, runs migrations, and restarts the service, all under human approval.  
5. **Benchmark Restart Procedure** stops any existing benchmark, starts a fresh run, and monitors for completion, again with approval gates.  
6. **Human Approval Gates** are placed before any action that could disrupt service or data.  
7. **Stop Conditions** define when the agent must abort to avoid harm.  
8. **Recovery Commands** provide a clear path back to a stable state if something goes wrong.  

By following this structured, cautious plan, the flight‑recorder benchmark can be restored to reliable operation while safeguarding all existing data.

# Version Control Strategy  
To maintain a reliable deployment pipeline, the flight‑recorder project should adhere to a strict Git workflow.  

1. **Branching Model**  
   - `main` – stable, production‑ready code.  
   - `develop` – integration branch for new features.  
   - Feature branches – `feature/<name>` for isolated work.  
   - Hotfix branches – `hotfix/<name>` for urgent patches.  

2. **Pull Request (PR) Process**  
   - All changes must be merged via PRs.  
   - Require at least one code‑reviewer approval.  
   - Run automated tests and linting in CI before merge.  

3. **Tagging Releases**  
   - Tag every merge to `main` with a semantic version: `vX.Y.Z`.  
   - Store tags in a dedicated `releases` folder for rollback.  

4. **Git Hooks**  
   - Pre‑commit hook to run `black`, `flake8`, and unit tests.  
   - Pre‑push hook to enforce commit message format.  

5. **Backup of Git Repository**  
   ```bash
   ssh user@192.168.1.116 "cd /models/flight-recorder && git bundle create /tmp/flight-recorder.bundle --all"
   ```
   - Transfer the bundle to a secure off‑site location.  

# Environment Variable Management  
Sensitive configuration should be externalized to environment variables or a secrets manager.

1. **`.env` File** – Store non‑sensitive defaults.  
   ```bash
   DB_PATH=/models/flight-recorder/data/flight.db
   LOG_LEVEL=INFO
   ```

2. **Systemd Service File** – Export variables.  
   ```ini
   [Service]
   EnvironmentFile=/etc/flight-recorder/.env
   ```

3. **Secrets Manager** – For production, use HashiCorp Vault or AWS Secrets Manager.  
   ```bash
   export SECRET_API_KEY=$(vault kv get -field=api_key secret/flight-recorder)
   ```

4. **Validation Script** – Ensure all required env vars are set before start.  
   ```bash
   ssh user@192.168.1.116 "python /models/flight-recorder/app/validate_env.py"
   ```

# Containerization and Docker  
Encapsulating the application in a Docker container simplifies deployment and isolation.

1. **Dockerfile**  
   ```dockerfile
   FROM python:3.12-slim
   WORKDIR /app
   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt
   COPY . .
   CMD ["gunicorn", "-b", "0.0.0.0:8080", "app:app"]
   ```

2. **Build Image**  
   ```bash
   ssh user@192.168.1.116 "docker build -t flight-recorder:latest /models/flight-recorder/app"
   ```

3. **Run Container**  
   ```bash
   ssh user@192.168.1.116 "docker run -d --name flight-recorder -p 8080:8080 -v /models/flight-recorder/data:/app/data flight-recorder:latest"
   ```

4. **Health Check** – Docker healthcheck in Dockerfile.  
   ```dockerfile
   HEALTHCHECK --interval=30s --timeout=5s CMD curl -f http://localhost:8080/health || exit 1
   ```

5. **Rollback** – Keep previous image tags.  
   ```bash
   docker tag flight-recorder:latest flight-recorder:$(date +%Y%m%d_%H%M%S)
   ```

# Continuous Integration Pipeline  
Automate testing, linting, and deployment with a CI system (GitHub Actions, GitLab CI, or Jenkins).

1. **CI Workflow** – Example GitHub Actions YAML.  
   ```yaml
   name: CI
   on: [push, pull_request]
   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
         - name: Set up Python
           uses: actions/setup-python@v5
           with:
             python-version: '3.12'
         - name: Install dependencies
           run: pip install -r requirements.txt
         - name: Lint
           run: flake8 .
         - name: Test
           run: pytest tests/
         - name: Build Docker
           run: docker build -t flight-recorder:${{ github.sha }} .
   ```

2. **Deployment Trigger** – On merge to `main`, push Docker image to registry and trigger remote deploy script.  

3. **Rollback on Failure** – If CI fails, the pipeline aborts and no new image is pushed.  

# Security Hardening  
Protect the server and application from common threats.

1. **Firewall Rules**  
   ```bash
   ssh user@192.168.1.116 "sudo ufw allow 8080/tcp"
   ssh user@192.168.1.116 "sudo ufw enable"
   ```

2. **SSH Hardening**  
   - Disable root login: `PermitRootLogin no`.  
   - Use key‑based authentication only.  
   - Change default SSH port to 2222.  

3. **Application Hardening**  
   - Enable HTTPS with self‑signed cert or Let's Encrypt.  
   - Set `SECURE_SSL_REDIRECT=True` in config.  
   - Use `helmet` middleware (if Node) or equivalent.  

4. **Database Security**  
   - Store SQLite file with permissions `600`.  
   - Regularly run `PRAGMA quick_check;` and `PRAGMA integrity_check;`.  

5. **Secrets Management**  
   - Never commit secrets to Git.  
   - Use environment variables or Vault.  

# Performance Tuning  
Optimize resource usage to ensure the benchmark runs smoothly.

1. **CPU Affinity**  
   ```bash
   ssh user@192.168.1.116 "taskset -c 0-3 python /models/flight-recorder/benchmark/benchmark_script.py"
   ```

2. **Memory Limits**  
   - Use cgroups or Docker `--memory` flag.  
   ```bash
   docker run -d --name flight-recorder --memory=512m -p 8080:8080 flight-recorder:latest
   ```

3. **Database Indexing**  
   ```sql
   sqlite3 /models/flight-recorder/data/flight.db "CREATE INDEX IF NOT EXISTS idx_timestamp ON flights(timestamp);"
   ```

4. **Connection Pooling**  
   - Configure SQLAlchemy pool size: `pool_size=10`.  

5. **Benchmark Profiling**  
   ```bash
   ssh user@192.168.1.116 "python -m cProfile -o profile.prof /models/flight-recorder/benchmark/benchmark_script.py"
   ```

# Monitoring and Alerting  
Continuous visibility into system health and benchmark progress.

1. **Prometheus Exporter** – Expose metrics.  
   ```python
   from prometheus_client import start_http_server, Counter
   start_http_server(8000)
   requests_counter = Counter('requests_total', 'Total HTTP requests')
   ```

2. **Grafana Dashboard** – Visualize CPU, memory, DB latency.  

3. **Alertmanager** – Trigger alerts on thresholds.  
   ```yaml
   alert: HighCPU
   expr: node_cpu_seconds_total{mode="idle"} < 0.2
   ```

4. **Log Aggregation** – Ship logs to Loki or ELK stack.  

5. **Health Checks** – Periodic curl to `/health` endpoint.  

# Documentation and Knowledge Base  
Maintain up‑to‑date documentation for future operators.

1. **README.md** – Project overview, setup, and deployment steps.  
2. **CONTRIBUTING.md** – Git workflow, coding standards.  
3. **SECURITY.md** – Vulnerability reporting process.  
4. **CHANGELOG.md** – Version history.  
5. **Wiki** – Detailed runbooks, troubleshooting guides.  

# Future Improvements  
Plan for long‑term resilience and scalability.

1. **Horizontal Scaling** – Deploy multiple instances behind a load balancer.  
2. **Database Migration** – Move from SQLite to PostgreSQL for concurrent writes.  
3. **CI/CD Automation** – Full GitOps pipeline with ArgoCD.  
4. **Observability** – Integrate OpenTelemetry for distributed tracing.  
5. **Automated Rollback** – Detect benchmark failures and revert to previous stable state automatically.  

---  

By extending the plan with these additional layers—version control discipline, environment isolation, containerization, CI/CD, security hardening, performance tuning, monitoring, comprehensive documentation, and a roadmap for future scalability—the flight‑recorder benchmark can be maintained reliably, safely, and with minimal risk of data loss or downtime.