# Mission The primary objective of this operation is to restore the reliable execution of benchmarks on the remote model server located at `192.168.1.116`. The system, referred to as the "Flight Recorder," is currently experiencing instability or failure in its benchmark pipeline. The mission is governed by a "Safety-First" mandate. The agent must ensure that no production data—specifically the SQLite database located in `/models/flight-recorder/data`—is lost, corrupted, or deleted during the remediation process. The goal is to move the system from an unstable or non-functional state to a verified, healthy state where benchmarks can be executed and recorded without manual intervention or intermittent crashes. **Key Objectives:** 1. **Diagnostic Baseline:** Establish a clear understanding of the current system state (CPU, Memory, Disk, Process status). 2. **Integrity Verification:** Confirm that the SQLite database is healthy and that the application code is intact. 3. **Safe Redeployment:** Update or restart the application environment to clear transient faults or apply necessary patches. 4. **Validation:** Execute a controlled benchmark run to verify the fix. 5. **Persistence:** Ensure the fix is stable and does not introduce regressions in data logging. --- # Read-Only Inspection Before any changes are made, the agent will perform a comprehensive read-only audit of the environment. This phase is designed to identify the root cause of the benchmark failure without altering the state of the machine. ### 1. Connectivity and Environment Audit The agent will first verify the ability to communicate with the host and map the filesystem. - **Command:** `ssh user@192.168.1.116 "ls -la /models/flight-recorder/app && ls -la /models/flight-recorder/data"` - **Purpose:** Confirm directory existence and check ownership/permissions. If the agent cannot write to the data directory, the benchmark will fail. - **Expected Outcome:** Confirmation that `/app` contains the source code and `/data` contains the `.sqlite` files. ### 2. Resource Utilization Analysis Since the server "may be busy," it is critical to determine if the failure is due to resource exhaustion (OOM, CPU pinning, or Disk I/O wait). - **Command:** `ssh user@192.168.1.116 "top -b -n 1 | head -n 20"` - **Command:** `ssh user@192.168.1.116 "df -h /models"` - **Command:** `ssh user@192.168.1.116 "free -m"` - **Analysis:** - If CPU load is > 80% consistently, the agent will investigate which process is consuming cycles. - If disk space is < 5%, the SQLite database may be failing to write journals, leading to "database is locked" errors. ### 3. Process State Inspection Identify if the model server is currently running, hung, or crashing in a loop. - **Command:** `ssh user@192.168.1.116 "ps aux | grep -E 'flight-recorder|python|model_server'"` - **Purpose:** Check for zombie processes or multiple instances of the server running simultaneously, which would cause SQLite locking conflicts. ### 4. Log Aggregation and Error Hunting The agent will scan the application logs for specific keywords: `ERROR`, `CRITICAL`, `Timeout`, `Locked`, or `OutOfMemory`. - **Command:** `ssh user@192.168.1.116 "tail -n 500 /models/flight-recorder/app/logs/server.log"` (assuming standard log path). - **Command:** `ssh user@192.168.1.116 "grep -i 'exception' /models/flight-recorder/app/logs/server.log | tail -n 20"` - **Analysis:** Look for `sqlite3.OperationalError: database is locked` or `CUDA out of memory`. --- # Health Checks Once the initial inspection is complete, the agent will run a series of active health checks to determine the functional status of the service. ### 1. API Responsiveness If the model server provides a REST API, the agent will test the heartbeat endpoint. - **Command:** `ssh user@192.168.1.116 "curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/health"` - **Interpretation:** - `200 OK`: Service is alive. - `500/503`: Service is alive but unhealthy. - `000/Connection Refused`: Service is down. ### 2. Port Binding Verification Ensure the application is listening on the expected ports. - **Command:** `ssh user@192.168.1.116 "netstat -tulpn | grep LISTEN"` - **Purpose:** Verify that the server hasn't crashed and left a "ghost" port open, or that it failed to bind to the port due to a conflict. ### 3. Model Loading Status Check if the model is actually loaded into VRAM/RAM. - **Command:** `ssh user@192.168.1.116 "nvidia-smi"` (if GPU based). - **Analysis:** If VRAM is occupied but the API is unresponsive, the process is likely hung in a deadlock. --- # Data Safety Checks Because the database is SQLite, it is susceptible to corruption if the server crashed during a write operation or if the disk is full. ### 1. Database Integrity Check The agent will use the SQLite CLI to verify the internal structure of the database. - **Command:** `ssh user@192.168.1.116 "sqlite3 /models/flight-recorder/data/flight_recorder.db 'PRAGMA integrity_check;'"` - **Expected Result:** `ok`. Any other result indicates corruption. ### 2. Write-Ability Test Verify that the application can still write to the data directory. - **Command:** `ssh user@192.168.1.116 "touch /models/flight-recorder/data/test_write.tmp && rm /models/flight-recorder/data/test_write.tmp"` - **Purpose:** Ensure the filesystem is not mounted as Read-Only due to a kernel error. ### 3. Mandatory Backup (Pre-Remediation) Before any redeploy or restart, a snapshot of the data must be taken. - **Command:** `ssh user@192.168.1.116 "cp /models/flight-recorder/data/flight_recorder.db /models/flight-recorder/data/flight_recorder.db.bak_$(date +%Y%m%d_%H%M%S)"` - **Safety Note:** This is a non-destructive copy. It ensures that if a redeploy triggers a database migration that fails, the original data is preserved. --- # Redeploy Procedure If the health checks indicate the service is hung or the code is in an inconsistent state, a redeploy will be initiated. ### 1. Graceful Shutdown Attempt to stop the service without killing processes abruptly. - **Command:** `ssh user@192.168.1.116 "systemctl stop flight-recorder"` or `ssh user@192.168.1.116 "pm2 stop all"` - **Verification:** Run `ps aux` again to ensure no lingering processes remain. ### 2. Forced Termination (If Necessary) If the service is unresponsive to SIGTERM. - **Command [REQUIRES APPROVAL]:** `ssh user@192.168.1.116 "pkill -9 -f flight-recorder"` - **Reasoning:** Only used if the process is in a "D" state or unresponsive to standard stop commands. ### 3. Code Refresh Ensure the application is running the correct version of the code. - **Command:** `ssh user@192.168.1.116 "cd /models/flight-recorder/app && git fetch && git reset --hard origin/main"` - **Purpose:** Remove any local "hotfixes" that might have caused instability and return to a known good state. ### 4. Dependency Validation Ensure that no missing libraries are causing the crash. - **Command:** `ssh user@192.168.1.116 "cd /models/flight-recorder/app && pip install -r requirements.txt"` ### 5. Service Restart Bring the server back online. - **Command:** `ssh user@192.168.1.116 "systemctl start flight-recorder"` - **Verification:** Check logs immediately: `tail -f /models/flight-recorder/app/logs/server.log`. --- # Benchmark Restart Procedure Once the server is healthy, the agent will trigger the benchmark suite to verify the restoration of reliability. ### 1. Small-Scale Smoke Test Run a single, low-resource benchmark case to ensure the pipeline is connected. - **Command:** `ssh user@192.168.1.116 "python3 /models/flight-recorder/app/run_benchmark.py --test-mode --cases 1"` - **Purpose:** Verify that the app can read the model, process a request, and write the result to the SQLite DB. ### 2. Full Benchmark Execution Trigger the standard benchmark suite. - **Command:** `ssh user@192.168.1.116 "nohup python3 /models/flight-recorder/app/run_benchmark.py > /models/flight-recorder/app/logs/benchmark_run.log 2>&1 &"` - **Purpose:** Run the full suite in the background to avoid SSH timeout issues. ### 3. Real-time Monitoring Monitor the progress and the database growth. - **Command:** `ssh user@192.168.1.116 "tail -f /models/flight-recorder/app/logs/benchmark_run.log"` - **Command:** `ssh user@192.168.1.116 "watch -n 5 'ls -lh /models/flight-recorder/data/flight_recorder.db'"` - **Analysis:** If the `.db` file size is increasing, the benchmark is successfully recording data. --- # Human Approval Gates To prevent accidental data loss or service disruption, the agent will pause and request explicit approval at the following junctions: 1. **Gate 1: Pre-Redeploy.** After the "Read-Only Inspection" and "Health Checks" are complete, the agent will present the findings (e.g., "CPU is at 99%, process is hung") and ask for permission to proceed with the `systemctl stop` and `git reset` commands. 2. **Gate 2: Forced Termination.** If `systemctl stop` fails, the agent will request approval to use `pkill -9`. 3. **Gate 3: Database Modification.** If the `PRAGMA integrity_check` fails, the agent will **not** attempt to repair the database. It will stop and ask the human for a decision on whether to restore from the `.bak` file or attempt a `VACUUM` repair. 4. **Gate 4: Final Validation.** After the benchmark completes, the agent will present the results and ask for confirmation that the system is considered "Restored." --- # Stop Conditions The agent will immediately cease all operations and alert the human operator if any of the following conditions are met: - **Critical Disk Space:** If `df -h` shows the partition containing `/models` is at 100% capacity. (Continuing could corrupt the SQLite DB). - **Hardware Failure:** If `dmesg` shows I/O errors or GPU XID errors. - **Unexpected Data Loss:** If the `ls -la /models/flight-recorder/data` command returns an empty directory or the database file is missing. - **Permission Denied:** If the agent discovers it lacks the necessary `sudo` or write permissions to perform the redeploy, and cannot be granted them. - **Kernel Panic/Reboot Loop:** If the server becomes unreachable via SSH during a restart. --- # Recovery Commands In the event that the redeploy or benchmark restart causes a regression, the following recovery steps will be taken: ### 1. Database Rollback If the database becomes corrupted or data is accidentally modified during the redeploy. - **Command [REQUIRES APPROVAL]:** `ssh user@192.168.1.116 "mv /models/flight-recorder/data/flight_recorder.db /models/flight-recorder/data/flight_recorder.db.corrupt && cp /models/flight-recorder/data/flight_recorder.db.bak_[TIMESTAMP] /models/flight-recorder/data/flight_recorder.db"` ### 2. Code Rollback If the `git reset --hard origin/main` introduced a bug. - **Command:** `ssh user@192.168.1.116 "cd /models/flight-recorder/app && git checkout [PREVIOUS_COMMIT_HASH]"` ### 3. Emergency Service Stop If the benchmark restart causes a CPU/RAM spike that threatens the rest of the server. - **Command:** `ssh user@192.168.1.116 "pkill -9 -f run_benchmark.py"` --- # Summary The proposed plan follows a strict linear progression from **Observation $\rightarrow$ Validation $\rightarrow$ Remediation $\rightarrow$ Verification**. By prioritizing read-only inspections and mandatory backups, the agent ensures that the "Flight Recorder" data remains safe. The use of human approval gates for destructive or high-impact commands (like `pkill -9` or database restoration) mitigates the risk of accidental downtime. The final success metric is the successful completion of the benchmark suite and the verification of new entries in the SQLite database, ensuring that the system is not just "running," but "functioning reliably." **Execution Flow Recap:** 1. **Inspect:** Check CPU, RAM, Disk, and Logs. 2. **Verify:** Test API and SQLite integrity. 3. **Protect:** Create a timestamped backup of the `.db` file. 4. **Fix:** Stop service $\rightarrow$ Reset code $\rightarrow$ Update deps $\rightarrow$ Start service. 5. **Test:** Smoke test $\rightarrow$ Full benchmark $\rightarrow$ Log monitoring. 6. **Close:** Final report and human sign-off.