# Mission The primary objective of this mission is to restore reliable benchmark execution on the remote model server located at 192.168.1.116. The server hosts a model application under `/models/flight-recorder/app` and relies on a SQLite database located under `/models/flight-recorder/data`. The current state of the server is unknown, but the goal is to ensure that benchmark runs complete successfully, consistently, and without data loss. The agent must operate with a "safety-first" philosophy. Non-destructive inspection and application redeployment are permitted. However, any action that could potentially delete, overwrite, or corrupt the data directory requires explicit human approval. The mission will be executed in a phased approach: 1. **Read-Only Inspection:** Gather comprehensive information about the server's current state without making any changes. 2. **Health Checks:** Verify the operational status of the application, the database, and the underlying system resources. 3. **Data Safety Checks:** Ensure the SQLite database is intact and create a safe backup before any redeployment occurs. 4. **Redeploy Procedure:** Gracefully restart the application to clear any stale states or memory leaks. 5. **Benchmark Restart Procedure:** Execute the benchmark and monitor its output for reliability. Throughout this process, the agent will pause at critical junctures to request human approval, ensuring that no data is inadvertently lost. # Read-Only Inspection Before taking any action, the agent must establish a baseline understanding of the server's current state. This phase involves gathering information about the network, filesystem, running processes, logs, and the database. All commands in this phase are strictly read-only. ## 1. Network and Connectivity First, verify that the server is reachable and that the application port is listening. ```bash # Verify SSH connectivity ssh root@192.168.1.116 # Check if the application port is listening (assuming port 8080, adjust as needed) ss -tlnp | grep 8080 # Check for any established connections to the application port ss -tnp | grep 8080 ``` *Expected Output:* The `ss` command should show the application process listening on the expected port. If no process is listening, the application may have crashed or failed to start. ## 2. Filesystem and Directory Structure Inspect the application and data directories to check for stale files, permission issues, or disk space problems. ```bash # Check overall disk space df -h # Check the specific data directory for disk space df -h /models/flight-recorder/data # List the application directory contents ls -la /models/flight-recorder/app # List the data directory contents ls -la /models/flight-recorder/data # Check for any stale lock files or temporary files in the data directory find /models/flight-recorder/data -name "*.lock" -o -name "*.tmp" -o -name "*.pid" ``` *Expected Output:* `df -h` should show sufficient free space (ideally >20%). The `ls -la` commands should show the expected files. The `find` command should return no results, or if it does, the agent must note them for later inspection. ## 3. Process Inspection Identify running processes related to the application and the benchmark. ```bash # List all running processes ps aux # Filter for the application process ps aux | grep flight-recorder # Filter for any benchmark processes ps aux | grep benchmark ``` *Expected Output:* The `ps aux` output should show the application process. If the application is running, note its PID. If a benchmark process is running, note its PID and state (e.g., is it stuck in `D` state, indicating disk I/O wait?). ## 4. Log Inspection Review application and system logs for errors or warnings. ```bash # Check application logs (assuming systemd journal) journalctl -u flight-recorder --no-pager -n 100 # Check system logs for disk or I/O errors journalctl -k --no-pager -n 100 # Check for any recent kernel OOM (Out of Memory) events dmesg | grep -i "oom" ``` *Expected Output:* The logs should show the application starting up and running without critical errors. If there are errors, note the timestamps and error messages. ## 5. Database Inspection Inspect the SQLite database file without modifying it. ```bash # Check the database file size and permissions ls -lh /models/flight-recorder/data/benchmark.db # Check for SQLite WAL (Write-Ahead Logging) files ls -lh /models/flight-recorder/data/benchmark.db-wal ls -lh /models/flight-recorder/data/benchmark.db-shm # Check the database journal mode (read-only) sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA journal_mode;" # Check the database integrity (read-only) sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;" ``` *Expected Output:* The database file should exist and have a reasonable size. The `PRAGMA journal_mode` should return `wal` or `delete`. The `PRAGMA integrity_check` should return `ok`. If it returns anything else, the database is corrupted, and the agent must halt and request human approval. # Health Checks After the read-only inspection, the agent must perform active health checks to determine if the server is in a healthy state. These checks will help identify if the application is responsive, if the database is accessible, and if the system resources are adequate. ## 1. Application Health Check Verify that the application is responding to requests. ```bash # Check the application health endpoint (assuming /health endpoint exists) curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health # Check the application root endpoint curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/ ``` *Expected Output:* A `200` status code indicates the application is healthy. A `500` or `503` code indicates an error. A `000` code indicates the application is not running or not responding. ## 2. Database Health Check Verify that the database is accessible and not locked. ```bash # Check if the database is locked sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA busy_timeout(5000);" sqlite3 /models/flight-recorder/data/benchmark.db "SELECT 1;" # Check the database size sqlite3 /models/flight-recorder/data/benchmark.db "SELECT page_count * page_size as size FROM pragma_page_count(), pragma_page_size();" ``` *Expected Output:* The `SELECT 1` command should return `1` without hanging. If it hangs, the database is locked, and the agent must investigate the locking process. ## 3. System Resource Health Check Verify that the system has adequate CPU, memory, and disk I/O. ```bash # Check CPU and memory usage top -b -n 1 | head -n 10 # Check disk I/O iostat -x 1 5 # Check for memory leaks or high memory usage free -h ``` *Expected Output:* CPU usage should be normal. Memory usage should not be critically high. Disk I/O should not show high `await` times, which would indicate a bottleneck. ## 4. Benchmark Health Check Verify if there are any leftover benchmark processes or state files. ```bash # Check for any benchmark state files ls -la /models/flight-recorder/data/benchmark_state* # Check for any benchmark log files ls -la /models/flight-recorder/data/benchmark_log* ``` *Expected Output:* If there are leftover state files, the agent must note them. If the benchmark is currently running, the agent must determine if it is stuck or just taking a long time. # Data Safety Checks Before any redeployment or benchmark restart, the agent must ensure that the data is safe. This is the most critical phase of the mission. The agent will create a backup of the SQLite database and verify its integrity. ## 1. Database Backup Create a backup of the SQLite database. Since SQLite is a file-based database, the safest way to back it up is to use the `sqlite3 .backup` command, which ensures a consistent snapshot even if the database is being written to. ```bash # Create a backup of the database sqlite3 /models/flight-recorder/data/benchmark.db ".backup /models/flight-recorder/data/benchmark.db.bak" # Verify the backup was created ls -lh /models/flight-recorder/data/benchmark.db.bak # Verify the backup integrity sqlite3 /models/flight-recorder/data/benchmark.db.bak "PRAGMA integrity_check;" ``` *Expected Output:* The backup file should be created, and the integrity check should return `ok`. If the integrity check fails, the agent must halt and request human approval. ## 2. Data Directory Backup If the database backup is successful, the agent can create a full backup of the data directory. This provides an additional layer of safety. ```bash # Create a full backup of the data directory cp -a /models/flight-recorder/data /models/flight-recorder/data.bak # Verify the backup was created ls -lh /models/flight-recorder/data.bak ``` *Expected Output:* The backup directory should be created with the same permissions and contents as the original data directory. ## 3. Permission Verification Verify that the application has the correct permissions to read and write to the data directory. ```bash # Check the permissions of the data directory ls -la /models/flight-recorder/data # Check the permissions of the database file ls -lh /models/flight-recorder/data/benchmark.db ``` *Expected Output:* The application user should have read and write permissions to the data directory and the database file. If the permissions are incorrect, the agent must note them for later correction. # Redeploy Procedure If the health checks and data safety checks pass, the agent will proceed with the redeployment of the application. The goal is to clear any stale states, memory leaks, or file locks that may be causing benchmark failures. ## 1. Graceful Shutdown Stop the application gracefully to ensure that all pending operations are completed and the database is properly closed. ```bash # Stop the application service systemctl stop flight-recorder # Verify the application is stopped ps aux | grep flight-recorder ``` *Expected Output:* The `ps aux` command should show no running application processes. If the process is still running, the agent must force kill it (requires human approval). ## 2. Clean Build Artifacts Remove any stale build artifacts or temporary files from the application directory. ```bash # Remove any stale build artifacts rm -rf /models/flight-recorder/app/node_modules/.cache rm -rf /models/flight-recorder/app/dist/.cache # Verify the artifacts are removed ls -la /models/flight-recorder/app/node_modules/.cache ls -la /models/flight-recorder/app/dist/.cache ``` *Expected Output:* The cache directories should be empty or removed. ## 3. Install Dependencies Install the application dependencies to ensure that all required packages are up to date. ```bash # Navigate to the application directory cd /models/flight-recorder/app # Install dependencies npm install # Verify the dependencies are installed ls -la /models/flight-recorder/app/node_modules ``` *Expected Output:* The `npm install` command should complete without errors. The `node_modules` directory should contain the required packages. ## 4. Build the Application Build the application to ensure that all code is compiled and ready for execution. ```bash # Build the application npm run build # Verify the build was successful ls -la /models/flight-recorder/app/dist ``` *Expected Output:* The `npm run build` command should complete without errors. The `dist` directory should contain the built application files. ## 5. Start the Application Start the application and verify that it is running correctly. ```bash # Start the application service systemctl start flight-recorder # Verify the application is running ps aux | grep flight-recorder # Check the application health endpoint curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health ``` *Expected Output:* The `ps aux` command should show the application process running. The `curl` command should return a `200` status code. ## 6. Verify Database Access Verify that the application can access the database after the redeployment. ```bash # Check if the database is accessible sqlite3 /models/flight-recorder/data/benchmark.db "SELECT 1;" # Check if the database is locked sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA busy_timeout(5000);" sqlite3 /models/flight-recorder/data/benchmark.db "SELECT 1;" ``` *Expected Output:* The `SELECT 1` command should return `1` without hanging. The database should not be locked. # Benchmark Restart Procedure Once the application is redeployed and verified, the agent will restart the benchmark. The goal is to ensure that the benchmark runs reliably and produces accurate results. ## 1. Reset Benchmark State If there are any leftover benchmark state files, the agent will reset them to ensure a clean benchmark run. ```bash # Reset benchmark state files rm -f /models/flight-recorder/data/benchmark_state* # Verify the state files are removed ls -la /models/flight-recorder/data/benchmark_state* ``` *Expected Output:* The state files should be removed. ## 2. Start the Benchmark Start the benchmark and monitor its output for errors. ```bash # Start the benchmark cd /models/flight-recorder/app && ./run_benchmark.sh # Monitor the benchmark output tail -f /var/log/flight-recorder/benchmark.log ``` *Expected Output:* The benchmark should start and run without errors. The log file should show the benchmark progress and results. ## 3. Monitor Benchmark Progress Monitor the benchmark progress to ensure that it is running reliably. ```bash # Check the benchmark progress tail -f /var/log/flight-recorder/benchmark.log # Check the application health endpoint curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health # Check the database health sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;" ``` *Expected Output:* The benchmark should progress without errors. The application health endpoint should return a `200` status code. The database integrity check should return `ok`. ## 4. Verify Benchmark Results Once the benchmark completes, verify the results to ensure that they are accurate and reliable. ```bash # Check the benchmark results cat /var/log/flight-recorder/benchmark.log # Check the database for any new data sqlite3 /models/flight-recorder/data/benchmark.db "SELECT COUNT(*) FROM benchmark_results;" ``` *Expected Output:* The benchmark results should be accurate and reliable. The database should contain the new benchmark results. # Human Approval Gates The agent will pause at critical junctures to request human approval. These gates are designed to prevent accidental data loss or system damage. ## Gate 1: Force Kill Application Process If the application process is not responding to the `systemctl stop` command, the agent will request approval to force kill the process. ```bash # Force kill the application process kill -9 ``` *Approval Required:* Yes. Killing a process forcefully can lead to data corruption if the database is being written to. ## Gate 2: Delete Stale Lock Files If there are stale lock files in the data directory, the agent will request approval to delete them. ```bash # Delete stale lock files rm -f /models/flight-recorder/data/*.lock ``` *Approval Required:* Yes. Deleting lock files can lead to data corruption if the database is being written to. ## Gate 3: Overwrite Database Backup If the agent needs to overwrite the database backup, the agent will request approval. ```bash # Overwrite the database backup cp -f /models/flight-recorder/data/benchmark.db /models/flight-recorder/data/benchmark.db.bak ``` *Approval Required:* Yes. Overwriting the backup can lead to data loss if the backup is needed for recovery. ## Gate 4: Delete Benchmark State Files If there are leftover benchmark state files, the agent will request approval to delete them. ```bash # Delete benchmark state files rm -f /models/flight-recorder/data/benchmark_state* ``` *Approval Required:* Yes. Deleting benchmark state files can lead to inaccurate benchmark results. ## Gate 5: Run Destructive Benchmark If the benchmark is destructive and will overwrite existing data, the agent will request approval. ```bash # Run the destructive benchmark cd /models/flight-recorder/app && ./run_benchmark.sh --destructive ``` *Approval Required:* Yes. Running a destructive benchmark can lead to data loss. # Stop Conditions The agent will stop the mission and request human intervention if any of the following conditions are met: 1. **Database Corruption:** If the SQLite database integrity check fails, the agent will stop and request human intervention. 2. **Disk Full:** If the disk space drops below 10%, the agent will stop and request human intervention. 3. **Unresponsive System:** If the server is unresponsive to SSH or the application is not responding to health checks, the agent will stop and request human intervention. 4. **Unexpected Errors:** If the application fails to start after redeployment, the agent will stop and request human intervention. 5. **User Abort:** If the user explicitly requests the agent to stop, the agent will stop immediately. # Recovery Commands If the mission fails or the system becomes unstable, the agent will use the following recovery commands to restore the system to a stable state. ## 1. Restore Database Backup If the database is corrupted, the agent will restore the database from the backup. ```bash # Restore the database from the backup sqlite3 /models/flight-recorder/data/benchmark.db ".restore /models/flight-recorder/data/benchmark.db.bak" # Verify the database integrity sqlite3 /models/flight-recorder/data/benchmark.db "PRAGMA integrity_check;" ``` ## 2. Rollback Application If the application fails to start after redeployment, the agent will rollback the application to the previous version. ```bash # Rollback the application cd /models/flight-recorder/app && git checkout # Rebuild the application npm install && npm run build # Start the application systemctl start flight-recorder ``` ## 3. Force Kill Stuck Processes If a process is stuck and not responding, the agent will force kill it. ```bash # Force kill the stuck process kill -9 ``` ## 4. Clear Stale Lock Files If the database is locked due to stale lock files, the agent will clear them. ```bash # Clear stale lock files rm -f /models/flight-recorder/data/*.lock ``` ## 5. Restart System If the system is completely unresponsive, the agent will request human approval to restart the system. ```bash # Restart the system reboot ``` # Summary This mission plan outlines a cautious and methodical approach to restoring reliable benchmark execution on the remote model server at 192.168.1.116. The plan prioritizes data safety and system stability, ensuring that no data is lost during the remediation process. The mission begins with a read-only inspection to gather information about the server's current state. This is followed by health checks to verify the operational status of the application, the database, and the system resources. Data safety checks are then performed to ensure that the SQLite database is intact and to create a safe backup before any redeployment occurs. If the health checks and data safety checks pass, the agent will proceed with the redeployment of the application. The redeployment process involves a graceful shutdown, cleaning build artifacts, installing dependencies, building the application, and starting the application. The agent will verify that the application is running correctly and that the database is accessible after the redeployment. Once the application is redeployed, the agent will restart the benchmark and monitor its output for errors. The agent will reset any leftover benchmark state files to ensure a clean benchmark run. The benchmark will be monitored to ensure that it runs reliably and produces accurate results. Throughout the mission, the agent will pause at critical junctures to request human approval, ensuring that no data is inadvertently lost. The agent will stop the mission and request human intervention if any of the stop conditions are met. If the mission fails or the system becomes unstable, the agent will use the recovery commands to restore the system to a stable state. By following this plan, the agent will be able to restore reliable benchmark execution on the remote model server without losing data.