# Engineering Dossier: AI Flight Recorder Production Readiness

**Project Codename:** AI Flight Recorder  
**Status:** Pre-Deployment Review  
**Target Environment:** Private Home-Lab (Production-Grade)  
**Primary Model Profile:** Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf (llama.cpp reasoning enabled)

---

# Executive Summary
The AI Flight Recorder is a telemetry and benchmarking suite designed to provide deep observability into LLM interactions within a private infrastructure. Currently, the system functions as a functional prototype, but it faces significant scalability and reliability hurdles regarding large-scale output handling, reasoning-specific metrics, and data privacy.

This dossier outlines the transition from a "lab-only" script to a production-ready service. The primary objectives are:
1.  **Decoupling Large Payloads:** Moving from inline SQLite storage to a file-based artifact system to prevent database bloat and UI rendering lag.
2.  **Reasoning Observability:** Implementing a parser to isolate and measure "thinking" tokens vs. "content" tokens.
3.  **Privacy Guardrails:** Integrating a multi-stage redaction pipeline to ensure PII (emails, Teams messages) never hits the persistent storage.
4.  **Auditability:** Implementing a versioned model profile system to track exactly which weights and parameters produced which result.
5.  **Benchmark Durability:** Transitioning from ephemeral CLI outputs to persistent "Campaign" records.

---

# Current Architecture
The current system follows a linear request-response flow with side-effect logging:

1.  **Entry Point:** `app/main.py` (FastAPI) receives requests via `/v1/chat/completions`.
2.  **Proxy Layer:** `app/proxy.py` acts as a middleware. It initiates a request to `llama.cpp-server` (192.168.1.116).
3.  **Persistence Layer:** `app/store.py` captures:
    *   `llm_requests`: Prompt, response, raw timings.
    *   `client_sessions`: User-specific metadata.
    *   `benchmarks`: Raw results from `app/bench.py`.
4.  **Reporting Layer:** `app/reports.py` performs `SELECT` aggregations on the SQLite database to populate the dashboard.
5.  **Execution:** `app/bench.py` runs prompts and writes results directly to the database.

**Current Infrastructure:**
*   **Database:** SQLite (Single file on `/models/flight-recorder/data/`).
*   **Inference:** `llama.cpp` running on a local GPU/CPU node.
*   **Storage:** All data (including 10k+ token responses) is currently stored as `TEXT` in SQLite.

---

# Failure Modes Found
During the pre-deployment audit, the following critical failure modes were identified:

### 1. SQLite "Database is Locked" (Concurrency)
Under high-load benchmark campaigns (multiple concurrent threads writing to `llm_requests`), SQLite's file-level locking will cause `Database is locked` errors. This is exacerbated by the `app/store.py` helpers if they do not use a connection pool or WAL (Write-Ahead Logging) mode.

### 2. UI/Memory Bloat (The "Giant JSON" Problem)
Storing 20,000+ token responses in a single SQLite row causes the FastAPI `GET /dashboard` route to fetch massive strings into memory. This leads to:
*   High memory pressure on the FastAPI worker.
*   Slow UI rendering (browser hangs trying to parse a 5MB JSON string).
*   Slow query execution for the `reports.py` aggregation logic.

### 3. Reasoning Token Leakage
Currently, reasoning tokens (the "thinking" phase) are bundled with the content. If the goal is to measure "Reasoning Efficiency," the current system cannot distinguish between a model "thinking" for 500 tokens and "writing" 500 tokens of actual content.

### 4. PII Leakage (Privacy Risk)
The proxy currently forwards raw input. If a user pastes a Teams message containing a private email or a phone number, that data is persisted in the `llm_requests` table in plaintext.

### 5. Model Profile Drift
Model profiles are currently defined in a way that is easily mutable. If a user changes a temperature or a system prompt in the config, the historical benchmarks are no longer comparable because the "profile" wasn't versioned.

---

# Data Model Changes
To resolve the identified failure modes, the following schema migrations are required.

### New Table: `model_profiles`
Instead of a static config, we need a versioned registry.
```sql
CREATE TABLE model_profiles (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL, -- e.g., "Qwen3.6-35B-Balanced"
    model_path TEXT NOT NULL,
    temperature REAL,
    top_p REAL,
    max_tokens INTEGER,
    reasoning_enabled BOOLEAN,
    system_prompt TEXT,
    version_hash TEXT NOT NULL, -- SHA256 of the config
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

### Modified Table: `llm_requests`
We will move the `response_content` to a reference system.
```sql
ALTER TABLE llm_requests ADD COLUMN artifact_id TEXT; -- Path to the file
ALTER TABLE llm_requests ADD COLUMN reasoning_tokens INTEGER DEFAULT 0;
ALTER TABLE llm_requests ADD COLUMN content_tokens INTEGER DEFAULT 0;
ALTER TABLE llm_requests ADD COLUMN redacted_flag BOOLEAN DEFAULT FALSE;
ALTER TABLE llm_requests ADD COLUMN profile_id INTEGER REFERENCES model_profiles(id);
```

### New Table: `benchmark_campaigns`
To support "Campaigns" for better reporting.
```sql
CREATE TABLE benchmark_campaigns (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    description TEXT,
    created_by TEXT,
    status TEXT -- 'pending', 'running', 'completed', 'failed'
);

CREATE TABLE benchmark_runs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_id INTEGER REFERENCES benchmark_campaigns(id),
    profile_id INTEGER REFERENCES model_profiles(id),
    start_time TIMESTAMP,
    end_time TIMESTAMP,
    avg_ttft REAL, -- Time to First Token
    avg_tps REAL,  -- Tokens per second
    total_reasoning_tokens INTEGER,
    total_content_tokens INTEGER,
    metadata JSON -- For screenshots, environment info, etc.
);
```

---

# Artifact Storage Design
To solve the "Giant JSON" problem, we will implement a **Content Addressable Storage (CAS)** style approach for long outputs.

### Logic Flow:
1.  **Proxy receives response.**
2.  **Check Length:** If `len(response) > 2048` characters:
    *   Generate a unique ID: `req_{timestamp}_{uuid.uuid4().hex[:8]}`.
    *   Write the full JSON response to `/models/flight-recorder/data/artifacts/{id}.json`.
    *   Store only the `id` in the SQLite `llm_requests` table.
3.  **Dashboard Fetch:** The dashboard will only fetch the first 500 characters and the `artifact_id`. If the user clicks "View Full Response," the frontend fetches the file directly from the artifact path.

### Directory Structure:
```text
/models/flight-recorder/data/
├── database.db
├── artifacts/
│   ├── 20231027_1400_a1b2c3d4.json
│   └── 20231027_1405_e5f6g7h8.json
└── profiles/
    └── qwen-35b-balanced.json
```

---

# Reasoning Budget Handling
Since `llama.cpp` reasoning is enabled, the model outputs a specific block (e.g., `<thought>...</thought>`). We must parse this to calculate the "Reasoning Budget."

### Parsing Logic (`app/proxy.py`):
The proxy will use a streaming parser to identify reasoning blocks.
1.  **Identify Tags:** Use a regex or a state-machine to detect `<thought>` and `</thought>`.
2.  **Token Counting:**
    *   Count tokens inside the tags $\rightarrow$ `reasoning_tokens`.
    *   Count tokens outside the tags $\rightarrow$ `content_tokens`.
3.  **Metrics Calculation:**
    *   **Reasoning Ratio:** $\frac{reasoning\_tokens}{total\_tokens}$
    *   **Reasoning Latency:** Time spent while the model was inside the `<thought>` block.

### Example JSON Metadata:
```json
{
  "request_id": "req_9921",
  "reasoning_tokens": 450,
  "content_tokens": 120,
  "reasoning_ratio": 0.789,
  "thought_duration_ms": 4500,
  "content_duration_ms": 1200
}
```

---

# Benchmark Runner Design
The `app/bench.py` script will be upgraded to a "Campaign" runner.

### CLI Interface:
```bash
# Run a specific campaign with a specific profile
python -m app.bench --campaign "coding_eval_v1" --profile "qwen-35b-balanced" --output-dir "./results"

# Run a new campaign with custom prompts
python -m app.bench --new-campaign "creative_writing_test" --prompts "file://prompts.txt" --profile "qwen-35b-balanced"
```

### Campaign Execution Logic:
1.  **Initialization:** Create a `benchmark_campaign` entry.
2.  **Profile Locking:** Fetch the `model_profile` and lock it (ensure no one changes the temperature mid-run).
3.  **Parallel Execution:** Use a `ThreadPoolExecutor` to send prompts to the proxy.
4.  **Real-time Progress:** Update the `benchmark_runs` table with "Current Progress" (e.g., 45/100 prompts completed).
5.  **Finalization:** Calculate aggregate metrics (TTFT, TPS, Reasoning Ratio) and write the final `benchmark_runs` row.

---

# Reporting Plane
The `app/reports.py` will be updated to provide "Comparison Views."

### Key Metrics to Expose:
1.  **Throughput (TPS):** Average tokens per second across a campaign.
2.  **TTFT (Time to First Token):** Critical for UX.
3.  **Reasoning Overhead:** Average time spent in `<thought>` blocks vs. content generation.
4.  **Cost Efficiency:** (If applicable) Tokens used per prompt.
5.  **Model Comparison:** A side-by-side table comparing `Campaign X` across `Profile A` and `Profile B`.

### Dashboard SQL Example:
```sql
-- Get average reasoning ratio for a specific campaign
SELECT 
    avg(reasoning_tokens * 1.0 / (reasoning_tokens + content_tokens)) as avg_reasoning_ratio
FROM llm_requests 
WHERE profile_id = (SELECT id FROM model_profiles WHERE name = 'Qwen3.6-35B-Balanced')
AND request_id IN (SELECT request_id FROM benchmark_runs WHERE campaign_id = 12);
```

---

# Privacy And Redaction
To prevent PII leakage, a `RedactionEngine` will be implemented in `app/proxy.py`.

### Redaction Pipeline:
1.  **Input Scrubbing:** Before sending to `llama.cpp`, the prompt is scanned.
2.  **Output Scrubbing:** Before saving to SQLite/Artifacts, the response is scanned.
3.  **Regex Patterns:**
    *   **Email:** `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`
    *   **Phone:** `\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b`
    *   **Teams/Slack IDs:** `\b[A-Z0-9]{10,15}\b` (Custom patterns for internal IDs).
4.  **Replacement:** Replace detected PII with `[REDACTED_EMAIL]`, `[REDACTED_PHONE]`.

### Implementation Detail:
The `proxy.py` will use a `RedactionMiddleware` class:
```python
class RedactionMiddleware:
    def __init__(self):
        self.patterns = [EMAIL_REGEX, PHONE_REGEX, TEAMS_ID_REGEX]

    def scrub(self, text: str) -> str:
        for pattern in self.patterns:
            text = re.sub(pattern, "[REDACTED]", text)
        return text
```

---

# Deployment Plan
The deployment will move from a single script to a containerized multi-service architecture.

### Docker Compose Configuration:
```yaml
services:
  ai-flight-recorder:
    build: ./app
    ports:
      - "8000:8000"
    environment:
      - LLAMA_SERVER_URL=http://192.168.1.116:8080
      - DATABASE_URL=sqlite:////models/flight-recorder/data/database.db
      - ARTIFACT_PATH=/models/flight-recorder/data/artifacts
    volumes:
      - ./data:/models/flight-recorder/data
    depends_on:
      - llama-cpp-server

  llama-cpp-server:
    image: ghcr.io/ggerganov/llama.cpp:full-gpu
    ports:
      - "8080:8080"
    volumes:
      - ./models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

### Migration Steps:
1.  **Backup:** Copy existing `database.db` to `database_backup_v1.db`.
2.  **Schema Update:** Run `python -m app.migrations.upgrade_v2` (creates new tables and adds columns).
3.  **Data Migration:** Script to move existing `response_content` into the new `artifacts/` folder and update `llm_requests` with IDs.
4.  **Environment Sync:** Update all `.env` files to point to the new `ARTIFACT_PATH`.

---

# Test Plan
The following 20 acceptance tests must pass before deployment.

### Unit Tests (Redaction & Parsing)
1.  **Redaction_Email:** Verify `test@example.com` becomes `[REDACTED]`.
2.  **Redaction_Phone:** Verify `555-0199` becomes `[REDACTED]`.
3.  **Redaction_Teams:** Verify internal ID `ABC123XYZ` is caught.
4.  **Reasoning_Parser_Basic:** Verify `<thought>...</thought>` is correctly split.
5.  **Reasoning_Parser_Nested:** Verify nested tags (if any) don't break the parser.
6.  **Reasoning_Token_Count:** Verify token counts match `tiktoken` counts for the specific model.

### Integration Tests (Proxy & Storage)
7.  **Proxy_Forward_Success:** Verify request reaches `llama.cpp` and returns 200.
8.  **Proxy_Timeout:** Verify 504 error when `llama.cpp` takes > 60s.
9.  **Artifact_Storage_Large:** Verify a 10,000 token response is saved to a file, not the DB.
10. **Artifact_Storage_Duplicate:** Verify unique IDs are generated for identical prompts.
11. **Database_WAL_Mode:** Verify SQLite is running in WAL mode (check `PRAGMA journal_mode`).
12. **Profile_Audit:** Verify that changing a profile's temperature creates a new `version_hash`.

### Benchmark Tests (Runner)
13. **Campaign_Creation:** Verify a new campaign is created in the DB.
14. **Campaign_Parallelism:** Verify 5 concurrent prompts are handled by the runner.
15. **Benchmark_Aggregation:** Verify `avg_tps` is calculated correctly after a run.
16. **Benchmark_Failure_Recovery:** Verify a failed run is marked as `failed` and doesn't crash the runner.

### UI & Reporting Tests
17. **Dashboard_Load_Speed:** Verify dashboard loads in < 2s with 1,000+ records.
18. **Dashboard_Artifact_Link:** Verify clicking "View Full" opens the correct file.
19. **Report_Comparison:** Verify the "Model Comparison" table shows correct data for two different profiles.
20. **Redaction_Persistence:** Verify that no `[REDACTED]` strings are missing from the final stored artifact.

---

# Rollback Plan
In the event of a critical failure during deployment:

1.  **Immediate Rollback:**
    *   Stop the `ai-flight-recorder` container.
    *   Revert the `database.db` to the `database_backup_v1.db` file.
    *   Restart the previous Docker image version.
2.  **Data Recovery:**
    *   If the new artifact system was used, the `llm_requests` table will need to be re-populated from the `artifacts/` folder using a provided `recovery_script.py`.
3.  **Trigger Points:**
    *   Rollback if:
        *   Proxy latency increases by > 50% compared to baseline.
        *   SQLite "Database is locked" errors occur more than 5 times in 1 minute.
        *   Redaction fails to catch a known PII string in the first 10 requests.

---

# Open Questions
1.  **Tokenization Consistency:** Should we use the `llama.cpp` tokenizer or a standard `tiktoken` library for counting? (Recommendation: Use `llama.cpp` tokenizer to ensure 1:1 parity with the inference engine).
2.  **Artifact Retention:** Do we need a TTL (Time To Live) for artifacts? (Recommendation: Keep all artifacts for 30 days, then archive to cold storage).
3.  **Advanced Redaction:** Should we implement a secondary LLM-based redaction pass for complex PII (e.g., "The secret project name is X")?
4.  **Multi-GPU Scaling:** If we move to a multi-GPU setup, how will the proxy handle load balancing across multiple `llama.cpp` instances?
5.  **User Auth:** Should we add an API Key requirement to the proxy to prevent unauthorized internal access?

6. **Multi-GPU Scaling:** If we move to a multi-GPU setup, how will the proxy handle load balancing across multiple `llama.cpp` instances? (Recommendation: Implement a round-robin or least-connections load balancer in the proxy layer, or utilize a dedicated load balancer like HAProxy in front of the inference nodes.)
7. **Tokenization Consistency:** Should we use the `llama.cpp` tokenizer or a standard `tiktoken` library for counting? (Recommendation: Use `llama.cpp` tokenizer to ensure 1:1 parity with the inference engine.)
8. **Artifact Retention:** Do we need a TTL (Time To Live) for artifacts? (Recommendation: Keep all artifacts for 30 days, then archive to cold storage.)
9. **Advanced Redaction:** Should we implement a secondary LLM-based redaction pass for complex PII (e.g., "The secret project name is X")? (Recommendation: Start with regex-based redaction and move to a small, local "Redaction-Model" (e.g., a 1B parameter model) for high-sensitivity data.)
10. **User Auth:** Should we add an API Key requirement to the proxy to prevent unauthorized internal access? (Recommendation: Yes, implement a simple `X-API-Key` header check in the FastAPI middleware.)

---

# Technical Deep Dive: Implementation Specifications

## 1. Proxy Layer Architecture (`app/proxy.py`)
The proxy is the most critical component for reliability. It must handle streaming, redaction, and artifact offloading.

### Request Lifecycle:
1.  **Ingest:** Receive `POST /v1/chat/completions`.
2.  **Auth:** Validate `X-API-Key`.
3.  **Redact Input:** Run the `RedactionEngine` on the `messages` array.
4.  **Profile Lookup:** Retrieve the `model_profile` based on the requested model ID.
5.  **Forward:** Send the request to `llama.cpp-server` with the correct parameters (temperature, top_p, etc.).
6.  **Stream Processing:**
    *   As chunks arrive, identify `<thought>` tags.
    *   Calculate `reasoning_tokens` vs `content_tokens` on the fly.
    *   Track `ttft` (Time to First Token).
7.  **Finalize:**
    *   Once the stream ends, perform a final redaction on the full response.
    *   Check response length. If > 2048 chars, call `ArtifactManager.save()`.
    *   Write the final record to `llm_requests` via `app/store.py`.
8.  **Respond:** Return the redacted, streamed response to the client.

### Redaction Engine Implementation:
```python
import re
import json
from typing import List, Dict

class RedactionEngine:
    def __init__(self):
        # Pre-compiled regex for performance
        self.patterns = {
            "email": re.compile(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'),
            "phone": re.compile(r'\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b'),
            "teams_id": re.compile(r'\b[A-Z0-9]{10,15}\b'),
            "ipv4": re.compile(r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b')
        }

    def scrub(self, text: str) -> str:
        if not text:
            return text
        
        scrubbed_text = text
        for label, pattern in self.patterns.items():
            scrubbed_text = pattern.sub(f"[REDACTED_{label.upper()}]", scrubbed_text)
        
        return scrubbed_text

    def scrub_json(self, data: Dict) -> Dict:
        """Recursively scrub all string values in a JSON object."""
        if isinstance(data, dict):
            return {k: self.scrub(v) if isinstance(v, str) else self.scrub_json(v) for k, v in data.items()}
        elif isinstance(data, list):
            return [self.scrub(i) if isinstance(i, str) else self.scrub_json(i) for i in data]
        elif isinstance(data, str):
            return self.scrub(data)
        return data
```

### Artifact Manager Implementation:
```python
import os
import json
import uuid
from datetime import datetime

class ArtifactManager:
    def __init__(self, base_path: str):
        self.base_path = base_path
        os.makedirs(base_path, exist_ok=True)

    def save_large_response(self, content: dict) -> str:
        """
        Saves a large response to a file and returns the unique ID.
        """
        unique_id = f"req_{datetime.now().strftime('%Y%m%d_%H%M%S')}_{uuid.uuid4().hex[:8]}"
        file_path = os.path.join(self.base_path, f"{unique_id}.json")
        
        with open(file_path, 'w') as f:
            json.dump(content, f, indent=2)
            
        return unique_id

    def get_artifact(self, artifact_id: str) -> dict:
        file_path = os.path.join(self.base_path, f"{artifact_id}.json")
        if not os.path.exists(file_path):
            return {}
        with open(file_path, 'r') as f:
            return json.load(f)
```

---

## 2. Data Persistence Layer (`app/store.py`)
To handle SQLite's concurrency limitations, we will use a connection pool and ensure WAL mode is enabled.

### Database Initialization:
```python
import sqlite3
from contextlib import contextmanager

DATABASE_PATH = "/models/flight-recorder/data/database.db"

def init_db():
    with get_db_connection() as conn:
        cursor = conn.cursor()
        # Enable WAL mode for better concurrency
        cursor.execute("PRAGMA journal_mode=WAL;")
        cursor.execute("PRAGMA synchronous=NORMAL;")
        
        # Create tables if they don't exist (simplified for brevity)
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS model_profiles (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                name TEXT NOT NULL,
                model_path TEXT NOT NULL,
                temperature REAL,
                top_p REAL,
                max_tokens INTEGER,
                reasoning_enabled BOOLEAN,
                system_prompt TEXT,
                version_hash TEXT NOT NULL,
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            );
        """)
        # ... other table creations ...
        conn.commit()

@contextmanager
def get_db_connection():
    conn = sqlite3.connect(DATABASE_PATH, timeout=30)
    conn.row_factory = sqlite3.Row
    try:
        yield conn
    finally:
        conn.close()
```

---

## 3. Benchmark Runner Logic (`app/bench.py`)
The benchmark runner is designed to be a stateful machine that tracks progress and handles failures gracefully.

### Campaign State Machine:
1.  **PENDING:** Campaign created, but no prompts have been sent.
2.  **RUNNING:** Prompts are being dispatched to the proxy.
3.  **COMPLETED:** All prompts in the campaign have received a response.
4.  **FAILED:** A critical error occurred (e.g., proxy unreachable) and the run was aborted.

### Benchmark Execution Loop:
```python
import time
import concurrent.futures
from app.store import update_benchmark_progress

def run_campaign(campaign_id: int, profile_id: int, prompts: List[str]):
    # 1. Update status to RUNNING
    update_campaign_status(campaign_id, "RUNNING")
    
    results = []
    start_time = time.time()
    
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        future_to_prompt = {
            executor.submit(call_proxy_api, prompt, profile_id): prompt 
            for prompt in prompts
        }
        
        completed_count = 0
        for future in concurrent.futures.as_completed(future_to_prompt):
            try:
                res = future.result()
                results.append(res)
                completed_count += 1
                
                # Update progress every 5 prompts
                if completed_count % 5 == 0:
                    update_benchmark_progress(campaign_id, completed_count, len(prompts))
            except Exception as e:
                print(f"Error processing prompt: {e}")
                update_campaign_status(campaign_id, "FAILED")
                return

    # 2. Final Aggregation
    total_time = time.time() - start_time
    avg_tps = calculate_tps(results, total_time)
    avg_ttft = calculate_ttft(results)
    
    finalize_benchmark_run(campaign_id, profile_id, avg_tps, avg_ttft, results)
    update_campaign_status(campaign_id, "COMPLETED")
```

---

## 4. Reporting and Dashboard Metrics
The dashboard will provide a high-level overview and a granular "Deep Dive" view.

### Dashboard Metrics (SQL Aggregations):
*   **Total Tokens Processed:** `SELECT SUM(content_tokens + reasoning_tokens) FROM llm_requests`
*   **Average Reasoning Ratio:** `SELECT AVG(reasoning_tokens * 1.0 / (reasoning_tokens + content_tokens)) FROM llm_requests`
*   **Model Comparison Table:**
    | Model Profile | Avg TPS | Avg TTFT | Reasoning % | Success Rate |
    | :--- | :--- | :--- | :--- | :--- |
    | Qwen-35B-Balanced | 45.2 | 120ms | 15.4% | 99.8% |
    | Llama-3-70B-Fast | 32.1 | 250ms | 5.2% | 98.5% |

### API Endpoints for Dashboard:
*   `GET /api/dashboard/summary`: Returns aggregate stats for the last 24 hours.
*   `GET /api/dashboard/campaign/{id}`: Returns detailed metrics for a specific benchmark campaign.
*   `GET /api/dashboard/compare?profile_a=ID&profile_b=ID`: Returns a side-by-side comparison of two profiles.
*   `GET /api/dashboard/logs?limit=50`: Returns the last 50 requests (redacted).

---

## 5. Infrastructure and Deployment Details

### Docker Environment Variables:
| Variable | Description | Default Value |
| :--- | :--- | :--- |
| `LLAMA_SERVER_URL` | URL of the llama.cpp server | `http://192.168.1.116:8080` |
| `DATABASE_URL` | Path to the SQLite file | `/models/flight-recorder/data/database.db` |
| `ARTIFACT_PATH` | Path to the artifact folder | `/models/flight-recorder/data/artifacts` |
| `REDACTION_LEVEL` | Level of scrubbing (0-3) | `2` |
| `MAX_WORKERS` | Max concurrent proxy requests | `10` |
| `LOG_LEVEL` | Logging verbosity | `INFO` |

### Storage Strategy:
*   **Database:** SQLite (WAL Mode) for metadata, counts, and status.
*   **Artifacts:** Local filesystem (Ext4 or XFS) for large JSON blobs.
*   **Logs:** Standard Python `logging` output to `stdout`, captured by Docker logs.

---

## 6. Maintenance and Operations

### Log Rotation:
The system will use `RotatingFileHandler` to ensure that logs do not consume all available disk space.
*   Max Files: 10
*   Max Bytes: 10MB

### Artifact Cleanup:
A cron job or a scheduled task will run every Sunday at 03:00 AM to archive artifacts older than 30 days.
```bash
# Example cleanup script
find /models/flight-recorder/data/artifacts/ -mtime +30 -exec mv {} /models/flight-recorder/data/archive/ \;
```

### Model Profile Auditing:
Every time a `model_profile` is updated, the system will:
1.  Generate a new `version_hash` (SHA256 of the profile parameters).
2.  Log the change in a `profile_audit_log` table.
3.  Ensure that any benchmark run associated with the *old* hash remains unchanged.

---

## 7. Troubleshooting Guide

### Common Error Codes:
*   **ERR_PROXY_TIMEOUT:** The `llama.cpp` server did not respond within the configured timeout (default 60s).
    *   *Solution:* Check GPU utilization or increase `LLAMA_SERVER_TIMEOUT`.
*   **ERR_DB_LOCKED:** SQLite is unable to write to the database.
    *   *Solution:* Ensure WAL mode is enabled and check for long-running `SELECT` queries in `app/reports.py`.
*   **ERR_ARTIFACT_WRITE:** Permission denied or disk full in the artifact directory.
    *   *Solution:* Check Docker volume permissions and disk space.
*   **ERR_REDACTION_FAIL:** A known PII string was detected in the output but not redacted.
    *   *Solution:* Update the `RedactionEngine` regex patterns.

### Diagnostic Commands:
*   `python -m app.diagnose --db-check`: Verifies database integrity and WAL mode.
*   `python -m app.diagnose --proxy-ping`: Tests connectivity to the `llama.cpp` server.
*   `python -m app.diagnose --redaction-test`: Runs a suite of PII strings through the redaction engine to verify success.

---

## 8. Synthetic Data for Testing
To verify the system's ability to handle large outputs and redaction, use the following test cases.

### Test Case 1: Large Output (Stress Test)
**Prompt:** "Write a 5,000-word technical manual for a fictional spaceship's propulsion system, including detailed safety protocols, maintenance schedules, and emergency procedures."
**Expected Behavior:**
*   Proxy should detect length > 2048.
*   Proxy should save the full output to `artifacts/`.
*   Database should store the `artifact_id`.
*   Dashboard should show a "View Full" link.

### Test Case 2: PII Redaction (Privacy Test)
**Prompt:** "Send an email to john.doe@example.com regarding the project. Include my phone number 555-0199 and the Teams ID ABC123XYZ."
**Expected Behavior:**
*   Input should be scrubbed before reaching `llama.cpp`.
*   Output should contain `[REDACTED_EMAIL]`, `[REDACTED_PHONE]`, and `[REDACTED_TEAMS_ID]`.

### Test Case 3: Reasoning Analysis (Metric Test)
**Prompt:** "Solve this complex math problem: [Insert 10-step calculus problem]. Think step-by-step."
**Expected Behavior:**
*   Parser should identify the `<thought>` block.
*   `reasoning_tokens` should be significantly higher than `content_tokens`.
*   `reasoning_ratio` should be accurately reflected in the dashboard.

---

## 9. Future Roadmap

### Phase 2: Advanced Observability
*   **Visual Trace:** A timeline view of the request, showing when the model was "thinking" vs. "writing."
*   **Token Costing:** Integration with OpenAI/Anthropic pricing models to estimate the cost of local runs.
*   **Multi-Model Comparison:** A "Battle" mode where the same prompt is sent to two different models simultaneously and results are compared side-by-side.

### Phase 3: Enterprise Features
*   **RBAC:** Role-Based Access Control for the dashboard (Admin, Viewer, Auditor).
*   **Webhooks:** Trigger a webhook when a benchmark campaign completes.
*   **S3 Integration:** Automatically upload artifacts to an S3-compatible bucket (e.g., MinIO) for long-term storage.
*   **Dynamic Redaction:** Use a dedicated local LLM to perform "Context-Aware Redaction" to catch nuanced PII.

---

## 10. Appendix: Glossary of Terms
*   **TTFT:** Time to First Token. The latency between the user hitting "Enter" and the first character appearing.
*   **TPS:** Tokens Per Second. The speed at which the model generates text.
*   **Reasoning Budget:** The proportion of total tokens allocated to the internal "thinking" process of the model.
*   **Artifact:** A persistent file containing a full, un-truncated LLM response.
*   **WAL Mode:** Write-Ahead Logging. A journaling mode for SQLite that allows multiple readers and one writer to operate simultaneously.
*   **Redaction:** The process of identifying and masking sensitive information (PII) from text.
*   **Campaign:** A collection of benchmark prompts run against a specific model profile to gather aggregate performance data.
*   **Profile:** A versioned set of hyperparameters (temperature, top_p, system prompt) used to ensure benchmark reproducibility.
*   **Proxy:** A middleware service that sits between the user and the inference engine to handle logging, redaction, and telemetry.
*   **Flight Recorder:** The overarching project name, signifying the system's role in recording and "replaying" LLM interactions for analysis.

---

## 11. Final Deployment Checklist
- [ ] **Database:** SQLite WAL mode confirmed (`PRAGMA journal_mode=WAL;`).
- [ ] **Volumes:** Docker volumes for `/models/flight-recorder/data` are mounted with `rw` permissions.
- [ ] **Network:** `llama-cpp-server` is reachable from the `ai-flight-recorder` container.
- [ ] **Redaction:** All 4 primary regex patterns are verified against the test suite.
- [ ] **Artifacts:** The `ARTIFACT_PATH` is writable and has sufficient quota.
- [ ] **Profiles:** At least one `model_profile` is initialized with a valid `version_hash`.
- [ ] **Benchmarks:** A "Smoke Test" campaign has been run and successfully completed.
- [ ] **Dashboard:** The dashboard successfully loads and displays the "Smoke Test" results.
- [ ] **Rollback:** A backup of the initial `database.db` is stored in a secure location.
- [ ] **Security:** The `X-API-Key` is set in the environment and is not hardcoded in the source.

---

## 12. Migration Script (SQL)
To be executed during the transition from the prototype to the production-ready version.

```sql
-- 1. Create the new model_profiles table
CREATE TABLE model_profiles (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    model_path TEXT NOT NULL,
    temperature REAL,
    top_p REAL,
    max_tokens INTEGER,
    reasoning_enabled BOOLEAN,
    system_prompt TEXT,
    version_hash TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- 2. Add columns to llm_requests
ALTER TABLE llm_requests ADD COLUMN artifact_id TEXT;
ALTER TABLE llm_requests ADD COLUMN reasoning_tokens INTEGER DEFAULT 0;
ALTER TABLE llm_requests ADD COLUMN content_tokens INTEGER DEFAULT 0;
ALTER TABLE llm_requests ADD COLUMN redacted_flag BOOLEAN DEFAULT FALSE;
ALTER TABLE llm_requests ADD COLUMN profile_id INTEGER;

-- 3. Create benchmark_campaigns and benchmark_runs
CREATE TABLE benchmark_campaigns (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    description TEXT,
    created_by TEXT,
    status TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE benchmark_runs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_id INTEGER REFERENCES benchmark_campaigns(id),
    profile_id INTEGER REFERENCES model_profiles(id),
    start_time TIMESTAMP,
    end_time TIMESTAMP,
    avg_ttft REAL,
    avg_tps REAL,
    total_reasoning_tokens INTEGER,
    total_content_tokens INTEGER,
    metadata JSON
);

-- 4. Indexing for performance
CREATE INDEX idx_llm_requests_profile ON llm_requests(profile_id);
CREATE INDEX idx_llm_requests_artifact ON llm_requests(artifact_id);
CREATE INDEX idx_benchmark_runs_campaign ON benchmark_runs(campaign_id);
```

---

## 13. Example Configuration File (`config.yaml`)
This file will be used to initialize the `app/proxy.py` and `app/bench.py` settings.

```yaml
proxy:
  llama_server_url: "http://192.168.1.116:8080"
  request_timeout: 60
  max_workers: 10
  redaction:
    enabled: true
    level: 2
    custom_patterns:
      - "INTERNAL_PROJECT_CODE"
  artifacts:
    enabled: true
    max_length: 2048
    base_path: "/models/flight-recorder/data/artifacts"

database:
  path: "/models/flight-recorder/data/database.db"
  wal_mode: true
  timeout: 30

benchmarking:
  default_profile: "qwen-35b-balanced"
  parallel_workers: 5
  report_interval: 5
  save_screenshots: true

logging:
  level: "INFO"
  format: "json"
  file_path: "/models/flight-recorder/data/logs/app.log"
```

---

## 14. Detailed Benchmark Runner Logic (Python Snippet)
This is the core logic for `app/bench.py` to ensure robust execution.

```python
import logging
from typing import List, Dict
from app.store import update_benchmark_progress, update_campaign_status, finalize_benchmark_run
from app.proxy import call_proxy_api

logger = logging.getLogger(__name__)

class BenchmarkRunner:
    def __init__(self, campaign_id: int, profile_id: int, prompts: List[str]):
        self.campaign_id = campaign_id
        self.profile_id = profile_id
        self.prompts = prompts
        self.results = []

    def execute(self):
        logger.info(f"Starting campaign {self.campaign_id} with profile {self.profile_id}")
        update_campaign_status(self.campaign_id, "RUNNING")
        
        start_time = time.time()
        success_count = 0
        failure_count = 0

        for i, prompt in enumerate(self.prompts):
            try:
                logger.info(f"Processing prompt {i+1}/{len(self.prompts)}")
                response = call_proxy_api(prompt, self.profile_id)
                self.results.append(response)
                success_count += 1
                
                # Update progress every 5 prompts
                if (i + 1) % 5 == 0:
                    update_benchmark_progress(self.campaign_id, i + 1, len(self.prompts))
            except Exception as e:
                logger.error(f"Failed at prompt {i+1}: {str(e)}")
                failure_count += 1
                # We continue the run even if one prompt fails, but we log it
                continue

        total_time = time.time() - start_time
        
        # Calculate metrics
        avg_tps = self._calculate_tps(total_time)
        avg_ttft = self._calculate_ttft()
        
        finalize_benchmark_run(
            self.campaign_id, 
            self.profile_id, 
            avg_tps, 
            avg_ttft, 
            self.results
        )
        
        status = "COMPLETED" if failure_count == 0 else "PARTIAL_SUCCESS"
        update_campaign_status(self.campaign_id, status)
        logger.info(f"Campaign {self.campaign_id} finished with status {status}")

    def _calculate_tps(self, total_time: float) -> float:
        total_tokens = sum(r.get('content_tokens', 0) + r.get('reasoning_tokens', 0) for r in self.results)
        return total_tokens / total_time if total_time > 0 else 0

    def _calculate_ttft(self) -> float:
        ttfts = [r.get('ttft_ms', 0) for r in self.results if r.get('ttft_ms') is not None]
        return sum(ttfts) / len(ttfts) if ttfts else 0
```

---

## 15. Final Acceptance Test Suite (Expanded)
These tests are to be run using `pytest` before the final production push.

### Test Suite: `tests/test_production_readiness.py`

```python
import pytest
from app.proxy import RedactionEngine, ArtifactManager
from app.store import init_db

# --- Redaction Tests ---
def test_redaction_email():
    engine = RedactionEngine()
    input_text = "Contact me at secret@company.com for details."
    output = engine.scrub(input_text)
    assert "[REDACTED_EMAIL]" in output
    assert "secret@company.com" not in output

def test_redaction_phone():
    engine = RedactionEngine()
    input_text = "Call me at 555-0199."
    output = engine.scrub(input_text)
    assert "[REDACTED_PHONE]" in output

def test_redaction_teams_id():
    engine = RedactionEngine()
    input_text = "User ID: ABC123XYZ999"
    output = engine.scrub(input_text)
    assert "[REDACTED_TEAMS_ID]" in output

# --- Artifact Tests ---
def test_artifact_storage_creation():
    manager = ArtifactManager("/tmp/test_artifacts")
    data = {"content": "A" * 3000, "meta": "test"}
    artifact_id = manager.save_large_response(data)
    assert artifact_id.startswith("req_")
    
    # Verify retrieval
    retrieved = manager.get_artifact(artifact_id)
    assert retrieved["content"] == "A" * 3000

def test_artifact_non_existent():
    manager = ArtifactManager("/tmp/test_artifacts")
    assert manager.get_artifact("invalid_id") == {}

# --- Database Tests ---
def test_db_initialization():
    init_db()
    # Check if WAL mode is active
    import sqlite3
    conn = sqlite3.connect("/models/flight-recorder/data/database.db")
    cursor = conn.cursor()
    cursor.execute("PRAGMA journal_mode;")
    mode = cursor.fetchone()[0]
    assert "wal" in mode.lower()
    conn.close()

# --- Proxy Logic Tests ---
def test_reasoning_parser_logic():
    # Mocking a raw response from llama.cpp
    raw_response = "<thought>I should calculate the area of the circle.</thought>The area is 3.14 * r^2."
    # Logic to be tested:
    thought_part = raw_response.split("<thought>")[1].split("</thought>")[0]
    content_part = raw_response.split("</thought>")[1]
    
    assert len(thought_part) > 0
    assert "The area is" in content_part

# --- Benchmark Runner Tests ---
def test_benchmark_progress_update(mocker):
    # Mock the update_benchmark_progress function
    mocker.patch("app.store.update_benchmark_progress")
    from app.bench import BenchmarkRunner
    
    runner = BenchmarkRunner(campaign_id=1, profile_id=1, prompts=["Test prompt"])
    runner.execute()
    
    # Verify that progress was updated
    assert mocker.patch("app.store.update_benchmark_progress").called

# --- Performance Tests ---
def test_dashboard_query_speed():
    import time
    # Simulate 1000 records
    # Measure time to run a complex aggregation
    start = time.time()
    # (Simulated query)
    time.sleep(0.01) 
    end = time.time()
    assert (end - start) < 0.5 # Should be very fast with indexing

# --- Security Tests ---
def test_api_key_rejection():
    # Mock a request without an X-API-Key header
    # Verify that the proxy returns 401 Unauthorized
    pass

def test_redaction_recursive_json():
    engine = RedactionEngine()
    data = {
        "user": "John Doe",
        "contact": {
            "email": "john@doe.com",
            "phone": "123-456-7890"
        },
        "tags": ["work", "private@email.com"]
    }
    scrubbed = engine.scrub_json(data)
    assert "[REDACTED_EMAIL]" in scrubbed["contact"]["email"]
    assert "[REDACTED_PHONE]" in scrubbed["contact"]["phone"]
    assert "[REDACTED_EMAIL]" in scrubbed["tags"][1]

# --- Concurrency Tests ---
def test_sqlite_concurrent_writes(mocker):
    # Simulate 5 threads writing to the DB simultaneously
    # Verify no "Database is locked" errors occur
    pass

# --- Model Profile Tests ---
def test_profile_versioning():
    # Verify that changing a profile creates a new version_hash
    pass

# --- Timeout Tests ---
def test_proxy_timeout_handling():
    # Mock a slow llama.cpp response
    # Verify that the proxy returns a 504 Gateway Timeout
    pass

# --- UI Rendering Tests ---
def test_dashboard_json_payload_size():
    # Verify that the dashboard API does not return the full artifact content
    # but only the artifact_id and a snippet.
    pass

# --- Data Integrity Tests ---
def test_artifact_id_uniqueness():
    manager = ArtifactManager("/tmp/test_artifacts")
    ids = set()
    for _ in range(100):
        ids.add(manager.save_large_response({"data": "test"}))
    assert len(ids) == 100

# --- Redaction Consistency ---
def test_redaction_does_not_corrupt_non_pii():
    engine = RedactionEngine()
    text = "The project is called 'Apollo'. No emails here."
    assert engine.scrub(text) == text

# --- Benchmark Aggregation Tests ---
def test_tps_calculation():
    # Verify that TPS calculation handles zero time correctly
    # and correctly sums tokens.
    pass

# --- Log Format Tests ---
def test_log_is_json():
    # Verify that the logger outputs valid JSON
    pass

# --- Environment Variable Tests ---
def test_env_vars_loaded():
    import os
    assert os.getenv("LLAMA_SERVER_URL") is not None

# --- Volume Mount Tests ---
def test_artifact_path_writable():
    # Verify that the app can write to the designated artifact path
    pass

# --- Final Integration Test ---
def test_full_request_flow():
    # Simulate a full request from user -> proxy -> llama.cpp -> proxy -> db -> response
    pass
```

---

## 16. Deployment Script (`deploy.sh`)
A script to automate the deployment and migration.

```bash
#!/bin/bash
set -e

echo "Starting Deployment of AI Flight Recorder..."

# 1. Backup existing data
if [ -f "./data/database.db" ]; then
    echo "Backing up database..."
    cp ./data/database.db ./data/database_backup_$(date +%Y%m%d).db
fi

# 2. Run Migrations
echo "Running database migrations..."
python3 -m app.migrations.upgrade_v2

# 3. Build and Start Containers
echo "Building Docker images..."
docker-compose build

echo "Starting services..."
docker-compose up -d

# 4. Verify Health
echo "Verifying health..."
sleep 10
curl -f http://localhost:8000/health || (echo "Health check failed!" && exit 1)

echo "Deployment successful."
```

---

## 17. Final Summary of Engineering Decisions
*   **SQLite WAL Mode:** Chosen over PostgreSQL to maintain the simplicity of a single-file home-lab setup while solving the primary concurrency bottleneck.
*   **File-based Artifacts:** Chosen over `TEXT` columns in SQLite to ensure the dashboard remains responsive and the database remains lean.
*   **Regex Redaction:** Chosen as the first line of defense for PII. It is deterministic and fast.
*   **Versioned Profiles:** Chosen to ensure that every benchmark result is tied to a specific, immutable configuration of the LLM.
*   **Campaign Runner:** Chosen to allow for structured, repeatable benchmarking that can be easily compared over time.

This dossier provides the complete blueprint for the production-grade deployment of the AI Flight Recorder. All components are designed to be modular, scalable, and auditable.