# Engineering Dossier: AI Flight Recorder Infrastructure Upgrade
**Project Codename:** Project Sentinel
**Status:** Pre-Deployment Review
**Target Environment:** Private Home-Lab (Docker-based)
**Primary Model:** Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf

---

# Executive Summary
The "AI Flight Recorder" is a critical infrastructure component designed to provide observability, benchmarking, and auditing for LLM interactions within a private environment. Currently, the system serves as a proxy between a frontend/benchmark suite and a `llama.cpp` backend. While functional, the current architecture faces significant scaling bottlenecks regarding large-scale benchmark artifacts, lack of granular reasoning metrics, and potential PII leakage.

This dossier outlines a transition from a "monolithic-log" approach to a "decoupled-artifact" architecture. Key improvements include:
1.  **Artifact Externalization:** Moving large JSON outputs from SQLite to a structured filesystem to prevent database bloat and UI lag.
2.  **Reasoning Telemetry:** Implementing a dedicated parsing layer to isolate "thinking" tokens from "response" tokens for accurate cost/performance analysis.
3.  **Privacy Guardrails:** A middleware redaction layer to scrub PII (Emails, Teams IDs) before persistence.
4.  **Auditability:** A versioned Model Profile system to track changes in system prompts, temperature, and quantization parameters.
5.  **Robust Benchmarking:** A stateful runner capable of handling 10k+ token outputs with progress tracking and automatic retry logic.

---

# Current Architecture
The current system operates as a synchronous proxy chain:
1.  **Client/Bench Runner:** Sends requests to the FastAPI Proxy.
2.  **FastAPI Proxy (`app/proxy.py`):** Receives the request, identifies the model, forwards it to `llama.cpp` (192.168.1.116), and streams the response.
3.  **Persistence Layer (`app/store.py`):** Captures the full request/response cycle and writes it into a single SQLite row in the `llm_requests` table.
4.  **Reporting (`app/reports.py`):** Queries the `llm_requests` table to aggregate metrics for the dashboard.
5.  **Storage:** All data resides in `/models/flight-recorder/data/flight_recorder.db`.

**Current Limitations:**
- **SQLite Bloat:** Storing 20,000-token responses in a `TEXT` column causes significant overhead during `SELECT *` queries for the dashboard.
- **Timeout Vulnerability:** Long-running benchmarks (e.g., "Write a 5,000-word technical manual") often trigger gateway timeouts because the proxy waits for the full response before finalizing the DB write.
- **Reasoning Obscurity:** Reasoning tokens are currently mixed with completion tokens, making it impossible to calculate "Reasoning Efficiency" (Reasoning Tokens / Completion Tokens).

---

# Failure Modes Found

### 1. Database Serialization Failure (Critical)
When a benchmark generates a massive output (e.g., a full codebase), the `store.py` helper attempts to insert a multi-megabyte string into SQLite. This can lead to `SQLITE_FULL` errors or extreme latency in the `INSERT` transaction, potentially causing the proxy to hang and drop the connection.

### 2. PII Leakage (High)
The proxy currently records the raw input. If a user pastes a Teams message containing a private email or a corporate ID, that data is persisted in plain text in the SQLite database. This violates the "Private Home-Lab" security posture.

### 3. Model Profile Drift (Medium)
Model parameters (temperature, top_p, repeat_penalty) are currently passed as request-time overrides or hardcoded in the proxy. There is no "Source of Truth" for what the "Standard Qwen3.6 Profile" actually is, making it impossible to reproduce a benchmark from three weeks ago.

### 4. Reasoning Token Inflation (Medium)
Because `llama.cpp` reasoning is enabled, the "thinking" process can consume significant context window. Without a way to separate these tokens, the "Tokens Per Second" metric is skewed, as it includes the time spent "thinking" which may not be desired in the final "Response Speed" metric.

### 5. UI Rendering Bottleneck (Low)
The dashboard attempts to load the last 50 requests. If those requests contain 10k tokens each, the browser's DOM will struggle to render the text blocks, leading to a sluggish user experience.

---

# Data Model Changes

To resolve the identified failure modes, the following schema migrations are required.

### New Table: `model_profiles`
Tracks the "Golden Configuration" for each model.
```sql
CREATE TABLE model_profiles (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    model_name TEXT NOT NULL, -- e.g., "Qwen3.6-35B-A3B-APEX"
    version TEXT NOT NULL,     -- e.g., "v1.0.2"
    system_prompt TEXT,
    temperature REAL,
    top_p REAL,
    repeat_penalty REAL,
    max_tokens INTEGER,
    reasoning_enabled BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

### Modified Table: `llm_requests`
Remove the `full_response` column and replace it with an `artifact_id`.
```sql
ALTER TABLE llm_requests RENAME TO llm_requests_old;

CREATE TABLE llm_requests (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT,
    model_profile_id INTEGER,
    request_prompt TEXT,
    redacted_prompt TEXT,       -- The prompt after PII scrubbing
    reasoning_content TEXT,      -- Isolated "thinking" block
    completion_content TEXT,     -- Isolated "answer" block
    reasoning_tokens INTEGER,
    completion_tokens INTEGER,
    total_tokens INTEGER,
    latency_ms INTEGER,
    artifact_path TEXT,          -- Path to the full JSONL file
    status TEXT,                 -- 'success', 'failed', 'timeout'
    error_message TEXT,
    FOREIGN KEY(model_profile_id) REFERENCES model_profiles(id)
);
```

### New Table: `benchmark_runs`
To group multiple requests into a single "Campaign."
```sql
CREATE TABLE benchmark_runs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_name TEXT,
    description TEXT,
    total_requests INTEGER,
    success_count INTEGER,
    avg_latency REAL,
    metadata JSON,               -- Store screenshots, hardware stats, etc.
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

---

# Artifact Storage Design

To handle the "10,000+ token" problem, we will implement a **Sidecar Artifact Pattern**.

### Storage Logic:
1.  **Database:** Stores metadata, IDs, and small snippets (first 200 characters).
2.  **Filesystem:** Stores the full, raw JSON output in a structured directory.

### Directory Structure:
`/models/flight-recorder/data/artifacts/{run_id}/{timestamp}_{request_id}.jsonl`

### Implementation Detail:
The `app/store.py` will be updated with an `ArtifactManager` class:
```python
class ArtifactManager:
    def __init__(self, base_path: str):
        self.base_path = base_path

    def save_artifact(self, request_id: str, data: dict) -> str:
        path = f"{self.base_path}/{request_id}/"
        os.makedirs(path, exist_ok=True)
        file_path = f"{path}{request_id}.jsonl"
        with open(file_path, 'w') as f:
            f.write(json.dumps(data) + "\n")
        return file_path

    def get_artifact_preview(self, path: str) -> str:
        # Returns only the first 500 chars for the UI
        with open(path, 'r') as f:
            return f.read(500) + "..."
```

---

# Reasoning Budget Handling

To measure reasoning without turning it off, the proxy must implement a **Reasoning Parser**.

### The Logic:
`llama.cpp` often outputs reasoning within specific tags (e.g., `<thought>...</thought>`) or as a distinct prefix.

1.  **Extraction:** The proxy will scan the stream for the reasoning delimiter.
2.  **Counting:**
    *   `reasoning_tokens`: Count of tokens inside the thinking block.
    *   `completion_tokens`: Count of tokens after the thinking block.
3.  **Metrics:**
    *   **Reasoning Ratio:** `reasoning_tokens / total_tokens`.
    *   **Thinking Latency:** Time elapsed from request start until the closing `</thought>` tag.

### Example Schema for Reasoning:
| Request ID | Reasoning Tokens | Completion Tokens | Thinking Latency | Reasoning Ratio |
| :--- | :--- | :--- | :--- | :--- |
| req_001 | 450 | 120 | 4200ms | 0.78 |
| req_002 | 12 | 800 | 150ms | 0.01 |

---

# Benchmark Runner Design

The `app/bench.py` script will be upgraded from a simple loop to a **Stateful Campaign Runner**.

### Features:
1.  **Checkpointing:** If a benchmark of 100 prompts fails at #50, the runner should resume from #51.
2.  **Concurrency Control:** Ability to run `N` concurrent requests to stress-test the `llama.cpp` server.
3.  **Progress Reporting:** A `progress.json` file updated in real-time, allowing a frontend to show a progress bar.

### CLI Example:
```bash
python app/bench.py \
  --campaign "Qwen3.6-Reasoning-Stress" \
  --model "Qwen3.6-35B-A3B-APEX" \
  --prompts ./prompts/long_form_tasks.jsonl \
  --concurrency 2 \
  --output-dir ./results/2023-10-27 \
  --report-on-completion
```

### State Machine:
- `PENDING`: Prompt queued.
- `RUNNING`: Request sent to proxy, awaiting stream.
- `COMPLETED`: Response received, artifact saved.
- `FAILED`: Error caught (timeout, 500, etc.).
- `RETRYING`: Automatic retry (max 3 attempts).

---

# Reporting Plane

The dashboard will be updated to support **Comparative Analysis**.

### New Dashboard Metrics:
1.  **Throughput (Tokens/Sec):** Calculated as `(completion_tokens / completion_latency)`.
2.  **Reasoning Overhead:** The delta between "Total Latency" and "Completion Latency."
3.  **Success Rate:** Percentage of requests that didn't hit the `max_tokens` limit or a timeout.
4.  **Cost Efficiency:** (Optional) Estimated cost based on token counts.

### Comparison View:
A "Model vs. Model" view will allow the user to select two `benchmark_runs` and overlay:
- Latency (Bar Chart)
- Reasoning Ratio (Line Chart)
- Success Rate (Gauge)

---

# Privacy And Redaction

To ensure no private data leaks into the SQLite database, a **Redaction Middleware** will be inserted into `app/proxy.py`.

### Redaction Engine:
The engine will use a multi-pass regex approach.

```python
import re

class RedactionEngine:
    def __init__(self):
        self.patterns = {
            "email": r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}',
            "teams_id": r'\[\w+\]\s\d+', # Example Teams ID pattern
            "ipv4": r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b',
            "phone": r'\+?\d{1,3}?[-.\s]?\(?\d{1,3}?\)?[-.\s]?\d{3,4}[-.\s]?\d{4}'
        }

    def redact(self, text: str) -> str:
        redacted_text = text
        for label, pattern in self.patterns.items():
            redacted_text = re.sub(pattern, f"[REDACTED_{label.upper()}]", redacted_text)
        return redacted_text
```

### Implementation Flow:
1.  **Request Received** $\rightarrow$ **Redaction Engine** $\rightarrow$ **Store Redacted Prompt in DB**.
2.  **Original Prompt** $\rightarrow$ **Forward to llama.cpp**.
3.  **Response Received** $\rightarrow$ **Redaction Engine** $\rightarrow$ **Store Redacted Response in DB**.

---

# Deployment Plan

### Phase 1: Database Migration
1.  Stop the `ai-flight-recorder` container.
2.  Run the migration script: `python app/migrations/upgrade_v2.py`.
    *   This script will create the new tables and move data from `llm_requests` to the new schema.
3.  Verify row counts match.

### Phase 2: Infrastructure Update
1.  Update `deploy/docker-compose.yml` to include a new volume for artifacts:
    ```yaml
    volumes:
      - /models/flight-recorder/data:/data
      - /models/flight-recorder/artifacts:/artifacts
    ```
2.  Update environment variables:
    *   `ARTIFACT_PATH=/artifacts`
    *   `REDACTION_ENABLED=true`
    *   `MODEL_PROFILE_ID=1`

### Phase 3: Rolling Update
1.  `docker-compose down`
2.  `docker-compose up -d --build`
3.  Verify the proxy is reachable and `llama.cpp` is responding.

---

# Test Plan

### Unit Tests (Python)
1.  **Redaction Test:** Verify `test@example.com` becomes `[REDACTED_EMAIL]`.
2.  **Redaction Test:** Verify Teams IDs are correctly identified.
3.  **Artifact Manager Test:** Verify file creation and path return.
4.  **Reasoning Parser Test:** Verify `<thought>` blocks are correctly split.
5.  **Schema Test:** Verify `model_profiles` inserts correctly.

### Integration Tests
6.  **Proxy-to-Llama Flow:** Send a request, verify 200 OK and DB entry.
7.  **Timeout Test:** Send a request that exceeds 60s, verify `status='timeout'` in DB.
8.  **Artifact Persistence:** Verify a 10k token response creates a file in `/artifacts`.
9.  **Profile Switch:** Change `MODEL_PROFILE_ID` and verify parameters change.

### Acceptance Tests (User Stories)
10. **User Story 1:** "As a researcher, I can run a 100-prompt benchmark and see a progress bar."
11. **User Story 2:** "As a privacy officer, I can verify that no emails appear in the SQLite database."
12. **User Story 3:** "As a developer, I can compare the reasoning speed of Qwen3.6 vs a previous model."
13. **User Story 4:** "As a user, I can view a preview of a 5,000-word response without the UI freezing."
14. **User Story 5:** "As an admin, I can audit who changed the system prompt for the Qwen model."
15. **User Story 6:** "As a user, I can resume a failed benchmark from the middle."
16. **User Story 7:** "As a user, I can see the 'Reasoning Ratio' for every request."
17. **User Story 8:** "As a user, I can see a 'Success Rate' percentage for a campaign."
18. **User Story 9:** "As a user, I can see the 'Thinking Latency' vs 'Completion Latency'."
19. **User Story 10:** "As a user, I can see a list of all active model profiles."
20. **User Story 11:** "As a user, I can see a 'Redacted' version of the prompt in the dashboard."
21. **User Story 12:** "As a user, I can see a 'Full Artifact' link that opens the JSONL file."
22. **User Story 13:** "As a user, I can see a 'Reasoning Content' block separately from the answer."
23. **User Story 14:** "As a user, I can see a 'Total Tokens' count for every request."
24. **User Story 15:** "As a user, I can see a 'Latency' bar chart for a specific campaign."
25. **User Story 16:** "As a user, I can see a 'Reasoning Overhead' metric."
26. **User Story 17:** "As a user, I can see a 'Model Version' tag on every request."
27. **User Story 18:** "As a user, I can see a 'Status' indicator (Success/Fail/Timeout)."
28. **User Story 19:** "As a user, I can see an 'Error Message' if a request fails."
29. **User Story 20:** "As a user, I can see a 'Created At' timestamp for every request."

---

# Rollback Plan

### Scenario 1: Database Migration Failure
1.  **Action:** Restore `flight_recorder.db` from the pre-migration snapshot.
2.  **Command:** `cp /models/flight-recorder/data/flight_recorder.db.bak /models/flight-recorder/data/flight_recorder.db`
3.  **Verification:** Check that the old `llm_requests` table is populated.

### Scenario 2: Proxy Connectivity Issues
1.  **Action:** Revert `docker-compose.yml` to the previous image tag.
2.  **Command:** `docker-compose down && docker-compose up -d`
3.  **Verification:** Ensure the proxy can reach `192.168.1.116`.

### Scenario 3: Redaction Engine Over-Aggression
1.  **Action:** Toggle `REDACTION_ENABLED=false` in the environment variables.
2.  **Command:** `docker-compose up -d`
3.  **Verification:** Confirm that raw text is appearing in the database (for debugging purposes only).

---

# Open Questions

1.  **Hardware Limits:** How will the system handle concurrent requests if the GPU VRAM is saturated by `llama.cpp`? (Proposed: Implement a request queue in the proxy).
2.  **Context Window:** Should we implement a "Context Window Warning" in the UI if a benchmark prompt is too close to the model's limit?
3.  **Multi-Model Support:** Should we support multiple `llama.cpp` instances on different ports?
4.  **Artifact Cleanup:** How long should we retain artifacts? (Proposed: A cron job to delete artifacts older than 30 days).
5.  **Advanced Redaction:** Do we need to support "Named Entity Recognition" (NER) for more complex redaction (e.g., names of people)?
6.  **Streaming Artifacts:** Should we stream the artifact to the filesystem *during* the generation, or only at the end? (Proposed: Stream to a temporary file and rename on completion to ensure atomicity).
7.  **Reasoning Delimiters:** Does `llama.cpp` have a standard for reasoning tags, or do we need to support multiple formats (e.g., `<thought>`, `[THINK]`, etc.)?
8.  **Dashboard Performance:** If we have 10,000+ requests, how do we implement pagination for the dashboard?
9.  **Model Profile Versioning:** Should we allow "Draft" profiles that aren't active but can be used for testing?
10. **Log Rotation:** How should we handle the logs of the proxy itself (standard out/error)?

11. **Log Rotation and Observability:**
To ensure the proxy doesn't consume all available disk space with standard output logs, we will implement a rotation policy using `RotatingFileHandler` or a sidecar container like `Fluentbit`.
- **Log Levels:** `DEBUG` for development, `INFO` for production.
- **Rotation Policy:** Max 10 files, 100MB each.
- **Metrics Export:** Implement a `/metrics` endpoint for Prometheus to track:
    - `proxy_request_count_total` (labeled by status_code, model_id)
    - `proxy_request_latency_seconds` (histogram)
    - `proxy_tokens_generated_total` (counter)
    - `proxy_reasoning_tokens_total` (counter)
    - `proxy_artifact_creation_success` (counter)

---

# Detailed Migration Script (Python)
The following script (`app/migrations/upgrade_v2.py`) must be executed to transition the database from the legacy flat structure to the new relational/artifact-based structure.

```python
import sqlite3
import os
import json
import logging

# Configuration
DB_PATH = "/models/flight-recorder/data/flight_recorder.db"
LOG_FILE = "/models/flight-recorder/data/migration.log"

logging.basicConfig(
    filename=LOG_FILE,
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

def migrate():
    if not os.path.exists(DB_PATH):
        logging.error(f"Database not found at {DB_PATH}")
        return

    conn = sqlite3.connect(DB_PATH)
    cursor = conn.cursor()

    try:
        logging.info("Starting migration to Version 2.0...")

        # 1. Create new tables
        logging.info("Creating new tables...")
        cursor.executescript("""
            CREATE TABLE IF NOT EXISTS model_profiles (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                model_name TEXT NOT NULL,
                version TEXT NOT NULL,
                system_prompt TEXT,
                temperature REAL,
                top_p REAL,
                repeat_penalty REAL,
                max_tokens INTEGER,
                reasoning_enabled BOOLEAN,
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            );

            CREATE TABLE IF NOT EXISTS benchmark_runs (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                campaign_name TEXT,
                description TEXT,
                total_requests INTEGER,
                success_count INTEGER,
                avg_latency REAL,
                metadata JSON,
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            );

            CREATE TABLE IF NOT EXISTS llm_requests (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                session_id TEXT,
                model_profile_id INTEGER,
                request_prompt TEXT,
                redacted_prompt TEXT,
                reasoning_content TEXT,
                completion_content TEXT,
                reasoning_tokens INTEGER,
                completion_tokens INTEGER,
                total_tokens INTEGER,
                latency_ms INTEGER,
                artifact_path TEXT,
                status TEXT,
                error_message TEXT,
                FOREIGN KEY(model_profile_id) REFERENCES model_profiles(id)
            );
        """)

        # 2. Migrate existing data from old table to new table
        # Assuming the old table was named 'llm_requests_old' or we rename it now
        logging.info("Renaming old table...")
        cursor.execute("ALTER TABLE llm_requests RENAME TO llm_requests_old;")

        logging.info("Migrating data from llm_requests_old to llm_requests...")
        # We need to handle the fact that the old table had 'full_response'
        # and we need to split it into reasoning/completion if possible.
        cursor.execute("""
            INSERT INTO llm_requests (
                session_id, 
                model_profile_id, 
                request_prompt, 
                redacted_prompt, 
                reasoning_content, 
                completion_content, 
                reasoning_tokens, 
                completion_tokens, 
                total_tokens, 
                latency_ms, 
                artifact_path, 
                status, 
                error_message
            )
            SELECT 
                session_id,
                1, -- Default to first profile for legacy data
                request_prompt,
                request_prompt, -- Redaction will be applied on next write
                NULL, -- Legacy data doesn't have split reasoning
                full_response,
                0, 
                0, 
                0, 
                latency_ms,
                NULL,
                'success',
                NULL
            FROM llm_requests_old;
        """)

        # 3. Seed initial model profile
        logging.info("Seeding initial model profiles...")
        cursor.execute("""
            INSERT INTO model_profiles (model_name, version, system_prompt, temperature, top_p, repeat_penalty, max_tokens, reasoning_enabled)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?)
        """, ("Qwen3.6-35B-A3B-APEX", "v1.0.0", "You are a helpful assistant.", 0.7, 0.9, 1.1, 4096, 1))

        conn.commit()
        logging.info("Migration completed successfully.")

    except Exception as e:
        conn.rollback()
        logging.error(f"Migration failed: {str(e)}")
        raise
    finally:
        conn.close()

if __name__ == "__main__":
    migrate()
```

---

# Proxy Middleware Implementation (Code)
This module handles the core logic of the proxy: Redaction, Reasoning Parsing, and Artifact Persistence.

```python
import re
import json
import time
import uuid
import os
from typing import Dict, Any, Tuple
from fastapi import Request
from pydantic import BaseModel

# Configuration Constants
ARTIFACT_BASE_PATH = os.getenv("ARTIFACT_PATH", "/artifacts")
REDACTION_ENABLED = os.getenv("REDACTION_ENABLED", "true").lower() == "true"

class RedactionEngine:
    """Handles PII scrubbing for both inputs and outputs."""
    def __init__(self):
        self.patterns = {
            "email": r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}',
            "teams_id": r'\[\w+\]\s\d+',
            "ipv4": r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b',
            "phone": r'\+?\d{1,3}?[-.\s]?\(?\d{1,3}?\)?[-.\s]?\d{3,4}[-.\s]?\d{4}'
        }

    def redact(self, text: str) -> str:
        if not text:
            return text
        redacted_text = text
        for label, pattern in self.patterns.items():
            redacted_text = re.sub(pattern, f"[REDACTED_{label.upper()}]", redacted_text)
        return redacted_text

class ReasoningParser:
    """Extracts thinking blocks from llama.cpp streams."""
    @staticmethod
    def parse_stream(stream_generator) -> Tuple[str, str, int, int]:
        """
        Parses a stream and splits it into reasoning and completion.
        Returns: (reasoning_text, completion_text, reasoning_tokens, completion_tokens)
        """
        reasoning_text = ""
        completion_text = ""
        is_reasoning = False
        
        # This is a simplified logic; actual implementation would handle 
        # chunked stream processing and token counting.
        for chunk in stream_generator:
            if "<thought>" in chunk:
                is_reasoning = True
            if "</thought>" in chunk:
                is_reasoning = False
            
            if is_reasoning:
                reasoning_text += chunk
            else:
                completion_text += chunk
                
        # Token counting (simplified)
        r_tokens = len(reasoning_text.split())
        c_tokens = len(completion_text.split())
        
        return reasoning_text, completion_text, r_tokens, c_tokens

class ArtifactManager:
    """Handles saving large JSONL artifacts to the filesystem."""
    def __init__(self, base_path: str):
        self.base_path = base_path

    def save(self, request_id: str, data: Dict[str, Any]) -> str:
        path = os.path.join(self.base_path, request_id)
        os.makedirs(path, exist_ok=True)
        file_path = os.path.join(path, f"{request_id}.jsonl")
        
        with open(file_path, 'w') as f:
            f.write(json.dumps(data) + "\n")
        return file_path

# Global instances
redactor = RedactionEngine()
artifact_mgr = ArtifactManager(ARTIFACT_BASE_PATH)

async def process_request(request_data: Dict[str, Any]):
    """
    Main processing pipeline for the proxy.
    1. Redact Input
    2. Call llama.cpp
    3. Parse Reasoning
    4. Save Artifact
    5. Save Metadata to DB
    """
    start_time = time.time()
    request_id = str(uuid.uuid4())
    
    # 1. Redaction
    raw_prompt = request_data.get("prompt", "")
    redacted_prompt = redactor.redact(raw_prompt) if REDACTION_ENABLED else raw_prompt
    
    # 2. Call llama.cpp (Mocked for this example)
    # In reality, this would be an aiohttp call to 192.168.1.116:8080
    # response_stream = await call_llama_cpp(redacted_prompt)
    
    # 3. Parse Reasoning
    # reasoning_text, completion_text, r_tokens, c_tokens = ReasoningParser.parse_stream(response_stream)
    
    # 4. Save Artifact
    artifact_data = {
        "request_id": request_id,
        "prompt": raw_prompt,
        "redacted_prompt": redacted_prompt,
        "reasoning": reasoning_text,
        "completion": completion_text,
        "timestamp": time.time()
    }
    path = artifact_mgr.save(request_id, artifact_data)
    
    # 5. DB Update
    # db.execute("INSERT INTO llm_requests ...", (..., path, ...))
    
    latency = (time.time() - start_time) * 1000
    return {"request_id": request_id, "latency": latency, "artifact_path": path}
```

---

# Benchmark Runner State Machine (Detailed Logic)
The `app/bench.py` runner will follow a strict state machine to ensure durability during long-running campaigns.

### State Definitions:
- **IDLE:** Waiting for user input or campaign start.
- **INITIALIZING:** Validating prompt files, checking database connectivity, and verifying model profile existence.
- **QUEUING:** Adding prompts to an internal priority queue.
- **EXECUTING:** Actively sending requests to the proxy.
- **PAUSED:** Execution halted by user or due to a rate limit/error.
- **RETRYING:** A request failed and is being re-attempted (max 3).
- **COMPLETING:** Finalizing the `benchmark_runs` table entry and generating the final report.
- **FAILED:** A critical error occurred (e.g., DB connection lost).

### Retry Logic:
- **Transient Errors (Retryable):** 503 Service Unavailable, 504 Gateway Timeout, Connection Refused.
- **Permanent Errors (Non-Retryable):** 400 Bad Request, 401 Unauthorized, 404 Not Found.

### Progress Tracking:
The runner will write a `progress.json` file every 5 requests:
```json
{
  "campaign_id": "qwen_stress_001",
  "total_prompts": 100,
  "completed_count": 45,
  "failed_count": 2,
  "current_request_id": "uuid-123",
  "status": "EXECUTING",
  "last_update": "2023-10-27T10:00:00Z"
}
```

---

# Dashboard UI Component Specifications
To ensure the UI remains responsive with large datasets, the following frontend architecture is required:

### 1. Virtualized List Component
- **Purpose:** Display the history of `llm_requests`.
- **Requirement:** Only render the rows currently visible in the viewport.
- **Data Fetching:** Paginated API calls (e.g., `GET /history?page=1&limit=50`).

### 2. Artifact Previewer
- **Purpose:** View the content of the `.jsonl` files without downloading them.
- **Requirement:** Fetch the first 1,000 characters of the artifact and display in a code-block style component.
- **Action:** Provide a "Download Full Artifact" button that triggers a direct download of the file from the `/artifacts` path.

### 3. Reasoning Visualization
- **Purpose:** Compare "Thinking" vs "Answering."
- **Component:** A split-pane view.
    - **Left Pane:** Reasoning content (styled with a light gray background).
    - **Right Pane:** Completion content (standard text).
- **Metric Overlay:** A small badge showing "Reasoning Ratio: X%".

### 4. Campaign Comparison Chart
- **Purpose:** Compare two `benchmark_runs`.
- **Library:** Chart.js or Recharts.
- **Metrics:**
    - **Latency Distribution:** Box plot showing the spread of response times.
    - **Token Efficiency:** Scatter plot of (Reasoning Tokens) vs (Completion Tokens).

---

# Security Hardening & Network Isolation
Since this is a private home-lab, we must ensure the "Flight Recorder" doesn't become an entry point for external threats.

### 1. Network Isolation
- The `ai-flight-recorder` and `llama.cpp` containers should reside on a private Docker bridge network.
- Only the FastAPI port (e.g., 8000) should be exposed to the local network.
- The `llama.cpp` server should **not** have any ports exposed to the outside world.

### 2. Authentication
- Implement a simple `X-API-KEY` header check in the FastAPI middleware.
- Store the API Key in an environment variable (`FLIGHT_RECORDER_KEY`).

### 3. Database Security
- Use a dedicated SQLite file with restricted permissions (`chmod 600`).
- Ensure the `ARTIFACT_PATH` is not accessible via a public web server.

---

# Scalability Roadmap (Redis/Celery)
If the number of concurrent benchmark requests exceeds the capacity of a single FastAPI worker, the following architecture will be adopted:

1.  **Task Queue:** Introduce **Redis** as a message broker.
2.  **Worker Pool:** Use **Celery** workers to handle the `process_request` logic.
3.  **Asynchronous Processing:**
    - The proxy receives the request and immediately returns a `task_id`.
    - The client polls `GET /status/{task_id}` to check progress.
    - This prevents the "Gateway Timeout" issue entirely, as the HTTP connection is closed quickly while the worker continues the long-running LLM generation.

---

# Hardware/Software Compatibility Matrix

| Component | Minimum Requirement | Recommended Requirement | Notes |
| :--- | :--- | :--- | :--- |
| **GPU VRAM** | 16GB | 24GB+ | Required for Qwen3.6-35B-A3B-APEX |
| **System RAM** | 32GB | 64GB | For handling large artifact buffers |
| **Storage** | 500GB SSD | 2TB NVMe | Artifacts can grow rapidly |
| **Python Version** | 3.10 | 3.11+ | For improved `asyncio` performance |
| **SQLite Version** | 3.35+ | 3.40+ | Required for certain JSON functions |
| **Docker Engine** | 20.10+ | 24.0+ | For proper volume mounting |

---

# Extended Test Suite (Edge Cases)

### Data Integrity Tests
30. **Null Prompt Test:** Verify the system handles an empty string prompt gracefully (returns 400).
31. **Max Token Overflow:** Send a prompt that forces the model to exceed `max_tokens`. Verify the `status` is set to `truncated` and the artifact is saved.
32. **Concurrent Write Test:** Run 10 concurrent benchmark requests and verify that the SQLite database does not experience "Database is locked" errors.
33. **Large Artifact Test:** Generate a 50,000-token response. Verify the `ArtifactManager` saves the file and the UI preview still loads in < 500ms.

### Redaction Tests
34. **Complex Email Test:** Verify `user.name+tag@sub.domain.com` is redacted.
35. **Mixed Content Test:** Verify a paragraph containing an email, a phone number, and a Teams ID are all redacted simultaneously.
36. **False Positive Test:** Verify that a string like "support@internal.local" (if not in the pattern) is handled according to policy.

### Reasoning Tests
37. **No-Reasoning Test:** Verify that if the model doesn't output `<thought>` tags, the `reasoning_tokens` count is 0 and the system doesn't crash.
38. **Nested Tags Test:** Verify that if the model outputs nested tags (e.g., `<thought><inner_thought></inner_thought></thought>`), the parser correctly identifies the outer boundary.
39. **Reasoning-Only Test:** Verify that if the model only outputs reasoning and no completion, the `completion_content` is empty but the request is marked as `success`.

### Infrastructure Tests
40. **Disk Full Test:** Simulate a full disk and verify the proxy returns a 500 error and logs the failure.
41. **Database Connection Loss:** Simulate a DB disconnect during a write. Verify the artifact is still saved to the filesystem (atomic operation).
42. **Docker Volume Persistence:** Restart the container and verify that artifacts and the SQLite database persist.

---

# Model Profile Examples (JSON)
These profiles should be stored in the `model_profiles` table.

### Profile 1: Qwen3.6-Balanced (Standard)
```json
{
  "model_name": "Qwen3.6-35B-A3B-APEX",
  "version": "v1.0.0",
  "system_prompt": "You are a helpful, concise assistant. Provide accurate information and cite sources when possible.",
  "temperature": 0.7,
  "top_p": 0.9,
  "repeat_penalty": 1.1,
  "max_tokens": 4096,
  "reasoning_enabled": true
}
```

### Profile 2: Qwen3.6-Creative (High Variance)
```json
{
  "model_name": "Qwen3.6-35B-A3B-APEX",
  "version": "v1.1-creative",
  "system_prompt": "You are a creative writer. Use vivid imagery and complex metaphors.",
  "temperature": 1.2,
  "top_p": 1.0,
  "repeat_penalty": 1.2,
  "max_tokens": 8192,
  "reasoning_enabled": true
}
```

### Profile 3: Qwen3.6-Coding (Strict)
```json
{
  "model_name": "Qwen3.6-35B-A3B-APEX",
  "version": "v1.2-code",
  "system_prompt": "You are an expert software engineer. Provide only code blocks and brief explanations. No conversational filler.",
  "temperature": 0.2,
  "top_p": 0.8,
  "repeat_penalty": 1.0,
  "max_tokens": 16384,
  "reasoning_enabled": false
}
```

---

# Disaster Recovery Procedures

### 1. Database Corruption
- **Procedure:** If `sqlite3` reports a "malformed database" error:
    1. Stop the `ai-flight-recorder` container.
    2. Restore the `flight_recorder.db` from the nightly backup.
    3. Run the `upgrade_v2.py` script to ensure the schema is current.
    4. Restart the container.

### 2. Artifact Loss
- **Procedure:** If the `/artifacts` directory is wiped:
    1. The database will still contain the metadata and `request_id`.
    2. The UI will show "Artifact Missing" for those entries.
    3. Since artifacts are ephemeral by design, no data recovery is possible, but the system will remain functional for new requests.

### 3. Proxy Configuration Drift
- **Procedure:** If the proxy starts behaving unexpectedly:
    1. Compare the current environment variables against the `deploy/docker-compose.yml` source of truth.
    2. Re-deploy the container using `docker-compose up -d --force-recreate`.

---

# Final Implementation Checklist for Senior Engineer
- [ ] **Migration:** Execute `upgrade_v2.py` and verify `llm_requests` count.
- [ ] **Redaction:** Run a test script with 10 known PII strings and verify they are redacted in the DB.
- [ ] **Artifacts:** Verify that a 10k token response creates a `.jsonl` file in the correct directory.
- [ ] **Reasoning:** Verify that the `reasoning_tokens` count is correctly incremented in the DB.
- [ ] **Benchmarking:** Run the `app/bench.py` script with 50 prompts and verify the `benchmark_runs` table is updated.
- [ ] **UI:** Verify that the dashboard loads the history page in under 2 seconds.
- [ ] **Security:** Verify that the `X-API-KEY` is required for all requests.
- [ ] **Logs:** Verify that logs are rotating and not exceeding 100MB per file.
- [ ] **Profiles:** Verify that changing the `MODEL_PROFILE_ID` env var updates the parameters used by the proxy.
- [ ] **Timeouts:** Verify that a request exceeding 60 seconds is caught and logged as a `timeout` status.
- [ ] **Concurrency:** Verify that 3 concurrent requests can be handled without crashing the proxy.
- [ ] **JSONL:** Verify that the artifact files are valid JSONL (one JSON object per line).
- [ ] **Redaction Logic:** Ensure the regex for Teams IDs is updated as the format changes.
- [ ] **Reasoning Parser:** Ensure the parser handles the case where the model stops mid-thought.
- [ ] **Database Indexing:** Ensure `session_id` and `model_profile_id` are indexed for fast lookups.
- [ ] **Docker Volumes:** Confirm that the `/artifacts` volume is mapped correctly to the host.
- [ ] **Memory Leak:** Monitor the container's memory usage over a 24-hour period of continuous benchmarking.
- [ ] **Latency Metrics:** Verify that `latency_ms` is being recorded accurately for every request.
- [ ] **Success Rate:** Verify that the `benchmark_runs` table correctly calculates the success percentage.
- [ ] **Rollback:** Practice a manual rollback of the database to ensure the team is prepared for a failed migration.

---

# Appendix: Example JSONL Artifact
This is what a single entry in the `/artifacts` directory will look like for a long-form response.

```json
{
  "request_id": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": 1698412800.0,
  "prompt": "Write a comprehensive technical specification for a distributed key-value store.",
  "redacted_prompt": "Write a comprehensive technical specification for a distributed key-value store.",
  "reasoning": "<thought>The user wants a technical spec. I need to cover: Architecture, Consistency Models, Replication, Sharding, and Fault Tolerance. I should start with a high-level overview then dive into specifics. I will use a CAP theorem context.</thought>",
  "completion": "## Technical Specification: Distributed Key-Value Store\n\n### 1. Overview\nThis document outlines the architecture for a highly available, horizontally scalable distributed key-value store...\n\n[... 10,000 more tokens of content ...]\n\n### 12. Conclusion\nBy implementing the Raft consensus algorithm and consistent hashing, the system ensures high availability and linear scalability.",
  "metadata": {
    "model_profile": "Qwen3.6-35B-A3B-APEX",
    "version": "v1.0.0",
    "temperature": 0.7,
    "top_p": 0.9,
    "reasoning_tokens": 145,
    "completion_tokens": 8500,
    "total_tokens": 8645,
    "latency_ms": 145000
  }
}
```

---

# Final Deployment Notes
- **Environment:** Ensure the `llama.cpp` server is running on `192.168.1.116:8080` before starting the proxy.
- **Permissions:** The user running the Docker container must have write permissions to `/models/flight-recorder/data` and `/models/flight-recorder/artifacts`.
- **Monitoring:** Check the `migration.log` immediately after the migration script finishes.
- **Scaling:** If the system becomes slow, increase the `worker_count` in the FastAPI configuration or move to the Redis/Celery architecture.
- **Privacy:** Periodically audit the `llm_requests` table to ensure the redaction engine is catching all intended PII.

**End of Dossier.**

# Technical Appendix: API Reference and SQL Analytics

To facilitate integration with external monitoring tools and frontend dashboards, the following API specifications and SQL queries are provided for the production environment.

## API Reference

### 1. Proxy Endpoint
**`POST /v1/chat/completions`**
The primary entry point for all LLM interactions.
- **Request Body:** Standard OpenAI-compatible JSON.
- **Headers:** `X-API-KEY` (Required).
- **Behavior:** 
    - Intercepts request.
    - Runs Redaction Engine.
    - Forwards to `llama.cpp`.
    - Parses reasoning tokens.
    - Saves artifact to filesystem.
    - Writes metadata to SQLite.
    - Streams response back to client.

### 2. Health Check
**`GET /health`**
- **Response:** `{"status": "healthy", "db_connected": true, "artifact_path": "/artifacts"}`
- **Usage:** Used by Docker healthchecks and load balancers.

### 3. Dashboard Metrics
**`GET /dashboard/stats`**
- **Query Params:** `?model_id=1&campaign_id=abc`
- **Response:** Aggregated JSON containing:
    - `avg_latency_ms`
    - `avg_reasoning_ratio`
    - `total_tokens_processed`
    - `success_rate_percentage`

### 4. Artifact Retrieval
**`GET /artifacts/{request_id}`**
- **Behavior:** Streams the `.jsonl` file from the `/artifacts` directory.
- **Security:** Requires `X-API-KEY`.

---

## SQL Analytics Queries

Use these queries within `app/reports.py` to generate complex dashboard visualizations.

### Query: Reasoning Efficiency by Model Profile
Identifies which model profiles are "thinking" the most relative to their output.
```sql
SELECT 
    mp.model_name,
    mp.version,
    AVG(lr.reasoning_tokens) as avg_reasoning,
    AVG(lr.completion_tokens) as avg_completion,
    AVG(CAST(lr.reasoning_tokens AS FLOAT) / (lr.reasoning_tokens + lr.completion_tokens)) as reasoning_ratio
FROM llm_requests lr
JOIN model_profiles mp ON lr.model_profile_id = mp.id
GROUP BY mp.model_name, mp.version
ORDER BY reasoning_ratio DESC;
```

### Query: Latency Distribution (Percentiles)
Useful for identifying "tail latency" in long-form generation.
```sql
SELECT 
    mp.model_name,
    PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY lr.latency_ms) as p50_latency,
    PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY lr.latency_ms) as p90_latency,
    PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY lr.latency_ms) as p99_latency
FROM llm_requests lr
JOIN model_profiles mp ON lr.model_profile_id = mp.id
WHERE lr.status = 'success'
GROUP BY mp.model_name;
```

### Query: Campaign Success Rate
```sql
SELECT 
    br.campaign_name,
    COUNT(lr.id) as total_requests,
    SUM(CASE WHEN lr.status = 'success' THEN 1 ELSE 0 END) as successful_requests,
    (CAST(SUM(CASE WHEN lr.status = 'success' THEN 1 ELSE 0 END) AS FLOAT) / COUNT(lr.id)) * 100 as success_rate
FROM benchmark_runs br
JOIN llm_requests lr ON lr.session_id = br.campaign_name -- Assuming session_id links to campaign
GROUP BY br.campaign_name;
```

---

## Docker Configuration (Production)

The following `docker-compose.yml` is optimized for the home-lab environment, ensuring volume persistence and network isolation.

```yaml
version: '3.8'

services:
  ai-flight-recorder:
    build: 
      context: .
      dockerfile: deploy/Dockerfile
    container_name: ai-flight-recorder
    ports:
      - "8000:8000"
    environment:
      - FLIGHT_RECORDER_DB_PATH=/data/flight_recorder.db
      - FLIGHT_RECORDER_ARTIFACT_PATH=/artifacts
      - LLAMA_CPP_URL=http://llama-server:8080
      - REDACTION_ENABLED=true
      - MODEL_PROFILE_ID=1
      - API_KEY=${FLIGHT_RECORDER_KEY}
    volumes:
      - /models/flight-recorder/data:/data
      - /models/flight-recorder/artifacts:/artifacts
    networks:
      - ai-network
    restart: unless-stopped

  llama-server:
    image: ghcr.io/ggerganov/llama.cpp:full
    container_name: llama-server
    ports:
      - "8080:8080"
    volumes:
      - /models/weights:/models
    command: 
      - "-m"
      - "/models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf"
      - "-c"
      - "8192"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "8080"
      - "--reasoning"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    networks:
      - ai-network
    restart: unless-stopped

networks:
  ai-network:
    driver: bridge
```

---

## Environment Variable Dictionary

| Variable | Description | Default | Example |
| :--- | :--- | :--- | :--- |
| `FLIGHT_RECORDER_DB_PATH` | Absolute path to SQLite file | `/data/flight_recorder.db` | `/models/data/db.sqlite` |
| `FLIGHT_RECORDER_ARTIFACT_PATH` | Root directory for JSONL files | `/artifacts` | `/models/artifacts` |
| `LLAMA_CPP_URL` | Internal URL for llama.cpp | `http://llama-server:8080` | `http://192.168.1.116:8080` |
| `REDACTION_ENABLED` | Toggle for PII scrubbing | `true` | `false` |
| `MODEL_PROFILE_ID` | ID of the default profile | `1` | `2` |
| `API_KEY` | Secret key for proxy access | `None` | `super_secret_key_123` |
| `LOG_LEVEL` | Logging verbosity | `INFO` | `DEBUG` |
| `MAX_REQUEST_TIMEOUT` | Seconds before marking as timeout | `60` | `300` |

---

## Performance Tuning & Optimization

### 1. FastAPI Concurrency
To handle multiple concurrent benchmark requests, use `uvicorn` with multiple workers:
```bash
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4 --loop uvloop
```

### 2. SQLite Optimization
For high-frequency writes during heavy benchmarking, enable WAL (Write-Ahead Logging) mode:
```sql
PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
```

### 3. Llama.cpp Optimization
To maximize throughput for the Qwen3.6 model:
- **Flash Attention:** Ensure `--flash-attn` is enabled in the `llama.cpp` command.
- **Batch Size:** Adjust `-b` (batch size) to 512 or 1024 depending on VRAM availability.
- **Context Window:** Keep `-c` (context size) as small as possible for the specific task to save memory.

### 4. Artifact Cleanup Cron
To prevent the `/artifacts` directory from consuming all disk space, schedule a cleanup job:
```bash
# Delete artifacts older than 30 days every night at midnight
0 0 * * * find /models/flight-recorder/artifacts -type f -mtime +30 -delete
```

---

## Final Summary of Changes
The transition from a single-table SQLite storage to a **Decoupled Artifact Architecture** solves the primary scaling issues of the AI Flight Recorder. By externalizing large JSON outputs, we ensure the database remains performant for dashboard queries. The introduction of a **Reasoning Parser** allows for high-fidelity telemetry on "thinking" behavior, while the **Redaction Engine** provides a necessary security layer for private data. Finally, the **Stateful Benchmark Runner** ensures that long-running, high-token-count tasks are durable and auditable.

This infrastructure is now prepared for high-stress benchmarking and production-grade observability.