# Engineering Dossier: AI Flight Recorder Evolution
**Project Code Name:** `flight-recorder-v2`
**Status:** Pre-Deployment Review
**Target Environment:** Home-lab (192.168.1.116)
**Primary Model:** Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf (Reasoning Enabled)

---

# Executive Summary

The `ai-flight-recorder` is currently a functional proxy and recording layer for llama.cpp. However, as the project moves toward rigorous benchmarking of reasoning models (like the Qwen3.6-35B series), the current architecture faces critical bottlenecks. The primary risks involve database bloat from long-form reasoning outputs, potential leakage of sensitive home-lab/corporate data (Teams/Email), and a lack of structured artifact management for large-scale benchmark campaigns.

This dossier outlines the transition from a simple "recorder" to a "benchmark and observability platform." The core of the upgrade involves decoupling raw output storage from the metadata database, implementing a regex-based redaction pipeline, and introducing a formal "Reasoning Budget" tracking mechanism to differentiate between "thinking" and "answering" latency and token usage.

The goal is to enable the execution of campaigns involving tens of thousands of tokens per request without crashing the FastAPI event loop or corrupting the SQLite state, while ensuring that every model profile change is auditable for scientific reproducibility.

---

# Current Architecture

The system operates as a transparent middleware layer.

### Data Flow
1. **Request Phase:** Client $\rightarrow$ FastAPI Proxy (`/v1/chat/completions`) $\rightarrow$ Request Metadata captured $\rightarrow$ Forwarded to llama.cpp.
2. **Response Phase:** llama.cpp $\rightarrow$ FastAPI Proxy $\rightarrow$ Response streamed to Client $\rightarrow$ Full response captured in memory $\rightarrow$ Written to SQLite.
3. **Analysis Phase:** Dashboard $\rightarrow$ `reports.py` $\rightarrow$ SQLite $\rightarrow$ Aggregated Metrics.
4. **Benchmark Phase:** `bench.py` $\rightarrow$ Loop of requests to Proxy $\rightarrow$ Results stored in SQLite.

### Component Stack
- **API:** FastAPI (Python 3.11+)
- **Backend:** llama.cpp server (OpenAI compatible)
- **Storage:** SQLite (Single file on `/models/flight-recorder/data`)
- **Deployment:** Docker Compose (Host: 192.168.1.116)
- **Model:** Qwen3.6-35B (GGUF) with reasoning capabilities enabled.

---

# Failure Modes Found

During the pre-deployment audit, the following failure modes were identified:

### 1. The "Giant Blob" Memory Exhaustion
The current proxy captures the full response in a string before writing to SQLite. For reasoning models generating 20k+ tokens, this leads to:
- **Memory Spikes:** High RAM usage per request, potentially triggering OOM (Out of Memory) kills in the Docker container.
- **SQLite Page Overflow:** Extremely large text fields in SQLite can degrade query performance for the dashboard.
- **UI Freeze:** The dashboard attempts to render these giant strings in the browser, leading to DOM crashes.

### 2. The "Reasoning Blindspot"
The system treats the entire response as a single block of text. Because the Qwen3.6 model uses internal reasoning (thinking), the current metrics conflate "thinking time" with "generation time." This makes it impossible to determine if a model is slow because it is "thinking hard" or because the output generation is lagging.

### 3. PII Leakage (Privacy Risk)
The proxy records everything. If a user pastes a Teams chat or an email into the prompt for summarization, that PII (Personally Identifiable Information) is stored in plaintext in the SQLite DB. There is currently no scrubbing mechanism.

### 4. Benchmark Non-Reproducibility
Model profiles (temperature, top_p, system prompts) are changed via environment variables or manual config updates. There is no versioning. If a benchmark is run today and another next week, there is no guarantee the "Balanced" profile hasn't been tweaked, rendering the comparison invalid.

### 5. SQLite Write Contention
During high-concurrency benchmarks, the `app/store.py` helpers perform synchronous writes. This creates `database is locked` errors when the benchmark runner and the proxy attempt to write simultaneously.

---

# Data Model Changes

To resolve the identified failure modes, the SQLite schema must be evolved. We move from a "flat" recording style to a "relational artifact" style.

### New Schema Definitions

```sql
-- 1. Model Profile Versioning
CREATE TABLE model_profiles (
    profile_id INTEGER PRIMARY KEY AUTOINCREMENT,
    profile_name TEXT NOT NULL, -- e.g., 'Qwen3.6-Balanced'
    config_json TEXT NOT NULL,   -- Store temp, top_p, penalty, etc.
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    is_active BOOLEAN DEFAULT 0
);

-- 2. Benchmark Campaigns
CREATE TABLE benchmark_campaigns (
    campaign_id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    description TEXT,
    start_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    end_time TIMESTAMP,
    profile_id INTEGER,
    FOREIGN KEY(profile_id) REFERENCES model_profiles(profile_id)
);

-- 3. Enhanced LLM Requests
CREATE TABLE llm_requests (
    request_id TEXT PRIMARY KEY,
    campaign_id INTEGER,
    session_id TEXT,
    prompt_text TEXT,
    -- Split response into metadata and a pointer to a file
    response_preview TEXT,       -- First 1000 chars for UI
    artifact_path TEXT,          -- Path to .jsonl or .txt file on disk
    
    -- Reasoning Metrics
    reasoning_tokens INTEGER DEFAULT 0,
    generation_tokens INTEGER DEFAULT 0,
    total_tokens INTEGER DEFAULT 0,
    
    -- Timing Metrics (in milliseconds)
    time_to_first_token INTEGER,
    reasoning_duration INTEGER,  -- Time spent in <thought>
    generation_duration INTEGER, -- Time spent in final answer
    total_duration INTEGER,
    
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(campaign_id) REFERENCES benchmark_campaigns(campaign_id)
);

-- 4. Audit Log for Profile Changes
CREATE TABLE profile_audit_log (
    audit_id INTEGER PRIMARY KEY AUTOINCREMENT,
    profile_id INTEGER,
    changed_by TEXT,
    change_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    old_value TEXT,
    new_value TEXT
);
```

### Migration Strategy
1. **Backup:** `cp data/flight_recorder.db data/flight_recorder.db.bak`.
2. **Schema Update:** Run a migration script to create new tables.
3. **Data Migration:** 
   - Extract existing `response_text` from `llm_requests`.
   - Write them to `/models/flight-recorder/artifacts/{id}.txt`.
   - Update `llm_requests` to set `artifact_path` and truncate `response_text` into `response_preview`.

---

# Artifact Storage Design

To handle "tens of thousands of tokens," we must stop treating the database as a file system.

### Storage Architecture
- **Root Directory:** `/models/flight-recorder/artifacts/`
- **Structure:** `/artifacts/{campaign_id}/{request_id}.json`
- **Format:** JSONL (JSON Lines) to allow streaming writes.

### The Artifact Lifecycle
1. **Stream Capture:** As the proxy receives chunks from llama.cpp, it writes them immediately to a temporary file on disk.
2. **Finalization:** Once the stream ends, the proxy calculates the final token counts and reasoning duration.
3. **Database Commit:** The proxy writes the metadata (timings, token counts, and the file path) to SQLite.
4. **Cleanup:** A background task (cron) archives artifacts older than 30 days to a compressed `.tar.gz` or deletes them if they aren't part of a "Saved Campaign."

### File Content Example (`/artifacts/12/req_abc123.json`)
```json
{
  "metadata": {
    "model": "Qwen3.6-35B",
    "timestamp": "2023-10-27T10:00:00Z",
    "system_prompt": "You are a helpful assistant..."
  },
  "turns": [
    {
      "role": "user",
      "content": "Analyze this 50-page legal document..."
    },
    {
      "role": "assistant",
      "reasoning": "<thought>\nStep 1: Identify parties...\nStep 2: Analyze clause 4...\n</thought>",
      "content": "The document states that..."
    }
  ]
}
```

---

# Reasoning Budget Handling

Reasoning models (like Qwen3.6) typically wrap their internal chain-of-thought in specific tags (e.g., `<thought>...</thought>`). We must treat this as a first-class citizen in our metrics.

### Measurement Logic
The proxy will implement a state-machine parser during the streaming phase:

1. **State: `IDLE`** $\rightarrow$ Wait for first token.
2. **State: `REASONING`** $\rightarrow$ Triggered when `<thought>` is detected.
   - Start `reasoning_timer`.
   - Increment `reasoning_token_count`.
3. **State: `GENERATING`** $\rightarrow$ Triggered when `</thought>` is detected.
   - Stop `reasoning_timer`.
   - Start `generation_timer`.
   - Increment `generation_token_count`.

### Metrics to Track
- **Reasoning Overhead Ratio:** $\frac{\text{Reasoning Tokens}}{\text{Total Tokens}}$. High ratios indicate the model is "over-thinking" simple tasks.
- **Thinking Latency:** The wall-clock time from the first `<thought>` token to the last `</thought>` token.
- **Reasoning Efficiency:** $\frac{\text{Correctness Score}}{\text{Reasoning Tokens}}$. (Used during benchmark evaluation).

---

# Benchmark Runner Design

The `app/bench.py` module needs to be upgraded from a simple loop to a "Campaign Manager."

### Campaign Workflow
1. **Initialization:**
   - Define a `Campaign` object: `Name`, `ModelProfileID`, `DatasetPath`.
   - Create a directory in `/artifacts/{campaign_id}/`.
2. **Execution (Async Batching):**
   - Use `asyncio.gather` with a semaphore (e.g., `limit=2`) to avoid overloading the llama.cpp server.
   - Each request is tagged with the `campaign_id`.
3. **Evaluation:**
   - After the run, the runner reads the artifacts.
   - It can run a "Judge LLM" (e.g., a larger model or a script) to score the outputs.
   - Scores are written back to a `benchmark_results` table.

### CLI Example
```bash
# Start a new benchmark campaign
python -m app.bench run \
  --name "Legal-Summarization-Test-V1" \
  --profile "Qwen3.6-Balanced" \
  --dataset "./data/legal_prompts.jsonl" \
  --concurrency 1 \
  --save-artifacts true
```

---

# Reporting Plane

The dashboard must move from "List of Requests" to "Comparative Analysis."

### Key Dashboard Metrics
- **Tokens Per Second (TPS):**
  - $\text{TPS}_{\text{reasoning}}$: Speed of the thinking process.
  - $\text{TPS}_{\text{generation}}$: Speed of the final answer.
- **Latency Heatmap:** Distribution of `total_duration` across different prompt lengths.
- **Model Comparison Table:**
  | Profile | Avg Reasoning Tokens | Avg Gen Tokens | Success Rate | Avg Latency |
  | :--- | :--- | :--- | :--- | :--- |
  | Balanced | 450 | 1,200 | 88% | 12.4s |
  | Precise | 1,200 | 800 | 94% | 22.1s |

### Report Generation
The `app/reports.py` will now generate a PDF/HTML summary for each campaign, including:
- **Sample Pairs:** "Worst performing" vs "Best performing" requests.
- **Token Distribution:** A histogram showing how many tokens were spent thinking vs answering.
- **Resource Usage:** Peak RAM/CPU during the campaign.

---

# Privacy And Redaction

To prevent PII leakage, a `RedactionEngine` will be inserted into the proxy pipeline *before* any data hits the SQLite DB or the Artifact storage.

### Redaction Pipeline
The engine uses a series of compiled regex patterns to replace sensitive data with placeholders.

```python
import re

class RedactionEngine:
    PATTERNS = {
        "EMAIL": r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+",
        "TEAMS_ID": r"teams:[\w\d-]+",
        "IPV4": r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b",
        "INTERNAL_URL": r"https?://(internal-corp|home-lab)\.local/\S*"
    }

    def redact(self, text: str) -> str:
        for label, pattern in self.PATTERNS.items():
            text = re.sub(pattern, f"<{label}_REDACTED>", text)
        return text
```

### Implementation Point
The `redact()` method is called in `app/proxy.py` at two points:
1. **Incoming Prompt:** Before saving `prompt_text` to the DB.
2. **Outgoing Response:** Before saving the response to the artifact file and the `response_preview` field.

---

# Deployment Plan

The deployment will follow a "Blue-Green" style approach on the home-lab server to ensure zero data loss.

### Step 1: Infrastructure Preparation
- Create the artifact directory: `mkdir -p /models/flight-recorder/artifacts`.
- Set permissions: `chmod -R 775 /models/flight-recorder/artifacts`.
- Update `docker-compose.yml` to mount this volume:
  ```yaml
  services:
    ai-flight-recorder:
      volumes:
        - /models/flight-recorder/data:/app/data
        - /models/flight-recorder/artifacts:/app/artifacts
  ```

### Step 2: Database Migration
- Run the migration script `python app/migrations/v2_schema.py`.
- Verify that the `model_profiles` table is populated with the current Qwen3.6 configuration.

### Step 3: Application Rollout
- Build the new Docker image: `docker build -t flight-recorder:v2 .`.
- Deploy the container: `docker-compose up -d`.
- Monitor logs for `SQLiteCrashes` or `PermissionDenied` errors.

### Step 4: Validation
- Run a "Smoke Test" request.
- Verify that a `.json` file is created in `/artifacts/` and that the DB contains the path, not the full text.

---

# Test Plan

The following 20 acceptance tests must pass before the deployment is considered successful.

### Functional Tests (Proxy & Storage)
1. **[T1] Basic Proxy:** Request to `/v1/chat/completions` returns a valid LLM response.
2. **[T2] Artifact Creation:** Every request creates a corresponding `.json` file in the artifacts folder.
3. **[T3] DB Pointer:** The `llm_requests` table contains the correct path to the artifact file.
4. **[T4] Preview Truncation:** `response_preview` is exactly 1000 characters or less, regardless of response size.
5. **[T5] Reasoning Split:** A response with `<thought>` tags correctly populates `reasoning_tokens` and `generation_tokens`.
6. **[T6] Timing Accuracy:** `total_duration` equals the sum of reasoning and generation durations (within a 50ms margin).
7. **[T7] Large Output Stability:** A request generating 30,000 tokens does not crash the proxy or the DB.
8. **[T8] Concurrent Writes:** 5 simultaneous benchmark requests do not trigger a `database is locked` error (WAL mode check).

### Privacy Tests
9. **[T9] Email Redaction:** Prompt containing `user@example.com` is stored as `<EMAIL_REDACTED>`.
10. **[T10] Teams Redaction:** Prompt containing `teams:12345` is stored as `<TEAMS_ID_REDACTED>`.
11. **[T11] IP Redaction:** Prompt containing `192.168.1.116` is stored as `<IPV4_REDACTED>`.
12. **[T12] Redaction Consistency:** Redaction is applied to both the prompt and the response.

### Benchmark & Reporting Tests
13. **[T13] Campaign Isolation:** Requests in Campaign A do not appear in the reports for Campaign B.
14. **[T14] Profile Audit:** Changing a model parameter creates a new entry in `profile_audit_log`.
15. **[T15] TPS Calculation:** The dashboard correctly calculates Tokens Per Second based on the `duration` fields.
16. **[T16] Artifact Retrieval:** The dashboard can load the full text of a response by reading the artifact file from disk.
17. **[T17] Dataset Loop:** The benchmark runner correctly iterates through a `.jsonl` dataset without skipping entries.
18. **[T18] Memory Ceiling:** RAM usage remains stable during a 100-request benchmark campaign.

### Edge Case Tests
19. **[T19] Malformed Reasoning:** A response that starts a `<thought>` tag but never closes it is handled gracefully (marked as reasoning until the end).
20. **[T20] Disk Full:** The system handles a "Disk Full" error during artifact writing without crashing the proxy (returns 500 error to client).

---

# Rollback Plan

In the event of a critical failure (e.g., data corruption or systemic instability):

### Immediate Action (The "Panic Button")
1. **Stop Container:** `docker-compose down`.
2. **Revert Image:** `docker-compose up -d` using the `v1` image tag.
3. **Restore DB:** `mv data/flight_recorder.db.bak data/flight_recorder.db`.

### Recovery Steps
- **Data Salvage:** If the `v2` DB is partially populated, run a script to extract the `artifact_path` and `prompt_text` into a CSV for manual review.
- **Log Analysis:** Inspect `/var/log/docker` and the FastAPI application logs to identify the specific failure mode (e.g., `OSError: [Errno 28] No space left on device`).
- **Post-Mortem:** Update the `Failure Modes` section of this dossier before attempting a second deployment.

---

# Open Questions

1. **S3 Migration:** At what volume of artifacts (e.g., 100GB+) should we migrate from local disk to a MinIO instance?
2. **Judge LLM Integration:** Should the benchmark runner include an automated "Judge" (e.g., using GPT-4o or a larger Qwen model) to score the reasoning quality, or keep it manual?
3. **Tokenization Accuracy:** Since we are using a proxy, we rely on the `usage` field from llama.cpp. If llama.cpp's token count differs from the actual GGUF tokenizer, how should we handle the discrepancy?
4. **UI Rendering:** For 50k+ token responses, should the dashboard implement a "Virtual Scroll" or a "Download Artifact" button instead of attempting to render the text?
5. **Reasoning Tag Variations:** If the model changes its reasoning tags (e.g., from `<thought>` to `<reasoning>`), should the tags be configurable via the `model_profiles` table?