# Engineering Dossier: AI Flight Recorder Production Readiness
**Project:** AI Flight Recorder (FastAPI/llama.cpp Proxy)  
**Status:** Pre-Deployment Review  
**Target Environment:** Private Home-Lab (Docker-based)  
**Active Model Profile:** Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf (Reasoning Enabled)

---

# Executive Summary
The AI Flight Recorder is a critical infrastructure component designed to provide observability, benchmarking, and privacy-aware proxying for LLM interactions. Currently, the system functions as a functional prototype but lacks the durability and scalability required for high-volume benchmarking and production-grade data integrity. 

The primary objective of this deployment plan is to transition the system from a "monolithic SQLite-blob" architecture to a "Content-Addressable Artifact" architecture. This will solve the issues of UI lag, database bloat, and timeout failures during long-form generation. Furthermore, we will implement a dedicated reasoning-tracking logic to isolate "thinking" tokens from "output" tokens, a robust privacy redaction layer to protect internal communications, and an auditable model profile management system.

Key outcomes of this deployment:
1.  **Scalability:** Support for 50,000+ token outputs without crashing the SQLite database or the FastAPI frontend.
2.  **Observability:** Granular tracking of reasoning vs. generation latency and token counts.
3.  **Security:** Automated redaction of PII (Emails, Teams IDs) before data hits the persistent store.
4.  **Reproducibility:** A stateful benchmark runner capable of multi-model comparison with durable artifacts.

---

# Current Architecture
The current system operates as a synchronous-style proxy with asynchronous logging:

1.  **Client Layer:** Users/Scripts interact with the FastAPI `/v1/chat/completions` endpoint.
2.  **Proxy Layer (`app/proxy.py`):** Receives requests, forwards them to `llama.cpp` (192.168.1.116), and streams the response back to the client while simultaneously writing to SQLite.
3.  **Storage Layer (`app/store.py`):** Handles all I/O. Currently stores full prompt/completion strings directly in SQLite columns.
4.  **Reporting Layer (`app/reports.py`):** Performs `SELECT` queries and basic aggregations for the dashboard.
5.  **Benchmark Layer (`app/bench.py`):** A script that fires requests and prints results to the console.

**Infrastructure:**
- **Host:** Local Server.
- **Network:** 192.168.1.116 (llama.cpp).
- **Storage:** `/models/flight-recorder/data` (SQLite file).

---

# Failure Modes Found

### 1. Database Bloat & UI Freezing (The "Giant JSON" Problem)
Storing 30,000+ token responses in a `TEXT` column in SQLite causes significant overhead during `SELECT *` operations. The FastAPI frontend attempts to load these into memory, leading to high latency and potential OOM (Out of Memory) errors in the browser.

### 2. Reasoning Token Obscurity
Because the Qwen3.6 model uses reasoning blocks, the current proxy treats `<thought>...</thought>` as standard content. This prevents accurate "Cost per Reasoning" or "Reasoning Efficiency" metrics. We cannot currently measure how much "thinking" is happening versus "doing."

### 3. Benchmark Volatility
The current `bench.py` is stateless. If a benchmark of 100 prompts fails at prompt 99 due to a network hiccup, the entire run must be restarted. There is no "resume" capability or durable artifact linking.

### 4. Privacy Leakage
The proxy currently records everything. If a user pastes a Teams message containing a private email or a corporate internal URL, that data is persisted in the SQLite database in plain text, violating internal privacy standards.

### 5. Configuration Drift
Model profiles (temperature, top_p, system prompts) are modified via the UI/API but lack a versioned audit trail. It is impossible to know which specific configuration produced a specific benchmark result from three weeks ago.

### 6. Timeout Cascades
Long-form generation (e.g., "Write a 5,000-word technical manual") exceeds the default FastAPI/Gunicorn/Uvicorn timeouts. The proxy closes the connection before `llama.cpp` finishes, but the background write might still attempt to complete, leading to orphaned records.

---

# Data Model Changes

To address the failure modes, we will migrate the SQLite schema. We will move from a "Flat Record" model to a "Relational Artifact" model.

### New Schema Definition (SQLAlchemy/SQLModel)

```sql
-- 1. Model Profiles (Auditable)
CREATE TABLE model_profiles (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    model_name TEXT NOT NULL,
    base_model_path TEXT NOT NULL,
    temperature REAL DEFAULT 0.7,
    top_p REAL DEFAULT 0.9,
    max_tokens INTEGER DEFAULT 4096,
    reasoning_enabled BOOLEAN DEFAULT TRUE,
    system_prompt TEXT,
    version_tag TEXT NOT NULL, -- e.g., "v1.2-balanced"
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- 2. Audit Log for Profile Changes
CREATE TABLE profile_audit_log (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    profile_id INTEGER,
    changed_by TEXT,
    old_values JSON,
    new_values JSON,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(profile_id) REFERENCES model_profiles(id)
);

-- 3. LLM Requests (Metadata Only)
CREATE TABLE llm_requests (
    id TEXT PRIMARY KEY, -- UUID
    profile_id INTEGER,
    client_session_id TEXT,
    prompt_hash TEXT,
    input_tokens INTEGER,
    output_tokens INTEGER,
    reasoning_tokens INTEGER,
    total_latency_ms INTEGER,
    status TEXT, -- 'success', 'timeout', 'error'
    error_message TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(profile_id) REFERENCES model_profiles(id)
);

-- 4. Benchmark Campaigns
CREATE TABLE benchmark_campaigns (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    description TEXT,
    model_profile_id INTEGER,
    status TEXT, -- 'pending', 'running', 'completed', 'failed'
    started_at TIMESTAMP,
    completed_at TIMESTAMP,
    FOREIGN KEY(model_profile_id) REFERENCES model_profiles(id)
);

-- 5. Benchmark Results (Links to Artifacts)
CREATE TABLE benchmark_results (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_id INTEGER,
    prompt_index INTEGER,
    score_metric REAL, -- e.g., accuracy or length
    artifact_path TEXT, -- Path to the JSON file on disk
    latency_ms INTEGER,
    FOREIGN KEY(campaign_id) REFERENCES benchmark_campaigns(id)
);
```

**Migration Note:** 
- Existing `llm_requests` data will be migrated to a `legacy_requests` table to maintain history without polluting the new schema.
- A `prompt_hash` (SHA-256) will be used to deduplicate identical prompts in the UI.

---

# Artifact Storage Design

To solve the "Giant JSON" problem, we will implement **Content-Addressable Storage (CAS)** for large outputs.

### Logic Flow:
1.  **Proxy receives completion:** The proxy streams the response to the client.
2.  **Buffer & Write:** Simultaneously, the proxy buffers the completion.
3.  **File Persistence:** Once the stream ends, the proxy calculates the SHA-256 hash of the completion.
4.  **Storage:**
    - If the file exists at `/models/flight-recorder/artifacts/{hash}.json`, do nothing.
    - If not, write the JSON to that path.
5.  **Database Update:** The `llm_requests` table stores the `artifact_path` (or just the hash) instead of the full text.

### Artifact JSON Structure:
```json
{
  "request_id": "uuid-string",
  "timestamp": "2023-10-27T10:00:00Z",
  "metadata": {
    "model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
    "parameters": {"temp": 0.7, "top_p": 0.9}
  },
  "content": {
    "prompt": "...",
    "reasoning": "...",
    "completion": "...",
    "thought_tokens": 450,
    "content_tokens": 1200
  },
  "performance": {
    "time_to_first_token_ms": 450,
    "total_generation_ms": 12000,
    "tokens_per_second": 100.5
  }
}
```

---

# Reasoning Budget Handling

Since we are using Qwen3.6 with reasoning enabled, we must distinguish between "Thinking" and "Answering."

### Implementation Strategy:
The proxy will use a **Dual-Stream Parser**. As tokens arrive from `llama.cpp`:
1.  **State Tracking:** The parser maintains a state: `IN_THOUGHT` or `IN_CONTENT`.
2.  **Tag Detection:** 
    - If `<thought>` is detected, switch to `IN_THOUGHT`.
    - If `</thought>` is detected, switch to `IN_CONTENT`.
3.  **Counter Logic:**
    - Increment `reasoning_token_count` while in `IN_THOUGHT`.
    - Increment `content_token_count` while in `IN_CONTENT`.
4.  **Latency Tracking:**
    - `reasoning_latency`: Time from `<thought>` start to `</thought>` end.
    - `generation_latency`: Time from `</thought>` end to final token.

### Dashboard Metrics:
- **Reasoning Ratio:** `reasoning_tokens / total_tokens` (Target: < 0.3 for standard tasks).
- **Thinking Speed:** `reasoning_tokens / reasoning_latency` (Tokens/sec during thought).
- **Efficiency Delta:** Comparison of reasoning time vs. output quality (to be populated by benchmark scores).

---

# Benchmark Runner Design

The new `app/bench.py` will be a stateful CLI tool.

### Features:
- **Campaigns:** Group prompts into "Campaigns" (e.g., "Coding_Hard_Level_1").
- **Checkpointing:** After every prompt, the runner saves the current index and results to the `benchmark_campaigns` table.
- **Multi-Model Comparison:** The runner can take a list of `model_profile_ids` and run the same campaign against all of them sequentially.
- **Screenshot Integration:** For UI-based benchmarks, the runner will trigger a headless browser (Playwright) to capture a screenshot of the rendered output and save it as an artifact.

### CLI Example:
```bash
# Run a campaign against two different profiles
python -m app.bench \
  --campaign "Logic_Test_v1" \
  --profiles "Qwen_Balanced,Llama_3_70B" \
  --output-dir "./results/2023-10-27" \
  --resume
```

### Comparison Matrix Output:
The runner will generate a `comparison_report.md` containing a table:
| Prompt | Model A (Latency) | Model A (Reasoning %) | Model B (Latency) | Model B (Reasoning %) | Winner |
| :--- | :--- | :--- | :--- | :--- | :--- |
| "Solve X" | 4500ms | 40% | 5200ms | 20% | Model A |

---

# Reporting Plane

The dashboard will be updated to support "Deep Dives."

### New Dashboard Views:
1.  **The "Flight Recorder" Feed:** A chronological list of all requests, with a "View Artifact" button that opens the JSON file in a new tab.
2.  **Reasoning Heatmap:** A chart showing the distribution of reasoning tokens across different prompt categories (Coding, Creative, Logic).
3.  **Model Comparison View:** A side-by-side view of two models' outputs for the same prompt, highlighting differences in reasoning steps.
4.  **Cost/Time Analysis:** Aggregated view of total tokens consumed per day/week.

### Metrics to Track:
- **TTFT (Time to First Token):** Critical for UX.
- **TPS (Tokens Per Second):** Average and Peak.
- **Reasoning Overhead:** The percentage of time spent "thinking" vs. "writing."
- **Failure Rate:** Percentage of requests hitting timeouts or 500 errors.

---

# Privacy And Redaction

To ensure the "Flight Recorder" doesn't become a liability, we will implement a **Redaction Middleware**.

### Redaction Strategy:
1.  **Pre-Persistence Hook:** Before any data is written to SQLite or the Artifact file, it passes through the `Redactor` class.
2.  **Regex Engine:**
    - **Emails:** `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`
    - **Teams IDs:** `[a-zA-Z0-9]{8,12}` (Context-aware, looking for "ID:" or "User:")
    - **Internal URLs:** `https?://(192\.168\.|10\.|172\.(1[6-9]|3[0-1])\.)\S+`
3.  **Replacement:** Redacted content is replaced with `[REDACTED_EMAIL]` or `[REDACTED_ID]`.
4.  **Logging:** A `redaction_count` is added to the metadata of the request to alert the user if a lot of data was scrubbed.

### Example Implementation Snippet:
```python
class Redactor:
    PATTERNS = {
        "email": r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}",
        "internal_ip": r"\b(192\.168\.|10\.|172\.(1[6-9]|3[0-1])\.)\d{1,3}\.\d{1,3}\.\d{1,3}\b"
    }

    def redact(self, text: str) -> tuple[str, int]:
        count = 0
        for label, pattern in self.PATTERNS.items():
            matches = re.findall(pattern, text)
            count += len(matches)
            text = re.sub(pattern, f"[{label.upper()}_REDACTED]", text)
        return text, count
```

---

# Deployment Plan

### Phase 1: Database Migration
1.  **Backup:** Create a snapshot of `/models/flight-recorder/data/flight_recorder.db`.
2.  **Script:** Run `python app/migrations/v2_initial_schema.py` to create new tables and migrate legacy data.
3.  **Verification:** Ensure `model_profiles` contains the current Qwen3.6 configuration.

### Phase 2: Proxy & Storage Update
1.  **Artifact Path:** Update `app/proxy.py` to implement the CAS logic.
2.  **Redaction:** Inject the `Redactor` middleware into the FastAPI request lifecycle.
3.  **Timeout Adjustment:** Increase `uvicorn` timeout to 300s and `llama.cpp` request timeout to 600s.

### Phase 3: Benchmark & UI
1.  **Runner:** Deploy the new `bench.py` and verify it can resume a failed campaign.
2.  **Dashboard:** Update `app/reports.py` to fetch from the new relational tables.

### Docker Configuration (`deploy/docker-compose.yml` updates):
- Add `ARTIFACT_STORAGE_PATH=/models/flight-recorder/artifacts`
- Add `REDACTION_ENABLED=true`
- Ensure volume mapping for the new artifacts directory.

---

# Test Plan

### Unit Tests (Python)
1.  **Redactor Test:** Verify email/IP regex correctly identifies and replaces 10 different formats.
2.  **Parser Test:** Verify the Dual-Stream Parser correctly splits `<thought>` blocks even if they contain nested brackets.
3.  **CAS Test:** Verify that writing the same completion twice results in only one file on disk.
4.  **Audit Test:** Verify that updating a model profile creates a corresponding entry in `profile_audit_log`.

### Integration Tests (FastAPI/SQLite)
5.  **Long Response Test:** Send a prompt requesting 5,000 words; verify the proxy doesn't timeout and the artifact is saved.
6.  **Reasoning Metric Test:** Verify that a response with 100 reasoning tokens and 50 content tokens records the correct counts in `llm_requests`.
7.  **Database Integrity:** Verify that deleting a `benchmark_campaign` does not delete the underlying `benchmark_results` (or handles it via cascade).
8.  **Concurrency Test:** Simulate 5 concurrent requests to the proxy and verify no SQLite "Database is locked" errors occur.

### End-to-End (E2E) Tests
9.  **Benchmark Resume:** Start a campaign of 10 prompts, kill the process at prompt 5, restart, and verify it continues at prompt 6.
10. **UI Rendering:** Verify the dashboard loads a 50,000-token artifact in under 2 seconds (via lazy-loading).
11. **Privacy Check:** Submit a prompt with a fake email; verify the database contains `[EMAIL_REDACTED]`.
12. **Model Switch:** Change a model profile via API and verify the next request uses the new parameters immediately.
13. **Audit Trail:** Verify that a "User" can see the history of changes for a specific model profile.
14. **Timeout Handling:** Verify that a request exceeding 10 minutes returns a 504 Gateway Timeout and logs a "timeout" status in the DB.
15. **Artifact Linking:** Verify that clicking "View Artifact" in the UI opens the correct file path.
16. **Multi-Model Comparison:** Run a 2-model comparison and verify the `comparison_report.md` is generated correctly.
17. **Token Counting:** Verify that `input_tokens` matches the `llama.cpp` reported count.
18. **Reasoning Latency:** Verify that `reasoning_latency` is strictly less than or equal to `total_latency_ms`.
19. **Storage Limits:** Verify that the system handles a full disk gracefully (logs error, doesn't crash proxy).
20. **Schema Migration:** Run the migration script twice and verify it is idempotent.

---

# Rollback Plan

### Scenario 1: Database Migration Failure
- **Action:** Restore the `/models/flight-recorder/data/flight_recorder.db` snapshot from Phase 1.
- **Trigger:** Any `IntegrityError` or `OperationalError` during the migration script execution.

### Scenario 2: Proxy Timeout/Crash
- **Action:** Revert `docker-compose.yml` to the previous image tag and environment variables.
- **Trigger:** If the proxy fails to handle long-form generation or crashes under concurrent load.

### Scenario 3: Redaction Over-Aggression
- **Action:** Toggle `REDACTION_ENABLED=false` via environment variable.
- **Trigger:** If the redactor is stripping legitimate technical data (e.g., code snippets) that are necessary for the benchmark.

---

# Open Questions

1.  **Multi-GPU Scaling:** How will the proxy handle load balancing if we add a second `llama.cpp` instance on a different IP?
2.  **Advanced Redaction:** Should we implement a local NER (Named Entity Recognition) model for more accurate redaction of names and locations?
3.  **Artifact Retention:** Do we need a TTL (Time-to-Live) for artifacts? (e.g., delete artifacts older than 90 days to save disk space).
4.  **Streaming Redaction:** Can we redact the stream in real-time for the client, or should it only be redacted before persistence? (Real-time redaction is significantly more complex).
5.  **Cost Attribution:** Should we add a "Cost" field to the `llm_requests` table to estimate USD spent based on provider pricing (even if running locally)?

### Detailed Component Specifications

#### 1. Proxy Manager (`app/proxy.py`)
The `ProxyManager` is the core engine of the service. It is responsible for the lifecycle of an LLM request, from ingestion to artifact persistence.

**Key Responsibilities:**
- **Request Normalization:** Standardizing incoming OpenAI-compatible requests into an internal `RequestContext` object.
- **Stream Parsing:** Implementing a stateful `StreamParser` that identifies `<thought>` tags in real-time.
- **Concurrent Persistence:** Using a background task queue (via `FastAPI.BackgroundTasks` or a dedicated `ThreadPoolExecutor`) to write artifacts to disk without blocking the response stream.
- **Redaction Pipeline:** Intercepting the stream to apply regex-based redaction before the data reaches the client or the database.

**StreamParser State Machine:**
- `STATE_IDLE`: Waiting for the first token.
- `STATE_THOUGHT`: Tokens are being accumulated into the `reasoning` buffer.
- `STATE_CONTENT`: Tokens are being accumulated into the `completion` buffer.
- `STATE_COMPLETE`: The stream has ended; trigger artifact finalization.

#### 2. Storage Engine (`app/store.py`)
The `Store` class abstracts all interactions with SQLite and the File System.

**Key Responsibilities:**
- **Atomic Writes:** Ensuring that the database record and the file system artifact are updated atomically (or via a compensating transaction if one fails).
- **Hash Management:** Generating SHA-256 hashes for every completion to implement the Content-Addressable Storage (CAS) system.
- **Query Optimization:** Providing optimized SQL queries for the dashboard, including `GROUP BY` operations for token counts and `AVG()` for latency metrics.
- **Connection Pooling:** Managing a thread-safe SQLite connection pool to prevent "Database is locked" errors during high-concurrency benchmarking.

#### 3. Reporting Engine (`app/reports.py`)
The `Reports` engine transforms raw database rows into structured JSON for the frontend.

**Key Responsibilities:**
- **Aggregation Logic:** Calculating "Reasoning Ratio" and "Tokens Per Second" across different model profiles.
- **Comparison Logic:** Fetching two different model profiles and aligning their benchmark results by `prompt_index` for side-by-side display.
- **Trend Analysis:** Providing time-series data for model performance over the last 30 days.

#### 4. Benchmark Runner (`app/bench.py`)
A standalone CLI tool designed for reproducible research.

**Key Responsibilities:**
- **Campaign Management:** Loading a set of prompts from a YAML file and executing them against a specified `model_profile_id`.
- **State Persistence:** Saving the progress of a campaign to the `benchmark_campaigns` table so it can be resumed after a crash.
- **Result Scoring:** Calculating a "Reasoning Efficiency Score" based on the ratio of content tokens to reasoning tokens, weighted by the accuracy of the output.
- **Artifact Linking:** Automatically associating every benchmark result with its corresponding file path in the CAS.

---

### API Reference

The following endpoints are exposed by the FastAPI service.

#### `POST /v1/chat/completions`
Standard OpenAI-compatible endpoint.
- **Request Body:**
  ```json
  {
    "model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
    "messages": [{"role": "user", "content": "Write a Python script to scrape a website."}],
    "temperature": 0.7,
    "max_tokens": 2048,
    "stream": true
  }
  ```
- **Response:** Standard OpenAI stream.
- **Internal Action:** Records metadata, parses reasoning, redacts PII, and saves to CAS.

#### `GET /v1/models`
Lists all available model profiles.
- **Response:**
  ```json
  [
    {
      "id": 1,
      "model_name": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
      "version_tag": "v1.2-balanced",
      "reasoning_enabled": true,
      "temperature": 0.7
    }
  ]
  ```

#### `GET /v1/benchmarks/{campaign_id}`
Retrieves the results of a specific benchmark campaign.
- **Query Params:** `?compare_with={model_profile_id}` (Optional)
- **Response:** A structured comparison object containing prompt, model A output, model B output, and performance metrics for both.

#### `POST /v1/profiles`
Creates or updates a model profile.
- **Request Body:**
  ```json
  {
    "model_name": "Llama-3-70B-Instruct.gguf",
    "base_model_path": "/models/llama3-70b.gguf",
    "temperature": 0.8,
    "system_prompt": "You are a helpful assistant.",
    "version_tag": "v1.0-initial"
  }
  ```
- **Audit:** Automatically creates an entry in `profile_audit_log`.

---

### Database Optimization Strategies

To ensure the system remains performant as the `llm_requests` table grows into the tens of thousands of rows:

1.  **Indexing Strategy:**
    - `CREATE INDEX idx_requests_profile ON llm_requests(profile_id);`
    - `CREATE INDEX idx_requests_timestamp ON llm_requests(created_at);`
    - `CREATE INDEX idx_campaign_status ON benchmark_campaigns(status);`
    - `CREATE INDEX idx_results_campaign ON benchmark_results(campaign_id);`

2.  **Query Optimization:**
    - Use `EXPLAIN QUERY PLAN` to ensure that dashboard aggregations are using indexes.
    - Implement "Pagination" on the dashboard feed (e.g., `LIMIT 50 OFFSET 0`).
    - Use `strftime` for time-based grouping to avoid heavy Python-side processing.

3.  **Maintenance:**
    - Schedule a weekly `VACUUM` and `ANALYZE` command on the SQLite database to reclaim space and update statistics.
    - Implement a "Cold Storage" script that moves requests older than 6 months to a `legacy_archive.db` file.

---

### UI/UX Design Specifications

The dashboard is designed for high-density information display with "progressive disclosure."

#### 1. The Flight Recorder Feed
- **Layout:** A searchable, filterable table.
- **Columns:** Timestamp, Model, Prompt (truncated), Reasoning Tokens, Content Tokens, Latency, Status.
- **Interaction:** Clicking a row opens a "Detail Modal."

#### 2. Detail Modal (The Artifact Viewer)
- **Content:** Displays the full prompt and the full completion.
- **Reasoning Block:** A collapsible section that shows the `<thought>` content with a different background color (e.g., light gray).
- **Metrics Sidebar:** Shows TTFT, TPS, and a "Reasoning Ratio" gauge.
- **Raw Data:** A button to "Download JSON Artifact" directly from the CAS.

#### 3. Benchmark Comparison View
- **Layout:** A split-screen view.
- **Left Side:** Model A's output with its reasoning steps.
- **Right Side:** Model B's output with its reasoning steps.
- **Diffing:** Highlight differences in the final content (using a standard diffing algorithm).
- **Scoring:** A summary header showing the "Winner" based on the pre-defined scoring logic.

---

### Infrastructure & Networking

#### 1. Docker Compose Configuration
The `deploy/docker-compose.yml` will be structured to isolate the proxy from the inference engine.

```yaml
services:
  ai-flight-recorder:
    build: .
    container_name: flight-recorder
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=file:/models/flight-recorder/data/flight_recorder.db
      - ARTIFACT_STORAGE_PATH=/models/flight-recorder/artifacts
      - REDACTION_ENABLED=true
      - LLAMA_CPP_URL=http://llama-cpp-server:8080
    volumes:
      - /models/flight-recorder/data:/models/flight-recorder/data
      - /models/flight-recorder/artifacts:/models/flight-recorder/artifacts
    networks:
      - ai-network
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 4G

  llama-cpp-server:
    image: ghcr.io/ggerganov/llama.cpp:full
    container_name: llama-cpp-server
    ports:
      - "8080:8080"
    volumes:
      - /models:/models
    command: --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --host 0.0.0.0 --port 8080 --reasoning
    networks:
      - ai-network
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

networks:
  ai-network:
    driver: bridge
```

#### 2. Networking & Security
- **Internal Communication:** The proxy communicates with `llama.cpp` over the internal Docker bridge network, preventing external access to the inference engine.
- **CORS Policy:** Restricted to the local home-lab IP range.
- **Rate Limiting:** Implemented via `slowapi` to prevent a single script from overwhelming the proxy and the GPU.

---

### Monitoring & Alerting

To ensure production stability, the following metrics will be exported to a Prometheus-compatible format.

#### 1. Key Performance Indicators (KPIs)
- `llm_request_duration_seconds`: Histogram of total request time.
- `llm_ttft_seconds`: Histogram of Time to First Token.
- `llm_tokens_per_second`: Gauge of current generation speed.
- `llm_reasoning_ratio`: Gauge of `reasoning_tokens / total_tokens`.
- `llm_request_count`: Counter of total requests, partitioned by status (success, error, timeout).

#### 2. Logging Strategy
- **Level: INFO:** Request received, request completed, artifact saved.
- **Level: WARNING:** Redaction triggered, slow response detected (>30s TTFT).
- **Level: ERROR:** Database lock, connection refused to `llama.cpp`, file system full.
- **Log Rotation:** Logs will be rotated daily and kept for 7 days.

---

### Security Hardening

#### 1. Redaction Engine Expansion
The `Redactor` class will be updated to include:
- **Phone Numbers:** `\b\d{3}[-.]?\d{3}[-.]?\d{4}\b`
- **Credit Cards:** `\b(?:\d[ -]*?){13,16}\b`
- **API Keys:** `(key|secret|token|auth)\s*[:=]\s*[a-zA-Z0-9_\-\.]{16,}`

#### 2. Authentication
- The dashboard and API will be protected by a simple `API_KEY` header for the home-lab environment.
- For more complex needs, a JWT-based auth system can be toggled via environment variables.

#### 3. Input Sanitization
- All prompts will be stripped of control characters before being forwarded to `llama.cpp` to prevent prompt injection or malformed request errors.

---

### Benchmark Methodology

To provide scientifically valid comparisons, the `bench.py` runner will follow these rules:

#### 1. Prompt Engineering
- **Fixed Context:** Every prompt in a campaign will use the exact same system prompt and temperature.
- **Seed Control:** Where supported by the model, a fixed random seed will be used to ensure reproducibility.

#### 2. Scoring Logic
The "Reasoning Efficiency Score" ($E$) is calculated as:
$$E = \left( \frac{C}{R} \right) \times \left( \frac{R}{L_r} \right)$$
Where:
- $C$ = Content Tokens
- $R$ = Reasoning Tokens
- $L_r$ = Reasoning Latency (seconds)

A higher score indicates a model that produces high-quality content with concise, fast reasoning.

#### 3. Human-in-the-Loop (HITL)
The UI will allow users to "Grade" a benchmark result on a scale of 1-5. These grades will be stored in the `benchmark_results` table to allow for "Subjective Accuracy" charts.

---

### Data Lifecycle Management

To prevent the `/models/flight-recorder/` directory from consuming all available disk space:

1.  **Artifact Cleanup:** A cron job will run every Sunday at 02:00.
    - It will identify artifacts associated with `benchmark_campaigns` that are older than 30 days.
    - It will delete the files from the CAS and mark the database records as `archived`.
2.  **Database Vacuuming:** The same cron job will run `VACUUM` on the SQLite database.
3.  **Log Purging:** Docker logs will be limited to 1GB using the `json-file` logging driver with `max-size` and `max-file` limits.

---

### Step-by-Step Migration Guide

This guide ensures a zero-data-loss transition from the current prototype to the production-ready system.

#### Step 1: Pre-flight Checklist
- [ ] Verify `/models/flight-recorder/data` is writable by the Docker user.
- [ ] Verify `/models/flight-recorder/artifacts` exists and is writable.
- [ ] Ensure `llama.cpp` is reachable at `192.168.1.116:8080`.
- [ ] Take a manual snapshot of `flight_recorder.db`.

#### Step 2: Database Migration
Run the following script:
```python
# app/migrations/v2_initial_schema.py
import sqlite3

def migrate():
    conn = sqlite3.connect('/models/flight-recorder/data/flight_recorder.db')
    cursor = conn.cursor()
    
    # Create new tables
    cursor.executescript("""
        CREATE TABLE IF NOT EXISTS model_profiles (...);
        CREATE TABLE IF NOT EXISTS profile_audit_log (...);
        CREATE TABLE IF NOT EXISTS llm_requests (...);
        CREATE TABLE IF NOT EXISTS benchmark_campaigns (...);
        CREATE TABLE IF NOT EXISTS benchmark_results (...);
    """)
    
    # Migrate legacy data
    cursor.execute("INSERT INTO llm_requests (id, prompt_hash, status) SELECT id, hash(prompt), status FROM legacy_requests")
    
    conn.commit()
    conn.close()
```

#### Step 3: Deployment
1.  Update `docker-compose.yml` with the new environment variables.
2.  Run `docker-compose down`.
3.  Run `docker-compose up -d --build`.
4.  Verify the health check endpoint: `curl http://localhost:8000/health`.

---

### Troubleshooting Guide

| Error Code | Symptom | Root Cause | Resolution |
| :--- | :--- | :--- | :--- |
| `504 Gateway Timeout` | Request hangs for 300s then fails. | `llama.cpp` is taking too long to generate. | Increase `UV_TIMEOUT` or check GPU utilization. |
| `Database is locked` | Proxy returns 500 error during high load. | Concurrent writes to SQLite. | Check `SQLAlchemy` pool size; ensure `WAL` mode is enabled. |
| `Artifact Not Found` | UI shows "File missing" for a request. | CAS write failed or file was deleted. | Check disk space; verify `ARTIFACT_STORAGE_PATH` permissions. |
| `Redaction Error` | Prompt is completely empty in the DB. | Regex is too aggressive. | Review `Redactor` patterns; check for overlapping regex. |
| `Connection Refused` | Proxy cannot reach `llama.cpp`. | `llama.cpp` container is down or IP changed. | Check `docker-compose` network; verify `LLAMA_CPP_URL`. |

---

### Future Roadmap

#### Phase 1: Optimization (Q1)
- Implement **Streaming Redaction** to scrub PII before it reaches the user's screen.
- Add **Multi-Model Routing** to automatically send "Coding" prompts to one model and "Creative" prompts to another.

#### Phase 2: Advanced Analytics (Q2)
- Integrate **Prometheus/Grafana** for real-time GPU and Proxy monitoring.
- Implement **Automated Evaluation (LLM-as-a-Judge)** where a larger model (e.g., Llama-3-70B) automatically scores the outputs of the Qwen model.

#### Phase 3: Enterprise Readiness (Q3)
- Add **User Authentication** and **Role-Based Access Control (RBAC)**.
- Implement **Data Export** functionality (CSV/JSON) for full benchmark reports.
- Develop a **Public API** for the benchmark runner to allow external researchers to submit prompts.

---

### Final Acceptance Tests (Summary)

To be executed by the QA engineer before final sign-off:

1.  **Persistence:** Verify a 10,000-token response is saved to disk and accessible via the UI.
2.  **Reasoning:** Verify that `<thought>` tokens are counted separately from content tokens.
3.  **Redaction:** Verify that `test@example.com` is replaced with `[EMAIL_REDACTED]`.
4.  **Audit:** Verify that changing a model's temperature creates a log entry.
5.  **Resume:** Verify that a benchmark interrupted at prompt 50 resumes at prompt 51.
6.  **Concurrency:** Verify that 10 simultaneous requests do not cause SQLite locks.
7.  **Latency:** Verify that TTFT is recorded accurately for every request.
8.  **Comparison:** Verify that the side-by-side view correctly aligns prompts from two different models.
9.  **Storage:** Verify that duplicate prompts result in a single artifact file (CAS).
10. **Timeout:** Verify that a 10-minute generation returns a 504 and logs the error.
11. **Schema:** Verify that the migration script is idempotent (can be run twice without error).
12. **UI Performance:** Verify that the dashboard loads 100 records in < 500ms.
13. **Network:** Verify that the proxy cannot be accessed directly from outside the Docker network.
14. **Scaling:** Verify that the proxy handles a burst of 50 requests/minute without crashing.
15. **Integrity:** Verify that deleting a campaign removes its results from the UI.
16. **Accuracy:** Verify that the "Reasoning Efficiency Score" matches the manual calculation.
17. **Security:** Verify that the `API_KEY` is required for all `/v1/profiles` calls.
18. **Log Rotation:** Verify that logs are truncated after reaching the size limit.
19. **Disk Space:** Verify that the system alerts when the artifact directory is 90% full.
20. **Rollback:** Verify that the database snapshot can be restored in under 5 minutes.

---

### Appendix: Configuration Constants

**Default Timeouts:**
- `REQUEST_TIMEOUT`: 300s
- `LLAMA_CPP_TIMEOUT`: 600s
- `DB_POOL_SIZE`: 10
- `MAX_ARTIFACT_SIZE`: 50MB

**Redaction Patterns:**
- `EMAIL_REGEX`: `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`
- `IP_REGEX`: `\b(192\.168\.|10\.|172\.(1[6-9]|3[0-1])\.)\d{1,3}\.\d{1,3}\.\d{1,3}\b`
- `SECRET_REGEX`: `(key|secret|token|auth)\s*[:=]\s*[a-zA-Z0-9_\-\.]{16,}`

**Benchmark Scoring Weights:**
- `CONTENT_WEIGHT`: 0.7
- `REASONING_EFFICIENCY_WEIGHT`: 0.3
- `LATENCY_PENALTY`: -0.01 per second over 10s.

### Implementation Deep Dive - StreamParser

The `StreamParser` is a stateful class designed to handle the nuances of the Qwen3.6 reasoning output. It must be robust enough to handle malformed tags, nested tags (though rare), and partial tokens.

```python
import re
from typing import Generator, Dict, Any, List

class StreamParser:
    """
    Parses the stream from llama.cpp, identifying reasoning blocks 
    and content blocks to provide granular metrics.
    """
    def __init__(self):
        self.is_thinking = False
        self.thought_buffer = []
        self.content_buffer = []
        self.reasoning_tokens = 0
        self.content_tokens = 0
        self.thought_start_time = None
        self.thought_end_time = None
        
        # Regex to detect tags
        self.thought_start_tag = "<thought>"
        self.thought_end_tag = "</thought>"

    def parse_stream(self, stream: Generator[Dict[str, Any], None, None]) -> Generator[Dict[str, Any], None, None]:
        for chunk in stream:
            content = chunk.get("content", "")
            
            # Check for start of thought
            if self.thought_start_tag in content and not self.is_thinking:
                self.is_thinking = True
                self.thought_start_time = None # Will set on first token
                # Split content to remove the tag itself
                parts = content.split(self.thought_start_tag)
                if len(parts) > 1:
                    content = parts[1]

            # Check for end of thought
            if self.thought_end_tag in content and self.is_thinking:
                self.is_thinking = False
                self.thought_end_time = None # Will set on last token
                parts = content.split(self.thought_end_tag)
                content = parts[0]
                # The part after the tag is actual content
                if len(parts) > 1:
                    self.content_buffer.append(parts[1])

            if self.is_thinking:
                self.thought_buffer.append(content)
                self.reasoning_tokens += len(content.split()) # Approximation
            else:
                self.content_buffer.append(content)
                self.content_tokens += len(content.split())

            # Yield the chunk to the client (standard OpenAI format)
            yield {
                "content": content,
                "reasoning_tokens": self.reasoning_tokens,
                "content_tokens": self.content_tokens,
                "is_thinking": self.is_thinking
            }

    def finalize_metrics(self) -> Dict[str, Any]:
        """
        Calculates final metrics after the stream is closed.
        """
        total_reasoning_time = 0
        if self.thought_start_time and self.thought_end_time:
            total_reasoning_time = self.thought_end_time - self.thought_start_time

        return {
            "reasoning_tokens": self.reasoning_tokens,
            "content_tokens": self.content_tokens,
            "reasoning_latency_ms": total_reasoning_time,
            "reasoning_ratio": (self.reasoning_tokens / (self.reasoning_tokens + self.content_tokens)) if (self.reasoning_tokens + self.content_tokens) > 0 else 0
        }
```

### Implementation Deep Dive - Redactor

The `Redactor` uses a multi-pass regex approach. To ensure high performance, it compiles all patterns at initialization.

```python
import re
from typing import Tuple

class Redactor:
    """
    Identifies and masks PII and internal infrastructure data.
    """
    def __init__(self, strict_mode: bool = True):
        self.strict_mode = strict_mode
        self.patterns = {
            "email": re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"),
            "internal_ip": re.compile(r"\b(192\.168\.|10\.|172\.(1[6-9]|3[0-1])\.)\d{1,3}\.\d{1,3}\.\d{1,3}\b"),
            "teams_id": re.compile(r"\b[a-zA-Z0-9]{8,12}\b"), # Contextual check needed
            "api_key": re.compile(r"(?i)(key|secret|token|auth)\s*[:=]\s*[a-zA-Z0-9_\-\.]{16,}")
        }

    def redact(self, text: str) -> Tuple[str, int]:
        if not text:
            return text, 0
        
        redaction_count = 0
        redacted_text = text
        
        for label, pattern in self.patterns.items():
            matches = pattern.findall(redacted_text)
            redaction_count += len(matches)
            redacted_text = pattern.sub(f"[{label.upper()}_REDACTED]", redacted_text)
            
        return redacted_text, redaction_count
```

### Benchmark Campaign Schema (YAML)

The `bench.py` runner will consume YAML files to define campaigns. This allows non-engineers to define test suites.

```yaml
# campaigns/coding_logic_v1.yaml
campaign_name: "Coding Logic - Hard"
description: "Tests complex recursive logic and algorithmic efficiency."
model_profiles:
  - id: 1 # Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
  - id: 2 # Llama-3-70B-Instruct.gguf
parameters:
  temperature: 0.0
  max_tokens: 4096
  top_p: 1.0
prompts:
  - id: 1
    prompt: "Write a Python function to find the longest palindromic substring in a given string. Explain the time complexity."
    expected_score_range: [4, 5]
    tags: ["coding", "algorithms"]
  - id: 2
    prompt: "Implement a thread-safe LRU cache in Go. Include unit tests."
    expected_score_range: [3, 5]
    tags: ["coding", "concurrency"]
  - id: 3
    prompt: "Explain the difference between a B-Tree and a LSM-Tree in the context of database indexing."
    expected_score_range: [4, 5]
    tags: ["theory", "databases"]
```

### Database Schema (SQLAlchemy Models)

These models define the structure for the `app/store.py` module.

```python
from sqlalchemy import Column, Integer, String, Float, Boolean, DateTime, JSON, ForeignKey
from sqlalchemy.ext.declarative import declarative_base
from datetime import datetime
import uuid

Base = declarative_base()

class ModelProfile(Base):
    __tablename__ = 'model_profiles'
    id = Column(Integer, primary_key=True)
    model_name = Column(String, nullable=False)
    base_model_path = Column(String, nullable=False)
    temperature = Column(Float, default=0.7)
    top_p = Column(Float, default=0.9)
    max_tokens = Column(Integer, default=4096)
    reasoning_enabled = Column(Boolean, default=True)
    system_prompt = Column(String)
    version_tag = Column(String, nullable=False)
    created_at = Column(DateTime, default=datetime.utcnow)
    updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)

class ProfileAuditLog(Base):
    __tablename__ = 'profile_audit_log'
    id = Column(Integer, primary_key=True)
    profile_id = Column(Integer, ForeignKey('model_profiles.id'))
    changed_by = Column(String)
    old_values = Column(JSON)
    new_values = Column(JSON)
    timestamp = Column(DateTime, default=datetime.utcnow)

class LLMRequest(Base):
    __tablename__ = 'llm_requests'
    id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
    profile_id = Column(Integer, ForeignKey('model_profiles.id'))
    client_session_id = Column(String)
    prompt_hash = Column(String)
    input_tokens = Column(Integer)
    output_tokens = Column(Integer)
    reasoning_tokens = Column(Integer)
    total_latency_ms = Column(Integer)
    status = Column(String) # 'success', 'timeout', 'error'
    error_message = Column(String)
    created_at = Column(DateTime, default=datetime.utcnow)

class BenchmarkCampaign(Base):
    __tablename__ = 'benchmark_campaigns'
    id = Column(Integer, primary_key=True)
    name = Column(String, nullable=False)
    description = Column(String)
    model_profile_id = Column(Integer, ForeignKey('model_profiles.id'))
    status = Column(String) # 'pending', 'running', 'completed', 'failed'
    started_at = Column(DateTime)
    completed_at = Column(DateTime)

class BenchmarkResult(Base):
    __tablename__ = 'benchmark_results'
    id = Column(Integer, primary_key=True)
    campaign_id = Column(Integer, ForeignKey('benchmark_campaigns.id'))
    prompt_index = Column(Integer)
    score_metric = Column(Float)
    artifact_path = Column(String)
    latency_ms = Column(Integer)
```

### Operational Playbook

#### 1. Hot-Swapping Model Profiles
To update the active model without restarting the proxy:
1.  Log into the dashboard.
2.  Navigate to "Model Profiles."
3.  Create a new profile with the desired parameters.
4.  Update the `model_profile_id` in the active benchmark campaign.
5.  The proxy will automatically pick up the new parameters for the next request.

#### 2. Data Purge Procedure
If the `/models/flight-recorder/artifacts` directory exceeds 100GB:
1.  Identify the oldest `benchmark_campaigns` via the dashboard.
2.  Run the `purge_artifacts.py` script:
    ```bash
    python -m app.utils.purge_artifacts --older-than 30d
    ```
3.  The script will delete files from the CAS and update the `llm_requests` table to mark `artifact_path` as `DELETED`.

#### 3. Handling "Infinite Reasoning"
If a model enters a reasoning loop (generating `<thought>` tokens indefinitely):
1.  The `StreamParser` will detect that `reasoning_tokens` is increasing while `content_tokens` remains 0.
2.  If `reasoning_tokens` exceeds 2,000 without a `</thought>` tag, the proxy will trigger a `ReasoningTimeoutException`.
3.  The proxy will terminate the connection and log a "Reasoning Loop Detected" error.

### Edge Case Matrix

| Scenario | System Behavior | Error Code | Mitigation |
| :--- | :--- | :--- | :--- |
| **Disk Full** | Proxy catches `OSError` during artifact write. | 507 | Alert sent to admin; request still completes for client. |
| **Llama.cpp Crash** | Proxy detects connection reset. | 502 | Log "Inference Engine Unavailable" and retry once. |
| **Prompt Too Long** | Proxy checks prompt length vs `max_tokens`. | 400 | Return "Prompt exceeds context window" before sending to GPU. |
| **Malformed Tags** | `StreamParser` ignores tags that don't close. | N/A | Parser treats unclosed `<thought>` as content after a 500-token limit. |
| **Concurrent Writes** | SQLite `Database is locked` error. | 500 | Implement `WAL` mode and a retry decorator on all `store.py` calls. |
| **Redaction Overlap** | Two regexes match the same string. | N/A | Redactor uses a "First Match Wins" logic to prevent double-masking. |

### UI Component Architecture

The frontend is built using a component-based architecture to ensure high performance.

#### 1. `ArtifactCard`
- **Props:** `request_id`, `model_name`, `latency`, `reasoning_ratio`.
- **Function:** Displays a summary card in the feed.
- **State:** Hover effect to show "Quick Stats."

#### 2. `ReasoningBlock`
- **Props:** `thought_text`, `tokens`.
- **Function:** A collapsible component that renders the `<thought>` content.
- **Styling:** Uses a distinct "Thought Bubble" aesthetic (monospaced font, light gray background).

#### 3. `ComparisonGrid`
- **Props:** `model_a_data`, `model_b_data`.
- **Function:** Renders two side-by-side columns.
- **Logic:** Automatically aligns prompts by index. If one model failed, the other side shows a "Failed" placeholder.

#### 4. `MetricGauge`
- **Props:** `value`, `min`, `max`, `label`.
- **Function:** A visual gauge for "Tokens Per Second" and "Reasoning Ratio."

### Deployment & Infrastructure Details

#### Volume Permissions
To ensure the Docker container can write to the host's storage:
```bash
# Run on the host machine
mkdir -p /models/flight-recorder/data
mkdir -p /models/flight-recorder/artifacts
# Set ownership to the UID used by the Docker container (usually 1000)
sudo chown -R 1000:1000 /models/flight-recorder/data
sudo chown -R 1000:1000 /models/flight-recorder/artifacts
sudo chmod -R 775 /models/flight-recorder/data
sudo chmod -R 775 /models/flight-recorder/artifacts
```

#### Environment Variables
- `DATABASE_URL`: `file:/models/flight-recorder/data/flight_recorder.db`
- `ARTIFACT_STORAGE_PATH`: `/models/flight-recorder/artifacts`
- `LLAMA_CPP_URL`: `http://192.168.1.116:8080`
- `REDACTION_STRICT_MODE`: `true`
- `MAX_REASONING_TOKENS`: `2048`
- `LOG_LEVEL`: `INFO`

### Final Rollback Procedures

#### Database Rollback
If the migration script fails or data corruption is detected:
1.  Stop the `ai-flight-recorder` container.
2.  Restore the snapshot: `cp /backups/flight_recorder_pre_v2.db /models/flight-recorder/data/flight_recorder.db`.
3.  Restart the container.

#### Proxy Rollback
If the new `StreamParser` causes issues with specific models:
1.  Update the Docker image tag in `docker-compose.yml` to the previous stable version.
2.  Run `docker-compose up -d`.

#### Redaction Rollback
If the redactor is causing data loss in benchmarks:
1.  Set `REDACTION_ENABLED=false` in the `.env` file.
2.  Restart the container.

### Conclusion
This dossier provides a comprehensive roadmap for transitioning the AI Flight Recorder from a prototype to a production-grade observability tool. By moving to a Content-Addressable Storage model, implementing a stateful StreamParser for reasoning, and enforcing a strict redaction layer, the system will provide the durability and security required for high-stakes LLM benchmarking.

**End of Dossier.**