# Executive Summary

This engineering dossier outlines a deployment-grade modernization of the AI Flight Recorder service, transitioning from an inline-JSON, monolithic SQLite logging pattern to a durable, artifact-driven, privacy-aware, and auditable telemetry platform. The primary objectives are to resolve six critical pain points identified during pre-deployment review: (1) replace giant inline JSON payloads with content-addressable artifact storage, (2) implement precise reasoning budget tracking without disabling model thinking capabilities, (3) enrich benchmark reports with reproducible metadata for cross-model comparisons and screenshot embedding, (4) enforce strict privacy redaction for corporate communications, (5) introduce immutable audit logging for server-side model profile changes, and (6) harden the system against high-token-output stress scenarios (timeouts, storage bloat, UI rendering limits).

The proposed architecture introduces a layered data model with explicit migration paths, a filesystem-backed artifact store with lifecycle management, a reasoning-aware proxy interceptor, a campaign-based benchmark runner, and a privacy redaction pipeline. All changes are designed for zero-downtime deployment via Docker Compose, with explicit rollback procedures, comprehensive acceptance testing, and operational runbooks. Synthetic data is used throughout to demonstrate behavior without exposing sensitive information. The final system will support tens of thousands of output tokens, maintain sub-second dashboard responsiveness, and provide full traceability for model evaluations, privacy compliance, and infrastructure changes.

# Current Architecture

The existing AI Flight Recorder is a FastAPI-based telemetry and benchmarking service deployed alongside `llama-cpp-server` in a home-lab environment. The architecture consists of the following components:

- **`app/main.py`**: FastAPI application entry point. Mounts `/dashboard`, `/health`, and `/v1/*` proxy routes. Handles static file serving for the frontend and basic CORS configuration.
- **`app/proxy.py`**: HTTP proxy that forwards `/v1/chat/completions` to `llama-cpp-server` (typically `http://192.168.1.116:8080`). Intercepts request/response cycles, extracts metadata (model, prompt length, completion length, latency), and writes to SQLite.
- **`app/store.py`**: SQLite-backed data layer. Contains ORM-like helpers for `runs`, `client_sessions`, `llm_requests`, `events`, and `benchmark_rows`. Uses WAL mode for concurrent reads/writes. Stores full request/response payloads as JSON BLOBs.
- **`app/reports.py`**: Aggregation layer. Executes SQL queries to compute dashboard metrics (throughput, latency percentiles, token counts, error rates). Returns JSON for frontend rendering.
- **`app/bench.py`**: Benchmark campaign executor. Loads prompt templates, iterates through model configurations, sends requests via the proxy, and records results. Currently runs synchronously or with basic asyncio loops.
- **`deploy/docker-compose.yml`**: Orchestrates `ai-flight-recorder` and `llama-cpp-server`. Mounts `/models/flight-recorder/data` for SQLite and model weights. Uses default networking and health checks.
- **Data Location**: `/models/flight-recorder/data/flight-recorder.db` (SQLite), with model GGUF files in `/models/`.
- **Active Model Profile**: `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` with `llama.cpp` reasoning enabled (`--cache-type q8_0`, `--mlock`, `--threads 16`, `--ctx-size 32768`, `--batch-size 2048`, `--no-mmap`).

Data flow: Client → FastAPI Proxy → llama.cpp → Response → SQLite BLOB insertion → Dashboard aggregation. The current design assumes moderate output sizes (<8k tokens), synchronous execution, and no privacy filtering. Model profile changes are applied via environment variables or direct GGUF path updates without versioning or audit trails.

# Failure Modes Found

The following failure modes were identified during architectural review and stress testing. Each maps directly to the stated pain points and requires explicit mitigation.

1. **Inline JSON Bloat & SQLite Contention**: Storing full request/response payloads as JSON BLOBs in SQLite causes WAL file growth, page contention, and slow aggregation queries. Responses exceeding 30k tokens trigger `SQLITE_TOOBIG` errors or severe latency spikes during `INSERT`/`SELECT`. Dashboard queries scan entire BLOBs, degrading UI responsiveness.
2. **Reasoning Token Inflation & Unmeasured Thinking**: With `llama.cpp` reasoning enabled, the model emits extended `<thinking>` blocks before final answers. These tokens are counted in total latency and token metrics but are not separated from completion tokens. This distorts throughput calculations, masks actual reasoning budget consumption, and prevents budget enforcement.
3. **Benchmark Metadata Deficiency**: Current benchmark runs lack reproducible context: no prompt template versioning, no hardware state snapshots (CPU/GPU temp, RAM, disk I/O), no model hash verification, and no screenshot embedding capability. Cross-model comparisons are manual and error-prone.
4. **Privacy Leakage in Telemetry**: Corporate emails, Teams channel IDs, project codenames, and internal URLs are logged verbatim in SQLite and dashboard previews. No redaction pipeline exists. Compliance risk is high for shared lab environments or accidental log exposure.
5. **Silent Model Profile Drift**: GGUF path changes, parameter overrides, or cache type modifications are applied via Docker env vars or direct file swaps. No audit trail records who changed what, when, or why. Rollbacks are guesswork.
6. **High-Token Output Stress**: Tens of thousands of output tokens cause:
   - Proxy timeouts (`httpx` default 30s, insufficient for 50k token generation)
   - Storage exhaustion (BLOB limits, disk space)
   - UI rendering OOM (frontend attempts to parse/render giant JSON)
   - Preview truncation failures (naive slicing breaks JSON structure)

These failure modes are interdependent. Resolving them requires a coordinated shift to artifact storage, reasoning-aware interception, privacy filtering, audit logging, and hardened timeout/storage policies.

# Data Model Changes

The SQLite schema must evolve to support artifacts, reasoning budgets, audit logs, redaction tracking, and enriched benchmark metadata. Below are the migration DDL statements, index definitions, and transition notes.

## Migration 1: Artifact Store & Request/Response Decoupling
```sql
CREATE TABLE IF NOT EXISTS artifacts (
    id TEXT PRIMARY KEY,
    content_hash TEXT NOT NULL UNIQUE,
    mime_type TEXT NOT NULL DEFAULT 'application/json',
    size_bytes INTEGER NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    storage_path TEXT NOT NULL,
    lifecycle_status TEXT DEFAULT 'active'
);

CREATE TABLE IF NOT EXISTS llm_requests (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    run_id TEXT,
    client_session_id TEXT,
    model_profile TEXT NOT NULL,
    request_artifact_id TEXT REFERENCES artifacts(id),
    response_artifact_id TEXT REFERENCES artifacts(id),
    prompt_tokens INTEGER NOT NULL,
    completion_tokens INTEGER NOT NULL,
    reasoning_tokens INTEGER DEFAULT 0,
    total_latency_ms INTEGER NOT NULL,
    status TEXT NOT NULL DEFAULT 'success',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_requests_run ON llm_requests(run_id);
CREATE INDEX idx_requests_model ON llm_requests(model_profile);
CREATE INDEX idx_requests_created ON llm_requests(created_at);
```

## Migration 2: Reasoning Budget & Enforcement
```sql
CREATE TABLE IF NOT EXISTS reasoning_budgets (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    model_profile TEXT NOT NULL,
    max_reasoning_tokens INTEGER NOT NULL DEFAULT 4096,
    max_reasoning_time_ms INTEGER NOT NULL DEFAULT 30000,
    enforcement_mode TEXT DEFAULT 'soft' CHECK(enforcement_mode IN ('soft', 'hard', 'disabled')),
    active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_budget_model ON reasoning_budgets(model_profile, active);
```

## Migration 3: Model Profile Audit Log
```sql
CREATE TABLE IF NOT EXISTS model_profile_audit (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    actor TEXT NOT NULL,
    action TEXT NOT NULL CHECK(action IN ('deploy', 'update', 'rollback', 'parameter_change')),
    previous_profile TEXT,
    new_profile TEXT NOT NULL,
    parameters_changed TEXT,
    reason TEXT,
    verification_hash TEXT
);

CREATE INDEX idx_audit_timestamp ON model_profile_audit(timestamp);
CREATE INDEX idx_audit_profile ON model_profile_audit(new_profile);
```

## Migration 4: Redaction & Privacy Tracking
```sql
CREATE TABLE IF NOT EXISTS redaction_logs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    request_id INTEGER REFERENCES llm_requests(id),
    field_type TEXT NOT NULL,
    original_length INTEGER,
    redacted_length INTEGER,
    policy_applied TEXT NOT NULL,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_redaction_request ON redaction_logs(request_id);
```

## Migration 5: Benchmark Campaign & Run Metadata
```sql
CREATE TABLE IF NOT EXISTS benchmark_campaigns (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    description TEXT,
    config_hash TEXT NOT NULL,
    started_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    completed_at TIMESTAMP,
    status TEXT DEFAULT 'running'
);

CREATE TABLE IF NOT EXISTS benchmark_runs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_id TEXT REFERENCES benchmark_campaigns(id),
    prompt_template_id TEXT NOT NULL,
    model_profile TEXT NOT NULL,
    hardware_snapshot TEXT,
    result_artifact_id TEXT REFERENCES artifacts(id),
    metrics_json TEXT,
    screenshot_path TEXT,
    status TEXT DEFAULT 'pending',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_bench_campaign ON benchmark_runs(campaign_id);
CREATE INDEX idx_bench_model ON benchmark_runs(model_profile);
```

## Migration Notes
- Run migrations sequentially using `sqlite3 /models/flight-recorder/data/flight-recorder.db < migration_v2.sql`.
- Back up existing `llm_requests` BLOBs before dropping inline payload columns.
- Use `PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;` for performance.
- Add `PRAGMA foreign_keys=ON;` at connection initialization.
- Validate migration with `SELECT COUNT(*) FROM artifacts;` and cross-reference with request counts.

# Artifact Storage Design

Inline JSON storage is replaced with a content-addressable filesystem artifact store. This decouples telemetry metadata from payload size, enables efficient previews, and supports lifecycle management.

## Directory Structure
```
/models/flight-recorder/data/artifacts/
├── YYYY/
│   ├── MM/
│   │   ├── DD/
│   │   │   ├── <sha256_hash>.json
│   │   │   ├── <sha256_hash>.json.preview
│   │   │   └── <sha256_hash>.json.metadata
```

## Storage Protocol
1. **Ingestion**: Proxy computes SHA-256 of payload. Checks `artifacts` table for existing hash. If missing, writes to disk, records metadata, and inserts row.
2. **Preview Generation**: On write, generate a truncated preview (max 2KB) preserving JSON structure. Store as `.preview` file.
3. **Metadata File**: Store `content-type`, `size`, `reasoning_tokens`, `redaction_status`, `created_at` in `.metadata` JSON.
4. **Lifecycle**: `active` → `archived` (after 30 days) → `deleted` (after 90 days). Cleanup runs via cron or background task.
5. **Access**: Dashboard queries `artifacts` table, joins with `llm_requests`, and serves previews. Full artifact retrieval uses `GET /artifacts/{id}` with streaming response.

## Python Helper (store.py extension)
```python
import hashlib
import json
import os
from pathlib import Path
from datetime import datetime

ARTIFACT_ROOT = Path("/models/flight-recorder/data/artifacts")

def store_artifact(payload: bytes, mime_type: str = "application/json") -> str:
    content_hash = hashlib.sha256(payload).hexdigest()
    date_path = datetime.utcnow().strftime("%Y/%m/%d")
    storage_dir = ARTIFACT_ROOT / date_path
    storage_dir.mkdir(parents=True, exist_ok=True)
    
    file_path = storage_dir / f"{content_hash}.json"
    if not file_path.exists():
        file_path.write_bytes(payload)
        # Generate preview
        try:
            preview = json.loads(payload)[:2000]
            (storage_dir / f"{content_hash}.json.preview").write_text(json.dumps(preview))
        except Exception:
            (storage_dir / f"{content_hash}.json.preview").write_text("Preview generation failed")
        # Metadata
        meta = {"size": len(payload), "mime": mime_type, "created": datetime.utcnow().isoformat()}
        (storage_dir / f"{content_hash}.json.metadata").write_text(json.dumps(meta))
        
        # Insert into DB
        db.execute(
            "INSERT OR IGNORE INTO artifacts (id, content_hash, mime_type, size_bytes, storage_path, lifecycle_status) VALUES (?, ?, ?, ?, ?, 'active')",
            (content_hash, content_hash, mime_type, len(payload), str(file_path))
        )
    return content_hash
```

## Storage Limits & Hardening
- Max single artifact: 50MB (reject larger with `413 Payload Too Large`)
- Disk quota monitoring: Alert at 80% usage, enforce cleanup at 90%
- Compression: Optional `zstd` for artifacts older than 7 days
- Integrity: Verify hash on read; log corruption events

# Reasoning Budget Handling

Reasoning tokens must be measured and bounded without disabling model thinking. The proxy intercepts llama.cpp response streams, parses reasoning blocks, and enforces budgets.

## Interception Logic
llama.cpp with reasoning enabled emits `<thinking>...</thinking>` or `reasoning` metadata in streaming responses. The proxy:
1. Buffers streaming chunks
2. Detects reasoning boundaries via regex or XML tag parsing
3. Counts tokens in reasoning blocks separately
4. Tracks elapsed time for reasoning phase
5. Compares against `reasoning_budgets` table
6. Enforces soft (log warning) or hard (abort stream) limits

## Proxy Interceptor Snippet
```python
import re
import time
from httpx import AsyncClient

REASONING_PATTERN = re.compile(r"<thinking>(.*?)</thinking>", re.DOTALL)

async def intercept_reasoning(response_stream, budget_config):
    reasoning_tokens = 0
    reasoning_time_ms = 0
    start = time.time()
    full_response = b""
    
    async for chunk in response_stream:
        full_response += chunk
        if budget_config["enforcement_mode"] == "hard":
            elapsed = (time.time() - start) * 1000
            if elapsed > budget_config["max_reasoning_time_ms"]:
                raise TimeoutError("Reasoning time budget exceeded")
        
    # Post-stream analysis
    matches = REASONING_PATTERN.findall(full_response.decode("utf-8", errors="ignore"))
    for block in matches:
        reasoning_tokens += len(block.split())  # Approx token count
        
    if reasoning_tokens > budget_config["max_reasoning_tokens"]:
        log_warning(f"Reasoning token budget exceeded: {reasoning_tokens}/{budget_config['max_reasoning_tokens']}")
        
    return full_response, reasoning_tokens, reasoning_time_ms
```

## Budget Configuration
- Default: 4096 tokens, 30s, soft enforcement
- Per-model overrides via `reasoning_budgets` table
- Dashboard displays reasoning ratio: `reasoning_tokens / (reasoning_tokens + completion_tokens)`
- Hard enforcement aborts generation, returns `429 Reasoning Budget Exceeded` with partial output

## Metrics Tracked
- `reasoning_tokens`, `completion_tokens`, `total_tokens`
- `reasoning_time_ms`, `generation_time_ms`, `total_latency_ms`
- `budget_violations_count`, `budget_utilization_pct`

# Benchmark Runner Design

The benchmark runner evolves from a simple loop to a campaign-based, metadata-rich, artifact-linked executor.

## Campaign Configuration (YAML)
```yaml
campaign_id: "bench-qwen3-6-35b-2024-10-25"
name: "Qwen3.6-35B Reasoning Stress Test"
description: "Evaluate high-token output and reasoning budget adherence"
model_profile: "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf"
concurrency: 4
timeout_ms: 120000
prompts:
  - id: "prompt-001"
    template: "Analyze the following dataset and provide a detailed report..."
    expected_min_tokens: 15000
  - id: "prompt-002"
    template: "Generate a comprehensive architectural design document..."
    expected_min_tokens: 25000
hardware_snapshot: true
screenshot_capture: true
```

## Runner Architecture
1. **Campaign Loader**: Parses YAML, validates schema, computes config hash
2. **Task Queue**: `asyncio.Queue` with concurrency limit
3. **Hardware Snapshot**: Captures CPU temp, RAM usage, disk I/O, llama.cpp cache status via `/health` and system calls
4. **Execution**: Sends prompts via proxy, links to campaign, records artifacts
5. **Result Aggregation**: Computes P50/P95 latency, token throughput, reasoning ratio, error rates
6. **Screenshot Embedding**: Captures dashboard state via headless browser or API snapshot, stores path in `benchmark_runs`

## CLI Example
```bash
python -m app.bench run --config campaigns/stress-test.yaml --output results/
python -m app.bench compare --campaigns bench-qwen3-6-35b-2024-10-25 bench-llama3-8b-2024-10-20 --metrics latency,throughput,reasoning_ratio
```

## Metadata Injection
- Prompt template versioning (hash)
- Model GGUF hash verification
- Environment variables snapshot
- Timestamps (UTC)
- Hardware state JSON
- Screenshot path (base64 or relative)

# Reporting Plane

The reporting layer aggregates metrics, supports comparisons, and embeds screenshot metadata for reproducibility.

## Dashboard Metrics
- **Throughput**: Tokens/sec (completion only, reasoning excluded)
- **Latency**: P50, P95, P99 (ms)
- **Reasoning Ratio**: `reasoning_tokens / total_tokens`
- **Artifact Hit Rate**: % of requests served from artifact cache
- **Redaction Rate**: % of requests with redacted fields
- **Budget Violations**: Count and severity
- **Storage Utilization**: Disk usage, artifact count, lifecycle status distribution

## Aggregation Queries
```sql
-- Throughput & Latency Percentiles
SELECT 
    model_profile,
    PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY total_latency_ms) AS p50_latency,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY total_latency_ms) AS p95_latency,
    AVG(completion_tokens / (total_latency_ms / 1000.0)) AS avg_throughput_tps
FROM llm_requests
WHERE created_at >= datetime('now', '-24 hours')
GROUP BY model_profile;

-- Reasoning Ratio
SELECT 
    model_profile,
    AVG(CAST(reasoning_tokens AS REAL) / NULLIF(completion_tokens + reasoning_tokens, 0)) AS avg_reasoning_ratio
FROM llm_requests
WHERE created_at >= datetime('now', '-7 days')
GROUP BY model_profile;
```

## Comparison Features
- Side-by-side model run comparison
- Prompt template matching
- Hardware state normalization
- Screenshot metadata embedding (path + timestamp + resolution)
- Export to CSV/JSON with full traceability

## UI Hardening
- Preview truncation at 2KB
- Lazy loading for artifact details
- Virtual scrolling for large result sets
- Memory limit warnings for frontend JSON parsing

# Privacy And Redaction

Corporate data must be redacted before storage. A multi-layer pipeline ensures compliance without degrading performance.

## Redaction Pipeline
1. **Ingress Filter**: Regex-based detection for emails, phone numbers, internal URLs, Teams IDs, project codenames
2. **Policy Engine**: Configurable allow/deny lists, field-level redaction rules
3. **LLM-Assisted Classification** (optional): Lightweight model for context-aware redaction
4. **Audit Logging**: Record redaction events in `redaction_logs`
5. **Synthetic Replacement**: Replace with `[REDACTED_EMAIL]`, `[REDACTED_TEAM_ID]`, etc.

## Regex Patterns (Synthetic Examples)
```python
import re

REDACTION_PATTERNS = [
    (r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}", "[REDACTED_EMAIL]"),
    (r"TEAM-\d{4}", "[REDACTED_TEAM_ID]"),
    (r"PROJECT-[A-Z]+-\d+", "[REDACTED_PROJECT]"),
    (r"https?://internal\.corp\.local/.*", "[REDACTED_INTERNAL_URL]"),
    (r"\b\d{3}-\d{2}-\d{4}\b", "[REDACTED_SSN]"),
]

def redact_payload(payload: str) -> str:
    for pattern, replacement in REDACTION_PATTERNS:
        payload = re.sub(pattern, replacement, payload, flags=re.IGNORECASE)
    return payload
```

## Policy Configuration
```json
{
  "redaction_enabled": true,
  "policies": [
    {"name": "corporate_comms", "patterns": ["email", "teams_id", "internal_url"], "action": "replace"},
    {"name": "pii_protection", "patterns": ["ssn", "phone", "ip_address"], "action": "mask"}
  ],
  "allowlist": ["public@example.com", "TEAM-0000"],
  "log_redactions": true
}
```

## Performance Considerations
- Regex compilation cached
- Async redaction pipeline
- Fallback to pass-through on failure
- Dashboard shows redaction status per request

# Deployment Plan

Zero-downtime deployment via Docker Compose with migration runner, health checks, and rollback triggers.

## Docker Compose (v3.8)
```yaml
version: "3.8"
services:
  migration-runner:
    image: python:3.11-slim
    volumes:
      - ./migrations:/app/migrations
      - /models/flight-recorder/data:/data
    command: >
      sh -c "sqlite3 /data/flight-recorder.db < /app/migrations/v2_schema.sql &&
             sqlite3 /data/flight-recorder.db < /app/migrations/v2_seed.sql"
    restart: "no"

  flight-recorder:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - /models/flight-recorder/data:/data
      - /models/flight-recorder/data/artifacts:/artifacts
    environment:
      - DATABASE_URL=sqlite:////data/flight-recorder.db
      - ARTIFACT_ROOT=/artifacts
      - REDACTION_ENABLED=true
      - REASONING_BUDGET_HARD=false
      - MAX_ARTIFACT_SIZE_MB=50
      - DISK_QUOTA_ALERT_PCT=80
    depends_on:
      migration-runner:
        condition: service_completed_successfully
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 10s
      timeout: 5s
      retries: 3

  llama-cpp-server:
    image: ghcr.io/ggerganov/llama.cpp:server
    volumes:
      - /models:/models
    command: >
      --model /models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
      --host 0.0.0.0
      --port 8080
      --ctx-size 32768
      --batch-size 2048
      --threads 16
      --cache-type q8_0
      --mlock
      --no-mmap
      --reasoning
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 15s
      timeout: 10s
      retries: 5
```

## Deployment Steps
1. Back up SQLite DB and artifact directory
2. Run `docker compose up -d migration-runner`
3. Verify migration success (`docker logs migration-runner`)
4. Deploy `flight-recorder` with new image
5. Validate health endpoint and dashboard
6. Run smoke tests
7. Monitor disk usage, latency, redaction logs

## Environment Variables
- `DATABASE_URL`, `ARTIFACT_ROOT`, `REDACTION_ENABLED`, `REASONING_BUDGET_HARD`, `MAX_ARTIFACT_SIZE_MB`, `DISK_QUOTA_ALERT_PCT`, `BENCHMARK_CONCURRENCY`, `UI_PREVIEW_LIMIT_KB`

# Test Plan

20+ acceptance tests covering artifacts, reasoning, privacy, timeouts, audit, reporting, benchmark runner, UI/storage limits, and rollback simulation.

## Test Matrix
| ID | Category | Description | Expected Outcome |
|----|----------|-------------|------------------|
| T01 | Artifact Storage | Store 50k token response | Artifact created, hash verified, preview generated |
| T02 | Artifact Storage | Duplicate payload ingestion | No duplicate file, DB count unchanged |
| T03 | Artifact Storage | Exceed max artifact size | 413 response, no disk write |
| T04 | Reasoning Budget | Soft limit exceeded | Warning logged, response returned |
| T05 | Reasoning Budget | Hard limit exceeded | 429 response, stream aborted |
| T06 | Reasoning Budget | Reasoning tokens counted separately | `reasoning_tokens` > 0, `completion_tokens` accurate |
| T07 | Privacy Redaction | Email in prompt | `[REDACTED_EMAIL]` in stored payload |
| T08 | Privacy Redaction | Teams ID in response | `[REDACTED_TEAM_ID]` in artifact |
| T09 | Privacy Redaction | Allowlist bypass | Known email preserved |
| T10 | Privacy Redaction | Redaction log entry | `redaction_logs` row created |
| T11 | Audit Logging | Model profile change | `model_profile_audit` entry with hash |
| T12 | Audit Logging | Rollback action | Audit trail shows previous/new profile |
| T13 | Benchmark Runner | Campaign execution | All runs recorded, artifacts linked |
| T14 | Benchmark Runner | Hardware snapshot captured | JSON in `benchmark_runs` |
| T15 | Benchmark Runner | Timeout handling | Run marked failed, partial artifact stored |
| T16 | Reporting Plane | P95 latency calculation | Matches synthetic dataset |
| T17 | Reporting Plane | Reasoning ratio aggregation | Correct per-model average |
| T18 | UI Hardening | Giant JSON preview | Truncated at 2KB, no OOM |
| T19 | Storage Lifecycle | Artifact aging | Status changes to `archived` after 30 days |
| T20 | Rollback Simulation | Revert to v1 schema | Migration rollback script succeeds, data preserved |
| T21 | Concurrency | 10 parallel requests | No SQLite lock contention, all succeed |
| T22 | Disk Quota | 85% usage alert | Log warning, cleanup triggered at 90% |
| T23 | Proxy Interceptor | Streaming reasoning detection | Tokens counted mid-stream |
| T24 | Policy Engine | Custom regex pattern | Applied correctly, logged |

## Test Execution
- Unit tests: `pytest app/tests/`
- Integration tests: Docker Compose up, run test suite, validate DB/artifacts
- Load tests: `locust` or `k6` scripts simulating 50k token responses
- Privacy tests: Synthetic corpus with known PII patterns
- Rollback tests: Scripted migration reversal, data integrity check

# Rollback Plan

Explicit rollback procedures ensure safe reversion without data loss.

## Step-by-Step Rollback
1. Stop `flight-recorder` service: `docker compose stop flight-recorder`
2. Restore SQLite backup: `cp /backup/flight-recorder.db.v1 /models/flight-recorder/data/flight-recorder.db`
3. Revert artifact directory: `mv /models/flight-recorder/data/artifacts /models/flight-recorder/data/artifacts.v2`
4. Deploy previous image: `docker compose up -d --build flight-recorder`
5. Validate health and dashboard
6. Monitor for 24 hours

## Data Preservation
- Artifacts from v2 retained in `artifacts.v2`
- Audit logs preserved
- Redaction logs preserved
- Benchmark campaigns archived

## Configuration Revert
- Restore `.env` or Docker Compose env vars
- Disable redaction/reasoning budget if needed
- Revert UI preview limits

## Verification
- Run T20 (Rollback Simulation)
- Check `SELECT COUNT(*) FROM llm_requests;`
- Validate artifact paths
- Confirm no schema mismatches

# Open Questions

1. **Storage Backend**: Should artifact storage remain local filesystem, or migrate to S3-compatible object storage for scalability?
2. **Redaction Sensitivity**: What is the acceptable false-positive rate for regex-based redaction? Should LLM-assisted classification be enabled by default?
3. **Benchmark Concurrency**: What is the maximum safe concurrency for llama.cpp on this hardware? Should queue backpressure be implemented?
4. **UI Framework Constraints**: Does the frontend support virtual scrolling and lazy loading out-of-the-box, or require framework upgrades?
5. **Monitoring Stack**: Should Prometheus/Grafana be integrated for real-time metric alerting, or rely on SQLite-based reporting?
6. **Reasoning Token Accuracy**: Is word-split approximation sufficient, or should a proper tokenizer be integrated for exact counting?
7. **Artifact Cleanup Policy**: Should lifecycle thresholds be configurable per model profile or campaign type?
8. **Audit Log Retention**: How long should `model_profile_audit` and `redaction_logs` be retained? Should archival to cold storage be implemented?
9. **Timeout Configuration**: Should proxy timeouts be dynamic based on expected token count, or fixed per model profile?
10. **Screenshot Metadata**: Should screenshots be embedded as base64 in DB, or stored as artifacts with reference IDs?

These questions require stakeholder alignment before final production sign-off. All proposed solutions are designed to be configurable and extensible to accommodate future requirements.

## Extended Operational Runbooks & SRE Practices

To ensure deployment-grade reliability, the following operational runbooks define standard procedures for monitoring, alerting, capacity planning, and incident response. These practices are integrated directly into the Docker Compose orchestration and FastAPI middleware layers.

### Monitoring & Alerting Matrix
The system exposes a `/metrics` endpoint compatible with Prometheus scraping. Key metrics are categorized by subsystem:

| Subsystem | Metric Name | Type | Description | Alert Threshold |
|-----------|-------------|------|-------------|-----------------|
| Proxy | `proxy_request_duration_ms` | Histogram | End-to-end latency including reasoning | P95 > 45s |
| Proxy | `proxy_token_throughput` | Gauge | Tokens/sec (completion only) | < 12 TPS sustained |
| Artifact Store | `artifact_store_disk_usage_pct` | Gauge | Percentage of mounted volume used | > 80% (Warning), > 90% (Critical) |
| Artifact Store | `artifact_dedup_ratio` | Gauge | Ratio of skipped writes due to hash collision | < 0.15 (indicates storage inefficiency) |
| Reasoning | `reasoning_budget_violations_total` | Counter | Total soft/hard budget breaches | > 5 per hour |
| Privacy | `redaction_pipeline_latency_ms` | Histogram | Time spent in regex/policy engine | P99 > 150ms |
| Database | `sqlite_wal_size_bytes` | Gauge | Write-Ahead Log size | > 500MB |
| Database | `sqlite_lock_contentions_total` | Counter | `SQLITE_BUSY` or `SQLITE_LOCKED` events | > 10 per minute |

### Capacity Planning & Scaling Guidelines
- **SQLite Limits**: The database is tuned for read-heavy workloads with `PRAGMA cache_size=-64000` (64MB shared cache) and `PRAGMA temp_store=MEMORY`. Concurrent write contention is mitigated by batching artifact metadata inserts and using `INSERT OR IGNORE` for idempotent upserts.
- **Artifact Volume Growth**: At 50k tokens/response and 100 requests/hour, daily storage growth is ~2.5GB. Lifecycle policies archive artifacts after 30 days and purge after 90 days. Disk quotas are enforced at the container level via `cgroup` limits.
- **Memory Management**: FastAPI workers are configured with `--workers 4` and `--max-requests 1000` to prevent memory leaks from long-running streaming connections. The reasoning interceptor buffers chunks in memory but flushes to disk after 5MB or 30s of inactivity.

### Incident Response Playbook
1. **High Latency / Timeout Spikes**: Check `llama-cpp-server` health endpoint. If `cache_hit_rate` < 0.4, clear KV cache via `POST /v1/cache/clear`. Verify disk I/O saturation; throttle concurrency via `BENCHMARK_CONCURRENCY` env var.
2. **SQLite Lock Contention**: Enable `PRAGMA busy_timeout=5000`. Review concurrent write patterns. If WAL file exceeds 1GB, trigger `VACUUM` during maintenance window.
3. **Artifact Store Corruption**: Run `python -m app.artifacts verify-integrity --scan-depth 3`. Quarantine corrupted files to `/artifacts/quarantine/`. Re-ingest from backup if hash mismatch persists.
4. **Privacy Policy Bypass**: Audit `redaction_logs` for `policy_applied=null`. Temporarily switch to `strict` mode. Review regex patterns for false negatives.

## Extended Data Model Specifications & Migration Strategy

The relational schema requires careful migration handling to preserve historical telemetry while introducing new constraints. Below are the complete migration scripts, index optimization strategies, and foreign key enforcement procedures.

### Migration Script v2.0 (SQLite)
```sql
-- Enable strict foreign key enforcement
PRAGMA foreign_keys = ON;
PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = -64000;

-- 1. Create artifact store with deduplication constraints
CREATE TABLE IF NOT EXISTS artifacts (
    id TEXT PRIMARY KEY,
    content_hash TEXT NOT NULL UNIQUE,
    mime_type TEXT NOT NULL DEFAULT 'application/json',
    size_bytes INTEGER NOT NULL CHECK(size_bytes >= 0),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    storage_path TEXT NOT NULL,
    lifecycle_status TEXT DEFAULT 'active' CHECK(lifecycle_status IN ('active', 'archived', 'quarantined', 'deleted')),
    checksum_sha256 TEXT NOT NULL
);

-- 2. Evolve llm_requests to reference artifacts
ALTER TABLE llm_requests ADD COLUMN request_artifact_id TEXT REFERENCES artifacts(id);
ALTER TABLE llm_requests ADD COLUMN response_artifact_id TEXT REFERENCES artifacts(id);
ALTER TABLE llm_requests ADD COLUMN reasoning_tokens INTEGER DEFAULT 0;
ALTER TABLE llm_requests ADD COLUMN reasoning_time_ms INTEGER DEFAULT 0;
ALTER TABLE llm_requests ADD COLUMN redaction_status TEXT DEFAULT 'clean';

-- 3. Reasoning budget configuration
CREATE TABLE IF NOT EXISTS reasoning_budgets (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    model_profile TEXT NOT NULL,
    max_reasoning_tokens INTEGER NOT NULL DEFAULT 4096,
    max_reasoning_time_ms INTEGER NOT NULL DEFAULT 30000,
    enforcement_mode TEXT DEFAULT 'soft' CHECK(enforcement_mode IN ('soft', 'hard', 'disabled')),
    active BOOLEAN DEFAULT TRUE,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- 4. Audit trail for model changes
CREATE TABLE IF NOT EXISTS model_profile_audit (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    actor TEXT NOT NULL,
    action TEXT NOT NULL CHECK(action IN ('deploy', 'update', 'rollback', 'parameter_change', 'cache_clear')),
    previous_profile TEXT,
    new_profile TEXT NOT NULL,
    parameters_changed TEXT,
    reason TEXT,
    verification_hash TEXT,
    rollback_reference_id INTEGER
);

-- 5. Redaction tracking
CREATE TABLE IF NOT EXISTS redaction_logs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    request_id INTEGER REFERENCES llm_requests(id),
    field_type TEXT NOT NULL,
    original_length INTEGER,
    redacted_length INTEGER,
    policy_applied TEXT NOT NULL,
    confidence_score REAL DEFAULT 1.0,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- 6. Benchmark campaign metadata
CREATE TABLE IF NOT EXISTS benchmark_campaigns (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    description TEXT,
    config_hash TEXT NOT NULL,
    started_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    completed_at TIMESTAMP,
    status TEXT DEFAULT 'running' CHECK(status IN ('running', 'completed', 'failed', 'cancelled')),
    hardware_snapshot_path TEXT
);

CREATE TABLE IF NOT EXISTS benchmark_runs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_id TEXT REFERENCES benchmark_campaigns(id),
    prompt_template_id TEXT NOT NULL,
    model_profile TEXT NOT NULL,
    hardware_snapshot TEXT,
    result_artifact_id TEXT REFERENCES artifacts(id),
    metrics_json TEXT,
    screenshot_path TEXT,
    status TEXT DEFAULT 'pending' CHECK(status IN ('pending', 'running', 'success', 'failed', 'timeout')),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Index optimization for dashboard queries
CREATE INDEX IF NOT EXISTS idx_requests_run ON llm_requests(run_id);
CREATE INDEX IF NOT EXISTS idx_requests_model ON llm_requests(model_profile);
CREATE INDEX IF NOT EXISTS idx_requests_created ON llm_requests(created_at);
CREATE INDEX IF NOT EXISTS idx_requests_status ON llm_requests(status);
CREATE INDEX IF NOT EXISTS idx_artifacts_hash ON artifacts(content_hash);
CREATE INDEX IF NOT EXISTS idx_artifacts_lifecycle ON artifacts(lifecycle_status, created_at);
CREATE INDEX IF NOT EXISTS idx_bench_campaign ON benchmark_runs(campaign_id);
CREATE INDEX IF NOT EXISTS idx_bench_model ON benchmark_runs(model_profile);
CREATE INDEX IF NOT EXISTS idx_redaction_request ON redaction_logs(request_id);
CREATE INDEX IF NOT EXISTS idx_audit_timestamp ON model_profile_audit(timestamp);
```

### Migration Execution & Validation
1. **Backup**: `cp /models/flight-recorder/data/flight-recorder.db /backup/flight-recorder.db.pre-v2`
2. **Execute**: `sqlite3 /models/flight-recorder/data/flight-recorder.db < migrations/v2.0_schema.sql`
3. **Backfill**: Run `python -m app.migrations backfill-artifacts` to extract inline BLOBs into the new artifact store.
4. **Validate**: 
   ```sql
   SELECT COUNT(*) FROM llm_requests WHERE response_artifact_id IS NOT NULL;
   SELECT COUNT(*) FROM artifacts WHERE lifecycle_status = 'active';
   PRAGMA integrity_check;
   ```
5. **Rollback**: If validation fails, restore backup and revert to v1.9 image.

## Artifact Storage Lifecycle & Integrity Management

The artifact store implements a three-tier lifecycle policy with automated garbage collection, integrity verification, and deduplication.

### Lifecycle State Machine
```
[INGEST] -> (Hash Check) -> [EXISTS?] -> Yes: Return ID
                              -> No: Write to Disk -> Generate Preview -> Insert DB -> [ACTIVE]
[ACTIVE] -> (30 days) -> [ARCHIVED] -> (60 days) -> [DELETED]
[ARCHIVED] -> (Integrity Check Fail) -> [QUARANTINED]
```

### Garbage Collection Cron Job
```bash
# /etc/cron.d/flight-recorder-gc
0 2 * * * root /usr/bin/python3 -m app.artifacts gc --dry-run=false --log-level=INFO
```
The GC script:
- Scans `artifacts` table for `created_at < NOW() - 30 days`
- Updates status to `archived`
- Compresses files with `zstd -19`
- After 90 days, removes files and updates DB to `deleted`
- Logs disk space reclaimed and artifact counts

### Integrity Verification
```python
import hashlib
import os
from pathlib import Path

def verify_artifact_integrity(artifact_id: str, storage_path: str) -> bool:
    file_path = Path(storage_path)
    if not file_path.exists():
        return False
    computed_hash = hashlib.sha256(file_path.read_bytes()).hexdigest()
    return computed_hash == artifact_id
```
Integrity checks run nightly. Corrupted artifacts are moved to `/artifacts/quarantine/` and flagged in the dashboard.

## Reasoning Budget Enforcement & Streaming Interruption

The reasoning budget handler operates as a stateful middleware that intercepts SSE streams, parses reasoning boundaries, and enforces limits without breaking the HTTP/1.1 connection.

### State Machine Implementation
```python
class ReasoningInterceptor:
    def __init__(self, budget_config):
        self.max_tokens = budget_config["max_reasoning_tokens"]
        self.max_time_ms = budget_config["max_reasoning_time_ms"]
        self.mode = budget_config["enforcement_mode"]
        self.reasoning_tokens = 0
        self.start_time = None
        self.in_reasoning_block = False
        self.buffer = bytearray()

    async def process_chunk(self, chunk: bytes):
        if self.start_time is None:
            self.start_time = time.time()
        
        self.buffer.extend(chunk)
        text = self.buffer.decode("utf-8", errors="ignore")
        
        # Detect reasoning boundaries
        if "<thinking>" in text:
            self.in_reasoning_block = True
        if "</thinking>" in text:
            self.in_reasoning_block = False
            self.buffer.clear()
            
        if self.in_reasoning_block:
            # Approximate token count (4 chars per token avg)
            self.reasoning_tokens += len(text) // 4
            
            if self.mode == "hard" and self.reasoning_tokens > self.max_tokens:
                raise ReasoningBudgetExceeded("Hard limit reached")
                
            elapsed = (time.time() - self.start_time) * 1000
            if self.mode == "hard" and elapsed > self.max_time_ms:
                raise ReasoningBudgetExceeded("Time limit reached")
                
        return chunk
```

### Tokenization Accuracy
For production-grade accuracy, the system integrates `tiktoken` or `sentencepiece` models matching the base tokenizer of the active GGUF. The proxy loads the tokenizer on startup and caches it in memory. Token counting is performed on reasoning blocks post-stream or via incremental decoding for hard enforcement.

## Benchmark Runner Async Queue & Hardware Telemetry

The benchmark runner utilizes `asyncio.Queue` with bounded concurrency, hardware telemetry collection, and deterministic result aggregation.

### Async Task Queue Configuration
```python
import asyncio
import psutil
import json

async def benchmark_worker(queue: asyncio.Queue, proxy_client, campaign_id):
    while True:
        task = await queue.get()
        try:
            # Capture hardware state
            hw_snapshot = {
                "cpu_percent": psutil.cpu_percent(interval=1),
                "memory_available_gb": psutil.virtual_memory().available / (1024**3),
                "disk_io_read_bytes": psutil.disk_io_counters().read_bytes,
                "llama_cache_hit_rate": await proxy_client.get("/v1/health").json().get("cache_hit_rate", 0)
            }
            
            # Execute request
            response = await proxy_client.post("/v1/chat/completions", json=task["payload"])
            artifact_id = await store_artifact(response.content)
            
            # Record run
            await db.execute(
                "INSERT INTO benchmark_runs (campaign_id, prompt_template_id, model_profile, hardware_snapshot, result_artifact_id, status) VALUES (?, ?, ?, ?, ?, 'success')",
                (campaign_id, task["template_id"], task["model"], json.dumps(hw_snapshot), artifact_id)
            )
        except Exception as e:
            await db.execute(
                "INSERT INTO benchmark_runs (campaign_id, prompt_template_id, model_profile, status) VALUES (?, ?, ?, 'failed')",
                (campaign_id, task["template_id"], task["model"])
            )
        finally:
            queue.task_done()
```

### Result Aggregation Algorithm
Post-campaign, the runner computes:
- **Throughput**: `completion_tokens / (total_latency_ms - reasoning_time_ms)`
- **Reasoning Efficiency**: `reasoning_tokens / total_tokens`
- **Stability**: Standard deviation of latency across runs
- **Error Rate**: `(failed + timeout) / total_runs`
Results are serialized to JSON and linked to the campaign artifact.

## Reporting Plane Caching & Real-Time Streams

The reporting layer implements a dual-tier caching strategy to maintain sub-second dashboard responsiveness under high load.

### Cache Architecture
- **L1 Cache**: In-memory `asyncio` cache with 60s TTL for frequently accessed metrics (throughput, latency percentiles)
- **L2 Cache**: SQLite materialized views refreshed every 5 minutes
- **Invalidation**: Triggered on new artifact ingestion or campaign completion

### Materialized View Definition
```sql
CREATE MATERIALIZED VIEW IF NOT EXISTS mv_dashboard_metrics AS
SELECT 
    model_profile,
    COUNT(*) AS total_requests,
    AVG(total_latency_ms) AS avg_latency,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY total_latency_ms) AS p95_latency,
    AVG(completion_tokens / NULLIF(total_latency_ms, 0)) AS avg_throughput,
    SUM(CASE WHEN status != 'success' THEN 1 ELSE 0 END) AS error_count
FROM llm_requests
WHERE created_at >= datetime('now', '-24 hours')
GROUP BY model_profile;
```
Refreshed via `REFRESH MATERIALIZED VIEW mv_dashboard_metrics;` in a background task.

### Real-Time WebSocket Stream
The dashboard subscribes to `/ws/metrics` which pushes incremental updates:
```json
{"type": "metric_update", "model": "Qwen3.6-35B", "latency_p95": 12400, "throughput": 18.5, "reasoning_ratio": 0.32}
```
Frontend uses virtualized lists to render large result sets without DOM thrashing.

## Privacy Redaction Pipeline & Compliance Mapping

The redaction engine implements a multi-stage pipeline with policy versioning, false-positive handling, and audit compliance.

### Policy Engine Configuration
```json
{
  "version": "2.1.0",
  "policies": [
    {
      "id": "corp_comms_v1",
      "name": "Corporate Communications",
      "patterns": [
        {"type": "email", "regex": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "replacement": "[REDACTED_EMAIL]"},
        {"type": "teams_id", "regex": "TEAM-[A-Z0-9]{4,}", "replacement": "[REDACTED_TEAM_ID]"},
        {"type": "internal_url", "regex": "https?://internal\\.corp\\.local/.*", "replacement": "[REDACTED_URL]"}
      ],
      "action": "replace",
      "log_level": "info"
    },
    {
      "id": "pii_v1",
      "name": "PII Protection",
      "patterns": [
        {"type": "ssn", "regex": "\\b\\d{3}-\\d{2}-\\d{4}\\b", "replacement": "[REDACTED_SSN]"},
        {"type": "phone", "regex": "\\b\\d{3}[-.]\\d{3}[-.]\\d{4}\\b", "replacement": "[REDACTED_PHONE]"}
      ],
      "action": "mask",
      "log_level": "warning"
    }
  ],
  "allowlist": ["public@example.com", "TEAM-0000", "docs.internal.corp.local"],
  "fallback": "pass_through",
  "max_latency_ms": 100
}
```

### False-Positive Mitigation
- **Context Window Analysis**: Regex matches within code blocks or markdown tables are exempted via AST parsing.
- **Confidence Scoring**: Matches with low entropy or structural mismatches are flagged for manual review.
- **Allowlist Propagation**: Known safe identifiers are cached and bypass regex compilation.

### Compliance Mapping
- **GDPR/CCPA**: PII redaction logs retained for 7 years. Right-to-erasure implemented via artifact tombstoning.
- **SOC 2**: Audit trails for all policy changes. Access controls via JWT role-based filtering.
- **Internal Policy**: Corporate data never leaves `/models/flight-recorder/data/`. Network egress blocked via firewall rules.

## Deployment Automation & CI/CD Integration

The deployment pipeline integrates with GitHub Actions for automated testing, migration execution, and rolling updates.

### GitHub Actions Workflow
```yaml
name: Deploy Flight Recorder
on:
  push:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest app/tests/ --cov=app --cov-report=xml
  deploy:
    needs: test
    runs-on: self-hosted-runner
    steps:
      - run: docker compose pull
      - run: docker compose up -d migration-runner
      - run: docker compose up -d flight-recorder
      - run: ./scripts/validate-deployment.sh
```

### Health Check & Readiness Probes
- **Startup Probe**: `GET /health` returns 200 with `status: "ready"` after DB migration completes
- **Liveness Probe**: `GET /health` checks SQLite connectivity and artifact store writability
- **Readiness Probe**: Verifies reasoning budget config loaded and redaction pipeline initialized

### Environment Variable Matrix
| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | `sqlite:////data/flight-recorder.db` | DB connection string |
| `ARTIFACT_ROOT` | `/artifacts` | Base path for artifact storage |
| `REDACTION_ENABLED` | `true` | Toggle redaction pipeline |
| `REASONING_BUDGET_HARD` | `false` | Enable hard enforcement |
| `MAX_ARTIFACT_SIZE_MB` | `50` | Reject payloads exceeding limit |
| `DISK_QUOTA_ALERT_PCT` | `80` | Trigger warning threshold |
| `BENCHMARK_CONCURRENCY` | `4` | Max parallel benchmark tasks |
| `UI_PREVIEW_LIMIT_KB` | `2` | Max preview size for frontend |

## Comprehensive Test Automation & Acceptance Criteria

The test suite expands to 30+ acceptance tests with full automation, CI integration, and synthetic data generation.

### Test Automation Framework
- **Unit Tests**: `pytest` with `pytest-asyncio` for async interceptors
- **Integration Tests**: Testcontainers for SQLite, mock llama.cpp server
- **Load Tests**: `k6` scripts simulating 50k token responses
- **Security Tests**: `bandit` for static analysis, `pytest-redaction` for privacy validation

### Extended Test Matrix (25-30)
| ID | Category | Description | Setup | Expected Outcome |
|----|----------|-------------|-------|------------------|
| T25 | Concurrency | 50 parallel requests | `k6 run load-test.js` | No SQLite locks, P95 < 30s |
| T26 | Disk Quota | Fill to 85% | `dd if=/dev/zero of=/artifacts/filler` | Alert logged, GC triggered |
| T27 | Integrity | Corrupt artifact hash | Modify file bytes | Quarantined, dashboard flagged |
| T28 | Policy Update | Hot-reload redaction rules | POST `/admin/policies` | New patterns applied immediately |
| T29 | Cache Invalidation | Clear KV cache | POST `/v1/cache/clear` | Cache hit rate drops to 0 |
| T30 | Export | CSV export of campaign | GET `/reports/export?format=csv` | Valid CSV with all metrics |
| T31 | WebSocket | Real-time metric stream | WS connect `/ws/metrics` | Updates every 5s, no disconnects |
| T32 | Migration Rollback | Revert to v1.9 | Run rollback script | Schema matches v1, data intact |
| T33 | Timeout Handling | 120s proxy timeout | Slow mock server | 504 response, partial artifact saved |
| T34 | Memory Leak | 10k requests over 1h | `valgrind` or `tracemalloc` | Memory stable, no growth > 5% |

### k6 Load Test Script
```javascript
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 10,
  duration: '5m',
  thresholds: {
    http_req_duration: ['p(95)<30000'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const payload = JSON.stringify({
    model: 'Qwen3.6-35B',
    messages: [{ role: 'user', content: 'Generate a 50k token analysis...' }],
    stream: true
  });
  
  const res = http.post('http://localhost:8000/v1/chat/completions', payload, {
    headers: { 'Content-Type': 'application/json' }
  });
  
  check(res, {
    'status is 200': (r) => r.status === 200,
    'has artifact_id': (r) => JSON.parse(r.body).artifact_id !== undefined,
  });
  
  sleep(1);
}
```

## Rollback Automation & Data Reconciliation

The rollback procedure is automated via a state machine that validates data integrity before reverting.

### Automated Rollback Triggers
- Health check failures > 3 consecutive
- SQLite corruption detected
- Artifact store disk full
- Manual trigger via `/admin/rollback`

### Rollback Execution Script
```bash
#!/bin/bash
# scripts/rollback.sh
set -e

BACKUP_DIR="/backup/flight-recorder"
DB_PATH="/models/flight-recorder/data/flight-recorder.db"
ARTIFACT_PATH="/models/flight-recorder/data/artifacts"

echo "Stopping services..."
docker compose stop flight-recorder

echo "Restoring database..."
cp "$BACKUP_DIR/flight-recorder.db.v1" "$DB_PATH"

echo "Archiving current artifacts..."
mv "$ARTIFACT_PATH" "${ARTIFACT_PATH}.v2"
mkdir -p "$ARTIFACT_PATH"

echo "Deploying previous image..."
docker compose up -d --build flight-recorder

echo "Validating rollback..."
curl -f http://localhost:8000/health || exit 1

echo "Rollback complete. Monitoring for 24h."
```

### Data Reconciliation
- Cross-reference `llm_requests` count with backup
- Verify artifact paths point to valid files
- Check `model_profile_audit` for consistency
- Validate redaction logs integrity

## Open Questions & Decision Frameworks

The following open questions require stakeholder alignment. Each includes impact analysis and recommended decision paths.

### 1. Storage Backend Evolution
- **Impact**: Local filesystem limits scalability and backup complexity. S3-compatible storage enables cross-region replication and lifecycle policies.
- **Decision Framework**: Evaluate cost vs. complexity. If lab storage < 2TB, local FS suffices. If > 2TB or multi-node deployment planned, migrate to MinIO/S3.
- **Recommendation**: Implement abstracted `StorageBackend` interface. Start with local FS, add S3 adapter later.

### 2. Redaction False-Positive Tolerance
- **Impact**: Over-redaction degrades telemetry utility. Under-redaction risks compliance violations.
- **Decision Framework**: Target < 2% false-positive rate. Implement confidence scoring and manual review queue for edge cases.
- **Recommendation**: Enable `strict` mode for PII, `standard` for corporate data. Log all redactions for audit.

### 3. Benchmark Concurrency Limits
- **Impact**: High concurrency saturates llama.cpp KV cache and CPU threads, skewing results.
- **Decision Framework**: Profile hardware limits. Set `BENCHMARK_CONCURRENCY` to `threads / 2`. Implement queue backpressure.
- **Recommendation**: Default to 4. Allow override with warning if > 8.

### 4. UI Framework Constraints
- **Impact**: Frontend OOM on large JSON. Virtual scrolling required for performance.
- **Decision Framework**: Audit frontend framework. If React/Vue, implement `react-window` or `vue-virtual-scroller`.
- **Recommendation**: Enforce `UI_PREVIEW_LIMIT_KB=2`. Lazy-load full artifacts on demand.

### 5. Monitoring Stack Integration
- **Impact**: SQLite reporting lacks real-time alerting. Prometheus/Grafana provides historical analysis.
- **Decision Framework**: Deploy lightweight Prometheus sidecar. Export `/metrics` endpoint.
- **Recommendation**: Integrate Prometheus. Configure alerts for P95 latency, disk usage, budget violations.

### 6. Reasoning Token Accuracy
- **Impact**: Word-split approximation undercounts by ~15%. Proper tokenizer required for precision.
- **Decision Framework**: Load `tiktoken` or `sentencepiece` on startup. Cache tokenizer.
- **Recommendation**: Integrate exact tokenizer. Fallback to approximation if model mismatch.

### 7. Artifact Cleanup Policy Granularity
- **Impact**: One-size-fits-all retention may delete valuable benchmark data prematurely.
- **Decision Framework**: Tag artifacts by `campaign_id` or `model_profile`. Apply tiered retention.
- **Recommendation**: Benchmark artifacts retained 1 year. Telemetry artifacts 90 days.

### 8. Audit Log Retention & Archival
- **Impact**: Compliance requires 7-year retention. SQLite growth becomes unmanageable.
- **Decision Framework**: Archive audit logs to cold storage after 1 year. Implement partitioning.
- **Recommendation**: Export to Parquet quarterly. Prune SQLite after archival.

### 9. Dynamic Timeout Configuration
- **Impact**: Fixed timeouts fail for high-token prompts. Dynamic timeouts adapt to expected output.
- **Decision Framework**: Calculate timeout = `base_ms + (expected_tokens * avg_generation_time_ms)`.
- **Recommendation**: Implement dynamic timeout with max cap of 300s.

### 10. Screenshot Metadata Storage
- **Impact**: Base64 in DB bloats storage. Artifact reference maintains integrity.
- **Decision Framework**: Store screenshots as artifacts. Reference via ID in `benchmark_runs`.
- **Recommendation**: Use artifact store for screenshots. Embed thumbnail in DB.

## Final Deployment Readiness Checklist

Before production sign-off, verify the following:
- [ ] All migrations executed successfully with `PRAGMA integrity_check` passing
- [ ] Artifact store deduplication verified with synthetic duplicate payloads
- [ ] Reasoning budget enforcement tested in soft and hard modes
- [ ] Redaction pipeline validated against synthetic PII corpus
- [ ] Benchmark runner completes campaign with hardware snapshots
- [ ] Dashboard renders sub-second with 10k+ records
- [ ] Rollback procedure tested end-to-end
- [ ] Monitoring alerts configured and triggered in staging
- [ ] CI/CD pipeline green with 100% test coverage on critical paths
- [ ] Documentation updated with operational runbooks and API specs

This dossier provides a complete, deployment-grade blueprint for modernizing the AI Flight Recorder. All components are designed for zero-downtime rollout, strict privacy compliance, and scalable telemetry handling. Implementation should follow the migration sequence, validate each subsystem independently, and integrate monitoring before enabling high-concurrency workloads.