# Executive Summary

This engineering dossier outlines the architectural evolution, operational hardening, and deployment strategy for the `ai-flight-recorder` home-lab service. The system currently functions as a functional prototype: a FastAPI proxy that forwards OpenAI-compatible requests to a local `llama.cpp` server, records request/response metadata into SQLite, and exposes dashboard/reporting endpoints. While operational, the current implementation exhibits critical limitations when pushed toward sustained, high-volume, or production-adjacent workloads.

The six identified pain points drive the scope of this redesign:
1. **Durable Artifact Storage:** Inline JSON payloads in SQLite become unwieldy and degrade query performance as benchmark campaigns generate multi-megabyte outputs.
2. **Reasoning-Aware Token Accounting:** The current proxy counts all output tokens uniformly, obscuring the distinction between chain-of-thought reasoning and final completion, which is critical for cost/latency modeling and model comparison.
3. **Benchmark Metadata Richness:** Reports lack the structured metadata required for reproducible screenshots, model-to-model comparisons, and statistical significance tracking.
4. **Privacy Leakage Mitigation:** The proxy currently persists raw request/response bodies, creating a compliance and security risk when handling internal communications (email, Teams, proprietary code).
5. **Model Profile Auditability:** Swapping GGUF files or adjusting llama.cpp inference parameters currently leaves no trace, making reproducibility and incident investigation impossible.
6. **Long-Output Stress Handling:** Tens of thousands of output tokens trigger timeout cascades, UI rendering freezes, SQLite write contention, and preview generation failures.

This dossier provides a complete, deployment-grade blueprint to resolve these issues. It introduces a hybrid storage architecture (SQLite metadata + compressed artifact files), a reasoning-segment parser, a privacy redaction pipeline, an auditable model registry, a resilient benchmark runner, and a scalable reporting plane. The deployment plan includes Docker Compose hardening, health checks, backup strategies, and a phased rollout. The test plan contains 24 acceptance tests covering schema evolution, artifact lifecycle, reasoning accounting, privacy filtering, benchmark execution, reporting accuracy, and rollback procedures. All synthetic data examples use fabricated identifiers to maintain security hygiene.

The target audience for this document is a senior backend engineer or DevOps operator responsible for implementing, testing, and deploying the changes. No follow-up questions should be required to execute the plan, provided the home-lab environment matches the described topology (192.168.1.116, `/models/flight-recorder/data`, Python 3.11+, Docker Compose v2).

---

# Current Architecture

The `ai-flight-recorder` service is a self-contained Python/FastAPI application designed to intercept, proxy, and record LLM interactions. The architecture follows a monolithic deployment pattern optimized for local development and home-lab experimentation.

**Component Breakdown:**
- `app/main.py`: FastAPI application entry point. Mounts routers for `/dashboard`, `/health`, `/v1/chat/completions` (proxy), and `/reports`. Handles CORS, middleware, and basic routing.
- `app/proxy.py`: Core proxy logic. Accepts OpenAI-compatible JSON payloads, forwards them to `llama-cpp-server` running on `192.168.1.116:8080`, captures streaming or non-streaming responses, and writes metadata to SQLite via `store.py`.
- `app/store.py`: SQLite persistence layer. Uses `sqlite3` or `aiosqlite` with raw SQL or lightweight ORM. Tables include `runs`, `client_sessions`, `llm_requests`, `events`, and `benchmark_rows`. Writes are synchronous by default.
- `app/reports.py`: Aggregation engine. Runs SQL queries against SQLite to compute latency percentiles, token throughput, error rates, and usage summaries. Exposes endpoints for dashboard JSON and CSV exports.
- `app/bench.py`: Benchmark runner. Accepts campaign configurations (prompts, model parameters, concurrency), executes requests against the proxy, and stores results in `benchmark_rows`. Lacks retry logic, artifact linking, and progress tracking.
- `deploy/docker-compose.yml`: Orchestrates two services: `ai-flight-recorder` (Python app) and `llama-cpp-server` (C++ binary). Shares `/models/flight-recorder/data` via volume mount. Network is bridge-based, internal to the host.
- **Model Profile:** `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` loaded with `llama.cpp` reasoning mode enabled (`--reasoning` or equivalent context-aware thinking flags). Context window typically set to 32k or 64k tokens.

**Data Flow:**
1. Client sends `POST /v1/chat/completions` with `messages`, `model`, `max_tokens`, `stream` flags.
2. `proxy.py` validates payload, generates `request_id`, `session_id`, and `run_id`.
3. Request is forwarded to `llama-cpp-server` on `192.168.1.116:8080/v1/chat/completions`.
4. Response (streaming or non-streaming) is captured. Token counts, latency, and a truncated preview (first 500 chars) are written to SQLite.
5. `reports.py` aggregates data on demand. `bench.py` drives campaigns.
6. Dashboard serves aggregated metrics and recent runs.

**Current Limitations:**
- SQLite is used for both metadata and large JSON payloads, causing page cache bloat and write lock contention during concurrent benchmarks.
- No distinction between reasoning tokens and completion tokens; `usage.completion_tokens` includes `<think>` blocks.
- Privacy filtering is absent; raw prompts and responses are persisted verbatim.
- Model swaps (GGUF replacement, `--ctx-size`, `--gpu-layers`) are not logged or versioned.
- Benchmark runner lacks durability; process termination loses in-flight campaigns.
- UI preview generation fails or freezes on outputs >10k tokens due to synchronous string slicing and JSON serialization.

---

# Failure Modes Found

A systematic failure mode and effects analysis (FMEA) reveals the following critical risks in the current architecture:

| ID | Failure Mode | Severity | Likelihood | Impact | Root Cause |
|----|--------------|----------|------------|--------|------------|
| FM-01 | SQLite Write Lock Contention | High | High | Benchmark timeouts, dropped requests, dashboard lag | Synchronous writes during streaming responses; no connection pooling or WAL optimization |
| FM-02 | OOM on Long Outputs | Critical | Medium | Proxy crash, llama.cpp OOM, data loss | Inline JSON storage of 50k+ token responses; no chunking or streaming persistence |
| FM-03 | Timeout Cascades | High | High | Benchmark campaigns fail silently; UI hangs | Default `httpx`/`aiohttp` timeouts (30s) insufficient for long reasoning/completion; no retry/backoff |
| FM-04 | Privacy Leakage | Critical | Medium | Compliance violation, credential exposure | Raw request/response bodies persisted without filtering; logs contain PII |
| FM-05 | Untracked Model Swaps | Medium | High | Irreproducible benchmarks, silent performance regression | No model registry; GGUF replacement not audited; llama.cpp flags not recorded |
| FM-06 | Reasoning Token Miscounting | Medium | High | Inaccurate latency/cost modeling; flawed model comparisons | `usage.completion_tokens` includes `<think>` blocks; no segment parsing |
| FM-07 | UI Rendering Freeze | Medium | Medium | Dashboard unusable during/after long benchmarks | Synchronous JSON serialization of large payloads; no virtualization or pagination |
| FM-08 | Benchmark State Loss | High | Medium | Campaigns must be rerun; data inconsistency | No checkpointing; process termination drops in-flight runs; no artifact linking |
| FM-09 | Preview Generation Failure | Low | Medium | Dashboard shows empty/error previews | String slicing on malformed UTF-8 or extremely long strings; no fallback |
| FM-10 | Storage Bloat | Medium | High | Disk exhaustion, backup failures | No cleanup policy; artifacts stored inline; no compression |

**Mitigation Strategy:** Each failure mode maps directly to a section in this dossier. FM-01/02/07/09/10 are addressed via Artifact Storage Design and Data Model Changes. FM-03 via Benchmark Runner Design and Deployment Plan. FM-04 via Privacy And Redaction. FM-05 via Data Model Changes and Deployment Plan. FM-06 via Reasoning Budget Handling. FM-08 via Benchmark Runner Design and Artifact Storage.

---

# Data Model Changes

SQLite schema evolution must be backward-compatible, idempotent, and optimized for write-heavy benchmark workloads. The following DDL additions and modifications are required.

**Migration Strategy:**
- Use `alembic` or raw SQL migration scripts stored in `deploy/migrations/`.
- Enable WAL mode: `PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; PRAGMA cache_size=-64000;`
- Create indexes on foreign keys and timestamp columns.
- Run migrations before service startup via `app/migrate.py`.

**New Tables:**

```sql
-- Audit log for model profile changes
CREATE TABLE IF NOT EXISTS model_audit_log (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    run_id TEXT NOT NULL,
    model_hash TEXT NOT NULL,
    model_filename TEXT NOT NULL,
    llama_cpp_flags TEXT NOT NULL,
    ctx_size INTEGER NOT NULL,
    gpu_layers INTEGER NOT NULL,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
    operator TEXT DEFAULT 'system',
    FOREIGN KEY (run_id) REFERENCES runs(id)
);

-- Reasoning segment tracking
CREATE TABLE IF NOT EXISTS reasoning_segments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    request_id TEXT NOT NULL,
    segment_index INTEGER NOT NULL,
    start_token INTEGER NOT NULL,
    end_token INTEGER NOT NULL,
    token_count INTEGER NOT NULL,
    content_preview TEXT,
    FOREIGN KEY (request_id) REFERENCES llm_requests(id)
);

-- Extended benchmark metadata
CREATE TABLE IF NOT EXISTS benchmark_runs_extended (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_id TEXT NOT NULL,
    run_id TEXT NOT NULL,
    artifact_path TEXT NOT NULL,
    artifact_hash TEXT NOT NULL,
    artifact_size_bytes INTEGER NOT NULL,
    reasoning_token_count INTEGER DEFAULT 0,
    completion_token_count INTEGER DEFAULT 0,
    total_token_count INTEGER NOT NULL,
    latency_ms REAL NOT NULL,
    error_code TEXT,
    screenshot_metadata TEXT,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (run_id) REFERENCES runs(id)
);

-- Artifact metadata (if not using filesystem-only)
CREATE TABLE IF NOT EXISTS artifact_registry (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    hash TEXT UNIQUE NOT NULL,
    path TEXT NOT NULL,
    size_bytes INTEGER NOT NULL,
    compression TEXT DEFAULT 'zstd',
    chunk_count INTEGER DEFAULT 1,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    expires_at DATETIME
);
```

**Indexing Strategy:**
```sql
CREATE INDEX idx_llm_requests_run_id ON llm_requests(run_id);
CREATE INDEX idx_llm_requests_created_at ON llm_requests(created_at);
CREATE INDEX idx_benchmark_extended_campaign ON benchmark_runs_extended(campaign_id);
CREATE INDEX idx_benchmark_extended_created ON benchmark_runs_extended(created_at);
CREATE INDEX idx_model_audit_model_hash ON model_audit_log(model_hash);
CREATE INDEX idx_reasoning_request_id ON reasoning_segments(request_id);
```

**Migration Notes:**
- Run `PRAGMA journal_mode=WAL;` immediately after schema upgrade.
- Backfill existing `llm_requests` with `reasoning_token_count=0` and `completion_token_count=total_tokens`.
- Populate `artifact_registry` with existing inline payloads by extracting and hashing them.
- Verify foreign key constraints: `PRAGMA foreign_keys=ON;`
- Test migration on a copy of `/models/flight-recorder/data/flight_recorder.db` before production.

---

# Artifact Storage Design

Inline JSON storage is replaced with a hybrid architecture: SQLite holds metadata and pointers, while actual payloads are stored as compressed, chunked files on the host filesystem.

**Directory Structure:**
```
/models/flight-recorder/data/
├── artifacts/
│   ├── 2024/
│   │   ├── 11/
│   │   │   ├── 15/
│   │   │   │   ├── a1b2c3d4e5f6..._chunk_0.zst
│   │   │   │   ├── a1b2c3d4e5f6..._chunk_1.zst
│   │   │   │   └── a1b2c3d4e5f6..._meta.json
│   │   │   └── ...
│   │   └── ...
│   └── ...
├── previews/
│   ├── thumb_a1b2c3d4e5f6..._500.txt
│   └── ...
└── flight_recorder.db
```

**Naming Convention:**
- Hash: `SHA256(payload)[:16]`
- Path: `{year}/{month}/{day}/{hash}_chunk_{n}.zst`
- Metadata: `{hash}_meta.json` containing `original_size`, `compressed_size`, `chunk_count`, `created_at`, `request_id`, `campaign_id`

**Chunking Strategy:**
- Max chunk size: 2MB uncompressed.
- Compression: `zstd` level 3 (balance of speed/ratio).
- Streaming write: Proxy writes chunks as they arrive; no full payload buffering in memory.
- Atomicity: Write to temp file, then rename. Use `os.replace()` for atomicity.

**Artifact Lifecycle:**
- Creation: Proxy writes chunks on response completion.
- Preview Generation: Extract first 500 chars, strip control characters, save to `previews/`.
- Cleanup: Cron job runs nightly:
  ```bash
  find /models/flight-recorder/data/artifacts -name "*.zst" -mtime +30 -delete
  find /models/flight-recorder/data/previews -name "*.txt" -mtime +30 -delete
  sqlite3 flight_recorder.db "DELETE FROM artifact_registry WHERE expires_at < datetime('now');"
  ```
- Retention Policy: Configurable via `ARTIFACT_RETENTION_DAYS=30` env var.

**UI Integration:**
- Dashboard requests `/api/artifacts/{hash}/preview` → serves `previews/{hash}_500.txt`
- Full download: `/api/artifacts/{hash}/download` → streams chunks, verifies hash
- Pagination: `?page=1&limit=50` on `/api/benchmarks/{campaign_id}/runs`

**Performance Considerations:**
- Filesystem: Use `ext4` or `xfs` with `noatime`. Avoid network mounts for artifact storage.
- I/O: Async chunk writing via `asyncio.to_thread()` or `concurrent.futures`.
- Memory: Never load full artifact into RAM; stream to client or preview generator.

---

# Reasoning Budget Handling

The current proxy treats all output tokens uniformly. With `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` reasoning enabled, outputs contain `<think>` blocks that must be measured separately for accurate latency/cost modeling and model comparison.

**Token Segmentation Logic:**
1. **Parsing:** Detect `<think>` and `</think>` delimiters in the response text.
2. **Counting:** Use tiktoken or llama.cpp's internal tokenizer to count tokens in each segment.
3. **Tracking:** Store `reasoning_token_count`, `completion_token_count`, and `total_token_count` in `benchmark_runs_extended`.
4. **Streaming Support:** For streaming responses, accumulate tokens per chunk, segment boundaries, and flush to SQLite on completion.

**Implementation Sketch (`app/proxy.py`):**
```python
import tiktoken
import re

THINK_PATTERN = re.compile(r"<think>(.*?)</think>", re.DOTALL)

def split_reasoning_and_completion(text: str) -> dict:
    match = THINK_PATTERN.search(text)
    if match:
        reasoning = match.group(1).strip()
        completion = text[:match.start()] + text[match.end():]
    else:
        reasoning = ""
        completion = text
    
    enc = tiktoken.get_encoding("cl100k_base")
    return {
        "reasoning_tokens": len(enc.encode(reasoning)),
        "completion_tokens": len(enc.encode(completion)),
        "total_tokens": len(enc.encode(text))
    }
```

**llama.cpp Configuration:**
- Do not disable `--reasoning` or `--thinking` flags.
- Set `--max-tokens` appropriately (e.g., 8192) to prevent runaway generation.
- Use `--ctx-size 65536` to accommodate long reasoning chains.
- Enable `--log-disable` in production to reduce noise.

**Budget Enforcement:**
- Middleware checks `max_tokens` vs `reasoning_token_count + completion_token_count`.
- If `reasoning_tokens > 0.7 * max_tokens`, log warning and continue (do not abort, as reasoning is intentional).
- Record ratio in `benchmark_runs_extended.reasoning_ratio = reasoning_tokens / total_tokens`.

**Dashboard Metrics:**
- `reasoning_token_percentage` (histogram)
- `avg_reasoning_latency_ms`
- `completion_vs_reasoning_token_distribution`
- `reasoning_budget_exceeded_count`

**Testing:**
- Verify with synthetic prompts containing explicit `<think>` blocks.
- Confirm token counts match tiktoken output.
- Ensure streaming responses correctly segment tokens.

---

# Benchmark Runner Design

`bench.py` must evolve from a simple script to a resilient, auditable campaign engine.

**Architecture:**
- **Campaign Config:** YAML/JSON schema defining prompts, model parameters, concurrency, retries, artifact linking.
- **Worker Pool:** `asyncio.Semaphore`-bounded pool for concurrent requests.
- **Checkpointing:** Save in-flight state to `bench_state.json` every 10 seconds.
- **Retry Logic:** Exponential backoff (1s, 2s, 4s, 8s) with max 3 retries.
- **Progress Tracking:** Emit events to `events` table; dashboard polls `/api/benchmarks/{id}/progress`.
- **Graceful Shutdown:** Catch `SIGTERM`/`SIGINT`, flush checkpoints, close connections, exit cleanly.

**Campaign Schema (`bench_campaign.yaml`):**
```yaml
campaign_id: "qwen36-35b-reasoning-baseline-20241115"
model: "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf"
prompts:
  - id: "p1"
    text: "Explain quantum entanglement in simple terms."
  - id: "p2"
    text: "Write a Python function to merge two sorted arrays."
params:
  max_tokens: 4096
  temperature: 0.7
  top_p: 0.9
  stream: false
concurrency: 5
retries: 3
timeout_seconds: 120
artifact_policy: "compress_and_store"
metadata:
  author: "lab_admin"
  purpose: "reasoning_accuracy_baseline"
```

**CLI Interface:**
```bash
# Start campaign
python -m app.bench run --config bench_campaign.yaml --output-dir /models/flight-recorder/data/benchmarks

# Resume interrupted campaign
python -m app.bench resume --campaign-id qwen36-35b-reasoning-baseline-20241115

# Export results
python -m app.bench export --campaign-id qwen36-35b-reasoning-baseline-20241115 --format csv --output results.csv

# View progress
python -m app.bench status --campaign-id qwen36-35b-reasoning-baseline-20241115
```

**State Management:**
- `bench_state.json` contains `campaign_id`, `completed_prompts`, `failed_prompts`, `in_progress`, `last_checkpoint`.
- On resume, skip completed prompts, retry failed ones, continue in-progress.
- Atomic writes: write to `.tmp`, then rename.

**Error Handling:**
- Network errors: retry with backoff.
- Timeout: mark as `timeout`, log latency, continue.
- Parse error: mark as `malformed`, store raw response in artifact.
- OOM: abort campaign, log error, notify dashboard.

**Integration with Proxy:**
- Proxy returns `request_id`, `artifact_path`, `reasoning_tokens`, `completion_tokens`.
- Runner links these to `benchmark_runs_extended` and `artifact_registry`.
- Dashboard aggregates campaign-level metrics.

---

# Reporting Plane

`reports.py` must provide actionable insights, reproducible screenshots, and model comparison capabilities.

**Dashboard Metrics:**
- **Latency:** p50, p90, p99, max (ms)
- **Throughput:** tokens/sec, requests/min
- **Token Distribution:** reasoning vs completion ratio
- **Error Rates:** 4xx, 5xx, timeout, malformed
- **Privacy:** redaction hits, PII types detected
- **Model Audit:** version changes, flag adjustments

**Aggregation Queries:**
```sql
-- Campaign summary
SELECT 
    campaign_id,
    COUNT(*) as total_runs,
    AVG(latency_ms) as avg_latency,
    AVG(total_token_count) as avg_tokens,
    AVG(reasoning_token_count) as avg_reasoning,
    AVG(completion_token_count) as avg_completion,
    SUM(CASE WHEN error_code IS NOT NULL THEN 1 ELSE 0 END) as errors
FROM benchmark_runs_extended
WHERE created_at >= datetime('now', '-7 days')
GROUP BY campaign_id;

-- Model comparison matrix
SELECT 
    model_hash,
    model_filename,
    AVG(latency_ms) as avg_latency,
    AVG(total_token_count) as avg_tokens,
    AVG(reasoning_token_count) as avg_reasoning,
    AVG(completion_token_count) as avg_completion,
    COUNT(*) as total_runs
FROM benchmark_runs_extended br
JOIN model_audit_log mal ON br.run_id = mal.run_id
WHERE br.created_at >= datetime('now', '-30 days')
GROUP BY model_hash, model_filename;
```

**Export Pipeline:**
- CSV: `SELECT * FROM benchmark_runs_extended WHERE campaign_id = ?`
- JSON: Full metadata + artifact hashes
- PDF: Dashboard screenshot + summary table (via `weasyprint` or `reportlab`)
- Screenshot Metadata: Embed campaign ID, model hash, timestamp, prompt IDs in PNG metadata or sidecar JSON.

**UI Rendering Optimization:**
- Virtualized table for runs list (React/VirtualList or server-side pagination).
- Lazy-load artifact previews.
- Debounce filter/search inputs.
- Cache aggregation results for 5 minutes.

**Model Comparison Features:**
- Side-by-side latency/token charts.
- Reasoning ratio distribution.
- Error rate comparison.
- Export comparison matrix as CSV/JSON.

---

# Privacy And Redaction

The proxy must filter PII before persistence or logging. A multi-layered approach ensures compliance without significant performance degradation.

**Redaction Pipeline:**
1. **Regex Layer:** Fast, deterministic patterns for emails, phone numbers, SSNs, credit cards, Teams IDs, internal domains.
2. **LLM Fallback (Optional):** For ambiguous cases, send to a local small model (e.g., `Phi-3-mini`) for PII classification. Disabled by default for performance.
3. **Configurable Rules:** YAML-based rule engine allows lab admins to add custom patterns.
4. **Audit Trail:** Log redaction hits with type, count, and timestamp. Do not log redacted content.

**Regex Patterns (Synthetic Examples):**
```python
PII_PATTERNS = {
    "email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
    "phone": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b",
    "ssn": r"\b\d{3}-\d{2}-\d{4}\b",
    "teams_id": r"\b\d{10,}\b",  # Simplified; real Teams IDs are UUIDs
    "internal_domain": r"\b@(corp|internal|lab)\.example\.com\b"
}
```

**Implementation (`app/redact.py`):**
```python
import re
import json

class PrivacyFilter:
    def __init__(self, config_path: str = "privacy_rules.yaml"):
        self.rules = self._load_rules(config_path)
        self.compiled = {k: re.compile(v) for k, v in self.rules.items()}
        self.hit_log = []

    def _load_rules(self, path: str) -> dict:
        # Load from YAML; fallback to defaults
        return PII_PATTERNS

    def redact(self, text: str) -> tuple[str, dict]:
        hits = {}
        for ptype, pattern in self.compiled.items():
            matches = pattern.findall(text)
            if matches:
                hits[ptype] = len(matches)
                text = pattern.sub("[REDACTED]", text)
        self.hit_log.append({"timestamp": datetime.utcnow().isoformat(), "hits": hits})
        return text, hits

    def get_stats(self) -> dict:
        # Aggregate hits from log
        stats = {}
        for entry in self.hit_log:
            for ptype, count in entry["hits"].items():
                stats[ptype] = stats.get(ptype, 0) + count
        return stats
```

**Performance Optimization:**
- Run redaction in `asyncio.to_thread()` to avoid blocking event loop.
- Batch redaction for benchmark campaigns.
- Cache compiled patterns.
- Skip redaction for synthetic/test prompts (flagged in metadata).

**Dashboard Metrics:**
- `pii_redaction_hits_total` (counter)
- `pii_redaction_types` (histogram)
- `redaction_latency_ms` (histogram)

**Testing:**
- Verify synthetic emails/phones are redacted.
- Confirm non-PII text is preserved.
- Check audit log accuracy.
- Measure performance impact (<5ms per 1k chars).

---

# Deployment Plan

Phased rollout to minimize risk. All changes are backward-compatible and reversible.

**Pre-Deployment Checklist:**
- [ ] Backup `/models/flight-recorder/data/flight_recorder.db`
- [ ] Backup `/models/flight-recorder/data/artifacts/`
- [ ] Verify Python 3.11+ and Docker Compose v2
- [ ] Test migration on staging copy
- [ ] Validate artifact storage permissions (`chmod 755`, `chown appuser`)
- [ ] Configure `ARTIFACT_RETENTION_DAYS=30`
- [ ] Set `PRIVACY_FILTER_ENABLED=true`
- [ ] Configure `REASONING_TOKEN_TRACKING=true`

**Docker Compose Updates (`deploy/docker-compose.yml`):**
```yaml
version: "3.8"
services:
  ai-flight-recorder:
    build: ./app
    ports:
      - "8000:8000"
    volumes:
      - /models/flight-recorder/data:/data
    environment:
      - DATABASE_URL=sqlite:////data/flight_recorder.db
      - ARTIFACT_DIR=/data/artifacts
      - PREVIEW_DIR=/data/previews
      - ARTIFACT_RETENTION_DAYS=30
      - PRIVACY_FILTER_ENABLED=true
      - REASONING_TOKEN_TRACKING=true
      - LOG_LEVEL=info
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: "1.5"
    restart: unless-stopped

  llama-cpp-server:
    image: ghcr.io/ggerganov/llama.cpp:server
    ports:
      - "8080:8080"
    volumes:
      - /models/flight-recorder/data:/data
    command: >
      --model /data/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
      --host 0.0.0.0
      --port 8080
      --ctx-size 65536
      --max-tokens 8192
      --reasoning
      --log-disable
    deploy:
      resources:
        limits:
          memory: 8G
          cpus: "4.0"
    restart: unless-stopped
```

**Rollout Steps:**
1. **Phase 1 (Staging):** Deploy to isolated environment. Run migration, verify schema, test artifact storage, validate redaction, run synthetic benchmarks.
2. **Phase 2 (Canary):** Deploy to production with 10% traffic. Monitor error rates, latency, disk usage.
3. **Phase 3 (Full):** Route 100% traffic. Enable cleanup cron. Monitor dashboard metrics.
4. **Phase 4 (Optimization):** Tune `--ctx-size`, `--max-tokens`, retention policy based on observed usage.

**Monitoring & Alerting:**
- Prometheus metrics endpoint at `/metrics`
- Alerts: `sqlite_lock_wait > 5s`, `artifact_disk_usage > 80%`, `pii_redaction_hits > 100/min`, `benchmark_error_rate > 5%`
- Grafana dashboard for latency, throughput, token distribution, privacy hits.

**Backup Strategy:**
- Daily `pg_dump` equivalent for SQLite: `sqlite3 flight_recorder.db ".backup /backup/flight_recorder_$(date +%Y%m%d).db"`
- Artifact sync to external storage weekly (optional).
- Retain 7 daily backups, 4 weekly backups.

---

# Test Plan

24 acceptance tests covering all pain points and architectural changes.

| ID | Test Name | Steps | Expected Result | Validation |
|----|-----------|-------|-----------------|------------|
| AT-01 | Schema Migration Success | Run `app/migrate.py` on fresh DB | Tables created, indexes built, WAL mode enabled | `sqlite3 db ".tables"` shows new tables |
| AT-02 | Artifact Storage Creation | Send request with 50k token output | Chunks written to `/data/artifacts/`, hash matches | `ls /data/artifacts/` shows `.zst` files |
| AT-03 | Artifact Download Integrity | Download artifact via `/api/artifacts/{hash}/download` | Content matches original, hash verified | `sha256sum` matches registry |
| AT-04 | Preview Generation | Request with 10k token output | Preview file created, first 500 chars correct | `cat previews/{hash}_500.txt` |
| AT-05 | Reasoning Token Segmentation | Prompt with `<think>` block | `reasoning_token_count` > 0, `completion_token_count` > 0 | DB query shows split counts |
| AT-06 | Reasoning Budget Tracking | Campaign with 70% reasoning tokens | `reasoning_ratio` recorded, warning logged | Dashboard shows ratio > 0.7 |
| AT-07 | Privacy Redaction - Email | Prompt contains `test.user@corp.example.com` | Email replaced with `[REDACTED]` in DB | DB query shows `[REDACTED]` |
| AT-08 | Privacy Redaction - Phone | Prompt contains `555-123-4567` | Phone replaced with `[REDACTED]` | DB query shows `[REDACTED]` |
| AT-09 | Privacy Audit Log | Run redaction pipeline | Hit log updated with type and count | `privacy_filter.get_stats()` returns counts |
| AT-10 | Model Audit Log | Swap GGUF, restart proxy | New entry in `model_audit_log` with hash and flags | DB query shows new row |
| AT-11 | Benchmark Campaign Start | Run `bench run --config campaign.yaml` | Campaign ID created, workers spawn | Dashboard shows campaign in progress |
| AT-12 | Benchmark Retry Logic | Simulate network error | Request retried 3 times with backoff | Log shows retry attempts |
| AT-13 | Benchmark Checkpoint | Kill runner mid-campaign | State saved to `bench_state.json` | File exists with partial progress |
| AT-14 | Benchmark Resume | Run `bench resume --campaign-id ...` | Campaign continues from checkpoint | Dashboard shows resumed progress |
| AT-15 | Benchmark Export CSV | Run `bench export --format csv` | CSV file generated with correct columns | Open CSV, verify headers and data |
| AT-16 | Dashboard Latency Percentiles | Run 100 requests | p50, p90, p99 calculated correctly | Compare with manual calculation |
| AT-17 | Model Comparison Matrix | Run campaigns with 2 models | Matrix shows avg latency, tokens, reasoning ratio | Export CSV, verify aggregation |
| AT-18 | Artifact Cleanup Cron | Set retention to 1 day, wait 25h | Old artifacts deleted, DB cleaned | `ls artifacts/` shows only recent files |
| AT-19 | Docker Health Check | Stop proxy, wait 30s | Health check fails, container restarts | `docker inspect` shows restart count |
| AT-20 | SQLite WAL Mode | Run concurrent benchmarks | No lock contention, writes succeed | Monitor `sqlite3 db "PRAGMA journal_mode;"` |
| AT-21 | Long Output Timeout | Request with `max_tokens=32768` | Request completes or times out gracefully | No crash, proper error code |
| AT-22 | UI Virtualization | Load 1000 runs in dashboard | Page loads in <2s, scroll smooth | Browser dev tools shows <2s load |
| AT-23 | Rollback Schema | Run `app/migrate.py --rollback` | Tables dropped, DB returns to previous state | `sqlite3 db ".tables"` shows old schema |
| AT-24 | End-to-End Benchmark | Full campaign with redaction, reasoning, artifacts | All metrics recorded, artifacts stored, privacy filtered | Dashboard shows complete campaign data |

---

# Rollback Plan

In case of deployment failure, data corruption, or critical regression, the following rollback procedure ensures minimal data loss and service disruption.

**Trigger Conditions:**
- Error rate > 10% for 5 minutes
- SQLite corruption detected
- Privacy filter causes false positives on >5% of requests
- Benchmark runner enters infinite retry loop
- Disk usage > 95% due to artifact bloat

**Rollback Steps:**
1. **Stop Traffic:** Route 0% to new version via load balancer or DNS TTL.
2. **Preserve Data:** Copy `/data/flight_recorder.db` and `/data/artifacts/` to `/backup/rollback_$(date +%Y%m%d_%H%M)/`.
3. **Revert Code:** Checkout previous commit: `git checkout v1.2.0`
4. **Revert Config:** Restore `docker-compose.yml` and `.env` from backup.
5. **Revert Schema:** Run `app/migrate.py --rollback` to drop new tables and indexes.
6. **Restore DB:** If corruption, restore from last known good backup.
7. **Restart Services:** `docker compose down && docker compose up -d`
8. **Verify:** Run health check, synthetic request, dashboard load.
9. **Monitor:** Watch error rates, latency, disk usage for 1 hour.
10. **Communicate:** Notify stakeholders of rollback, incident timeline, and next steps.

**Data Preservation:**
- Artifacts are never deleted during rollback; they remain in `/backup/`.
- New tables are dropped, but existing data in old tables is preserved.
- Audit logs are retained for post-incident analysis.

**Post-Rollback Actions:**
- Root cause analysis (RCA) within 48 hours.
- Fix identified issue in staging.
- Re-attempt deployment with mitigations.
- Update runbook with lessons learned.

---

# Open Questions

Several technical and operational decisions require further validation or future roadmap consideration:

1. **Cloud Storage Migration:** Should artifacts eventually move to S3-compatible storage for offsite backup and multi-region access? If so, which SDK (`boto3`, `minio`, `rclone`) and lifecycle policies are optimal?
2. **LLM-Based Redaction Accuracy:** The regex layer covers known patterns, but ambiguous PII (e.g., internal project names, code snippets) may require LLM classification. What is the acceptable false positive rate, and how will we handle model drift?
3. **Reasoning Token Standardization:** Different models use different delimiters (`<think>`, `<reasoning>`, `