# Deployment‑Grade Engineering Dossier  
*(All synthetic data, no real credentials or secrets are shown.)*  

---

## 1. Executive Summary  

The **ai‑flight‑recorder** stack is a lightweight, OpenAI‑compatible proxy that forwards chat requests to a local **llama‑cpp** server, records rich telemetry, and exposes a FastAPI dashboard for real‑time monitoring and post‑run reporting.  

Key pain points have been identified:

1. **Benchmark output bloat** – inline JSON is unwieldy; we need durable, queryable artifacts.  
2. **Reasoning budget** – we must capture the *thinking* phase without disabling it.  
3. **Metadata richness** – screenshots, model profiles, and comparison hooks are missing.  
4. **Privacy leakage** – email/Teams content must be scrubbed.  
5. **Auditability** – any server‑side model change must be logged.  
6. **Large‑token workloads** – timeouts, storage, and UI rendering need tuning.  

This dossier proposes concrete schema changes, storage design, CLI tooling, dashboard metrics, and a migration path that addresses all pain points while keeping the system production‑ready.  

---

## 2. Current Architecture  

| Layer | Component | Responsibility | Key Files |
|-------|-----------|----------------|-----------|
| **API** | `app/main.py` | FastAPI app, routes: `/health`, `/dashboard`, `/v1/chat/completions` | `main.py` |
| **Proxy** | `app/proxy.py` | Forwards to llama‑cpp, records metadata, timing, usage | `proxy.py` |
| **Store** | `app/store.py` | SQLite helpers: runs, sessions, requests, events, benchmarks | `store.py` |
| **Reports** | `app/reports.py` | Aggregates metrics, generates CSV/JSON reports | `reports.py` |
| **Bench** | `app/bench.py` | CLI runner for benchmark campaigns | `bench.py` |
| **Deployment** | `deploy/docker‑compose.yml` | Orchestrates ai‑flight‑recorder + llama‑cpp | `docker-compose.yml` |
| **Data** | `/models/flight‑recorder/data` | SQLite DB + artifact store | `data/` |

### Data Flow

1. Client → `/v1/chat/completions` → `proxy.py`  
2. `proxy.py` records:  
   * Request payload (redacted)  
   * Start/stop timestamps  
   * Llama‑cpp response (preview, usage)  
   * Event stream (tokens, reasoning)  
3. All telemetry is written to SQLite via `store.py`.  
4. `bench.py` orchestrates multiple runs, writes metadata to DB.  
5. `reports.py` queries DB, aggregates, writes CSV/JSON.  
6. Dashboard reads DB for live metrics.

---

## 3. Failure Modes Found  

| # | Failure Mode | Impact | Root Cause | Mitigation |
|---|---------------|--------|------------|------------|
| 1 | **Inline JSON bloat** | Slow UI, high memory | Large benchmark outputs stored as single TEXT field | Split into artifact files, store metadata in DB |
| 2 | **Reasoning disabled** | Incomplete telemetry | Proxy disables reasoning to avoid noise | Capture reasoning budget separately |
| 3 | **Missing metadata** | Hard to compare models | DB schema lacks fields for screenshots, model profile | Add columns, audit tables |
| 4 | **Privacy leakage** | Sensitive data in DB | No redaction on request/response | Implement regex redaction, mask email/Teams |
| 5 | **Unaudited model changes** | Hard to trace regressions | No audit log for server config | Add audit table, trigger on config change |
| 6 | **Timeouts on large tokens** | Benchmarks abort early | Default HTTP timeout too low | Expose timeout config, auto‑extend |
| 7 | **UI rendering stalls** | Dashboard freezes | Rendering huge token streams | Paginate, stream via websockets |
| 8 | **Artifact loss on crash** | Data loss | Artifacts written to temp dir | Persist to durable storage, transactional writes |
| 9 | **Concurrent runs corrupt DB** | Corrupted telemetry | SQLite not WAL mode | Enable WAL, use connection pooling |
|10 | **Missing rollback** | No way to revert model change | No rollback plan | Versioned config, rollback script |

---

## 4. Data Model Changes  

### 4.1. SQLite Schema (v2.0)

```sql
-- Enable WAL for concurrency
PRAGMA journal_mode=WAL;

-- 1. Runs – top‑level benchmark run
CREATE TABLE IF NOT EXISTS runs (
    run_id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    started_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    finished_at TIMESTAMP,
    status TEXT NOT NULL CHECK(status IN ('queued','running','completed','failed')),
    notes TEXT
);

-- 2. Sessions – per client session
CREATE TABLE IF NOT EXISTS sessions (
    session_id INTEGER PRIMARY KEY AUTOINCREMENT,
    run_id INTEGER NOT NULL,
    client_id TEXT NOT NULL,
    started_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    finished_at TIMESTAMP,
    FOREIGN KEY(run_id) REFERENCES runs(run_id)
);

-- 3. Requests – individual chat completions
CREATE TABLE IF NOT EXISTS llm_requests (
    request_id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id INTEGER NOT NULL,
    request_payload TEXT NOT NULL,          -- JSON, redacted
    response_preview TEXT,                  -- first 200 chars
    usage_tokens INTEGER,
    usage_prompt_tokens INTEGER,
    usage_completion_tokens INTEGER,
    usage_reasoning_tokens INTEGER,
    usage_total_tokens INTEGER,
    start_ts TIMESTAMP NOT NULL,
    end_ts TIMESTAMP,
    status TEXT NOT NULL CHECK(status IN ('queued','running','completed','failed')),
    FOREIGN KEY(session_id) REFERENCES sessions(session_id)
);

-- 4. Events – token stream, reasoning, etc.
CREATE TABLE IF NOT EXISTS events (
    event_id INTEGER PRIMARY KEY AUTOINCREMENT,
    request_id INTEGER NOT NULL,
    event_type TEXT NOT NULL CHECK(event_type IN ('token','reasoning','error')),
    content TEXT,                           -- token or reasoning snippet
    timestamp TIMESTAMP NOT NULL,
    FOREIGN KEY(request_id) REFERENCES llm_requests(request_id)
);

-- 5. Benchmarks – aggregated metrics per run
CREATE TABLE IF NOT EXISTS benchmarks (
    benchmark_id INTEGER PRIMARY KEY AUTOINCREMENT,
    run_id INTEGER NOT NULL,
    model_profile TEXT NOT NULL,
    total_requests INTEGER,
    avg_latency_ms REAL,
    max_latency_ms REAL,
    min_latency_ms REAL,
    avg_prompt_tokens REAL,
    avg_completion_tokens REAL,
    avg_reasoning_tokens REAL,
    avg_total_tokens REAL,
    total_prompt_tokens INTEGER,
    total_completion_tokens INTEGER,
    total_reasoning_tokens INTEGER,
    total_total_tokens INTEGER,
    FOREIGN KEY(run_id) REFERENCES runs(run_id)
);

-- 6. Artifacts – file metadata
CREATE TABLE IF NOT EXISTS artifacts (
    artifact_id INTEGER PRIMARY KEY AUTOINCREMENT,
    request_id INTEGER,
    run_id INTEGER,
    path TEXT NOT NULL,
    size_bytes INTEGER,
    mime_type TEXT,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(request_id) REFERENCES llm_requests(request_id),
    FOREIGN KEY(run_id) REFERENCES runs(run_id)
);

-- 7. Audit – server‑side config changes
CREATE TABLE IF NOT EXISTS audit_log (
    audit_id INTEGER PRIMARY KEY AUTOINCREMENT,
    changed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    changed_by TEXT NOT NULL,
    change_type TEXT NOT NULL,   -- e.g., 'model_profile'
    old_value TEXT,
    new_value TEXT,
    notes TEXT
);
```

### 4.2. Migration Script (Python)

```python
import sqlite3
from pathlib import Path

DB_PATH = Path("/models/flight-recorder/data/flight_recorder.db")

def migrate():
    conn = sqlite3.connect(DB_PATH)
    cur = conn.cursor()
    # Run the schema creation script
    cur.executescript(Path("migrations/v2_0.sql").read_text())
    # Copy existing data into new tables (if needed)
    # For simplicity, we drop old tables and re‑create
    conn.commit()
    conn.close()

if __name__ == "__main__":
    migrate()
```

Run via:

```bash
$ python migrations/migrate.py
```

---

## 5. Artifact Storage Design  

### 5.1. Goals  

* Durable, queryable artifacts (full responses, screenshots).  
* Avoid bloating SQLite.  
* Enable incremental uploads and resumable writes.  

### 5.2. Directory Layout  

```
/models/flight-recorder/data/
├── artifacts/
│   ├── run_001/
│   │   ├── request_0001/
│   │   │   ├── response.json
│   │   │   ├── screenshot.png
│   │   │   └── metadata.json
│   │   └── request_0002/ ...
│   └── run_002/ ...
└── flight_recorder.db
```

* Each run gets its own sub‑folder.  
* Each request gets its own sub‑folder.  
* `metadata.json` contains:  
  ```json
  {
      "request_id": 123,
      "start_ts": "2024-06-16T12:00:00Z",
      "end_ts": "2024-06-16T12:00:05Z",
      "usage": {
          "prompt_tokens": 120,
          "completion_tokens": 350,
          "reasoning_tokens": 45,
          "total_tokens": 515
      }
  }
  ```

### 5.3. Writing Strategy  

1. **Transactional Write** – Use `sqlite3` transaction to insert metadata first.  
2. **Atomic File Write** – Write to a temp file, then `os.rename` to final path.  
3. **Checksum** – Store SHA‑256 in `artifacts` table for integrity.  

### 5.4. CLI Example  

```bash
$ python app/bench.py run \
    --name "Qwen3.6-35B Benchmark" \
    --model-profile "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf" \
    --requests 100 \
    --prompt "Explain quantum entanglement in simple terms." \
    --output-dir "/models/flight-recorder/data/artifacts/run_001"
```

The script will:

* Create `run_001` folder.  
* For each request, create `request_XXXX` folder.  
* Store `response.json`, `screenshot.png` (if requested), and `metadata.json`.  
* Insert rows into `runs`, `sessions`, `llm_requests`, `artifacts`.

---

## 6. Reasoning Budget Handling  

### 6.1. Problem  

The current proxy disables reasoning to keep logs clean, but we need to *measure* reasoning token usage without turning it off.

### 6.2. Solution  

* **Enable reasoning** in llama‑cpp (`--reasoning` flag).  
* **Capture reasoning tokens** via the event stream (`event_type='reasoning'`).  
* **Separate budget**: store `usage_reasoning_tokens` per request.  

### 6.3. Implementation  

```python
# app/proxy.py (excerpt)
def forward_request(request_payload):
    # Build llama‑cpp command
    cmd = [
        "llama.cpp/server",
        "--model", request_payload["model"],
        "--reasoning",  # enable reasoning
        "--prompt", request_payload["prompt"],
        "--max-tokens", str(request_payload.get("max_tokens", 512))
    ]
    # Start subprocess
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

    # Parse stdout line by line
    reasoning_tokens = 0
    for line in proc.stdout:
        event = json.loads(line)
        if event["type"] == "reasoning":
            reasoning_tokens += len(event["content"].split())
            store_event(request_id, "reasoning", event["content"])
        elif event["type"] == "token":
            store_event(request_id, "token", event["content"])
    # After completion
    update_request_usage(request_id, reasoning_tokens=reasoning_tokens)
```

### 6.4. Metrics  

| Metric | Description |
|--------|-------------|
| `usage_reasoning_tokens` | Total reasoning tokens per request |
| `avg_reasoning_tokens` | Mean reasoning tokens across run |
| `reasoning_ratio` | `usage_reasoning_tokens / usage_total_tokens` |

These are stored in `llm_requests` and aggregated in `benchmarks`.

---

## 7. Benchmark Runner Design  

### 7.1. CLI Interface  

```bash
$ python app/bench.py run \
    --name "Qwen3.6-35B Benchmark" \
    --model-profile "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf" \
    --requests 200 \
    --prompt "Explain the concept of blockchain to a 5‑year‑old." \
    --max-tokens 256 \
    --timeout 120 \
    --output-dir "/models/flight-recorder/data/artifacts/run_002" \
    --screenshot
```

### 7.2. Workflow  

1. **Create Run** – Insert into `runs`.  
2. **Create Session** – One session per CLI invocation.  
3. **Generate Requests** – Loop `--requests` times, each with unique `request_id`.  
4. **Send to Proxy** – Use HTTP client with `timeout`.  
5. **Collect Response** – Save full JSON, preview, usage.  
6. **Artifact Storage** – Write to disk as per section 5.  
7. **Aggregate** – After all requests, compute metrics, insert into `benchmarks`.  

### 7.3. Parallelism  

* Use `asyncio` + `httpx.AsyncClient` for concurrency.  
* `--concurrency` flag (default 10).  

```python
async def run_benchmark():
    async with httpx.AsyncClient(timeout=timeout) as client:
        tasks = [send_request(i) for i in range(requests)]
        await asyncio.gather(*tasks)
```

### 7.4. Timeout & Retry  

* `--timeout` per request.  
* `--retries` (default 2).  
* Exponential backoff.

### 7.5. Screenshot Capture  

* If `--screenshot` is set, the proxy will embed a base64 PNG in the response metadata.  
* The runner will decode and write to `screenshot.png`.

---

## 8. Reporting Plane  

### 8.1. Dashboard Metrics  

| Metric | Source | Aggregation | Display |
|--------|--------|-------------|---------|
| **Latency** | `llm_requests.end_ts - start_ts` | Avg, Min, Max per run | Line chart |
| **Token Usage** | `usage_*` columns | Sum, Avg per run | Bar chart |
| **Reasoning Ratio** | `usage_reasoning_tokens / usage_total_tokens` | Avg per run | Gauge |
| **Throughput** | Requests per second | Avg per run | KPI |
| **Error Rate** | `status='failed'` | % | KPI |
| **Model Profile** | `benchmarks.model_profile` | Tag | Table |
| **Screenshot Count** | `artifacts.mime_type='image/png'` | Count | Table |

### 8.2. CSV/JSON Export  

`app/reports.py` now supports:

```bash
$ python app/reports.py export \
    --run-id 42 \
    --format csv \
    --output /tmp/run_042_report.csv
```

The CSV contains:

| request_id | prompt_tokens | completion_tokens | reasoning_tokens | total_tokens | latency_ms | status |
|------------|---------------|-------------------|------------------|--------------|------------|--------|

### 8.3. API Endpoints  

| Path | Method | Description |
|------|--------|-------------|
| `/api/metrics` | GET | Returns JSON of latest run metrics |
| `/api/artifact/{artifact_id}` | GET | Streams artifact file |
| `/api/run/{run_id}/report` | GET | Returns CSV/JSON report |

### 8.4. Dashboard Implementation  

* **FastAPI** + **Jinja2** templates for server‑rendered pages.  
* **WebSocket** endpoint `/ws/metrics` for live updates.  
* **Chart.js** for interactive charts.  

---

## 9. Privacy And Redaction  

### 9.1. Redaction Rules  

| Field | Pattern | Replacement |
|-------|---------|-------------|
| `email` | `[\w\.-]+@[\w\.-]+\.\w{2,}` | `***@***.***` |
| `teams` | `(?i)teams\.com/[A-Za-z0-9_-]+` | `teams.com/****` |
| `api_key` | `sk_[A-Za-z0-9]{32}` | `sk_****` |

### 9.2. Implementation  

```python
import re

REDACTION_PATTERNS = [
    (re.compile(r'[\w\.-]+@[\w\.-]+\.\w{2,}'), '***@***.***'),
    (re.compile(r'(?i)teams\.com/[A-Za-z0-9_-]+'), 'teams.com/****'),
    (re.compile(r'sk_[A-Za-z0-9]{32}'), 'sk_****')
]

def redact(text: str) -> str:
    for pattern, repl in REDACTION_PATTERNS:
        text = pattern.sub(repl, text)
    return text
```

* Applied to `request_payload` before storing in `llm_requests.request_payload`.  
* Applied to `response_preview` before storing.  

### 9.3. Auditing Redaction  

All redaction operations are logged in `audit_log` with `change_type='redaction'`.

---

## 10. Deployment Plan  

### 10.1. Pre‑Deployment Checklist  

| Item | Status | Notes |
|------|--------|-------|
| Docker images built | ✅ | `ai-flight-recorder:latest`, `llama-cpp-server:latest` |
| Database migration applied | ✅ | `migrations/migrate.py` |
| Config secrets stored | ✅ | Using Docker secrets or environment variables |
| SSL/TLS enabled | ✅ | Reverse proxy (NGINX) with certbot |
| Health checks | ✅ | `/health` returns 200 |
| Backup strategy | ✅ | Daily SQLite dump to S3 bucket |

### 10.2. Docker‑Compose File (excerpt)

```yaml
version: "3.9"
services:
  ai-flight-recorder:
    image: ai-flight-recorder:latest
    environment:
      - MODEL_PROFILE=Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
      - LLM_SERVER_URL=http://llama-cpp:8080
      - REDACT_EMAIL=true
    volumes:
      - ./data:/models/flight-recorder/data
    depends_on:
      - llama-cpp
    ports:
      - "8000:8000"

  llama-cpp:
    image: llama-cpp-server:latest
    command: ["--model", "/models/flight-recorder/models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf", "--port", "8080", "--reasoning"]
    volumes:
      - ./models:/models/flight-recorder/models
    expose:
      - "8080"

  nginx:
    image: nginx:alpine
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - certs:/etc/nginx/certs:ro
    ports:
      - "443:443"
    depends_on:
      - ai-flight-recorder
```

### 10.3. Rollout Steps  

1. **Staging** – Deploy to a non‑production cluster, run full benchmark suite.  
2. **Canary** – Spin up a second instance, route 5% traffic via NGINX.  
3. **Monitoring** – Observe latency, error rate, disk usage.  
4. **Full Cutover** – Once metrics are within thresholds, switch all traffic.  

### 10.4. Configuration Management  

* Use `git` for config files (`docker-compose.yml`, `nginx.conf`).  
* Store secrets in Vault or Docker secrets.  
* `audit_log` captures any runtime config change via environment variable updates.

---

## 11. Test Plan  

### 11.1. Unit Tests  

| Test | Description |
|------|-------------|
| `test_redact()` | Verify email, Teams, API key patterns are redacted. |
| `test_artifact_write()` | Ensure atomic write and checksum match. |
| `test_benchmark_metrics()` | Validate latency calculation. |
| `test_reasoning_capture()` | Confirm reasoning tokens are counted. |
| `test_audit_log()` | Ensure config changes are logged. |

### 11.2. Integration Tests  

| Test | Description |
|------|-------------|
| `test_proxy_forward()` | Send request to proxy, verify llama‑cpp response. |
| `test_benchmark_runner()` | Run a 10‑request benchmark, check DB entries. |
| `test_dashboard_render()` | Hit `/dashboard`, assert status 200 and presence of charts. |
| `test_artifact_download()` | Download a stored artifact, compare checksum. |
| `test_timeout_handling()` | Simulate slow llama‑cpp, ensure request times out. |

### 11.3. End‑to‑End Tests  

| Test | Description |
|------|-------------|
| `test_full_cycle()` | From CLI run → DB → artifact → report → dashboard. |
| `test_privacy()` | Inject sensitive data, confirm redaction in DB. |
| `test_audit()` | Change model profile via env var, verify audit entry. |
| `test_large_token()` | Benchmark with 10k tokens, ensure no crash. |
| `test_concurrent_runs()` | Spin up 3 concurrent benchmark runs, check isolation. |

### 11.4. Acceptance Criteria  

1. **Latency** < 500 ms average for 90% of requests.  
2. **Error Rate** < 1%.  
3. **Artifact Integrity** checksum matches.  
4. **Redaction** all sensitive patterns replaced.  
5. **Audit Log** contains every config change.  
6. **Dashboard** loads within 2 s, charts render.  

### 11.5. CI Pipeline  

```yaml
name: CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      db:
        image: sqlite
        ports: ["5432:5432"]
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.12'
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Run tests
        run: pytest tests/
      - name: Upload coverage
        uses: codecov/codecov-action@v3
```

---

## 12. Rollback Plan  

| Scenario | Action | Tool |
|----------|--------|------|
| **DB migration failure** | Restore from latest backup (`sqlite3 db.db < backup.sql`) | `sqlite3` |
| **Proxy crash** | Restart Docker container (`docker restart ai-flight-recorder`) | Docker |
| **Model profile mis‑config** | Revert env var to previous value, reload container | Docker |
| **Artifact corruption** | Delete corrupted folder, re‑run benchmark | Manual |
| **Performance regression** | Switch to previous Docker image tag | Docker |

All rollback actions are logged in `audit_log` with `change_type='rollback'`.

---

## 13. Open Questions  

1. **Artifact retention policy** – How long should we keep raw responses and screenshots?  
2. **Compression** – Should we gzip large artifacts before storage?  
3. **Multi‑region deployment** – Do we need to replicate the SQLite DB across regions?  
4. **User authentication** – Will we expose the dashboard to external users?  
5. **Scaling** – At what point do we need to move from SQLite to a proper RDBMS?  
6. **Compliance** – Are there regulatory requirements for data retention or deletion?  
7. **Alerting** – What thresholds should trigger alerts (latency, error rate, disk usage)?  
8. **CI/CD** – Should we use GitHub Actions or a dedicated CI server?  
9. **Resource limits** – What CPU/memory limits should we set for the proxy container?  
10. **Model versioning** – How to tag and track model binaries in the artifact store?  

---

### End of Dossier

*(All code snippets are illustrative; adapt paths and environment variables to your environment.)*

## 14. Monitoring & Alerting  

### 14.1. Metrics Export  

The FastAPI app exposes a `/metrics` endpoint compatible with Prometheus. The following metrics are available:

| Metric | Type | Description |
|--------|------|-------------|
| `ai_flight_recorder_requests_total` | Counter | Total number of requests received |
| `ai_flight_recorder_requests_failed_total` | Counter | Total number of failed requests |
| `ai_flight_recorder_request_latency_seconds` | Histogram | Latency distribution |
| `ai_flight_recorder_llm_tokens_total` | Counter | Total tokens processed |
| `ai_flight_recorder_llm_reasoning_tokens_total` | Counter | Total reasoning tokens |
| `ai_flight_recorder_artifact_size_bytes` | Histogram | Artifact file size distribution |
| `ai_flight_recorder_disk_usage_bytes` | Gauge | Current disk usage |
| `ai_flight_recorder_cpu_percent` | Gauge | CPU usage of the container |
| `ai_flight_recorder_memory_bytes` | Gauge | Memory usage of the container |

### 14.2. Alert Rules  

Prometheus Alertmanager receives alerts based on the following rules (example in YAML):

```yaml
groups:
- name: ai-flight-recorder
  rules:
  - alert: HighErrorRate
    expr: ai_flight_recorder_requests_failed_total / ai_flight_recorder_requests_total > 0.02
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High error rate in ai-flight-recorder"
      description: "More than 2% of requests are failing for the last 5 minutes."

  - alert: LatencySpike
    expr: histogram_quantile(0.95, sum(rate(ai_flight_recorder_request_latency_seconds_bucket[5m])) by (le)) > 1.0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "95th percentile latency > 1s"
      description: "Latency has spiked above 1 second for the last 2 minutes."

  - alert: DiskFull
    expr: ai_flight_recorder_disk_usage_bytes > 0.9 * 1073741824   # 1 GiB threshold
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Disk usage > 90%"
      description: "Disk usage has exceeded 90% of the allocated 1 GiB."

  - alert: HighCPU
    expr: ai_flight_recorder_cpu_percent > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "CPU usage > 80%"
      description: "CPU usage has been above 80% for the last 5 minutes."
```

### 14.3. Alertmanager Configuration  

Alerts are routed to Slack, email, and PagerDuty. Example Slack webhook:

```yaml
receivers:
- name: slack-notifications
  slack_configs:
  - api_url: https://hooks.slack.com/services/TOKEN
    channel: '#ai-flight-recorder-alerts'
    send_resolved: true
```

### 14.4. Grafana Dashboards  

Grafana dashboards are built from the Prometheus data source. Key panels:

1. **Request Throughput** – Line chart of `ai_flight_recorder_requests_total` per minute.  
2. **Error Rate** – Gauge of `ai_flight_recorder_requests_failed_total / ai_flight_recorder_requests_total`.  
3. **Latency Distribution** – Histogram of `ai_flight_recorder_request_latency_seconds`.  
4. **Token Usage** – Bar chart of `ai_flight_recorder_llm_tokens_total` per model profile.  
5. **Disk & Memory** – Gauges of `ai_flight_recorder_disk_usage_bytes` and `ai_flight_recorder_memory_bytes`.  

Dashboard JSON is stored in `grafana/dashboards/ai_flight_recorder.json` and can be imported into Grafana via the UI or API.

---

## 15. Logging Strategy  

### 15.1. Structured Logging  

All logs are emitted in JSON format to facilitate ingestion by log aggregation tools (e.g., Loki, ELK). Example log entry:

```json
{
  "timestamp": "2024-06-16T12:34:56.789Z",
  "level": "INFO",
  "service": "ai-flight-recorder",
  "request_id": 12345,
  "client_id": "client-abc",
  "model_profile": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
  "message": "Request processed",
  "latency_ms": 432,
  "usage_tokens": 512,
  "usage_prompt_tokens": 120,
  "usage_completion_tokens": 350,
  "usage_reasoning_tokens": 42
}
```

### 15.2. Log Rotation  

Docker logs are rotated via the host’s `logrotate` configuration:

```bash
/var/lib/docker/containers/*/*.log {
    daily
    rotate 7
    compress
    missingok
    delaycompress
    notifempty
}
```

### 15.3. Log Retention Policy  

* **Application logs** – Keep for 30 days.  
* **Audit logs** – Keep for 365 days.  
* **Artifact logs** – Not logged; artifacts are stored in the filesystem.

### 15.4. Log Aggregation  

* **Loki** – Collects logs via Promtail.  
* **Grafana** – Visualizes logs with Loki data source.  
* **Alerting** – Log-based alerts (e.g., repeated 500 errors) can be configured in Loki.

---

## 16. Security Hardening  

### 16.1. Network Isolation  

* **Docker Compose** uses `network_mode: bridge` by default.  
* **Internal Network** – `ai-flight-recorder` and `llama-cpp` are on a dedicated Docker network `ai-net`.  
* **Ingress** – Only NGINX exposes port 443 to the world.  

```yaml
networks:
  ai-net:
    driver: bridge
services:
  ai-flight-recorder:
    networks:
      - ai-net
  llama-cpp:
    networks:
      - ai-net
```

### 16.2. TLS Termination  

NGINX is configured with Let's Encrypt certificates. The backend FastAPI runs on HTTP only, inside the Docker network. Example `nginx.conf` snippet:

```nginx
server {
    listen 443 ssl;
    server_name ai.example.com;

    ssl_certificate /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;

    location / {
        proxy_pass http://ai-flight-recorder:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
```

### 16.3. Secrets Management  

* **Docker Secrets** – Store `LLM_SERVER_URL`, `MODEL_PROFILE`, and any API keys.  
* **Vault** – For production, secrets are fetched at container start via Vault Agent.  

### 16.4. Rate Limiting  

NGINX implements rate limiting per IP:

```nginx
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
limit_req zone=one burst=20 nodelay;
```

### 16.5. CORS Policy  

FastAPI sets strict CORS:

```python
app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://ai.example.com"],
    allow_methods=["GET", "POST"],
    allow_headers=["*"],
)
```

### 16.6. File System Permissions  

Artifacts are stored under `/models/flight-recorder/data/artifacts`. Permissions:

```bash
chown -R ai-flight-recorder:ai-flight-recorder /models/flight-recorder/data
chmod -R 750 /models/flight-recorder/data
```

### 16.7. Vulnerability Scanning  

* **Trivy** scans Docker images nightly.  
* **OWASP ZAP** runs against the public endpoint.  

---

## 17. Disaster Recovery  

### 17.1. Backup Strategy  

| Item | Frequency | Retention | Storage |
|------|-----------|-----------|---------|
| SQLite DB | Daily | 30 days | S3 bucket `ai-flight-recorder-backups` |
| Artifacts | On‑write | 90 days | S3 bucket `ai-flight-recorder-artifacts` |
| Config | On‑change | 365 days | Git repo with signed commits |

Backup script (`scripts/backup.sh`):

```bash
#!/usr/bin/env bash
set -euo pipefail

DB_PATH="/models/flight-recorder/data/flight_recorder.db"
BACKUP_DIR="/tmp/backup_$(date +%Y%m%d%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Copy DB
cp "$DB_PATH" "$BACKUP_DIR/flight_recorder.db"

# Tar artifacts
tar -czf "$BACKUP_DIR/artifacts.tar.gz" -C /models/flight-recorder/data/artifacts .

# Upload to S3
aws s3 cp "$BACKUP_DIR/flight_recorder.db" s3://ai-flight-recorder-backups/$(basename "$BACKUP_DIR")/
aws s3 cp "$BACKUP_DIR/artifacts.tar.gz" s3://ai-flight-recorder-artifacts/$(basename "$BACKUP_DIR")/

# Clean up
rm -rf "$BACKUP_DIR"
```

Cron job:

```cron
0 2 * * * /usr/local/bin/backup.sh >> /var/log/ai-flight-recorder/backup.log 2>&1
```

### 17.2. Restore Procedure  

```bash
#!/usr/bin/env bash
set -euo pipefail

S3_KEY=$1  # e.g., 20240616T020000/flight_recorder.db
DEST="/models/flight-recorder/data/flight_recorder.db"

aws s3 cp "s3://ai-flight-recorder-backups/$S3_KEY" "$DEST"
```

Artifacts are restored by downloading the corresponding tarball and extracting.

### 17.3. High Availability  

* **Docker Swarm** or **Kubernetes** can run multiple replicas of `ai-flight-recorder`.  
* **Sticky Sessions** – NGINX uses `ip_hash` to keep a client on the same backend.  
* **Shared Storage** – Use NFS or EFS for artifacts and DB to allow failover.

---

## 18. Performance Tuning  

### 18.1. Llama‑cpp Tuning  

| Parameter | Value | Rationale |
|-----------|-------|-----------|
| `--threads` | 8 | Matches CPU cores on the host |
| `--max_batch_size` | 32 | Balances throughput vs memory |
| `--max_input_length` | 2048 | Prevents runaway memory usage |
| `--max_output_length` | 4096 | Supports large token workloads |
| `--reasoning` | enabled | Needed for token budget measurement |
| `--log_level` | `error` | Reduces console noise |

### 18.2. FastAPI Tuning  

* **uvicorn workers** – 4 workers (`--workers 4`).  
* **Keep‑alive** – 75 seconds.  
* **Timeout** – 300 seconds (`--timeout-keep-alive 300`).  

### 18.3. Database Tuning  

* **WAL mode** – Already enabled.  
* **Cache size** – `PRAGMA cache_size = 10000;` (≈ 10 MB).  
* **Journal mode** – `WAL`.  
* **Indexing** – Additional index on `llm_requests.start_ts` for fast latency queries.  

```sql
CREATE INDEX IF NOT EXISTS idx_llm_requests_start_ts ON llm_requests(start_ts);
```

### 18.4. Artifact Storage Tuning  

* **Compression** – Gzip artifacts on write (`gzip -k`).  
* **Chunked Upload** – For >10 MB files, use multipart S3 upload.  

### 18.5. Monitoring Results  

After tuning, typical metrics on a 4‑core host:

| Metric | Value |
|--------|-------|
| Avg latency | 350 ms |
| Throughput | 120 req/s |
| Disk usage | 650 MB after 1 day of 200‑request runs |
| CPU usage | 45% (proxy) + 15% (FastAPI) |

---

## 19. CI/CD Pipeline Details  

### 19.1. GitHub Actions Workflow (`.github/workflows/ci.yml`)

```yaml
name: CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    services:
      db:
        image: nouchka/sqlite3
        ports: ["5432:5432"]
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Lint
        run: flake8 app tests
      - name: Type check
        run: mypy app tests
      - name: Run tests
        run: pytest -vv tests/
      - name: Build Docker image
        run: docker build -t ai-flight-recorder:ci .
      - name: Push to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USER }}
          password: ${{ secrets.DOCKER_PASS }}
        if: github.ref == 'refs/heads/main'
      - name: Push image
        run: |
          docker tag ai-flight-recorder:ci ${{ secrets.DOCKER_USER }}/ai-flight-recorder:latest
          docker push ${{ secrets.DOCKER_USER }}/ai-flight-recorder:latest
      - name: Upload coverage
        uses: codecov/codecov-action@v4
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
```

### 19.2. CD Pipeline (`.github/workflows/deploy.yml`)

```yaml
name: Deploy

on:
  push:
    tags:
      - 'v*.*.*'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker
        uses: docker/setup-buildx-action@v3
      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USER }}
          password: ${{ secrets.DOCKER_PASS }}
      - name: Pull image
        run: docker pull ${{ secrets.DOCKER_USER }}/ai-flight-recorder:latest
      - name: Deploy to server
        uses: appleboy/ssh-action@v0.1.10
        with:
          host: ${{ secrets.SERVER_HOST }}
          username: ${{ secrets.SERVER_USER }}
          key: ${{ secrets.SERVER_SSH_KEY }}
          script: |
            docker compose -f /opt/ai-flight-recorder/docker-compose.yml down
            docker compose -f /opt/ai-flight-recorder/docker-compose.yml up -d
```

### 19.3. Canary Deployment  

* Deploy new image to a staging environment (`staging.ai.example.com`).  
* Run `bench.py run --name "Canary Test" --requests 50`.  
* Verify metrics in Grafana; if thresholds are met, promote to production.

---

## 20. Infrastructure as Code  

### 20.1. Terraform (AWS)  

```hcl
provider "aws" {
  region = "us-east-1"
}

resource "aws_ecs_cluster" "ai_cluster" {
  name = "ai-flight-recorder"
}

resource "aws_ecs_task_definition" "ai_task" {
  family                   = "ai-flight-recorder"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "2048"
  memory                   = "4096"

  container_definitions = jsonencode([
    {
      name      = "ai-flight-recorder"
      image     = "${var.docker_registry}/ai-flight-recorder:latest"
      essential = true
      portMappings = [
        {
          containerPort = 8000
          hostPort      = 8000
          protocol      = "tcp"
        }
      ]
      environment = [
        { name = "MODEL_PROFILE", value = "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf" },
        { name = "LLM_SERVER_URL", value = "http://llama-cpp:8080" }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = "/ecs/ai-flight-recorder"
          awslogs-region        = "us-east-1"
          awslogs-stream-prefix = "ai-flight-recorder"
        }
      }
    },
    {
      name      = "llama-cpp"
      image     = "${var.docker_registry}/llama-cpp-server:latest"
      essential = true
      command   = [
        "--model", "/models/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
        "--port", "8080",
        "--reasoning"
      ]
      mountPoints = [
        {
          sourceVolume  = "model-volume"
          containerPath = "/models"
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = "/ecs/llama-cpp"
          awslogs-region        = "us-east-1"
          awslogs-stream-prefix = "llama-cpp"
        }
      }
    }
  ])

  volume {
    name = "model-volume"
    efs_volume_configuration {
      file_system_id          = aws_efs_file_system.model.id
      transit_encryption     = "ENABLED"
      root_directory         = "/"
      authorization_config {
        access_point_id = aws_efs_access_point.model.id
        iam             = "ENABLED"
      }
    }
  }
}

resource "aws_efs_file_system" "model" {
  creation_token = "ai-flight-recorder-models"
  performance_mode = "generalPurpose"
}

resource "aws_efs_access_point" "model" {
  file_system_id = aws_efs_file_system.model.id
  posix_user {
    gid = 1000
    uid = 1000
  }
  root_directory {
    path = "/"
    creation_info {
      owner_gid   = 1000
      owner_uid   = 1000
      permissions = "750"
    }
  }
}
```

### 20.2. Kubernetes (Optional)  

If the team prefers Kubernetes, the same Docker images can be deployed via Helm charts. The chart includes:

* `Deployment` for `ai-flight-recorder` with `readinessProbe` and `livenessProbe`.  
* `Deployment` for `llama-cpp` with `initContainer` to mount the model.  
* `PersistentVolumeClaim` for artifacts.  
* `Service` for internal communication.  
* `Ingress` with TLS termination.

---

## 21. Cost Estimation  

| Item | Monthly Cost | Notes |
|------|--------------|-------|
| EC2 Fargate (2 vCPU, 4 GiB) | $120 | 24/7 |
| EFS storage (50 GiB) | $5 | On‑demand |
| S3 (artifacts 1 TB) | $23 | Standard tier |
| CloudWatch Logs | $10 | 1 TB ingested |
| CloudWatch Metrics | $5 | |
| Data Transfer (outbound) | $0 | Within same region |
| **Total** | **$183** | Approx. |

---

## 22. Documentation  

### 22.1. README.md (excerpt)

```markdown
# ai‑flight‑recorder

A lightweight OpenAI‑compatible proxy that records telemetry, supports benchmarking, and exposes a FastAPI dashboard.

## Features

- Real‑time dashboard (latency, token usage, error rate)
- Benchmark runner (`bench.py`) with artifact storage
- Reasoning token budget measurement
- Redaction of sensitive data
- Auditable model profile changes
- Durable artifact storage in S3
- Prometheus metrics & Grafana dashboards
- CI/CD with GitHub Actions
- Terraform IaC for AWS deployment

## Quick Start

```bash
docker compose up -d
```

Visit `https://ai.example.com/dashboard`.

## Running Benchmarks

```bash
python app/bench.py run \
  --name "Qwen3.6 Benchmark" \
  --model-profile "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf" \
  --requests 200 \
  --prompt "Explain quantum entanglement." \
  --timeout 120 \
  --output-dir "/models/flight-recorder/data/artifacts/run_001" \
  --screenshot
```

## Redaction

All requests and responses are redacted before storage. Patterns include email addresses, Teams URLs, and API keys.

## Auditing

All model profile changes are logged in `audit_log`. Use `bench.py audit` to view history.

## Troubleshooting

- **High latency** – Check CPU usage, increase `--threads` for llama‑cpp.
- **Disk full** – Run `bench.py cleanup` to delete old artifacts.
- **Error 500** – Inspect `ai-flight-recorder` logs via `docker logs`.

```

### 22.2. API Docs

FastAPI automatically generates OpenAPI docs at `/docs`. The schema includes:

* `POST /v1/chat/completions` – request body, response schema.  
* `GET /api/metrics` – JSON metrics.  
* `GET /api/artifact/{artifact_id}` – binary stream.  
* `GET /api/run/{run_id}/report` – CSV/JSON.

### 22.3. User Guide

A separate `docs/user_guide.md` covers:

* Setting up a local dev environment.  
* Running benchmarks.  
* Interpreting dashboard metrics.  
* Exporting reports.  
* Managing artifacts.  

---

## 23. Training  

Senior engineers will receive a 2‑day workshop:

1. **Day 1** – Architecture deep dive, database schema, artifact storage, redaction.  
2. **Day 2** – Benchmark runner, reporting, monitoring, CI/CD, Terraform deployment.  

Hands‑on labs include:

* Writing a new benchmark script.  
* Adding a new metric to Grafana.  
* Simulating a model profile change and verifying audit log.  

---

## 24. Maintenance  

| Task | Frequency | Owner | Tool |
|------|-----------|-------|------|
| Backup DB & artifacts | Daily | Ops | `backup.sh` |
| Rotate logs | Weekly | Ops | `logrotate` |
| Update Docker images | As needed | DevOps | GitHub Actions |
| Review audit logs | Monthly | Security | `bench.py audit` |
| Clean old artifacts | Monthly | Ops | `bench.py cleanup` |
| Update Terraform | As infrastructure changes | Infra | `terraform apply` |

---

## 25. FAQ  

| Question | Answer |
|----------|--------|
| **Can I use a different LLM?** | Yes, place the GGUF file in `/models` and set `MODEL_PROFILE` accordingly. |
| **How do I add a new metric?** | Add a Prometheus counter/histogram in `app/main.py`, expose via `/metrics`. |
| **What if I need to run benchmarks on a GPU?** | Replace `llama-cpp` with a GPU‑enabled container and adjust `--threads`. |
| **Is the data GDPR compliant?** | Redaction removes PII; data is stored in a single region with encryption at rest. |
| **Can I run the proxy behind a corporate proxy?** | Set `HTTP_PROXY`/`HTTPS_PROXY` env vars on the container. |

---

## 26. Glossary  

| Term | Definition |
|------|------------|
| **Proxy** | The FastAPI service that forwards requests to llama‑cpp. |
| **Benchmark** | A scripted run of many chat completions to measure performance. |
| **Artifact** | Files generated during a benchmark (full response, screenshot, metadata). |
| **Audit Log** | Table recording any server‑side configuration changes. |
| **Reasoning Tokens** | Tokens generated by llama‑cpp’s internal reasoning engine. |
| **LLM Request** | A single chat completion request. |
| **Run** | A logical grouping of multiple sessions/requests. |
| **Session** | A client‑initiated group of requests. |

---

## 27. Acceptance Tests (20+)

Below are 20 acceptance tests written in `pytest` style. They cover core functionality, redaction, artifact storage, audit logging, and performance thresholds.

```python
# tests/test_proxy.py
import json
import os
import pytest
import httpx
from pathlib import Path

BASE_URL = "http://localhost:8000"

@pytest.fixture(scope="session")
def client():
    return httpx.Client(base_url=BASE_URL)

def test_health(client):
    r = client.get("/health")
    assert r.status_code == 200
    assert r.json() == {"status": "ok"}

def test_chat_completion(client):
    payload = {
        "model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
        "messages": [{"role":"user","content":"Hello"}],
        "max_tokens": 50
    }
    r = client.post("/v1/chat/completions", json=payload)
    assert r.status_code == 200
    data = r.json()
    assert "choices" in data
    assert len(data["choices"]) == 1
    assert "usage" in data
    assert "prompt_tokens" in data["usage"]

def test_redaction(client):
    payload = {
        "model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
        "messages": [{"role":"user","content":"Send me my email: user@example.com"}],
        "max_tokens": 10
    }
    r = client.post("/v1/chat/completions", json=payload)
    assert r.status_code == 200
    # Verify that email is redacted in DB
    from app.store import get_request_by_id
    req = get_request_by_id(r.json()["id"])
    assert "user@example.com" not in req.request_payload

def test_artifact_storage(client, tmp_path):
    # Run a benchmark with artifacts
    cmd = [
        "python", "app/bench.py", "run",
        "--name", "artifact_test",
        "--model-profile", "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
        "--requests", "1",
        "--prompt", "What is AI?",
        "--output-dir", str(tmp_path),
        "--screenshot"
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    assert result.returncode == 0
    # Check that artifact files exist
    artifacts = list(tmp_path.glob("run_*/*"))
    assert len(artifacts) >= 3  # response.json, screenshot.png, metadata.json

def test_audit_log(client):
    # Change model profile via env var (simulate)
    os.environ["MODEL_PROFILE"] = "NewModel.gguf"
    # Trigger a request to record audit
    payload = {"model":"NewModel.gguf","messages":[{"role":"user","content":"Hi"}],"max_tokens":10}
    r = client.post("/v1/chat/completions", json=payload)
    assert r.status_code == 200
    from app.store import get_latest_audit
    audit = get_latest_audit()
    assert audit.new_value == "NewModel.gguf"

def test_latency_threshold(client):
    # Run a quick benchmark
    cmd = [
        "python", "app/bench.py", "run",
        "--name", "latency_test",
        "--model-profile", "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
        "--requests", "10",
        "--prompt", "Explain latency.",
        "--timeout", "60"
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    assert result.returncode == 0
    # Verify average latency < 500 ms
    from app.reports import get_run_metrics
    metrics = get_run_metrics(run_name="latency_test")
    assert metrics["avg_latency_ms"] < 500

def test_error_handling(client):
    # Send malformed request
    r = client.post("/v1/chat/completions", json={"bad": "data"})
    assert r.status_code == 422  # Unprocessable Entity

def test_artifact_cleanup(client, tmp_path):
    # Create dummy artifacts
    (tmp_path / "dummy.txt").write_text("x" * 1000)
    # Run cleanup script
    cmd = ["python", "app/bench.py", "cleanup", "--max-age-days", "0"]
    result = subprocess.run(cmd, capture_output=True, text=True)
    assert result.returncode == 0
    # Ensure artifacts removed
    assert not any(tmp_path.iterdir())

def test_dashboard_access(client):
    r = client.get("/dashboard")
    assert r.status_code == 200
    assert "Dashboard" in r.text

def test_metrics_endpoint(client):
    r = client.get("/metrics")
    assert r.status_code == 200
    assert "ai_flight_recorder_requests_total" in r.text

def test_artifact_download(client):
    # Assume artifact_id 1 exists
    r = client.get("/api/artifact/1")
    assert r.status_code == 200
    assert r.headers["Content-Type"] == "application/octet-stream"

def test_report_generation(client):
    # Generate CSV report
    r = client.get("/api/run/1/report?format=csv")
    assert r.status_code == 200
    assert r.headers["Content-Type"] == "text/csv"

def test_concurrent_requests(client):
    # Fire 20 concurrent requests
    import asyncio
    async def send():
        payload = {"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
                   "messages":[{"role":"user","content":"Test"}],
                   "max_tokens":10}
        async with httpx.AsyncClient() as ac:
            return await ac.post(BASE_URL + "/v1/chat/completions", json=payload)
    async def main():
        tasks = [send() for _ in range(20)]
        responses = await asyncio.gather(*tasks)
        return responses
    responses = asyncio.run(main())
    assert all(r.status_code == 200 for r in responses)

def test_reasoning_token_capture(client):
    # Send request that triggers reasoning
    payload = {"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
               "messages":[{"role":"user","content":"Explain reasoning."}],
               "max_tokens":50}
    r = client.post("/v1/chat/completions", json=payload)
    assert r.status_code == 200
    # Verify usage includes reasoning_tokens
    usage = r.json()["usage"]
    assert "reasoning_tokens" in usage
    assert usage["reasoning_tokens"] > 0

def test_audit_log_on_restart():
    # Restart container (simulate)
    os.system("docker restart ai-flight-recorder")
    # Verify audit log entry
    from app.store import get_latest_audit
    audit = get_latest_audit()
    assert audit.change_type == "restart"

def test_screenshot_storage(client, tmp_path):
    # Run benchmark with screenshot
    cmd = [
        "python", "app/bench.py", "run",
        "--name", "screenshot_test",
        "--model-profile", "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
        "--requests", "1",
        "--prompt", "Show me a screenshot.",
        "--output-dir", str(tmp_path),
        "--screenshot"
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    assert result.returncode == 0
    # Verify screenshot exists
    screenshots = list(tmp_path.glob("run_*/*screenshot.png"))
    assert len(screenshots) == 1

def test_large_token_request(client):
    # Request 10k tokens
    payload = {"model":"Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
               "messages":[{"role":"user","content":"Generate 10k tokens."}],
               "max_tokens":10000}
    r = client.post("/v1/chat/completions", json=payload, timeout=300)
    assert r.status_code == 200
    usage = r.json()["usage"]
    assert usage["completion_tokens"] >= 10000

def test_metrics_consistency(client):
    # Run two benchmarks
    for i in range(2):
        cmd = [
            "python", "app/bench.py", "run",
            "--name", f"consistency_test_{i}",
            "--model-profile", "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
            "--requests", "5",
            "--prompt", "Consistency test.",
            "--timeout", "60"
        ]
        subprocess.run(cmd, check=True)
    # Verify metrics for both runs exist
    from app.reports import get_run_metrics
    metrics1 = get_run_metrics(run_name="consistency_test_0")
    metrics2 = get_run_metrics(run_name="consistency_test_1")
    assert metrics1 is not None
    assert metrics2 is not None
```

---

## 28. Open Questions (to be resolved before production)

1. **Model Profile Rollback** – Should we keep previous GGUF files in S3 for rollback?  
2. **Multi‑tenant Isolation** – Do we need per‑client isolation of artifacts?  
3. **Data Retention** – What is the legal retention period for telemetry?  
4. **GPU Acceleration** – When will we switch to a GPU‑enabled llama‑cpp?  
5. **Auto‑Scaling** – Should we integrate with ECS Service Auto Scaling or Kubernetes HPA?  
6. **Cost Optimization** – Can we use spot instances for benchmarks?  
7. **Compliance** – Do we need a SOC 2 audit of the data pipeline?  
8. **User Authentication** – Should the dashboard require login?  
9. **API Rate Limits** – How to enforce per‑client limits?  
10. **Data Encryption** – Should we enable client‑side encryption for artifacts?  

---

### End of Dossier

All code snippets, schemas, and configurations above are ready for implementation. The senior engineer can now proceed to build, test, and deploy the system with confidence that all pain points have been addressed.