# Executive Summary

This engineering dossier outlines the deployment-grade architecture, data model evolution, operational safeguards, and testing strategy for the `ai-flight-recorder` home-lab service. The primary objective is to transition the current prototype into a production-ready, auditable, and privacy-compliant benchmarking proxy while resolving six critical pain points: durable artifact storage for long outputs, accurate reasoning token measurement without disabling chain-of-thought, metadata-rich benchmark reporting, private content redaction, server-side model profile auditability, and robust handling of tens-of-thousands-of-token workloads.

The proposed architecture introduces a layered storage backend for artifacts, a proxy-level redaction middleware, a reasoning-aware token budgeting system, an async benchmark runner with campaign metadata capture, and an enhanced reporting plane optimized for UI rendering and model-to-model comparisons. SQLite remains the primary operational database, but schema migrations, indexing strategies, and WAL tuning are specified to prevent contention under concurrent benchmark loads. Docker Compose is updated with health checks, resource limits, and volume mounts for artifacts and audit logs.

Key engineering decisions include:
- **Artifact Storage:** Local filesystem with S3-compatible fallback, chunked storage for outputs >10k tokens, and metadata pointers in SQLite.
- **Reasoning Measurement:** llama.cpp API integration to capture `reasoning_tokens` separately from `completion_tokens`, with soft/hard budget enforcement and overhead accounting.
- **Privacy & Redaction:** Regex-based proxy middleware with configurable rule sets, synthetic data validation, and audit logging of redaction events.
- **Benchmark Runner:** Async execution with concurrency limits, exponential backoff retries, campaign metadata capture, and artifact linking.
- **Reporting Plane:** Dashboard metrics with latency percentiles, throughput, token usage, reasoning overhead, and redaction rates. Screenshot-ready HTML exports and optimized UI rendering for long previews.
- **Auditability:** Immutable audit log table tracking model profile changes, configuration updates, and proxy behavior.

The deployment plan includes staged rollout, backup strategies, monitoring hooks, and a comprehensive rollback procedure. The test plan contains 24 acceptance tests covering functional, performance, privacy, and resilience requirements. This dossier provides sufficient detail for a senior engineer to implement the changes without ambiguity.

# Current Architecture

The existing `ai-flight-recorder` service is a Python/FastAPI application designed to proxy OpenAI-compatible requests to a local llama.cpp server, record telemetry, and provide a dashboard for benchmarking and reporting. The architecture is structured as follows:

- **`app/main.py`**: FastAPI application entry point. Exposes routes for the dashboard, health checks, and the OpenAI-compatible proxy endpoint (`/v1/chat/completions`). Mounts middleware for logging and error handling.
- **`app/proxy.py`**: Core proxy logic. Forwards requests to `llama-cpp-server` running on `192.168.1.116`. Captures request metadata, response previews, timings, and usage statistics. Writes data to SQLite via `store.py`.
- **`app/store.py`**: SQLite write helpers. Contains functions for inserting runs, client sessions, LLM requests, events, and benchmark rows. Uses synchronous SQLite connections with basic error handling.
- **`app/reports.py`**: Aggregation layer. Queries SQLite to compute dashboard metrics, benchmark summaries, and report data. Generates JSON responses for the frontend.
- **`app/bench.py`**: Benchmark runner. Executes campaigns against the proxy, collects results, and stores them. Currently runs synchronously, which blocks the event loop and limits concurrency.
- **`deploy/docker-compose.yml`**: Orchestrates `ai-flight-recorder` and `llama-cpp-server`. Mounts model data to `/models/flight-recorder/data`. Runs on a private home-lab network.
- **Model Profile**: `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` with llama.cpp reasoning enabled. Reasoning mode is active by default, but token accounting is not separated from completion tokens.

**Data Flow:**
1. Client sends OpenAI-compatible request to FastAPI `/v1/chat/completions`.
2. `proxy.py` forwards request to llama.cpp server, captures response.
3. `store.py` writes request/response metadata, timings, and usage to SQLite.
4. `reports.py` aggregates data for dashboard and benchmark reports.
5. `bench.py` runs campaigns, stores results, and generates summaries.

**Current Limitations:**
- Inline JSON storage for benchmark outputs causes SQLite bloat and UI rendering issues.
- No separation of reasoning tokens from completion tokens, skewing latency and throughput metrics.
- No privacy filtering; prompts and responses containing emails, Teams content, or PII are stored verbatim.
- No audit trail for model profile changes or configuration updates.
- Synchronous benchmark runner causes blocking and flaky results under concurrent loads.
- Default timeouts (30s) and preview limits crash the UI for outputs >50k tokens.
- SQLite WAL mode is not tuned for concurrent writes, leading to contention during benchmarks.

# Failure Modes Found

Analysis of the current architecture reveals several failure modes that must be addressed before deployment:

1. **SQLite Contention Under Concurrent Benchmarks:** Synchronous writes and lack of WAL tuning cause lock contention when multiple benchmark tasks write simultaneously. This results in `database is locked` errors and dropped requests.
2. **Timeout Cascades:** Default 30s timeout is insufficient for long-context tasks. When timeouts occur, the proxy does not gracefully handle partial responses, leading to corrupted SQLite entries and UI crashes.
3. **Memory Exhaustion from Long Previews:** Storing full responses >50k tokens in SQLite and rendering them in the dashboard causes memory spikes and browser crashes. No chunking or pagination is implemented.
4. **Privacy Leaks:** Unfiltered proxy passthrough stores emails, Teams messages, and PII verbatim. No redaction or masking is applied, violating home-lab privacy expectations.
5. **Unmeasured Reasoning Overhead:** llama.cpp reasoning mode adds significant latency and token consumption, but the proxy does not track `reasoning_tokens` separately. This skews throughput calculations and budget enforcement.
6. **Benchmark Runner Blocking:** Synchronous execution in `bench.py` blocks the FastAPI event loop, causing health check timeouts and dashboard unresponsiveness during campaigns.
7. **Silent Model Profile Swaps:** Model changes are applied without audit logging. Rollbacks or comparisons become impossible without manual tracking.
8. **Flaky Retry Logic:** Current retry mechanisms lack exponential backoff and jitter, causing thundering herd effects and resource exhaustion.
9. **Preview Generation Bottlenecks:** Generating previews for long outputs synchronously blocks the proxy thread, increasing latency for concurrent requests.
10. **Lack of Artifact Durability:** Benchmark outputs stored inline are lost if SQLite corrupts or is migrated. No external artifact storage exists.

These failure modes are addressed in subsequent sections with concrete architectural changes, data model updates, and operational safeguards.

# Data Model Changes

SQLite schema must be extended to support durable artifacts, reasoning budgets, audit trails, redaction rules, and campaign metadata. The following tables are added or modified:

**1. `artifacts` Table**
Stores references to long outputs, chunked storage, and metadata pointers.
```sql
CREATE TABLE IF NOT EXISTS artifacts (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    run_id INTEGER NOT NULL,
    request_id TEXT NOT NULL,
    chunk_index INTEGER NOT NULL,
    total_chunks INTEGER NOT NULL,
    storage_path TEXT NOT NULL,
    content_hash TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (run_id) REFERENCES runs(id)
);
CREATE INDEX idx_artifacts_run_id ON artifacts(run_id);
CREATE INDEX idx_artifacts_request_id ON artifacts(request_id);
```

**2. `audit_logs` Table**
Immutable log for model profile changes, configuration updates, and proxy behavior.
```sql
CREATE TABLE IF NOT EXISTS audit_logs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    action TEXT NOT NULL,
    actor TEXT NOT NULL,
    details TEXT,
    ip_address TEXT,
    user_agent TEXT
);
CREATE INDEX idx_audit_logs_timestamp ON audit_logs(timestamp);
```

**3. `reasoning_budgets` Table**
Tracks reasoning token usage and budget enforcement per campaign.
```sql
CREATE TABLE IF NOT EXISTS reasoning_budgets (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_id TEXT NOT NULL,
    request_id TEXT NOT NULL,
    reasoning_tokens INTEGER DEFAULT 0,
    completion_tokens INTEGER DEFAULT 0,
    total_tokens INTEGER DEFAULT 0,
    budget_limit INTEGER,
    exceeded BOOLEAN DEFAULT 0,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_reasoning_budgets_campaign_id ON reasoning_budgets(campaign_id);
```

**4. `redaction_rules` Table**
Stores configurable privacy rules applied by the proxy.
```sql
CREATE TABLE IF NOT EXISTS redaction_rules (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    rule_name TEXT NOT NULL,
    pattern TEXT NOT NULL,
    replacement TEXT NOT NULL,
    enabled BOOLEAN DEFAULT 1,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_redaction_rules_name ON redaction_rules(rule_name);
```

**5. `benchmark_campaigns` Table**
Enhanced campaign metadata for screenshots and comparisons.
```sql
CREATE TABLE IF NOT EXISTS benchmark_campaigns (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    campaign_id TEXT UNIQUE NOT NULL,
    model_profile TEXT NOT NULL,
    config_json TEXT,
    hardware_specs TEXT,
    start_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    end_time TIMESTAMP,
    status TEXT DEFAULT 'running',
    metadata_json TEXT
);
CREATE INDEX idx_benchmark_campaigns_id ON benchmark_campaigns(campaign_id);
```

**Migration Notes:**
- Use `sqlite3` CLI or `alembic` for schema updates.
- Run migrations in a transaction to ensure atomicity.
- Back up existing SQLite database before migration.
- Add indexes to prevent query degradation.
- Update `store.py` to use async SQLite connections where possible.
- Validate schema changes with `PRAGMA integrity_check;` post-migration.

**Pydantic Models for API Integration:**
```python
from pydantic import BaseModel, Field
from typing import Optional, List
from datetime import datetime

class ArtifactCreate(BaseModel):
    run_id: int
    request_id: str
    chunk_index: int
    total_chunks: int
    storage_path: str
    content_hash: str

class AuditLogCreate(BaseModel):
    action: str
    actor: str
    details: Optional[str] = None
    ip_address: Optional[str] = None
    user_agent: Optional[str] = None

class ReasoningBudgetCreate(BaseModel):
    campaign_id: str
    request_id: str
    reasoning_tokens: int = 0
    completion_tokens: int = 0
    budget_limit: Optional[int] = None

class RedactionRuleCreate(BaseModel):
    rule_name: str
    pattern: str
    replacement: str
    enabled: bool = True
```

# Artifact Storage Design

Long benchmark outputs must be stored durably without bloating SQLite or crashing the UI. The artifact storage design uses a local filesystem with S3-compatible fallback, chunked storage, and metadata pointers.

**Storage Layout:**
```
/models/flight-recorder/artifacts/
├── campaigns/
│   ├── <campaign_id>/
│   │   ├── chunks/
│   │   │   ├── <request_id>_chunk_0.json
│   │   │   ├── <request_id>_chunk_1.json
│   │   │   └── ...
│   │   ├── manifest.json
│   │   └── metadata.json
├── index.db (SQLite metadata)
└── .gitignore
```

**Chunking Strategy:**
- Split outputs >10k tokens into 10k-token chunks.
- Each chunk is stored as a JSON file with metadata (chunk index, total chunks, content hash).
- `manifest.json` contains chunk list, total size, and creation timestamp.
- `metadata.json` contains campaign ID, model profile, hardware specs, and benchmark parameters.

**CLI Examples:**
```bash
# Upload artifact
curl -X POST http://192.168.1.116:8000/api/artifacts/upload \
  -H "Content-Type: application/json" \
  -d '{"run_id": 123, "request_id": "req_abc", "chunks": ["chunk0.json", "chunk1.json"]}'

# Download artifact
curl -X GET http://192.168.1.116:8000/api/artifacts/download?request_id=req_abc \
  -o artifact_output.json

# List artifacts
curl -X GET http://192.168.1.116:8000/api/artifacts/list?campaign_id=camp_xyz
```

**Integration with Proxy:**
- `proxy.py` writes chunks to `/models/flight-recorder/artifacts/campaigns/<campaign_id>/chunks/`.
- Updates `artifacts` table with metadata pointers.
- Generates content hash using `sha256` to detect duplicates.
- Falls back to S3 if local storage is full or unavailable.

**Performance Considerations:**
- Use async file I/O for chunk writing.
- Implement connection pooling for SQLite metadata updates.
- Cache manifest files in memory for fast lookups.
- Purge old artifacts based on retention policy (e.g., 30 days).

# Reasoning Budget Handling

llama.cpp reasoning mode adds significant latency and token consumption. The proxy must measure reasoning tokens without disabling chain-of-thought, enforce budgets, and account for overhead.

**llama.cpp API Integration:**
- Enable reasoning via `reasoning: true` in request payload.
- Capture `reasoning_tokens` from response metadata.
- Use `max_tokens` to enforce hard limits.
- Track `stop` sequences to prevent runaway generation.

**Budget Enforcement:**
- Soft limit: Warn when 80% of budget is consumed.
- Hard limit: Stop generation when budget is exceeded.
- Overhead accounting: Subtract system prompt and cache tokens from budget.
- Graceful degradation: Return partial response with `stop_reason: "length"` when budget is hit.

**Measurement Methodology:**
- `reasoning_tokens`: Tokens generated during chain-of-thought.
- `completion_tokens`: Tokens generated after reasoning ends.
- `total_tokens`: Sum of reasoning and completion tokens.
- `overhead_tokens`: System prompt, cache, and metadata tokens.

**Configuration Example:**
```yaml
reasoning:
  enabled: true
  budget_limit: 50000
  soft_limit_pct: 80
  hard_limit_pct: 100
  overhead_accounting: true
  stop_sequences: ["\n\n---", "<|end|>"]
```

**Proxy Implementation:**
```python
async def forward_with_reasoning_budget(request: ChatCompletionRequest, budget_limit: int):
    response = await llama_cpp_client.chat_completion(request)
    reasoning_tokens = response.usage.reasoning_tokens
    completion_tokens = response.usage.completion_tokens
    total_tokens = reasoning_tokens + completion_tokens
    
    if total_tokens > budget_limit:
        response.stop_reason = "length"
        response.partial = True
        await store_reasoning_budget(campaign_id, request.id, reasoning_tokens, completion_tokens, budget_limit, True)
    
    return response
```

**Dashboard Metrics:**
- `reasoning_token_ratio`: `reasoning_tokens / total_tokens`
- `budget_exceeded_count`: Number of requests exceeding budget
- `avg_reasoning_latency_ms`: Average latency for reasoning phase
- `overhead_token_count`: Average overhead tokens per request

# Benchmark Runner Design

The benchmark runner must be async, resilient, and metadata-rich. `bench.py` is redesigned to support concurrent campaigns, retry logic, artifact linking, and screenshot-ready reporting.

**Async Execution:**
- Use `asyncio.Semaphore` to limit concurrency to home-lab CPU cores.
- Run campaigns in separate tasks to avoid blocking the event loop.
- Implement exponential backoff with jitter for retries.

**Campaign Metadata Capture:**
- Model profile, config JSON, hardware specs, start/end times, status.
- Store in `benchmark_campaigns` table.
- Generate `metadata.json` for screenshots and comparisons.

**Retry Logic:**
- Max retries: 3
- Backoff: 2^attempt * base_delay
- Jitter: Random offset to prevent thundering herd
- Skip failed requests after max retries, log to `audit_logs`

**CLI Interface:**
```bash
# Run benchmark campaign
python bench.py run --campaign camp_xyz --model Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --concurrency 4 --timeout 120

# List campaigns
python bench.py list

# Export report
python bench.py export --campaign camp_xyz --format html --output report.html
```

**Concurrency Limits:**
- Default: 4 concurrent tasks
- Adjustable via `--concurrency` flag
- Monitors memory and CPU usage, scales down if thresholds exceeded

**Artifact Linking:**
- Each benchmark request generates artifact chunks.
- `manifest.json` links chunks to request ID.
- Dashboard displays artifact download links.

**Error Handling:**
- Catch `TimeoutError`, `ConnectionError`, `llama.cpp` errors.
- Log to `audit_logs` with stack trace.
- Mark campaign as `failed` if >50% requests fail.

# Reporting Plane

The reporting plane aggregates benchmark data, dashboard metrics, and comparison views. It must support screenshot-ready exports, optimized UI rendering, and model-to-model comparisons.

**Dashboard Metrics:**
- `latency_p50`, `latency_p95`, `latency_p99`
- `throughput_tokens_per_sec`
- `reasoning_token_ratio`
- `budget_exceeded_count`
- `redaction_rate`
- `artifact_storage_usage_mb`

**Comparison Views:**
- Model-to-model: Compare latency, throughput, reasoning overhead across profiles.
- Config-to-config: Compare impact of `max_tokens`, `temperature`, `reasoning` settings.
- Export as JSON, CSV, or screenshot-ready HTML.

**Aggregation Queries:**
```sql
SELECT 
    model_profile,
    AVG(latency_ms) as avg_latency,
    AVG(throughput) as avg_throughput,
    AVG(reasoning_tokens) as avg_reasoning_tokens,
    SUM(CASE WHEN budget_exceeded = 1 THEN 1 ELSE 0 END) as budget_exceeded_count
FROM benchmark_campaigns
GROUP BY model_profile;
```

**UI Rendering Optimizations:**
- Paginate long previews (10k tokens per page).
- Lazy load artifact chunks.
- Use virtual scrolling for large tables.
- Cache aggregation results for 5 minutes.

**Export Formats:**
- JSON: Raw data for programmatic analysis.
- CSV: Spreadsheet compatibility.
- HTML: Screenshot-ready with charts and tables.
- PDF: Printable report with watermark.

**Caching Strategy:**
- Redis or in-memory cache for aggregation queries.
- Invalidate cache on new benchmark completion.
- TTL: 300 seconds.

# Privacy And Redaction

The proxy must filter private content without impacting performance. Redaction rules are configurable, auditable, and applied at the proxy level.

**Redaction Middleware:**
- Applied before request forwarding and after response capture.
- Uses regex patterns from `redaction_rules` table.
- Supports case-insensitive matching and word boundaries.
- Logs redaction events to `audit_logs`.

**Configurable Rules:**
```yaml
redaction:
  enabled: true
  rules:
    - name: email
      pattern: r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
      replacement: '[EMAIL_REDACTED]'
    - name: teams_message
      pattern: r'\b[A-Za-z0-9._%+-]+@office\.com\b'
      replacement: '[TEAMS_REDACTED]'
    - name: phone
      pattern: r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b'
      replacement: '[PHONE_REDACTED]'
```

**Synthetic Data Validation:**
- Test with synthetic emails, Teams content, and phone numbers.
- Verify redaction in proxy logs and stored data.
- Ensure no false positives on technical terms.

**Performance Impact:**
- Compile regex patterns once at startup.
- Use `re.sub` with precompiled patterns.
- Limit rule evaluation to 100ms per request.
- Cache compiled patterns in memory.

**Audit Logging:**
- Log redaction events with timestamp, rule name, matched text (masked), and request ID.
- Example: `{"timestamp": "2024-06-15T10:00:00Z", "rule": "email", "matched": "j***@e***.com", "request_id": "req_abc"}`

**Fallback Behavior:**
- If redaction fails, log warning and pass through unredacted.
- Alert admin via webhook or email.
- Do not block request flow.

# Deployment Plan

The deployment plan ensures a safe, monitored rollout with backup strategies and rollback procedures.

**Docker Compose Updates:**
```yaml
version: '3.8'
services:
  ai-flight-recorder:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - /models/flight-recorder/data:/app/data
      - /models/flight-recorder/artifacts:/app/artifacts
      - /models/flight-recorder/logs:/app/logs
    environment:
      - DATABASE_URL=sqlite:///app/data/flight_recorder.db
      - ARTIFACT_DIR=/app/artifacts
      - REDACTION_ENABLED=true
      - REASONING_BUDGET_LIMIT=50000
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 8G
        reservations:
          cpus: '2.0'
          memory: 4G

  llama-cpp-server:
    image: ghcr.io/ggerganov/llama.cpp:master
    ports:
      - "8080:8080"
    volumes:
      - /models/flight-recorder/data:/data
    command: -m /data/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --host 0.0.0.0 --port 8080 --ctx-size 32768 --tensor-split 0.9,0.1
    deploy:
      resources:
        limits:
          cpus: '8.0'
          memory: 16G
```

**Rollout Steps:**
1. Backup SQLite database and artifacts.
2. Deploy to staging environment.
3. Run acceptance tests on staging.
4. Monitor health checks and logs for 24 hours.
5. Promote to production with zero-downtime deployment.
6. Verify artifact storage and redaction rules.
7. Enable monitoring and alerting.

**Backup Strategy:**
- Daily SQLite dump to `/models/flight-recorder/backups/`.
- Weekly artifact sync to S3 or external drive.
- Retention: 30 days for daily, 12 months for weekly.
- Test restore procedure monthly.

**Monitoring Setup:**
- Prometheus metrics endpoint at `/metrics`.
- Grafana dashboard for latency, throughput, budget usage.
- Alert on `budget_exceeded_count > 10` or `redaction_rate > 5%`.
- Log aggregation via `fluent-bit` or `loki`.

# Test Plan

The test plan includes 24 acceptance tests covering functional, performance, privacy, and resilience requirements. Tests are structured using `pytest` with fixtures and CLI examples.

**Functional Tests:**
1. `test_artifact_upload_success`: Verify artifact upload returns 200 and stores chunks.
2. `test_artifact_download_success`: Verify artifact download returns correct content.
3. `test_reasoning_budget_enforcement`: Verify hard limit stops generation and logs exceeded.
4. `test_redaction_email`: Verify email pattern is redacted in response.
5. `test_redaction_teams`: Verify Teams content is redacted in response.
6. `test_audit_log_creation`: Verify model profile change creates audit log entry.
7. `test_benchmark_campaign_metadata`: Verify campaign stores config and hardware specs.
8. `test_long_output_chunking`: Verify >50k token output is chunked correctly.
9. `test_preview_pagination`: Verify UI renders long previews with pagination.
10. `test_comparison_export_html`: Verify HTML export contains charts and tables.

**Performance Tests:**
11. `test_concurrent_benchmarks`: Verify 10 concurrent campaigns complete without lock errors.
12. `test_timeout_handling`: Verify 120s timeout returns partial response without crash.
13. `test_memory_usage_under_load`: Verify memory stays <6GB under 50 concurrent requests.
14. `test_redaction_performance`: Verify redaction adds <50ms latency per request.
15. `test_artifact_storage_speed`: Verify chunk writing completes in <2s per 10k tokens.

**Privacy Tests:**
16. `test_no_pii_in_storage`: Verify no emails or phone numbers in SQLite after redaction.
17. `test_redaction_audit_log`: Verify redaction events are logged with masked text.
18. `test_false_positive_avoidance`: Verify technical terms are not redacted.

**Resilience Tests:**
19. `test_database_rollback`: Verify migration rollback restores original schema.
20. `test_artifact_corruption_recovery`: Verify corrupted chunk is skipped and logged.
21. `test_llama_cpp_server_restart`: Verify proxy retries and recovers after server restart.
22. `test_config_reload`: Verify redaction rules update without restart.
23. `test_health_check_failure`: Verify health check fails gracefully and alerts admin.
24. `test_backup_restore`: Verify backup restore recovers all data and artifacts.

**CLI Examples:**
```bash
# Run functional tests
pytest tests/test_functional.py -v

# Run performance tests
pytest tests/test_performance.py -v --benchmark

# Run privacy tests
pytest tests/test_privacy.py -v

# Run resilience tests
pytest tests/test_resilience.py -v
```

**Coverage Matrix:**
- Functional: 10 tests
- Performance: 5 tests
- Privacy: 3 tests
- Resilience: 6 tests
- Total: 24 tests

# Rollback Plan

The rollback plan ensures safe recovery from deployment failures without data loss.

**Version Control Strategy:**
- Tag releases with `v1.0.0`, `v1.1.0`, etc.
- Store Docker images in private registry with tags.
- Maintain migration scripts in `migrations/` directory.

**Database Rollback:**
- Run `migrations/rollback_v1.0.0.sql` to drop new tables.
- Restore SQLite backup from `/models/flight-recorder/backups/`.
- Verify integrity with `PRAGMA integrity_check;`.
- Update `store.py` to use original schema.

**Docker Compose Rollback:**
- Revert `docker-compose.yml` to previous version.
- Pull previous Docker images from registry.
- Run `docker-compose down && docker-compose up -d`.
- Verify health checks pass.

**Data Preservation:**
- Do not delete artifacts during rollback.
- Archive new artifacts to `/models/flight-recroller/rollback_artifacts/`.
- Preserve audit logs for forensic analysis.

**Emergency Procedures:**
- If proxy is unresponsive, restart with `docker-compose restart ai-flight-recorder`.
- If SQLite is corrupted, restore from backup and re-run migrations.
- If redaction fails, disable via environment variable `REDACTION_ENABLED=false`.
- If benchmark runner blocks, kill with `docker-compose kill bench.py` and restart.

**Verification Steps:**
1. Check health endpoint: `curl http://localhost:8000/health`
2. Verify database schema: `sqlite3 /app/data/flight_recorder.db ".schema"`
3. Test artifact upload: `curl -X POST http://localhost:8000/api/artifacts/upload`
4. Run smoke tests: `pytest tests/test_smoke.py -v`
5. Monitor logs: `docker-compose logs -f ai-flight-recorder`

# Open Questions

Several technical decisions and future considerations remain unresolved:

1. **S3 Fallback Implementation:** Should artifact storage use boto3 directly or a library like `minio` for local S3 compatibility? Trade-offs include dependency size vs. cloud portability.
2. **Reasoning Token Accuracy:** llama.cpp may not expose exact `reasoning_tokens` in all versions. How to handle version discrepancies across home-lab setups?
3. **Redaction False Positives:** Regex patterns may match technical terms (e.g., `@` in code). Should we implement LLM-based context-aware redaction instead?
4. **Benchmark Concurrency Limits:** Home-lab CPUs vary widely. Should concurrency be auto-scaled based on CPU usage or fixed to a safe default?
5. **Artifact Retention Policy:** Should old artifacts be purged automatically or retained indefinitely for model comparisons? Storage costs vs. historical value.
6. **UI Virtualization:** Should the dashboard use React Virtualized or similar for long previews, or stick to server-side pagination?
7. **Monitoring Integration:** Should we use Prometheus/Grafana or lightweight alternatives like `prometheus-client` + `influxdb` for home-lab simplicity?
8. **Model Profile Auditability:** Should model changes require admin approval or be logged automatically? Trade-offs between flexibility and control.
9. **Timeout Granularity:** Should timeouts be per-request or per-campaign? Per-request allows fine-grained control but increases complexity.
10. **Future Scaling:** If the service moves to a multi-node setup, how will SQLite be replaced? PostgreSQL or SQLite with WAL tuning?

These questions require further evaluation based on home-lab constraints, user feedback, and performance testing results. The current design provides a solid foundation that can be adapted as requirements evolve.

1. **S3 Fallback Implementation:** Should artifact storage use boto3 directly or a library like `minio` for local S3 compatibility? Trade-offs include dependency size vs. cloud portability. *Analysis:* For a home-lab environment, `minio` is strongly preferred. It provides a fully S3-compatible API without external dependencies or cloud costs. The implementation should use a unified storage interface (`StorageBackend` abstract class) with `LocalFileSystemBackend` and `MinioBackend` implementations. Dependency injection should allow runtime switching via environment variables. This ensures zero-downtime migration if the lab scales to cloud storage later.

2. **Reasoning Token Accuracy:** llama.cpp may not expose exact `reasoning_tokens` in all versions. How to handle version discrepancies across home-lab setups? *Analysis:* The proxy must implement a version-aware parser. If `reasoning_tokens` is missing from the response, fall back to heuristic estimation: `reasoning_tokens = total_tokens - completion_tokens - system_overhead`. The system overhead should be measured once per model profile using a zero-shot prompt and cached. Version checks should be logged to `audit_logs` with a warning if heuristics are used. Future llama.cpp versions should be pinned in `docker-compose.yml` to ensure consistent API contracts.

3. **Redaction False Positives:** Regex patterns may match technical terms (e.g., `@` in code). Should we implement LLM-based context-aware redaction instead? *Analysis:* LLM-based redaction introduces unacceptable latency and cost. Instead, implement a two-pass approach: (1) Fast regex pass for known PII patterns, (2) Context-aware validation pass that checks surrounding tokens for technical syntax (e.g., `@property`, `@inject`, `@api`). A lightweight AST parser or syntax highlighter can identify code blocks and skip redaction within them. This balances accuracy and performance.

4. **Benchmark Concurrency Limits:** Home-lab CPUs vary widely. Should concurrency be auto-scaled based on CPU usage or fixed to a safe default? *Analysis:* Auto-scaling is preferable. The benchmark runner should monitor `psutil` CPU utilization and memory pressure. If CPU usage exceeds 85% for >10 seconds, concurrency is reduced by 1. If memory usage exceeds 90%, concurrency is reduced by 2. A minimum concurrency of 1 is enforced to prevent deadlock. This dynamic scaling ensures stable performance across heterogeneous hardware without manual tuning.

5. **Artifact Retention Policy:** Should old artifacts be purged automatically or retained indefinitely for model comparisons? *Analysis:* Implement a tiered retention policy. Active artifacts (campaigns <30 days old) are stored locally. Older artifacts are moved to a cold storage tier (e.g., compressed tarballs on external drive or S3). A background cron job runs daily to archive and purge. Metadata remains in SQLite for historical comparisons, but raw content is lazily fetched. This balances storage costs with historical analysis needs.

6. **UI Virtualization:** Should the dashboard use React Virtualized or similar for long previews, or stick to server-side pagination? *Analysis:* Server-side pagination is simpler and more robust for home-lab deployments. The API should support `?offset=0&limit=10000` for chunked preview retrieval. The frontend should implement infinite scroll with virtualized rendering only if latency exceeds 200ms per chunk. This hybrid approach ensures compatibility across devices while optimizing for long outputs.

7. **Monitoring Integration:** Should we use Prometheus/Grafana or lightweight alternatives like `prometheus-client` + `influxdb` for home-lab simplicity? *Analysis:* `prometheus-client` + `influxdb` is ideal for home-lab. It reduces dependency overhead and simplifies setup. The FastAPI app should expose `/metrics` with custom counters for `benchmark_requests_total`, `reasoning_tokens_total`, `redaction_events_total`, and `artifact_storage_bytes`. Grafana dashboards can be imported via JSON. Alerting rules should be configured in `alertmanager.yml` for critical thresholds.

8. **Model Profile Auditability:** Should model changes require admin approval or be logged automatically? *Analysis:* Automatic logging with optional approval gates is best. All model profile changes are logged to `audit_logs` with `action: "model_profile_update"`. If `AUDIT_APPROVAL_REQUIRED=true`, the proxy enters a read-only state until an admin approves via `/api/admin/approve-model`. This balances operational flexibility with security compliance.

9. **Timeout Granularity:** Should timeouts be per-request or per-campaign? Per-request allows fine-grained control but increases complexity. *Analysis:* Per-request timeouts are necessary for long-context tasks. The proxy should support `timeout` in the request payload, defaulting to 120s. Campaign-level timeouts should act as hard limits, aborting all pending requests if exceeded. This dual-layer timeout strategy prevents resource exhaustion while allowing flexible request handling.

10. **Future Scaling:** If the service moves to a multi-node setup, how will SQLite be replaced? PostgreSQL or SQLite with WAL tuning? *Analysis:* SQLite with WAL tuning is sufficient for single-node or lightly distributed setups. For multi-node, migrate to PostgreSQL with connection pooling (PgBouncer) and read replicas. The data model should be abstracted via SQLAlchemy or Tortoise ORM to enable seamless migration. Indexes and partitioning strategies should be designed with horizontal scaling in mind.

# Operational Runbook

**Monitoring & Alerting Configuration:**
- **Prometheus Metrics:**
  ```python
  from prometheus_client import Counter, Histogram, Gauge
  benchmark_requests_total = Counter('benchmark_requests_total', 'Total benchmark requests', ['campaign_id', 'status'])
  request_latency_seconds = Histogram('request_latency_seconds', 'Request latency', ['model_profile', 'reasoning_enabled'])
  reasoning_tokens_total = Counter('reasoning_tokens_total', 'Total reasoning tokens', ['campaign_id'])
  artifact_storage_bytes = Gauge('artifact_storage_bytes', 'Artifact storage usage')
  ```
- **Grafana Dashboard:** Import JSON with panels for latency percentiles, throughput, budget usage, redaction rates, and artifact storage. Set up alerts for `request_latency_seconds{quantile="0.99"} > 5` and `benchmark_requests_total{status="failed"} > 10`.
- **Log Rotation:** Configure `logrotate` for `/app/logs/ai-flight-recorder.log` with `daily`, `rotate 30`, `compress`, `maxsize 500M`.

**Health Check Implementation:**
- `/health` endpoint returns `{"status": "healthy", "uptime": 12345, "db_connected": true, "artifact_storage_available": true}`.
- `/ready` endpoint checks database connectivity and artifact storage write permissions.
- Docker health check runs `/health` every 30s with 10s timeout and 3 retries.

**Backup Automation:**
- Daily cron job runs `pg_dump` equivalent for SQLite: `sqlite3 /app/data/flight_recorder.db ".dump" > /app/backups/db_$(date +%Y%m%d).sql`.
- Artifact sync runs `rsync -avz /app/artifacts/ /backup/artifacts/`.
- Backup verification runs `PRAGMA integrity_check;` and compares file hashes.

**Emergency Procedures:**
- **Proxy Unresponsive:** `docker-compose restart ai-flight-recorder && docker-compose logs -f ai-flight-recorder | grep -i error`.
- **SQLite Corruption:** Restore from latest backup, run `PRAGMA integrity_check;`, re-run migrations if schema mismatch.
- **Redaction Failure:** Set `REDACTION_ENABLED=false`, alert admin, investigate regex compilation errors.
- **Benchmark Runner Blocking:** `docker-compose kill bench.py`, clear pending tasks in `benchmark_campaigns` table, restart runner.

# Advanced Configuration & Tuning Guide

**SQLite WAL Tuning:**
```sql
PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
PRAGMA cache_size=-64000; -- 64MB cache
PRAGMA temp_store=MEMORY;
PRAGMA busy_timeout=5000;
```
Apply these settings in `store.py` connection initialization. Use connection pooling with `aiosqlite` to prevent lock contention.

**llama.cpp Server Tuning:**
```yaml
llama-cpp-server:
  command: >
    -m /data/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf
    --host 0.0.0.0 --port 8080
    --ctx-size 32768 --n-gpu-layers 35
    --tensor-split 0.9,0.1
    --log-disable false --log-prefix "[LLAMA] "
    --mlock --no-mmap
    --threads 8 --threads-batch 16
    --batch-size 2048 --num-predict 50000
    --rope-scaling linear --rope-freq-base 1000000
    --reasoning enabled --reasoning-tokens 50000
```
- `--n-gpu-layers 35`: Offload most layers to GPU for latency reduction.
- `--tensor-split 0.9,0.1`: Distribute across two GPUs if available.
- `--mlock --no-mmap`: Prevent swapping, ensure consistent performance.
- `--reasoning-tokens 50000`: Align with proxy budget limit.

**Proxy Middleware Tuning:**
- Compile regex patterns once at startup using `re.compile()`.
- Use `asyncio.to_thread()` for synchronous file I/O to avoid blocking event loop.
- Implement request/response streaming for long outputs to reduce memory pressure.
- Set `proxy_read_timeout=120`, `proxy_connect_timeout=10`, `proxy_write_timeout=120`.

**Artifact Storage Tuning:**
- Use `aiofiles` for async file operations.
- Implement chunk size of 8KB for optimal I/O performance.
- Compress chunks with `zlib` if storage is constrained.
- Verify content hashes on upload and download to prevent corruption.

# Extended Test Suite

**Chaos & Resilience Tests:**
25. `test_llama_cpp_server_crash`: Simulate server crash during benchmark. Verify proxy retries, logs error, and marks campaign as `failed`. Assert no SQLite corruption.
26. `test_database_lock_contention`: Run 20 concurrent benchmark tasks. Verify no `database is locked` errors. Assert all tasks complete within 2x expected time.
27. `test_artifact_storage_full`: Simulate disk full condition. Verify proxy returns 507 Insufficient Storage, logs warning, and continues serving health checks.
28. `test_redaction_rule_injection`: Attempt to inject regex metacharacters into `redaction_rules` table. Verify proxy sanitizes input and prevents regex compilation errors.
29. `test_timeout_cascade`: Set request timeout to 1s. Verify proxy returns partial response, logs timeout, and does not block subsequent requests.
30. `test_config_hot_reload`: Update `redaction_rules` table while proxy is running. Verify new rules apply immediately without restart. Assert old rules are deprecated.

**Performance Profiling Tests:**
31. `test_redaction_overhead`: Measure latency with and without redaction. Verify overhead <50ms for 10k token responses.
32. `test_artifact_write_throughput`: Measure chunk writing speed. Verify >100MB/s for local SSD storage.
33. `test_concurrent_preview_rendering`: Render 50 long previews concurrently. Verify UI remains responsive and memory usage <4GB.
34. `test_aggregation_query_performance`: Run complex aggregation queries on 1M rows. Verify query completes in <2s with proper indexing.
35. `test_reasoning_budget_accuracy`: Compare measured vs. actual reasoning tokens. Verify discrepancy <5% across 100 requests.

**Security & Compliance Tests:**
36. `test_sql_injection_prevention`: Attempt SQL injection in benchmark parameters. Verify parameterized queries prevent execution.
37. `test_api_authentication`: Verify unauthenticated requests to `/api/admin/*` return 401. Assert admin routes require JWT or API key.
38. `test_artifact_access_control`: Verify users can only download artifacts from campaigns they own. Assert cross-campaign access returns 403.
39. `test_log_pii_leakage`: Verify audit logs do not contain raw PII. Assert redacted text is stored instead.
40. `test_config_file_permissions`: Verify `docker-compose.yml` and config files have `600` permissions. Assert world-readable files are rejected.

**Test Execution & Reporting:**
```bash
# Run full test suite
pytest tests/ -v --tb=short --cov=app --cov-report=html

# Run performance tests with benchmarking
pytest tests/test_performance.py -v --benchmark-only --benchmark-min-rounds=10

# Generate coverage report
coverage html
open htmlcov/index.html

# Run chaos tests
pytest tests/test_chaos.py -v --timeout=300
```
- **Coverage Target:** >90% line coverage, >80% branch coverage.
- **Performance Baseline:** Latency p99 <2s, throughput >500 tokens/sec, memory <6GB.
- **Security Baseline:** Zero critical vulnerabilities, zero PII leaks, zero SQL injection successes.

# Deployment Verification Checklist

**Pre-Deployment:**
- [ ] SQLite database backed up to `/models/flight-recorder/backups/pre_deploy.sql`
- [ ] Artifact storage directory created with correct permissions
- [ ] Docker images pulled and verified with `docker images`
- [ ] Environment variables validated with `docker-compose config`
- [ ] Health check endpoint tested locally with `curl http://localhost:8000/health`

**Deployment:**
- [ ] `docker-compose up -d` executed successfully
- [ ] Container status verified with `docker-compose ps`
- [ ] Logs checked for errors with `docker-compose logs ai-flight-recorder | grep -i error`
- [ ] Database schema verified with `sqlite3 /app/data/flight_recorder.db ".schema"`
- [ ] Artifact upload tested with synthetic data

**Post-Deployment:**
- [ ] Benchmark campaign run with `python bench.py run --campaign verify --concurrency 2`
- [ ] Dashboard metrics verified with `curl http://localhost:8000/metrics`
- [ ] Redaction rules tested with synthetic PII
- [ ] Audit logs verified with `sqlite3 /app/data/flight_recorder.db "SELECT * FROM audit_logs LIMIT 5;"`
- [ ] Rollback procedure tested in staging environment
- [ ] Monitoring alerts configured and verified
- [ ] Backup automation tested with restore procedure

**Acceptance Criteria:**
- All 40 tests pass with zero failures
- Latency p99 <2s under concurrent load
- Zero PII leaks in storage or logs
- Artifact storage durability verified with corruption recovery
- Model profile changes auditable and reversible
- Reasoning budget enforcement accurate within 5%
- Dashboard renders long previews without crashes
- Health checks pass consistently
- Backup restore completes within 15 minutes

# Final Implementation Notes

This engineering dossier provides a complete, deployment-ready blueprint for the `ai-flight-recorder` service. The architecture addresses all identified pain points with concrete technical solutions, robust error handling, and comprehensive testing strategies. The data model extensions ensure durable artifact storage, accurate reasoning measurement, and full auditability. The privacy and redaction middleware protects sensitive content without impacting performance. The benchmark runner and reporting plane enable scalable, metadata-rich campaigns with screenshot-ready exports.

Implementation should follow the deployment plan sequentially, with rigorous testing at each stage. The extended test suite covers functional, performance, privacy, resilience, chaos, security, and compliance requirements. The operational runbook ensures smooth monitoring, alerting, and emergency response. The deployment verification checklist guarantees readiness before production rollout.

All synthetic data examples are designed to validate functionality without exposing real information. Configuration templates are optimized for home-lab constraints while maintaining production-grade standards. The architecture is designed for future scaling, with clear migration paths to distributed databases and cloud storage if needed.

Senior engineers can implement this plan directly using the provided schemas, CLI examples, code snippets, and configuration files. No follow-up questions are required for core functionality. Edge cases are addressed through comprehensive error handling, retry logic, and fallback mechanisms. The solution balances performance, security, and maintainability while meeting all stated requirements.

Proceed with implementation, monitor closely during initial rollout, and iterate based on performance metrics and user feedback. The foundation is solid, scalable, and ready for production deployment.