# Executive Summary This engineering dossier outlines the architectural evolution and deployment strategy for the **ai-flight-recorder** service, a private home-lab Python/FastAPI application designed to proxy, record, and benchmark LLM interactions. The current deployment, running alongside a llama.cpp server on `192.168.1.116`, is facing critical scaling and operational pain points as usage shifts toward high-complexity, long-context workloads. The primary objectives of this deployment are to resolve six specific pain points: 1. **Durable Artifacts:** Replace inline JSON payloads for long benchmark outputs with a disk-backed artifact storage system to prevent SQLite bloat and UI crashes. 2. **Reasoning Measurement:** Implement precise tokenization and duration tracking for the `reasoning` block in the Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf model without disabling the reasoning capability. 3. **Benchmark Metadata:** Enrich benchmark reports with granular metadata (model profile hashes, hardware telemetry, prompt fingerprints) to enable accurate model-to-model comparisons. 4. **Privacy Redaction:** Introduce a server-side redaction layer to strip PII (emails, Teams IDs, credit cards) from prompts and responses before they are persisted to SQLite. 5. **Auditability:** Track server-side model profile changes (e.g., switching from Qwen3.6 to a different quantization) via immutable event logs and model profile hashes. 6. **Stress Resilience:** Address timeouts, storage limits, and UI rendering issues caused by tens of thousands of output tokens by implementing streaming-aware timeouts, artifact offloading, and pagination. This dossier provides the concrete data model changes, artifact storage design, privacy mechanisms, and deployment plan required to transition the service from a prototype to a production-grade, deployment-ready system. --- # Current Architecture The ai-flight-recorder service is a monolithic Python application built on FastAPI, designed to act as an OpenAI-compatible proxy and telemetry recorder. **Core Components:** * **`app/main.py`**: The FastAPI application entry point. Exposes the dashboard UI, health check routes (`/health`), and the OpenAI-compatible proxy endpoint (`/v1/chat/completions`). * **`app/proxy.py`**: The core proxy logic. Forwards requests to the llama.cpp server at `192.168.1.116`. Captures request metadata, response previews, timings, and usage statistics. Currently writes these directly into SQLite. * **`app/store.py`**: SQLite write helpers. Contains functions for inserting runs, client sessions, LLM requests, events, and benchmark rows. Currently, large JSON payloads (prompts, responses) are serialized and stored directly in the `llm_requests` table. * **`app/reports.py`**: Aggregation logic. Queries SQLite to generate dashboard metrics and report summaries. * **`app/bench.py`**: Benchmark campaign runner. Iterates over a list of prompts, sends them to the proxy, and records the results. * **`deploy/docker-compose.yml`**: Orchestrates the ai-flight-recorder container and the llama-cpp-server container. Mounts the data volume at `/models/flight-recorder/data`. **Data Flow:** 1. Client sends a POST request to `http://ai-flight-recorder:8000/v1/chat/completions`. 2. `proxy.py` intercepts the request, extracts metadata (user, model, prompt), and forwards the request to `192.168.1.116:8080/v1/chat/completions`. 3. llama.cpp processes the request and returns a response. 4. `proxy.py` captures the response, calculates timings, and calls `store.py` to write the data to SQLite. 5. The response is streamed back to the client. **Current Infrastructure:** * **LLM Server:** llama-cpp-server running on `192.168.1.116`. * **Model:** `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` with reasoning enabled. * **Storage:** SQLite database located at `/models/flight-recorder/data/flight_recorder.db`. * **Network:** Internal home-lab network. --- # Failure Modes Found During the review of the current architecture, the following failure modes were identified, particularly under high-load or long-context scenarios: 1. **SQLite Bloat and Corruption:** Storing full JSON responses (especially those with tens of thousands of tokens) directly in the `llm_requests` table causes the SQLite database to grow rapidly. This leads to slow queries, UI rendering lag, and potential database corruption if the database file exceeds 10GB. 2. **HTTP Timeouts:** The default FastAPI timeout is 30 seconds. For long-context tasks requiring 32k output tokens, the proxy times out before the response is fully received, resulting in incomplete data and broken client connections. 3. **Privacy Leaks:** The proxy currently stores raw prompts and responses. If a user accidentally pastes an email address, Teams ID, or credit card number, it is persisted in plaintext in the SQLite database, violating privacy policies. 4. **Reasoning Blindness:** The current proxy does not distinguish between the `reasoning` block and the final answer in the llama.cpp response. This makes it impossible to measure reasoning token usage or duration, which is critical for evaluating the Qwen3.6 model's reasoning capabilities. 5. **Audit Gaps:** If the model profile is changed server-side (e.g., switching from `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` to `Qwen3.6-35B-A3B-APEX-MTP-I-Heavy.gguf`), the proxy does not record this change. Benchmark results become incomparable across different model profiles. 6. **Benchmark Fragility:** The `bench.py` runner lacks resilience. If a single request fails due to a network blip or timeout, the entire benchmark campaign may halt or produce incomplete results. There is no mechanism to retry failed requests or to track partial progress. --- # Data Model Changes To address the failure modes, the SQLite data model must be refactored. The primary changes involve decoupling large payloads from the main tables, adding reasoning-specific fields, and introducing auditability. **New Tables:** 1. **`artifacts`**: Stores large payloads (prompts, responses, reasoning blocks) as file paths or S3 URIs. ```sql CREATE TABLE artifacts ( id INTEGER PRIMARY KEY AUTOINCREMENT, run_id INTEGER NOT NULL, artifact_type TEXT NOT NULL CHECK(artifact_type IN ('prompt', 'response', 'reasoning', 'benchmark_report')), content_hash TEXT NOT NULL, file_path TEXT NOT NULL, size_bytes INTEGER NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, FOREIGN KEY (run_id) REFERENCES runs(id) ); ``` 2. **`model_profiles`**: Tracks model profile changes for auditability. ```sql CREATE TABLE model_profiles ( id INTEGER PRIMARY KEY AUTOINCREMENT, profile_name TEXT NOT NULL, model_hash TEXT NOT NULL, quantization TEXT NOT NULL, reasoning_enabled INTEGER NOT NULL DEFAULT 1, activated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, deactivated_at TIMESTAMP ); ``` 3. **`benchmark_campaigns`**: Tracks benchmark campaign status and metadata. ```sql CREATE TABLE benchmark_campaigns ( id INTEGER PRIMARY KEY AUTOINCREMENT, campaign_name TEXT NOT NULL, status TEXT NOT NULL CHECK(status IN ('pending', 'running', 'completed', 'failed')), total_requests INTEGER NOT NULL DEFAULT 0, completed_requests INTEGER NOT NULL DEFAULT 0, failed_requests INTEGER NOT NULL DEFAULT 0, started_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, completed_at TIMESTAMP ); ``` **Modified Tables:** 1. **`runs`**: Add model profile hash and artifact references. ```sql ALTER TABLE runs ADD COLUMN model_profile_hash TEXT; ALTER TABLE runs ADD COLUMN reasoning_tokens INTEGER DEFAULT 0; ALTER TABLE runs ADD COLUMN reasoning_duration_ms INTEGER DEFAULT 0; ALTER TABLE runs ADD COLUMN artifact_prompt_id INTEGER; ALTER TABLE runs ADD COLUMN artifact_response_id INTEGER; ALTER TABLE runs ADD COLUMN artifact_reasoning_id INTEGER; ``` 2. **`llm_requests`**: Remove large text columns, add artifact references. ```sql ALTER TABLE llm_requests DROP COLUMN prompt_text; ALTER TABLE llm_requests DROP COLUMN response_text; ALTER TABLE llm_requests DROP COLUMN reasoning_text; ALTER TABLE llm_requests ADD COLUMN artifact_prompt_id INTEGER; ALTER TABLE llm_requests ADD COLUMN artifact_response_id INTEGER; ALTER TABLE llm_requests ADD COLUMN artifact_reasoning_id INTEGER; ALTER TABLE llm_requests ADD COLUMN prompt_redacted TEXT; ALTER TABLE llm_requests ADD COLUMN response_redacted TEXT; ``` 3. **`events`**: Add event types for auditability. ```sql -- Add new event types: 'model_profile_changed', 'artifact_uploaded', 'benchmark_campaign_started' ``` **Migration Notes:** * Run the migration script to create the new tables and modify existing ones. * For existing data, migrate large payloads from `llm_requests` to the `artifacts` table. Compress the payloads using `zstd` before writing to disk. * Populate the `model_profiles` table with the current model profile hash. --- # Artifact Storage Design To solve the SQLite bloat and UI rendering issues, large payloads must be offloaded to durable artifact storage. **Storage Backend:** * **Primary:** Local filesystem (`/models/flight-recorder/artifacts/`). * **Secondary (Future):** S3-compatible object storage (e.g., MinIO) for distributed deployments. **Directory Structure:** ``` /models/flight-recorder/artifacts/ ├── runs/ │ ├── {run_id}/ │ │ ├── prompt.zst │ │ ├── response.zst │ │ └── reasoning.zst ├── benchmarks/ │ ├── {campaign_id}/ │ │ ├── report.json │ │ └── artifacts/ │ │ ├── {request_id}_prompt.zst │ │ └── {request_id}_response.zst ``` **Compression:** * Use `zstd` (Zstandard) for compression. It offers a good balance of speed and compression ratio, which is critical for high-throughput environments. * Compression level: 3 (fast compression, good ratio). **Artifact Lifecycle:** * **Creation:** When a request is completed, the proxy writes the payload to the artifact storage and records the file path and hash in the `artifacts` table. * **Retrieval:** The dashboard and reports query the `artifacts` table to get the file path, then stream the file from the filesystem. * **Retention:** Artifacts older than 30 days are automatically deleted by a background cron job. **CLI Example:** ```bash # Upload an artifact python -m app.artifacts upload --run-id 123 --type response --file /tmp/response.json # Download an artifact python -m app.artifacts download --run-id 123 --type response --output /tmp/response.json # List artifacts for a run python -m app.artifacts list --run-id 123 ``` --- # Reasoning Budget Handling To measure reasoning output without turning thinking off, we must parse the `reasoning` field in the llama.cpp response and tokenize it separately. **Tokenization Strategy:** * Use the `tiktoken` library with the `cl100k_base` encoding (compatible with Qwen models). * Tokenize the `reasoning` block and the final answer block separately. * Calculate the token count and duration for each block. **Implementation:** 1. **Response Parsing:** When the proxy receives the response from llama.cpp, it checks for the `reasoning` field. 2. **Token Counting:** ```python import tiktoken enc = tiktoken.get_encoding("cl100k_base") reasoning_tokens = len(enc.encode(response.get("reasoning", ""))) answer_tokens = len(enc.encode(response.get("content", ""))) ``` 3. **Duration Tracking:** Measure the time between the start of the `reasoning` block and the start of the final answer block. 4. **Budget Enforcement:** If the reasoning tokens exceed a threshold (e.g., 80% of the total output tokens), log a warning. This helps identify models that are "over-thinking" and wasting compute. **Schema Update:** * Add `reasoning_tokens` (INTEGER) and `reasoning_duration_ms` (INTEGER) to the `runs` table. **Dashboard Metrics:** * `llm_reasoning_tokens_total`: Counter of reasoning tokens generated. * `llm_reasoning_duration_seconds`: Histogram of reasoning duration. * `llm_reasoning_ratio`: Gauge of reasoning tokens / total output tokens. --- # Benchmark Runner Design The `bench.py` runner must be refactored to handle long outputs, provide resilience, and capture rich metadata. **Architecture:** * **Async Concurrency:** Use `asyncio.Semaphore` to limit the number of concurrent requests to llama.cpp. * **Campaign Definition:** Define campaigns in YAML files. * **Resilience:** Implement retry logic for failed requests. * **Artifact Integration:** Write responses to artifact storage instead of inline JSON. **Campaign YAML Example:** ```yaml campaign_name: "qwen36_reasoning_benchmark" model_profile: "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced" timeout_seconds: 600 max_concurrent_requests: 4 prompts: - id: "prompt_1" text: "Solve this complex math problem..." expected_tokens: 32000 - id: "prompt_2" text: "Write a detailed technical report..." expected_tokens: 16000 ``` **CLI Example:** ```bash # Run a benchmark campaign python -m app.bench run --campaign /models/flight-recorder/campaigns/qwen36_reasoning_benchmark.yaml --timeout 600 # Resume a failed campaign python -m app.bench resume --campaign-id 456 ``` **Metadata Capture:** * Record the model profile hash, hardware telemetry (CPU/GPU usage), and prompt fingerprint (SHA-256 of the prompt) for each request. * This enables accurate model-to-model comparisons by ensuring that the same prompts are used across different model profiles. --- # Reporting Plane The reporting plane must be updated to generate rich, metadata-enriched reports that are suitable for screenshots and later comparisons. **Dashboard Metrics:** * `llm_request_duration_seconds`: Histogram of request duration. * `llm_output_tokens_total`: Counter of output tokens generated. * `llm_artifact_size_bytes`: Histogram of artifact sizes. * `llm_redaction_events_total`: Counter of PII redaction events. **Report Generation:** * **HTML Reports:** Generate interactive HTML reports using `Jinja2` templates. Include charts (using `Plotly` or `Chart.js`) for token usage, duration, and reasoning ratios. * **PDF Reports:** Generate PDF reports using `WeasyPrint` for executive summaries. * **Comparison Views:** Provide side-by-side views of Model A vs Model B on the same prompt, highlighting differences in reasoning duration, token count, and output quality. **CLI Example:** ```bash # Generate a report for a benchmark campaign python -m app.reports generate --campaign-id 456 --format html --output /models/flight-recorder/reports/qwen36_reasoning_benchmark.html # Generate a comparison report python -m app.reports compare --campaign-id 456 --campaign-id 457 --output /models/flight-recorder/reports/comparison.html ``` --- # Privacy And Redaction To prevent privacy leaks, a server-side redaction layer must be implemented to strip PII from prompts and responses before they are persisted. **Redaction Strategy:** * **Regex-Based Redaction:** Use regex patterns to detect and redact common PII (emails, credit cards, phone numbers). * **LLM-Based Redaction:** Use a small, local LLM (e.g., `Phi-3-mini`) to detect and redact context-aware PII (e.g., Teams IDs, project names). **Regex Patterns:** ```python PII_PATTERNS = { "email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "credit_card": r"\b(?:\d[ -]*?){13,16}\b", "phone": r"\b(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b", "teams_id": r"\b[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}\b", } ``` **Implementation:** 1. **Pre-Write Redaction:** Before writing to SQLite, the proxy calls the redaction layer to strip PII from the prompt and response. 2. **Redacted Storage:** Store the redacted text in the `prompt_redacted` and `response_redacted` columns of the `llm_requests` table. 3. **Raw Storage:** Store the raw text in the artifact storage (if required for debugging, but access is restricted). **CLI Example:** ```bash # Scan a directory for PII python -m app.redactor scan --path /models/flight-recorder/data/raw --output /models/flight-recorder/reports/pii_scan.html # Redact a file python -m app.redactor redact --input /models/flight-recorder/data/raw/response.json --output /models/flight-recorder/data/raw/response_redacted.json ``` --- # Deployment Plan The deployment will be executed in phases to minimize risk. **Phase 1: Database Migration** * Run the migration script to create the new tables and modify existing ones. * Migrate large payloads from `llm_requests` to the `artifacts` table. * Verify the integrity of the database. **Phase 2: Artifact Storage Setup** * Create the artifact storage directory structure. * Update the Docker Compose configuration to mount the artifact storage volume. * Test artifact upload and retrieval. **Phase 3: Redaction Layer Integration** * Deploy the redaction layer. * Test PII detection and redaction. * Verify that redacted data is stored correctly. **Phase 4: Benchmark Runner Update** * Deploy the updated benchmark runner. * Run a small benchmark campaign to verify resilience and metadata capture. * Test artifact integration. **Phase 5: Dashboard/Reporting Update** * Deploy the updated dashboard and reporting plane. * Test report generation and comparison views. * Verify that all metrics are being captured correctly. **Docker Compose Changes:** ```yaml services: ai-flight-recorder: image: ai-flight-recorder:latest volumes: - /models/flight-recorder/data:/models/flight-recorder/data - /models/flight-recorder/artifacts:/models/flight-recorder/artifacts environment: - REDACTION_ENABLED=true - ARTIFACT_STORAGE_PATH=/models/flight-recorder/artifacts - TIMEOUT_SECONDS=600 ``` --- # Test Plan The test plan includes unit tests, integration tests, and end-to-end tests to ensure the new features work correctly. **Unit Tests:** 1. **UT-01:** Verify regex-based PII redaction for emails. 2. **UT-02:** Verify regex-based PII redaction for credit cards. 3. **UT-03:** Verify tokenization of the `reasoning` block. 4. **UT-04:** Verify artifact compression using `zstd`. 5. **UT-05:** Verify artifact decompression using `zstd`. **Integration Tests:** 6. **IT-01:** Verify that the proxy writes artifacts to disk and records the file path in SQLite. 7. **IT-02:** Verify that the proxy reads artifacts from disk and serves them via the API. 8. **IT-03:** Verify that the proxy redacts PII before writing to SQLite. 9. **IT-04:** Verify that the benchmark runner retries failed requests. 10. **IT-05:** Verify that the benchmark runner captures model profile hashes. **End-to-End Tests:** 11. **E2E-01:** Run a full benchmark campaign and verify that the report is generated correctly. 12. **E2E-02:** Verify that the dashboard displays the correct metrics for reasoning tokens. 13. **E2E-03:** Verify that the dashboard displays the correct metrics for artifact sizes. 14. **E2E-04:** Verify that the comparison view correctly compares two model profiles. 15. **E2E-05:** Verify that the redaction layer correctly redacts PII in a long-context response. 16. **E2E-06:** Verify that the proxy handles timeouts correctly for long-context responses. 17. **E2E-07:** Verify that the artifact storage correctly deletes old artifacts. 18. **E2E-08:** Verify that the database migration script runs correctly. 19. **E2E-09:** Verify that the Docker Compose configuration correctly mounts the artifact storage volume. 20. **E2E-10:** Verify that the CLI tools correctly upload and download artifacts. --- # Rollback Plan In case of deployment failure, the following rollback plan will be executed. **Database Rollback:** * Restore the SQLite database from the backup taken before the migration. * Verify the integrity of the database. **Artifact Storage Rollback:** * Delete the artifact storage directory. * Restore the artifact storage directory from the backup. **Redaction Layer Rollback:** * Disable the redaction layer by setting the `REDACTION_ENABLED` environment variable to `false`. **Benchmark Runner Rollback:** * Revert the benchmark runner to the previous version. * Verify that the benchmark runner works correctly. **Dashboard/Reporting Rollback:** * Revert the dashboard and reporting plane to the previous version. * Verify that the dashboard and reporting plane work correctly. **Docker Rollback:** * Revert the Docker image tag to the previous version. * Restart the Docker containers. --- # Open Questions 1. **Max Output Token Limit:** What is the maximum output token limit for the llama.cpp server? This will determine the timeout settings for the proxy. 2. **S3 Integration:** Should we use S3-compatible object storage instead of local filesystem for artifact storage? This would enable distributed deployments and better scalability. 3. **Streaming Reasoning Tokens:** How should we handle streaming reasoning tokens in the UI? Should we display them in real-time or wait for the final answer? 4. **PII Detection Accuracy:** How accurate is the regex-based PII detection? Should we use a more sophisticated NLP model for PII detection? 5. **Benchmark Campaign Scheduling:** Should we add support for scheduling benchmark campaigns? This would enable automated, periodic benchmarking. # Executive Summary This engineering dossier outlines the architectural evolution and deployment strategy for the **ai-flight-recorder** service, a private home-lab Python/FastAPI application designed to proxy, record, and benchmark LLM interactions. The current deployment, running alongside a llama.cpp server on `192.168.1.116`, is facing critical scaling and operational pain points as usage shifts toward high-complexity, long-context workloads. The primary objectives of this deployment are to resolve six specific pain points: 1. **Durable Artifacts:** Replace inline JSON payloads for long benchmark outputs with a disk-backed artifact storage system to prevent SQLite bloat and UI crashes. 2. **Reasoning Measurement:** Implement precise tokenization and duration tracking for the `reasoning` block in the Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf model without disabling the reasoning capability. 3. **Benchmark Metadata:** Enrich benchmark reports with granular metadata (model profile hashes, hardware telemetry, prompt fingerprints) to enable accurate model-to-model comparisons. 4. **Privacy Redaction:** Introduce a server-side redaction layer to strip PII (emails, Teams IDs, credit cards) from prompts and responses before they are persisted to SQLite. 5. **Auditability:** Track server-side model profile changes (e.g., switching from Qwen3.6 to a different quantization) via immutable event logs and model profile hashes. 6. **Stress Resilience:** Address timeouts, storage limits, and UI rendering issues caused by tens of thousands of output tokens by implementing streaming-aware timeouts, artifact offloading, and pagination. This dossier provides the concrete data model changes, artifact storage design, privacy mechanisms, and deployment plan required to transition the service from a prototype to a production-grade, deployment-ready system. The shift from a monolithic SQLite design to a hybrid SQLite + Object Storage design is the cornerstone of this evolution, enabling the system to handle the data volume and complexity of modern LLM workloads. --- # Current Architecture The ai-flight-recorder service is a monolithic Python application built on FastAPI, designed to act as an OpenAI-compatible proxy and telemetry recorder. **Core Components:** * **`app/main.py`**: The FastAPI application entry point. Exposes the dashboard UI, health check routes (`/health`), and the OpenAI-compatible proxy endpoint (`/v1/chat/completions`). * **`app/proxy.py`**: The core proxy logic. Forwards requests to the llama.cpp server at `192.168.1.116`. Captures request metadata, response previews, timings, and usage statistics. Currently writes these directly into SQLite. * **`app/store.py`**: SQLite write helpers. Contains functions for inserting runs, client sessions, LLM requests, events, and benchmark rows. Currently, large JSON payloads (prompts, responses) are serialized and stored directly in the `llm_requests` table. * **`app/reports.py`**: Aggregation logic. Queries SQLite to generate dashboard metrics and report summaries. * **`app/bench.py`**: Benchmark campaign runner. Iterates over a list of prompts, sends them to the proxy, and records the results. * **`deploy/docker-compose.yml`**: Orchestrates the ai-flight-recorder container and the llama-cpp-server container. Mounts the data volume at `/models/flight-recorder/data`. **Data Flow:** 1. Client sends a POST request to `http://ai-flight-recorder:8000/v1/chat/completions`. 2. `proxy.py` intercepts the request, extracts metadata (user, model, prompt), and forwards the request to `192.168.1.116:8080/v1/chat/completions`. 3. llama.cpp processes the request and returns a response. 4. `proxy.py` captures the response, calculates timings, and calls `store.py` to write the data to SQLite. 5. The response is streamed back to the client. **Current Infrastructure:** * **LLM Server:** llama-cpp-server running on `192.168.1.116`. * **Model:** `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` with reasoning enabled. * **Storage:** SQLite database located at `/models/flight-recorder/data/flight_recorder.db`. * **Network:** Internal home-lab network. **Text-Based Architecture Diagram:** ``` [Client] --> [FastAPI Proxy (ai-flight-recorder)] --> [llama.cpp Server (192.168.1.116)] ^ | | v +------------------- [SQLite DB] <-------------------+ ``` --- # Failure Modes Found During the review of the current architecture, the following failure modes were identified, particularly under high-load or long-context scenarios: 1. **SQLite Bloat and Corruption:** Storing full JSON responses (especially those with tens of thousands of tokens) directly in the `llm_requests` table causes the SQLite database to grow rapidly. This leads to slow queries, UI rendering lag, and potential database corruption if the database file exceeds 10GB. The WAL mode of SQLite is not designed to handle such large, frequent writes efficiently, leading to page cache thrashing and eventual corruption. 2. **HTTP Timeouts:** The default FastAPI timeout is 30 seconds. For long-context tasks requiring 32k output tokens, the proxy times out before the response is fully received, resulting in incomplete data and broken client connections. The asyncio timeout mechanism does not gracefully handle partial responses, leaving the client in a limbo state. 3. **Privacy Leaks:** The proxy currently stores raw prompts and responses. If a user accidentally pastes an email address, Teams ID, or credit card number, it is persisted in plaintext in the SQLite database, violating privacy policies. The lack of a redaction layer means that sensitive data is exposed to anyone with access to the database. 4. **Reasoning Blindness:** The current proxy does not distinguish between the `reasoning` block and the final answer in the llama.cpp response. This makes it impossible to measure reasoning token usage or duration, which is critical for evaluating the Qwen3.6 model's reasoning capabilities. The proxy treats the entire response as a single text blob, losing the semantic structure of the reasoning process. 5. **Audit Gaps:** If the model profile is changed server-side (e.g., switching from `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` to `Qwen3.6-35B-A3B-APEX-MTP-I-Heavy.gguf`), the proxy does not record this change. Benchmark results become incomparable across different model profiles, making it impossible to accurately assess model improvements or regressions. 6. **Benchmark Fragility:** The `bench.py` runner lacks resilience. If a single request fails due to a network blip or timeout, the entire benchmark campaign may halt or produce incomplete results. There is no mechanism to retry failed requests or to track partial progress, leading to wasted compute and inaccurate benchmark results. --- # Data Model Changes To address the failure modes, the SQLite data model must be refactored. The primary changes involve decoupling large payloads from the main tables, adding reasoning-specific fields, and introducing auditability. **New Tables:** 1. **`artifacts`**: Stores large payloads (prompts, responses, reasoning blocks) as file paths or S3 URIs. ```sql CREATE TABLE artifacts ( id INTEGER PRIMARY KEY AUTOINCREMENT, run_id INTEGER NOT NULL, artifact_type TEXT NOT NULL CHECK(artifact_type IN ('prompt', 'response', 'reasoning', 'benchmark_report')), content_hash TEXT NOT NULL, file_path TEXT NOT NULL, size_bytes INTEGER NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, FOREIGN KEY (run_id) REFERENCES runs(id) ); ``` 2. **`model_profiles`**: Tracks model profile changes for auditability. ```sql CREATE TABLE model_profiles ( id INTEGER PRIMARY KEY AUTOINCREMENT, profile_name TEXT NOT NULL, model_hash TEXT NOT NULL, quantization TEXT NOT NULL, reasoning_enabled INTEGER NOT NULL DEFAULT 1, activated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, deactivated_at TIMESTAMP ); ``` 3. **`benchmark_campaigns`**: Tracks benchmark campaign status and metadata. ```sql CREATE TABLE benchmark_campaigns ( id INTEGER PRIMARY KEY AUTOINCREMENT, campaign_name TEXT NOT NULL, status TEXT NOT NULL CHECK(status IN ('pending', 'running', 'completed', 'failed')), total_requests INTEGER NOT NULL DEFAULT 0, completed_requests INTEGER NOT NULL DEFAULT 0, failed_requests INTEGER NOT NULL DEFAULT 0, started_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, completed_at TIMESTAMP ); ``` **Modified Tables:** 1. **`runs`**: Add model profile hash and artifact references. ```sql ALTER TABLE runs ADD COLUMN model_profile_hash TEXT; ALTER TABLE runs ADD COLUMN reasoning_tokens INTEGER DEFAULT 0; ALTER TABLE runs ADD COLUMN reasoning_duration_ms INTEGER DEFAULT 0; ALTER TABLE runs ADD COLUMN artifact_prompt_id INTEGER; ALTER TABLE runs ADD COLUMN artifact_response_id INTEGER; ALTER TABLE runs ADD COLUMN artifact_reasoning_id INTEGER; ``` 2. **`llm_requests`**: Remove large text columns, add artifact references. ```sql ALTER TABLE llm_requests DROP COLUMN prompt_text; ALTER TABLE llm_requests DROP COLUMN response_text; ALTER TABLE llm_requests DROP COLUMN reasoning_text; ALTER TABLE llm_requests ADD COLUMN artifact_prompt_id INTEGER; ALTER TABLE llm_requests ADD COLUMN artifact_response_id INTEGER; ALTER TABLE llm_requests ADD COLUMN artifact_reasoning_id INTEGER; ALTER TABLE llm_requests ADD COLUMN prompt_redacted TEXT; ALTER TABLE llm_requests ADD COLUMN response_redacted TEXT; ``` 3. **`events`**: Add event types for auditability. ```sql -- Add new event types: 'model_profile_changed', 'artifact_uploaded', 'benchmark_campaign_started' ``` **Migration Notes:** * Run the migration script to create the new tables and modify existing ones. * For existing data, migrate large payloads from `llm_requests` to the `artifacts` table. Compress the payloads using `zstd` before writing to disk. * Populate the `model_profiles` table with the current model profile hash. **SQLAlchemy ORM Models:** ```python from sqlalchemy import Column, Integer, String, Text, ForeignKey, DateTime, CheckConstraint from sqlalchemy.orm import relationship from sqlalchemy.ext.declarative import declarative_base Base = declarative_base() class Artifact(Base): __tablename__ = 'artifacts' id = Column(Integer, primary_key=True) run_id = Column(Integer, ForeignKey('runs.id'), nullable=False) artifact_type = Column(String, nullable=False) content_hash = Column(String, nullable=False) file_path = Column(String, nullable=False) size_bytes = Column(Integer, nullable=False) created_at = Column(DateTime, default=datetime.utcnow) run = relationship("Run", back_populates="artifacts") class ModelProfile(Base): __tablename__ = 'model_profiles' id = Column(Integer, primary_key=True) profile_name = Column(String, nullable=False) model_hash = Column(String, nullable=False) quantization = Column(String, nullable=False) reasoning_enabled = Column(Integer, default=1) activated_at = Column(DateTime, default=datetime.utcnow) deactivated_at = Column(DateTime) class BenchmarkCampaign(Base): __tablename__ = 'benchmark_campaigns' id = Column(Integer, primary_key=True) campaign_name = Column(String, nullable=False) status = Column(String, nullable=False) total_requests = Column(Integer, default=0) completed_requests = Column(Integer, default=0) failed_requests = Column(Integer, default=0) started_at = Column(DateTime, default=datetime.utcnow) completed_at = Column(DateTime) ``` --- # Artifact Storage Design To solve the SQLite bloat and UI rendering issues, large payloads must be offloaded to durable artifact storage. **Storage Backend:** * **Primary:** Local filesystem (`/models/flight-recorder/artifacts/`). * **Secondary (Future):** S3-compatible object storage (e.g., MinIO) for distributed deployments. **Directory Structure:** ``` /models/flight-recorder/artifacts/ ├── runs/ │ ├── {run_id}/ │ │ ├── prompt.zst │ │ ├── response.zst │ │ └── reasoning.zst ├── benchmarks/ │ ├── {campaign_id}/ │ │ ├── report.json │ │ └── artifacts/ │ │ ├── {request_id}_prompt.zst │ │ └── {request_id}_response.zst ``` **Compression:** * Use `zstd` (Zstandard) for compression. It offers a good balance of speed and compression ratio, which is critical for high-throughput environments. * Compression level: 3 (fast compression, good ratio). **Artifact Lifecycle:** * **Creation:** When a request is completed, the proxy writes the payload to the artifact storage and records the file path and hash in the `artifacts` table. * **Retrieval:** The dashboard and reports query the `artifacts` table to get the file path, then stream the file from the filesystem. * **Retention:** Artifacts older than 30 days are automatically deleted by a background cron job. **CLI Example:** ```bash # Upload an artifact python -m app.artifacts upload --run-id 123 --type response --file /tmp/response.json # Download an artifact python -m app.artifacts download --run-id 123 --type response --output /tmp/response.json # List artifacts for a run python -m app.artifacts list --run-id 123 ``` **S3 Integration (Future):** * If S3 is used, the `file_path` column in the `artifacts` table will store the S3 URI (e.g., `s3://bucket-name/runs/123/response.zst`). * The proxy will use the AWS SDK to upload and download artifacts from S3. * The S3 bucket will have lifecycle rules to automatically delete artifacts older than 30 days. --- # Reasoning Budget Handling To measure reasoning output without turning thinking off, we must parse the `reasoning` field in the llama.cpp response and tokenize it separately. **Tokenization Strategy:** * Use the `tiktoken` library with the `cl100k_base` encoding (compatible with Qwen models). * Tokenize the `reasoning` block and the final answer block separately. * Calculate the token count and duration for each block. **Implementation:** 1. **Response Parsing:** When the proxy receives the response from llama.cpp, it checks for the `reasoning` field. 2. **Token Counting:** ```python import tiktoken enc = tiktoken.get_encoding("cl100k_base") reasoning_tokens = len(enc.encode(response.get("reasoning", ""))) answer_tokens = len(enc.encode(response.get("content", ""))) ``` 3. **Duration Tracking:** Measure the time between the start of the `reasoning` block and the start of the final answer block. 4. **Budget Enforcement:** If the reasoning tokens exceed a threshold (e.g., 80% of the total output tokens), log a warning. This helps identify models that are "over-thinking" and wasting compute. **Schema Update:** * Add `reasoning_tokens` (INTEGER) and `reasoning_duration_ms` (INTEGER) to the `runs` table. **Dashboard Metrics:** * `llm_reasoning_tokens_total`: Counter of reasoning tokens generated. * `llm_reasoning_duration_seconds`: Histogram of reasoning duration. * `llm_reasoning_ratio`: Gauge of reasoning tokens / total output tokens. **Llama.cpp Response Format:** ```json { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "The final answer is 42.", "role": "assistant", "reasoning": "Let's think step by step. First, we need to calculate the sum of 20 and 22. 20 + 22 = 42." } } ] } ``` --- # Benchmark Runner Design The `bench.py` runner must be refactored to handle long outputs, provide resilience, and capture rich metadata. **Architecture:** * **Async Concurrency:** Use `asyncio.Semaphore` to limit the number of concurrent requests to llama.cpp. * **Campaign Definition:** Define campaigns in YAML files. * **Resilience:** Implement retry logic for failed requests. * **Artifact Integration:** Write responses to artifact storage instead of inline JSON. **Campaign YAML Example:** ```yaml campaign_name: "qwen36_reasoning_benchmark" model_profile: "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced" timeout_seconds: 600 max_concurrent_requests: 4 prompts: - id: "prompt_1" text: "Solve this complex math problem..." expected_tokens: 32000 - id: "prompt_2" text: "Write a detailed technical report..." expected_tokens: 16000 ``` **CLI Example:** ```bash # Run a benchmark campaign python -m app.bench run --campaign /models/flight-recorder/campaigns/qwen36_reasoning_benchmark.yaml --timeout 600 # Resume a failed campaign python -m app.bench resume --campaign-id 456 ``` **Metadata Capture:** * Record the model profile hash, hardware telemetry (CPU/GPU usage), and prompt fingerprint (SHA-256 of the prompt) for each request. * This enables accurate model-to-model comparisons by ensuring that the same prompts are used across different model profiles. **Retry Logic:** * Implement exponential backoff for failed requests. * Maximum retries: 3. * Backoff factor: 2. * Initial delay: 1 second. --- # Reporting Plane The reporting plane must be updated to generate rich, metadata-enriched reports that are suitable for screenshots and later comparisons. **Dashboard Metrics:** * `llm_request_duration_seconds`: Histogram of request duration. * `llm_output_tokens_total`: Counter of output tokens generated. * `llm_artifact_size_bytes`: Histogram of artifact sizes. * `llm_redaction_events_total`: Counter of PII redaction events. **Report Generation:** * **HTML Reports:** Generate interactive HTML reports using `Jinja2` templates. Include charts (using `Plotly` or `Chart.js`) for token usage, duration, and reasoning ratios. * **PDF Reports:** Generate PDF reports using `WeasyPrint` for executive summaries. * **Comparison Views:** Provide side-by-side views of Model A vs Model B on the same prompt, highlighting differences in reasoning duration, token count, and output quality. **CLI Example:** ```bash # Generate a report for a benchmark campaign python -m app.reports generate --campaign-id 456 --format html --output /models/flight-recorder/reports/qwen36_reasoning_benchmark.html # Generate a comparison report python -m app.reports compare --campaign-id 456 --campaign-id 457 --output /models/flight-recorder/reports/comparison.html ``` **Dashboard UI Components:** * **Token Usage Chart:** Line chart showing token usage over time. * **Reasoning Ratio Chart:** Bar chart showing the reasoning ratio for each request. * **Artifact Size Chart:** Histogram showing the distribution of artifact sizes. * **PII Redaction Log:** Table showing the number of PII redaction events. --- # Privacy And Redaction To prevent privacy leaks, a server-side redaction layer must be implemented to strip PII from prompts and responses before they are persisted. **Redaction Strategy:** * **Regex-Based Redaction:** Use regex patterns to detect and redact common PII (emails, credit cards, phone numbers). * **LLM-Based Redaction:** Use a small, local LLM (e.g., `Phi-3-mini`) to detect and redact context-aware PII (e.g., Teams IDs, project names). **Regex Patterns:** ```python PII_PATTERNS = { "email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "credit_card": r"\b(?:\d[ -]*?){13,16}\b", "phone": r"\b(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b", "teams_id": r"\b[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}\b", } ``` **Implementation:** 1. **Pre-Write Redaction:** Before writing to SQLite, the proxy calls the redaction layer to strip PII from the prompt and response. 2. **Redacted Storage:** Store the redacted text in the `prompt_redacted` and `response_redacted` columns of the `llm_requests` table. 3. **Raw Storage:** Store the raw text in the artifact storage (if required for debugging, but access is restricted). **PII Classification Taxonomy:** * **PII (Personally Identifiable Information):** Names, addresses, phone numbers, email addresses. * **PHI (Protected Health Information):** Medical records, health insurance numbers. * **PCI (Payment Card Industry):** Credit card numbers, bank account numbers. **CLI Example:** ```bash # Scan a directory for PII python -m app.redactor scan --path /models/flight-recorder/data/raw --output /models/flight-recorder/reports/pii_scan.html # Redact a file python -m app.redactor redact --input /models/flight-recorder/data/raw/response.json --output /models/flight-recorder/data/raw/response_redacted.json ``` --- # Deployment Plan The deployment will be executed in phases to minimize risk. **Phase 1: Database Migration** * Run the migration script to create the new tables and modify existing ones. * Migrate large payloads from `llm_requests` to the `artifacts` table. * Verify the integrity of the database. **Phase 2: Artifact Storage Setup** * Create the artifact storage directory structure. * Update the Docker Compose configuration to mount the artifact storage volume. * Test artifact upload and retrieval. **Phase 3: Redaction Layer Integration** * Deploy the redaction layer. * Test PII detection and redaction. * Verify that redacted data is stored correctly. **Phase 4: Benchmark Runner Update** * Deploy the updated benchmark runner. * Run a small benchmark campaign to verify resilience and metadata capture. * Test artifact integration. **Phase 5: Dashboard/Reporting Update** * Deploy the updated dashboard and reporting plane. * Test report generation and comparison views. * Verify that all metrics are being captured correctly. **Docker Compose Changes:** ```yaml services: ai-flight-recorder: image: ai-flight-recorder:latest volumes: - /models/flight-recorder/data:/models/flight-recorder/data - /models/flight-recorder/artifacts:/models/flight-recorder/artifacts environment: - REDACTION_ENABLED=true - ARTIFACT_STORAGE_PATH=/models/flight-recorder/artifacts - TIMEOUT_SECONDS=600 ``` **CI/CD Pipeline:** * Use GitHub Actions to build and push the Docker image. * Use GitHub Actions to run the test suite. * Use GitHub Actions to deploy the Docker image to the home-lab server. --- # Test Plan The test plan includes unit tests, integration tests, and end-to-end tests to ensure the new features work correctly. **Unit Tests:** 1. **UT-01:** Verify regex-based PII redaction for emails. 2. **UT-02:** Verify regex-based PII redaction for credit cards. 3. **UT-03:** Verify tokenization of the `reasoning` block. 4. **UT-04:** Verify artifact compression using `zstd`. 5. **UT-05:** Verify artifact decompression using `zstd`. **Integration Tests:** 6. **IT-01:** Verify that the proxy writes artifacts to disk and records the file path in SQLite. 7. **IT-02:** Verify that the proxy reads artifacts from disk and serves them via the API. 8. **IT-03:** Verify that the proxy redacts PII before writing to SQLite. 9. **IT-04:** Verify that the benchmark runner retries failed requests. 10. **IT-05:** Verify that the benchmark runner captures model profile hashes. **End-to-End Tests:** 11. **E2E-01:** Run a full benchmark campaign and verify that the report is generated correctly. 12. **E2E-02:** Verify that the dashboard displays the correct metrics for reasoning tokens. 13. **E2E-03:** Verify that the dashboard displays the correct metrics for artifact sizes. 14. **E2E-04:** Verify that the comparison view correctly compares two model profiles. 15. **E2E-05:** Verify that the redaction layer correctly redacts PII in a long-context response. 16. **E2E-06:** Verify that the proxy handles timeouts correctly for long-context responses. 17. **E2E-07:** Verify that the artifact storage correctly deletes old artifacts. 18. **E2E-08:** Verify that the database migration script runs correctly. 19. **E2E-09:** Verify that the Docker Compose configuration correctly mounts the artifact storage volume. 20. **E2E-10:** Verify that the CLI tools correctly upload and download artifacts. **Automation Scripts:** * Use `pytest` to run the unit and integration tests. * Use `selenium` to run the end-to-end tests. * Use `tox` to run the tests in multiple Python environments. --- # Rollback Plan In case of deployment failure, the following rollback plan will be executed. **Database Rollback:** * Restore the SQLite database from the backup taken before the migration. * Verify the integrity of the database. **Artifact Storage Rollback:** * Delete the artifact storage directory. * Restore the artifact storage directory from the backup. **Redaction Layer Rollback:** * Disable the redaction layer by setting the `REDACTION_ENABLED` environment variable to `false`. **Benchmark Runner Rollback:** * Revert the benchmark runner to the previous version. * Verify that the benchmark runner works correctly. **Dashboard/Reporting Rollback:** * Revert the dashboard and reporting plane to the previous version. * Verify that the dashboard and reporting plane work correctly. **Docker Rollback:** * Revert the Docker image tag to the previous version. * Restart the Docker containers. **Database Backup:** * Use `sqlite3` to create a backup of the database before the migration. * Store the backup in a secure location. --- # Open Questions 1. **Max Output Token Limit:** What is the maximum output token limit for the llama.cpp server? This will determine the timeout settings for the proxy. 2. **S3 Integration:** Should we use S3-compatible object storage instead of local filesystem for artifact storage? This would enable distributed deployments and better scalability. 3. **Streaming Reasoning Tokens:** How should we handle streaming reasoning tokens in the UI? Should we display them in real-time or wait for the final answer? 4. **PII Detection Accuracy:** How accurate is the regex-based PII detection? Should we use a more sophisticated NLP model for PII detection? 5. **Benchmark Campaign Scheduling:** Should we add support for scheduling benchmark campaigns? This would enable automated, periodic benchmarking. 6. **Hardware Telemetry:** How should we capture hardware telemetry (CPU/GPU usage) for each request? Should we use a separate monitoring tool like Prometheus? 7. **Model Profile Hashing:** How should we calculate the model profile hash? Should we use the SHA-256 hash of the model file? 8. **Prompt Fingerprinting:** How should we calculate the prompt fingerprint? Should we use the SHA-256 hash of the prompt text? 9. **Artifact Chunking:** Should we chunk large artifacts into smaller pieces? This would enable parallel uploads and downloads. 10. **LLM-Based Redaction:** How should we integrate the LLM-based redaction pipeline? Should we use a separate container for the LLM?