# Root Cause Hypotheses

Based on the symptoms (intermittent duplicate benchmark rows) and the architecture (FastAPI, SQLite, concurrent clients like OpenWebUI and agents), there are four primary hypotheses for the root cause:

### 1. Lack of Request Idempotency (Primary Suspect)
The most likely cause is that the clients (OpenWebUI, Agent Client) are retrying requests due to network timeouts or slow responses. If the `create_run` function does not check for a unique identifier provided by the client, every retry results in a new row in the SQLite database. Because the proxy handles concurrent requests, a "slow" request might time out on the client side, triggering a retry while the original request is still being processed by the server.

### 2. "Check-then-Act" Race Condition
The application logic likely follows a pattern like:
1. `SELECT` to see if a run exists for this parameters.
2. If not, `INSERT` a new run.
In a concurrent environment, two threads/tasks can perform the `SELECT` simultaneously, both see that no run exists, and both proceed to `INSERT`. Even with SQLite's locking, if the application logic isn't wrapped in a proper transaction or using `ON CONFLICT` clauses, this leads to duplicates.

### 3. SQLite Concurrency Limitations
While SQLite supports multiple readers, it only supports one writer at a time. If the `aiosqlite` or `sqlite3` connection pool is not configured correctly, or if the `journal_mode` is not set to `WAL` (Write-Ahead Logging), concurrent writes can lead to `Database is locked` errors. If the application catches these errors and retries the entire request flow, it creates a new row.

### 4. Streaming Connection Re-establishment
For streaming requests, if the TCP connection drops but the server-side generator continues to run, the client might reconnect. If the reconnection logic triggers `create_run` again instead of resuming the previous run ID, a duplicate "run" entry will be created for what is effectively the same logical operation.

---

# Evidence To Collect

To confirm these hypotheses, the following data points must be gathered:

1.  **Request Logs with Correlation IDs:**
    *   Capture `X-Request-ID` or a similar header from the proxy.
    *   Log the timestamp, client IP, and the specific payload of every request hitting `create_run`.
2.  **Database Transaction Logs:**
    *   Enable SQLite's `trace` or use a wrapper to log every SQL statement executed.
    *   Identify if multiple `INSERT INTO runs` statements occur with identical parameters within a <500ms window.
3.  **Client-Side Retry Logs:**
    *   Check OpenWebUI and Agent logs to see if they are issuing retries on 504 (Gateway Timeout) or 500 (Internal Server Error) responses.
4.  **SQLite Lock Contention Metrics:**
    *   Monitor the frequency of `sqlite3.OperationalError: database is locked` errors in the FastAPI logs.
5.  **Streaming State Analysis:**
    *   Log the lifecycle of a stream: `Stream_Started`, `Stream_Chunk_Sent`, `Stream_Finished`, and `Stream_Error`.
    *   Check if `Stream_Started` appears twice for the same user session without a `Stream_Finished` in between.

---

# Patch Plan

The fix follows a multi-layered approach: **Idempotency at the API level**, **Atomicity at the Database level**, and **Concurrency optimization at the Engine level**.

### Phase 1: Idempotency Layer
Introduce an `idempotency_key` header. Clients (or the proxy) must generate a UUID for each unique benchmark request.
*   If the key exists, return the existing run metadata.
*   If it doesn't, create it.

### Phase 2: Database Schema Update
Modify the `runs` table to enforce uniqueness at the engine level. This is the "engineering-grade" safety net.

### Phase 3: Code Refactoring
Update `create_run` to use an atomic "Upsert" pattern.

### Phase 4: SQLite Optimization
Enable WAL mode to allow concurrent reads and writes.

---

# SQLite Constraints And Indexes

To ensure data integrity, apply the following schema changes.

### SQL Schema Update
```sql
-- 1. Enable WAL Mode (Run this on connection initialization)
PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
PRAGMA busy_timeout=5000; -- Wait up to 5s for locks to clear

-- 2. Add Idempotency Key to Runs Table
-- If table already exists, use ALTER TABLE or a migration script
ALTER TABLE runs ADD COLUMN idempotency_key TEXT UNIQUE;

-- 3. Create Index for faster lookups on common query patterns
CREATE INDEX IF NOT EXISTS idx_runs_status ON runs(status);
CREATE INDEX IF NOT EXISTS idx_runs_client_id ON runs(client_id);
```

### Python Implementation (SQLAlchemy/aiosqlite style)
Instead of:
```python
# BAD: Check-then-act
run = await db.fetch("SELECT id FROM runs WHERE idempotency_key = ?", (key,))
if not run:
    await db.execute("INSERT INTO runs ...")
```

Use:
```python
# GOOD: Atomic Upsert
query = """
INSERT INTO runs (idempotency_key, metadata, status)
VALUES (:key, :meta, :status)
ON CONFLICT(idempotency_key) DO NOTHING;
"""
# Then fetch the result (either the new row or the existing one)
```

---

# Streaming Edge Cases

Streaming introduces specific complexities that standard REST endpoints do not face:

1.  **The "Zombie" Stream:**
    *   *Scenario:* Client disconnects, but the `create_llm_request` generator keeps running.
    *   *Fix:* Use a `ContextManager` or a `TaskGroup` to ensure that if the FastAPI `StreamingResponse` context closes, the underlying LLM request is cancelled immediately.

2.  **Partial Success / Resume:**
    *   *Scenario:* A stream fails at 50% completion. The client retries.
    *   *Fix:* The `idempotency_key` must remain the same. The `create_run` logic should detect the existing run and check if it's in a `FAILED` or `IN_PROGRESS` state. If `IN_PROGRESS`, the system should decide whether to resume or restart (usually restart for LLM benchmarks).

3.  **Duplicate `add_event` calls:**
    *   *Scenario:* A retry logic in the streaming loop causes the same event to be logged twice.
    *   *Fix:* Each event should have a deterministic ID (e.g., `hash(run_id + sequence_number)`). Use `INSERT OR IGNORE` for events.

---

# Test Plan

To ensure the fix is robust, execute the following 12 acceptance tests:

### Concurrency & Idempotency Tests
1.  **Identical Concurrent Requests:** Fire 10 identical requests with the same `idempotency_key` simultaneously. *Expected: 1 row created, 9 successes returning the same ID.*
2.  **Rapid Retry Test:** Simulate a 2-second network delay on the first request, then fire a retry with the same key. *Expected: Second request returns the first request's ID.*
3.  **High-Volume Concurrent Writes:** Fire 50 different requests with different keys simultaneously. *Expected: All 50 rows created without `Database is locked` errors.*
4.  **Agent "Burst" Test:** Simulate an agent client firing 5 requests in 100ms. *Expected: No duplicates, all requests handled.*

### Streaming & Network Tests
5.  **Client Disconnect (Mid-Stream):** Kill the client connection during a stream. *Expected: Server-side task terminates; no duplicate run created on reconnect.*
6.  **Partial Stream Failure:** Force a 500 error halfway through a stream. *Expected: Run status updated to `FAILED`; subsequent retry with same key creates a new run or resumes correctly.*
7.  **Proxy Timeout Simulation:** Use a proxy to inject a 10s delay. *Expected: Client retries are caught by the idempotency key.*
8.  **Slow Consumer Test:** Stream to a very slow client. *Expected: Server does not hang or block other concurrent runs.*

### Database & Integrity Tests
9.  **Unique Constraint Violation:** Manually attempt to insert a duplicate `idempotency_key` via raw SQL. *Expected: Database rejects the insert.*
10. **WAL Mode Verification:** Verify `PRAGMA journal_mode` is `wal` under load. *Expected: Concurrent reads do not block writes.*
11. **Large Payload Integrity:** Send a benchmark with a 10k token prompt. *Expected: Row created correctly without truncation.*
12. **Rollback Integrity:** Simulate a crash during `create_run`. *Expected: Database remains in a consistent state (no partial rows).*

---

# Rollback Plan

In the event of a regression (e.g., the `ON CONFLICT` logic causes unexpected behavior or the `idempotency_key` causes collisions):

1.  **Immediate Code Revert:** Revert the FastAPI service to the previous stable commit.
2.  **Database Schema Reversion:**
    *   If the `idempotency_key` column was added, it can remain (it won't affect the old code).
    *   If the `UNIQUE` constraint causes issues, drop the constraint: `ALTER TABLE runs DROP CONSTRAINT runs_idempotency_key_key;`
3.  **Cache/Registry Flush:** If an in-memory registry was used for idempotency, clear the cache.
4.  **Manual Cleanup:** Run a script to delete duplicate rows based on timestamp and metadata similarity:
    ```sql
    DELETE FROM runs 
    WHERE id IN (
        SELECT id FROM (
            SELECT id, ROW_NUMBER() OVER (PARTITION BY metadata ORDER BY created_at DESC) as rn
            FROM runs
        ) WHERE rn > 1
    );
    ```

---

# Decision Summary

**Selected Solution:** **Database-level Idempotency with WAL Mode.**

*   **Why Idempotency Keys?** It is the industry standard for handling retries in distributed systems. By requiring a key from the client (or generating a deterministic hash of the request body), we eliminate the "Check-then-Act" race condition entirely.
*   **Why `ON CONFLICT`?** It moves the atomicity requirement from the Python application layer to the Database engine. This is significantly safer and faster than manual checks.
*   **Why WAL Mode?** It is the standard for production-grade SQLite. It allows multiple readers and one writer to operate simultaneously without blocking, which is critical for a service handling concurrent requests from OpenWebUI and agents.
*   **Engineering Grade?** Yes. This approach handles network instability, concurrent execution, and database locking—the three pillars of production reliability—without requiring the complexity of a full Postgres/Redis stack.

## Detailed Implementation Guide

To move from theory to production, the following implementation provides a robust, thread-safe, and idempotent way to handle benchmark runs using `aiosqlite` and FastAPI.

### 1. Database Manager and Schema
We encapsulate the database logic to ensure that every connection is initialized with the correct `PRAGMA` settings.

```python
import aiosqlite
import json
import logging
import uuid
from typing import Optional, Dict, Any
from fastapi import HTTPException

logger = logging.getLogger("benchmark_service")

class DatabaseManager:
    def __init__(self, db_path: str):
        self.db_path = db_path

    async def get_connection(self) -> aiosqlite.Connection:
        conn = await aiosqlite.connect(self.db_path)
        # Enable WAL mode and other performance/concurrency settings
        await conn.execute("PRAGMA journal_mode=WAL;")
        await conn.execute("PRAGMA synchronous=NORMAL;")
        await conn.execute("PRAGMA busy_timeout=5000;")
        await conn.execute("PRAGMA cache_size=-64000;")  # ~64MB cache
        await conn.execute("PRAGMA mmap_size=268435456;")  # 256MB memory map
        conn.row_factory = aiosqlite.Row
        return conn

    async def initialize_schema(self):
        async with await self.get_connection() as conn:
            await conn.execute("""
                CREATE TABLE IF NOT EXISTS runs (
                    id TEXT PRIMARY KEY,
                    idempotency_key TEXT UNIQUE,
                    client_id TEXT,
                    metadata TEXT,
                    status TEXT,
                    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                );
            """)
            await conn.execute("""
                CREATE TABLE IF NOT EXISTS events (
                    id INTEGER PRIMARY KEY AUTOINCREMENT,
                    run_id TEXT,
                    event_type TEXT,
                    content TEXT,
                    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                    FOREIGN KEY(run_id) REFERENCES runs(id)
                );
            """)
            await conn.commit()
```

### 2. Idempotent Run Creation
The `create_run` function now uses the `INSERT ... ON CONFLICT` pattern. This ensures that even if two requests hit the server at the exact same microsecond with the same key, the database engine handles the atomicity.

```python
async def create_run(
    db_manager: DatabaseManager, 
    idempotency_key: str, 
    client_id: str, 
    metadata: Dict[str, Any]
) -> Dict[str, Any]:
    """
    Creates a new run or returns the existing one if the key is already present.
    """
    run_id = str(uuid.uuid4())
    meta_json = json.dumps(metadata)
    
    async with await db_manager.get_connection() as conn:
        # Atomic Upsert: If key exists, do nothing and we will fetch the existing row
        await conn.execute("""
            INSERT INTO runs (id, idempotency_key, client_id, metadata, status)
            VALUES (?, ?, ?, ?, ?)
            ON CONFLICT(idempotency_key) DO NOTHING;
        """, (run_id, idempotency_key, client_id, meta_json, "started"))
        
        await conn.commit()
        
        # Fetch the row (either the one we just inserted or the existing one)
        async with conn.execute(
            "SELECT * FROM runs WHERE idempotency_key = ?", (idempotency_key,)
        ) as cursor:
            row = await cursor.fetchone()
            if not row:
                # This should technically be impossible due to the transaction logic
                raise HTTPException(status_code=500, detail="Failed to retrieve run record.")
            
            return dict(row)
```

### 3. Event Logging with Deduplication
To prevent duplicate events during retries, we use a deterministic event ID or a simple `INSERT OR IGNORE` if the event content is unique.

```python
async def add_event(
    db_manager: DatabaseManager, 
    run_id: str, 
    event_type: str, 
    content: str
):
    """
    Logs an event to the run. Uses a simple insert. 
    For higher precision, a unique constraint on (run_id, event_type, content) 
    could be added to the events table.
    """
    async with await db_manager.get_connection() as conn:
        await conn.execute("""
            INSERT INTO events (run_id, event_type, content)
            VALUES (?, ?, ?)
        """, (run_id, event_type, content))
        await conn.commit()
```

### 4. Streaming Response with Context Management
This ensures that if a client disconnects, the LLM generation is halted, preventing "zombie" processes from consuming resources and potentially creating inconsistent states.

```python
from fastapi.responses import StreamingResponse
import asyncio

async def stream_llm_response(
    db_manager: DatabaseManager,
    run_data: Dict[str, Any],
    llm_generator: asyncio.Generator
):
    run_id = run_data["id"]
    
    async def event_generator():
        try:
            async for chunk in llm_generator:
                # Check if client is still connected
                # (FastAPI handles this via the generator's lifecycle)
                yield f"data: {json.dumps({'chunk': chunk})}\n\n"
                await add_event(db_manager, run_id, "chunk", chunk)
            
            await update_run_status(db_manager, run_id, "completed")
        except asyncio.CancelledError:
            # This is triggered when the client disconnects
            logger.info(f"Stream cancelled for run {run_id}")
            await update_run_status(db_manager, run_id, "cancelled")
            raise
        except Exception as e:
            logger.error(f"Error in stream {run_id}: {e}")
            await update_run_status(db_manager, run_id, "failed")
            yield f"data: {json.dumps({'error': str(e)})}\n\n"

    return StreamingResponse(event_generator(), media_type="text/event-stream")

async def update_run_status(db_manager: DatabaseManager, run_id: str, status: str):
    async with await db_manager.get_connection() as conn:
        await conn.execute(
            "UPDATE runs SET status = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?",
            (status, run_id)
        )
        await conn.commit()
```

---

# Observability and Monitoring

To maintain an engineering-grade service, we must move beyond "it works" to "we know why it failed."

### 1. Structured Logging
Replace standard print statements with structured JSON logging. This allows for easy parsing by tools like ELK or Grafana Loki.
*   **Fields to include:** `request_id`, `idempotency_key`, `run_id`, `client_id`, `latency_ms`, `db_lock_time_ms`.
*   **Log Levels:**
    *   `INFO`: Request received, Run created, Stream finished.
    *   `WARNING`: Idempotency hit (client retried), Database lock contention.
    *   `ERROR`: SQL errors, LLM timeouts, Connection drops.

### 2. Prometheus Metrics
Expose a `/metrics` endpoint to track the following:
*   `benchmark_requests_total{status="success|failure|duplicate"}`: Tracks the volume of requests.
*   `benchmark_idempotency_hits_total`: A high number here indicates network instability or aggressive client retries.
*   `benchmark_db_lock_wait_seconds`: Measures how long threads are waiting for the SQLite write lock.
*   `benchmark_stream_duration_seconds`: Histogram of how long benchmarks take.
*   `benchmark_active_streams`: Gauge of concurrent active streams.

### 3. Alerting Rules
*   **High Duplicate Rate:** Alert if `idempotency_hits` exceeds 20% of total requests over a 5-minute window (indicates proxy/network issues).
*   **Database Lock Contention:** Alert if `db_lock_wait_seconds` exceeds 1 second for more than 5% of requests.
*   **Stream Failure Rate:** Alert if the ratio of `status="failed"` to `status="completed"` exceeds 10%.

---

# Security and Input Validation

Since this is a production-grade service, we must protect the SQLite database and the LLM logic from malicious inputs.

### 1. Pydantic Schema Validation
Never pass raw JSON directly into the database. Use Pydantic to enforce types and constraints.
```python
from pydantic import BaseModel, Field, validator
from typing import List, Optional

class BenchmarkRequest(BaseModel):
    model_id: str = Field(..., min_length=3, max_length=50)
    prompt: str = Field(..., max_length=10000)
    temperature: float = Field(default=0.7, ge=0.0, le=2.0)
    max_tokens: int = Field(default=2048, gt=0)
    
    @validator('model_id')
    def validate_model_id(cls, v):
        allowed = ["gpt-4", "claude-3", "llama-3"]
        if v not in allowed:
            raise ValueError(f"Model {v} is not supported.")
        return v
```

### 2. SQL Injection Prevention
While `aiosqlite` handles parameter binding, ensure that no f-strings are used in SQL queries.
*   **Bad:** `conn.execute(f"INSERT INTO runs VALUES ('{run_id}')")`
*   **Good:** `conn.execute("INSERT INTO runs VALUES (?)", (run_id,))`

### 3. Rate Limiting
Implement a per-`client_id` rate limit using a simple leaky bucket algorithm or a Redis-backed counter (if moving beyond a single SQLite file). This prevents a single agent client from saturating the SQLite write lock and starving other users.

---

# Advanced SQLite Tuning & Filesystem

For a home-lab environment, SQLite is excellent, but it requires specific tuning to behave like a production database.

### 1. Filesystem Considerations
*   **Journaling:** Ensure the underlying filesystem (e.g., EXT4, XFS) is not mounted with `sync` options, as this will cripple SQLite performance.
*   **Disk I/O:** If possible, place the SQLite database on an NVMe SSD. SQLite's performance is heavily bound by disk latency during `fsync` operations.

### 2. PRAGMA Deep Dive
*   **`mmap_size`:** By setting this to a large value (e.g., 256MB), SQLite can map the database file into memory. This significantly speeds up reads and reduces the overhead of system calls.
*   **`cache_size`:** A negative value (e.g., `-64000`) specifies the cache size in kilobytes. Increasing this keeps more of the index in memory, reducing disk hits.
*   **`busy_timeout`:** This is critical. It tells SQLite to wait for a specified duration (e.g., 5000ms) for a lock to clear before throwing an error. This handles the "intermittent" nature of concurrent writes.

### 3. Connection Pooling
While `aiosqlite` is asynchronous, SQLite itself is not a client-server DB.
*   **Single Writer Pattern:** For high-concurrency, consider a dedicated "Writer Thread" or a queue where all `INSERT/UPDATE` operations are serialized, while `SELECT` operations remain concurrent. This eliminates `Database is locked` errors entirely.

---

# Client-Side Integration Strategy

The fix is only complete if the clients (OpenWebUI, Agent Client) are updated to cooperate with the idempotency logic.

### 1. Idempotency Key Generation
Clients should generate a unique key for every *logical* request.
*   **Deterministic Keys:** For an Agent client, the key should be a hash of the prompt, the model ID, and the temperature.
    *   `key = sha256(prompt + model_id + temperature).hexdigest()`
*   **UUID Keys:** For a UI-driven request (OpenWebUI), the key should be generated when the "Submit" button is clicked and persisted in the frontend state.

### 2. Handling 409 Conflict vs. 200 OK
The server's `ON CONFLICT DO NOTHING` logic means that if a client retries a request that was already successfully created, the server will return the *existing* run.
*   **Client Logic:** The client should check the `run_id` in the response. If the `run_id` matches the one from a previous (timed-out) attempt, the client should resume listening to that stream or fetch the existing status.

### 3. Proxy Configuration
If using Nginx or Traefik as a proxy:
*   Ensure `proxy_buffering` is off for streaming.
*   Ensure `proxy_read_timeout` is high enough to accommodate long LLM generations (e.g., 300s).
*   Pass the `X-Request-ID` header through to the FastAPI app.

---

# Migration Script & Data Cleanup

To transition from the current "broken" state to the "fixed" state, a migration script is required.

### 1. Schema Migration Script
```python
import asyncio
import aiosqlite

async def migrate_database(db_path: str):
    async with aiosqlite.connect(db_path) as conn:
        # 1. Add the idempotency_key column
        try:
            await conn.execute("ALTER TABLE runs ADD COLUMN idempotency_key TEXT")
            print("Added idempotency_key column.")
        except aiosqlite.OperationalError:
            print("Column already exists.")

        # 2. Create a unique index
        await conn.execute("CREATE UNIQUE INDEX IF NOT EXISTS idx_runs_idempotency ON runs(idempotency_key)")
        
        # 3. Backfill idempotency_key for existing rows
        # We use a hash of the metadata to create a deterministic key for old data
        await conn.execute("""
            UPDATE runs 
            SET idempotency_key = hex(randomblob(16)) 
            WHERE idempotency_key IS NULL
        """)
        
        await conn.commit()
        print("Migration complete.")

if __name__ == "__main__":
    asyncio.run(migrate_database("benchmark.db"))
```

### 2. Duplicate Cleanup
Run this once to clean up the existing duplicates before enabling the new logic:
```sql
-- Identify duplicates based on metadata and timestamp
WITH DuplicateCTE AS (
    SELECT id, 
           ROW_NUMBER() OVER (
               PARTITION BY metadata 
               ORDER BY created_at DESC
           ) as rn
    FROM runs
)
DELETE FROM runs WHERE id IN (SELECT id FROM DuplicateCTE WHERE rn > 1);
```

---

# Scalability Roadmap

While SQLite is sufficient for a home lab, the following steps outline the path to a multi-user production environment.

### Phase 1: SQLite + WAL (Current State)
*   **Pros:** Simple, zero-config, extremely fast for single-node.
*   **Limit:** Single-writer bottleneck.

### Phase 2: SQLite + Write Queue
*   **Implementation:** Use a `multiprocessing.Queue` or `asyncio.Queue` to funnel all writes to a single background worker.
*   **Pros:** Eliminates `Database is locked` errors completely.
*   **Limit:** Still restricted to a single machine.

### Phase 3: PostgreSQL Migration
*   **Trigger:** When the number of concurrent users exceeds 10 or when you need to run multiple instances of the FastAPI service behind a load balancer.
*   **Changes:**
    *   Replace `aiosqlite` with `SQLAlchemy` + `asyncpg`.
    *   Use `pg_advisory_locks` for complex distributed locks.
    *   Move idempotency keys to a Redis cache for sub-millisecond lookups.

### Phase 4: Distributed Architecture
*   **Implementation:** FastAPI instances are stateless. Redis handles idempotency and rate limiting. PostgreSQL handles persistent storage.
*   **Pros:** Infinite horizontal scaling.
*   **Limit:** Increased operational complexity.

---

# Detailed Error Handling Matrix

To ensure the system is "engineering-grade," the application must map internal errors to meaningful HTTP responses.

| Error Scenario | Internal Exception | HTTP Status | Client Action |
| :--- | :--- | :--- | :--- |
| **Database Locked** | `sqlite3.OperationalError` | 503 Service Unavailable | Retry with exponential backoff |
| **Duplicate Key** | `sqlite3.IntegrityError` | 409 Conflict | Use the `run_id` provided in the response |
| **LLM Timeout** | `asyncio.TimeoutError` | 504 Gateway Timeout | Retry with same `idempotency_key` |
| **Invalid Input** | `pydantic.ValidationError` | 422 Unprocessable Entity | Fix request payload |
| **Auth Failure** | `PermissionError` | 401 Unauthorized | Check API keys |
| **Stream Drop** | `ConnectionResetError` | (No status code) | Client should reconnect with same key |

### Implementation of Error Mapping:
```python
from fastapi import Request
from fastapi.responses import JSONResponse

@app.exception_handler(aiosqlite.OperationalError)
async def handle_db_lock(request: Request, exc: aiosqlite.OperationalError):
    if "locked" in str(exc).lower():
        return JSONResponse(
            status_code=503,
            content={"error": "Database is busy. Please retry in a moment.", "retry_after": 1}
        )
    return JSONResponse(status_code=500, content={"error": "Internal Database Error"})

@app.exception_handler(Exception)
async def handle_generic_exception(request: Request, exc: Exception):
    logger.error(f"Unhandled exception: {exc}", exc_info=True)
    return JSONResponse(
        status_code=500,
        content={"error": "An unexpected error occurred."}
    )
```

---

# Final Verification Checklist

Before deploying the patch, verify the following:

1.  [ ] **WAL Mode:** Run `PRAGMA journal_mode;` and confirm it returns `wal`.
2.  [ ] **Idempotency:** Send 5 identical requests with the same key; confirm only 1 row exists in `runs`.
3.  [ ] **Concurrency:** Run a script that spawns 20 concurrent `create_run` calls; confirm 0 `Database is locked` errors.
4.  [ ] **Streaming:** Disconnect a client mid-stream; confirm the server logs `Stream cancelled` and the status updates to `cancelled`.
5.  [ ] **Metadata:** Verify that complex nested JSON in the `metadata` field is correctly serialized and retrieved.
6.  [ ] **Logs:** Confirm that every request generates a unique `request_id` in the logs.
7.  [ ] **Timeouts:** Verify that `busy_timeout` is actually preventing immediate failures on write contention.
8.  [ ] **Index Performance:** Run `EXPLAIN QUERY PLAN` on the `idempotency_key` lookup to ensure it uses the index.
9.  [ ] **Memory Map:** Confirm `mmap_size` is active (useful for large benchmark history).
10. [ ] **Pydantic:** Verify that a request with a negative `temperature` returns a 422 error.
11. [ ] **Rollback:** Ensure the migration script can be run in reverse or that the old schema is still compatible.
12. [ ] **Agent Integration:** Confirm the Agent client correctly handles the `200 OK` response when it receives a pre-existing `run_id`.