# Root Cause Hypotheses

1. **Missing Idempotency in `create_run`**: The most likely driver of duplicate benchmark rows is a non-atomic check-then-insert pattern. If `create_run` first queries for an existing run ID or timestamp, then issues an `INSERT`, concurrent requests (especially from proxy retries or OpenWebUI polling) can pass the check simultaneously. SQLite's row-level locking only activates at commit time, so both transactions succeed, producing duplicates.

2. **Async/Sync Boundary Mismatch with SQLite**: FastAPI runs on an asyncio event loop, but standard `sqlite3` is synchronous and thread-hostile. If the codebase uses synchronous SQLite calls inside async routes without `run_in_executor` or `aiosqlite`, the event loop can be blocked. Under concurrent load, multiple coroutines may interleave DB operations, causing partial commits, race conditions on auto-increment sequences, or duplicate inserts when connection state is shared incorrectly across tasks.

3. **Streaming Generator Lifecycle Bugs**: Streaming endpoints often yield chunks while simultaneously calling `add_event` or `finish_llm_request`. If the generator is interrupted (client disconnect, proxy timeout, or network blip), the cleanup/finalization logic may not execute, or may execute multiple times due to retry middleware. This can cause `finish_llm_request` to be called twice for the same logical run, or `add_event` to emit duplicate sequence numbers.

4. **Proxy/Client Retry Amplification**: The proxy handling OpenWebUI, benchmark runners, and agent clients likely implements automatic retries on HTTP 5xx or timeouts. Without idempotency keys or `Idempotency-Key` headers, a single logical benchmark submission can be retried 2-3 times. If the backend lacks conflict resolution, each retry creates a new row.

5. **SQLite Concurrency Configuration**: Default SQLite settings (`journal_mode=DELETE`, `synchronous=FULL`) are conservative but can cause `database is locked` errors under concurrent writes. If the application catches these errors and retries blindly without idempotency, duplicates emerge. Alternatively, if `WAL` mode is enabled but `busy_timeout` is too low, transactions may abort and retry, again without deduplication.

# Evidence To Collect

1. **Database Schema & Constraints Audit**:
   ```sql
   -- Check existing constraints
   PRAGMA table_info(runs);
   PRAGMA index_list(runs);
   PRAGMA index_info(idx_name);
   -- Identify duplicates
   SELECT run_id, COUNT(*) as cnt FROM runs GROUP BY run_id HAVING cnt > 1;
   SELECT run_id, event_type, COUNT(*) FROM events GROUP BY run_id, event_type HAVING COUNT(*) > 1;
   ```

2. **SQLite Runtime Configuration**:
   ```sql
   PRAGMA journal_mode;
   PRAGMA synchronous;
   PRAGMA busy_timeout;
   PRAGMA foreign_keys;
   PRAGMA wal_checkpoint(PASSIVE);
   ```
   Verify if WAL mode is active, what the busy timeout is, and whether foreign keys are enforced.

3. **Application Logs & Tracing**:
   - Collect correlation IDs or request IDs across the proxy, FastAPI, and DB layer.
   - Look for `sqlite3.OperationalError: database is locked` or `IntegrityError` traces.
   - Check for repeated `POST /runs` or `POST /events` with identical payloads within <500ms windows.
   - Verify if `finish_llm_request` is logged multiple times per `run_id`.

4. **Code Review Artifacts**:
   - Inspect `create_run`, `add_event`, `finish_llm_request` for transaction boundaries (`BEGIN`/`COMMIT`/`ROLLBACK`).
   - Check if UUIDs are generated client-side or server-side.
   - Verify if `aiosqlite` or synchronous `sqlite3` is used, and how connections are scoped (per-request vs. global).
   - Look for missing `ON CONFLICT` clauses or idempotency key validation.

5. **Network/Proxy Metrics**:
   - Count retry rates per endpoint.
   - Measure time-to-first-byte vs. time-to-last-byte for streaming routes.
   - Check if the proxy implements `Retry-After` or idempotency header forwarding.

# Patch Plan

1. **Enforce Idempotency at the API Layer**:
   - Require an `Idempotency-Key` header for `create_run` and `create_llm_request`.
   - Store processed keys in a short-lived cache (Redis or in-memory TTL dict) keyed by `(endpoint, idempotency_key)`.
   - If a key exists and the request is within the TTL, return the cached response without hitting the DB.

2. **Atomic Insert with Conflict Resolution**:
   - Replace check-then-insert with `INSERT OR IGNORE` or `ON CONFLICT DO NOTHING`.
   - Example Python pattern using `aiosqlite`:
     ```python
     async def create_run(db: aiosqlite.Connection, run_data: dict):
         async with db.execute(
             "INSERT OR IGNORE INTO runs (run_id, benchmark_id, status, created_at) VALUES (?, ?, ?, ?)",
             (run_data["run_id"], run_data["benchmark_id"], "pending", run_data["created_at"])
         ) as cursor:
             if cursor.rowcount == 0:
                 # Conflict detected, fetch existing
                 async with db.execute("SELECT * FROM runs WHERE run_id = ?", (run_data["run_id"],)) as cur:
                     existing = await cur.fetchone()
                 return existing
             await db.commit()
             return run_data
     ```

3. **Transaction Scoping & Async Safety**:
   - Wrap all DB mutations in explicit `BEGIN`/`COMMIT` blocks.
   - Use `aiosqlite` with per-request connection pooling or a connection factory that yields isolated connections.
   - Never share a single `sqlite3.Connection` across async tasks. Use `asyncio.Lock` per DB file if sharing is unavoidable, but prefer connection pooling.

4. **State Machine Validation**:
   - Enforce valid transitions: `pending -> running -> completed/failed`.
   - Add `CHECK` constraints or application-level guards in `finish_llm_request` to prevent double-finalization.
   - Example guard:
     ```python
     if current_status not in ("pending", "running"):
         raise ValueError(f"Invalid state transition from {current_status}")
     ```

5. **Proxy/Client Coordination**:
   - Return `409 Conflict` with a `Retry-After` header on idempotency collisions.
   - Document idempotency requirements for OpenWebUI and agent clients.
   - Implement exponential backoff with jitter on the client side.

6. **Python-Level Guardrails**:
   - Use context managers for DB connections to guarantee cleanup.
   - Validate all inputs with Pydantic before DB interaction.
   - Log all `IntegrityError` and `OperationalError` with correlation IDs.
   - Set `sqlite3.isolation_level = None` if using manual transaction control, or rely on `aiosqlite`'s default autocommit behavior with explicit `commit()` calls.

# SQLite Constraints And Indexes

1. **Schema Hardening**:
   ```sql
   -- Enable safety features
   PRAGMA journal_mode = WAL;
   PRAGMA synchronous = NORMAL;
   PRAGMA busy_timeout = 5000;
   PRAGMA foreign_keys = ON;

   -- Runs table with unique constraint
   CREATE TABLE IF NOT EXISTS runs (
       run_id TEXT PRIMARY KEY,
       benchmark_id TEXT NOT NULL,
       status TEXT NOT NULL CHECK(status IN ('pending', 'running', 'completed', 'failed')),
       created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
       updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
       metadata TEXT
   );

   -- Events table with composite unique key
   CREATE TABLE IF NOT EXISTS events (
       event_id INTEGER PRIMARY KEY AUTOINCREMENT,
       run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE,
       event_type TEXT NOT NULL,
       payload TEXT,
       sequence INTEGER NOT NULL,
       created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
       UNIQUE(run_id, sequence)
   );
   ```

2. **Index Strategy**:
   ```sql
   -- Accelerate lookups by benchmark and status
   CREATE INDEX idx_runs_benchmark_status ON runs(benchmark_id, status);
   CREATE INDEX idx_runs_created_at ON runs(created_at DESC);

   -- Accelerate event retrieval per run
   CREATE INDEX idx_events_run_id ON events(run_id);
   CREATE INDEX idx_events_run_sequence ON events(run_id, sequence);
   ```

3. **Conflict Resolution Patterns**:
   - Use `INSERT OR IGNORE` for idempotent creation.
   - Use `ON CONFLICT DO UPDATE SET updated_at = excluded.updated_at` for upserts where state changes are expected.
   - Avoid `REPLACE` as it deletes and re-inserts, breaking foreign key references and auto-increment sequences.

4. **Write Amplification Mitigation**:
   - Batch `add_event` calls when possible (e.g., accumulate 10 events, then insert in a single transaction).
   - Use `PRAGMA wal_autocheckpoint = 1000` to balance WAL file growth and checkpoint frequency.
   - Monitor `PRAGMA wal_checkpoint(PASSIVE)` during high-throughput periods.

# Streaming Edge Cases

1. **Client Disconnect Mid-Stream**:
   - FastAPI's `StreamingResponse` may not trigger cleanup if the client drops the connection abruptly.
   - **Fix**: Use `asyncio.TaskGroup` or `try/finally` blocks to guarantee `finish_llm_request` runs. Wrap the generator in a context manager that catches `GeneratorExit` or `ConnectionClosed`.
   - Example:
     ```python
     async def stream_benchmark(run_id: str):
         await start_run(run_id)
         try:
             async for chunk in generate_chunks():
                 yield chunk
                 await add_event(run_id, "chunk", chunk)
         except (GeneratorExit, asyncio.CancelledError):
             await finish_llm_request(run_id, status="interrupted")
             raise
         else:
             await finish_llm_request(run_id, status="completed")
     ```

2. **Duplicate Event Emission**:
   - Streaming generators may yield the same logical event multiple times due to retry logic or buffer flushing.
   - **Fix**: Enforce `UNIQUE(run_id, sequence)` at the DB level. In Python, maintain a monotonic sequence counter per run. Use `INSERT OR IGNORE` for events.

3. **Backpressure & Event Loop Blocking**:
   - Synchronous SQLite writes inside an async generator will block the event loop, causing timeouts for other requests.
   - **Fix**: Use `aiosqlite` exclusively. If batching events, use `asyncio.gather` with bounded concurrency. Set `PRAGMA busy_timeout` to prevent indefinite blocking.

4. **Partial Finalization**:
   - If `finish_llm_request` fails after some events are written, the run may remain in `running` state indefinitely.
   - **Fix**: Implement a background reconciliation job that scans for `running` runs older than a threshold (e.g., 10 minutes) and transitions them to `failed` or `completed` based on event history.

5. **Idempotent Streaming Chunks**:
   - Clients may request the same stream twice due to proxy retries.
   - **Fix**: Cache the final response for completed runs. For in-progress streams, return `409 Conflict` with a pointer to the existing run ID.

# Test Plan

1. **Concurrent `create_run` Idempotency**: Fire 50 concurrent requests with the same `run_id` and `Idempotency-Key`. Verify exactly 1 row inserted, 49 return cached/409 responses.
2. **SQLite WAL Concurrency**: Run 100 parallel `add_event` calls across 10 different runs. Verify no `database is locked` errors, all events committed, WAL file size remains bounded.
3. **Streaming Client Disconnect**: Simulate client drop at 50% stream completion. Verify `finish_llm_request` is called exactly once with `status=interrupted`, no duplicate finalization rows.
4. **State Transition Guard**: Attempt `finish_llm_request` on a `completed` run. Verify `IntegrityError` or application-level rejection, DB state unchanged.
5. **Proxy Retry Deduplication**: Configure proxy to retry on 503. Send request that triggers 503, then succeeds. Verify single DB row, idempotency key cached.
6. **Event Sequence Uniqueness**: Insert events with duplicate `(run_id, sequence)`. Verify `UNIQUE` constraint triggers, `INSERT OR IGNORE` drops duplicates, no crash.
7. **Async/Sync Boundary Safety**: Run FastAPI with `uvicorn --workers 2`. Verify no shared connection state leaks, each worker uses isolated `aiosqlite` connections.
8. **Transaction Rollback on Failure**: Simulate DB constraint violation mid-transaction. Verify `ROLLBACK` executes, no partial writes, connection returned to pool.
9. **Background Reconciliation**: Create a `running` run, pause event emission for 15 minutes. Verify reconciliation job transitions it to `failed` and logs alert.
10. **High-Throughput Event Batching**: Generate 10,000 events across 100 runs. Verify batch inserts complete within SLA, no event loop starvation, CPU/memory stable.
11. **Idempotency Key TTL Expiry**: Insert key, wait past TTL, retry request. Verify new row created, old key purged, no stale cache hits.
12. **Foreign Key Cascade Delete**: Delete a `run_id`. Verify all associated `events` are cascaded, no orphaned rows, constraint enforced.
13. **PRAGMA Configuration Validation**: On startup, verify `journal_mode=WAL`, `synchronous=NORMAL`, `busy_timeout=5000`. Fail fast if mismatched.
14. **Payload Size Limits**: Send oversized event payloads. Verify truncation or rejection before DB write, no SQLite page size violations.

# Rollback Plan

1. **Feature Flag Idempotency**: Wrap idempotency key validation behind a config flag (`ENABLE_IDEMPOTENCY=true`). If issues arise, disable to revert to previous behavior while monitoring.
2. **Database Migration Rollback**:
   ```sql
   -- Rollback unique constraints if needed
   DROP INDEX IF EXISTS idx_events_run_sequence;
   ALTER TABLE events DROP CONSTRAINT IF EXISTS unique_run_sequence;
   -- Note: SQLite lacks ALTER TABLE DROP CONSTRAINT; use table recreation if strict rollback needed
   ```
   Maintain versioned migration scripts with forward/backward compatibility.
3. **Code Revert Strategy**: Keep the patch in a separate branch. Use git revert or deployment pipeline rollback. Ensure no breaking API changes are deployed without versioning.
4. **Duplicate Data Cleanup**:
   ```sql
   -- Safe deduplication script
   DELETE FROM runs WHERE rowid NOT IN (
       SELECT MIN(rowid) FROM runs GROUP BY run_id
   );
   DELETE FROM events WHERE rowid NOT IN (
       SELECT MIN(rowid) FROM events GROUP BY run_id, sequence
   );
   ```
   Run during maintenance window with backup.
5. **Monitoring & Alerts**:
   - Alert on `IntegrityError` rate > 1/min.
   - Alert on duplicate `run_id` detection.
   - Track `PRAGMA wal_checkpoint` failures.
   - Log idempotency cache hit/miss ratios.
6. **Communication**: Notify OpenWebUI and agent client maintainers of idempotency requirements. Provide retry guidelines and expected response codes.

# Decision Summary

The duplicate benchmark rows stem from a combination of missing idempotency, non-atomic inserts, async/sync SQLite boundary issues, and streaming lifecycle gaps. The fix enforces idempotency via request keys and `INSERT OR IGNORE`, hardens the schema with `UNIQUE` constraints and WAL mode tuning, wraps all DB operations in explicit async-safe transactions, and adds state machine guards to prevent double-finalization. Streaming edge cases are handled with `try/finally` cleanup, monotonic sequence counters, and background reconciliation for stuck runs. The test plan covers concurrency, idempotency, constraint enforcement, async safety, and failure recovery. Rollback procedures include feature flags, migration reversals, and safe deduplication scripts. This approach balances home-lab simplicity with production-grade reliability, ensuring data integrity under concurrent load while maintaining operational safety. Deploy behind a feature flag, monitor idempotency hit rates and WAL checkpoint health, and iterate based on telemetry.

# Advanced Concurrency & Async Safety Patterns

FastAPI's async event loop and SQLite's synchronous C-binding require explicit isolation boundaries. Sharing a single `sqlite3.Connection` across coroutines violates SQLite's thread-affinity rules and will trigger `sqlite3.ProgrammingError: Cannot operate on a closed database` or silent data corruption under load. The following patterns enforce strict async safety:

1. **Per-Request Connection Factory with `aiosqlite`**:
   ```python
   import aiosqlite
   from contextlib import asynccontextmanager

   @asynccontextmanager
   async def get_db_connection(db_path: str):
       conn = await aiosqlite.connect(db_path)
       conn.row_factory = aiosqlite.Row
       try:
           yield conn
       finally:
           await conn.close()
   ```
   Inject this via FastAPI's `Depends()` to guarantee lifecycle alignment with the HTTP request. Never cache connections globally unless using a dedicated async pool manager.

2. **Asyncio Lock for Schema Mutations**:
   While data writes are safe under WAL mode, `ALTER TABLE` or `PRAGMA` changes require exclusive locks. Wrap schema migrations in a module-level `asyncio.Lock`:
   ```python
   _schema_lock = asyncio.Lock()
   async def run_migration(conn: aiosqlite.Connection, migration_sql: str):
       async with _schema_lock:
           await conn.execute(migration_sql)
           await conn.commit()
   ```

3. **Coroutine Cancellation Safety**:
   FastAPI cancels background tasks if the client disconnects. Use `asyncio.shield()` for critical finalization steps, but ensure they are idempotent:
   ```python
   async def finalize_run(run_id: str, status: str):
       try:
           await finish_llm_request(run_id, status)
       except asyncio.CancelledError:
           # Re-raise after ensuring partial state is cleaned
           await cleanup_partial_events(run_id)
           raise
   ```

4. **Bounded Concurrency for Event Batching**:
   Use `asyncio.Semaphore` to prevent event loop starvation during high-throughput streaming:
   ```python
   _db_semaphore = asyncio.Semaphore(10)
   async def safe_add_event(run_id: str, event_type: str, payload: dict):
       async with _db_semaphore:
           await add_event(run_id, event_type, payload)
   ```

# SQLite Performance & Durability Tuning for Home Labs

Home-lab environments often run on consumer SSDs or even SD cards. SQLite's default settings are conservative but suboptimal for concurrent async workloads. Apply the following tuning matrix:

1. **Journaling & Synchronization**:
   ```sql
   PRAGMA journal_mode = WAL;          -- Enables concurrent readers + single writer
   PRAGMA synchronous = NORMAL;        -- Balances speed vs. crash safety (OS sync on commit)
   PRAGMA wal_autocheckpoint = 1000;   -- Checkpoint every 1000 pages (~64MB)
   PRAGMA cache_size = -64000;         -- 64MB shared cache (negative = KB)
   PRAGMA mmap_size = 268435456;       -- 256MB memory-mapped I/O for faster reads
   ```
   `NORMAL` synchronous mode is safe for home labs; `FULL` adds unnecessary fsync overhead, while `OFF` risks corruption on power loss.

2. **File System & Permissions**:
   - Mount the SQLite directory on a filesystem with reliable journaling (ext4, APFS, NTFS with proper alignment).
   - Set `chmod 660` on the `.db` file and ensure the FastAPI process runs under a dedicated user.
   - Disable OS-level antivirus real-time scanning on the DB directory to prevent file locking interference.

3. **Crash Recovery & Backup Strategy**:
   ```sql
   -- Point-in-time recovery via WAL backup
   PRAGMA wal_checkpoint(TRUNCATE);
   -- Automated backup hook
   VACUUM INTO 'backup_$(date +%Y%m%d_%H%M%S).db';
   ```
   Schedule `wal_checkpoint(TRUNCATE)` during low-traffic windows. Use `sqlite3 .backup` or `rqlite`-style replication if multi-node redundancy is required. Never copy the `.db` file while the process is running; always checkpoint first.

# Observability, Telemetry & Alerting

Engineering-grade reliability requires visibility into the async/SQLite boundary. Implement the following telemetry stack:

1. **Structured Logging with Correlation IDs**:
   ```python
   import logging
   import uuid
   from fastapi import Request

   logger = logging.getLogger("benchmark_service")

   async def log_middleware(request: Request, call_next):
       request.state.correlation_id = str(uuid.uuid4())
       logger.info("request_start", extra={"correlation_id": request.state.correlation_id, "path": request.url.path})
       response = await call_next(request)
       logger.info("request_end", extra={"correlation_id": request.state.correlation_id, "status": response.status_code})
       return response
   ```

2. **SQLite-Specific Metrics**:
   - Track `PRAGMA page_count`, `PRAGMA freelist_count`, and `PRAGMA wal_checkpoint` return codes.
   - Monitor `busy_timeout` hits via custom counters:
     ```python
     from prometheus_client import Counter
     db_lock_counter = Counter("sqlite_busy_timeout_hits", "Count of busy timeout retries")
     ```
   - Alert when `IntegrityError` rate exceeds 0.1% of total requests.

3. **Distributed Tracing**:
   Propagate `traceparent` headers from OpenTelemetry across the proxy, FastAPI, and DB layer. Instrument `aiosqlite` with custom spans to measure query latency, lock wait times, and transaction commit duration. Set SLOs: p95 query latency < 50ms, p99 transaction commit < 100ms.

# Extended Test Scenarios & Automation

The initial 12 acceptance tests must be integrated into a CI/CD pipeline with deterministic reproducibility. Expand coverage with the following automation strategy:

1. **Tooling Stack**:
   - `pytest-asyncio` for async route testing
   - `locust` for concurrent load simulation
   - `sqlite-utils` for schema validation and data integrity checks
   - `docker-compose` for isolated test environments with ephemeral volumes

2. **Chaos & Failure Injection**:
   - Simulate network partitions between proxy and FastAPI using `tc` (traffic control) or `chaos-mesh`.
   - Force SQLite `database is locked` by spawning a long-running `BEGIN IMMEDIATE` transaction in a background thread.
   - Verify graceful degradation: requests should queue, retry with backoff, or return `503 Service Unavailable` with `Retry-After`.

3. **Data Consistency Assertions**:
   ```python
   def test_no_orphaned_events(db_conn):
       orphans = db_conn.execute(
           "SELECT COUNT(*) FROM events WHERE run_id NOT IN (SELECT run_id FROM runs)"
       ).fetchone()[0]
       assert orphans == 0, "Foreign key cascade failed or constraint violated"
   ```
   Run this after every load test. Integrate with `pytest` fixtures that seed deterministic benchmark payloads.

4. **Performance Regression Guards**:
   - Baseline p95 latency for `create_run` and `add_event` under 50 concurrent users.
   - Fail CI if latency increases >15% or if WAL file size exceeds 500MB during test execution.
   - Use `pytest-benchmark` to track Python-level overhead from async context managers and lock acquisitions.

# API Contract & Client Integration Guidelines

OpenWebUI, benchmark runners, and agent clients must adhere to strict idempotency and retry contracts to prevent duplicate rows:

1. **Idempotency Header Specification**:
   - `Idempotency-Key`: UUIDv4 generated client-side per logical benchmark submission.
   - TTL: 24 hours. Keys are stored in an in-memory LRU cache with disk fallback.
   - Response: `200 OK` with cached payload if key exists; `409 Conflict` with `Retry-After: 5` if key is locked but not yet committed.

2. **Retry Policy**:
   - Exponential backoff: `base=1s, max=30s, jitter=0.5`
   - Retry only on `5xx` or `408 Request Timeout`. Never retry `409` or `422`.
   - Implement `Transfer-Encoding: chunked` awareness: clients must not retry mid-stream unless the server explicitly signals resumability.

3. **Error Response Schema**:
   ```json
   {
     "error": "idempotency_conflict",
     "message": "Request already processed or in-flight",
     "run_id": "abc-123",
     "retry_after": 5,
     "trace_id": "otel-xyz"
   }
   ```
   Document this in OpenAPI/Swagger specs. Provide SDK snippets for Python, Go, and Bash to standardize client behavior.

# Final Engineering Validation Checklist

Before promoting to production, verify the following:

- [ ] All `INSERT` statements use `OR IGNORE` or `ON CONFLICT DO NOTHING` for idempotent creation.
- [ ] `aiosqlite` connections are scoped per-request; no global connection state leaks.
- [ ] `PRAGMA journal_mode=WAL` and `synchronous=NORMAL` are enforced at startup.
- [ ] Streaming generators wrap DB calls in `try/finally` with explicit `finish_llm_request` invocation.
- [ ] Idempotency cache TTL matches business requirements; stale keys are purged automatically.
- [ ] Foreign key constraints are enabled (`PRAGMA foreign_keys=ON`) and cascade deletes are tested.
- [ ] Load tests confirm p95 latency < 50ms under 100 concurrent users with zero duplicate rows.
- [ ] Monitoring alerts trigger on `IntegrityError` spikes, WAL checkpoint failures, and busy timeout exhaustion.
- [ ] Backup scripts checkpoint WAL before copying; restore procedures are documented and tested quarterly.
- [ ] Client documentation includes idempotency key generation, retry policies, and error handling examples.
- [ ] Rollback procedures are automated; feature flags can disable idempotency enforcement without downtime.
- [ ] Security audit confirms no SQL injection vectors; all parameters are bound via `?` placeholders.

This architecture transforms a fragile, race-prone SQLite integration into a deterministic, observable, and resilient data pipeline. By enforcing idempotency at the API boundary, hardening schema constraints, isolating async execution contexts, and instrumenting every failure mode, the service will withstand concurrent proxy traffic, streaming interruptions, and home-lab hardware variability without producing duplicate benchmark rows. Deploy incrementally, validate telemetry, and iterate based on observed lock contention and cache hit ratios.