# Situation

The current operational state of the home-lab environment is centered on the evaluation and benchmarking of a specific large language model (LLM) configuration: the `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` [E6]. The primary objective is to transition from "toy tests" to a rigorous, representative benchmark suite that accurately reflects real-world usage patterns, specifically focusing on long-form reasoning and high-context window performance [E7].

The project is currently facing several technical hurdles, most notably a degradation in prompt evaluation speed when utilizing a massive 262,144 token context window [E1]. Additionally, the hardware environment—specifically the R9700 GPU—is operating near its memory ceiling, with the model consuming approximately 95% of available VRAM after warmup [E2]. 

From a governance perspective, the project is operating under strict privacy constraints. Because the benchmark involves WorkDash, which processes sensitive email and Teams content, there is a mandate to keep raw source data within the lab and utilize redacted previews for database storage [E5, E9, E13]. The goal is to produce a public-facing aggregate report while maintaining absolute confidentiality of the underlying private synthetic data [E13].

The project is currently in the "Design and Validation" phase, moving toward a "Production Benchmark" phase. This requires the implementation of a comprehensive data capture strategy [E8], a refined reporting dashboard [E3, E10], and a diversified traffic simulation model that includes various agentic and interface types [E11].

# Evidence Timeline

The following timeline reconstructs the events of the synthetic week, detailing the progression from initial performance observations to the establishment of privacy and reporting protocols.

**Monday**
*   **08:14** – Alex identifies a performance bottleneck: prompt evaluation in `llama.cpp` is slow following the enablement of a 262,144 token context [E1].
*   **09:02** – Priya provides hardware telemetry for the R9700, noting a total VRAM of 32,624 MiB and a model load footprint of 31,016 MiB post-warmup [E2].
*   **10:30** – A requirement is established via Teams for the benchmark dashboard to include specific visual evidence: throughput, reliability, latency, MTP (Multi-Token Prediction) acceptance, and output quality [E3].
*   **13:42** – Server logs are analyzed, revealing that the current MTP profile yields a `draft_n_accepted / draft_n` ratio typically between 0.75 and 0.90 [E4].

**Tuesday**
*   **07:55** – A project note is issued regarding data sensitivity, stating that benchmark outputs must remain private due to WorkDash's processing of email and Teams content [E5].
*   **11:12** – The active model configuration is documented as `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` with a reasoning budget set to 8,192 [E6].
*   **14:40** – User feedback indicates that current "toy tests" are insufficient. The requirement is shifted toward real tasks that necessitate reasoning and produce long final outputs [E7].

**Wednesday**
*   **09:25** – A maintenance standard is defined for every benchmark run. Mandatory capture fields include: model path, quantization, context size, backend, MTP settings, reasoning budget, prompt tokens, completion tokens, duration, and GPU memory [E8].
*   **10:01** – A privacy architecture is defined: raw messages are to be stored as local artifacts only, while SQLite will hold redacted previews [E9].
*   **15:16** – The dashboard requirements are expanded to include a model leaderboard, failure drilldown, long-output histogram, context-fit table, and a comparative analysis of MTP vs. non-MTP performance [E10].

**Thursday**
*   **08:08** – Traffic type definitions are established. The benchmark must compare OpenWebUI, AgentSSH, Cline, opencode, and WorkDash, though these should be treated as traffic categories rather than requiring deep app-specific integrations [E11].
*   **12:34** – A reliability protocol is implemented: benchmark tasks must terminate upon failure unless the `--keep-going` flag is explicitly provided [E12].

**Friday**
*   **09:00** – Planning for public dissemination is finalized. The public writeup will describe aggregate model behavior, but private source data is strictly forbidden from leaving the lab [E13].
*   **16:50** – A server note highlights the operational challenge of long outputs, which can run for several minutes. This necessitates the implementation of progress tracking, artifact saving, and partial failure reporting [E14].

# Technical Findings

### 1. Memory and Hardware Constraints
The hardware environment is characterized by the R9700 GPU, which possesses 32,624 MiB of VRAM [E2]. The current model, `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`, consumes 31,016 MiB after the warmup phase [E2]. 

**Analysis of VRAM Headroom:**
The remaining available VRAM is approximately 1,608 MiB (32,624 - 31,016). This represents a very slim margin (approx. 4.9% of total VRAM). Given that the project is attempting to utilize a 262,144 token context [E1], this limited headroom is a critical risk factor. KV cache growth for such a large context window typically requires significant memory; the fact that the model loads but prompt evaluation is slow suggests that the system may be swapping or struggling with memory bandwidth/allocation at the edge of the VRAM limit.

### 2. Model Configuration and Performance
The active model is a balanced MTP (Multi-Token Prediction) variant of Qwen3.6-35B [E6]. 

*   **Reasoning Budget:** The model is configured with a `--reasoning-budget` of 8,192 [E6]. This budget is intended to support the "long final outputs and reasoning" requested by users [E7].
*   **MTP Efficiency:** The MTP profile is performing well, with an acceptance rate (`draft_n_accepted / draft_n`) of 0.75 to 0.90 [E4]. This indicates that the draft model is highly accurate, significantly reducing the number of verification steps required by the main model and theoretically increasing throughput.
*   **Context Bottleneck:** Despite the MTP efficiency, the enablement of the 262,144 token context has led to slow prompt evaluation [E1]. This suggests that the bottleneck is not in the token generation phase (where MTP helps) but in the initial processing of the massive prompt (the "prefill" phase).

### 3. Execution and Reliability
The current execution logic is designed to be fail-fast [E12]. The use of the `--keep-going` flag allows for optional persistence, but the default is to stop after failure. This is a necessary safeguard given that long outputs can run for "many minutes" [E14], and allowing a failing process to continue could waste significant GPU resources.

# Privacy Findings

The project handles synthetic but private data, necessitating a multi-tiered privacy strategy to prevent the leakage of sensitive email and Teams content [E5].

### 1. Data Segregation and Storage
The architecture follows a "Local Artifact vs. Redacted Database" model [E9]:
*   **Local Artifacts:** Raw messages are stored exclusively as local files. These are the "ground truth" but are isolated from the primary database to prevent accidental exposure.
*   **SQLite Redacted Previews:** The SQLite database, used for tracking and analysis, contains only redacted versions of the messages. This allows for metadata analysis and progress tracking without exposing the actual private content of the communications.

### 2. Access and Publication Control
There is a strict boundary between "Lab Data" and "Public Data" [E13]:
*   **Internal/Lab:** All private source data and raw benchmark outputs must remain within the lab environment.
*   **External/Public:** Only aggregate model behavior (e.g., average latency, overall reliability percentages, general throughput trends) may be published. No individual samples or raw logs from the private dataset are permitted in public writeups.

### 3. Risk of Content Processing
The sensitivity is heightened because WorkDash is designed to process email and Teams content [E5]. Any benchmark output that captures the model's reasoning process may inadvertently leak snippets of the input data if not properly scrubbed, reinforcing the need for the "private" status of all benchmark outputs [E5].

# Benchmark Design Requirements

Based on the requirements gathered throughout the week, the benchmark suite must be redesigned to move beyond "toy tests" [E7].

### 1. Task Selection and Nature
*   **Real-World Complexity:** Tasks must be representative of actual work, requiring the model to perform complex reasoning [E7].
*   **Output Length:** The benchmark must specifically target "long final outputs" [E7].
*   **Duration Handling:** Because these outputs can take several minutes, the system must implement:
    *   Real-time progress indicators [E14].
    *   Intermediate artifact saving [E14].
    *   Partial failure reporting (identifying exactly where a long output failed) [E14].

### 2. Traffic Simulation
The benchmark must simulate five distinct traffic types to ensure the model's versatility across different interfaces [E11]:
1.  **OpenWebUI**
2.  **AgentSSH**
3.  **Cline**
4.  **opencode**
5.  **WorkDash**

*Note: These are to be treated as categories of traffic; the benchmark does not require full API integration with these specific applications [E11].*

### 3. Execution Logic
*   **Failure Handling:** Default behavior is to stop on failure [E12].
*   **Override:** The `--keep-going` flag must be supported for exhaustive testing runs [E12].

# Reporting Requirements

The reporting framework is divided into visual dashboard elements and raw data capture.

### 1. Dashboard Visualizations
The benchmark dashboard must provide the following views [E3, E10]:
*   **Performance Metrics:**
    *   Throughput (tokens/sec).
    *   Latency (time to first token and total time).
    *   MTP Acceptance Rate (visualizing the 0.75–0.90 range).
*   **Quality and Reliability:**
    *   Reliability scores (success vs. failure rates).
    *   Output quality assessments.
    *   Failure Drilldown (detailed analysis of why tasks failed).
*   **Distribution and Fit:**
    *   Long-output histogram (distribution of output lengths).
    *   Context-fit table (how the model handles various context sizes up to 262k).
*   **Comparative Analysis:**
    *   Model Leaderboard (comparing different versions/quants).
    *   MTP vs. Non-MTP comparison (to quantify the gain from the MTP profile).

### 2. Mandatory Data Capture (Per Run)
Every single run must log the following parameters to ensure reproducibility [E8]:
*   **Model Metadata:** Model path, quantization level.
*   **Configuration:** Context window size, backend used, MTP settings, reasoning budget.
*   **Execution Metrics:** Prompt tokens, completion tokens, total duration.
*   **Hardware Metrics:** GPU memory usage.

# Risks

### 1. Hardware Instability (VRAM Exhaustion)
The R9700 is operating with only ~1.6 GB of free VRAM [E2]. Any increase in context window usage or a slight increase in the reasoning budget could trigger an Out-of-Memory (OOM) error. The slow prompt evaluation [E1] is a leading indicator that the system is under extreme memory pressure.

### 2. Data Leakage
The use of private email and Teams content [E5] creates a high risk of PII (Personally Identifiable Information) leakage. If the "aggregate" public report [E13] is not strictly audited, specific examples from the "long outputs" [E7] could potentially reveal private lab data.

### 3. Timeout and Resource Hanging
Since long outputs can run for many minutes [E14], there is a risk of "zombie" processes or timeouts in the orchestration layer. Without the progress and partial failure reporting [E14], it will be difficult to distinguish between a slow-running successful task and a hung process.

### 4. Benchmark Validity
The transition from "toy tests" to "real tasks" [E7] may reveal that the current MTP acceptance rates (0.75–0.90) [E4] drop significantly when faced with complex reasoning. The current performance metrics may be overly optimistic if they were derived from simpler tests.

# Recommended Next Actions

| Owner | Action | Evidence | Priority | Validation Method |
| :--- | :--- | :--- | :--- | :--- |
| **DevOps** | Implement progress tracking and partial failure reporting for long-running tasks. | [E14] | High | Verify that a 5-minute output shows incremental progress in logs. |
| **ML Eng** | Conduct a comparative run: MTP enabled vs. MTP disabled for the same "real task" set. | [E10] | High | Compare throughput and latency in the dashboard. |
| **Privacy Officer** | Audit the SQLite redaction script to ensure no raw email/Teams content is stored. | [E9] | Critical | Query SQLite for known synthetic PII and confirm it is redacted. |
| **Perf Eng** | Profile the 262k context prompt evaluation to identify the cause of the slowdown. | [E1] | Medium | Compare prefill time vs. token generation time. |
| **QA** | Develop a set of "Real Tasks" involving long-form reasoning to replace toy tests. | [E7] | High | Peer review of task prompts for "real-world" complexity. |
| **Data Eng** | Build the mandatory data capture pipeline (Model path, Quant, VRAM, etc.). | [E8] | Medium | Ensure every run generates a complete metadata JSON/CSV. |
| **UI/UX** | Create the dashboard views for the long-output histogram and context-fit table. | [E10] | Medium | Visual confirmation of the five required dashboard components. |
| **Lead** | Establish the "Aggregate Only" filter for the public writeup. | [E13] | Critical | Review the draft public report for any specific source data. |

# Source-Backed Claims

*   **Claim:** The model is experiencing performance issues with very large contexts.
    *   **Source:** [E1] (Alex reports slow prompt evaluation at 262,144 tokens).
*   **Claim:** The GPU is nearly at full capacity.
    *   **Source:** [E2] (R9700 has 32,624 MiB; model uses 31,016 MiB).
*   **Claim:** The MTP (Multi-Token Prediction) is currently effective.
    *   **Source:** [E4] (Acceptance rate is 0.75 to 0.90).
*   **Claim:** Current testing methodology is insufficient.
    *   **Source:** [E7] (User complaint that toy tests are not representative).
*   **Claim:** Data must be handled with extreme privacy.
    *   **Source:** [E5, E9, E13] (WorkDash processes email/Teams; raw data stays local; only aggregates are public).
*   **Claim:** The model is configured for high-reasoning tasks.
    *   **Source:** [E6] (Reasoning budget set to 8,192).
*   **Claim:** The benchmark must cover a variety of traffic sources.
    *   **Source:** [E11] (OpenWebUI, AgentSSH, Cline, opencode, WorkDash).
*   **Claim:** Long outputs present a technical challenge for monitoring.
    *   **Source:** [E14] (Outputs run for many minutes; need progress/partial failure reporting).

# Publishable Summary

**Project Overview: LLM Performance Evaluation (Internal Lab)**

This report summarizes the benchmarking of the `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced` model. The evaluation focused on the intersection of high-context window utilization and Multi-Token Prediction (MTP) efficiency.

**Key Performance Indicators:**
*   **MTP Efficiency:** The model demonstrated a strong MTP acceptance rate, consistently falling between 75% and 90%, indicating high efficiency in token drafting and verification.
*   **Context Handling:** Testing was extended to a 262,144 token context window. While functional, a measurable decrease in prompt evaluation speed was observed at this scale.
*   **Hardware Utilization:** The model was deployed on an R9700 GPU, utilizing approximately 31 GB of the available 32 GB VRAM, leaving a narrow margin for operational overhead.

**Methodology Improvements:**
The benchmark suite has been evolved from basic "toy tests" to complex, real-world reasoning tasks. These tasks are designed to produce long-form outputs, requiring the implementation of new progress-tracking and partial-failure reporting mechanisms to ensure reliability over extended generation times.

**Traffic Simulation:**
The evaluation simulated diverse traffic patterns, including various agentic interfaces and web-based UIs, to ensure consistent performance across different interaction modalities.

**Privacy and Governance:**
All testing was conducted using synthetic private data. In accordance with lab security protocols, all raw source data remains isolated within the local environment. This public summary contains only aggregate behavioral data and contains no private source material.