# Thesis

The current landscape of local Large Language Model (LLM) deployment is shifting from a "feasibility" phase to an "optimization and observability" phase. For the home-lab enthusiast, the primary question is no longer simply "Can I run a 35B parameter model on my hardware?" but rather "How does this model perform under production-grade constraints, and how can I telemetry that performance accurately?"

This technical draft explores the intersection of high-parameter open-weight models, Multi-Token Prediction (MTP) draft decoding, and deep reasoning budgets within a constrained 32GB memory environment. By utilizing a local "AI Flight Recorder" to capture every nuance of the inference loop—from proxy metadata to reasoning-step timings—we aim to move beyond anecdotal "it feels fast" reports toward a rigorous, data-driven understanding of local AI capabilities.

The core hypothesis of this benchmark is that the integration of MTP draft decoding and a dedicated reasoning budget (8192) allows a 35B-class model to punch significantly above its weight class in complex reasoning and long-context tasks, provided the inference engine (llama.cpp via Vulkan) is tuned to handle the massive KV cache requirements of a 262,144 context window.

# Hardware And Runtime

The testbed centers on a 32GB-class R9700 server. In the context of local LLM hosting, 32GB represents a critical "tipping point." It is sufficient to host highly quantized 35B models while leaving enough headroom for a substantial KV cache, but it requires disciplined memory management to avoid OOM (Out of Memory) errors during high-context operations.

### Inference Engine: llama.cpp and Vulkan
We are utilizing `llama.cpp` as the primary inference backend. While CUDA remains the gold standard for NVIDIA hardware, the Vulkan backend provides a crucial cross-platform abstraction that allows for high-performance execution on a variety of GPUs. In our configuration, Vulkan serves as the bridge, ensuring that the heavy lifting of matrix multiplication is offloaded efficiently while maintaining compatibility with the server's heterogeneous compute environment.

### The MTP Advantage
A key differentiator in this benchmark is the use of **Multi-Token Prediction (MTP) draft decoding**. Traditional autoregressive decoding predicts one token at a time, which can become a bottleneck as model size increases. MTP allows the model to predict multiple potential future tokens simultaneously. By using these as "drafts" for a verification step, we can significantly increase the tokens-per-second (TPS) throughput without sacrificing the integrity of the output. We are testing how this draft decoding behaves specifically with the `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` model, which was architected with MTP in mind.

### Context and Memory Allocation
The target context window is set to **262,144 tokens**. This is an aggressive target for a 32GB system. To achieve this, we must balance the quantization of the model weights against the memory allocated for the KV cache. The "Balanced" variant of the Qwen model is chosen specifically to find the "sweet spot" where the model retains its cognitive capabilities while remaining small enough to allow the KV cache to breathe.

# Why Thinking Budgets Matter

One of the most significant shifts in recent model releases is the move toward "thinking" or "reasoning" models. These models are designed to generate an internal monologue or a chain-of-thought (CoT) before arriving at a final answer.

### The Reasoning Budget (8192)
We have initialized the server with a `--reasoning-budget 8192`. This parameter defines the maximum number of tokens the model is permitted to spend on "thinking" before it must commit to a final response. 

In a home-lab environment, understanding the "Reasoning Density" is vital. If the budget is too low, the model may truncate its internal logic, leading to "hallucinated" conclusions or shallow answers. If it is too high, the model may enter recursive loops or spend an excessive amount of time on trivial problems. By setting a fixed budget of 8192, we create a controlled environment to measure how the model utilizes its "internal workspace" across different categories of difficulty.

### Telemetry of Thought
The AI Flight Recorder is essential here. By capturing the reasoning tokens separately from the final output, we can calculate a "Reasoning-to-Output Ratio." This allows us to see if the model is actually "thinking" more for harder problems, or if it is simply wasting its budget on easy queries.

# Why Toy Benchmarks Failed

Early iterations of this benchmark were discarded because they relied on "toy" prompts—simple questions like "What is the capital of France?" or "Write a poem about a cat." 

### The Truncation Trap
The primary reason these tests failed was the `max_tokens` setting. Because thinking models generate a significant amount of internal monologue, a standard `max_tokens` limit (e.g., 512 or 1024) often cuts off the model *during* its thinking phase. When a model is interrupted mid-thought, the resulting output is often nonsensical or incomplete.

### The Validity of the Result
If the model is forced to stop before it finishes its reasoning, the benchmark measures the model's ability to be interrupted, not its ability to reason. To get valid data, the `max_tokens` must be large enough to accommodate:
1. The full reasoning budget (8192).
2. The expected length of the final output.
3. A safety buffer for the MTP draft decoding overhead.

By failing to account for this, early tests produced "garbage-in, garbage-out" data that skewed the perceived performance of the Qwen3.6-35B model.

# The Real-World Test Suite

To replace the failed toy benchmarks, we have constructed a multi-dimensional test suite designed to stress-test the model across different cognitive domains. Each test category is designed to push the model's specific strengths and weaknesses.

### 1. Coding and Scripting
*   **Task:** Complex refactoring of a Python script with multiple dependencies, and writing a Rust-based CLI tool.
*   **Goal:** Measure the model's ability to maintain syntax, follow logic across multiple functions, and handle "edge case" error handling.

### 2. RAG (Retrieval-Augmented Generation)
*   **Task:** Providing the model with 10-20 disparate snippets of technical documentation and asking it to synthesize a cohesive installation guide.
*   **Goal:** Test "Needle in a Haystack" performance and the model's ability to ignore irrelevant information in a large context window.

### 3. Agentic Work
*   **Task:** A multi-step task where the model must "plan" a series of actions (e.g., "Research a topic, summarize it, then write a tweet based on the summary").
*   **Goal:** Evaluate the stability of the "Plan-Act-Observe" loop and the model's ability to follow instructions over a long sequence of turns.

### 4. Server-Admin Work
*   **Task:** Generating complex Docker Compose files, Nginx configurations, and Bash scripts for automated backups.
*   **Goal:** Test the model's "knowledge" of infrastructure-as-code and its ability to produce valid, runnable configurations.

### 5. Chat Assistant Behavior
*   **Task:** Nuanced persona adoption and handling of ambiguous queries.
*   **Goal:** Measure "vibe" and conversational flow—how well does the model stick to a persona without drifting into generic AI-speak?

### 6. Creative Writing
*   **Task:** Long-form narrative prose with specific stylistic constraints (e.g., "Write in the style of a hard-boiled noir detective").
*   **Goal:** Evaluate stylistic consistency and the avoidance of repetitive "AI-isms."

### 7. Long-Output Reliability
*   **Task:** Generating a 2,000-word technical whitepaper on a fictional technology.
*   **Goal:** Check for "drift"—does the model lose the thread of the argument halfway through? Does it start repeating itself?

### 8. Context Fit
*   **Task:** Feeding the model a massive log file (near the 262k limit) and asking for a specific error trace.
*   **Goal:** Determine the practical limits of the 32GB VRAM/RAM setup when the KV cache is fully saturated.

### 9. MTP vs. Non-MTP Behavior
*   **Task:** Running the same prompts with and without MTP draft decoding enabled.
*   **Goal:** Quantify the speedup provided by MTP and identify any "drift" where the draft decoding might lead the model into suboptimal paths.

# What To Measure

To turn these tests into a publishable benchmark, we are collecting a specific set of metrics via the Flight Recorder.

### Quantitative Metrics
1.  **TTFT (Time to First Token):** How long does the model take to "start" after the prompt is submitted? This measures the overhead of the prompt processing and the initial reasoning step.
2.  **TPOT (Time Per Output Token):** The average speed of the generation phase.
3.  **MTP Speedup Factor:** The ratio of TPS with MTP enabled vs. disabled.
4.  **Reasoning Density:** The percentage of the reasoning budget used vs. the complexity of the task.
5.  **Context Saturation Point:** The point at which latency begins to scale exponentially as the context window fills.
6.  **Token Usage Efficiency:** Total tokens consumed vs. the quality of the final output (scored qualitatively).

### Qualitative Metrics
1.  **Instruction Adherence:** Did the model follow all constraints (e.g., "Do not use the word 'delve'")?
2.  **Coherence Score:** A 1-10 rating on the logical flow of long-form outputs.
3.  **Syntax Validity:** For coding tasks, does the code actually run?
4.  **Hallucination Rate:** How often does the model invent facts when the information is present in the RAG context?

# How To Read The Charts

Once the data is processed, the following charts will be the primary vehicles for communication.

### The "Context vs. Latency" Curve
This chart will plot the number of input tokens (from 1k to 262k) on the X-axis and TTFT/TPOT on the Y-axis. We expect to see a linear increase until a certain "knee" point, after which the memory pressure of the KV cache will cause a non-linear spike in latency.

### The "MTP Gain" Heatmap
A heatmap showing the speedup percentage across different categories. We hypothesize that MTP will provide the highest gains in "Creative Writing" (high-entropy text) and lower gains in "Coding" (low-entropy, structured text).

### Reasoning Budget Utilization
A bar chart showing how much of the 8192 budget is consumed by different task types. This will reveal if the model is "over-thinking" simple tasks or if it is hitting the ceiling on complex ones.

### The "Drift" Analysis
A comparison of MTP vs. Non-MTP outputs. This will show if the draft decoding introduces any significant deviations in the "reasoning" path compared to the standard autoregressive path.

# Privacy Boundaries

A critical component of this benchmark is the "Local-First" ethos. 

### Data Sovereignty
All raw inference data—including the full reasoning traces, the raw prompt/response pairs, and the internal metadata from the Flight Recorder—will remain strictly on the local R9700 server. This is a private draft; the raw logs are not to be uploaded to any third-party service or public repository.

### Sanitization Protocol
Before any findings are shared publicly, the data will undergo a rigorous sanitization process:
1.  **PII Scrubbing:** Removal of any personal identifiers or local file paths.
2.  **Aggregation:** Converting raw timings into averages, medians, and percentiles.
3.  **De-identification:** Removing specific "private" prompts and replacing them with generic "Task Descriptions" (e.g., "Task: Complex Python Refactor" instead of "Task: Refactor my private company's payroll script").
4.  **Artifact Stripping:** Only the final benchmark charts and summary tables will be exported.

# Expected Model Categories

The benchmark will categorize the performance of the `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` into four distinct performance tiers:

1.  **High-Fidelity Reasoning:** Tasks where the 8192 budget is fully utilized to solve complex logic or math.
2.  **High-Throughput Synthesis:** Tasks where MTP draft decoding provides the primary value (e.g., creative writing, summarization).
3.  **Context-Heavy Retrieval:** Tasks where the 262k context window is the primary stressor (e.g., RAG, log analysis).
4.  **Instruction-Strict Execution:** Tasks where the model must adhere to rigid formatting (e.g., JSON output, server-admin configs).

# Limits And Caveats

We must acknowledge the inherent limitations of this specific setup to ensure the benchmark is not misinterpreted.

### The 32GB Ceiling
While 32GB is powerful, it is not infinite. At 262k context, the KV cache can become massive. Users should expect that as the context window fills, the available memory for the model weights may be squeezed, potentially requiring higher quantization (e.g., 4-bit or even 3-bit) to maintain stability.

### Vulkan vs. CUDA
While Vulkan is highly capable, it may not achieve the same peak TFLOPS as a dedicated CUDA implementation. The benchmark should be viewed as a "Real-World Compatibility" test rather than a "Peak Hardware Performance" test.

### MTP Drift
MTP draft decoding is a heuristic. In some cases, the "draft" might lead the model into a logical dead-end that it has to "correct" in the verification step. This can occasionally lead to slightly higher TPOT in very complex logical sequences compared to standard decoding.

### Reasoning Budget Exhaustion
If a task is truly "impossible" for the model, it may spend the entire 8192 budget and still fail. This should be noted as a "Model Limitation" rather than a "Benchmark Failure."

# Next Experiments

Following the completion of this initial benchmark, the next phase of research will focus on:

1.  **Quantization Sensitivity:** Testing 3-bit vs. 4-bit vs. 6-bit versions of the Qwen3.6-35B model to find the exact point where reasoning logic begins to degrade.
2.  **Dynamic Reasoning Budgets:** Implementing a system that scales the reasoning budget based on the detected complexity of the input prompt.
3.  **Multi-Model Routing:** Using a smaller, faster model (e.g., a 7B-class model) to "triage" prompts, only routing complex reasoning tasks to the R9700's 35B-class model.
4.  **KV Cache Compression:** Exploring techniques like 4-bit KV cache quantization to see if we can extend the 262k context window even further within the 32GB limit.
5.  **MTP Fine-Tuning:** Investigating if specific "drafting" models can be used to improve the MTP speedup for the Qwen3.6-35B base.

### Next Experiments (Expanded)

Beyond the initial suite, the second phase of research will focus on the "Stress-Limit" analysis. This involves pushing the R9700 server to its absolute hardware ceilings to identify the point of catastrophic failure—not just in terms of OOM errors, but in terms of "cognitive collapse."

1.  **Quantization Sensitivity Analysis:** We will perform a side-by-side comparison of the `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` model across multiple quantization levels (Q3_K_L, Q4_K_M, Q5_K_M, and Q8_0). The goal is to identify the "Reasoning Cliff"—the specific quantization point where the model's ability to maintain a coherent 8192-token reasoning chain begins to degrade. We hypothesize that for a 35B model, the jump from 4-bit to 5-bit provides a disproportionately large boost in complex coding tasks compared to the memory overhead.

2.  **Dynamic Reasoning Budgets:** Rather than a static `--reasoning-budget 8192`, we will experiment with a "Complexity-Aware" routing system. This would involve a lightweight pre-processor (a smaller 7B-class model) that analyzes the prompt and assigns a reasoning budget. A simple "What is the capital of France?" would get a budget of 128, while a "Refactor this legacy C++ codebase" would get the full 8192. This will allow us to measure the efficiency of "Reasoning Density" across a wider spectrum of difficulty.

3.  **Multi-Model Routing and Orchestration:** We will explore a "MoE-style" (Mixture of Experts) approach at the application layer. By using a local orchestrator, we can route "creative" tasks to a high-throughput MTP-enabled model and "logical/mathematical" tasks to a model optimized for raw precision. This will help us determine if a single 35B model is truly the optimal balance for a 32GB home-lab, or if a multi-model ensemble provides a better "Total Intelligence" per watt.

4.  **KV Cache Compression Techniques:** To maximize the utility of the 262,144 context window, we will test various KV cache quantization methods (e.g., FP8 or INT4 KV cache). This is critical for the R9700 server, as the memory footprint of a 262k context window can easily exceed the remaining 32GB of available memory once the model weights are loaded. We need to find the threshold where KV cache quantization introduces "memory drift" in long-context RAG tasks.

5.  **MTP Fine-Tuning and Draft Model Selection:** We will investigate whether using a specialized "Draft" model (a smaller model specifically trained to predict the tokens of the Qwen3.6-35B) yields better results than the internal MTP draft decoding of the "Balanced" model. This involves measuring the "Acceptance Rate" of draft tokens—the percentage of tokens predicted by the draft that are accepted by the verification step.

6.  **Cross-Lingual Code Translation:** A high-complexity task where the model must translate logic between three different programming languages (e.g., Python to Rust to Go) while maintaining the same functional logic. This will be the ultimate test of the model's "Internal Workspace" and its ability to hold multiple complex structures in its "mind" simultaneously during the reasoning phase.

# Deep Dive: The Mechanics of MTP Draft Decoding

To understand why this benchmark is significant, we must look closely at Multi-Token Prediction (MTP). In standard autoregressive inference, the model predicts token $n$, then uses that prediction as input to predict token $n+1$. This is a serial process. For a 35B model, this serial nature creates a latency floor.

MTP draft decoding changes this by allowing the model to predict a sequence of future tokens (e.g., tokens $n$ through $n+k$) in a single forward pass. These are "draft" tokens. The system then performs a "verification" step, where the model checks if these drafts are likely to be correct. If they are, the model skips the intermediate steps, effectively "jumping" ahead in the generation process.

The `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` model is specifically architected to support this. The "Balanced" designation implies a specific weight distribution designed to maintain high accuracy in the draft phase without requiring a massive increase in parameter count. In our benchmark, we are looking for the "MTP Efficiency Gain"—the ratio of tokens-per-second (TPS) with MTP enabled versus the standard autoregressive path.

However, MTP is not a free lunch. There is a risk of "Draft Drift," where the draft tokens lead the model into a logical cul-de-sac. Because the reasoning budget is set to 8192, the model has the space to "think" through these errors, but if the draft drift is too severe, the model may spend its entire reasoning budget simply trying to correct the errors introduced by the draft decoder. Our Flight Recorder will capture these "Correction Events" by monitoring the variance in the reasoning tokens before the final output is committed.

# The Mathematics of the KV Cache at 262k Context

One of the most ambitious aspects of this benchmark is the 262,144 context window. To understand the technical challenge, we must look at the memory requirements of the Key-Value (KV) cache.

The KV cache stores the "keys" and "values" of the attention mechanism for every token in the context window. The memory required for the KV cache is roughly calculated as:
$Memory = 2 \times \text{Layers} \times \text{Heads} \times \text{Head\_Dim} \times \text{Context\_Length} \times \text{Precision}$

For a 35B-class model, these numbers are substantial. If we assume a model with 32 layers, 32 heads, and a head dimension of 128, using FP16 precision (2 bytes per element), the calculation for a 262,144 token context looks like this:
$2 \times 32 \times 32 \times 128 \times 262,144 \times 2 \text{ bytes} \approx 108 \text{ GB}$

This immediately reveals the core challenge: a 262k context window at full precision is impossible on a 32GB server. This is why the benchmark relies on two critical technologies:
1.  **GGUF Quantization:** By using a quantized model, we reduce the memory footprint of the weights themselves.
2.  **KV Cache Quantization:** We must employ techniques to compress the KV cache (e.g., to 4-bit or 8-bit). By quantizing the KV cache to 4-bit (0.5 bytes per element), the memory requirement drops to approximately 27 GB. 

Even at 27 GB, we are dangerously close to the 32GB limit of the R9700 server. This creates a "Memory Pressure" environment where the OS, the Vulkan driver, and the `llama.cpp` overhead must all coexist in the remaining 5 GB of headroom. This benchmark will specifically measure the "Stability Threshold"—how many tokens can be processed before the system begins to swap memory to disk, which would cause a catastrophic drop in TPS.

# Vulkan Backend: Performance and Memory Mapping

The choice of the Vulkan backend for `llama.cpp` is a deliberate technical decision for the R9700 server. Unlike CUDA, which is tied to NVIDIA's proprietary ecosystem, Vulkan provides a high-performance, cross-platform API that allows for more granular control over memory allocation.

In our setup, the Vulkan backend handles the offloading of the tensor operations to the GPU. However, the "bottleneck" in home-lab environments is often the PCIe bus or the memory bandwidth between the CPU and GPU. Because we are running on a 32GB-class system, we are likely using a mix of system RAM and VRAM. 

The Vulkan implementation in `llama.cpp` allows us to manage "Unified Memory" more effectively. We are monitoring how the driver handles the "paging" of weights. When the KV cache grows, the system must decide which weights to keep in the fastest memory and which to move to slower tiers. Our Flight Recorder will capture the "Memory Access Latency" by measuring the delta between the start of a forward pass and the completion of the attention mechanism. This will tell us if the Vulkan backend is efficiently utilizing the available hardware or if it is getting bogged down by memory management overhead.

# Flight Recorder Architecture: Telemetry Pipeline

The "AI Flight Recorder" is the backbone of this benchmark's data integrity. It is not merely a logger; it is a full-stack telemetry pipeline designed to capture the "hidden" life of an LLM inference.

### The Proxy Layer
The Flight Recorder sits as an OpenAI-compatible proxy between the user's frontend and the `llama.cpp` backend. Every request passes through this proxy, which injects a unique `trace_id` into the metadata.

### Metadata Capture
For every request, the recorder captures:
*   **Prompt Complexity:** A heuristic score based on token count and "instruction density."
*   **Reasoning Budget Utilization:** The number of tokens generated within the reasoning block.
*   **MTP Acceptance Rate:** The percentage of draft tokens that were validated by the model.
*   **Latency Breakdown:** We split the total time into:
    *   *Pre-fill Latency:* Time taken to process the input prompt.
    *   *Reasoning Latency:* Time spent in the 8192-token reasoning block.
    *   *Generation Latency:* Time spent producing the final output.

### Response Previews and Artifacts
The recorder also captures "Response Previews"—intermediate states of the model's output. This is crucial for "Long-Output Reliability" tests. If a model starts to drift or repeat itself at token 1,500 of a 2,000-token response, the Flight Recorder will flag that specific timestamp. By correlating this with the "Reasoning Density" at that moment, we can determine if the model "lost its train of thought" because the reasoning budget was exhausted or because the context window became too saturated.

# OS-Level System Tuning for the R9700

To ensure the benchmark results are reproducible and not skewed by OS-level interference, several system-level optimizations are being applied to the R9700 server.

1.  **HugePages Allocation:** We are enabling 2MB HugePages in the Linux kernel. This reduces the overhead of the Translation Lookaside Buffer (TLB) when the system is managing the massive memory blocks required for the 262k context window.
2.  **CPU Pinning and Isolation:** To prevent the OS scheduler from moving the `llama.cpp` threads across different physical cores, we are using `taskset` to pin the inference process to specific high-performance cores. We also isolate these cores from general OS tasks to minimize context switching.
3.  **Memory Locking:** We are using `mlockall` to prevent the system from swapping the model weights to disk. In a 32GB environment, a single swap event can cause a multi-second hang in the inference loop, which would invalidate the TPS metrics.
4.  **Vulkan Driver Tuning:** We are monitoring the `amdgpu` or `nvidia` (depending on the specific R9700 hardware) driver's memory management. Specifically, we are looking at the "Video Memory Management" (VMM) behavior to ensure that the Vulkan buffers are being allocated in "pinned" memory where possible.

# Quantization Analysis: The "Balanced" Strategy

The choice of the `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` model is central to the thesis. In the world of GGUF quantization, "Balanced" usually refers to a specific optimization of the `K-Quants` method.

Unlike standard 4-bit quantization, which applies a uniform bit-depth across all weights, K-Quants (and the "Balanced" variant) identifies which weights are most critical to the model's "intelligence." For example, the attention heads and the final output layers are often kept at a higher precision (e.g., 6-bit), while the less critical intermediate layers are compressed to 3-bit or 4-bit.

In our benchmark, we are testing if this "Balanced" approach is sufficient for the 8192-token reasoning budget. A major risk with aggressive quantization is "Reasoning Decay"—where the model can still generate fluent text, but its ability to follow complex, multi-step logic fails. By using the "Balanced" model, we are testing the hypothesis that we can maintain 90% of the "Reasoning Fidelity" of a full-precision 35B model while only using 30% of the memory.

# Reasoning Density and Cognitive Load

A new metric we are introducing in this benchmark is **Reasoning Density (RD)**. 

$RD = \frac{\text{Reasoning Tokens}}{\text{Complexity Score} \times \text{Output Length}}$

The "Complexity Score" is a qualitative measure of the prompt's difficulty (e.g., 1 for a greeting, 10 for a complex architectural refactor). 

By tracking RD, we can see how "hard-working" the model is. A high RD suggests the model is dedicating a significant amount of internal thought to a difficult problem. A low RD on a difficult problem suggests the model is "skimming"—it is providing an answer without sufficient internal deliberation. 

This is particularly important for the "Agentic Work" and "Coding" categories. For an agent to successfully plan a multi-step task, it needs a high RD during the "Planning" phase. If the model produces a plan too quickly (low RD), it is likely to fail in the "Execution" phase. Our Flight Recorder will allow us to see exactly where the RD drops off as the context window fills up.

# Comparative Analysis: MTP vs. Non-MTP

To truly validate the MTP draft decoding, we will run a "Control Group" of tests. For every prompt in our suite, we will run two versions:
1.  **Standard Autoregressive (SA):** No MTP, standard `llama.cpp` inference.
2.  **MTP Draft Decoding (MTPD):** Using the `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` with MTP enabled.

We will measure the **"Quality-Speed Tradeoff."** In some cases, MTPD might be 3x faster but result in a 5% drop in "Instruction Adherence." In other cases, it might be 2x faster with 0% loss in quality. 

We will specifically look for "MTP Collapse"—scenarios where the draft decoder becomes so "confident" in an incorrect path that the verification step fails to correct it in time, leading to a hallucination. This is the "frontier" of local LLM optimization: finding the point where speed begins to cannibalize accuracy.

# Conclusion and Methodology Summary

This benchmark is designed to move the home-lab community away from "vibe-based" testing and toward "telemetry-based" evaluation. By combining a 32GB-class R9700 server, a high-parameter Qwen3.6-35B model, MTP draft decoding, and a rigorous 8192-token reasoning budget, we are creating a high-fidelity laboratory for local AI.

The data collected by the AI Flight Recorder will provide a blueprint for how to optimize local inference for production-grade tasks. It will tell us how much context we can actually use before the system breaks, how much reasoning budget is required for different types of work, and whether MTP draft decoding is a viable path for high-speed local AI.

All raw data will remain local, ensuring the privacy of the test environment. The final publication will focus on the sanitized aggregate findings—the charts, the RD scores, and the "Reasoning Cliff" analysis—providing a definitive guide for anyone looking to push the limits of what a 32GB home-lab can achieve with the next generation of open-weight models.

### Technical Appendix: Hardware Interconnects and Memory Bandwidth Analysis

To fully appreciate the performance of the R9700 server in this benchmark, we must look beyond the raw 32GB capacity and examine the underlying memory architecture. In local LLM inference, the primary bottleneck is rarely the raw compute power of the GPU or CPU; it is the bandwidth—the speed at which weights and KV cache data can be moved into the processing units.

The R9700 architecture utilizes a multi-channel memory controller that determines the maximum theoretical bandwidth. When running a 35B model, the system must constantly stream weights from memory to the compute cores. In a 32GB-class system, this is often a "shared" memory environment where the CPU and GPU are competing for the same memory bus. 

During the "Pre-fill" phase—where the model processes the initial 262,144 tokens—the bandwidth requirement is at its peak. The system must ingest the entire prompt and generate the initial KV cache. If the memory bandwidth is insufficient, the Time to First Token (TTFT) will skyrocket. Our benchmark will specifically measure the "Bandwidth Saturation Point," identifying the prompt length at which the memory controller becomes the primary bottleneck, effectively capping the model's responsiveness regardless of how many "reasoning tokens" the budget allows.

### Vulkan Memory Management and Buffer Allocation

The use of the Vulkan backend in `llama.cpp` provides a sophisticated way to handle memory, but it requires careful configuration to avoid fragmentation. In our R9700 setup, we are monitoring how Vulkan handles `VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT` versus `VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`.

Because we are operating near the 32GB limit, the way the driver allocates "heaps" for the KV cache is critical. If the driver allocates memory in small, non-contiguous chunks, the overhead of managing these pointers can slow down the attention mechanism. We are looking for "Buffer Contiguity"—ensuring that the 262k context window is stored in a way that allows the GPU to perform coalesced memory reads.

Furthermore, we are analyzing the `vkAllocateMemory` calls during the scaling of the context window. As the model generates tokens and the KV cache grows, the system must dynamically allocate more space. In a constrained 32GB environment, "Memory Fragmentation" can lead to a situation where there is enough *total* free memory, but not enough *contiguous* memory to house the next set of tokens. Our Flight Recorder will log "Allocation Latency" to identify if the system is struggling with memory management as the context window approaches the 262k limit.

### APEX-MTP Architecture and Draft Verification Logic

The `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` model utilizes a specific architectural optimization for Multi-Token Prediction. In the "Balanced" configuration, the model is trained to predict not just the next token, but a "lookahead" sequence. 

The core of the MTP logic in our benchmark is the **Verification Step**. When the draft decoder proposes a sequence of 3 to 5 tokens, the main model must "verify" them. This is done by comparing the logit distributions of the draft tokens against the model's own internal predictions. We are measuring the **"Draft Acceptance Rate" (DAR)**. 

A high DAR means the draft decoder is highly accurate, allowing the system to skip many autoregressive steps and achieve high TPS. A low DAR means the draft decoder is "hallucinating" paths, forcing the model to constantly correct itself. Because we have an 8192-token reasoning budget, we can observe a unique phenomenon: "Corrective Reasoning." This is where the model spends its reasoning budget not on the *answer*, but on correcting the errors introduced by the MTP draft decoder. By isolating these "Correction Events" in our Flight Recorder, we can determine the true cost of MTP speedups in high-complexity tasks.

### RAG at Scale: The 262k Context Challenge

Retrieval-Augmented Generation (RAG) at a 262,144 token context window is a frontier for home-lab hardware. Most RAG implementations focus on "Small Context" (e.g., 4k to 32k tokens), where the model only sees a few snippets of retrieved data. 

By pushing to 262k, we are testing "Massive Context RAG." This requires the model to maintain "Global Coherence"—the ability to understand how a piece of information on page 1 of the document relates to a piece of information on page 500. 

We will use a modified "Needle in a Haystack" test. Instead of a single fact, we will insert three "Interdependent Facts" across the 262k context. For example:
1.  Fact A: A specific variable name in a code snippet.
2.  Fact B: A dependency requirement for that variable.
3.  Fact C: A security constraint that overrides Fact B.

The model must synthesize all three to provide a correct answer. This tests the "Reasoning Density" of the 8192-token budget. If the model only retrieves Fact A and B but ignores C, it indicates a "Contextual Drift" where the model's attention is being diluted by the sheer volume of the 262k window.

### Flight Recorder Data Schema and Telemetry Processing

To ensure the data is ready for publication, the AI Flight Recorder will output a structured JSON schema for every request. This allows for automated post-processing and the generation of the charts described in the "How To Read The Charts" section.

A sample telemetry entry will look like this:

```json
{
  "trace_id": "bf_99283_x92",
  "model": "Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf",
  "parameters": {
    "context_window": 262144,
    "reasoning_budget": 8192,
    "mtp_enabled": true,
    "vulkan_backend": true
  },
  "input_metadata": {
    "prompt_tokens": 450,
    "complexity_score": 8,
    "category": "Coding_Refactor"
  },
  "inference_metrics": {
    "ttft_ms": 4200,
    "tpot_ms": 45,
    "total_tokens_generated": 1200,
    "reasoning_tokens_used": 7800,
    "mtp_acceptance_rate": 0.92,
    "memory_pressure_pct": 94.2
  },
  "reasoning_trace": [
    {"step": 1, "content": "Analyzing the legacy Python structure...", "ms": 120},
    {"step": 2, "content": "Identifying the bottleneck in the loop...", "ms": 150},
    {"step": 3, "content": "Planning the Rust transition...", "ms": 200}
  ],
  "artifacts": {
    "syntax_validity": true,
    "instruction_adherence": 0.98,
    "drift_detected": false
  }
}
```

By aggregating thousands of these JSON objects, we can perform a statistical analysis of the model's behavior. We can calculate the standard deviation of "Reasoning Density" across different categories and identify the exact "Context Saturation Point" where `memory_pressure_pct` correlates with a spike in `tpot_ms`. This level of granularity is what will separate this benchmark from standard "feel-based" reviews and provide a truly professional-grade analysis for the home-lab community.