# Thesis The convergence of open-weight GGUF models, Vulkan-accelerated inference runtimes, and structured local telemetry represents a pivotal shift in home-lab AI deployment. For years, the enthusiast community has operated under a fundamental constraint: consumer-grade hardware with limited VRAM struggles to accommodate modern large language models, particularly those designed for extended reasoning, multi-token prediction, or dense context utilization. The arrival of 32GB-class servers—such as the R9700 platform—breaks this ceiling, enabling native in-GPU execution of 30–40B parameter models with meaningful context windows and advanced decoding strategies. However, hardware capability alone does not guarantee predictable performance, reproducible benchmarks, or actionable insights. Without a disciplined telemetry pipeline, explicit reasoning budgets, and workloads that mirror real-world usage, benchmarking devolves into synthetic noise. This draft establishes the architectural, methodological, and analytical framework for evaluating what a 32GB-class R9700 server can accomplish when running open-weight GGUF models routed through a local AI Flight Recorder. The active first target is `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`, a model engineered for active-parameter efficiency, multi-token prediction (MTP) draft decoding, and balanced reasoning alignment. The inference stack runs on `llama.cpp` via the Vulkan backend, configured with a 262,144-token context window, draft decoding enabled, and an explicit `--reasoning-budget 8192` flag to cap internal deliberation tokens. All traffic is proxied through a local Flight Recorder that captures OpenAI-compatible request metadata, response previews, timing telemetry, usage accounting, and benchmark artifacts in a strictly local, privacy-preserving pipeline. The central thesis is straightforward: constrained home-lab hardware can deliver production-grade local AI inference when paired with explicit resource budgeting, architecture-aware workload design, and rigorous telemetry capture. The critical insight is that modern thinking models do not behave like traditional next-token predictors. They allocate internal scratchpad tokens, require extended output windows to preserve chain-of-thought coherence, and exhibit highly variable latency profiles depending on draft verification success rates and context-window utilization. Toy benchmarks that truncate output, ignore reasoning budgets, or measure only synthetic prompt-response pairs produce invalid conclusions. Real-world evaluation requires a structured test suite spanning coding, RAG, agentic tool use, server administration, conversational coherence, creative generation, long-output reliability, context-window stress, and MTP acceleration analysis. This document does not present benchmark results. It is a pre-data-collection draft that codifies the methodology, telemetry schema, evaluation criteria, and analytical frameworks necessary to produce publishable, reproducible findings. All raw telemetry will remain local. Only sanitized, aggregated metrics will be shared upon completion. The goal is to provide home-lab operators, system integrators, and open-weight model enthusiasts with a transparent, technically rigorous blueprint for evaluating 32GB-class inference hardware against modern GGUF models, with particular emphasis on thinking-model behavior, MTP decoding dynamics, and privacy-preserving benchmarking practices. # Hardware And Runtime The R9700-class server operating with 32GB of VRAM sits at a critical inflection point in consumer and prosumer AI hardware. It is large enough to accommodate full weight loading for 30–40B parameter models in moderate quantization (Q4_K_M, Q5_K_M), yet constrained enough to demand careful memory management, particularly when enabling extended context windows and advanced decoding strategies. The runtime stack centers on `llama.cpp`, leveraging the Vulkan backend for compute acceleration. This combination introduces specific architectural considerations that directly impact performance, stability, and telemetry accuracy. **VRAM Partitioning and Memory Layout** GGUF models are memory-mapped by design, allowing the runtime to load weights directly into GPU memory without intermediate conversion steps. On a 32GB VRAM pool, the allocation strategy must account for three primary consumers: model weights, KV cache, and scratch/working buffers. For a 35B-parameter model at Q4_K_M quantization, weights occupy approximately 17–18GB. This leaves ~14GB for KV cache and scratch space. The KV cache scales linearly with context length, batch size, and model depth. At 262,144 tokens, even with a batch size of 1, the KV cache can consume 4–6GB depending on layer count, head dimensions, and cache eviction policy. The remaining 8–10GB serves as scratch space for MTP draft verification, attention computation, and Vulkan memory barriers. Memory fragmentation is a persistent risk in long-context scenarios. Vulkan's memory allocation model differs significantly from CUDA's contiguous pool approach. It relies on heap management, memory type filtering, and explicit barrier synchronization. Improper allocation alignment or frequent reallocation during context window expansion can trigger memory thrashing, leading to stuttering latency or OOM conditions. The runtime must be launched with explicit layer offloading thresholds, Vulkan memory pool pre-allocation, and cache eviction policies that prioritize recent tokens while maintaining retrieval accuracy for needle-in-haystack patterns. **Vulkan Backend Characteristics** The Vulkan backend in `llama.cpp` provides cross-platform compute acceleration but introduces distinct performance characteristics. Unlike CUDA, which benefits from mature kernel fusion and automatic memory optimization, Vulkan requires explicit queue submission, command buffer recording, and memory synchronization. The compute shaders handle matrix multiplication, attention computation, and positional encoding. Performance is highly dependent on driver maturity, shader compilation caching, and GPU architecture. Key Vulkan considerations for this deployment: - **Compute Queue Priority:** Vulkan uses multiple queues (graphics, compute, transfer). The inference runtime must bind to the dedicated compute queue to avoid contention with display or transfer operations. - **Memory Barriers:** Frequent context window updates require explicit memory barriers. Improper barrier placement causes pipeline stalls, inflating time-to-first-token (TTFT) and time-between-tokens (TBT). - **Shader Compilation Overhead:** First-run compilation of compute shaders can introduce significant latency. Pre-warming the pipeline and caching compiled shaders mitigates this. - **Precision Handling:** Vulkan supports FP16, BF16, and FP32. The runtime must map GGUF quantization types to compatible Vulkan compute formats without precision loss that degrades reasoning quality. **Context Window Management at 262,144 Tokens** A 256K context window is not merely a storage metric; it is a computational boundary. Attention computation scales quadratically with sequence length in standard transformers, but modern implementations use sliding window attention, rope scaling, and KV cache compression to maintain feasibility. The runtime must enforce: - **Rope Scaling:** Dynamic frequency scaling to maintain positional accuracy across extended sequences. - **Sliding Window:** Active attention window (e.g., 32K–64K tokens) with historical KV cache retention for retrieval. - **Cache Eviction:** LRU or relevance-weighted eviction policies to prevent OOM while preserving retrieval fidelity. - **Tokenization Overhead:** Subword fragmentation increases with domain-specific prompts. The tokenizer must handle mixed-language, code, and technical documentation without excessive tokenization overhead. **MTP Draft Decoding Architecture** Multi-Token Prediction (MTP) introduces a draft-verify paradigm that fundamentally alters latency and throughput profiles. A smaller draft model predicts N tokens ahead, which are then verified by the base model in a single forward pass. Successful verifications yield N tokens at the cost of one base model inference, dramatically improving throughput. However, MTP introduces specific constraints: - **Draft Model Alignment:** The draft model must share architectural compatibility with the base model. Mismatched architectures increase rejection rates, negating throughput gains. - **Verification Overhead:** The base model must process the draft sequence alongside the verified prefix. This increases compute load per step but reduces total forward passes. - **Batch Sizing:** MTP performs best with moderate batch sizes. Overly aggressive batching increases memory pressure and draft rejection rates. - **Latency Variance:** MTP introduces bimodal latency distributions: fast paths (high verification success) and slow paths (draft rejection requiring full regeneration). **Launch Configuration and Runtime Flags** The server is launched with explicit configuration parameters: - `--reasoning-budget 8192`: Caps internal deliberation tokens for thinking models. Prevents runaway generation, ensures predictable latency, and maintains VRAM stability. - `--ctx-size 262144`: Sets maximum context window. Requires careful KV cache management and cache eviction tuning. - `--mtp-draft`: Enables MTP draft decoding. Requires draft model alignment and verification threshold tuning. - `--vulkan`: Activates Vulkan backend. Requires queue binding, memory barrier optimization, and shader caching. - `--threads`, `--batch-size`, `--n-gpu-layers`: Hardware-specific tuning for optimal utilization without OOM. The runtime stack is configured for reproducibility: fixed seeds, deterministic sampling parameters (temperature, top-p, top-k), and explicit logging flags. All telemetry is captured locally via the Flight Recorder proxy, ensuring raw data never leaves the host. This configuration establishes a stable, measurable baseline for evaluating model performance under real-world workloads. # Why Thinking Budgets Matter Thinking models represent a paradigm shift in large language model architecture. Unlike traditional next-token predictors that generate responses sequentially, thinking models allocate internal scratchpad tokens to deliberate, plan, and verify reasoning steps before producing final output. This internal deliberation manifests as chain-of-thought (CoT) sequences, self-correction loops, and multi-step verification. While this improves accuracy on complex reasoning tasks, it introduces significant challenges for constrained hardware: extended token generation, unpredictable latency profiles, and high VRAM consumption for KV cache retention. The `--reasoning-budget 8192` flag is not merely a configuration parameter; it is a critical resource management mechanism. It explicitly caps the number of tokens the model may allocate to internal deliberation before transitioning to final output generation. This cap serves three primary functions: **1. Predictable Latency and Throughput** Without a reasoning budget, thinking models can enter unbounded deliberation loops, particularly on ambiguous or multi-constraint prompts. This results in latency spikes that invalidate performance benchmarks and degrade user experience. A fixed budget ensures that internal scratchpad generation remains bounded, allowing the Flight Recorder to capture consistent timing metrics and enabling accurate throughput calculations. The budget acts as a circuit breaker, forcing the model to transition from deliberation to output generation once the threshold is reached. **2. VRAM and KV Cache Stability** Extended deliberation consumes KV cache memory linearly. For a 35B-parameter model at 256K context, every internal scratchpad token increases memory pressure. Without a budget, prolonged reasoning can trigger cache eviction, context window overflow, or OOM conditions. The 8192-token cap ensures that internal deliberation remains within predictable memory bounds, preserving cache stability and preventing runtime crashes. This is particularly critical for home-lab deployments where thermal throttling and memory fragmentation can exacerbate instability. **3. Benchmark Validity and Workload Alignment** Thinking models are designed for complex reasoning tasks: multi-step planning, code architecture design, debugging, and agentic tool use. These tasks require extended deliberation, but not unbounded generation. A reasoning budget aligns the model's behavior with real-world usage patterns, where users expect structured output within reasonable timeframes. It also enables consistent evaluation across workloads, as the budget provides a fixed parameter for measuring reasoning quality versus output speed. **Interaction with MTP Draft Decoding** The reasoning budget interacts critically with MTP draft decoding. Draft tokens are generated by a smaller model and verified by the base model. If the draft sequence extends beyond the reasoning budget, verification may be interrupted, leading to partial token acceptance or rejection. The budget must account for both internal deliberation and draft verification tokens to prevent premature termination. Proper budget accounting requires the Flight Recorder to track: - Internal scratchpad token count - Draft prediction token count - Verified token count - Total token consumption against budget threshold **Budget Tuning and Model-Specific Behavior** Different thinking models exhibit varying deliberation patterns. Some models allocate most budget to initial planning, while others distribute it across iterative verification. The `--reasoning-budget 8192` flag is calibrated for `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`, which demonstrates balanced reasoning alignment: sufficient deliberation for complex tasks without excessive scratchpad generation. However, budget tuning may be required for other models, particularly those with aggressive self-correction loops or multi-agent simulation behaviors. **Why Budgets Matter for Home-Lab Deployments** Home-lab operators lack the infinite compute resources of cloud deployments. They must optimize for predictable performance, resource stability, and reproducible benchmarks. Reasoning budgets provide explicit control over model behavior, preventing runaway generation while preserving reasoning quality. They enable accurate telemetry capture, consistent workload evaluation, and transparent performance reporting. Without budgets, thinking models behave unpredictably, invalidating benchmarks and degrading user experience. With budgets, they become reliable, measurable, and deployable on constrained hardware. # Why Toy Benchmarks Failed The initial benchmarking phase relied on synthetic, toy workloads designed for rapid turnaround. Prompts were short, `max_tokens` was constrained to 256–512 tokens, and evaluation focused on immediate response accuracy. This approach produced invalid conclusions for several technical reasons, fundamentally misunderstanding the architecture and behavior of modern thinking models. **1. Truncation of Chain-of-Thought Sequences** Thinking models allocate internal scratchpad tokens to deliberate before generating final output. When `max_tokens` is too small, the model is forced to truncate its deliberation sequence prematurely. This results in: - Incomplete reasoning steps - Logical collapse in multi-step planning - Invalid output formatting (e.g., unclosed code blocks, broken JSON) - Artificially low accuracy scores that reflect budget truncation, not model capability Toy benchmarks measured only the final output tokens, ignoring the internal deliberation phase. This created a false impression of model degradation, when in reality, the model was simply prevented from completing its intended reasoning process. **2. KV Cache and Context Window Skew** Short prompts with truncated responses create highly skewed prompt-to-response ratios. This skews KV cache utilization, attention computation, and MTP draft verification rates. The model experiences: - High attention density on short sequences - Frequent draft rejection due to insufficient context - Inaccurate timing metrics (TTFT and TBT do not reflect real-world load) - Misleading throughput calculations that favor synthetic workloads over complex tasks Real-world workloads involve extended prompts, multi-turn conversations, and long outputs. Toy benchmarks failed to replicate this distribution, producing metrics that do not generalize to production usage. **3. MTP Draft Decoding Artifacts** MTP draft decoding requires sufficient sequence length to demonstrate acceleration benefits. Draft models predict N tokens ahead, which are verified by the base model. When responses are truncated: - Draft verification is interrupted mid-sequence - Rejection rates spike due to incomplete context - Throughput gains are negated by frequent regeneration - Latency variance increases unpredictably Toy benchmarks measured MTP performance on artificially short sequences, producing invalid acceleration ratios and misleading draft rejection statistics. **4. Reasoning Budget Misalignment** Thinking models require extended output windows to accommodate internal deliberation. Constrained `max_tokens` forces the model to compress reasoning into limited tokens, resulting in: - Superficial analysis - Missing verification steps - Inaccurate tool-use sequences - Degraded agentic planning The `--reasoning-budget 8192` flag ensures internal deliberation remains bounded, but toy benchmarks ignored this parameter entirely, treating thinking models as standard next-token predictors. This fundamental mismatch invalidated all early measurements. **5. Proxy and Telemetry Blind Spots** The Flight Recorder captures request metadata, response previews, timings, and usage. When toy benchmarks truncate output, the proxy records incomplete sequences, leading to: - Underreported token counts - Inaccurate budget consumption metrics - Misleading latency percentiles - Invalid quality assessments Telemetry accuracy depends on complete sequence capture. Truncated responses produce corrupted telemetry, which propagates into aggregated metrics and invalidates benchmark conclusions. **Lessons Learned and Methodological Correction** The failure of toy benchmarks underscores a critical principle: benchmark design must match model architecture. Thinking models require extended output windows, explicit reasoning budgets, and workloads that mirror real-world complexity. The corrected methodology prioritizes: - Generous `max_tokens` limits (4096–8192 tokens per response) - Real-world workload categories (coding, RAG, agentic work, etc.) - Complete sequence capture for telemetry accuracy - MTP-aware timing and throughput calculations - Reasoning budget accounting for internal deliberation This correction ensures that benchmark results reflect actual model capability, not synthetic constraints. The real-world test suite, detailed in the following sections, is designed to produce valid, reproducible, and actionable insights for home-lab deployments. # The Real-World Test Suite The benchmark suite is structured around nine workload categories that mirror real-world home-lab and production usage patterns. Each category is designed to stress specific model capabilities, measure telemetry accuracy, and evaluate MTP versus non-MTP behavior under realistic constraints. Prompts are engineered for reproducibility, with fixed seeds, controlled sampling parameters, and explicit evaluation criteria. The Flight Recorder captures all traffic, ensuring complete sequence logging and accurate telemetry. **1. Coding Workloads** - **Objective:** Evaluate multi-file code generation, debugging, architecture design, and refactoring. - **Prompts:** Provide incomplete codebases, bug reports, and architectural constraints. Request full implementation, test cases, and documentation. - **Metrics:** Code compilation success rate, test pass rate, documentation completeness, MTP draft verification accuracy, latency percentiles. - **Expected Failure Modes:** Truncated code blocks, missing imports, incorrect type hints, logical errors in multi-step debugging. **2. RAG (Retrieval-Augmented Generation)** - **Objective:** Evaluate context retrieval, citation accuracy, chunking alignment, and hallucination resistance. - **Prompts:** Provide retrieved document chunks, ask for synthesized answers with explicit citations. Include contradictory or missing information to test retrieval fidelity. - **Metrics:** Citation precision/recall, hallucination rate, chunk alignment accuracy, context window utilization efficiency. - **Expected Failure Modes:** Fabricated citations, misaligned chunk references, over-reliance on training data, context window overflow. **3. Agentic Work** - **Objective:** Evaluate tool use, state management, multi-step planning, and error recovery. - **Prompts:** Provide environment constraints, tool definitions, and task objectives. Request step-by-step execution with tool calls, state updates, and error handling. - **Metrics:** Tool success rate, state consistency, planning accuracy, error recovery rate, MTP draft token acceptance. - **Expected Failure Modes:** Invalid tool parameters, missing state updates, infinite loops, broken JSON/tool schemas. **4. Server-Admin Work** - **Objective:** Evaluate bash scripting, networking configuration, log analysis, and system administration. - **Prompts:** Provide system logs, configuration files, and operational constraints. Request diagnostic scripts, configuration fixes, and automation workflows. - **Metrics:** Script syntax validity, command execution accuracy, log parsing precision, configuration safety, latency under complex prompts. - **Expected Failure Modes:** Unsafe commands, incorrect regex patterns, missing error handling, over-permissive configurations. **5. Chat Assistant Behavior** - **Objective:** Evaluate conversational coherence, memory retention, tone adaptation, and multi-turn consistency. - **Prompts:** Provide multi-turn dialogues with evolving context, user preferences, and topic shifts. Evaluate coherence, memory recall, and tone consistency. - **Metrics:** Turn consistency, memory accuracy, tone alignment, context retention, latency variance across turns. - **Expected Failure Modes:** Context drift, tone inconsistency, memory hallucination, repetitive responses. **6. Creative Writing** - **Objective:** Evaluate narrative structure, character consistency, stylistic adaptation, and long-form coherence. - **Prompts:** Provide genre constraints, character profiles, and plot outlines. Request chapter drafts, scene transitions, and stylistic variations. - **Metrics:** Narrative coherence, character consistency, stylistic fidelity, pacing accuracy, long-output formatting. - **Expected Failure Modes:** Plot contradictions, character drift, tonal inconsistency, formatting breaks. **7. Long-Output Reliability** - **Objective:** Evaluate 5,000+ token generation, coherence decay, formatting stability, and MTP acceleration over extended sequences. - **Prompts:** Provide detailed outlines, technical specifications, and structural constraints. Request comprehensive documentation, reports, or guides. - **Metrics:** Coherence decay rate, formatting compliance, MTP draft acceptance over length, latency scaling, budget adherence. - **Expected Failure Modes:** Coherence breakdown, formatting corruption, MTP rejection spikes, budget exhaustion. **8. Context Fit** - **Objective:** Evaluate 256K context window utilization, needle-in-haystack retrieval, repetition resistance, and KV cache efficiency. - **Prompts:** Provide extended context with embedded queries, contradictory information, and retrieval targets. Measure accuracy across window positions. - **Metrics:** Retrieval accuracy by position, repetition rate, KV cache utilization, context window efficiency, MTP draft performance in long contexts. - **Expected Failure Modes:** Positional decay, retrieval failure, repetition loops, cache eviction errors. **9. MTP vs Non-MTP Behavior** - **Objective:** Compare throughput, latency, accuracy, and draft rejection rates between MTP-enabled and baseline decoding. - **Prompts:** Use identical workloads across both configurations. Measure acceleration ratios, verification success rates, and quality preservation. - **Metrics:** Throughput ratio, latency percentiles, accuracy delta, draft rejection rate, budget consumption difference. - **Expected Failure Modes:** MTP degradation on complex tasks, draft model misalignment, verification overhead, throughput variance. Each workload is executed with fixed seeds, deterministic sampling parameters, and explicit timeout thresholds. The Flight Recorder captures complete sequences, ensuring telemetry accuracy and enabling post-hoc analysis. This suite provides a comprehensive evaluation framework that reflects real-world home-lab usage, rather than synthetic constraints. # What To Measure Telemetry accuracy is the foundation of valid benchmarking. The Flight Recorder captures request metadata, response previews, timings, usage accounting, and benchmark artifacts in a structured, local-only pipeline. The measurement schema is designed to capture both quantitative performance metrics and qualitative quality indicators, enabling comprehensive analysis across all workload categories. **Request Metadata** - Prompt token count (pre-tokenization, post-tokenization, subword fragmentation) - System prompt length and role alignment - Sampling parameters (temperature, top-p, top-k, repetition penalty) - Model version, quantization tier, GGUF hash - Vulkan backend flags, layer offloading count, batch size - Reasoning budget allocation and consumption tracking **Timing Telemetry** - Time-to-first-token (TTFT): Measures prompt processing and initial generation latency - Time-between-tokens (TBT): Measures generation throughput and MTP acceleration - Total generation time: End-to-end latency including draft verification and budget enforcement - Percentile distributions (p50, p90, p95, p99): Captures latency variance and outlier behavior - MTP draft verification time: Measures draft prediction and base model verification overhead **Usage Accounting** - Input token count (prompt + system + retrieved context) - Output token count (final response + internal scratchpad) - Draft token count (MTP predictions) - Verified token count (accepted draft tokens) - Rejected token count (draft tokens requiring regeneration) - Reasoning budget consumption (internal deliberation tokens) - Context window utilization (active KV cache size, eviction events) **Response Previews and Quality Indicators** - First 512 tokens preview (for formatting and coherence assessment) - JSON/tool schema validation status - Code compilation readiness (syntax check, import validation) - Citation accuracy markers (for RAG workloads) - Formatting compliance (markdown, JSON, XML structure) - Error detection flags (truncation, budget exhaustion, OOM warnings) **Performance Metrics** - Throughput (tokens/sec): Overall generation speed, MTP acceleration ratio - Latency percentiles: Distribution of TTFT and TBT across workloads - Context window efficiency: Tokens processed per GB of VRAM - MTP acceleration: Verified tokens per base model forward pass - Budget adherence: Percentage of workloads completing within reasoning budget - Draft rejection rate: Percentage of draft tokens requiring regeneration **Quality Metrics** - Factual accuracy (domain-specific validation) - Coherence score (n-gram consistency, logical flow) - Formatting compliance (structural integrity, syntax validity) - Tool-use success rate (agentic workloads) - Citation precision/recall (RAG workloads) - Long-output stability (coherence decay over 5k+ tokens) **Telemetry Schema and Storage** All telemetry is logged in JSONL format, with each line representing a single request. Fields are strictly typed, with timestamps, request IDs, and workload categories for traceability. Raw data is stored in encrypted local storage, with access restricted to the benchmark harness. Aggregated metrics are computed via local scripts, ensuring no PII or sensitive context leaves the host. This schema enables precise cross-workload analysis, MTP behavior tracking, and reasoning budget evaluation. # How To Read The Charts Visualizing benchmark data requires careful consideration of metric relationships, workload distribution, and hardware constraints. The following guidelines explain how to interpret the charting methodology, distinguish signal from noise, and cross-reference telemetry with raw logs. **Scatter Plots: Latency vs Throughput** - X-axis: Average TBT (tokens/sec) - Y-axis: TTFT (milliseconds) - Interpretation: Points in the upper-left quadrant indicate fast generation but high initial latency (typical of MTP with draft verification overhead). Points in the lower-right indicate low initial latency but slower generation (typical of non-MTP baseline). Outliers indicate draft rejection spikes or budget exhaustion. - Cross-reference: Match outlier points to request IDs in raw logs to identify specific failure modes (e.g., draft model misalignment, context window overflow). **Heatmaps: Context Window vs Accuracy** - X-axis: Context window utilization percentage - Y-axis: Retrieval accuracy (for RAG and context fit workloads) - Interpretation: High utilization with high accuracy indicates efficient KV cache management and effective rope scaling. High utilization with low accuracy indicates cache eviction errors or positional decay. - Cross-reference: Correlate with cache eviction event logs to identify memory fragmentation or LRU policy failures. **Line Charts: Budget vs Performance** - X-axis: Reasoning budget consumption percentage - Y-axis: Quality score (coherence, accuracy, formatting) - Interpretation: Linear decline indicates budget exhaustion degrading output quality. Plateau indicates optimal budget utilization. Spikes indicate draft verification interference or MTP acceleration artifacts. - Cross-reference: Match budget consumption to internal scratchpad token counts to verify deliberation alignment. **Box Plots: MTP vs Non-MTP Behavior** - X-axis: Decoding configuration (MTP enabled/disabled) - Y-axis: Latency percentiles (p50, p90, p95, p99) - Interpretation: Narrower boxes indicate consistent performance. Wider boxes indicate high variance (typical of MTP with variable draft rejection rates). Median shift indicates throughput improvement or degradation. - Cross-reference: Match variance to draft rejection rates and verification overhead logs. **Statistical Considerations** - Confidence intervals: All metrics include 95% confidence intervals to account for hardware variance and workload distribution. - Outlier handling: Points beyond 3 standard deviations are flagged for manual review, not automatically excluded. - Normalization: Metrics are normalized across workloads to enable cross-category comparison. - Proxy overhead: Flight Recorder latency is subtracted from raw timings to ensure accurate model performance measurement. **Reading the Charts Correctly** Charts are not standalone metrics; they are visualizations of telemetry data that require cross-referencing with raw logs, workload categories, and hardware constraints. Always verify chart interpretations against request-specific telemetry, budget consumption, and draft verification rates. Avoid overgeneralizing from aggregate metrics; home-lab deployments vary significantly based on GPU architecture, driver maturity, and thermal conditions. Use charts as diagnostic tools, not definitive conclusions. # Privacy Boundaries The home-lab AI ecosystem operates in a trust-sensitive environment. Users deploy models locally to maintain data sovereignty, avoid vendor lock-in, and preserve privacy. The Flight Recorder architecture is designed with strict privacy boundaries, ensuring that raw telemetry never leaves the host, while still enabling reproducible benchmarking and community knowledge sharing. **Data Handling Policy** - Raw traffic remains local: All OpenAI-compatible proxy traffic, request metadata, response previews, and timing logs are stored exclusively on the host machine. - No external telemetry: The Flight Recorder does not transmit data to external servers, cloud services, or third-party analytics platforms. - Encrypted local storage: Telemetry is stored in AES-256 encrypted containers, with access restricted to the benchmark harness and authorized system accounts. - PII stripping: All prompts and responses are scanned for personally identifiable information, API keys, credentials, and sensitive context. Detected PII is redacted before storage. - Model output redaction: Responses containing system prompts, internal instructions, or proprietary context are masked in aggregated metrics. **Proxy Architecture** - Local-only routing: All client requests are intercepted by a local proxy that forwards traffic to the inference runtime without external relay. - Request isolation: Each request is assigned a unique session ID, ensuring cross-request isolation and preventing telemetry leakage. - Timeout enforcement: Requests exceeding defined thresholds are terminated, preventing indefinite generation and resource exhaustion. - Access controls: Proxy configuration requires explicit authorization, with role-based access control (RBAC) for telemetry viewing and export. **Sharing Strategy** - Sanitized aggregates only: Only aggregated metrics, anonymized patterns, and methodology documentation are shared publicly. - Reproducible scripts: Benchmark execution scripts, telemetry schemas, and analysis pipelines are open-sourced to enable community verification. - Transparent methodology: All configuration parameters, workload definitions, and evaluation criteria are documented to ensure reproducibility. - Community contribution: Home-lab operators are encouraged to run the benchmark suite on their hardware, contributing anonymized aggregates to a shared repository. **Why Privacy Matters** Home-lab deployments often involve sensitive workloads: proprietary code, internal documentation, personal assistants, and confidential analysis. Privacy boundaries ensure that users maintain full control over their data, while still benefiting from community benchmarking and methodology sharing. The Flight Recorder architecture prioritizes data sovereignty without sacrificing analytical rigor, enabling transparent, reproducible, and privacy-preserving benchmarking. # Expected Model Categories The GGUF ecosystem spans diverse model families, quantization tiers, and architectural designs. Understanding these categories is essential for selecting appropriate models for the R9700 32GB platform and interpreting benchmark results. **Quantization Tiers** - Q2_Q3: Extreme compression, significant accuracy degradation, suitable only for exploratory testing. - Q4_K_M: Balanced accuracy and VRAM efficiency, ideal for 30–40B models on 32GB hardware. - Q5_K_M: Higher fidelity, increased VRAM consumption, suitable for reasoning-critical workloads. - Q8_F16: Near-lossless precision, high VRAM demand, limited to smaller models or CPU offloading. - FP16/BF16: Full precision, maximum VRAM consumption, typically reserved for development and validation. **Architecture Families** - Dense Models: All parameters active per token, predictable VRAM usage, standard attention computation. - MoE (Mixture of Experts): Sparse activation, lower active parameter count, higher throughput, requires careful expert routing. - Active-Parameter Models (e.g., APEX variants): Optimized for efficient activation, reduced compute overhead, improved MTP compatibility. - Thinking-Optimized Models: Designed with internal scratchpad allocation, chain-of-thought alignment, and reasoning budget support. **MTP Compatibility** - Native MTP support: Models explicitly trained with draft-verify paradigms, high verification success rates. - Adapted MTP support: Models requiring external draft models, variable verification rates, potential accuracy trade-offs. - Non-MTP models: Standard next-token prediction, baseline latency/throughput, no draft acceleration. **Selection Criteria for `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`** - Active parameter efficiency: 3B active parameters reduce compute overhead while maintaining 35B contextual capacity. - APEX optimization: Enhanced activation routing improves MTP draft verification rates and reduces latency variance. - MTP I-Balanced variant: Calibrated for balanced reasoning alignment, ensuring sufficient deliberation without excessive scratchpad generation. - GGUF compatibility: Optimized for Vulkan backend, with memory-mapped weights and efficient KV cache management. This model represents a strategic choice for the 32GB R9700 platform, balancing VRAM efficiency, reasoning capability, and MTP acceleration. Other models may require different quantization tiers, context window limits, or budget tuning, but the evaluation methodology remains consistent across categories. # Limits And Caveats No benchmarking framework is without constraints. Acknowledging limitations is essential for interpreting results accurately and avoiding overgeneralization. The following caveats apply to this evaluation methodology and the R9700 32GB platform. **Hardware Constraints** - VRAM fragmentation: Vulkan memory allocation can fragment over extended sessions, impacting context window stability. - Thermal throttling: Sustained compute load may trigger GPU thermal limits, reducing clock speeds and increasing latency. - PCIe bandwidth: CPU-to-GPU data transfer limits may bottleneck prompt processing and context window updates. - Driver maturity: Vulkan backend performance varies significantly across GPU architectures and driver versions. **Software Constraints** - llama.cpp version drift: Runtime updates may alter memory management, MTP implementation, and Vulkan backend behavior. - MTP draft model alignment: Draft model quality directly impacts verification success rates and throughput acceleration. - Proxy overhead: Flight Recorder interception adds minimal latency, but may introduce timing variance under high concurrency. - Context window implementation: Sliding window, rope scaling, and cache eviction policies vary across runtime versions. **Methodological Constraints** - Workload representativeness: Nine categories cover major use cases but cannot encompass all real-world scenarios. - Subjective quality metrics: Coherence, accuracy, and formatting scores require human validation or specialized evaluators. - Reproducibility across configs: Home-lab hardware varies significantly; results may not generalize to different GPUs, RAM, or storage configurations. - Model-specific variability: Thinking models exhibit different deliberation patterns; budget tuning may be required per model. **Transparency Statement** This document is a pre-data-collection draft. No benchmark results are fabricated or inferred. All metrics, charts, and conclusions will be derived from real telemetry collected during the benchmark suite execution. Raw data remains local; only sanitized aggregates will be shared. The methodology is designed for reproducibility, with full configuration parameters, workload definitions, and analysis scripts available for community verification. # Next Experiments The completion of the initial benchmark suite will enable a series of follow-up experiments designed to optimize performance, explore hardware limits, and expand the evaluation framework. These experiments will build upon the telemetry schema, workload categories, and privacy boundaries established in this draft. **Scaling Studies** - VRAM partitioning: Evaluate optimal weight/KV cache/scratch allocation ratios across quantization tiers. - Context window limits: Test 128K, 192K, and 256K windows to identify accuracy decay and cache eviction thresholds. - CPU offload thresholds: Measure performance degradation when offloading layers to system RAM, identifying viable fallback configurations. **MTP Optimization** - Draft model selection: Compare multiple draft architectures to identify optimal verification success rates and throughput acceleration. - Verification thresholds: Tune draft token acceptance criteria to balance accuracy and latency. - Batch size tuning: Evaluate batch size impact on MTP acceleration, memory pressure, and latency variance. **Proxy Enhancements** - Structured logging: Implement automated schema validation, error detection, and telemetry normalization. - Real-time budget monitoring: Develop live dashboards for reasoning budget consumption, draft verification rates, and VRAM utilization. - Automated evaluation: Integrate LLM-as-judge and rule-based validators for quality scoring across workloads. **Cross-Model Comparison** - Quantization sweeps: Benchmark Q4_K_M, Q5_K_M, and Q8_F16 variants across all workload categories. - Architecture benchmarks: Compare dense, MoE, and active-parameter models for throughput, accuracy, and MTP compatibility. - Thinking vs non-thinking: Evaluate reasoning budget impact on standard models, identifying optimal budget configurations. **Community Contribution** - Open methodology: Publish benchmark scripts, telemetry schemas, and analysis pipelines for community verification. - Shared telemetry repository: Aggregate anonymized metrics from home-lab operators to establish industry baselines. - Hardware collaboration: Partner with GPU manufacturers and runtime developers to optimize Vulkan backend performance for thinking models. These experiments will transform the initial benchmark suite into a comprehensive evaluation framework, enabling home-lab operators to make informed decisions about hardware selection, model deployment, and performance optimization. The goal is not to produce vendor marketing material, but to establish transparent, reproducible, and privacy-preserving benchmarking practices for the open-weight AI community.