# Thesis

The convergence of open-weight language models, consumer-grade GPU hardware, and localized inference proxies has fundamentally altered the trajectory of home-lab AI experimentation. For years, enthusiasts were constrained by proprietary API ecosystems, opaque evaluation metrics, and hardware that could not accommodate modern model sizes. Today, the landscape has shifted. A 32GB-class R9700 server, paired with carefully quantized GGUF models and routed through a local AI Flight Recorder, presents a unique computational environment capable of handling real-world, multi-step AI workloads. This draft outlines the architectural, methodological, and operational framework for evaluating what this specific stack can actually do, rather than what synthetic benchmarks claim it can do.

The active first target, `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`, represents a class of models that blend active parameter efficiency with multi-token prediction capabilities. When executed via `llama.cpp` on a Vulkan backend with a 262,144-token context window and an explicit `--reasoning-budget 8192`, the system is designed to prioritize deliberative generation over raw token throughput. The local AI Flight Recorder acts as both a proxy and an observability layer, capturing OpenAI-compatible traffic, request metadata, response previews, timing telemetry, usage statistics, and benchmark artifacts. This architecture enables granular, reproducible measurement without exposing raw prompts or responses to external networks.

Early testing revealed a critical flaw in conventional evaluation approaches: toy benchmarks with artificially constrained `max_tokens` values systematically misrepresent the behavior of thinking-capable models. These models require extended generation windows to execute chain-of-thought reasoning, self-correction, tool-use planning, and multi-step verification. When truncated prematurely, they appear slow, incoherent, or inefficient. The real benchmark suite must therefore be structured around sustained, realistic workloads that exercise coding, RAG, agentic workflows, server administration, conversational assistance, creative generation, long-output reliability, context window utilization, and MTP versus non-MTP decoding behavior.

This document serves as a private draft framework. It does not contain published results, as real data collection is still pending. All raw telemetry, prompt logs, and response payloads will remain strictly local. Only sanitized, aggregate findings will be shared at a later stage. The goal is to establish a rigorous, transparent, and reproducible methodology that home-lab operators, open-weight researchers, and local AI practitioners can adapt, extend, and validate. By focusing on architectural constraints, runtime behavior, reasoning budget allocation, and proxy-level observability, this draft provides a foundation for understanding what a 32GB-class server can realistically achieve with modern open-weight models when routed through a local AI Flight Recorder.

# Hardware And Runtime

The foundation of this evaluation rests on a 32GB-class R9700 server, a configuration that sits at a critical inflection point for local AI inference. Thirty-two gigabytes of VRAM is sufficient to load mid-sized dense models, moderate-parameter Mixture-of-Experts (MoE) architectures, and heavily quantized variants of larger families, but it demands careful memory management. The R9700 platform, typically equipped with high-bandwidth memory controllers, PCIe Gen4 lanes, and consumer-to-prosumer GPU architectures, provides a stable compute substrate. However, the bottleneck rarely lies in raw FLOPS; it lies in memory bandwidth, VRAM fragmentation, thermal headroom, and the efficiency of the inference runtime.

The runtime in question is `llama.cpp`, executed with a Vulkan compute backend. Vulkan is chosen not merely as an alternative to CUDA or Metal, but as a highly portable, low-overhead compute abstraction that maps efficiently to modern GPU architectures. The Vulkan backend in `llama.cpp` leverages compute shaders for tensor operations, asynchronous queue submission for overlapping prefill and decode phases, and explicit memory barriers to manage KV cache updates. This approach minimizes driver-level overhead and allows fine-grained control over compute scheduling, which is critical when running extended generation workloads.

GGUF (GGML Unified Format) is the quantization and serialization standard used for model weights. The target model, `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf`, utilizes a balanced quantization scheme that preserves critical attention heads and feed-forward network parameters while compressing less sensitive layers. The "A3B" designation indicates an active parameter count of approximately 3 billion, suggesting a sparse or MoE routing mechanism that activates only a subset of the total 35 billion parameters per forward pass. The "APEX" component refers to an optimized kernel implementation, likely targeting specific tensor core or compute shader pathways. The "I-Balanced" suffix denotes a quantization layout that prioritizes inference stability over extreme compression, striking a balance between memory footprint and numerical accuracy.

The context window is configured to 262,144 tokens. This is not merely a configuration flag; it dictates KV cache allocation, RoPE (Rotary Positional Encoding) scaling behavior, and attention computation complexity. At this scale, the KV cache alone can consume several gigabytes of VRAM, especially when using FP16 or BF16 precision for attention states. The runtime must manage cache eviction, sliding window optimization, or attention compaction strategies to prevent out-of-memory errors during long-context workloads. RoPE scaling is adjusted to maintain positional accuracy across the extended window, typically using NTK-aware or YaRN-based interpolation methods to mitigate attention decay.

Multi-Token Prediction (MTP) draft decoding is enabled as a core runtime feature. MTP speculative decoding works by employing a smaller, faster draft model to generate N candidate tokens, which are then verified in parallel by the larger target model. If the draft tokens match the target model's predictions, they are accepted in a single forward pass; otherwise, the target model recomputes the divergent position and continues. This mechanism can significantly improve token throughput, but it introduces additional complexity: draft model alignment, acceptance rate variance, memory overhead for draft KV states, and synchronization barriers. The Vulkan backend must handle draft and target compute queues concurrently, managing memory transfers and shader execution pipelines to avoid stalls.

The server is launched with `--reasoning-budget 8192`, a runtime flag that allocates a dedicated token window for deliberative generation. This budget is separate from the total context window and is specifically reserved for chain-of-thought reasoning, self-correction loops, tool-use planning, and multi-step verification. The runtime enforces this budget by tracking token consumption, pausing generation when the budget is exhausted, and optionally triggering a summary or fallback mechanism. This flag fundamentally changes how the model is evaluated: instead of measuring raw speed or single-turn accuracy, the system measures how effectively the model utilizes allocated reasoning space.

Vulkan compute shaders are compiled with explicit workgroup sizes, memory layout optimizations, and async queue prioritization. The runtime monitors VRAM allocation, kernel execution times, and queue submission latency. Memory fragmentation is mitigated through pre-allocated KV cache pools and tensor weight pinning. Thermal throttling is managed via dynamic frequency scaling and compute load balancing. The entire stack is designed to operate within the 32GB VRAM constraint while maintaining stable performance across extended generation workloads.

# Why Thinking Budgets Matter

The concept of a reasoning budget represents a paradigm shift in how we evaluate and deploy language models. Traditional inference pipelines treat generation as a linear sequence: prompt in, tokens out, stop at `max_tokens`. This approach works adequately for simple completion tasks, but it fails catastrophically for models designed to think. Thinking models—those trained with explicit chain-of-thought, self-reflection, tool-use planning, or multi-step verification objectives—require dedicated computational space to execute their internal reasoning processes before producing final output.

A reasoning budget of 8192 tokens is not arbitrary. It is calibrated to accommodate extended deliberation without exhausting the total context window. When a model receives a complex prompt requiring architecture design, multi-file refactoring, RAG synthesis, or agentic workflow planning, it must first decompose the problem, generate intermediate steps, verify assumptions, and refine its approach. This process consumes tokens. If the budget is too small, the model is forced to truncate its reasoning mid-stream, resulting in incomplete logic, premature conclusions, or fallback to pattern-matching heuristics. If the budget is too large, it wastes context window space, increases latency, and may introduce hallucination drift through excessive self-correction loops.

The `--reasoning-budget 8192` flag enforces a structural constraint on the generation pipeline. The runtime tracks token consumption within the budget window, monitors attention computation overhead, and dynamically adjusts decoding parameters to stay within limits. When the budget is exhausted, the model is expected to transition from deliberative generation to final output synthesis. This transition is critical: it tests the model's ability to summarize its reasoning, extract key conclusions, and produce structured output without losing coherence.

Measuring reasoning budget utilization requires more than counting tokens. The Flight Recorder captures budget allocation patterns, early stopping signals, draft token acceptance rates within the reasoning window, and attention head activation profiles. It tracks how the model distributes tokens across problem decomposition, verification, correction, and final output. It monitors whether the model consistently utilizes the full budget or truncates prematurely. It identifies cases where the model wastes budget on redundant self-correction or loops. It distinguishes between productive deliberation and unproductive token churn.

The implications for benchmark design are profound. Toy benchmarks that constrain `max_tokens` to 512 or 1024 tokens fundamentally misrepresent thinking model behavior. They force premature termination, skew latency metrics, and produce inaccurate accuracy scores. A model that requires 4000 tokens to reason through a coding task will appear broken if evaluated with a 2000-token limit. The real benchmark suite must allocate sufficient reasoning budget to allow models to operate as designed. It must measure not just what the model outputs, but how it arrives at that output.

Reasoning budgets also interact with MTP draft decoding. Draft models must align with the target model's reasoning patterns to maintain high acceptance rates. If the draft model generates tokens that deviate from the target's deliberative path, verification overhead increases, and throughput drops. The runtime must balance draft speed with reasoning fidelity. The Flight Recorder captures draft acceptance rates within the reasoning window, tracks verification latency, and measures how draft alignment impacts budget utilization.

Ultimately, thinking budgets transform evaluation from a throughput exercise into a reasoning quality assessment. They force practitioners to confront the computational cost of deliberation, the trade-offs between speed and accuracy, and the architectural requirements of modern LLMs. By enforcing a structured reasoning budget, the runtime ensures that models are evaluated under conditions that match their training objectives, rather than being punished for their design philosophy.

# Why Toy Benchmarks Failed

Early testing of the R9700 server with `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` revealed a consistent pattern of misleading results. The initial benchmark suite consisted of short prompts, constrained `max_tokens` values, and synthetic evaluation metrics. The results were uniformly poor: high latency, low coherence, frequent truncation, and inaccurate outputs. At first glance, the model appeared incapable of handling real-world workloads. Further investigation revealed that the failure was not in the model or the hardware, but in the benchmark design.

Toy benchmarks fail because they ignore the architectural reality of thinking models. These models are trained to generate extended reasoning traces before producing final output. They expect sufficient token budget to decompose problems, verify assumptions, and refine their approach. When `max_tokens` is set too low, the generation pipeline is forced to terminate prematurely. The model is cut off mid-reasoning, producing incomplete logic, truncated conclusions, or fallback responses. The benchmark measures this truncation as model failure, when in reality it measures benchmark failure.

The illusion of speed compounds the problem. Short `max_tokens` values produce low total latency numbers, which are often misinterpreted as high performance. In reality, the model is not completing the task; it is being silenced. The benchmark records a fast response, but the response is useless. This creates a false positive: the system appears efficient, but it is functionally broken. When the same prompts are evaluated with adequate reasoning budget, latency increases, but output quality improves dramatically. The trade-off is real, and it must be measured honestly.

MTP draft decoding exacerbates the issue under constrained budgets. Draft models generate N tokens in parallel, but if the budget is exhausted before verification completes, draft tokens are wasted. The runtime incurs overhead for draft generation, memory allocation, and queue synchronization, but produces no useful output. The benchmark records high draft acceptance rates or low acceptance rates, but neither metric reflects actual task completion. The system appears unstable, when in reality the benchmark is misaligned with the model's operational requirements.

Context window utilization is another casualty of toy benchmarks. Models with 262,144-token context windows are designed to ingest large documents, maintain long conversational state, and execute multi-step workflows. Toy benchmarks rarely exceed 2,000 tokens of input, leaving the majority of the context window unused. This creates a false impression of inefficiency: the model appears to waste memory, when in reality it is simply not being asked to utilize its full capacity. Real-world workloads require sustained context usage, and benchmarks must reflect that demand.

The failure of toy benchmarks is not unique to this hardware or runtime. It is a systemic issue in AI evaluation. Practitioners chase token throughput, latency numbers, and synthetic accuracy scores, ignoring the structural requirements of modern models. Thinking models require deliberative space. Agentic models require tool-use planning windows. RAG models require document ingestion and synthesis phases. Creative models require narrative development and stylistic refinement. Benchmarks that ignore these requirements produce misleading results that do not translate to real-world performance.

The lesson is clear: benchmark design must match model architecture and intended use case. Toy benchmarks are useful for quick sanity checks, but they are inadequate for evaluating thinking-capable models. The real benchmark suite must allocate sufficient reasoning budget, simulate realistic workloads, measure task completion rather than token count, and track deliberation quality alongside output accuracy. Only then can practitioners understand what the system can actually do.

# The Real-World Test Suite

The real benchmark suite is structured around nine core workload categories, each designed to exercise specific model capabilities, runtime behaviors, and proxy observability features. The suite avoids synthetic shortcuts, prioritizes sustained generation, and measures task completion rather than token throughput. Each category includes prompt templates, evaluation rubrics, retry logic, and seed control to ensure reproducibility.

Coding workloads test multi-file refactoring, debugging, architecture design, and syntax generation. Prompts include complex codebases, cross-module dependencies, and error logs requiring diagnostic reasoning. The model must decompose the problem, generate intermediate analysis steps, verify syntax and logic, and produce structured output. Evaluation measures code correctness, compilation success rate, architectural coherence, and reasoning budget utilization.

RAG workloads test document retrieval, synthesis, hallucination resistance, and context window utilization. Prompts include large document corpora, overlapping information, and contradictory statements. The model must ingest context, extract relevant information, synthesize responses, and cite sources. Evaluation measures retrieval accuracy, synthesis coherence, hallucination rate, and context fit. The Flight Recorder tracks KV cache pressure, attention head activation, and draft acceptance rates during retrieval phases.

Agentic workloads test tool calling, state management, multi-step workflows, and error recovery. Prompts require the model to plan actions, execute tool calls, parse responses, adjust strategy, and complete tasks. Evaluation measures plan validity, tool usage accuracy, state consistency, and budget utilization. The proxy captures tool call logs, response routing, and retry patterns. The runtime tracks draft alignment during planning phases.

Server-admin workloads test log analysis, configuration generation, troubleshooting, and policy enforcement. Prompts include system logs, error traces, configuration files, and security policies. The model must parse unstructured data, identify root causes, generate remediation steps, and produce structured output. Evaluation measures diagnostic accuracy, configuration validity, policy compliance, and reasoning budget efficiency.

Chat assistant behavior tests conversational coherence, memory retention, tone adaptation, and multi-turn consistency. Prompts include extended dialogues, persona switches, context carryover, and preference tracking. Evaluation measures response relevance, memory accuracy, tone consistency, and budget utilization. The Flight Recorder captures turn-by-turn timing, context window growth, and draft acceptance variance.

Creative writing tests narrative consistency, stylistic control, long-form generation, and structural integrity. Prompts include story outlines, character development, plot progression, and stylistic constraints. Evaluation measures narrative coherence, stylistic adherence, repetition detection, and output reliability. The runtime tracks generation stability, token distribution, and budget exhaustion patterns.

Long-output reliability tests 10,000+ token generation, structural integrity, attention decay, and coherence maintenance. Prompts require extended essays, technical documentation, or multi-chapter narratives. Evaluation measures structural consistency, repetition rate, factual accuracy, and budget utilization. The Flight Recorder captures KV cache eviction patterns, RoPE scaling behavior, and draft acceptance rates over extended generation.

Context fit tests how effectively the model utilizes the 262,144-token window. Prompts include large documents, multi-source synthesis, and long conversational state. Evaluation measures information retention, retrieval accuracy, attention distribution, and memory efficiency. The runtime tracks context window saturation, attention head activation profiles, and draft alignment during retrieval phases.

MTP vs non-MTP behavior tests speculative decoding performance, draft acceptance rates, verification latency, and quality trade-offs. Prompts are identical across runs, with MTP enabled and disabled. Evaluation measures token throughput, latency distribution, acceptance rate variance, and output quality. The Flight Recorder captures draft model overhead, queue synchronization latency, and budget utilization under both configurations.

Each workload category includes automated scoring rubrics, manual review checkpoints, and statistical validation methods. The suite runs on a controlled seed, uses deterministic decoding where possible, and tracks variance across multiple runs. The Flight Recorder captures all telemetry, enabling granular analysis of runtime behavior, proxy overhead, and model performance.

# What To Measure

Measurement is the foundation of reproducible evaluation. The Flight Recorder captures a comprehensive set of metrics, each designed to reflect specific aspects of model behavior, runtime performance, and proxy operation. These metrics are structured into latency, throughput, quality, resource utilization, and proxy overhead categories.

Time to First Token (TTFT) measures the duration from prompt submission to the first generated token. It reflects prefill computation, context window loading, and proxy routing latency. TTFT is critical for interactive workloads, where user experience depends on rapid initial response.

Time Between Tokens (TBT) measures the interval between consecutive generated tokens. It reflects decode computation, MTP verification overhead, and queue synchronization latency. TBT variance indicates generation stability and draft alignment quality.

Total Latency measures the duration from prompt submission to generation completion. It aggregates TTFT, TBT, and proxy overhead. Total latency is essential for batch workloads, where end-to-end timing determines system throughput.

Token Throughput measures the number of tokens generated per second. It is calculated as total tokens divided by total latency. Throughput reflects computational efficiency but must be interpreted alongside quality metrics to avoid misleading conclusions.

Reasoning Budget Utilization measures the percentage of the 8192-token budget consumed during generation. It tracks allocation patterns, early stopping signals, and budget exhaustion events. High utilization indicates effective deliberation; low utilization may indicate premature truncation or inefficient token usage.

Context Window Utilization measures the percentage of the 262,144-token window occupied by prompt, KV cache, and generated tokens. It tracks memory pressure, attention computation overhead, and RoPE scaling behavior. High utilization indicates effective context management; low utilization may indicate unused capacity or inefficient allocation.

MTP Acceptance Rate measures the percentage of draft tokens accepted by the target model during verification. It reflects draft alignment, verification latency, and throughput improvement. High acceptance rates indicate effective speculative decoding; low rates indicate misalignment or excessive verification overhead.

KV Cache Memory tracks VRAM allocation for attention states, key-value pairs, and cache eviction patterns. It monitors fragmentation, pool utilization, and memory pressure. Stable KV cache allocation indicates efficient context management; fragmentation indicates potential performance degradation.

Proxy Overhead measures the latency and resource cost introduced by the Flight Recorder. It tracks request routing, metadata capture, timing hooks, and response preview sampling. Low overhead indicates efficient proxy design; high overhead indicates bottlenecks or excessive telemetry collection.

Hallucination Rate measures the frequency of factually incorrect or unsupported claims in generated output. It is evaluated using automated fact-checking rubrics and manual review. Low hallucination rates indicate reliable generation; high rates indicate reasoning budget exhaustion or context window saturation.

Repetition Rate measures the frequency of duplicated phrases, paragraphs, or structural patterns in generated output. It is calculated using n-gram analysis and structural coherence scoring. Low repetition rates indicate stable generation; high rates indicate attention decay or budget mismanagement.

Error Rate measures the frequency of generation failures, including out-of-memory errors, timeout exceptions, and proxy routing failures. It tracks runtime stability, queue synchronization issues, and hardware thermal throttling. Low error rates indicate robust operation; high rates indicate configuration issues or hardware constraints.

These metrics are captured continuously, aggregated per workload category, and stored in structured formats for later analysis. The Flight Recorder ensures that all telemetry remains local, with only sanitized aggregates prepared for future publication. Measurement is not an afterthought; it is the core of the evaluation framework.

# How To Read The Charts

Charts are the primary visualization tool for interpreting benchmark results. They transform raw telemetry into actionable insights, but they require careful interpretation to avoid misreading variance, confusing correlation with causation, or misattributing proxy overhead to model behavior. The following guidelines establish a framework for chart reading that prioritizes accuracy, context, and practical utility.

Latency distributions should be read as histograms or violin plots, not single-point averages. Averages mask variance, which is critical in home-lab environments where thermal throttling, PCIe bandwidth contention, and driver scheduling can cause significant fluctuation. Look for tails, not just means. A model with a low average latency but a long tail may be unreliable for interactive workloads. A model with a slightly higher average but tight distribution may be preferable for production-like testing.

Throughput curves should be read alongside quality metrics. High token throughput is meaningless if output quality degrades. Look for inflection points where throughput increases but hallucination rate or repetition rate also rises. These points indicate trade-offs between speed and reliability. The goal is not maximum throughput, but optimal throughput for the target workload.

Budget utilization heatmaps should be read as time-series or stacked bar charts. High utilization is not inherently good; it must be correlated with task completion and output quality. A model that consumes 90% of its budget but produces incoherent output is mismanaging deliberative space. A model that consumes 60% but produces accurate, structured output is using budget efficiently. Look for patterns, not single data points.

MTP acceptance rate charts should be read alongside verification latency. High acceptance rates are desirable, but only if verification overhead does not negate throughput gains. Look for acceptance rate stability across different workload categories. A model with high acceptance rates for coding but low rates for RAG indicates draft alignment issues. The chart should reflect consistency, not isolated peaks.

Context window utilization charts should be read as memory pressure indicators. High utilization is not inherently bad; it reflects effective context management. However, it must be correlated with attention decay and retrieval accuracy. A model that utilizes 80% of its context window but fails to retrieve relevant information is misallocating attention. Look for utilization patterns that match workload complexity.

Proxy overhead charts should be read as baseline measurements. The Flight Recorder adds latency, but it should be consistent across runs. If overhead fluctuates significantly, it indicates telemetry bottlenecks, queue contention, or driver scheduling issues. The chart should reflect stability, not variance.

Common pitfalls include confusing prefill time with decode time, ignoring draft model overhead, misreading context window saturation, and attributing proxy latency to model inference. To avoid these pitfalls, always read charts in context, correlate multiple metrics, and validate findings with manual review. Charts are tools, not oracles. They guide interpretation, but they do not replace rigorous evaluation.

# Privacy Boundaries

Privacy is not an afterthought; it is a foundational constraint of this evaluation framework. Raw telemetry, prompt logs, response payloads, and metadata contain sensitive information that must remain strictly local. The Flight Recorder is designed with privacy-by-default architecture, ensuring that no raw data leaves the home-lab environment unless explicitly sanitized and aggregated.

Prompt logs contain user input, system instructions, tool calls, and contextual data. These may include proprietary code, configuration files, personal information, or sensitive business logic. Response payloads contain generated output, which may include hallucinated claims, factual errors, or unintended disclosures. Metadata contains timing information, VRAM allocation patterns, and proxy routing logs, which may reveal hardware constraints, workload complexity, or operational patterns.

The sanitization pipeline operates locally, using regex-based redaction, LLM-assisted filtering, and hash-based deduplication. PII, code snippets, proprietary configs, and prompt templates are stripped or obfuscated. Response payloads are summarized or truncated to remove sensitive content. Metadata is aggregated to preserve statistical utility while eliminating identifiable patterns. The pipeline runs before any data is prepared for future publication.

Aggregate-only sharing is the core privacy principle. Only statistical summaries, anonymized distributions, and methodology transparency are shared. Raw logs, prompt-response pairs, and unredacted telemetry are never published. This approach aligns with home-lab ethics, avoids accidental data leakage, and maintains trust with the community.

Technical implementation includes secure local storage, encrypted telemetry databases, and access-controlled export mechanisms. The Flight Recorder logs are stored in Parquet or CSV format, with encryption at rest and strict file permissions. Export scripts validate sanitization before generating public artifacts. The pipeline is auditable, reproducible, and transparent.

Privacy boundaries are not limitations; they are design constraints that improve evaluation quality. By keeping raw data local, practitioners avoid contamination from external networks, reduce attack surface, and maintain control over sensitive information. By sharing only sanitized aggregates, the community gains access to rigorous methodology without compromising privacy. This approach ensures that the evaluation framework is both scientifically valid and ethically sound.

# Expected Model Categories

The evaluation framework is designed to accommodate a range of open-weight model families, not just the target model. While `Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf` is the active first target, the Flight Recorder proxy and benchmark suite are architecture-agnostic, enabling comparison across different model families, quantization schemes, and reasoning paradigms.

Llama-family models offer dense architectures, strong general alignment, and extensive community support. They are well-suited for chat assistant behavior, creative writing, and server-admin workloads. Quantization trade-offs vary widely, with Q4_K_M offering balanced accuracy and memory efficiency, while IQ2_XS prioritizes extreme compression at the cost of reasoning quality.

Mistral-family models emphasize instruction following, tool-use planning, and agentic workflows. They are optimized for coding, RAG, and multi-step verification. Their attention mechanisms and RoPE scaling make them well-suited for extended context windows, though draft alignment may require tuning.

Gemma-family models focus on efficiency, low-latency inference, and educational use cases. They are well-suited for chat assistant behavior, long-output reliability, and context fit testing. Their smaller parameter counts make them ideal for budget-constrained environments, though reasoning budget utilization may differ from larger models.

Command R-family models emphasize retrieval-augmented generation, multi-document synthesis, and enterprise workflows. They are optimized for RAG, server-admin workloads, and agentic tool calling. Their training objectives align closely with real-world information retrieval, making them ideal for context fit and hallucination rate testing.

Specialized coding and math models focus on syntax generation, algorithmic reasoning, and structured output. They are well-suited for coding workloads, long-output reliability, and MTP vs non-MTP behavior testing. Their training data and objective functions prioritize precision over generality, requiring careful benchmark design to avoid skewing results.

Quantization trade-offs are a critical consideration across all families. IQ2_XS and Q3_K_M prioritize memory efficiency, making them suitable for 32GB VRAM constraints, but may degrade reasoning quality. Q4_K_M and Q5_K_M offer balanced accuracy and throughput, ideal for extended generation workloads. Q8_0 preserves maximum fidelity but may exceed VRAM limits for larger models. The Flight Recorder captures quantization impact on TTFT, TBT, acceptance rates, and hallucination rates, enabling data-driven quantization selection.

Architecture considerations include MoE vs dense routing, draft model pairing, and context scaling strategies. MoE models activate sparse parameter subsets, improving throughput but requiring careful budget allocation. Dense models provide consistent performance but may exceed VRAM limits. Draft model pairing must align with target model reasoning patterns to maintain high acceptance rates. Context scaling strategies must balance RoPE accuracy, attention decay, and memory pressure.

The Flight Recorder enables standardized model comparison by providing consistent routing, unified metrics, and proxy-level abstraction. Practitioners can swap models, adjust quantization, and evaluate performance without reconfiguring the benchmark suite. This flexibility ensures that the evaluation framework remains relevant as new models emerge and hardware constraints evolve.

# Limits And Caveats

No evaluation framework is perfect. The R9700 server, Vulkan runtime, and Flight Recorder proxy provide a robust foundation, but they operate within inherent constraints that must be acknowledged. These limits are not failures; they are boundaries that define the scope of valid inference.

Hardware constraints are the most immediate limitation. Thirty-two gigabytes of VRAM is sufficient for mid-sized models, but it restricts concurrent workloads, limits batch sizes, and increases sensitivity to memory fragmentation. Thermal throttling can reduce compute performance during extended generation, while PCIe Gen4 bandwidth may bottleneck data transfer between CPU and GPU. These constraints are real, but they reflect the reality of home-lab environments, not cloud data centers.

Software stack maturity introduces variability. Vulkan driver support varies across GPU architectures, with some drivers offering optimized compute shaders while others introduce scheduling overhead. `llama.cpp` version drift can alter KV cache management, MTP verification logic, and reasoning budget allocation. Experimental features like MTP draft decoding are powerful but require careful tuning to avoid performance degradation. These variables must be tracked, documented, and controlled.

Benchmark limitations exist in all evaluation frameworks. Synthetic benchmarks rarely capture real-world complexity, and even well-designed suites may miss edge cases. Prompt sensitivity can skew results, with minor wording changes altering output quality. Evaluation subjectivity remains a challenge, particularly for creative writing, chat assistant behavior, and agentic workflows. The Flight Recorder mitigates these issues through automated scoring, manual review checkpoints, and statistical validation, but it cannot eliminate them entirely.

Proxy overhead is an inherent cost of observability. The Flight Recorder captures telemetry, routes traffic, and logs metadata, all of which introduce latency. While optimized for minimal impact, the proxy adds a measurable overhead that must be accounted for in timing metrics. Practitioners must distinguish between model inference latency and proxy routing latency to avoid misattribution.

Generalizability is limited by hardware and runtime specificity. Results from the R9700 server with Vulkan backend and `llama.cpp` runtime may not transfer directly to cloud environments, different GPU architectures, or alternative inference frameworks. The evaluation framework is designed for home-lab reproducibility, not universal applicability. Practitioners should adapt the methodology to their specific stack, not assume direct transferability.

Responsible use is a critical consideration. Open-weight models may have licensing restrictions, ethical guidelines, or usage limitations. Agentic deployments require careful oversight, as tool-use planning and state management can lead to unintended consequences. The evaluation framework prioritizes transparency, reproducibility, and ethical compliance, but practitioners must exercise judgment when deploying models in production-like environments.

These limits are not weaknesses; they are boundaries that define the scope of valid inference. Acknowledging them ensures that the evaluation framework remains scientifically rigorous, practically useful, and ethically sound.

# Next Experiments

The evaluation framework is designed to evolve. As real data is collected, as hardware constraints are tested, and as new models emerge, the next experiments will expand the scope, refine the methodology, and deepen the analysis. These experiments are not guarantees; they are pathways for continued exploration.

Scale testing will explore multi-GPU configurations, CPU offloading strategies, and mixed-precision inference. The goal is to determine whether 32GB VRAM constraints can be mitigated through distributed computation, dynamic memory allocation, or hybrid CPU-GPU pipelines. Experiments will measure latency distribution, memory fragmentation, and thermal throttling under extended workloads.

MTP optimization will focus on draft model selection, token prediction algorithms, and speculative decoding variants. The goal is to improve acceptance rates, reduce verification overhead, and align draft models with target model reasoning patterns. Experiments will test alternative draft architectures, dynamic token allocation, and adaptive verification strategies.

Proxy enhancements will introduce dynamic routing, load balancing, caching mechanisms, and rate limiting. The goal is to reduce proxy overhead, improve telemetry collection efficiency, and enable multi-client testing. Experiments will measure routing latency, cache hit rates, and queue synchronization performance.

Benchmark automation will integrate CI/CD pipelines, nightly runs, and regression tracking. The goal is to ensure reproducibility, track variance over time, and enable continuous evaluation. Experiments will test automated prompt generation, scoring rubric validation, and statistical significance testing.

Community sharing will focus on sanitized datasets, methodology documentation, and reproducible scripts. The goal is to enable peer validation, foster collaboration, and establish standardized evaluation frameworks. Experiments will test data anonymization pipelines, export script validation, and community contribution workflows.

Long-term vision positions the home-lab as a research testbed, not just a consumer playground. The goal is to establish open evaluation frameworks, standardized reasoning budgets, and transparent benchmarking practices. Experiments will track hardware evolution, software stack maturity, and community adoption, ensuring that the evaluation framework remains relevant as the landscape shifts.

These experiments are not isolated; they are interconnected pathways that build upon the foundation established by this draft. As real data is collected, as constraints are tested, and as new models emerge, the framework will evolve to meet the demands of real-world AI inference. The goal is not perfection; it is progress. The goal is not publication; it is understanding. The goal is not speed; it is rigor. The R9700 server, Vulkan runtime, and Flight Recorder proxy provide the foundation. The next experiments will build upon it.