# Working Title
Benchmarks, Budgets, and Brains: Evaluating Open-Weight GGUF Models on a 32GB GPU with AI Flight Recorder

# Thesis
Running open-weight GGUF models on a 32GB-class GPU sits at the intersection of aspiration and constraint. The hardware can comfortably hold mid-sized models at moderate quantization levels, but it forces deliberate trade-offs between context window depth, generation speed, and answer fidelity. Token efficiency and answer quality are not opposing goals; they are coupled dimensions of the same system. A model that burns tokens lazily wastes compute, while a model that hoards brevity at the cost of correctness wastes user time and trust. AI Flight Recorder provides the telemetry layer needed to observe this coupling in real time, separating prompt ingestion from generation, tracking internal reasoning steps, and logging speculative decoding acceptance rates. Until real runs are executed, every number, latency curve, and quality annotation in this draft is synthetic and marked as such. The purpose is to establish a rigorous evaluation framework that home-lab operators can populate with actual data, rather than chasing vanity metrics or defaulting to single-turn toy prompts. The core argument is straightforward: on constrained hardware, the optimal configuration is the one that maximizes meaningful quality per token spent, not the one that minimizes tokens at all costs, nor the one that maximizes raw throughput while degrading output fidelity.

# The Lab Setup
A 32GB GPU home-lab environment typically centers around a consumer flagship or a retired enterprise accelerator. The VRAM budget is the primary constraint, dictating maximum context length, batch size, and quantization ceiling. A realistic stack includes:
- GPU: 32GB VRAM accelerator (e.g., RTX 3090/4090, or equivalent used hardware)
- Inference Engine: llama.cpp or a compatible backend supporting GGUF, with optional MLC or vLLM integration for mixed precision or CUDA graph optimizations
- Quantization Targets: Q4_K_M, Q5_K_M, Q6_K, and occasionally Q8_0 for comparison
- Telemetry Layer: AI Flight Recorder, configured as a local daemon or sidecar process that intercepts inference streams, logs token counts, timestamps, memory allocations, speculative decoding states, and quality annotations
- Storage/Network: NVMe dataset cache for rapid model loading, local JSONL/Parquet export pipelines, and a lightweight dashboard for visualizing Flight Recorder outputs

The AI Flight Recorder integration is the differentiator. Rather than relying on terminal stdout or third-party web UIs that obscure internals, Flight Recorder captures structured telemetry at the token level. It records:
- Prompt tokens consumed vs. GPU memory pinned
- Generation tokens emitted, including internal reasoning vs. final answer tokens (when the model outputs explicit chain-of-thought or verification steps)
- Latency breakdown: prefill time, first-token latency, inter-token latency, and speculative draft/verify cycles
- VRAM utilization, context window fragmentation, and offload patterns
- Quality flags annotated by the operator or automated heuristic checks (syntax validation, factuality markers, instruction adherence)

All telemetry is exported in a schema-agnostic format that can be replayed, filtered, and correlated across model versions and quantization levels. Until actual inference runs are performed, every metric placeholder, latency interval, and acceptance rate is synthetic. The architecture is designed to accept real data without schema changes, ensuring that the evaluation methodology remains stable as the lab matures.

# Why Toy Tests Failed
Home-lab evaluations often begin with single-turn prompts: “Write a poem,” “Solve 37 × 8,” or “Summarize this paragraph.” These tests are convenient but structurally misleading. They fail for several reasons:
- Context Window Illusion: Toy prompts rarely exceed 500 tokens. They do not stress KV cache management, attention head saturation, or quantization-induced drift at longer horizons.
- Reasoning Depth Avoidance: Trivial math or surface-level summarization does not trigger multi-step deduction, self-correction, or cross-reference verification. Quantization artifacts that manifest during complex reasoning remain invisible.
- Token Efficiency Distortion: A model can appear highly efficient on a toy prompt because the generation window is short. This masks inter-token latency degradation, speculative decoding inefficiency, or context window thrashing that appears at 8K+ tokens.
- Quality Measurement Blindness: Short outputs are easy to skim. They do not reveal instruction-following drift, hallucination cascades, or domain-specific failure modes that emerge under sustained generation.
- Speculative Decoding Misalignment: MTP and similar techniques optimize for token prediction confidence. Toy prompts often sit in high-confidence regions, inflating acceptance rates while hiding draft-model mismatch during complex reasoning.

A rigorous evaluation replaces toy prompts with structured workloads: multi-document synthesis, code generation with test-case validation, constraint-satisfying drafting, and iterative refinement loops. These workloads stress the same dimensions that matter in production-adjacent home-lab use cases. Until real runs populate the telemetry, all workload examples and failure modes described here are synthetic. The goal is to establish a testing protocol that captures the full inference lifecycle, not just the final output.

# Reasoning Tokens Versus Final Tokens
Modern open-weight models frequently emit internal reasoning steps before delivering a final answer. These may appear as explicit chain-of-thought, self-questioning, verification passes, or structured reasoning blocks. From a telemetry perspective, this creates a critical distinction:
- Reasoning Tokens: Tokens generated to work through the problem, validate assumptions, or draft intermediate states. They consume compute, VRAM, and time, but they do not always reach the user.
- Final Tokens: Tokens that constitute the delivered answer, summary, code block, or structured output. They are what the user reads and what typically drives satisfaction metrics.

AI Flight Recorder can parse this distinction when the model follows a consistent formatting convention or when the inference engine exposes reasoning boundaries. The telemetry pipeline separates prompt ingestion, reasoning generation, and final output emission, logging timestamps, token counts, and memory states for each phase. This separation matters because:
- Compute Allocation: Reasoning tokens often dominate total generation cost. A model that spends 70% of its tokens reasoning may deliver a marginally better answer, but the efficiency trade-off must be explicit.
- Latency Perception: Users perceive latency based on first-token time and final output completion. Reasoning steps hidden behind a single prompt may improve accuracy without degrading perceived responsiveness, but they still consume GPU cycles.
- Context Window Pressure: Reasoning tokens occupy KV cache slots. On a 32GB GPU, this can truncate available context for the prompt or subsequent turns, forcing trade-offs between depth and breadth.
- Quality Attribution: When an answer fails, knowing whether the failure occurred during reasoning or final emission helps diagnose quantization limits, prompt ambiguity, or model architecture constraints.

Until real runs are executed, all token-split examples, latency intervals, and cache pressure metrics are synthetic. The framework is designed to accommodate models that emit explicit reasoning, implicit reasoning, or no reasoning at all. The evaluation does not penalize reasoning tokens; it logs them, correlates them with final output quality, and lets the operator decide whether the trade-off aligns with their workload.

# Quality Per Token
Token efficiency and answer quality are frequently treated as rivals. In practice, they are coupled variables that must be optimized jointly. Quality per token is not a single metric; it is a composite of factuality, instruction adherence, coherence, safety, and domain accuracy, normalized against the total tokens consumed. The relationship follows several observable patterns:
- Diminishing Returns: Beyond a certain token budget, additional generation rarely improves correctness. It may add verbosity, hedge unnecessarily, or drift into repetition. Quality per token declines even if absolute quality plateaus.
- Reasoning Premium: Models that allocate extra tokens to verification, cross-checking, or structured reasoning often achieve higher absolute quality. The quality per token may dip during reasoning, but the final output quality can justify the expenditure, especially for complex or high-stakes tasks.
- Terse Excellence: A model that delivers a concise, accurate answer using fewer tokens can outperform a verbose model that burns context and compute for marginal gains. Brevity reduces latency, preserves VRAM for subsequent turns, and improves user experience without sacrificing correctness.
- Quantization Sensitivity: Lower quantization levels (e.g., Q4_K_M) may appear efficient but degrade quality per token on reasoning-heavy workloads. Higher quantization (Q6_K/Q8_0) increases token cost but can restore fidelity, shifting the quality-per-token curve favorably for complex tasks.

AI Flight Recorder enables quality-per-token analysis by logging generation length alongside operator-annotated or heuristic-derived quality flags. The telemetry pipeline calculates:
- Effective token cost (prompt + reasoning + final)
- Quality score (manual annotation, automated linting, or domain-specific validators)
- Quality per token ratio (quality score divided by effective token cost)
- Confidence intervals across repeated runs to account for stochastic variation

Until real runs populate the dataset, all quality scores, ratios, and confidence intervals are synthetic placeholders. The methodology emphasizes reproducibility: fixed prompts, controlled temperature/top_p settings, repeated trials, and consistent annotation rubrics. The goal is not to declare a winner, but to map the trade-off surface so operators can select configurations aligned with their actual workloads.

# MTP Acceptance
Multi-Token Prediction (MTP) and related speculative decoding techniques accelerate generation by having a smaller draft model propose multiple tokens, which the main model verifies in parallel. The efficiency gain hinges on the acceptance rate: the fraction of draft tokens that survive verification without triggering recomputation. AI Flight Recorder logs MTP telemetry at granular levels:
- Draft tokens proposed per step
- Verified tokens accepted
- Verification cycles triggered
- Draft-to-verify time ratio
- Overall throughput acceleration factor

Acceptance rate is highly workload-dependent. Factual, deterministic prompts often yield high acceptance because the draft model’s distribution closely matches the main model’s. Reasoning-heavy, constrained, or creative prompts often yield lower acceptance because the main model must correct divergent paths. On a 32GB GPU, MTP introduces a secondary memory footprint for the draft model, which must be accommodated within the VRAM budget alongside the main model and KV cache.

AI Flight Recorder captures MTP acceptance as a time-series metric, correlating it with latency curves, token efficiency, and quality flags. The telemetry pipeline also logs:
- Acceptance rate per prompt segment
- Draft model mismatch frequency
- Verification overhead vs. compute savings
- Context window impact when MTP alters generation pacing

Until real runs are executed, all acceptance rates, throughput multipliers, and draft/verify time splits are synthetic. The framework does not assume MTP is universally beneficial; it logs the conditions under which it helps or hurts, allowing operators to toggle speculative decoding based on workload characteristics. A high acceptance rate with degraded quality is worse than a low acceptance rate with preserved correctness. MTP is a tool, not a goal.

# What Screenshots Should Show
Documentation and reproducibility require clear, structured screenshots that capture the inference lifecycle without obscuring internals. AI Flight Recorder exports should be framed to show:
- Telemetry Dashboard Overview: Token counters (prompt, reasoning, final), latency breakdown, VRAM utilization, and MTP acceptance rate at a glance
- Time-Series Graphs: Generation tokens over time, acceptance rate fluctuations, inter-token latency spikes, and context window pressure
- Quality Annotation Panel: Operator flags, automated validation results, and quality-per-token calculations aligned with specific prompts
- KV Cache Visualization: Memory allocation per layer, offload patterns, and context truncation boundaries when limits are approached
- Speculative Decoding Breakdown: Draft vs. verify token streams, mismatch frequency, and compute savings vs. overhead
- Export Schema Sample: JSONL/Parquet structure demonstrating how Flight Recorder logs token-level events, timestamps, and quality metadata

All screenshot placeholders must be explicitly marked as synthetic until actual runs are performed. The documentation protocol emphasizes:
- Consistent prompt formatting across runs
- Fixed sampling parameters (temperature, top_p, frequency penalty)
- Repeated trials for statistical stability
- Clear labeling of quantization level, model version, and inference engine
- Annotation of workload type (reasoning, code, summarization, constraint satisfaction)

Screenshots are not decorative; they are evidence. They should allow a peer to reconstruct the evaluation conditions, verify the telemetry pipeline, and assess whether the observed trade-offs align with the thesis. Synthetic examples are provided only to illustrate layout, labeling conventions, and data correlation methods.

# Caveats
Several factors limit generalization and require careful documentation:
- Quantization Artifacts: Lower quantization levels may preserve efficiency but degrade reasoning fidelity, safety boundaries, or domain-specific accuracy. Effects vary by architecture and training data.
- VRAM Fragmentation: Context window growth, batch sizing, and MTP draft models can fragment VRAM, causing unexpected offloads or context truncation that skew telemetry.
- Stochastic Variation: Sampling introduces variance. Single runs are insufficient; repeated trials with fixed seeds or controlled randomness are necessary for stable metrics.
- Quality Subjectivity: Automated heuristics cannot fully capture nuance, domain expertise, or instruction intent. Operator annotation remains essential, and rubrics must be explicit.
- AI Flight Recorder Parsing Limits: Telemetry accuracy depends on model output formatting, inference engine hooks, and parsing rules. Unstructured reasoning or non-standard token boundaries may require manual correction.
- Hardware Throttling: Thermal limits, power delivery, and PCIe bandwidth can affect sustained throughput, especially during long generations or repeated runs.
- Model Architecture Differences: Attention patterns, normalization layers, and vocabulary design influence token efficiency, reasoning behavior, and MTP acceptance. Cross-model comparisons require normalized workloads.
- Generalization Boundaries: Findings on one model family or quantization level do not transfer to another. Evaluation must remain workload-specific and version-locked.

All caveats are documented to prevent overgeneralization. Synthetic examples and placeholder metrics are explicitly marked to avoid misinterpretation. The framework is designed to absorb real data without restructuring, ensuring that observations remain grounded in actual hardware behavior.

# Draft Conclusion
Running open-weight GGUF models on a 32GB-class GPU is a discipline of deliberate trade-offs. Token efficiency and answer quality are not competing objectives; they are interdependent variables that must be optimized together. A model that burns tokens lazily wastes compute and context, while a model that hoards brevity at the cost of correctness wastes user trust and iteration cycles. AI Flight Recorder provides the telemetry layer needed to observe this coupling, separating reasoning from final output, logging speculative decoding acceptance, and correlating quality with token expenditure. Until real runs populate the dataset, every number, latency curve, and quality annotation in this draft is synthetic and marked as such. The framework does not prescribe a single optimal configuration; it maps the trade-off surface so operators can select settings aligned with their actual workloads. On constrained hardware, the best model is not always the fastest, nor always the most concise. It is the one that delivers meaningful quality per token spent, preserves VRAM for sustained use, and provides clear telemetry to guide future iterations. Home-lab AI matures not by chasing benchmarks, but by measuring carefully, documenting thoroughly, and iterating with intent. When real data replaces synthetic placeholders, the evaluation will reveal which configurations earn their compute, which workloads justify their token budgets, and which models truly belong in the 32GB lab.