# Working Title
Beyond the Benchmarks: Evaluating Open-Weight GGUF Models on a 32GB GPU with AI Flight Recorder

# Thesis
The era of judging local inference solely by raw inference speed (tokens per second) or subjective answer length is over. With the proliferation of open-weight GGUF models and the availability of consumer-grade 32GB-class GPUs, we have the hardware to run models that previously required enterprise datacenter resources. However, raw speed is a deceptive metric. A model that generates 100 tokens per second while hallucinating or failing to reason through a complex prompt is less useful than a model generating 40 tokens per second that provides a deeply reasoned, accurate response. By leveraging AI Flight Recorder—a telemetry and observability framework for local LLM inference—we can move beyond superficial benchmarks. We can measure the ratio of reasoning tokens to final tokens, evaluate quality per token, and understand the mechanics of speculative and multi-token prediction. This article explores how to build a rigorous, grounded evaluation framework for home-lab inference, where token efficiency and answer quality are not mutually exclusive, but deeply intertwined.

# The Lab Setup
Before diving into the metrics, we must establish the baseline hardware and software configuration. The "32GB-class GPU" is the sweet spot for local inference. It allows for full GPU offloading of mid-sized models (7B–13B parameters) at high quantization levels, while still leaving enough VRAM for extended context windows or multi-token prediction buffers.

For this evaluation, we are using a single 32GB-class workstation GPU (e.g., an NVIDIA RTX 6000 Ada Generation or a dual-GPU RTX 4090 setup acting as a unified 32GB pool via NVLink/shared memory). The system is paired with 64GB of DDR5 system RAM, which serves as a crucial offload target for the KV cache when context exceeds VRAM limits, though we will aim to keep the active context within VRAM to maintain high tokens-per-second (t/s) rates.

The software stack consists of:
1. **llama.cpp**: The backbone of GGUF inference, providing the foundational CPU/GPU offloading capabilities.
2. **AI Flight Recorder**: A custom telemetry daemon that hooks into the inference pipeline. It captures granular metrics including per-token generation time, KV cache memory allocation, speculative decoding acceptance rates, and prompt/reasoning token segmentation.
3. **The Models**: We are testing a selection of open-weight GGUF models, specifically:
   *   Llama-3-8B-Instruct (Q4_K_M and Q5_K_M)
   *   Mistral-7B-Instruct-v0.3 (Q4_K_M)
   *   Mixtral-8x7B-Instruct (Q4_K_M)

Quantization is set to Q4_K_M as our baseline. This is the pragmatic choice for home labs, offering a massive reduction in VRAM footprint (fitting 8B and 13B models comfortably) with minimal degradation in reasoning capability compared to Q8_0. The AI Flight Recorder is configured to log telemetry at a 100ms interval, capturing the state of the GPU, the KV cache, and the token generation stream.

*Note: Until real runs are executed, all specific metric values, graphs, and screenshots referenced in this article are synthetic placeholders. They represent the *format* of the data we expect to collect, not actual results.*

# Why Toy Tests Failed
Historically, home-lab benchmarking has relied on "toy tests." You take a prompt—usually something generic like "Write a haiku about a robot" or "Explain quantum entanglement in simple terms"—and you time how long it takes for the model to output. You then subjectively grade the answer. If the answer looks decent and the t/s is high, you declare victory.

Toy tests fail for three critical reasons:
1. **The Illusion of Competence**: LLMs are incredibly good at sounding confident. A 7B model can generate a 50-token answer in 2 seconds that sounds brilliant but is factually wrong. A toy test rewards speed and confidence, not accuracy.
2. **Token Count Deception**: A model might output 100 tokens per second, but if 80 of those tokens are filler phrases ("It is important to note that...", "In conclusion..."), the actual information density is low. Conversely, a model outputting 40 tokens per second might be packing high-signal reasoning tokens into every single word. Toy tests don't measure signal density.
3. **Lack of Contextual Stress**: Toy tests rarely stress the KV cache. A 7B model might generate 500 tokens easily on a 32GB GPU, but what happens at 8,000 tokens? What happens at 32,000 tokens? The latency doesn't scale linearly; the KV cache grows, and the model's attention mechanism struggles to attend to earlier tokens. Without measuring KV cache utilization over time, you have no idea if your model is actually viable for real-world, long-context tasks.

AI Flight Recorder solves these problems by shifting the focus from *how fast* the model answers to *how the model answers*.

# Reasoning Tokens Versus Final Tokens
One of the most profound insights AI Flight Recorder provides is the ability to differentiate between reasoning tokens and final tokens. This distinction is vital for understanding why a model might appear "slow" but is actually performing a complex cognitive task.

When a model is prompted with a complex query—such as "Debug this Python script and explain why it fails," or "Solve this multi-step logic puzzle"—the model doesn't just output the answer. It generates a chain of thought. It reasons through the problem, identifies edge cases, and then formulates the final response.

In a standard llama.cpp run, these tokens are just a stream of text. But AI Flight Recorder, when integrated with reasoning-aware frameworks (like ReAct or structured prompt templates), can tag tokens based on their semantic function.
*   **Reasoning Tokens**: The internal monologue, the step-by-step breakdown, the self-correction.
*   **Final Tokens**: The actual answer provided to the user.

Let’s look at a synthetic example from AI Flight Recorder:
*   **Prompt**: "Solve this math problem: If a train leaves station A at 60mph..."
*   **Model A Output**: "The train arrives at station B in 4 hours." (10 tokens)
*   **Model B Output**: "Let's break this down. First, we need to find the distance... The distance is 240 miles. Next, we divide by speed... 240 / 60 = 4. The train arrives at station B in 4 hours." (80 tokens)

If we only look at speed, Model A is 8x faster. But if Model A is hallucinating the math, Model B is vastly superior. AI Flight Recorder allows us to calculate the **Reasoning Token Ratio (RTR)**:
`RTR = Reasoning Tokens / Final Tokens`

Model A has an RTR of 0 (0 reasoning tokens). Model B has an RTR of 7. This isn't just a metric; it's a diagnostic tool. If a model consistently has a high RTR, it means it is doing the heavy lifting. If it has a low RTR, it is either a very fast, direct-retrieval model, or it is a hallucinating model skipping the reasoning step.

This explains why a model using more tokens can still be worthwhile. If Model B's answer is 95% accurate and Model A's answer is 40% accurate, the 70 extra tokens in Model B are an investment, not a tax. They are the computational cost of reliability. For a home-lab user running local models, accuracy is paramount. A model that takes longer but doesn't lie is infinitely more useful than a model that answers instantly but fabricates facts.

# Quality Per Token
Building on the reasoning/final token distinction, we arrive at the core metric of our evaluation: **Quality Per Token (QPT)**.

QPT is a composite metric that combines objective accuracy scoring with token efficiency. It is calculated as:
`QPT = (Quality Score) / (Total Tokens Generated)`

The Quality Score is derived from a structured evaluation prompt, where the model's output is scored on a scale of 0-10 for accuracy, completeness, and relevance. The Total Tokens Generated is the sum of reasoning and final tokens.

Let’s apply this to our synthetic AI Flight Recorder data:
*   **Model A (Llama-3-8B-Q4_K_M)**: Quality Score = 6. Total Tokens = 50. QPT = 0.12
*   **Model B (Mixtral-8x7B-Q4_K_M)**: Quality Score = 8. Total Tokens = 120. QPT = 0.066
*   **Model C (Mistral-7B-Q4_K_M)**: Quality Score = 7. Total Tokens = 60. QPT = 0.116

At first glance, Model B looks terrible because its QPT is half of Model A's. It is generating 2.4x more tokens for a score that is only 33% higher. This is where the nuance comes in. Model B is a larger, more capable model (8x7B MoE) that is being forced to work harder to answer a complex prompt. The extra tokens are the "quality tax" of a deeper reasoning process. If we remove the quality score and just look at tokens, Model A is winning. But if we remove the token count and just look at quality, Model B is winning. QPT forces us to weigh both.

However, QPT also highlights the brilliance of the **terse model**. Model C (Mistral-7B) has a QPT of 0.116, almost matching Model A. It is generating fewer tokens than Model B, but delivering a significantly higher quality score. This is the sweet spot. Model C is highly efficient. It doesn't waste tokens on filler; it delivers high-signal answers with minimal overhead.

For a home-lab user, QPT is the ultimate tie-breaker. If you are running a chatbot for daily tasks, you want a high QPT. You want the model to answer quickly and accurately without unnecessary verbosity. If you are running a code interpreter or a math solver, you might tolerate a lower QPT (like Model B) because the reasoning tokens are necessary for correctness. AI Flight Recorder allows you to segment your QPT by task type, giving you a granular understanding of which models excel at which jobs.

# MTP Acceptance
Multi-Token Prediction (MTP) is the next frontier in local inference optimization, and AI Flight Recorder is essential for measuring its real-world impact. MTP is a speculative decoding technique where the model predicts the next $N$ tokens in a single forward pass, and then the main model verifies those tokens in parallel. If the tokens are accepted, they are written to the output buffer without sequential decoding, drastically cutting inference time.

However, MTP is not a magic bullet. It only works if the model is trained for it, and only if the acceptance rate is high. A low acceptance rate means the model is predicting wrong tokens, which forces the main model to backtrack and regenerate, potentially slowing down inference.

AI Flight Recorder captures MTP metrics at the token level:
*   `mtp_predicted_tokens`: The number of tokens predicted by the MTP head.
*   `mtp_accepted_tokens`: The number of those tokens that were verified and accepted by the main model.
*   `mtp_acceptance_rate`: `mtp_accepted_tokens / mtp_predicted_tokens`
*   `mtp_speedup`: The ratio of sequential decoding time vs. MTP decoding time.

In our synthetic setup, we are testing MTP on the Llama-3-8B model. The AI Flight Recorder logs show:
*   Average `mtp_acceptance_rate`: 0.65
*   Average `mtp_speedup`: 1.8x

This means that for every 10 tokens the MTP head predicts, 6.5 are accepted. Because the model is processing multiple tokens in parallel, the overall generation speed increases by 80%. This is a massive win. However, we must also track `mtp_rejection_penalties`. When a token is rejected, the model has to re-decode it sequentially. If the acceptance rate drops below 0.5, the MTP overhead (the cost of running the MTP head) outweighs the speedup, and inference actually slows down.

AI Flight Recorder allows us to visualize the MTP acceptance rate over the length of the prompt. We often see that MTP acceptance is high at the beginning of a response (where the model is confident) and drops off as the response gets longer and more complex (where the model starts hallucinating or losing coherence). This insight is crucial for tuning MTP. If the acceptance rate drops too low, we might need to reduce the MTP window size or switch to standard sequential decoding for the latter half of the response.

# What Screenshots Should Show
To make this evaluation transparent and reproducible, we need to document exactly what the AI Flight Recorder telemetry looks like. Since we are generating synthetic examples, these screenshots represent the *structure* of the data we expect to see.

**Screenshot 1: The Telemetry Dashboard**
The main dashboard should show a real-time graph of tokens/sec (t/s) on the Y-axis and time on the X-axis. We should see distinct phases:
*   **Phase 1: Prompt Processing**: The t/s is extremely high (e.g., 500+ t/s) as the model processes the input prompt. The AI Flight Recorder should overlay a shaded region indicating "Prompt Processing."
*   **Phase 2: Token Generation**: The t/s drops to the expected baseline (e.g., 40-60 t/s). The AI Flight Recorder should overlay a shaded region indicating "Token Generation."
*   **Phase 3: MTP Acceleration (if active)**: If MTP is enabled, we should see a sudden spike in t/s (e.g., 70-90 t/s) corresponding to the MTP acceptance rate. The AI Flight Recorder should overlay a shaded region indicating "MTP Acceleration."

**Screenshot 2: KV Cache Utilization Heatmap**
A heatmap showing VRAM usage over time. The X-axis is time, the Y-axis is VRAM. We should see a sharp spike during prompt processing as the KV cache is allocated, followed by a gradual, linear increase during token generation as the KV cache grows. This screenshot proves whether our 32GB GPU is actually holding the KV cache in VRAM or if it's offloading to system RAM. If the KV cache utilization line plateaus at the VRAM limit, we know the model is fully offloaded. If it spikes into the system RAM region, we know the context is too long, and we should see a corresponding drop in t/s.

**Screenshot 3: Reasoning vs. Final Token Ratio**
A bar chart comparing different models. The X-axis lists the models (Llama-3, Mistral, Mixtral). The Y-axis shows the token count. The bars are stacked: the bottom portion is "Reasoning Tokens" (colored blue), and the top portion is "Final Tokens" (colored green). This visualizes the RTR we discussed earlier. We expect to see the reasoning token portion grow significantly for complex prompts, especially in larger models like Mixtral.

**Screenshot 4: MTP Acceptance Rate Over Time**
A line graph showing the MTP acceptance rate over the course of a generation. The X-axis is the token number, the Y-axis is the acceptance rate (0.0 to 1.0). We expect to see the line start high (0.8-0.9) and gradually decline to 0.5-0.6 as the generation progresses. This screenshot is crucial for understanding MTP's real-world limits.

*Note: These screenshots are placeholders. Real runs will populate these graphs with actual data from our AI Flight Recorder setup.*

# Caveats
Before finalizing this evaluation framework, we must acknowledge the limitations and caveats of running AI Flight Recorder on a home-lab setup.

1. **AI Flight Recorder Overhead**: The telemetry daemon itself consumes CPU and memory. While we are measuring inference, we are also measuring the overhead of our measurement tool. We must ensure that the AI Flight Recorder is optimized and does not become a bottleneck that artificially lowers our t/s rates.
2. **Hardware Variability**: Home labs are not datacenters. A 32GB GPU on a PCIe 4.0 x16 slot will perform differently than one on a PCIe 3.0 x8 slot. System RAM speed and latency will also impact KV cache offloading. Our results are specific to our hardware and cannot be directly compared to others without normalizing for PCIe bandwidth and RAM latency.
3. **Quantization Artifacts**: We are using Q4_K_M quantization. This is a lossy compression scheme. While it preserves most of the model's capability, it introduces noise into the token embeddings. This noise can manifest as slightly higher perplexity or a slight increase in hallucination rates, which could skew our Quality Per Token metrics. We must compare models at the same quantization level to ensure fairness.
4. **Prompt Sensitivity**: LLMs are non-deterministic. The same prompt can yield different answers, and therefore different token counts and quality scores, on different runs. We must run each evaluation multiple times (at least 10-20) and use the median score, not the mean, to account for outliers. AI Flight Recorder's logging allows us to filter and analyze these multiple runs.
5. **MTP Model Availability**: Not all GGUF models support MTP. MTP requires the model to have a separate MTP head trained into it. We can only evaluate MTP on models specifically designed for it. For standard models, we are limited to speculative decoding with a draft model, which has a different telemetry profile.

# Draft Conclusion
The home-lab movement is no longer just about running large models on consumer hardware; it is about understanding how those models behave under real-world constraints. By leveraging AI Flight Recorder, we can move beyond the superficial metrics of raw speed and subjective answer length. We can measure the true cost of intelligence: the ratio of reasoning tokens to final tokens, the quality delivered per token, and the efficiency of speculative decoding.

A model that uses more tokens can be worthwhile if those tokens are reasoning tokens that lead to a meaningfully better answer. A terse model can be excellent if it delivers high-signal answers with minimal overhead. MTP can provide massive speedups, but only if the acceptance rate is high. By tracking these metrics, we can make informed decisions about which models to run, how to prompt them, and when to offload to system RAM.

The future of local inference is not just about faster t/s; it is about smarter t/s. With AI Flight Recorder, we have the tools to see the intelligence behind the tokens. The data will follow.