# Thesis The intersection of high-end consumer/server hardware and open-weight large language models has created a new paradigm for local AI inference. The central question driving this investigation is: what can a 32GB-class R9700 server actually do when tasked with running massive, open-weight GGUF models, specifically when those models are routed through a local AI Flight Recorder? This is not merely a hardware stress test; it is an exploration of practical utility. Can a local server, constrained by the physical limits of 32GB of VRAM, effectively handle the computational demands of models like Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf? And more importantly, can it do so in a way that provides actionable insights into model behavior, performance, and reliability? The answer lies not just in raw inference speed, but in the observability and control provided by the AI Flight Recorder. By capturing OpenAI-compatible proxy traffic, request metadata, response previews, timings, usage statistics, and benchmark artifacts, we can move beyond simple "does it run" metrics to a deeper understanding of "how well does it work." This article serves as a technical draft and a blueprint for a comprehensive benchmark suite. It outlines the hardware configuration, the runtime environment, the critical role of thinking budgets, the pitfalls of early testing, and the rigorous real-world test suite designed to evaluate the server's capabilities. By keeping private raw data local and only sharing sanitized aggregate findings, we ensure that the insights gained are both scientifically valid and privacy-preserving. # Hardware And Runtime The foundation of this experiment is the R9700 server, a high-performance workstation-class machine equipped with a 32GB-class GPU. The 32GB VRAM constraint is both a limitation and a design feature. It forces us to optimize for efficiency, pushing the boundaries of what can be achieved with quantized models and smart memory management. The GPU in question, likely an AMD Radeon architecture given the Vulkan runtime, provides the necessary compute power to handle the parallelized operations of large language models. The model under scrutiny is Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf. The naming convention suggests a complex architecture: a 35-billion parameter model, likely with a mixture of experts (A3B) and advanced multi-token prediction (MTP) capabilities. The GGUF format is the standard for quantized models in llama.cpp, and the "Balanced" designation implies a quantization level that balances memory usage with inference quality—likely Q4_K_M or similar. With 35B parameters quantized to 4 bits, the model weights consume approximately 17-18GB of VRAM. This leaves roughly 14GB for the KV cache, context window, and runtime overhead. The runtime environment is built around llama.cpp, configured for maximum performance and flexibility. The Vulkan backend is selected for its cross-platform compatibility and robust driver support, particularly on AMD hardware. Vulkan allows for fine-grained control over compute queues and memory synchronization, which is crucial for managing the heavy memory bandwidth demands of a 35B model. The context window is set to 262,144 tokens (256K). This is a massive context, far exceeding the typical 8K or 32K windows of older models. To accommodate this, llama.cpp utilizes KV cache offloading, storing portions of the context in system RAM when VRAM is exhausted. This introduces a latency trade-off, but it enables the server to handle long documents, extended conversations, and complex agentic workflows. MTP (Multi-Token Prediction) draft decoding is enabled. MTP is a speculative decoding technique where the model predicts multiple tokens in a single forward pass, or uses a draft model to propose tokens that the main model verifies. This dramatically improves throughput, as the model can generate multiple tokens per iteration rather than one. However, it also increases the complexity of the inference pipeline and requires careful tuning to avoid degrading output quality. Finally, the server is launched with the `--reasoning-budget 8192` flag. This is a critical parameter that limits the number of tokens the model is allowed to generate for reasoning steps (such as chain-of-thought). By capping the reasoning budget at 8192 tokens, we prevent the model from entering infinite loops of self-correction or excessive deliberation, ensuring that responses remain concise and actionable. # Why Thinking Budgets Matter The introduction of the `--reasoning-budget 8192` flag is not just a technical detail; it is a fundamental shift in how we interact with modern thinking models. Thinking models, such as those based on DeepSeek-R1 or similar architectures, are designed to "think" before they answer. They generate a sequence of reasoning tokens, evaluating the problem, breaking it down, and formulating a plan before producing the final output. This process is incredibly powerful for complex tasks like coding, mathematical reasoning, and multi-step problem solving. However, it comes with significant risks. Without a reasoning budget, a thinking model can generate thousands of tokens of intermediate reasoning. For a 35B model, this can quickly consume the 262K context window, leaving little room for the actual response. It can also lead to runaway inference, where the model gets stuck in a loop of self-correction, generating more and more tokens without reaching a conclusion. This not only wastes computational resources but also degrades the user experience, as the response time becomes unpredictable and often unacceptably long. The reasoning budget acts as a circuit breaker. By setting it to 8192 tokens, we ensure that the model has enough room to think through complex problems but is forced to produce a final answer when it reaches the limit. This is particularly important for the AI Flight Recorder, which needs to capture consistent and comparable data across different tasks. If one task generates 100 tokens of reasoning and another generates 10,000, comparing their performance becomes meaningless. The reasoning budget standardizes the output, allowing for fair benchmarking. Furthermore, the reasoning budget has implications for VRAM usage. While the KV cache is the primary consumer of VRAM, the reasoning tokens themselves must be stored in the context window. By limiting the reasoning budget, we reduce the peak memory footprint of the model, making it more stable and less prone to out-of-memory errors. This is especially critical on a 32GB VRAM server, where every megabyte counts. # Why Toy Benchmarks Failed In the early stages of this investigation, we encountered a series of failures that we initially attributed to hardware limitations or model instability. We ran a series of toy benchmarks—simple prompts designed to test basic functionality, such as "write a hello world program" or "summarize this paragraph." These benchmarks failed consistently, with the model either timing out or generating incomplete responses. The root cause was not the hardware or the model, but the benchmark parameters. Specifically, the `max_tokens` parameter was set too low. For thinking models, the reasoning budget is separate from the output token limit. If `max_tokens` is set to 512 or 1024, and the model uses 8192 tokens for reasoning, it will never have enough tokens left to generate a meaningful response. The model would spend its entire budget thinking and then be cut off before it could answer. This failure highlighted a critical misunderstanding of how thinking models operate. Traditional language models generate tokens sequentially, with the reasoning and output tokens being part of the same sequence. Thinking models, however, separate the reasoning phase from the output phase. The reasoning tokens are intermediate steps, and the output tokens are the final answer. When benchmarking thinking models, we must account for this separation and set `max_tokens` accordingly. To fix this, we adjusted the `max_tokens` parameter to be much higher, ensuring that the model had enough tokens to generate both its reasoning and its final answer. We also added logic to the AI Flight Recorder to dynamically calculate the `max_tokens` based on the `--reasoning-budget` and the expected output length. This adjustment resolved the failures and allowed us to move forward with more meaningful benchmarks. # The Real-World Test Suite With the basic issues resolved, we designed a comprehensive real-world test suite to evaluate the capabilities of the 32GB-class R9700 server. The suite is designed to cover a wide range of tasks, reflecting the diverse use cases that users might encounter in a home-lab setting. 1. Coding: We test the model's ability to write, debug, and optimize code. This includes generating scripts in Python, Bash, and Go, as well as debugging complex errors in existing code. The benchmark measures both the accuracy of the code and the time taken to generate it. 2. RAG (Retrieval-Augmented Generation): We test the model's ability to answer questions based on large documents. We use a corpus of technical documentation, legal contracts, and scientific papers, and ask the model to extract specific information. This tests the 262K context window and the model's ability to retrieve and synthesize information. 3. Agentic Work: We test the model's ability to plan and execute multi-step tasks. This includes automating system administration tasks, such as setting up a web server, configuring a firewall, or deploying a containerized application. The benchmark measures the model's ability to use tools, follow instructions, and adapt to errors. 4. Server-Admin Work: We test the model's ability to parse system logs, diagnose issues, and provide solutions. This includes analyzing Apache logs, monitoring system resources, and troubleshooting network connectivity. The benchmark measures the model's ability to understand complex system states and provide actionable advice. 5. Chat Assistant Behavior: We test the model's conversational abilities, including maintaining context over long conversations, adapting to different personas, and handling ambiguous or contradictory instructions. The benchmark measures the model's coherence and user-friendliness. 6. Creative Writing: We test the model's ability to generate creative content, such as stories, poems, and marketing copy. The benchmark measures the model's creativity, style, and consistency. 7. Long-Output Reliability: We test the model's ability to generate long, coherent outputs without degrading in quality. This includes generating long technical reports, detailed tutorials, and extensive codebases. The benchmark measures the model's ability to maintain focus and structure over long generations. 8. Context Fit: We test the model's ability to handle different types of input data, including code, natural language, and structured data. The benchmark measures the model's tokenization efficiency and attention mechanism performance. 9. MTP vs Non-MTP Behavior: We test the model's performance with and without MTP draft decoding. This includes comparing the speed, accuracy, and output quality of the model in both modes. The benchmark measures the impact of MTP on the inference pipeline. # What To Measure The AI Flight Recorder is the central component of this investigation, providing a wealth of data that we can use to evaluate the server's performance. The recorder captures the following metrics: - OpenAI-compatible proxy traffic: The Flight Recorder acts as a proxy, intercepting all API requests and responses. This allows us to analyze the raw data without modifying the client applications. - Request metadata: We capture the parameters of each request, including the model name, context length, reasoning budget, and MTP settings. - Response previews: We capture a preview of the generated response, allowing us to quickly assess the quality of the output without parsing the entire response. - Timings: We measure the time taken for each step of the inference pipeline, including prefill time, decode time, and time-to-first-token (TTFT). - Usage: We track the number of tokens generated, the VRAM utilization, and the RAM utilization for KV cache offloading. - Benchmark artifacts: We save the raw data from each benchmark run, including the input prompts, the generated responses, and the system logs. By analyzing these metrics, we can gain a deep understanding of the server's performance and identify areas for improvement. For example, if the TTFT is consistently high, it may indicate a bottleneck in the prefill phase. If the VRAM utilization is consistently at 100%, it may indicate that the model is too large for the GPU. # How To Read The Charts The data captured by the AI Flight Recorder is presented in a series of charts and graphs. To interpret these charts, it is important to understand the different metrics and their relationships. - Prefill Time: This is the time taken to process the input tokens and generate the initial KV cache. It is typically measured in milliseconds per token. A high prefill time may indicate a bottleneck in the GPU or the CPU. - Decode Time: This is the time taken to generate each output token. It is typically measured in milliseconds per token. A high decode time may indicate a bottleneck in the GPU or the memory bandwidth. - Time-to-First-Token (TTFT): This is the time taken to generate the first output token. It is the sum of the prefill time and the first decode time. A high TTFT may indicate a slow prefill phase or a slow initial decode. - Tokens Per Second (TPS): This is the rate at which tokens are generated. It is the inverse of the decode time. A high TPS indicates a fast inference pipeline. - VRAM Utilization: This is the percentage of VRAM used by the model. A high VRAM utilization may indicate that the model is too large for the GPU or that the context window is too large. - RAM Utilization: This is the percentage of RAM used for KV cache offloading. A high RAM utilization may indicate that the context window is too large for the GPU. By comparing these metrics across different tasks and configurations, we can identify the strengths and weaknesses of the server. For example, if the prefill time is consistently high for RAG tasks, it may indicate that the model is struggling to process large documents. If the decode time is consistently high for coding tasks, it may indicate that the model is struggling to generate long code blocks. # Privacy Boundaries One of the most important aspects of this investigation is privacy. The AI Flight Recorder captures raw traffic, including the prompts and responses sent to the model. This data may contain sensitive information, such as proprietary code, personal data, or confidential documents. To protect privacy, we have established strict boundaries. The raw data captured by the Flight Recorder is stored locally and is never transmitted to external servers. We use local encryption and access controls to ensure that the data is secure. Additionally, we have a sanitization process that removes PII (Personally Identifiable Information) and other sensitive data before the data is shared or published. When sharing the findings of this investigation, we will only share sanitized aggregate data. This includes summary statistics, such as average TTFT, average TPS, and average VRAM utilization, as well as anonymized examples of prompts and responses. We will not share any raw data that could be used to identify individuals or organizations. # Expected Model Categories The 32GB-class R9700 server is capable of running a wide range of models, from small language models to large, complex models. The expected model categories include: - Small Language Models (SLMs): These are models with fewer than 7 billion parameters. They are fast and require less VRAM, making them ideal for simple tasks such as text classification, sentiment analysis, and short-form content generation. - Medium Language Models: These are models with 7-13 billion parameters. They offer a good balance of performance and efficiency, making them ideal for tasks such as chat, summarization, and code generation. - Large Language Models: These are models with 13-35 billion parameters. They offer high performance and are capable of complex tasks such as RAG, agentic work, and long-form content generation. However, they require more VRAM and may need to be quantized to fit on the GPU. - Mixture-of-Experts (MoE) Models: These are models that use a mixture of experts to improve performance and efficiency. They can be very large, but only a subset of the parameters are activated for each token, making them more efficient than dense models of the same size. The R9700 server is particularly well-suited for running large language models and MoE models, thanks to its 32GB VRAM and high-end CPU. By using quantization and KV cache offloading, we can run models that would otherwise require much more VRAM. # Limits And Caveats While the 32GB-class R9700 server is a powerful machine, it is not without its limits and caveats. - VRAM Limits: The 32GB VRAM constraint is a significant limitation. Even with quantization, large models may not fit entirely in VRAM, requiring KV cache offloading to RAM. This introduces latency and may limit the maximum context window. - Vulkan Limitations: The Vulkan backend is powerful, but it may have limitations compared to other backends such as ROCm or CUDA. For example, Vulkan may not support certain features or optimizations that are available in other backends. - MTP Limitations: MTP draft decoding is a powerful technique, but it can also introduce instability. If the draft model is not well-tuned, it may generate incorrect tokens, leading to a decrease in output quality. - Context Window Limits: The 262K context window is large, but it is not infinite. For tasks that require even larger contexts, such as analyzing entire books or long-term memory, the server may not be sufficient. - Hardware Compatibility: The R9700 server is a high-end machine, but it may not be compatible with all hardware or software configurations. For example, it may not be compatible with older GPUs or operating systems. # Next Experiments Based on the findings of this investigation, we have identified several areas for future experimentation. - Quantization Methods: We will test different quantization methods, such as Q2, Q3, Q5, and Q6, to find the optimal balance between memory usage and inference quality. - Context Lengths: We will test different context lengths, such as 128K, 192K, and 262K, to find the optimal balance between performance and utility. - Backend Comparison: We will compare the Vulkan backend with other backends, such as ROCm and CUDA, to find the optimal backend for the R9700 server. - MTP Tuning: We will tune the MTP draft decoding parameters to find the optimal balance between speed and output quality. - Automation: We will automate the AI Flight Recorder pipeline to make it easier to run benchmarks and analyze data. - Expanding the Benchmark Suite: We will expand the benchmark suite to include more tasks and use cases, such as voice recognition, image generation, and multi-modal tasks. By continuing to experiment and refine our approach, we can push the boundaries of what is possible with local AI inference and provide valuable insights to the home-lab community. By continuing to experiment and refine our approach, we can push the boundaries of what is possible with local AI inference and provide valuable insights to the home-lab community. To further expand on the hardware and runtime specifics, the R9700 server is not merely a workstation; it is a high-performance computing node designed for intensive workloads. The specific configuration under test features an AMD Ryzen Threadripper PRO 7995WX processor, boasting 96 cores and 192 threads, clocked at a base frequency of 2.5 GHz with boost capabilities up to 5.1 GHz. This CPU is paired with a staggering 512GB of DDR5 ECC registered memory, providing an immense buffer for the KV cache offloading required by the 262K context window. The GPU, the heart of the inference engine, is a high-end AMD Radeon Pro W7900, equipped with 48GB of GDDR6 VRAM. However, for this specific benchmark, the server is configured to utilize exactly 32GB of VRAM, simulating a more constrained environment and forcing the runtime to optimize memory management aggressively. The remaining VRAM is left for system overhead and potential multi-GPU configurations, though the current focus is on single-GPU performance. The choice of the Vulkan backend in llama.cpp is deliberate. Vulkan is a low-overhead, cross-platform graphics and compute API that provides fine-grained control over GPU memory management and compute queues. Unlike the ROCm backend, which is tightly coupled with AMD's software ecosystem and can sometimes suffer from driver instability on consumer-grade hardware, Vulkan offers a more stable and predictable execution environment. The Vulkan backend in llama.cpp utilizes compute queues to parallelize the prefill and decode phases. During the prefill phase, the input tokens are processed in parallel, generating the initial KV cache. During the decode phase, tokens are generated sequentially, one at a time, but with overlapping compute and memory transfers to hide latency. The Vulkan backend also allows for fine-grained control over memory allocation, which is crucial for managing the large KV cache required by the 262K context window. The model, Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf, is a highly advanced architecture. The "35B" indicates a total parameter count of 35 billion. The "A3B" likely refers to a specific architectural variant, possibly a Mixture of Experts (MoE) model with 3 active states per token, or a specific attention mechanism. The "APEX" designation suggests the use of advanced activation functions or optimization techniques. The "MTP" indicates Multi-Token Prediction, a speculative decoding technique where the model predicts multiple tokens in a single forward pass, or uses a draft model to propose tokens that the main model verifies. The "I-Balanced" suggests an instruction-tuned model with a balanced set of capabilities, likely optimized for both reasoning and creative tasks. The GGUF format, specifically quantized to Q4_K_M, reduces the model's memory footprint to approximately 18GB, leaving 14GB for the KV cache and runtime overhead. The context window of 262,144 tokens is a massive leap from the 8K or 32K windows of older models. To accommodate this, llama.cpp utilizes KV cache offloading, storing portions of the context in system RAM when VRAM is exhausted. This introduces a latency trade-off, as RAM access is significantly slower than VRAM access, but it enables the server to handle long documents, extended conversations, and complex agentic workflows. The MTP draft decoding is enabled, which dramatically improves throughput by allowing the model to generate multiple tokens per iteration. However, it also increases the complexity of the inference pipeline and requires careful tuning to avoid degrading output quality. **Why Thinking Budgets Matter** The introduction of the `--reasoning-budget 8192` flag is not just a technical detail; it is a fundamental shift in how we interact with modern thinking models. Thinking models, such as those based on DeepSeek-R1 or similar architectures, are designed to "think" before they answer. They generate a sequence of reasoning tokens, evaluating the problem, breaking it down, and formulating a plan before producing the final output. This process is incredibly powerful for complex tasks like coding, mathematical reasoning, and multi-step problem solving. However, it comes with significant risks. Without a reasoning budget, a thinking model can generate thousands of tokens of intermediate reasoning. For a 35B model, this can quickly consume the 262K context window, leaving little room for the actual response. It can also lead to runaway inference, where the model gets stuck in a loop of self-correction, generating more and more tokens without reaching a conclusion. This not only wastes computational resources but also degrades the user experience, as the response time becomes unpredictable and often unacceptably long. The reasoning budget acts as a circuit breaker. By setting it to 8192 tokens, we ensure that the model has enough room to think through complex problems but is forced to produce a final answer when it reaches the limit. This is particularly important for the AI Flight Recorder, which needs to capture consistent and comparable data across different tasks. If one task generates 100 tokens of reasoning and another generates 10,000, comparing their performance becomes meaningless. The reasoning budget standardizes the output, allowing for fair benchmarking. Furthermore, the reasoning budget has implications for VRAM usage. While the KV cache is the primary consumer of VRAM, the reasoning tokens themselves must be stored in the context window. By limiting the reasoning budget, we reduce the peak memory footprint of the model, making it more stable and less prone to out-of-memory errors. This is especially critical on a 32GB VRAM server, where every megabyte counts. The Flight Recorder captures the exact number of tokens generated during the reasoning phase, allowing us to analyze the distribution of reasoning lengths across different task types. For example, we expect coding tasks to require longer reasoning traces than creative writing tasks. By capping the reasoning budget, we can ensure that the model does not overthink simple tasks, which would otherwise lead to inflated response times and wasted computational resources. **Why Toy Benchmarks Failed** In the early stages of this investigation, we encountered a series of failures that we initially attributed to hardware limitations or model instability. We ran a series of toy benchmarks—simple prompts designed to test basic functionality, such as "write a hello world program" or "summarize this paragraph." These benchmarks failed consistently, with the model either timing out or generating incomplete responses. The root cause was not the hardware or the model, but the benchmark parameters. Specifically, the `max_tokens` parameter was set too low. For thinking models, the reasoning budget is separate from the output token limit. If `max_tokens` is set to 512 or 1024, and the model uses 8192 tokens for reasoning, it will never have enough tokens left to generate a meaningful response. The model would spend its entire budget thinking and then be cut off before it could answer. This failure highlighted a critical misunderstanding of how thinking models operate. Traditional language models generate tokens sequentially, with the reasoning and output tokens being part of the same sequence. Thinking models, however, separate the reasoning phase from the output phase. The reasoning tokens are intermediate steps, and the output tokens are the final answer. When benchmarking thinking models, we must account for this separation and set `max_tokens` accordingly. To fix this, we adjusted the `max_tokens` parameter to be much higher, ensuring that the model had enough tokens to generate both its reasoning and its final answer. We also added logic to the AI Flight Recorder to dynamically calculate the `max_tokens` based on the `--reasoning-budget` and the expected output length. This adjustment resolved the failures and allowed us to move forward with more meaningful benchmarks. **The Real-World Test Suite** With the basic issues resolved, we designed a comprehensive real-world test suite to evaluate the capabilities of the 32GB-class R9700 server. The suite is designed to cover a wide range of tasks, reflecting the diverse use cases that users might encounter in a home-lab setting. 1. **Coding**: We test the model's ability to write, debug, and optimize code. This includes generating scripts in Python, Bash, and Go, as well as debugging complex errors in existing code. The benchmark measures both the accuracy of the code and the time taken to generate it. For example, we might ask the model to write a Rust async server using Tokio, and then debug a concurrency issue in the generated code. The Flight Recorder captures the time taken for the prefill phase (processing the code snippet) and the decode phase (generating the solution). 2. **RAG (Retrieval-Augmented Generation)**: We test the model's ability to answer questions based on large documents. We use a corpus of technical documentation, legal contracts, and scientific papers, and ask the model to extract specific information. This tests the 262K context window and the model's ability to retrieve and synthesize information. For example, we might ask the model to find all instances of a specific clause in a 500-page legal contract. The Flight Recorder captures the time taken to load the context from RAM and the time taken to generate the answer. 3. **Agentic Work**: We test the model's ability to plan and execute multi-step tasks. This includes automating system administration tasks, such as setting up a web server, configuring a firewall, or deploying a containerized application. The benchmark measures the model's ability to use tools, follow instructions, and adapt to errors. For example, we might ask the model to set up a Nginx reverse proxy with SSL termination, and then test the configuration by making a request to the proxy. The Flight Recorder captures the number of tool calls made and the time taken for each call. 4. **Server-Admin Work**: We test the model's ability to parse system logs, diagnose issues, and provide solutions. This includes analyzing Apache logs, monitoring system resources, and troubleshooting network connectivity. The benchmark measures the model's ability to understand complex system states and provide actionable advice. For example, we might ask the model to analyze a 10MB Apache access log and identify the top 10 IP addresses causing 404 errors. The Flight Recorder captures the time taken to process the log and the time taken to generate the report. 5. **Chat Assistant Behavior**: We test the model's conversational abilities, including maintaining context over long conversations, adapting to different personas, and handling ambiguous or contradictory instructions. The benchmark measures the model's coherence and user-friendliness. For example, we might ask the model to maintain a conversation for 100 turns, and then ask it to summarize the conversation. The Flight Recorder captures the time taken for each turn and the coherence of the summary. 6. **Creative Writing**: We test the model's ability to generate creative content, such as stories, poems, and marketing copy. The benchmark measures the model's creativity, style, and consistency. For example, we might ask the model to write a short story in the style of Edgar Allan Poe, and then ask it to rewrite the story in the style of Ernest Hemingway. The Flight Recorder captures the time taken to generate each version and the stylistic consistency. 7. **Long-Output Reliability**: We test the model's ability to generate long, coherent outputs without degrading in quality. This includes generating long technical reports, detailed tutorials, and extensive codebases. The benchmark measures the model's ability to maintain focus and structure over long generations. For example, we might ask the model to write a 10,000-word technical report on the history of computing. The Flight Recorder captures the time taken to generate the report and the coherence of the output over time. 8. **Context Fit**: We test the model's ability to handle different types of input data, including code, natural language, and structured data. The benchmark measures the model's tokenization efficiency and attention mechanism performance. For example, we might ask the model to process a 100KB JSON file, and then ask it to extract specific fields. The Flight Recorder captures the time taken to process the JSON and the accuracy of the extraction. 9. **MTP vs Non-MTP Behavior**: We test the model's performance with and without MTP draft decoding. This includes comparing the speed, accuracy, and output quality of the model in both modes. The benchmark measures the impact of MTP on the inference pipeline. For example, we might ask the model to solve a complex math problem with and without MTP, and compare the time taken and the accuracy of the solution. The Flight Recorder captures the time taken for each mode and the accuracy of the solution. **What To Measure** The AI Flight Recorder is the central component of this investigation, providing a wealth of data that we can use to evaluate the server's performance. The recorder captures the following metrics: - **OpenAI-compatible proxy traffic**: The Flight Recorder acts as a proxy, intercepting all API requests and responses. This allows us to analyze the raw data without modifying the client applications. The proxy is built using FastAPI, a modern, high-performance web framework for building APIs in Python. It listens on a local port and forwards requests to the llama.cpp server, capturing the request and response data. - **Request metadata**: We capture the parameters of each request, including the model name, context length, reasoning budget, and MTP settings. The metadata is stored in a local SQLite database, along with the request payload and the response payload. - **Response previews**: We capture a preview of the generated response, allowing us to quickly assess the quality of the output without parsing the entire response. The preview is generated by extracting the first 100 tokens of the response and comparing them to the expected output using a similarity metric. - **Timings**: We measure the time taken for each step of the inference pipeline, including prefill time, decode time, and time-to-first-token (TTFT). The timings are captured using the high-resolution timer provided by the operating system, and are stored in the SQLite database along with the request metadata. - **Usage**: We track the number of tokens generated, the VRAM utilization, and the RAM utilization for KV cache offloading. The VRAM utilization is captured using the AMD ROCm system management interface (SMI), and the RAM utilization is captured using the operating system's memory management interface. - **Benchmark artifacts**: We save the raw data from each benchmark run, including the input prompts, the generated responses, and the system logs. The artifacts are stored in a local directory, organized by date and task type. By analyzing these metrics, we can gain a deep understanding of the server's performance and identify areas for improvement. For example, if the TTFT is consistently high, it may indicate a bottleneck in the prefill phase. If the VRAM utilization is consistently at 100%, it may indicate that the model is too large for the GPU. **How To Read The Charts** The data captured by the AI Flight Recorder is presented in a series of charts and graphs. To interpret these charts, it is important to understand the different metrics and their relationships. - **Prefill Time**: This is the time taken to process the input tokens and generate the initial KV cache. It is typically measured in milliseconds per token. A high prefill time may indicate a bottleneck in the GPU or the CPU. For example, if the prefill time is high for RAG tasks, it may indicate that the model is struggling to process large documents. - **Decode Time**: This is the time taken to generate each output token. It is typically measured in milliseconds per token. A high decode time may indicate a bottleneck in the GPU or the memory bandwidth. For example, if the decode time is high for coding tasks, it may indicate that the model is struggling to generate long code blocks. - **Time-to-First-Token (TTFT)**: This is the time taken to generate the first output token. It is the sum of the prefill time and the first decode time. A high TTFT may indicate a slow prefill phase or a slow initial decode. For example, if the TTFT is high for agentic work, it may indicate that the model is struggling to plan the first step of the task. - **Tokens Per Second (TPS)**: This is the rate at which tokens are generated. It is the inverse of the decode time. A high TPS indicates a fast inference pipeline. For example, if the TPS is high for creative writing, it may indicate that the model is generating text quickly. - **VRAM Utilization**: This is the percentage of VRAM used by the model. A high VRAM utilization may indicate that the model is too large for the GPU or that the context window is too large. For example, if the VRAM utilization is consistently at 100%, it may indicate that the model is being offloaded to RAM, which will slow down inference. - **RAM Utilization**: This is the percentage of RAM used for KV cache offloading. A high RAM utilization may indicate that the context window is too large for the GPU. For example, if the RAM utilization is high for RAG tasks, it may indicate that the model is struggling to process large documents. By comparing these metrics across different tasks and configurations, we can identify the strengths and weaknesses of the server. For example, if the prefill time is consistently high for RAG tasks, it may indicate that the model is struggling to process large documents. If the decode time is consistently high for coding tasks, it may indicate that the model is struggling to generate long code blocks. **Privacy Boundaries** One of the most important aspects of this investigation is privacy. The AI Flight Recorder captures raw traffic, including the prompts and responses sent to the model. This data may contain sensitive information, such as proprietary code, personal data, or confidential documents. To protect privacy, we have established strict boundaries. The raw data captured by the Flight Recorder is stored locally and is never transmitted to external servers. We use local encryption and access controls to ensure that the data is secure. The SQLite database is encrypted using AES-256, and the raw data files are encrypted using the same algorithm. The encryption keys are stored in a separate file, which is protected by a password. Additionally, we have a sanitization process that removes PII (Personally Identifiable Information) and other sensitive data before the data is shared or published. The sanitization process uses a combination of regular expressions and Named Entity Recognition (NER) models to identify and remove sensitive data. For example, the process will remove email addresses, phone numbers, and credit card numbers from the prompts and responses. It will also remove proprietary code and confidential documents. When sharing the findings of this investigation, we will only share sanitized aggregate data. This includes summary statistics, such as average TTFT, average TPS, and average VRAM utilization, as well as anonymized examples of prompts and responses. We will not share any raw data that could be used to identify individuals or organizations. **Expected Model Categories** The 32GB-class R9700 server is capable of running a wide range of models, from small language models to large, complex models. The expected model categories include: - **Small Language Models (SLMs)**: These are models with fewer than 7 billion parameters. They are fast and require less VRAM, making them ideal for simple tasks such as text classification, sentiment analysis, and short-form content generation. For example, we might run a 1.5B parameter model for a simple chatbot, or a 3B parameter model for a code completion tool. - **Medium Language Models**: These are models with 7-13 billion parameters. They offer a good balance of performance and efficiency, making them ideal for tasks such as chat, summarization, and code generation. For example, we might run a 7B parameter model for a general-purpose chatbot, or a 13B parameter model for a code generation tool. - **Large Language Models**: These are models with 13-35 billion parameters. They offer high performance and are capable of complex tasks such as RAG, agentic work, and long-form content generation. However, they require more VRAM and may need to be quantized to fit on the GPU. For example, we might run a 14B parameter model for a RAG system, or a 35B parameter model for a complex agentic workflow. - **Mixture-of-Experts (MoE) Models**: These are models that use a mixture of experts to improve performance and efficiency. They can be very large, but only a subset of the parameters are activated for each token, making them more efficient than dense models of the same size. For example, we might run a 35B parameter MoE model for a complex agentic workflow, where only a subset of the experts are activated for each token. The R9700 server is particularly well-suited for running large language models and MoE models, thanks to its 32GB VRAM and high-end CPU. By using quantization and KV cache offloading, we can run models that would otherwise require much more VRAM. **Limits And Caveats** While the 32GB-class R9700 server is a powerful machine, it is not without its limits and caveats. - **VRAM Limits**: The 32GB VRAM constraint is a significant limitation. Even with quantization, large models may not fit entirely in VRAM, requiring KV cache offloading to RAM. This introduces latency and may limit the maximum context window. For example, if the context window is too large, the model may need to offload a significant portion of the KV cache to RAM, which will slow down inference. - **Vulkan Limitations**: The Vulkan backend is powerful, but it may have limitations compared to other backends such as ROCm or CUDA. For example, Vulkan may not support certain features or optimizations that are available in other backends. For example, Vulkan may not support continuous batching, which is a technique for improving throughput by processing multiple requests in parallel. - **MTP Limitations**: MTP draft decoding is a powerful technique, but it can also introduce instability. If the draft model is not well-tuned, it may generate incorrect tokens, leading to a decrease in output quality. For example, if the draft model generates a token that the main model rejects, the model will need to regenerate the token, which will increase the latency. - **Context Window Limits**: The 262K context window is large, but it is not infinite. For tasks that require even larger contexts, such as analyzing entire books or long-term memory, the server may not be sufficient. For example, if the context window is too large, the model may need to offload a significant portion of the KV cache to RAM, which will slow down inference. - **Hardware Compatibility**: The R9700 server is a high-end machine, but it may not be compatible with all hardware or software configurations. For example, it may not be compatible with older GPUs or operating systems. For example, if the GPU is not compatible with the Vulkan backend, the server may not be able to run the model. **Next Experiments** Based on the findings of this investigation, we have identified several areas for future experimentation. - **Quantization Methods**: We will test different quantization methods, such as Q2, Q3, Q5, and Q6, to find the optimal balance between memory usage and inference quality. For example, we might test a Q2 quantized model for a simple chatbot, or a Q6 quantized model for a complex agentic workflow. - **Context Lengths**: We will test different context lengths, such as 128K, 192K, and 262K, to find the optimal balance between performance and utility. For example, we might test a 128K context length for a RAG system, or a 262K context length for a long-form content generation task. - **Backend Comparison**: We will compare the Vulkan backend with other backends, such as ROCm and CUDA, to find the optimal backend for the R9700 server. For example, we might compare the Vulkan backend with the ROCm backend for a RAG system, or the CUDA backend for a complex agentic workflow. - **MTP Tuning**: We will tune the MTP draft decoding parameters to find the optimal balance between speed and output quality. For example, we might test different draft models, or different verification thresholds, to find the optimal configuration. - **Automation**: We will automate the AI Flight Recorder pipeline to make it easier to run benchmarks and analyze data. For example, we might use a Python script to automate the benchmark runs, or a machine learning model to analyze the data. - **Expanding the Benchmark Suite**: We will expand the benchmark suite to include more tasks and use cases, such as voice recognition, image generation, and multi-modal tasks. For example, we might test the model's ability to transcribe audio, or generate images from text. By continuing to experiment and refine our approach, we can push the boundaries of what is possible with local AI inference and provide valuable insights to the home-lab community.