# 1. Purpose And Scope The primary purpose of this master field manual is to establish a rigorous, reproducible, and comprehensive methodology for evaluating open-weight GGUF models within a constrained, high-performance home-lab environment. Specifically, this manual details the procedures for benchmarking models on a 32GB-class AMD Ryzen Threadripper PRO 7975WX (R9700) server utilizing the llama.cpp inference runtime, with telemetry and artifact collection managed by the AI Flight Recorder system. The scope of this benchmark campaign is strictly defined by the hardware constraints and the software stack. The 32GB RAM limitation necessitates a focus on quantized models (typically Q4_K_M, Q5_K_M, or Q6_K quantizations) and requires careful management of the Key-Value (KV) cache to prevent out-of-memory (OOM) errors during inference. The R9700 architecture provides a massive number of PCIe lanes and high memory bandwidth, which must be leveraged to maximize throughput, yet the CPU-only or limited-GPU offloading scenario means that inference speed is heavily dependent on memory bandwidth and cache locality rather than raw GPU compute. The AI Flight Recorder serves as the central nervous system of this benchmark. It is responsible for capturing every inference request, measuring latency, tracking token generation rates, recording system telemetry (CPU utilization, memory pressure, thermal throttling), and storing the resulting artifacts. The recorder ensures that no data is lost during the evaluation process and provides a structured, queryable database for post-hoc analysis. This manual does not merely provide a list of commands to run; it provides a holistic framework for understanding how open-weight models behave under real-world constraints. It covers the entire lifecycle of a benchmark run, from hardware profiling and runtime configuration to workload design, automated evaluation, and final reporting. The goal is to create a repeatable experiment that can be run by any operator, yielding comparable results across different model families, quantization levels, and inference parameters. By adhering to this manual, operators will gain deep insights into the trade-offs between model size, quantization, context length, and reasoning depth, ultimately allowing them to select the optimal model for their specific use cases. # 2. Hardware Profile The R9700 server is a workstation-class platform built around the AMD Ryzen Threadripper PRO 7975WX processor. This CPU features 32 cores and 64 threads, based on the Zen 4 architecture, and is paired with up to 2TB of DDR5 ECC memory. However, for this specific benchmark campaign, the system is configured with 32GB of DDR5-5600 ECC RAM. This configuration presents a unique set of challenges and opportunities. **CPU and Memory Architecture:** The 32 cores provide ample parallelism for multi-threaded inference, allowing llama.cpp to distribute the computational load across a large number of threads. The DDR5-5600 memory provides high bandwidth, which is critical for CPU-only inference where the model weights are loaded into RAM and accessed sequentially during the forward pass. The ECC (Error Correction Code) memory ensures data integrity, which is paramount for long-running inference tasks where a single bit flip could corrupt the KV cache or the generated output. **Storage Subsystem:** The server is equipped with enterprise-grade NVMe SSDs (e.g., Samsung PM9A3 or similar). The storage speed is crucial for loading large GGUF model files into memory. A fast NVMe drive reduces the time-to-first-token (TTFT) by ensuring that the model weights are streamed into RAM as quickly as possible. The storage layout must be optimized for sequential reads, as llama.cpp loads the model weights in a streaming fashion. **Thermal and Power Considerations:** The R9700 platform is designed for sustained high loads, but the 32GB RAM configuration means that the system is not memory-bound in the traditional sense of running out of RAM. However, the CPU will be heavily utilized during inference, generating significant heat. The thermal management system must be monitored to ensure that the CPU does not thermal throttle, which would cause unpredictable drops in inference speed. The power supply must be sufficient to handle the peak power draw of the CPU and any attached PCIe devices (e.g., a single GPU for offloading). **PCIe Topology:** The R9700 provides a massive number of PCIe lanes, allowing for the attachment of multiple high-speed devices. For this benchmark, the PCIe topology is configured to ensure that the NVMe drives and any GPU (if used for offloading) are connected to the CPU with the lowest possible latency. This is critical for maintaining high throughput during inference. **Hardware Profiling Checklist:** - [ ] Verify CPU frequency scaling is set to performance mode (no turbo boost throttling). - [ ] Confirm DDR5 memory is running at the rated 5600 MT/s speed. - [ ] Ensure NVMe drives are in AHCI mode and not in RAID, to maximize sequential read speeds. - [ ] Monitor CPU temperatures during inference to detect thermal throttling. - [ ] Verify that the system is not swapping memory to disk, which would severely degrade performance. # 3. llama.cpp Runtime Profile llama.cpp is the inference engine of choice for this benchmark due to its efficiency, flexibility, and support for a wide range of quantization formats. The runtime profile must be carefully tuned to maximize performance on the R9700 hardware. **Quantization Formats:** The primary quantization formats to be evaluated are Q4_K_M, Q5_K_M, and Q6_K. Q4_K_M offers the best balance between model size and quality, making it the default choice for most workloads. Q5_K_M provides a slight improvement in quality at the cost of increased memory usage, which may be necessary for models that require a larger context window. Q6_K is reserved for models where quality is paramount and memory is not a constraint. **KV Cache Management:** The KV cache is the most critical component for managing context length. In llama.cpp, the KV cache is stored in RAM, and its size is determined by the `--ctx-size` parameter. On a 32GB system, the maximum context size is limited by the available RAM. For a 7B parameter model in Q4_K_M, the KV cache can support a context size of up to 32,768 tokens. For a 13B parameter model, the context size may need to be reduced to 8,192 or 16,384 tokens to avoid OOM errors. **Thread Configuration:** The `--threads` and `--threads-batch` parameters control the number of threads used for inference. On the R9700, the optimal thread count is typically equal to the number of physical cores (32). However, for smaller models, fewer threads may be sufficient to achieve maximum throughput. The `--threads-batch` parameter controls the number of threads used for batch inference, which is critical for multi-token prediction (MTP) and speculative decoding. **GPU Offloading:** If a GPU is available, llama.cpp can offload layers to the GPU using the `--n-gpu-layers` parameter. However, on a 32GB system, the GPU VRAM may be limited, and offloading may not provide a significant performance benefit over CPU-only inference. The decision to offload must be made on a case-by-case basis, depending on the model size and the available VRAM. **llama.cpp Runtime Checklist:** - [ ] Select the appropriate quantization format (Q4_K_M, Q5_K_M, Q6_K). - [ ] Set the `--ctx-size` parameter to the maximum context size supported by the available RAM. - [ ] Configure the `--threads` and `--threads-batch` parameters for optimal performance. - [ ] Determine whether GPU offloading is beneficial based on the available VRAM. - [ ] Enable M-threads for multi-token prediction if supported by the model. # 4. Reasoning Budget Methodology The reasoning budget of 8192 tokens is a critical constraint that dictates how the model can be used for complex tasks. This budget represents the maximum number of tokens that the model can generate in a single inference call, including both the reasoning tokens (if using a reasoning model) and the final output tokens. **Defining the Reasoning Budget:** The reasoning budget is set using the `--n-predict` parameter in llama.cpp. For this benchmark, the budget is fixed at 8192 tokens. This means that the model can generate up to 8192 tokens in a single pass, after which the inference must be stopped. This constraint is particularly relevant for reasoning models, which may generate a large number of reasoning tokens before producing the final answer. **Measuring Reasoning Depth:** To evaluate the model's reasoning depth, the AI Flight Recorder must track the number of reasoning tokens generated versus the number of final output tokens. This can be achieved by parsing the model's output and identifying the reasoning tokens (e.g., using a specific delimiter or format). The ratio of reasoning tokens to output tokens provides a measure of the model's reasoning depth. **Managing the Reasoning Budget:** If the model generates more than 8192 tokens, the inference must be stopped, and the output must be truncated. This can lead to incomplete answers, which must be accounted for in the evaluation. To mitigate this, the prompt can be designed to encourage the model to generate concise answers, or the reasoning budget can be increased for tasks that require longer outputs. **Reasoning Budget Methodology Checklist:** - [ ] Set the `--n-predict` parameter to 8192. - [ ] Parse the model's output to identify reasoning tokens and final output tokens. - [ ] Calculate the ratio of reasoning tokens to output tokens. - [ ] Truncate the output if the model generates more than 8192 tokens. - [ ] Account for truncated outputs in the evaluation. # 5. MTP Versus Non-MTP Methodology Multi-Token Prediction (MTP) is a technique that allows the model to generate multiple tokens in a single forward pass, significantly improving inference speed. In llama.cpp, MTP is enabled using the `--m-threads` parameter, which specifies the number of threads used for MTP. **MTP Methodology:** To evaluate MTP, the model must be run with the `--m-threads` parameter set to a value greater than zero. The number of threads should be set to the number of physical cores (32) to maximize throughput. The MTP algorithm works by generating multiple tokens in parallel, using the previous tokens as context for the next tokens. This reduces the number of forward passes required to generate a sequence of tokens, resulting in a significant speedup. **Non-MTP Methodology:** To evaluate non-MTP inference, the model must be run with the `--m-threads` parameter set to zero. This disables MTP, and the model generates tokens one at a time, using the previous tokens as context for the next token. This is the standard autoregressive generation method. **Comparing MTP and Non-MTP:** To compare MTP and non-MTP inference, the AI Flight Recorder must track the time-to-first-token (TTFT) and the tokens-per-second (TPS) for both methods. The TTFT is the time it takes for the model to generate the first token, and the TPS is the number of tokens generated per second. MTP should result in a higher TPS than non-MTP, but the TTFT may be slightly longer due to the additional computation required for MTP. **MTP Versus Non-MTP Checklist:** - [ ] Run the model with `--m-threads` set to 32 for MTP inference. - [ ] Run the model with `--m-threads` set to 0 for non-MTP inference. - [ ] Track the TTFT and TPS for both methods. - [ ] Compare the TTFT and TPS to evaluate the performance of MTP. # 6. Context Fit Methodology The context fit methodology evaluates how well the model can handle different context lengths within the constraints of the 32GB RAM. This is critical for understanding the model's ability to process long documents, codebases, or conversations. **Context Lengths to Evaluate:** The context lengths to be evaluated are 4096, 8192, 16384, and 32768 tokens. These lengths represent a range of use cases, from short conversations to long documents. The maximum context length is limited by the available RAM, so the 32768 token context length may only be feasible for smaller models. **Measuring Context Fit:** To measure context fit, the AI Flight Recorder must track the time-to-first-token (TTFT) and the tokens-per-second (TPS) for each context length. The TTFT is the time it takes for the model to generate the first token, and the TPS is the number of tokens generated per second. As the context length increases, the TTFT and TPS may decrease due to the increased memory pressure and the larger KV cache. **Context Fit Methodology Checklist:** - [ ] Set the `--ctx-size` parameter to 4096, 8192, 16384, and 32768. - [ ] Track the TTFT and TPS for each context length. - [ ] Evaluate the model's ability to process long documents, codebases, or conversations. # 7. Coding Workloads **Realistic Task Design:** The coding workload evaluates the model's ability to generate, debug, and refactor code. The task involves presenting the model with a complex, multi-file Python refactoring scenario. The prompt includes a legacy codebase with deprecated patterns, requiring the model to identify anti-patterns, propose a modernized architecture, and generate the corresponding unit tests. **Prompt Shape:** ```json { "system": "You are an expert Python developer. Your task is to refactor the provided codebase to use modern Python best practices.", "user": "Here is the legacy codebase:\n\n```python\n# Legacy code with deprecated patterns\nimport os\nimport sys\n\ndef process_data(file_path):\n # Deprecated method\n data = open(file_path, 'r').read()\n # ... processing logic ...\n return data\n```\n\nPlease refactor this code to use modern Python best practices, including context managers and type hints. Also, provide unit tests for the refactored code." } ``` **Expected Answer Shape:** The expected answer includes a refactored version of the code using context managers and type hints, along with a set of unit tests using the `unittest` framework. The code should be clean, well-documented, and follow PEP 8 guidelines. **Automated Checks:** - Syntax validation using `py_compile`. - Unit test execution using `pytest`. - Code style validation using `flake8`. **Human Review Rubric:** - **Correctness (1-5):** Does the refactored code produce the same output as the original code? - **Best Practices (1-5):** Does the refactored code follow modern Python best practices? - **Completeness (1-5):** Are all the required unit tests provided? **Failure Examples:** - The model generates code that does not compile. - The model generates code that does not pass the unit tests. - The model fails to use context managers or type hints. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Code correctness score - Code style score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is sufficient for most coding tasks, but complex refactoring scenarios may require more tokens. If the model generates more than 8192 tokens, the output must be truncated, which may result in incomplete code or missing unit tests. # 8. Agentic Workloads **Realistic Task Design:** The agentic workload evaluates the model's ability to plan and execute multi-step tasks using tools. The task involves presenting the model with a complex scenario that requires the model to use a set of tools to gather information, make decisions, and take actions. **Prompt Shape:** ```json { "system": "You are an AI agent. Your task is to use the provided tools to complete the given objective.", "user": "Objective: Find the latest news about the stock market and summarize it.\n\nTools:\n- search(query): Search the web for information.\n- summarize(text): Summarize the provided text.\n\nPlease use the tools to complete the objective." } ``` **Expected Answer Shape:** The expected answer includes a sequence of tool calls, followed by a summary of the news. The tool calls should be valid, and the summary should be accurate and concise. **Automated Checks:** - Tool call validation using a JSON schema. - Summary accuracy using an LLM-as-a-judge. **Human Review Rubric:** - **Tool Call Validity (1-5):** Are the tool calls valid and correctly formatted? - **Task Completion (1-5):** Did the model successfully complete the objective? - **Summary Quality (1-5):** Is the summary accurate and concise? **Failure Examples:** - The model generates invalid tool calls. - The model fails to use the tools to gather information. - The model generates a summary that is inaccurate or incomplete. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Tool call validity score - Task completion score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is critical for agentic workloads, as the model may generate a large number of reasoning tokens before producing the final answer. If the model generates more than 8192 tokens, the output must be truncated, which may result in incomplete tool calls or missing actions. # 9. RAG Workloads **Realistic Task Design:** The RAG workload evaluates the model's ability to retrieve relevant information from a knowledge base and generate a response based on that information. The task involves presenting the model with a question and a set of documents, requiring the model to retrieve the relevant documents and generate a response. **Prompt Shape:** ```json { "system": "You are an AI assistant. Your task is to answer the question based on the provided documents.", "user": "Question: What is the capital of France?\n\nDocuments:\n- Document 1: The capital of France is Paris.\n- Document 2: The capital of Germany is Berlin.\n\nPlease answer the question based on the documents." } ``` **Expected Answer Shape:** The expected answer includes a response to the question, along with citations to the relevant documents. The response should be accurate and concise. **Automated Checks:** - Response accuracy using an LLM-as-a-judge. - Citation accuracy using a regex pattern. **Human Review Rubric:** - **Response Accuracy (1-5):** Is the response accurate? - **Citation Accuracy (1-5):** Are the citations accurate and correctly formatted? - **Conciseness (1-5):** Is the response concise? **Failure Examples:** - The model generates a response that is inaccurate. - The model fails to cite the relevant documents. - The model generates a response that is verbose or incomplete. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Response accuracy score - Citation accuracy score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is sufficient for most RAG workloads, but complex questions may require more tokens. If the model generates more than 8192 tokens, the output must be truncated, which may result in incomplete responses or missing citations. # 10. Chatbot Workloads **Realistic Task Design:** The chatbot workload evaluates the model's ability to engage in a multi-turn conversation. The task involves presenting the model with a series of user inputs, requiring the model to generate a response for each input. **Prompt Shape:** ```json { "system": "You are a helpful AI assistant. Your task is to respond to the user's inputs.", "user": "Hello, how are you?" } ``` **Expected Answer Shape:** The expected answer includes a response to the user's input. The response should be polite, helpful, and relevant. **Automated Checks:** - Toxicity detection using a toxicity classifier. - Relevance detection using an LLM-as-a-judge. **Human Review Rubric:** - **Politeness (1-5):** Is the response polite? - **Helpfulness (1-5):** Is the response helpful? - **Relevance (1-5):** Is the response relevant to the user's input? **Failure Examples:** - The model generates a response that is impolite or offensive. - The model generates a response that is not helpful. - The model generates a response that is irrelevant to the user's input. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Politeness score - Helpfulness score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is sufficient for most chatbot workloads, but long conversations may require more tokens. If the model generates more than 8192 tokens, the output must be truncated, which may result in incomplete responses. # 11. Creative And Editorial Workloads **Realistic Task Design:** The creative and editorial workload evaluates the model's ability to generate creative content and edit existing content. The task involves presenting the model with a creative prompt, requiring the model to generate a story, poem, or article. **Prompt Shape:** ```json { "system": "You are a creative writer. Your task is to write a story based on the provided prompt.", "user": "Prompt: Write a story about a robot that learns to love." } ``` **Expected Answer Shape:** The expected answer includes a story that is creative, engaging, and well-written. The story should follow the prompt and include a clear beginning, middle, and end. **Automated Checks:** - Creativity detection using an LLM-as-a-judge. - Grammar and spelling detection using a grammar checker. **Human Review Rubric:** - **Creativity (1-5):** Is the story creative and engaging? - **Grammar and Spelling (1-5):** Is the story free of grammar and spelling errors? - **Coherence (1-5):** Is the story coherent and well-structured? **Failure Examples:** - The model generates a story that is not creative or engaging. - The model generates a story with grammar and spelling errors. - The model generates a story that is incoherent or poorly structured. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Creativity score - Grammar and spelling score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is sufficient for most creative and editorial workloads, but long stories may require more tokens. If the model generates more than 8192 tokens, the output must be truncated, which may result in an incomplete story. # 12. Long-Output Reliability **Realistic Task Design:** The long-output reliability workload evaluates the model's ability to generate long outputs without losing coherence or hallucinating. The task involves presenting the model with a prompt that requires a long output, such as a detailed report or a long story. **Prompt Shape:** ```json { "system": "You are an AI assistant. Your task is to write a detailed report on the history of the internet.", "user": "Write a detailed report on the history of the internet, covering the following topics: the ARPANET, the development of TCP/IP, the creation of the World Wide Web, and the rise of social media." } ``` **Expected Answer Shape:** The expected answer includes a detailed report that covers all the required topics. The report should be well-structured, coherent, and free of hallucinations. **Automated Checks:** - Hallucination detection using an LLM-as-a-judge. - Coherence detection using a coherence classifier. **Human Review Rubric:** - **Completeness (1-5):** Does the report cover all the required topics? - **Coherence (1-5):** Is the report coherent and well-structured? - **Accuracy (1-5):** Is the report free of hallucinations? **Failure Examples:** - The model generates a report that is incomplete or missing topics. - The model generates a report that is incoherent or poorly structured. - The model generates a report with hallucinations. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Completeness score - Coherence score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is critical for long-output reliability workloads, as the model may generate a large number of tokens before producing the final answer. If the model generates more than 8192 tokens, the output must be truncated, which may result in an incomplete report. # 13. Privacy And Redaction Privacy and redaction are critical considerations for any benchmark campaign, especially when dealing with sensitive data. The AI Flight Recorder must ensure that all data is properly redacted before being stored or analyzed. **Data Redaction:** All data must be redacted using a redaction tool that removes or replaces sensitive information, such as names, addresses, and phone numbers. The redaction tool must be configured to handle a wide range of data types and formats. **Data Anonymization:** All data must be anonymized using a data anonymization tool that replaces sensitive information with synthetic data. The anonymization tool must be configured to handle a wide range of data types and formats. **Privacy and Redaction Checklist:** - [ ] Configure the redaction tool to remove or replace sensitive information. - [ ] Configure the anonymization tool to replace sensitive information with synthetic data. - [ ] Verify that all data is properly redacted and anonymized before being stored or analyzed. # 14. SQLite Storage And Artifact Layout The AI Flight Recorder stores all benchmark data in a SQLite database. The database schema must be carefully designed to ensure that all data is properly indexed and queryable. **Database Schema:** The database schema includes tables for storing model metadata, benchmark results, and system telemetry. The tables are indexed on the primary key and foreign keys to ensure fast queries. **Artifact Layout:** The artifact layout includes directories for storing model files, benchmark results, and system telemetry. The directories are organized by date and model name to ensure easy access and retrieval. **SQLite Storage And Artifact Layout Checklist:** - [ ] Design the database schema to ensure that all data is properly indexed and queryable. - [ ] Create directories for storing model files, benchmark results, and system telemetry. - [ ] Organize the directories by date and model name. # 15. Reporting Plane And Screenshots The reporting plane provides a web-based interface for visualizing benchmark results. The reporting plane must be carefully designed to ensure that all data is properly visualized and easy to understand. **Dashboard Design:** The dashboard includes charts and graphs for visualizing benchmark results, such as TTFT, TPS, and accuracy scores. The dashboard must be responsive and easy to navigate. **Screenshot Capture:** The AI Flight Recorder must capture screenshots of the dashboard at regular intervals to ensure that all data is properly recorded. The screenshots must be stored in the artifact layout. **Reporting Plane And Screenshots Checklist:** - [ ] Design the dashboard to include charts and graphs for visualizing benchmark results. - [ ] Capture screenshots of the dashboard at regular intervals. - [ ] Store the screenshots in the artifact layout. # 16. Model Leaderboards The model leaderboard provides a ranking of models based on their benchmark results. The leaderboard must be carefully designed to ensure that all models are properly ranked and easy to compare. **Ranking Methodology:** The ranking methodology includes a weighted average of the benchmark results, such as TTFT, TPS, and accuracy scores. The weights must be carefully chosen to reflect the importance of each metric. **Leaderboard Design:** The leaderboard includes a table for displaying the rankings, along with charts and graphs for visualizing the results. The leaderboard must be responsive and easy to navigate. **Model Leaderboards Checklist:** - [ ] Design the ranking methodology to include a weighted average of the benchmark results. - [ ] Design the leaderboard to include a table for displaying the rankings, along with charts and graphs for visualizing the results. # 17. Reproducibility Checklist Reproducibility is critical for any benchmark campaign. The reproducibility checklist ensures that all experiments can be reproduced by other operators. **Reproducibility Checklist:** - [ ] Set the random seed to a fixed value. - [ ] Use the same model files and quantization formats. - [ ] Use the same llama.cpp runtime parameters. - [ ] Use the same AI Flight Recorder configuration. - [ ] Use the same workload prompts and expected answers. # 18. Final Recommendations The final recommendations provide a summary of the best practices for evaluating open-weight GGUF models on a 32GB-class R9700 llama.cpp server using AI Flight Recorder. **Best Practices:** - Use the appropriate quantization format for the model and workload. - Configure the llama.cpp runtime parameters for optimal performance. - Use the AI Flight Recorder to capture all benchmark data. - Redact and anonymize all data before storing or analyzing. - Use the reporting plane to visualize benchmark results. - Use the model leaderboard to rank models based on their benchmark results. - Follow the reproducibility checklist to ensure that all experiments can be reproduced. **Future Work:** - Evaluate the performance of MTP and non-MTP inference on different models and workloads. - Evaluate the performance of the model on different context lengths. - Evaluate the performance of the model on different reasoning budgets. - Evaluate the performance of the model on different quantization formats. # 1. Purpose And Scope (Expanded) The primary purpose of this master field manual is to establish a rigorous, reproducible, and comprehensive methodology for evaluating open-weight GGUF models within a constrained, high-performance home-lab environment. Specifically, this manual details the procedures for benchmarking models on a 32GB-class AMD Ryzen Threadripper PRO 7975WX (R9700) server utilizing the llama.cpp inference runtime, with telemetry and artifact collection managed by the AI Flight Recorder system. The scope of this benchmark campaign is strictly defined by the hardware constraints and the software stack. The 32GB RAM limitation necessitates a focus on quantized models (typically Q4_K_M, Q5_K_M, or Q6_K quantizations) and requires careful management of the Key-Value (KV) cache to prevent out-of-memory (OOM) errors during inference. The R9700 architecture provides a massive number of PCIe lanes and high memory bandwidth, which must be leveraged to maximize throughput, yet the CPU-only or limited-GPU offloading scenario means that inference speed is heavily dependent on memory bandwidth and cache locality rather than raw GPU compute. The AI Flight Recorder serves as the central nervous system of this benchmark. It is responsible for capturing every inference request, measuring latency, tracking token generation rates, recording system telemetry (CPU utilization, memory pressure, thermal throttling), and storing the resulting artifacts. The recorder ensures that no data is lost during the evaluation process and provides a structured, queryable database for post-hoc analysis. This manual does not merely provide a list of commands to run; it provides a holistic framework for understanding how open-weight models behave under real-world constraints. It covers the entire lifecycle of a benchmark run, from hardware profiling and runtime configuration to workload design, automated evaluation, and final reporting. The goal is to create a repeatable experiment that can be run by any operator, yielding comparable results across different model families, quantization levels, and inference parameters. By adhering to this manual, operators will gain deep insights into the trade-offs between model size, quantization, context length, and reasoning depth, ultimately allowing them to select the optimal model for their specific use cases. **Detailed Scope Boundaries:** - **Model Families:** The benchmark will focus on Llama 3, Mistral, and Qwen model families, as these represent the current state-of-the-art in open-weight models. - **Quantization Levels:** The primary quantization levels to be evaluated are Q4_K_M, Q5_K_M, and Q6_K. Q8_0 will be included for reference, but its memory footprint may exceed the 32GB limit for larger context windows. - **Context Lengths:** The context lengths to be evaluated are 4096, 8192, 16384, and 32768 tokens. These lengths represent a range of use cases, from short conversations to long documents. - **Workloads:** The workloads to be evaluated include coding, agentic, RAG, chatbot, creative, and long-output reliability. - **Metrics:** The metrics to be evaluated include time-to-first-token (TTFT), tokens-per-second (TPS), accuracy, coherence, and hallucination rate. # 2. Hardware Profile (Expanded) The R9700 server is a workstation-class platform built around the AMD Ryzen Threadripper PRO 7975WX processor. This CPU features 32 cores and 64 threads, based on the Zen 4 architecture, and is paired with up to 2TB of DDR5 ECC memory. However, for this specific benchmark campaign, the system is configured with 32GB of DDR5-5600 ECC RAM. This configuration presents a unique set of challenges and opportunities. **CPU and Memory Architecture:** The 32 cores provide ample parallelism for multi-threaded inference, allowing llama.cpp to distribute the computational load across a large number of threads. The DDR5-5600 memory provides high bandwidth, which is critical for CPU-only inference where the model weights are loaded into RAM and accessed sequentially during the forward pass. The ECC (Error Correction Code) memory ensures data integrity, which is paramount for long-running inference tasks where a single bit flip could corrupt the KV cache or the generated output. **Storage Subsystem:** The server is equipped with enterprise-grade NVMe SSDs (e.g., Samsung PM9A3 or similar). The storage speed is crucial for loading large GGUF model files into memory. A fast NVMe drive reduces the time-to-first-token (TTFT) by ensuring that the model weights are streamed into RAM as quickly as possible. The storage layout must be optimized for sequential reads, as llama.cpp loads the model weights in a streaming fashion. **Thermal and Power Considerations:** The R9700 platform is designed for sustained high loads, but the 32GB RAM configuration means that the system is not memory-bound in the traditional sense of running out of RAM. However, the CPU will be heavily utilized during inference, generating significant heat. The thermal management system must be monitored to ensure that the CPU does not thermal throttle, which would cause unpredictable drops in inference speed. The power supply must be sufficient to handle the peak power draw of the CPU and any attached PCIe devices (e.g., a single GPU for offloading). **PCIe Topology:** The R9700 provides a massive number of PCIe lanes, allowing for the attachment of multiple high-speed devices. For this benchmark, the PCIe topology is configured to ensure that the NVMe drives and any GPU (if used for offloading) are connected to the CPU with the lowest possible latency. This is critical for maintaining high throughput during inference. **Hardware Profiling Checklist:** - [ ] Verify CPU frequency scaling is set to performance mode (no turbo boost throttling). - [ ] Confirm DDR5 memory is running at the rated 5600 MT/s speed. - [ ] Ensure NVMe drives are in AHCI mode and not in RAID, to maximize sequential read speeds. - [ ] Monitor CPU temperatures during inference to detect thermal throttling. - [ ] Verify that the system is not swapping memory to disk, which would severely degrade performance. - [ ] Check for any background processes that might interfere with the benchmark, such as antivirus scans or system updates. **Failure Modes:** - **Thermal Throttling:** If the CPU temperature exceeds the thermal limit, the CPU will reduce its clock speed, leading to a significant drop in inference speed. - **Memory Swapping:** If the system runs out of RAM, it will start swapping memory to disk, which will severely degrade performance and may cause the benchmark to fail. - **PCIe Bottleneck:** If the NVMe drive or GPU is not connected to the CPU with the lowest possible latency, it may cause a bottleneck in data transfer, leading to slower inference. # 3. llama.cpp Runtime Profile (Expanded) llama.cpp is the inference engine of choice for this benchmark due to its efficiency, flexibility, and support for a wide range of quantization formats. The runtime profile must be carefully tuned to maximize performance on the R9700 hardware. **Quantization Formats:** The primary quantization formats to be evaluated are Q4_K_M, Q5_K_M, and Q6_K. Q4_K_M offers the best balance between model size and quality, making it the default choice for most workloads. Q5_K_M provides a slight improvement in quality at the cost of increased memory usage, which may be necessary for models that require a larger context window. Q6_K is reserved for models where quality is paramount and memory is not a constraint. **KV Cache Management:** The KV cache is the most critical component for managing context length. In llama.cpp, the KV cache is stored in RAM, and its size is determined by the `--ctx-size` parameter. On a 32GB system, the maximum context size is limited by the available RAM. For a 7B parameter model in Q4_K_M, the KV cache can support a context size of up to 32,768 tokens. For a 13B parameter model, the context size may need to be reduced to 8,192 or 16,384 tokens to avoid OOM errors. **Thread Configuration:** The `--threads` and `--threads-batch` parameters control the number of threads used for inference. On the R9700, the optimal thread count is typically equal to the number of physical cores (32). However, for smaller models, fewer threads may be sufficient to achieve maximum throughput. The `--threads-batch` parameter controls the number of threads used for batch inference, which is critical for multi-token prediction (MTP) and speculative decoding. **GPU Offloading:** If a GPU is available, llama.cpp can offload layers to the GPU using the `--n-gpu-layers` parameter. However, on a 32GB system, the GPU VRAM may be limited, and offloading may not provide a significant performance benefit over CPU-only inference. The decision to offload must be made on a case-by-case basis, depending on the model size and the available VRAM. **llama.cpp Runtime Checklist:** - [ ] Select the appropriate quantization format (Q4_K_M, Q5_K_M, Q6_K). - [ ] Set the `--ctx-size` parameter to the maximum context size supported by the available RAM. - [ ] Configure the `--threads` and `--threads-batch` parameters for optimal performance. - [ ] Determine whether GPU offloading is beneficial based on the available VRAM. - [ ] Enable M-threads for multi-token prediction if supported by the model. - [ ] Set the `--n-predict` parameter to the desired output length. - [ ] Enable verbose logging to capture detailed inference metrics. **Failure Modes:** - **OOM Error:** If the KV cache is too large for the available RAM, the system will run out of memory and the inference will fail. - **Thermal Throttling:** If the CPU temperature exceeds the thermal limit, the CPU will reduce its clock speed, leading to a significant drop in inference speed. - **Thread Contention:** If the number of threads is too high, the threads may contend for CPU resources, leading to a decrease in throughput. # 4. Reasoning Budget Methodology (Expanded) The reasoning budget of 8192 tokens is a critical constraint that dictates how the model can be used for complex tasks. This budget represents the maximum number of tokens that the model can generate in a single inference call, including both the reasoning tokens (if using a reasoning model) and the final output tokens. **Defining the Reasoning Budget:** The reasoning budget is set using the `--n-predict` parameter in llama.cpp. For this benchmark, the budget is fixed at 8192 tokens. This means that the model can generate up to 8192 tokens in a single pass, after which the inference must be stopped. This constraint is particularly relevant for reasoning models, which may generate a large number of reasoning tokens before producing the final answer. **Measuring Reasoning Depth:** To evaluate the model's reasoning depth, the AI Flight Recorder must track the number of reasoning tokens generated versus the number of final output tokens. This can be achieved by parsing the model's output and identifying the reasoning tokens (e.g., using a specific delimiter or format). The ratio of reasoning tokens to output tokens provides a measure of the model's reasoning depth. **Managing the Reasoning Budget:** If the model generates more than 8192 tokens, the inference must be stopped, and the output must be truncated. This can lead to incomplete answers, which must be accounted for in the evaluation. To mitigate this, the prompt can be designed to encourage the model to generate concise answers, or the reasoning budget can be increased for tasks that require longer outputs. **Reasoning Budget Methodology Checklist:** - [ ] Set the `--n-predict` parameter to 8192. - [ ] Parse the model's output to identify reasoning tokens and final output tokens. - [ ] Calculate the ratio of reasoning tokens to output tokens. - [ ] Truncate the output if the model generates more than 8192 tokens. - [ ] Account for truncated outputs in the evaluation. - [ ] Monitor the model's behavior when the reasoning budget is reached to ensure that it does not hallucinate or generate irrelevant content. **Failure Modes:** - **Truncation:** If the model generates more than 8192 tokens, the output will be truncated, which may result in incomplete answers. - **Hallucination:** If the model is forced to generate a long output within a limited reasoning budget, it may hallucinate or generate irrelevant content to fill the budget. - **Inefficient Reasoning:** If the model generates too many reasoning tokens, it may waste the reasoning budget and leave insufficient tokens for the final answer. # 5. MTP Versus Non-MTP Methodology (Expanded) Multi-Token Prediction (MTP) is a technique that allows the model to generate multiple tokens in a single forward pass, significantly improving inference speed. In llama.cpp, MTP is enabled using the `--m-threads` parameter, which specifies the number of threads used for MTP. **MTP Methodology:** To evaluate MTP, the model must be run with the `--m-threads` parameter set to a value greater than zero. The number of threads should be set to the number of physical cores (32) to maximize throughput. The MTP algorithm works by generating multiple tokens in parallel, using the previous tokens as context for the next tokens. This reduces the number of forward passes required to generate a sequence of tokens, resulting in a significant speedup. **Non-MTP Methodology:** To evaluate non-MTP inference, the model must be run with the `--m-threads` parameter set to zero. This disables MTP, and the model generates tokens one at a time, using the previous tokens as context for the next token. This is the standard autoregressive generation method. **Comparing MTP and Non-MTP:** To compare MTP and non-MTP inference, the AI Flight Recorder must track the time-to-first-token (TTFT) and the tokens-per-second (TPS) for both methods. The TTFT is the time it takes for the model to generate the first token, and the TPS is the number of tokens generated per second. MTP should result in a higher TPS than non-MTP, but the TTFT may be slightly longer due to the additional computation required for MTP. **MTP Versus Non-MTP Checklist:** - [ ] Run the model with `--m-threads` set to 32 for MTP inference. - [ ] Run the model with `--m-threads` set to 0 for non-MTP inference. - [ ] Track the TTFT and TPS for both methods. - [ ] Compare the TTFT and TPS to evaluate the performance of MTP. - [ ] Evaluate the quality of the output for both methods to ensure that MTP does not degrade the quality of the generated text. - [ ] Monitor the memory usage for both methods to ensure that MTP does not cause OOM errors. **Failure Modes:** - **OOM Error:** If the MTP algorithm requires more memory than is available, the system will run out of memory and the inference will fail. - **Quality Degradation:** If the MTP algorithm is not implemented correctly, it may degrade the quality of the generated text. - **Increased TTFT:** If the MTP algorithm requires additional computation, it may increase the TTFT. # 6. Context Fit Methodology (Expanded) The context fit methodology evaluates how well the model can handle different context lengths within the constraints of the 32GB RAM. This is critical for understanding the model's ability to process long documents, codebases, or conversations. **Context Lengths to Evaluate:** The context lengths to be evaluated are 4096, 8192, 16384, and 32768 tokens. These lengths represent a range of use cases, from short conversations to long documents. The maximum context length is limited by the available RAM, so the 32768 token context length may only be feasible for smaller models. **Measuring Context Fit:** To measure context fit, the AI Flight Recorder must track the time-to-first-token (TTFT) and the tokens-per-second (TPS) for each context length. The TTFT is the time it takes for the model to generate the first token, and the TPS is the number of tokens generated per second. As the context length increases, the TTFT and TPS may decrease due to the increased memory pressure and the larger KV cache. **Context Fit Methodology Checklist:** - [ ] Set the `--ctx-size` parameter to 4096, 8192, 16384, and 32768. - [ ] Track the TTFT and TPS for each context length. - [ ] Evaluate the model's ability to process long documents, codebases, or conversations. - [ ] Monitor the memory usage for each context length to ensure that the system does not run out of memory. - [ ] Evaluate the quality of the output for each context length to ensure that the model does not hallucinate or generate irrelevant content. **Failure Modes:** - **OOM Error:** If the KV cache is too large for the available RAM, the system will run out of memory and the inference will fail. - **Quality Degradation:** If the model is forced to process a very long context, it may hallucinate or generate irrelevant content. - **Decreased Throughput:** If the context length is too large, the TTFT and TPS may decrease significantly. # 7. Coding Workloads (Expanded) **Realistic Task Design:** The coding workload evaluates the model's ability to generate, debug, and refactor code. The task involves presenting the model with a complex, multi-file Python refactoring scenario. The prompt includes a legacy codebase with deprecated patterns, requiring the model to identify anti-patterns, propose a modernized architecture, and generate the corresponding unit tests. **Prompt Shape:** ```json { "system": "You are an expert Python developer. Your task is to refactor the provided codebase to use modern Python best practices.", "user": "Here is the legacy codebase:\n\n```python\n# Legacy code with deprecated patterns\nimport os\nimport sys\n\ndef process_data(file_path):\n # Deprecated method\n data = open(file_path, 'r').read()\n # ... processing logic ...\n return data\n```\n\nPlease refactor this code to use modern Python best practices, including context managers and type hints. Also, provide unit tests for the refactored code." } ``` **Expected Answer Shape:** The expected answer includes a refactored version of the code using context managers and type hints, along with a set of unit tests using the `unittest` framework. The code should be clean, well-documented, and follow PEP 8 guidelines. **Automated Checks:** - Syntax validation using `py_compile`. - Unit test execution using `pytest`. - Code style validation using `flake8`. **Human Review Rubric:** - **Correctness (1-5):** Does the refactored code produce the same output as the original code? - **Best Practices (1-5):** Does the refactored code follow modern Python best practices? - **Completeness (1-5):** Are all the required unit tests provided? **Failure Examples:** - The model generates code that does not compile. - The model generates code that does not pass the unit tests. - The model fails to use context managers or type hints. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Code correctness score - Code style score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is sufficient for most coding tasks, but complex refactoring scenarios may require more tokens. If the model generates more than 8192 tokens, the output must be truncated, which may result in incomplete code or missing unit tests. **Additional Considerations:** - **Code Generation:** The model should be able to generate code from scratch based on a natural language description. - **Code Debugging:** The model should be able to identify and fix bugs in existing code. - **Code Refactoring:** The model should be able to refactor existing code to improve its readability, maintainability, and performance. # 8. Agentic Workloads (Expanded) **Realistic Task Design:** The agentic workload evaluates the model's ability to plan and execute multi-step tasks using tools. The task involves presenting the model with a complex scenario that requires the model to use a set of tools to gather information, make decisions, and take actions. **Prompt Shape:** ```json { "system": "You are an AI agent. Your task is to use the provided tools to complete the given objective.", "user": "Objective: Find the latest news about the stock market and summarize it.\n\nTools:\n- search(query): Search the web for information.\n- summarize(text): Summarize the provided text.\n\nPlease use the tools to complete the objective." } ``` **Expected Answer Shape:** The expected answer includes a sequence of tool calls, followed by a summary of the news. The tool calls should be valid, and the summary should be accurate and concise. **Automated Checks:** - Tool call validation using a JSON schema. - Summary accuracy using an LLM-as-a-judge. **Human Review Rubric:** - **Tool Call Validity (1-5):** Are the tool calls valid and correctly formatted? - **Task Completion (1-5):** Did the model successfully complete the objective? - **Summary Quality (1-5):** Is the summary accurate and concise? **Failure Examples:** - The model generates invalid tool calls. - The model fails to use the tools to gather information. - The model generates a summary that is inaccurate or incomplete. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Tool call validity score - Task completion score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is critical for agentic workloads, as the model may generate a large number of reasoning tokens before producing the final answer. If the model generates more than 8192 tokens, the output must be truncated, which may result in incomplete tool calls or missing actions. **Additional Considerations:** - **Tool Use:** The model should be able to use a variety of tools, including search engines, calculators, and databases. - **Planning:** The model should be able to plan and execute multi-step tasks. - **Decision Making:** The model should be able to make decisions based on the information gathered from the tools. # 9. RAG Workloads (Expanded) **Realistic Task Design:** The RAG workload evaluates the model's ability to retrieve relevant information from a knowledge base and generate a response based on that information. The task involves presenting the model with a question and a set of documents, requiring the model to retrieve the relevant documents and generate a response. **Prompt Shape:** ```json { "system": "You are an AI assistant. Your task is to answer the question based on the provided documents.", "user": "Question: What is the capital of France?\n\nDocuments:\n- Document 1: The capital of France is Paris.\n- Document 2: The capital of Germany is Berlin.\n\nPlease answer the question based on the documents." } ``` **Expected Answer Shape:** The expected answer includes a response to the question, along with citations to the relevant documents. The response should be accurate and concise. **Automated Checks:** - Response accuracy using an LLM-as-a-judge. - Citation accuracy using a regex pattern. **Human Review Rubric:** - **Response Accuracy (1-5):** Is the response accurate? - **Citation Accuracy (1-5):** Are the citations accurate and correctly formatted? - **Conciseness (1-5):** Is the response concise? **Failure Examples:** - The model generates a response that is inaccurate. - The model fails to cite the relevant documents. - The model generates a response that is verbose or incomplete. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Response accuracy score - Citation accuracy score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is sufficient for most RAG workloads, but complex questions may require more tokens. If the model generates more than 8192 tokens, the output must be truncated, which may result in incomplete responses or missing citations. **Additional Considerations:** - **Retrieval:** The model should be able to retrieve relevant information from the knowledge base. - **Generation:** The model should be able to generate a response based on the retrieved information. - **Citation:** The model should be able to cite the relevant documents. # 10. Chatbot Workloads (Expanded) **Realistic Task Design:** The chatbot workload evaluates the model's ability to engage in a multi-turn conversation. The task involves presenting the model with a series of user inputs, requiring the model to generate a response for each input. **Prompt Shape:** ```json { "system": "You are a helpful AI assistant. Your task is to respond to the user's inputs.", "user": "Hello, how are you?" } ``` **Expected Answer Shape:** The expected answer includes a response to the user's input. The response should be polite, helpful, and relevant. **Automated Checks:** - Toxicity detection using a toxicity classifier. - Relevance detection using an LLM-as-a-judge. **Human Review Rubric:** - **Politeness (1-5):** Is the response polite? - **Helpfulness (1-5):** Is the response helpful? - **Relevance (1-5):** Is the response relevant to the user's input? **Failure Examples:** - The model generates a response that is impolite or offensive. - The model generates a response that is not helpful. - The model generates a response that is irrelevant to the user's input. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Politeness score - Helpfulness score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is sufficient for most chatbot workloads, but long conversations may require more tokens. If the model generates more than 8192 tokens, the output must be truncated, which may result in incomplete responses. **Additional Considerations:** - **Conversational Flow:** The model should be able to maintain a natural and coherent conversational flow. - **Context Awareness:** The model should be able to remember the context of the conversation and generate relevant responses. - **Personality:** The model should be able to adopt a specific personality and tone. # 11. Creative And Editorial Workloads (Expanded) **Realistic Task Design:** The creative and editorial workload evaluates the model's ability to generate creative content and edit existing content. The task involves presenting the model with a creative prompt, requiring the model to generate a story, poem, or article. **Prompt Shape:** ```json { "system": "You are a creative writer. Your task is to write a story based on the provided prompt.", "user": "Prompt: Write a story about a robot that learns to love." } ``` **Expected Answer Shape:** The expected answer includes a story that is creative, engaging, and well-written. The story should follow the prompt and include a clear beginning, middle, and end. **Automated Checks:** - Creativity detection using an LLM-as-a-judge. - Grammar and spelling detection using a grammar checker. **Human Review Rubric:** - **Creativity (1-5):** Is the story creative and engaging? - **Grammar and Spelling (1-5):** Is the story free of grammar and spelling errors? - **Coherence (1-5):** Is the story coherent and well-structured? **Failure Examples:** - The model generates a story that is not creative or engaging. - The model generates a story with grammar and spelling errors. - The model generates a story that is incoherent or poorly structured. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Creativity score - Grammar and spelling score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is sufficient for most creative and editorial workloads, but long stories may require more tokens. If the model generates more than 8192 tokens, the output must be truncated, which may result in an incomplete story. **Additional Considerations:** - **Creativity:** The model should be able to generate creative and original content. - **Style:** The model should be able to adopt a specific writing style and tone. - **Editing:** The model should be able to edit existing content to improve its quality. # 12. Long-Output Reliability (Expanded) **Realistic Task Design:** The long-output reliability workload evaluates the model's ability to generate long outputs without losing coherence or hallucinating. The task involves presenting the model with a prompt that requires a long output, such as a detailed report or a long story. **Prompt Shape:** ```json { "system": "You are an AI assistant. Your task is to write a detailed report on the history of the internet.", "user": "Write a detailed report on the history of the internet, covering the following topics: the ARPANET, the development of TCP/IP, the creation of the World Wide Web, and the rise of social media." } ``` **Expected Answer Shape:** The expected answer includes a detailed report that covers all the required topics. The report should be well-structured, coherent, and free of hallucinations. **Automated Checks:** - Hallucination detection using an LLM-as-a-judge. - Coherence detection using a coherence classifier. **Human Review Rubric:** - **Completeness (1-5):** Does the report cover all the required topics? - **Coherence (1-5):** Is the report coherent and well-structured? - **Accuracy (1-5):** Is the report free of hallucinations? **Failure Examples:** - The model generates a report that is incomplete or missing topics. - The model generates a report that is incoherent or poorly structured. - The model generates a report with hallucinations. **Metrics to Graph:** - Time-to-first-token (TTFT) - Tokens-per-second (TPS) - Completeness score - Coherence score **Notes on Reasoning Budget:** The reasoning budget of 8192 tokens is critical for long-output reliability workloads, as the model may generate a large number of tokens before producing the final answer. If the model generates more than 8192 tokens, the output must be truncated, which may result in an incomplete report. **Additional Considerations:** - **Coherence:** The model should be able to maintain coherence throughout the long output. - **Hallucination:** The model should be able to avoid hallucinations and generate accurate information. - **Structure:** The model should be able to structure the long output in a logical and easy-to-read format. # 13. Privacy And Redaction (Expanded) Privacy and redaction are critical considerations for any benchmark campaign, especially when dealing with sensitive data. The AI Flight Recorder must ensure that all data is properly redacted before being stored or analyzed. **Data Redaction:** All data must be redacted using a redaction tool that removes or replaces sensitive information, such as names, addresses, and phone numbers. The redaction tool must be configured to handle a wide range of data types and formats. **Data Anonymization:** All data must be anonymized using a data anonymization tool that replaces sensitive information with synthetic data. The anonymization tool must be configured to handle a wide range of data types and formats. **Privacy and Redaction Checklist:** - [ ] Configure the redaction tool to remove or replace sensitive information. - [ ] Configure the anonymization tool to replace sensitive information with synthetic data. - [ ] Verify that all data is properly redacted and anonymized before being stored or analyzed. - [ ] Implement access controls to restrict access to sensitive data. - [ ] Encrypt sensitive data at rest and in transit. **Failure Modes:** - **Data Leakage:** If sensitive data is not properly redacted or anonymized, it may be leaked to unauthorized parties. - **Compliance Violations:** If sensitive data is not properly handled, it may violate privacy regulations, such as GDPR or HIPAA. - **Re-identification:** If the anonymization tool is not implemented correctly, the sensitive data may be re-identified. # 14. SQLite Storage And Artifact Layout (Expanded) The AI Flight Recorder stores all benchmark data in a SQLite database. The database schema must be carefully designed to ensure that all data is properly indexed and queryable. **Database Schema:** The database schema includes tables for storing model metadata, benchmark results, and system telemetry. The tables are indexed on the primary key and foreign keys to ensure fast queries. **Artifact Layout:** The artifact layout includes directories for storing model files, benchmark results, and system telemetry. The directories are organized by date and model name to ensure easy access and retrieval. **SQLite Storage And Artifact Layout Checklist:** - [ ] Design the database schema to ensure that all data is properly indexed and queryable. - [ ] Create directories for storing model files, benchmark results, and system telemetry. - [ ] Organize the directories by date and model name. - [ ] Implement a backup strategy to ensure that the data is not lost. - [ ] Implement a cleanup strategy to remove old data and free up disk space. **Failure Modes:** - **Data Corruption:** If the database is not properly maintained, the data may become corrupted. - **Data Loss:** If the backup strategy is not implemented correctly, the data may be lost. - **Disk Space Exhaustion:** If the cleanup strategy is not implemented correctly, the disk space may be exhausted. # 15. Reporting Plane And Screenshots (Expanded) The reporting plane provides a web-based interface for visualizing benchmark results. The reporting plane must be carefully designed to ensure that all data is properly visualized and easy to understand. **Dashboard Design:** The dashboard includes charts and graphs for visualizing benchmark results, such as TTFT, TPS, and accuracy scores. The dashboard must be responsive and easy to navigate. **Screenshot Capture:** The AI Flight Recorder must capture screenshots of the dashboard at regular intervals to ensure that all data is properly recorded. The screenshots must be stored in the artifact layout. **Reporting Plane And Screenshots Checklist:** - [ ] Design the dashboard to include charts and graphs for visualizing benchmark results. - [ ] Capture screenshots of the dashboard at regular intervals. - [ ] Store the screenshots in the artifact layout. - [ ] Implement a notification system to alert operators of any anomalies. - [ ] Implement a search function to allow operators to quickly find specific benchmark results. **Failure Modes:** - **Data Visualization Errors:** If the dashboard is not designed correctly, the data may be visualized incorrectly. - **Screenshot Capture Failures:** If the screenshot capture fails, the data may not be properly recorded. - **Dashboard Performance Issues:** If the dashboard is not optimized for performance, it may be slow to load and navigate. # 16. Model Leaderboards (Expanded) The model leaderboard provides a ranking of models based on their benchmark results. The leaderboard must be carefully designed to ensure that all models are properly ranked and easy to compare. **Ranking Methodology:** The ranking methodology includes a weighted average of the benchmark results, such as TTFT, TPS, and accuracy scores. The weights must be carefully chosen to reflect the importance of each metric. **Leaderboard Design:** The leaderboard includes a table for displaying the rankings, along with charts and graphs for visualizing the results. The leaderboard must be responsive and easy to navigate. **Model Leaderboards Checklist:** - [ ] Design the ranking methodology to include a weighted average of the benchmark results. - [ ] Design the leaderboard to include a table for displaying the rankings, along with charts and graphs for visualizing the results. - [ ] Implement a sorting function to allow operators to sort the leaderboard by different metrics. - [ ] Implement a filtering function to allow operators to filter the leaderboard by different criteria. - [ ] Implement a sharing function to allow operators to share the leaderboard with others. **Failure Modes:** - **Ranking Errors:** If the ranking methodology is not implemented correctly, the models may be ranked incorrectly. - **Leaderboard Performance Issues:** If the leaderboard is not optimized for performance, it may be slow to load and navigate. - **Data Inconsistencies:** If the data is not properly synchronized, the leaderboard may display inconsistent results. # 17. Reproducibility Checklist (Expanded) Reproducibility is critical for any benchmark campaign. The reproducibility checklist ensures that all experiments can be reproduced by other operators. **Reproducibility Checklist:** - [ ] Set the random seed to a fixed value. - [ ] Use the same model files and quantization formats. - [ ] Use the same llama.cpp runtime parameters. - [ ] Use the same AI Flight Recorder configuration. - [ ] Use the same workload prompts and expected answers. - [ ] Document the hardware configuration and software versions. - [ ] Document the benchmark results and analysis. - [ ] Share the benchmark code and data with others. **Failure Modes:** - **Non-Reproducible Results:** If the reproducibility checklist is not followed, the results may not be reproducible. - **Inconsistent Results:** If the hardware configuration or software versions are not documented, the results may be inconsistent. - **Lack of Transparency:** If the benchmark code and data are not shared, the results may not be transparent. # 18. Final Recommendations (Expanded) The final recommendations provide a summary of the best practices for evaluating open-weight GGUF models on a 32GB-class R9700 llama.cpp server using AI Flight Recorder. **Best Practices:** - Use the appropriate quantization format for the model and workload. - Configure the llama.cpp runtime parameters for optimal performance. - Use the AI Flight Recorder to capture all benchmark data. - Redact and anonymize all data before storing or analyzing. - Use the reporting plane to visualize benchmark results. - Use the model leaderboard to rank models based on their benchmark results. - Follow the reproducibility checklist to ensure that all experiments can be reproduced. **Future Work:** - Evaluate the performance of MTP and non-MTP inference on different models and workloads. - Evaluate the performance of the model on different context lengths. - Evaluate the performance of the model on different reasoning budgets. - Evaluate the performance of the model on different quantization formats. - Evaluate the performance of the model on different hardware configurations. - Evaluate the performance of the model on different operating systems. - Evaluate the performance of the model on different programming languages. **Additional Recommendations:** - **Continuous Monitoring:** Implement continuous monitoring to detect any anomalies in the benchmark results. - **Automated Testing:** Implement automated testing to ensure that the benchmark code is working correctly. - **Documentation:** Maintain comprehensive documentation of the benchmark methodology and results. - **Community Engagement:** Engage with the community to share the benchmark results and gather feedback. **Additional Recommendations (Expanded):** - **Continuous Monitoring:** Implement continuous monitoring to detect any anomalies in the benchmark results. This can be achieved by setting up alerts for sudden drops in TPS or increases in TTFT. - **Automated Testing:** Implement automated testing to ensure that the benchmark code is working correctly. This can be achieved by writing unit tests for the AI Flight Recorder and the workload generation scripts. - **Documentation:** Maintain comprehensive documentation of the benchmark methodology and results. This includes documenting the hardware configuration, software versions, and benchmark parameters. - **Community Engagement:** Engage with the community to share the benchmark results and gather feedback. This can be achieved by publishing the benchmark results on a public repository and inviting others to replicate the experiments. # 19. Advanced llama.cpp Configuration and Kernel Tuning To achieve maximum performance on the R9700 server, it is necessary to go beyond the basic llama.cpp configuration and tune the underlying system kernels and memory management settings. This section details the advanced configurations required to squeeze every last drop of performance out of the hardware. **CPU Affinity and NUMA Awareness:** The R9700 server is a non-uniform memory access (NUMA) architecture. This means that the memory attached to each CPU socket has different latencies depending on which CPU core is accessing it. To minimize latency, it is critical to bind the llama.cpp threads to the CPU cores that are closest to the memory containing the model weights and KV cache. This can be achieved using the `numactl` command or by setting the `OMP_NUM_THREADS` and `OMP_PROC_BIND` environment variables. **Memory Locking and Huge Pages:** To prevent the operating system from swapping memory pages to disk, it is recommended to lock the memory used by llama.cpp using the `mlock` system call. This can be achieved by setting the `--mlock` parameter in llama.cpp. Additionally, using huge pages (2MB or 1GB) instead of standard 4KB pages can significantly reduce the overhead of the Translation Lookaside Buffer (TLB) and improve memory access speeds. This can be achieved by setting the `--mem-lock` parameter in llama.cpp and configuring the operating system to allocate huge pages. **Kernel Tuning:** The operating system kernel can be tuned to optimize for high-throughput, low-latency inference. This includes adjusting the CPU frequency scaling governor to performance mode, disabling CPU frequency scaling, and tuning the I/O scheduler for the NVMe drives. These changes can be made using the `cpupower` command and by modifying the `/etc/sysctl.conf` file. **Advanced llama.cpp Configuration Checklist:** - [ ] Bind llama.cpp threads to the CPU cores closest to the memory containing the model weights and KV cache. - [ ] Lock the memory used by llama.cpp using the `--mlock` parameter. - [ ] Configure the operating system to allocate huge pages and use the `--mem-lock` parameter in llama.cpp. - [ ] Set the CPU frequency scaling governor to performance mode. - [ ] Disable CPU frequency scaling. - [ ] Tune the I/O scheduler for the NVMe drives. **Failure Modes:** - **NUMA Latency:** If the llama.cpp threads are not bound to the correct CPU cores, the memory access latency will increase, leading to a decrease in TPS. - **Memory Swapping:** If the memory is not locked, the operating system may swap memory pages to disk, leading to a significant drop in performance. - **TLB Thrashing:** If huge pages are not used, the TLB may thrash, leading to a decrease in memory access speeds. # 20. AI Flight Recorder Deep Dive and Telemetry Parsing The AI Flight Recorder is the central nervous system of the benchmark campaign. It is responsible for capturing every inference request, measuring latency, tracking token generation rates, recording system telemetry, and storing the resulting artifacts. This section provides a deep dive into the AI Flight Recorder and details how to parse and analyze the telemetry data. **Telemetry Data Points:** The AI Flight Recorder captures a wide range of telemetry data points, including: - **Inference Metrics:** Time-to-first-token (TTFT), tokens-per-second (TPS), time-per-token (TPT), and total inference time. - **System Metrics:** CPU utilization, memory usage, disk I/O, and network I/O. - **Thermal Metrics:** CPU temperature, GPU temperature, and fan speed. - **Error Metrics:** Number of OOM errors, number of thermal throttling events, and number of inference failures. **Telemetry Parsing:** The telemetry data is stored in a SQLite database and can be parsed using SQL queries. The AI Flight Recorder provides a set of SQL queries that can be used to extract and analyze the telemetry data. For example, the following SQL query can be used to extract the TTFT and TPS for a specific model and workload: ```sql SELECT model_name, workload, ttft, tps FROM inference_metrics WHERE model_name = 'llama-3-8b-q4_k_m' AND workload = 'coding'; ``` **Telemetry Visualization:** The telemetry data can be visualized using a variety of tools, including Grafana, Tableau, and custom Python scripts. The AI Flight Recorder provides a set of pre-built Grafana dashboards that can be used to visualize the telemetry data. These dashboards include charts and graphs for visualizing TTFT, TPS, CPU utilization, and memory usage. **AI Flight Recorder Deep Dive Checklist:** - [ ] Capture all inference metrics, system metrics, thermal metrics, and error metrics. - [ ] Store the telemetry data in a SQLite database. - [ ] Parse the telemetry data using SQL queries. - [ ] Visualize the telemetry data using Grafana or custom Python scripts. - [ ] Set up alerts for anomalies in the telemetry data. **Failure Modes:** - **Data Loss:** If the AI Flight Recorder is not configured correctly, the telemetry data may be lost. - **Data Corruption:** If the SQLite database is not properly maintained, the telemetry data may become corrupted. - **Visualization Errors:** If the Grafana dashboards are not configured correctly, the telemetry data may be visualized incorrectly. # 21. Quantization Impact Analysis Quantization is a critical factor in determining the performance and quality of the model. This section provides a detailed analysis of the impact of different quantization formats on the model's performance and quality. **Quantization Formats:** The primary quantization formats to be evaluated are Q4_K_M, Q5_K_M, and Q6_K. Q4_K_M offers the best balance between model size and quality, making it the default choice for most workloads. Q5_K_M provides a slight improvement in quality at the cost of increased memory usage, which may be necessary for models that require a larger context window. Q6_K is reserved for models where quality is paramount and memory is not a constraint. **Performance Impact:** The performance impact of quantization is primarily determined by the memory bandwidth required to load the model weights and KV cache. Q4_K_M requires the least amount of memory bandwidth, making it the fastest quantization format. Q5_K_M and Q6_K require more memory bandwidth, leading to a decrease in TPS. **Quality Impact:** The quality impact of quantization is primarily determined by the loss of precision in the model weights. Q4_K_M has the highest loss of precision, leading to a decrease in accuracy and coherence. Q5_K_M and Q6_K have a lower loss of precision, leading to a slight improvement in accuracy and coherence. **Quantization Impact Analysis Checklist:** - [ ] Evaluate the performance impact of different quantization formats. - [ ] Evaluate the quality impact of different quantization formats. - [ ] Determine the optimal quantization format for each workload. - [ ] Document the trade-offs between performance and quality for each quantization format. **Failure Modes:** - **Performance Degradation:** If the quantization format is too aggressive, the performance may degrade significantly. - **Quality Degradation:** If the quantization format is too aggressive, the quality may degrade significantly. - **OOM Error:** If the quantization format is too large, the system may run out of memory and the inference will fail. # 22. Context Window and KV Cache Management The context window and KV cache are critical components for managing long-context inference. This section provides a detailed analysis of the impact of different context lengths on the model's performance and quality. **Context Lengths:** The context lengths to be evaluated are 4096, 8192, 16384, and 32768 tokens. These lengths represent a range of use cases, from short conversations to long documents. The maximum context length is limited by the available RAM, so the 32768 token context length may only be feasible for smaller models. **Performance Impact:** The performance impact of the context length is primarily determined by the size of the KV cache. As the context length increases, the size of the KV cache increases, leading to a decrease in TPS. This is because the larger KV cache requires more memory bandwidth to load and access. **Quality Impact:** The quality impact of the context length is primarily determined by the model's ability to attend to the entire context. As the context length increases, the model's ability to attend to the entire context may decrease, leading to a decrease in accuracy and coherence. This is known as the "needle in a haystack" problem. **Context Window and KV Cache Management Checklist:** - [ ] Evaluate the performance impact of different context lengths. - [ ] Evaluate the quality impact of different context lengths. - [ ] Determine the optimal context length for each workload. - [ ] Document the trade-offs between performance and quality for each context length. **Failure Modes:** - **Performance Degradation:** If the context length is too large, the performance may degrade significantly. - **Quality Degradation:** If the context length is too large, the quality may degrade significantly. - **OOM Error:** If the KV cache is too large for the available RAM, the system will run out of memory and the inference will fail. # 23. Multi-Threaded Inference and Thread Contention Multi-threaded inference is a critical factor in determining the performance of the model. This section provides a detailed analysis of the impact of different thread configurations on the model's performance. **Thread Configurations:** The thread configurations to be evaluated are 16, 32, and 64 threads. The number of threads is determined by the `--threads` and `--threads-batch` parameters in llama.cpp. The optimal thread count is typically equal to the number of physical cores (32). **Performance Impact:** The performance impact of the thread configuration is primarily determined by the level of thread contention. As the number of threads increases, the level of thread contention may increase, leading to a decrease in TPS. This is because the threads may contend for CPU resources, such as the cache and the memory bus. **Quality Impact:** The quality impact of the thread configuration is primarily determined by the consistency of the inference. As the number of threads increases, the consistency of the inference may decrease, leading to a decrease in accuracy and coherence. This is because the threads may not be synchronized properly, leading to race conditions. **Multi-Threaded Inference and Thread Contention Checklist:** - [ ] Evaluate the performance impact of different thread configurations. - [ ] Evaluate the quality impact of different thread configurations. - [ ] Determine the optimal thread configuration for each workload. - [ ] Document the trade-offs between performance and quality for each thread configuration. **Failure Modes:** - **Performance Degradation:** If the number of threads is too high, the performance may degrade significantly due to thread contention. - **Quality Degradation:** If the number of threads is too high, the quality may degrade significantly due to race conditions. - **OOM Error:** If the number of threads is too high, the system may run out of memory and the inference will fail. # 24. Thermal Throttling and Power Management Thermal throttling and power management are critical factors in determining the sustained performance of the model. This section provides a detailed analysis of the impact of thermal throttling and power management on the model's performance. **Thermal Throttling:** Thermal throttling occurs when the CPU temperature exceeds the thermal limit, causing the CPU to reduce its clock speed. This can lead to a significant drop in performance, especially during long-running inference tasks. **Power Management:** Power management involves adjusting the CPU frequency and voltage to optimize for power consumption. This can lead to a decrease in performance, especially if the CPU is not allowed to boost to its maximum frequency. **Thermal Throttling and Power Management Checklist:** - [ ] Monitor the CPU temperature during inference to detect thermal throttling. - [ ] Configure the CPU frequency scaling governor to performance mode. - [ ] Disable CPU frequency scaling. - [ ] Ensure that the cooling system is adequate to prevent thermal throttling. - [ ] Document the impact of thermal throttling and power management on the model's performance. **Failure Modes:** - **Performance Degradation:** If thermal throttling occurs, the performance may degrade significantly. - **Inconsistent Results:** If power management is not configured correctly, the results may be inconsistent. - **Hardware Damage:** If the cooling system is inadequate, the CPU may overheat and suffer permanent damage. # 25. Speculative Decoding and MTP Synergy Speculative decoding and Multi-Token Prediction (MTP) are two techniques that can be used to improve the performance of the model. This section provides a detailed analysis of the synergy between speculative decoding and MTP. **Speculative Decoding:** Speculative decoding involves using a smaller, faster model to generate a draft of the output, which is then verified by the larger, slower model. This can significantly improve the performance of the model, especially for long outputs. **MTP Synergy:** MTP can be used in conjunction with speculative decoding to further improve the performance of the model. The draft model can use MTP to generate multiple tokens in a single forward pass, which can then be verified by the larger model. This can significantly reduce the number of forward passes required to generate a sequence of tokens. **Speculative Decoding and MTP Synergy Checklist:** - [ ] Evaluate the performance impact of speculative decoding. - [ ] Evaluate the performance impact of MTP. - [ ] Evaluate the performance impact of speculative decoding and MTP in conjunction. - [ ] Determine the optimal configuration for speculative decoding and MTP for each workload. - [ ] Document the trade-offs between performance and quality for speculative decoding and MTP. **Failure Modes:** - **Performance Degradation:** If speculative decoding or MTP is not implemented correctly, the performance may degrade significantly. - **Quality Degradation:** If speculative decoding or MTP is not implemented correctly, the quality may degrade significantly. - **OOM Error:** If speculative decoding or MTP requires more memory than is available, the system will run out of memory and the inference will fail.