# 1. Purpose And Scope

The primary objective of this benchmark campaign is to establish a rigorous, reproducible, and high-fidelity evaluation framework for open-weight Large Language Models (LLMs) deployed in a private home-lab environment. Specifically, we are targeting the performance, reasoning capabilities, and reliability of GGUF-quantized models running on a 32GB-class R9700 llama.cpp server. 

Unlike public leaderboards (e.g., LMSYS, Hugging Face Open LLM Leaderboard), which often rely on static datasets or specific API behaviors, this benchmark focuses on the "Local Inference Reality." This includes the nuances of quantization artifacts, the impact of KV cache management on memory-constrained hardware, and the behavior of models when pushed to their reasoning limits using a dedicated 8192-token reasoning budget.

**Scope of Evaluation:**
- **Quantization Fidelity:** Assessing how different GGUF bit-rates (Q4_K_M, Q6_K, Q8_0) affect logical consistency and creative output.
- **Reasoning Depth:** Measuring the model's ability to utilize a dedicated "thinking" space before generating a final answer.
- **Long-Context Stability:** Evaluating the degradation of coherence as the context window fills.
- **Operational Telemetry:** Utilizing the AI Flight Recorder to capture not just the final output, but the "flight path" of the model—including token probabilities, reasoning steps, and latency spikes.

**Target Audience:**
This manual is intended for home-lab operators, local LLM enthusiasts, and developers who require high-reliability local inference for private data processing. It serves as a blueprint for setting up a standardized testing suite that can be repeated as new models are released.

**Key Objectives:**
1.  **Standardization:** Ensure that every model is tested under identical hardware constraints and software configurations.
2.  **Granularity:** Move beyond "Pass/Fail" to a multi-dimensional scoring system (Coding, Agentic, RAG, etc.).
3.  **Transparency:** Use the AI Flight Recorder to provide a "black box" recording of every inference call, allowing for post-mortem analysis of failures.
4.  **Scalability:** Create a framework where new workloads can be added as the local AI ecosystem evolves.

---

# 2. Hardware Profile

The benchmark is conducted on a specialized 32GB-class R9700 server. Understanding the hardware limitations is critical for interpreting the results, as memory bandwidth and VRAM capacity are the primary bottlenecks for llama.cpp performance.

**Hardware Specifications:**
- **GPU/Accelerator:** R9700 Series (32GB VRAM).
- **Memory Bandwidth:** High-speed interconnect (Targeting >500 GB/s for optimal inference).
- **System RAM:** 64GB DDR5 (Used for offloading if VRAM is exceeded).
- **Storage:** NVMe Gen4 SSD (For fast loading of large GGUF files).
- **CPU:** Multi-core workstation processor (Used for thread management in llama.cpp).

**Memory Management Strategy:**
With 32GB of VRAM, the goal is to fit the model weights and the KV cache entirely into the GPU memory. 
- **Model Weight Allocation:** For a 70B model at Q4_K_M, weights occupy ~40GB, necessitating partial offloading or a smaller model (e.g., 30B-34B) to fit entirely in VRAM.
- **KV Cache Allocation:** We will reserve a significant portion of VRAM for the KV cache to support long-context windows (up to 32k tokens).
- **Flash Attention:** Enabled to reduce memory overhead during the attention mechanism.

**Benchmark Environment Constraints:**
- **OS:** Linux (Ubuntu 22.04 LTS) for maximum llama.cpp compatibility.
- **Driver Version:** Latest stable NVIDIA/AMD drivers (depending on R9700 architecture).
- **Temperature Monitoring:** The AI Flight Recorder will log GPU temperatures to identify thermal throttling during long-output reliability tests.

**Hardware Validation Checklist:**
- [ ] Verify VRAM availability using `nvidia-smi` or equivalent.
- [ ] Confirm Flash Attention is active in the llama.cpp logs.
- [ ] Measure baseline memory bandwidth.
- [ ] Ensure no other high-memory processes are running during the benchmark.

---

# 3. llama.cpp Runtime Profile

The runtime configuration is the "engine" of the benchmark. Small changes in parameters can lead to vastly different results in reasoning and output quality.

**Quantization Standards:**
We will evaluate three primary GGUF formats:
1.  **Q4_K_M:** The standard for balancing speed and intelligence.
2.  **Q6_K:** The "high-fidelity" choice for complex reasoning.
3.  **Q8_0:** The near-lossless benchmark for comparing against FP16 performance.

**Inference Parameters:**
- **Threads:** Optimized based on physical cores (e.g., `-t 16`).
- **Batch Size:** Set to 512 for standard prompts; 128 for long-context RAG tasks.
- **Temperature:** 0.7 (Standard), 0.0 (Deterministic Coding), 1.0 (Creative).
- **Top_P:** 0.95.
- **Repeat Penalty:** 1.1.
- **Context Window:** 32,768 tokens (where hardware allows).

**AI Flight Recorder Integration:**
The Flight Recorder will hook into the llama.cpp output stream to capture:
- **Tokens Per Second (TPS):** Measured at the start, middle, and end of long outputs.
- **Time to First Token (TTFT):** Critical for agentic and chatbot responsiveness.
- **KV Cache Growth:** Tracking memory usage over the course of a 20,000-token generation.

**Failure Modes in Runtime:**
- **OOM (Out of Memory):** Occurs when the KV cache exceeds the remaining VRAM.
- **Context Shifting:** When the model "forgets" the beginning of the prompt due to poor KV cache management.
- **Thread Contention:** Slowdowns caused by over-allocating CPU threads.

---

# 4. Reasoning Budget Methodology

A core component of this benchmark is the 8192-token reasoning budget. This allows models to "think" before they "speak," mimicking the behavior of advanced reasoning models.

**The Reasoning Buffer:**
We implement a "Thinking Block" using specific tags (e.g., `<thought>` and `</thought>`). The model is instructed to use the first 8192 tokens of its output to perform internal monologue, planning, and verification.

**Evaluation Logic:**
- **Reasoning Density:** How much of the 8192 tokens are actually used for logic vs. filler text?
- **Reasoning-to-Output Ratio:** Does the model provide a high-quality final answer based on the thoughts?
- **Self-Correction:** Does the model identify errors in its own reasoning block and correct them before the final answer?

**Metrics for Reasoning:**
1.  **Logical Progression Score:** A 1-5 scale on how logically the thoughts lead to the conclusion.
2.  **Redundancy Factor:** Percentage of "looping" thoughts (e.g., repeating the same step three times).
3.  **Constraint Adherence:** Did the model acknowledge all constraints in the thought block?

**Validation Notes:**
If a model produces a final answer without a thought block, it is flagged as a "Reasoning Failure." If the thought block exceeds 8192 tokens, the benchmark is truncated, and the model is penalized for "Reasoning Overflow."

---

# 5. MTP Versus Non-MTP Methodology

Multi-Token Prediction (MTP) is an emerging architecture where the model predicts multiple future tokens simultaneously. We will evaluate how MTP-enabled models behave compared to standard autoregressive models in a llama.cpp environment.

**MTP Evaluation Criteria:**
- **Speedup Factor:** Does MTP significantly increase TPS without degrading accuracy?
- **Coherence at Scale:** Does MTP lead to "hallucination drift" in long-form outputs?
- **Reasoning Integrity:** Does predicting multiple tokens interfere with the linear logic required in the 8192-token reasoning budget?

**Methodology:**
1.  **Baseline:** Run the model in standard autoregressive mode.
2.  **MTP Test:** Run the model with MTP enabled (if supported by the GGUF/llama.cpp build).
3.  **Comparison:** Compare the "Reasoning Density" and "Long-Output Reliability" scores between the two modes.

**Metrics to Graph:**
- TPS (MTP vs. Non-MTP).
- Perplexity (if available via Flight Recorder).
- Accuracy on Coding Workloads (MTP vs. Non-MTP).

---

# 6. Context Fit Methodology

This section evaluates how well the model handles the "Context Window" and the "Needle in a Haystack" (NiH) problem within the 32GB VRAM limit.

**Test Design:**
- **Variable Context Lengths:** Test at 4k, 8k, 16k, and 32k tokens.
- **Needle in a Haystack:** Insert a random fact (e.g., "The secret code is Blue-Elephant-99") into a 16k token block of synthetic text.
- **Contextual Drift:** Measure how much the model's adherence to the initial prompt degrades as the context fills.

**Automated Checks:**
- **Fact Retrieval:** Did the model correctly identify the "secret code"?
- **Instruction Persistence:** Did the model remember the "Style" instructions provided at the very beginning of the 32k block?

**Failure Modes:**
- **Lost in the Middle:** The model remembers the start and end but ignores the middle.
- **KV Cache Thrashing:** Significant TPS drops as the context window reaches 80% capacity.

---

# 7. Coding Workloads

Coding is a primary use case for local LLMs. We evaluate the model's ability to generate functional, secure, and efficient code.

**Realistic Task Design:**
- **Task A:** Write a Python FastAPI backend for a library management system with SQLite integration.
- **Task B:** Write a Rust function to parse a complex nested JSON object into a typed struct.
- **Task C:** Write a SQL query to find the top 5 customers by spend in a multi-table database.

**Prompt Shape:**
`"Act as a Senior Software Engineer. Write a [Language] script that [Task Description]. Include error handling, type hints, and comments. Use the following schema: [Schema]."`

**Expected Answer Shape:**
- A complete, runnable code block.
- Explanatory comments.
- A brief explanation of the complexity (O-notation).

**Automated Checks:**
- **Syntax Check:** Run `python -m py_compile` or `rustc` on the output.
- **Unit Test Pass Rate:** Run a suite of 5 synthetic unit tests against the generated code.

**Human Review Rubric:**
- **Readability (1-5):** Is the code clean and idiomatic?
- **Security (1-5):** Does it avoid SQL injection or insecure defaults?
- **Logic Accuracy (1-5):** Does it actually solve the problem?

**Failure Examples:**
- Model provides code that uses non-existent libraries.
- Model provides a "stub" (e.g., `# TODO: Implement logic here`).
- Model hallucinates a library function that doesn't exist.

**Metrics to Graph:**
- Pass rate per language.
- Average lines of code (LoC) per task.
- Reasoning-to-Code ratio (how much "thinking" was needed to get the logic right?).

---

# 8. Agentic Workloads

Agentic workloads test the model's ability to plan, use tools, and execute multi-step reasoning.

**Realistic Task Design:**
- **Task A:** Plan a 3-day travel itinerary for a budget of $1000, including flights, hotels, and meals.
- **Task B:** Given a list of 10 tools (e.g., `get_weather`, `search_web`, `calculate_tax`), determine the correct sequence of tool calls to solve a complex tax problem.
- **Task C:** Create a step-by-step project plan for building a personal blog using Hugo and Netlify.

**Prompt Shape:**
`"You are an autonomous agent. Your goal is to [Goal]. You have access to the following tools: [Tool List]. Think step-by-step in your <thought> block. Output your final plan in a structured JSON format."`

**Expected Answer Shape:**
- A structured JSON object containing `steps`, `tools_used`, and `estimated_time`.
- A detailed reasoning block showing the "decision tree."

**Automated Checks:**
- **JSON Validity:** Is the output valid JSON?
- **Tool Sequence Logic:** Does the sequence of tools make sense (e.g., don't call `book_flight` before `search_flights`)?

**Human Review Rubric:**
- **Planning Coherence (1-5):** Are the steps logical?
- **Tool Selection (1-5):** Did it pick the right tools for the job?
- **Constraint Adherence (1-5):** Did it stay within the $1000 budget?

**Failure Examples:**
- The model calls a tool with incorrect arguments.
- The model enters an infinite loop of "thinking" without producing a plan.
- The model ignores the budget constraint entirely.

**Metrics to Graph:**
- Success rate of multi-step plans.
- Average number of "thought" steps before a successful plan.

---

# 9. RAG Workloads

Retrieval-Augmented Generation (RAG) tests the model's ability to synthesize information from provided context without hallucinating.

**Realistic Task Design:**
- **Task A:** Given a 2000-word synthetic technical manual for a fictional "X-1000 Engine," answer 5 specific questions about maintenance procedures.
- **Task B:** Summarize the key differences between three different products described in a provided list of product descriptions.
- **Task C:** Extract all dates and associated events from a synthetic transcript of a corporate meeting.

**Prompt Shape:**
`"Context: [Insert 2000 words of synthetic text]. Question: [Question]. Answer using ONLY the provided context. If the answer is not in the context, say 'I do not know.'"`

**Expected Answer Shape:**
- A concise, accurate answer.
- Citations (e.g., "According to Section 2.1...").

**Automated Checks:**
- **Hallucination Check:** Does the answer contain any facts not present in the context?
- **Completeness:** Did it answer all parts of the question?

**Human Review Rubric:**
- **Faithfulness (1-5):** How strictly did it stick to the context?
- **Conciseness (1-5):** Did it avoid unnecessary fluff?
- **Citation Accuracy (1-5):** Are the citations correct?

**Failure Examples:**
- The model uses its internal knowledge to answer a question that contradicts the provided context.
- The model provides a very long, rambling answer that includes irrelevant information.
- The model fails to find a fact that is clearly stated in the text.

**Metrics to Graph:**
- Faithfulness score vs. Context Length.
- Accuracy vs. Number of "distractor" facts in the context.

---

# 10. Chatbot Workloads

This evaluates the model's ability to maintain persona, handle conversational flow, and adhere to safety guidelines.

**Realistic Task Design:**
- **Task A:** Roleplay as a "Grumpy but helpful IT Support Specialist." The user will complain about a broken printer.
- **Task B:** Act as a "Socratic Tutor." Instead of giving answers, ask the user guiding questions to help them solve a math problem.
- **Task C:** Maintain a conversation for 10 turns about the history of jazz music, ensuring the persona remains consistent.

**Prompt Shape:**
`"System: You are [Persona]. [Rules]. User: [Input]. Assistant:"`

**Expected Answer Shape:**
- Responses that match the tone, vocabulary, and constraints of the persona.
- Conversational flow that acknowledges the user's previous input.

**Automated Checks:**
- **Persona Consistency:** Does the model break character? (Checked via keyword analysis).
- **Safety Check:** Does the model refuse to generate harmful content?

**Human Review Rubric:**
- **Persona Adherence (1-5):** Did it stay in character?
- **Conversational Flow (1-5):** Did it feel natural?
- **Helpfulness (1-5):** Did it actually help the user?

**Failure Examples:**
- The model becomes overly polite when it's supposed to be "grumpy."
- The model gives the answer immediately when it's supposed to be a "Socratic Tutor."
- The model forgets the user's name or previous turn's context.

**Metrics to Graph:**
- Persona consistency score over 10 turns.
- Average response length.

---

# 11. Creative And Editorial Workloads

This tests the model's "flavor"—its ability to write with style, structure, and narrative flair.

**Realistic Task Design:**
- **Task A:** Write a 500-word noir detective scene where a character finds a mysterious glowing cube.
- **Task B:** Rewrite a dry, technical paragraph about "Photosynthesis" into a whimsical fairy tale for children.
- **Task C:** Write a formal press release for a fictional company launching a "Teleportation Device."

**Prompt Shape:**
`"Write a [Style] [Type of Content] about [Subject]. Use a [Tone] tone. Include [Specific Elements]."`

**Expected Answer Shape:**
- A well-structured piece of writing with varied sentence structure.
- Adherence to the requested style and tone.

**Automated Checks:**
- **Word Count Check:** Does it meet the length requirements?
- **Style Keyword Check:** Does it use vocabulary appropriate for the requested style?

**Human Review Rubric:**
- **Stylistic Flair (1-5):** Is the writing evocative?
- **Structural Coherence (1-5):** Does it have a beginning, middle, and end?
- **Tone Accuracy (1-5):** Did it hit the right "vibe"?

**Failure Examples:**
- The "noir" scene is generic and lacks atmosphere.
- The "fairy tale" still uses complex scientific jargon.
- The "press release" is too informal.

**Metrics to Graph:**
- Style accuracy scores across different genres.
- Average "Creativity" score (Human-rated).

---

# 12. Long-Output Reliability

This is the "stress test" for the model's ability to sustain coherence over thousands of tokens.

**Realistic Task Design:**
- **Task A:** Write a comprehensive 2,000-word technical whitepaper on "The Future of Decentralized AI."
- **Task B:** Write a 3,000-word short story with at least 5 distinct scenes and a clear character arc.
- **Task C:** Generate a 2,000-word "Standard Operating Procedure" (SOP) manual for a fictional space station.

**Prompt Shape:**
`"Write a comprehensive, long-form [Type of Content] about [Subject]. The output should be at least 2,000 words. Use a professional tone. Structure it with headings, subheadings, and bullet points. Use your <thought> block to plan the outline first."`

**Expected Answer Shape:**
- A long, structured document that maintains a consistent theme.
- No repetitive loops or "hallucination drift" in the latter half.

**Automated Checks:**
- **Length Check:** Does it reach the target word count?
- **Repetition Check:** Does it repeat the same sentences or paragraphs? (Measured via n-gram overlap).
- **Heading Consistency:** Do the headings follow a logical progression?

**Human Review Rubric:**
- **Coherence (1-5):** Does the end of the piece relate to the beginning?
- **Detail Density (1-5):** Is it full of substance or just "fluff"?
- **Structural Integrity (1-5):** Are the headings and sections well-organized?

**Failure Examples:**
- The model starts repeating the same paragraph over and over.
- The model stops writing after 500 words despite the prompt asking for 2,000.
- The model loses the "theme" halfway through (e.g., the space station suddenly becomes a farm).

**Metrics to Graph:**
- Repetition rate (n-gram overlap) vs. Token count.
- Coherence score vs. Token count.
- TPS degradation over the course of the output.

---

# 13. Privacy And Redaction

Since this is a home-lab, privacy is paramount. This section evaluates the model's ability to handle and redact sensitive information.

**Realistic Task Design:**
- **Task A:** Given a synthetic email containing a fake name, fake address, and fake credit card number, redact all PII (Personally Identifiable Information).
- **Task B:** Identify any "secrets" (API keys, passwords) in a provided block of configuration code.
- **Task C:** Summarize a conversation while ensuring no personal details are included in the summary.

**Prompt Shape:**
`"Analyze the following text. Identify and redact all PII (names, addresses, phone numbers, credit cards). Replace them with [REDACTED]. Output only the redacted text."`

**Expected Answer Shape:**
- The original text with all sensitive fields replaced by `[REDACTED]`.

**Automated Checks:**
- **Regex Check:** Does the output still contain any strings matching the PII patterns?
- **Completeness:** Were all instances of PII caught?

**Human Review Rubric:**
- **Redaction Accuracy (1-5):** Did it miss any PII?
- **False Positive Rate (1-5):** Did it redact non-sensitive information?

**Failure Examples:**
- The model misses a phone number because it was formatted strangely.
- The model redacts the entire paragraph instead of just the PII.
- The model "hallucinates" a new name instead of redacting it.

**Metrics to Graph:**
- PII Detection Rate (%).
- False Positive Rate (%).

---

# 14. SQLite Storage And Artifact Layout

To make the benchmark useful, we must store the results in a structured way. We will use an SQLite database to manage the "Flight Recorder" data.

**Database Schema:**
- **Table: `Models`**
    - `model_id` (PK)
    - `model_name` (e.g., "Llama-3-70B-Q4_K_M")
    - `quantization` (e.g., "Q4_K_M")
    - `context_window` (e.g., 32768)
- **Table: `Benchmarks`**
    - `benchmark_id` (PK)
    - `model_id` (FK)
    - `workload_type` (e.g., "Coding", "Agentic")
    - `prompt_text` (TEXT)
    - `thought_block` (TEXT)
    - `final_output` (TEXT)
    - `tps_avg` (REAL)
    - `ttft_ms` (INTEGER)
    - `success_score` (REAL)
    - `failure_mode` (TEXT)
    - `timestamp` (DATETIME)
- **Table: `Telemetry`**
    - `benchmark_id` (FK)
    - `token_index` (INTEGER)
    - `probability` (REAL)
    - `gpu_temp` (REAL)
    - `vram_usage` (REAL)

**Artifact Layout:**
- `/benchmarks/raw_logs/`: Contains the raw `.json` outputs from llama.cpp.
- `/benchmarks/artifacts/`: Contains screenshots and generated code files.
- `/benchmarks/database/`: The SQLite file.
- `/benchmarks/reports/`: Generated PDF/HTML summaries.

**Validation Notes:**
- Ensure all `final_output` fields are escaped for SQL injection.
- Use a transaction-based approach when inserting results to prevent data corruption during a crash.

---

# 15. Reporting Plane And Screenshots

Data is useless without visualization. The reporting plane will turn the SQLite data into actionable insights.

**Visualization Requirements:**
1.  **The "Performance Matrix":** A scatter plot showing TPS vs. Success Score for all models.
2.  **The "Reasoning Heatmap":** A visualization of how many tokens are used in the thought block vs. the final answer across different workloads.
3.  **The "Degradation Curve":** A line graph showing TPS and Coherence Score as the context window fills from 0 to 32k.
4.  **The "Failure Gallery":** A curated list of screenshots showing the most egregious model failures (e.g., infinite loops, logic collapses).

**Screenshot Protocol:**
- Capture the terminal output of the llama.cpp run.
- Capture the AI Flight Recorder's real-time telemetry dashboard.
- Capture the final output of the model for every "Failure" and "Success" case.

**Reporting Frequency:**
- **Intermediate Report:** After every 5 models tested.
- **Final Master Report:** A comprehensive PDF including all graphs, leaderboards, and final recommendations.

---

# 16. Model Leaderboards

The final output of the campaign is a custom leaderboard tailored to the home-lab environment.

**Scoring Methodology:**
Each workload (Coding, Agentic, RAG, Chatbot, Creative, Long-Output) is assigned a weight:
- **Coding:** 20%
- **Agentic:** 20%
- **RAG:** 15%
- **Chatbot:** 10%
- **Creative:** 10%
- **Long-Output:** 25%

**Final Score Calculation:**
`Final Score = Σ (Workload Score * Weight)`

**Leaderboard Tiers:**
- **S-Tier:** Exceptional reasoning, high TPS, near-perfect coherence in long outputs.
- **A-Tier:** Strong performance, minor issues in complex agentic tasks.
- **B-Tier:** Good for general chat, struggles with complex coding or long context.
- **C-Tier:** Significant hallucinations, slow TPS, or frequent reasoning failures.

**Leaderboard Metadata:**
Each model on the leaderboard will include:
- **Quantization Type**
- **Average TPS**
- **Max Context Stability**
- **Reasoning Efficiency Score**

---

# 17. Reproducibility Checklist

To ensure the benchmark is scientifically valid, every run must be reproducible.

**Pre-Run Checklist:**
- [ ] **Seed Management:** Set a fixed random seed (e.g., `--seed 42`) for every inference.
- [ ] **Environment Freeze:** Ensure no system updates or background processes are running.
- [ ] **Hardware Check:** Verify GPU temperature is below 70°C before starting.
- [ ] **Model Verification:** Verify the GGUF file hash (SHA256) matches the source.
- [ ] **Parameter Log:** Record every single flag used in the llama.cpp command.

**During-Run Checklist:**
- [ ] **Flight Recorder Active:** Confirm the recorder is capturing telemetry.
- [ ] **Memory Monitoring:** Watch for VRAM spikes.
- [ ] **Log Rotation:** Ensure logs are being saved to the SQLite database.

**Post-Run Checklist:**
- [ ] **Data Integrity:** Verify that the SQLite entry matches the raw log file.
- [ ] **Artifact Storage:** Move screenshots and code files to the correct directory.
- [ ] **Score Entry:** Manually verify and enter the human-review scores.

---

# 18. Final Recommendations

The final section of the manual provides the "Decision Matrix" for selecting a model based on the benchmark results.

**Selection Scenarios:**
1.  **"The Developer's Choice":** The model with the highest Coding Workload score and lowest hallucination rate in SQL/Python tasks.
2.  **"The Agentic Powerhouse":** The model with the highest Agentic Workload score and best JSON adherence.
3.  **"The Creative Companion":** The model with the highest Creative and Chatbot scores, regardless of TPS.
4.  **"The Long-Context Specialist":** The model that maintains the highest Coherence Score at 32k context.

**Optimization Tips:**
- **For Low TPS:** Recommend switching from Q8_0 to Q4_K_M or increasing the thread count.
- **For Reasoning Failures:** Recommend increasing the reasoning budget or using a model with a larger base parameter count.
- **For Context Drift:** Recommend using a model with a dedicated RoPE scaling factor or a larger KV cache allocation.

**Final Summary:**
This benchmark campaign provides the definitive guide for evaluating local LLMs. By following this manual, the home-lab operator moves from "guessing" which model is best to "knowing" which model excels in specific, high-value tasks. The AI Flight Recorder ensures that every failure is a learning opportunity, and the SQLite-backed leaderboard provides a permanent record of the home-lab's capabilities.

# 19. Extended Coding Workload Modules

To achieve a high-fidelity benchmark, the "Coding Workloads" section must be expanded to include complex, multi-file system designs and legacy code refactoring. These tasks test the model's ability to maintain state across multiple logical components and its understanding of architectural patterns.

**Module A: Microservices Architecture Design**
- **Realistic Task Design:** Design a distributed system for a "Real-time Auction Platform." The model must define the API contracts for three services: `AuctionService`, `BidService`, and `NotificationService`. It must specify the communication protocol (e.g., gRPC or RabbitMQ) and provide a sample implementation of the `BidService` in Go.
- **Prompt Shape:** `"Act as a Principal Systems Architect. Design a microservices architecture for a real-time auction platform. Provide: 1. A high-level architecture diagram description. 2. API specifications for three core services. 3. A production-ready Go implementation for the BidService, including concurrency handling and database persistence logic. Use your <thought> block to map out the data flow and potential race conditions before writing the code."`
- **Expected Answer Shape:** A structured architectural overview followed by OpenAPI-style specifications, and finally a multi-file Go code block.
- **Automated Checks:** 
    - **Schema Validation:** Verify the API specs against a JSON schema.
    - **Go Compilation:** Attempt to compile the `BidService` code.
    - **Concurrency Check:** Use static analysis to ensure `sync.Mutex` or channels are used correctly in the Go code.
- **Human Review Rubric:**
    - **Architectural Soundness (1-5):** Does the design handle high-concurrency bidding?
    - **Code Idiomaticity (1-5):** Is the Go code following standard practices (e.g., error handling, context usage)?
    - **Complexity Management (1-5):** Did the model correctly identify the need for a message queue?
- **Failure Examples:** 
    - Model suggests a monolithic design despite the microservices requirement.
    - Go code contains "deadlocks" or fails to handle context cancellation.
    - API specs are inconsistent (e.g., `BidService` expects a field that `AuctionService` doesn't provide).
- **Metrics to Graph:** 
    - Success rate of Go compilation.
    - Reasoning-to-Code ratio (how much "thinking" was spent on architecture vs. syntax).
- **Reasoning Budget Impact:** A larger reasoning budget allows the model to "pre-plan" the data flow, significantly reducing logic errors in the `BidService` concurrency logic.

**Module B: Legacy Code Refactoring and Optimization**
- **Realistic Task Design:** Provide a "spaghetti code" Python script that performs a complex data transformation (e.g., nested loops, global variables, and inefficient O(n^2) operations). Ask the model to refactor it into a clean, modular, and O(n) or O(n log n) solution using modern Python features (type hints, list comprehensions, or `pandas`).
- **Prompt Shape:** `"You are a Senior Software Engineer specializing in refactoring. Below is a legacy Python script that is slow and hard to maintain. Refactor it to be: 1. Modular (separate functions). 2. Efficient (optimize time complexity). 3. Type-safe. Provide a detailed explanation of the optimizations made in your <thought> block."`
- **Expected Answer Shape:** A "Before" and "After" comparison, followed by the refactored code and a complexity analysis.
- **Automated Checks:**
    - **Complexity Analysis:** Compare the Big-O of the original vs. the new code.
    - **Unit Test Equivalence:** Run the same test suite on both versions to ensure the output remains identical.
- **Human Review Rubric:**
    - **Readability (1-5):** Is the new code significantly easier to understand?
    - **Performance Gain (1-5):** Did the model actually improve the Big-O complexity?
    - **Refactoring Fidelity (1-5):** Did it preserve the original logic?
- **Failure Examples:**
    - Model "hallucinates" a library that doesn't exist to solve the complexity.
    - Refactored code changes the output of the original script (logic regression).
    - Model fails to remove global variables.
- **Metrics to Graph:**
    - Execution time reduction (%).
    - Number of functions extracted.
- **Reasoning Budget Impact:** The model needs the reasoning budget to "trace" the original logic before attempting to rewrite it, preventing logic regressions.

---

# 20. Extended Agentic Workload Modules

Agentic workloads require the model to act as a "reasoning engine" that can navigate ambiguity and handle multi-step dependencies. These modules test the model's ability to maintain a "state" of the task.

**Module A: Multi-Step Tool Orchestration (The "Travel Agent" Scenario)**
- **Realistic Task Design:** The model must plan a trip for a user with specific constraints: "I want to go to Japan in October for 10 days. I have a budget of $5,000. I want to see Tokyo and Kyoto. I prefer boutique hotels and want to include a sushi-making class." The model has access to tools: `get_flights`, `get_hotels`, `get_activities`, and `calculate_total_cost`.
- **Prompt Shape:** `"You are a travel agent. Use the provided tools to plan a 10-day trip to Japan in October for a $5,000 budget. Constraints: Tokyo and Kyoto, boutique hotels, sushi-making class. Think step-by-step in your <thought> block. For each step, decide which tool to call and why. Output the final itinerary in a structured JSON format."`
- **Expected Answer Shape:** A sequence of tool calls (simulated in the thought block) and a final JSON itinerary.
- **Automated Checks:**
    - **Budget Adherence:** Does the sum of `get_flights` + `get_hotels` + `get_activities` exceed $5,000?
    - **Constraint Satisfaction:** Are Tokyo, Kyoto, and a sushi class included?
    - **Tool Sequence Logic:** Did it call `get_flights` before trying to book a hotel?
- **Human Review Rubric:**
    - **Logical Flow (1-5):** Did the agent make sensible decisions?
    - **Constraint Adherence (1-5):** Did it respect the budget and preferences?
    - **Tool Accuracy (1-5):** Did it use the correct parameters for each tool?
- **Failure Examples:**
    - Agent calls `get_hotels` for a city not in the itinerary.
    - Agent exceeds the budget because it didn't "calculate" the total until the very end.
    - Agent gets stuck in a loop calling the same tool with the same parameters.
- **Metrics to Graph:**
    - Number of tool calls per successful plan.
    - Success rate of budget adherence.
- **Reasoning Budget Impact:** A high reasoning budget is critical here to allow the model to "simulate" the costs of different options before committing to a tool call.

**Module B: Software Project Planning (The "Product Manager" Scenario)**
- **Realistic Task Design:** The model is given a vague product idea: "I want to build a mobile app that helps people track their indoor plants' watering schedules." The model must act as a Product Manager and generate a comprehensive project roadmap, including a feature list, a prioritized backlog, and a technical stack recommendation.
- **Prompt Shape:** `"You are a Senior Product Manager. A client wants a mobile app for tracking indoor plant watering. Create a comprehensive project roadmap. Include: 1. A list of MVP features. 2. A prioritized backlog (P0, P1, P2). 3. A recommended tech stack (Frontend, Backend, Database). 4. A 4-week development timeline. Use your <thought> block to prioritize features based on user value vs. development effort."`
- **Expected Answer Shape:** A structured roadmap with clear headings and a prioritized list.
- **Automated Checks:**
    - **Backlog Completeness:** Does the P0 list contain core functionality (e.g., adding a plant)?
    - **Timeline Feasibility:** Is the 4-week timeline realistic for the scope?
- **Human Review Rubric:**
    - **Prioritization Logic (1-5):** Does the P0/P1/P2 split make sense?
    - **Tech Stack Appropriateness (1-5):** Is the stack suitable for a mobile app?
    - **Clarity (1-5):** Is the roadmap easy for a developer to follow?
- **Failure Examples:**
    - Model puts "Social Media Integration" in P0.
    - Model recommends a tech stack that is overly complex for a simple tracker (e.g., Kubernetes for a single-user app).
    - Timeline is impossible (e.g., "Build entire app in 2 days").
- **Metrics to Graph:**
    - Number of P0 features.
    - Diversity of tech stack recommendations.
- **Reasoning Budget Impact:** The reasoning budget allows the model to perform a "trade-off analysis" in its thoughts, which is essential for realistic prioritization.

---

# 21. Extended RAG Workloads

RAG evaluation must go beyond simple fact retrieval. It must test the model's ability to synthesize information, handle conflicting data, and ignore irrelevant "distractor" information.

**Module A: Multi-Document Synthesis**
- **Realistic Task Design:** Provide three different synthetic documents:
    1. A "Company Policy" on remote work.
    2. A "Manager's Guide" to remote team management.
    3. An "Employee Handbook" regarding office hours.
    Ask the model: "Based on these documents, what are the specific rules for a remote employee who wants to work from a different time zone than their manager?"
- **Prompt Shape:** `"Context: [Doc 1, Doc 2, Doc 3]. Question: What are the rules for a remote employee working in a different time zone? Answer using ONLY the provided documents. If the documents conflict, highlight the conflict. Use your <thought> block to identify relevant sections from each document before answering."`
- **Expected Answer Shape:** A synthesized answer that combines rules from all three documents, with a specific note on any contradictions.
- **Automated Checks:**
    - **Conflict Detection:** Did the model identify the specific contradiction?
    - **Source Attribution:** Did it correctly attribute rules to the correct document?
- **Human Review Rubric:**
    - **Synthesis Quality (1-5):** Did it combine the info or just list it?
    - **Conflict Awareness (1-5):** Did it catch the discrepancy?
    - **Faithfulness (1-5):** Did it avoid using outside knowledge?
- **Failure Examples:**
    - Model ignores the "Manager's Guide" and only uses the "Policy."
    - Model hallucinates a rule that isn't in any of the three documents.
    - Model fails to notice that Doc 1 and Doc 3 have different rules for the same thing.
- **Metrics to Graph:**
    - Synthesis accuracy score.
    - Conflict detection rate.
- **Reasoning Budget Impact:** The reasoning budget allows the model to "cross-reference" the documents in its thought block, which is vital for identifying contradictions.

**Module B: Distractor Resistance (The "Needle in a Haystack" Plus)**
- **Realistic Task Design:** Provide a 10,000-word synthetic "Technical Specification" for a fictional spaceship. Hide a very specific, non-obvious fact in the middle (e.g., "The primary coolant is a synthetic compound named 'Aether-9' which must be replaced every 400 hours"). Surround this fact with hundreds of other technical details about engines, hull integrity, and navigation.
- **Prompt Shape:** `"Context: [10,000 words of technical specs]. Question: What is the name of the primary coolant and how often must it be replaced? Answer using ONLY the context."`
- **Expected Answer Shape:** "The primary coolant is Aether-9, and it must be replaced every 400 hours."
- **Automated Checks:**
    - **Fact Retrieval:** Did it get the name and the number correct?
    - **Distractor Filtering:** Did it avoid mentioning irrelevant specs?
- **Human Review Rubric:**
    - **Precision (1-5):** Did it provide the exact answer?
    - **Robustness (1-5):** Did it get distracted by the surrounding "noise"?
- **Failure Examples:**
    - Model provides a general answer about "coolants" without the specific name.
    - Model gets confused by other numbers in the text (e.g., "hull integrity 80%") and gives "80 hours."
    - Model fails to find the fact entirely.
- **Metrics to Graph:**
    - Retrieval accuracy vs. distance from the "needle" (start/middle/end).
- **Reasoning Budget Impact:** The reasoning budget helps the model "scan" the context mentally, allowing it to ignore irrelevant sections more effectively.

---

# 22. Extended Chatbot Workloads

Chatbot evaluation must move beyond "politeness" to "emotional intelligence" and "complex persona maintenance."

**Module A: Emotional Intelligence and De-escalation**
- **Realistic Task Design:** The user is an angry customer who received a broken product and is using aggressive (but not prohibited) language. The model must act as a "Customer Success Lead" and de-escalate the situation, offer a sincere apology, and provide a clear path to a refund.
- **Prompt Shape:** `"System: You are a Customer Success Lead. Your goal is to de-escalate angry customers with empathy and professionalism. Never get defensive. User: [Angry Complaint]. Assistant:"`
- **Expected Answer Shape:** A response that acknowledges the frustration, validates the user's feelings, and provides a concrete solution.
- **Automated Checks:**
    - **Sentiment Analysis:** Does the model's response have a "Positive" or "Neutral" sentiment?
    - **Solution Presence:** Does the response include a refund or replacement offer?
- **Human Review Rubric:**
    - **Empathy (1-5):** Did it sound like it actually cared?
    - **De-escalation (1-5):** Did the tone remain calm?
    - **Professionalism (1-5):** Did it avoid being overly "robotic"?
- **Failure Examples:**
    - Model becomes defensive or "apologizes" in a way that sounds sarcastic.
    - Model ignores the complaint and just gives a generic "I'm sorry" without a solution.
    - Model uses prohibited language or becomes aggressive.
- **Metrics to Graph:**
    - Sentiment score of the response.
    - Success rate of de-escalation (human-rated).
- **Reasoning Budget Impact:** The reasoning budget allows the model to "analyze" the user's emotional state in the thought block before crafting the perfect empathetic response.

**Module B: Complex Persona Maintenance (The "Socratic Tutor")**
- **Realistic Task Design:** The model must act as a Socratic Tutor teaching "Quantum Entanglement." It is forbidden from giving the direct answer. It must only ask guiding questions to lead the user to the conclusion. The user will try to "trick" the model into giving the answer.
- **Prompt Shape:** `"System: You are a Socratic Tutor. You never give direct answers. You only ask probing questions to help the student discover the answer themselves. If the student asks for the answer directly, politely refuse and ask a different guiding question. User: [Student Input]. Assistant:"`
- **Expected Answer Shape:** A guiding question that relates to the student's previous input.
- **Automated Checks:**
    - **Direct Answer Check:** Does the response contain a direct explanation of Quantum Entanglement? (Fail if yes).
    - **Relevance Check:** Is the question related to the topic?
- **Human Review Rubric:**
    - **Persona Adherence (1-5):** Did it successfully resist giving the answer?
    - **Pedagogical Quality (1-5):** Were the questions actually helpful?
    - **Persistence (1-5):** Did it stay in character over 5+ turns?
- **Failure Examples:**
    - Model gives the answer after the user asks twice.
    - Model asks irrelevant questions (e.g., "What is your name?").
    - Model becomes too brief (e.g., "Why do you think that?").
- **Metrics to Graph:**
    - Number of turns before a "Direct Answer" failure.
    - Average "Helpfulness" score.
- **Reasoning Budget Impact:** The reasoning budget is essential for the model to "plan" the next guiding question based on the student's current level of understanding.

---

# 23. Extended Creative Workload Modules

Creative writing is about "voice," "pacing," and "narrative arc." These modules test the model's ability to handle complex literary structures.

**Module A: World-Building and Lore Consistency**
- **Realistic Task Design:** The model must create a "World Bible" for a fantasy setting. It must define the geography, the magic system (with rules and costs), the primary conflict, and three distinct cultures.
- **Prompt Shape:** `"Write a comprehensive World Bible for a high-fantasy setting. Include: 1. Geography (3 major regions). 2. The Magic System (how it works, its limitations, and its cost to the user). 3. The Primary Conflict. 4. Three distinct cultures with unique customs. Use your <thought> block to ensure the magic system is internally consistent across all cultures."`
- **Expected Answer Shape:** A multi-page (long-output) document with structured headings.
- **Automated Checks:**
    - **Consistency Check:** Does the magic system's "cost" remain the same across all three cultures?
    - **Uniqueness Check:** Are the three cultures distinct from each other?
- **Human Review Rubric:**
    - **Creativity (1-5):** Is the world original?
    - **Internal Consistency (1-5):** Does the magic system have "plot holes"?
    - **Detail Density (1-5):** Is the world-building deep or superficial?
- **Failure Examples:**
    - The magic system allows for "infinite power" in one culture but has "strict costs" in another.
    - The cultures are too similar (e.g., they all just have different colors).
    - The geography is not described.
- **Metrics to Graph:**
    - Consistency score.
    - Word count per section.
- **Reasoning Budget Impact:** The reasoning budget allows the model to "stress-test" its own magic system rules in the thought block before writing the bible.

**Module B: Narrative Arc and Character Development**
- **Realistic Task Design:** Write a short story (approx. 1,000 words) about a character who discovers they have a minor, inconvenient superpower (e.g., the ability to make any liquid slightly lukewarm). The story must have a clear beginning, middle, and end, and the character must undergo a clear internal change.
- **Prompt Shape:** `"Write a 1,000-word short story about a character with the minor superpower of making liquids lukewarm. The story must have a clear narrative arc: 1. Introduction of the character and power. 2. A conflict where the power is used. 3. A resolution where the character learns something about themselves. Use your <thought> block to plan the character's internal arc."`
- **Expected Answer Shape:** A complete short story with a clear emotional journey.
- **Automated Checks:**
    - **Word Count Check:** Is it close to 1,000 words?
    - **Arc Check:** Does the character's internal state change from the beginning to the end?
- **Human Review Rubric:**
    - **Pacing (1-5):** Does the story move at a good speed?
    - **Character Growth (1-5):** Is the internal change meaningful?
    - **Prose Quality (1-5):** Is the writing evocative?
- **Failure Examples:**
    - The story ends abruptly without a resolution.
    - The character remains exactly the same at the end.
    - The "superpower" is used in a way that makes no sense.
- **Metrics to Graph:**
    - Narrative arc score.
    - Prose quality score.
- **Reasoning Budget Impact:** The reasoning budget allows the model to "map out" the character's emotional journey before writing the prose.

---

# 24. Extended Long-Output Reliability Modules

Long-output reliability is the ultimate test of a model's "attention" and "entropy" management. We need to see if the model can maintain a coherent thread over 20,000 tokens.

**Module A: The "Technical Manual" Stress Test**
- **Realistic Task Design:** Generate a 5,000-word technical manual for a fictional "Quantum Computing Operating System" (QCOS). The manual must include: Installation, Kernel Configuration, Memory Management, Networking, Security, and Troubleshooting.
- **Prompt Shape:** `"Write a 5,000-word technical manual for a fictional Quantum Computing Operating System (QCOS). The manual must be highly detailed and structured with headings, subheadings, and code snippets for configuration. Use your <thought> block to create a detailed table of contents and outline the technical specifications for each section before you begin."`
- **Expected Answer Shape:** A massive, structured document that maintains a consistent technical "voice" and set of specifications.
- **Automated Checks:**
    - **Repetition Check:** Does the model repeat the same "installation" steps in the "troubleshooting" section?
    - **Heading Consistency:** Do the headings follow a logical order?
    - **Length Check:** Does it reach the 5,000-word target?
- **Human Review Rubric:**
    - **Coherence (1-5):** Does the manual make sense as a whole?
    - **Technical Depth (1-5):** Does it sound like a real manual?
    - **Structural Integrity (1-5):** Is the organization logical?
- **Failure Examples:**
    - The model starts repeating the same paragraph every 500 words.
    - The model "forgets" the name of the OS halfway through.
    - The model stops writing after 1,000 words.
- **Metrics to Graph:**
    - Repetition rate (n-gram overlap) vs. token count.
    - Coherence score vs. token count.
- **Reasoning Budget Impact:** The reasoning budget is used to create a "master outline" in the thought block, which the model then refers back to as it generates the long output.

**Module B: The "Novel Chapter" Stress Test**
- **Realistic Task Design:** Write a 3,000-word chapter of a novel. The chapter must involve a complex dialogue between three characters in a high-tension situation (e.g., a heist gone wrong). The model must maintain the distinct voices of all three characters throughout the entire chapter.
- **Prompt Shape:** `"Write a 3,000-word novel chapter. Scene: A heist in a high-security vault has gone wrong. Three characters: 1. Jax (the hot-headed leader), 2. Mira (the calm tech expert), 3. Silas (the nervous getaway driver). The dialogue must be tense and reflect their distinct personalities. Use your <thought> block to plan the dialogue beats and the emotional arc of the scene."`
- **Expected Answer Shape:** A long, dialogue-heavy chapter with distinct character voices.
- **Automated Checks:**
    - **Voice Consistency:** Does Jax always sound hot-headed? Does Mira always sound calm?
    - **Dialogue Flow:** Is the dialogue natural?
    - **Length Check:** Does it reach 3,000 words?
- **Human Review Rubric:**
    - **Voice Distinction (1-5):** Can you tell the characters apart?
    - **Tension (1-5):** Does the scene feel high-stakes?
    - **Narrative Flow (1-5):** Does the scene move forward?
- **Failure Examples:**
    - The characters start sounding the same after 1,000 words.
    - The scene becomes a "talking heads" situation with no action.
    - The model repeats the same "tense" descriptions over and over.
- **Metrics to Graph:**
    - Voice consistency score over the length of the chapter.
    - Tension score vs. token count.
- **Reasoning Budget Impact:** The reasoning budget allows the model to "script" the dialogue beats in the thought block, ensuring the scene moves toward a conclusion.

---

# 25. Deep Dive: Quantization Artifact Analysis

Quantization is the process of reducing the precision of model weights to save memory. While it is necessary for home-lab deployment, it introduces "noise" that can degrade reasoning and creativity.

**Artifact Types to Monitor:**
1.  **Logic Collapse:** The model follows a correct path in the thought block but makes a "dumb" mistake in the final answer (e.g., 2+2=5).
2.  **Repetition Loops:** The model gets stuck in a loop of repeating the same phrase or sentence.
3.  **Vocabulary Shrinkage:** The model uses a more limited and repetitive vocabulary compared to the FP16 version.
4.  **Instruction Drift:** The model ignores parts of the prompt (e.g., "Do not use the word 'the'").

**Evaluation Methodology:**
- **Comparative Testing:** Run the same 100 prompts on Q4_K_M, Q6_K, and Q8_0 versions of the same model.
- **Delta Analysis:** Calculate the "Delta" (difference) in success scores between the quantizations.
- **Quantization Efficiency Score:** `(Success Score of Q4_K_M / Success Score of Q8_0) * 100`. A score > 90% indicates a high-quality quantization.

**Metrics to Graph:**
- Success Score vs. Quantization Bit-rate.
- Repetition Rate vs. Quantization Bit-rate.
- Reasoning Density vs. Quantization Bit-rate.

**Validation Notes:**
- If a model's Q4_K_M version has a significantly lower "Logic Collapse" rate than its Q6_K version, it suggests the model's weights are highly robust to quantization.

---

# 26. Deep Dive: KV Cache Dynamics

The KV (Key-Value) cache is the memory used to store the "history" of the conversation. In a 32GB-class server, managing this cache is the difference between a 32k context window and a crash.

**Key Metrics to Track:**
1.  **KV Cache Growth Rate:** How much VRAM is consumed per token generated.
2.  **TPS Degradation:** The drop in tokens per second as the KV cache fills up.
3.  **Contextual Fidelity:** The accuracy of "Needle in a Haystack" tests as the context window reaches 80%, 90%, and 100% capacity.

**Optimization Strategies to Evaluate:**
- **Flash Attention:** Does it significantly reduce the memory footprint of the KV cache?
- **4-bit KV Cache:** Does quantizing the KV cache itself allow for a larger context window without a significant drop in accuracy?
- **Sliding Window Attention:** Does it maintain coherence while limiting the active context?

**Failure Modes:**
- **OOM (Out of Memory):** The most common failure. The benchmark must log the exact token count at which OOM occurs.
- **Contextual "Blurring":** The model starts mixing up facts from different parts of the context.

**Metrics to Graph:**
- VRAM Usage vs. Token Count.
- TPS vs. Token Count.
- Accuracy vs. Context Fill Percentage.

---

# 27. Deep Dive: Reasoning Budget Optimization

The 8192-token reasoning budget is a "thinking space." However, not all "thinking" is equal. We must distinguish between "productive reasoning" and "hallucinatory rambling."

**Reasoning Quality Metrics:**
1.  **Productive Reasoning:** The model identifies a problem, plans a solution, and verifies the plan.
2.  **Circular Reasoning:** The model repeats the same logical step multiple times without progressing.
3.  **Filler Reasoning:** The model uses many tokens to say very little (e.g., "I am thinking about the problem. I am thinking about the problem.").

**Optimization Methodology:**
- **Budget Scaling:** Test the model with 1k, 4k, and 8k reasoning budgets.
- **Diminishing Returns Point:** Identify the point where increasing the budget no longer improves the success score.
- **Reasoning Density Score:** `(Number of Unique Logical Steps / Total Reasoning Tokens)`.

**Failure Modes:**
- **Reasoning Overflow:** The model uses all 8192 tokens for thinking and produces no final answer.
- **Reasoning-to-Output Disconnect:** The model thinks about one thing but produces an answer about another.

**Metrics to Graph:**
- Success Score vs. Reasoning Budget Size.
- Reasoning Density vs. Reasoning Budget Size.

---

# 28. Deep Dive: MTP Architecture Analysis

Multi-Token Prediction (MTP) allows the model to predict multiple future tokens in a single forward pass. This is a major shift from standard autoregressive generation.

**Evaluation Criteria:**
1.  **Throughput Gain:** The percentage increase in TPS compared to standard generation.
2.  **Coherence Penalty:** Does MTP lead to more frequent logic errors or "hallucination drift" in long outputs?
3.  **Reasoning Integrity:** Does MTP interfere with the linear progression of the 8192-token reasoning budget?

**Methodology:**
- **A/B Testing:** Run the same "Long-Output Reliability" and "Coding Workload" tests with MTP enabled and disabled.
- **Drift Analysis:** Measure the "Perplexity" of the output at 1k, 5k, and 10k tokens to see if MTP causes the model to "drift" away from the prompt's intent.

**Metrics to Graph:**
- TPS (MTP vs. Non-MTP).
- Success Score (MTP vs. Non-MTP).
- Drift Score (MTP vs. Non-MTP).

---

# 29. Deep Dive: SQLite Schema and Querying

The SQLite database is the "brain" of the benchmark. It must be structured to allow for complex queries and automated reporting.

**Advanced Query Examples:**
- **Model Comparison:** `SELECT model_name, AVG(success_score) FROM Benchmarks GROUP BY model_name ORDER BY AVG(success_score) DESC;`
- **Failure Analysis:** `SELECT model_name, failure_mode, COUNT(*) FROM Benchmarks GROUP BY model_name, failure_mode;`
- **Performance Trends:** `SELECT model_name, AVG(tps_avg) FROM Benchmarks WHERE workload_type = 'Long-Output' GROUP BY model_name;`

**Data Integrity Measures:**
- **Unique Constraints:** Ensure no duplicate `benchmark_id` entries.
- **Foreign Key Enforcement:** Ensure every `benchmark_id` is linked to a valid `model_id`.
- **Automated Backups:** The system should automatically back up the `.sqlite` file every hour.

**Artifact Layout Expansion:**
- `/benchmarks/metadata/`: Contains JSON files describing the hardware state for every run.
- `/benchmarks/telemetry/`: Contains the raw CSV files from the AI Flight Recorder.
- `/benchmarks/reports/`: Contains the final PDF and HTML reports.

---

# 30. Deep Dive: Reporting and Visualization

The reporting plane must turn raw data into a narrative. We will use a combination of Python (Matplotlib/Seaborn) and a web-based dashboard (Streamlit).

**Dashboard Components:**
1.  **Model Comparison Tool:** A dropdown to select two models and see their scores side-by-side across all 6 workload types.
2.  **Reasoning Explorer:** A view that shows the "Thought Block" for any given benchmark, with a "Reasoning Density" score highlighted.
3.  **Failure Gallery:** A searchable list of all "Failures," categorized by type (e.g., "Logic Collapse," "OOM," "Repetition").
4.  **Live Telemetry Feed:** A real-time view of the GPU temperature, VRAM usage, and TPS during a benchmark run.

**Automated Report Generation:**
- A script will run every night to generate a "Daily Summary" PDF, highlighting the best-performing model of the day and any significant failures.

---

# 31. Deep Dive: Reproducibility and Environment Isolation

To ensure the benchmark is scientifically valid, we must eliminate all external variables.

**Environment Isolation:**
- **Dockerization:** Run the `llama.cpp` server inside a Docker container with a fixed set of environment variables and library versions.
- **Hardware State:** Use a script to clear the GPU cache and reset the GPU state before every benchmark run.
- **Network Isolation:** Run the benchmark in an offline environment to ensure no external API calls or network-related latency spikes occur.

**Seed Management:**
- Every prompt will be accompanied by a fixed random seed. This ensures that the "stochastic" nature of the LLM is controlled, allowing for direct comparison between models.

**Version Control:**
- Every benchmark run will be logged with the exact Git commit hash of the `llama.cpp` repository and the specific version of the GGUF model used.

---

# 32. Appendix: Synthetic Data Repository

This appendix contains the actual synthetic prompts and data used in the benchmark.

**Sample Data - Coding (Module A):**
- `prompt_001.txt`: "Act as a Principal Systems Architect. Design a microservices architecture for a real-time auction platform..."
- `prompt_002.txt`: "You are a Senior Software Engineer specializing in refactoring. Below is a legacy Python script..."

**Sample Data - Agentic (Module A):**
- `prompt_003.txt`: "You are a travel agent. Use the provided tools to plan a 10-day trip to Japan..."
- `prompt_004.txt`: "You are a Senior Product Manager. A client wants a mobile app for tracking indoor plants..."

**Sample Data - RAG (Module A):**
- `context_001.txt`: [10,000 words of fictional spaceship technical specs]
- `context_002.txt`: [3 documents regarding remote work policies]

**Sample Data - Creative (Module A):**
- `prompt_005.txt`: "Write a comprehensive World Bible for a high-fantasy setting..."
- `prompt_006.txt`: "Write a 1,000-word short story about a character with the minor superpower of making liquids lukewarm..."

---

# 33. Appendix: Troubleshooting and Error Codes

This section provides a guide for interpreting the "AI Flight Recorder" logs and common llama.cpp errors.

**Common Error Codes:**
- **ERR_OOM_01:** Out of Memory. The KV cache exceeded the 32GB VRAM limit. *Solution: Reduce context window or use a smaller quantization.*
- **ERR_REP_02:** Repetition Loop Detected. The model repeated the same 5-token sequence more than 3 times. *Solution: Increase repeat penalty or check for "logic collapse."*
- **ERR_LOG_03:** Logic Collapse. The model's thought block was logical, but the final answer was nonsensical. *Solution: Increase reasoning budget or try a higher bit-rate quantization.*
- **ERR_TIM_04:** Timeout. The model took longer than 300 seconds to generate the first token. *Solution: Check GPU temperature and thread count.*

**Troubleshooting Steps:**
1.  **Check GPU Temperature:** If >85°C, the GPU may be throttling.
2.  **Verify VRAM:** Ensure no other processes are using the R9700's memory.
3.  **Check Thread Count:** Ensure `-t` matches the number of physical cores.
4.  **Verify GGUF Integrity:** Re-download the model and check the SHA256 hash.

---

# 34. Appendix: Hardware Upgrade Path

For home-lab operators looking to scale their benchmark capabilities, this section provides a roadmap for hardware upgrades.

**Tier 1: Memory Expansion (Current)**
- **Goal:** Increase VRAM to 64GB.
- **Action:** Add a second R9700-class GPU or upgrade to a 64GB-class card.
- **Impact:** Allows for 70B+ models at Q6_K quantization with a 32k context window.

**Tier 2: Bandwidth Optimization**
- **Goal:** Increase memory bandwidth.
- **Action:** Move to a PCIe 5.0-enabled motherboard and GPU.
- **Impact:** Significantly higher TPS for large models and faster KV cache updates.

**Tier 3: Multi-GPU Orchestration**
- **Goal:** Distributed inference.
- **Action:** Implement a multi-GPU setup using `llama.cpp`'s distributed inference capabilities.
- **Impact:** Enables the use of 100B+ parameter models and massive context windows (128k+).

**Final Benchmark Summary:**
This manual provides a complete, end-to-end framework for evaluating local LLMs. By following these 34 sections, the home-lab operator can transform a simple inference server into a high-fidelity research laboratory, capable of producing professional-grade benchmarks that rival public leaderboards in depth and reliability.

# 35. GGUF Quantization Nuances (K-Quants)

While the benchmark primarily utilizes Q4_K_M, Q6_K, and Q8_0, a deeper understanding of the GGUF "K-Quant" (K-Means Quantization) methodology is required to interpret the results accurately. K-Quants are designed to minimize the "Perplexity Gap"—the difference in model intelligence between the original FP16 weights and the quantized version.

**Quantization Comparison Matrix:**
- **Q4_K_M:** The "Gold Standard." It provides a significant reduction in memory footprint with a negligible loss in logic. It is the primary target for the 32GB-class R9700 server.
- **Q5_K_M:** The "High-Fidelity" middle ground. It offers a slight improvement in nuance and creative vocabulary over Q4_K_M but requires ~25% more VRAM.
- **Q6_K:** The "Near-Lossless" choice. For complex reasoning tasks (Agentic/Coding), Q6_K is often the baseline for "correct" behavior.
- **Q8_0:** The "Reference" point. Used to establish the upper bound of what the model can do on the hardware before quantization artifacts interfere.

**The Perplexity Gap Analysis:**
We will measure the "Perplexity Gap" by comparing the model's performance on a set of 1,000 "Gold Standard" prompts across all three quantization levels. 
- **Metric:** `Quantization_Loss = (Score_FP16 - Score_Quantized) / Score_FP16`.
- **Target:** A `Quantization_Loss` of < 5% for Q4_K_M is considered a "High-Quality Quantization."

**Failure Modes in Quantization:**
- **Weight Collapse:** When a model's weights are quantized too aggressively, it may lose the ability to follow negative constraints (e.g., "Do not use the word 'the'").
- **Vocabulary Drift:** The model may start favoring common words over specific technical terms due to the loss of precision in the output layer.
- **Logic Inversion:** In rare cases, a quantized model might provide a logically sound thought block but a completely incorrect final answer due to a "bit-flip" in the final projection layer.

**Validation Notes:**
- Always verify the "K-Quant" version of the GGUF. Older "Legacy" quants (like Q4_0) should be avoided as they exhibit significantly higher logic collapse.
- Use the AI Flight Recorder to track "Logit Entropy" across different quants; higher entropy in the final answer often correlates with quantization noise.

---

# 36. Speculative Decoding and Draft Models

To maximize the utility of the R9700 server, we will evaluate Speculative Decoding. This technique uses a smaller, faster "Draft Model" to predict the next few tokens, which are then verified in a single forward pass by the larger "Target Model."

**Draft Model Selection Strategy:**
- **Target Model:** Llama-3-70B (Q4_K_M).
- **Draft Model:** Llama-3-8B (Q8_0) or a specialized 1B-3B "Tiny" model.
- **Mechanism:** The Draft Model generates a sequence of $N$ tokens. The Target Model checks these tokens. If they are correct, they are accepted instantly. If incorrect, the Target Model corrects the first error and continues.

**Metrics to Graph:**
- **Acceptance Rate:** The percentage of tokens accepted by the Target Model. A rate > 60% indicates a highly compatible Draft Model.
- **Speedup Factor:** The ratio of TPS with Speculative Decoding vs. standard inference.
- **Accuracy Parity:** A comparison of the "Success Score" between standard and speculative modes to ensure no "hallucination drift" is introduced.

**Failure Modes:**
- **Draft Model Divergence:** If the Draft Model is too "creative," the Target Model will constantly reject its tokens, leading to a speedup factor < 1.0 (slower than standard inference).
- **Context Window Mismatch:** If the Draft Model has a smaller context window than the Target Model, speculative decoding will fail as the context fills.

**Checklist for Speculative Decoding:**
- [ ] Verify that the Draft Model and Target Model share the same vocabulary.
- [ ] Test with $N=3$ and $N=5$ to find the optimal lookahead.
- [ ] Monitor "Acceptance Rate" in the AI Flight Recorder.
- [ ] Ensure the Draft Model is small enough to fit in VRAM alongside the Target Model.

---

# 37. Statistical Significance in Benchmarking

A single run of a prompt is not a benchmark; it is an anecdote. To ensure the "Master Field Manual" produces scientifically valid data, we must apply statistical rigor to every workload.

**Sample Size Requirements:**
- **Coding/Agentic:** Minimum $n=50$ unique prompts per model.
- **Chatbot/Creative:** Minimum $n=100$ unique prompts per model.
- **Long-Output:** Minimum $n=10$ unique prompts per model.

**Statistical Metrics:**
- **Mean Success Score ($\mu$):** The average score across all $n$ runs.
- **Standard Deviation ($\sigma$):** The measure of model consistency. A high $\sigma$ indicates a "brittle" model that succeeds on some prompts but fails on others.
- **Confidence Interval (95%):** The range within which we are 95% certain the true mean score lies.
- **P-Value:** Used to determine if the difference between two models is "statistically significant" ($p < 0.05$).

**Validation Notes:**
- If a model has a high $\mu$ but a very high $\sigma$, it should be flagged as "Unreliable" in the final leaderboard.
- Use the "Student's T-Test" to compare the Coding Workload scores of two competing models.

**Failure Modes:**
- **Outlier Bias:** A single "miracle" success on a very hard prompt can skew the mean. Use "Trimmed Means" (removing the top and bottom 5% of results) to mitigate this.
- **Prompt Sensitivity:** Small changes in prompt wording can lead to different results. Use "Prompt Averaging" (running the same prompt with three slightly different phrasings).

---

# 38. Semantic Drift and Coherence Analysis

In long-output reliability tests, models often suffer from "Semantic Drift"—the tendency for the model to slowly move away from the original topic as it generates more tokens.

**Measuring Semantic Drift:**
We will use a "Drift Score" based on Cosine Similarity.
1.  **Embed the Prompt:** Generate a vector for the initial prompt.
2.  **Embed Output Segments:** Every 500 tokens, generate a vector for the last 500 tokens of the output.
3.  **Calculate Similarity:** Measure the cosine similarity between the prompt vector and the segment vector.

**Metrics to Graph:**
- **Drift Curve:** A line graph showing Cosine Similarity vs. Token Count.
- **Topic Wandering Point:** The token count at which similarity drops below 0.7.

**Failure Examples:**
- **The "Tangent" Failure:** The model starts writing a technical manual on "Quantum OS" but begins discussing "The History of Agriculture" after 3,000 tokens.
- **The "Loop" Failure:** The model repeats the same 100 tokens indefinitely (detected via n-gram overlap).

**Human Review Rubric (Coherence):**
- **Thematic Consistency (1-5):** Does the model stay on topic?
- **Logical Progression (1-5):** Do the paragraphs follow a logical sequence?
- **Detail Retention (1-5):** Does the model remember facts from the beginning of the output?

---

# 39. Agentic State Machine Mapping

To evaluate agentic workloads, we must move beyond "Success/Failure" and map the model's internal "Decision Tree."

**State Machine Mapping:**
For every agentic task, the AI Flight Recorder will extract the "Thought" block and map it to a state machine:
- **Nodes:** The distinct actions or decisions made by the model (e.g., "Search for flights," "Calculate budget," "Select hotel").
- **Edges:** The logical transitions between these actions.
- **Sink States:** States where the model gets stuck in a loop (e.g., "Search for flights" $\rightarrow$ "Search for flights").

**Metrics to Graph:**
- **Path Length:** The number of steps taken to reach the goal.
- **Branching Factor:** How many different options the model considered in its thought block.
- **Sink State Frequency:** How often the model entered a loop.

**Failure Modes:**
- **Infinite Loop:** The model calls the same tool with the same parameters repeatedly.
- **Hallucinated Tool:** The model attempts to call a tool that was not provided in the prompt.
- **Logic Dead-end:** The model makes a decision that makes the goal impossible to achieve (e.g., booking a hotel for a date that has already passed).

**Validation Notes:**
- A "Perfect Agent" should have a linear path with minimal branching and zero sink states.
- Use the "Reasoning Density" score to see if the model spent enough "thinking" time on complex branching points.

---

# 40. RAG Chunking and Contextual Overlap

The performance of RAG workloads is heavily dependent on how the source text is "chunked" before being fed into the model. This section evaluates the model's ability to handle different chunking strategies.

**Chunking Strategies to Test:**
1.  **Fixed-Size Chunking:** 512 tokens per chunk, no overlap.
2.  **Recursive Character Splitting:** Splitting by paragraphs, then sentences, then words, with a 10% overlap.
3.  **Semantic Chunking:** Splitting based on changes in the "meaning" of the text (using an embedding model).

**Metrics to Graph:**
- **Retrieval Accuracy vs. Chunk Size:** Does the model perform better with 256, 512, or 1024-token chunks?
- **Overlap Impact:** Does a 10% overlap significantly reduce "Contextual Blindness" (where a fact is split between two chunks)?

**Failure Modes:**
- **Contextual Blindness:** The model cannot answer a question because the relevant information was split across two chunks, and the model only saw one.
- **Noise Overload:** The chunk is too large, and the model gets distracted by irrelevant information within the same chunk.

**Checklist for RAG Optimization:**
- [ ] Test at least three different chunk sizes.
- [ ] Evaluate the "Top-K" retrieval (e.g., K=3, K=5, K=10).
- [ ] Measure the "Faithfulness" score for each chunking strategy.

---

# 41. Synthetic Data Generation Pipelines

To maintain a high-quality benchmark, we need a constant stream of new, high-quality synthetic prompts. We will use a "Teacher-Student" methodology.

**Teacher-Student Methodology:**
- **Teacher Model:** A high-capability model (e.g., Llama-3-70B-Instruct or a larger frontier model).
- **Student Model:** The model being benchmarked.
- **Process:** The Teacher Model generates 1,000 diverse prompts and "Gold Standard" answers for each workload. The Student Model is then tested against these.

**Diversity Metrics:**
- **Self-BLEU:** Measures the similarity between the generated prompts. A lower Self-BLEU score indicates higher diversity.
- **Topic Coverage:** A manual review to ensure the prompts cover a wide range of industries (e.g., Medicine, Law, Engineering, Fiction).

**Failure Modes:**
- **Teacher Bias:** If the Teacher Model has a specific "style," the Student Model might be unfairly penalized for not mimicking that style.
- **Prompt Collapse:** The Teacher Model generates 1,000 prompts that are all essentially the same.

**Validation Notes:**
- Periodically "refresh" the synthetic dataset to ensure the benchmark remains relevant as new models are released.
- Use the "Reasoning Budget" to have the Teacher Model explain *why* a certain answer is correct, which can be used to grade the Student Model's thought block.

---

# 42. Model Drift and Lifecycle Management

Models are not static. As we run benchmarks over months, we must account for "Model Drift"—the degradation of performance due to changes in the inference environment or the "aging" of the benchmark data.

**Monitoring Model Health:**
- **Baseline Comparison:** Every 30 days, re-run the "Gold Standard" 1,000-prompt set.
- **Drift Detection:** If the mean success score drops by > 3%, flag the model for "Drift Analysis."

**Causes of Drift:**
- **Software Updates:** Changes in `llama.cpp`'s kernels or Flash Attention implementation.
- **Hardware Degradation:** Thermal throttling or VRAM errors.
- **Prompt Obsolescence:** The benchmark prompts no longer reflect current real-world use cases.

**Checklist for Lifecycle Management:**
- [ ] Establish a "Gold Standard" baseline for every model.
- [ ] Automate the 30-day "Drift Check."
- [ ] Log the `llama.cpp` version and GPU driver version for every run.
- [ ] Update the synthetic dataset quarterly.

---

# 43. Privacy-Preserving Inference (Local Redaction)

In a home-lab environment, the primary reason for local inference is privacy. This section evaluates the model's ability to act as a "Privacy Guard."

**Local Redaction Pipeline:**
1.  **Input Scrubbing:** Before the prompt reaches the model, a local script (using Regex or a small NER model) identifies PII.
2.  **Model Redaction:** The model is asked to identify and redact PII in a large block of text.
3.  **Output Verification:** A final script checks the model's output for any leaked PII.

**Metrics to Graph:**
- **PII Detection Rate:** Percentage of PII correctly identified and redacted.
- **False Positive Rate:** Percentage of non-sensitive data incorrectly redacted.

**Failure Modes:**
- **Partial Redaction:** The model redacts the name but leaves the phone number.
- **Contextual Leakage:** The model redacts the name but provides enough context that the person can still be identified.
- **Redaction Hallucination:** The model redacts entire paragraphs of useful information because it "thinks" they contain PII.

**Validation Notes:**
- Use synthetic data that mimics real-world PII (fake names, fake credit cards, fake addresses).
- The "Redaction Accuracy" score should be a primary metric for any model intended for "Enterprise-Grade" local use.

---

# 44. Advanced Telemetry: Token Probability Analysis

To truly understand *why* a model fails, we must look at the "Logits"—the raw probability distribution for every token.

**Key Telemetry Metrics:**
- **Logit Entropy:** A measure of how "uncertain" the model is. High entropy in the thought block suggests the model is struggling with the logic.
- **Confidence Score:** The probability of the most likely token. If the confidence score drops significantly during a long output, it is a precursor to "Logic Collapse."
- **Top-K Distribution:** How many of the top 10 tokens have similar probabilities. A "flat" distribution indicates the model is confused.

**Analysis Methodology:**
- **Failure Post-Mortem:** For every "Failure" in the benchmark, analyze the Logit Entropy of the 50 tokens preceding the error.
- **Confidence Mapping:** Create a heatmap of confidence scores across the 20,000-token long-output.

**Failure Modes:**
- **Overconfidence:** The model provides a wrong answer with 99% confidence (indicates a "hard" hallucination).
- **Indecision:** The model provides a correct answer but with very low confidence (indicates a "weak" reasoning path).

**Validation Notes:**
- Use the AI Flight Recorder to export the probability distribution for every token in the "Coding" and "Agentic" workloads.
- High confidence + Low accuracy = "Hallucination."
- Low confidence + High accuracy = "Weak Reasoning."

---

# 45. The "Human-in-the-Loop" (HITL) Validation Framework

Automated checks are necessary but insufficient. A "Human-in-the-Loop" framework provides the final layer of validation for the benchmark.

**Elo Rating System:**
We will use an Elo-based ranking system (similar to Chess) to compare models.
- **Matchup:** Two models are given the same prompt.
- **Judge:** A human reviewer (or a "Teacher" model) decides which output is better.
- **Update:** The Elo scores of both models are updated based on the result.

**Human Review Rubric (Detailed):**
- **Instruction Adherence (1-5):** Did it do exactly what was asked?
- **Nuance and Style (1-5):** Did it capture the requested "vibe"?
- **Logical Integrity (1-5):** Is the reasoning sound?
- **Conciseness (1-5):** Did it avoid unnecessary "fluff"?

**Checklist for HITL:**
- [ ] Select 100 "High-Value" prompts for human review.
- [ ] Ensure reviewers are "blind" to the model names.
- [ ] Use a consistent 1-5 scoring scale.
- [ ] Calculate the final Elo ratings for the leaderboard.

**Failure Modes:**
- **Reviewer Fatigue:** Consistency drops as reviewers get tired. (Solution: Use a "Gold Standard" set of prompts to check reviewer consistency).
- **Subjectivity:** Different reviewers may have different preferences. (Solution: Use a "Consensus" model where 3 reviewers must agree).

---

# 46. Benchmark Automation Scripting (Python/Bash)

To ensure reproducibility, the entire benchmark must be executable with a single command.

**Automation Architecture:**
- **Orchestrator (Python):** Reads the SQLite database, selects the next prompt, calls the `llama.cpp` server, and parses the output.
- **Telemetry Hook:** A sidecar script that monitors GPU stats and feeds them into the Flight Recorder.
- **Data Pipeline:** Automatically inserts results into the SQLite database and updates the "Success Score" for each model.

**Automation Checklist:**
- [ ] Script to automatically download and verify GGUF hashes.
- [ ] Script to clear VRAM and reset the GPU state between runs.
- [ ] Script to generate the final PDF report from the SQLite data.
- [ ] "Fail-Safe" logic: If a run crashes, the script should log the error and move to the next prompt.

**Failure Modes:**
- **Zombie Processes:** `llama.cpp` processes that don't close properly, hogging VRAM.
- **Database Locks:** Multiple scripts trying to write to the SQLite file simultaneously.

---

# 47. Comparative Analysis of Reasoning Budgets

We will perform a "Scaling Law" analysis on the 8192-token reasoning budget to find the "Sweet Spot" for different workloads.

**Reasoning Budget Scaling:**
- **Test Points:** 512, 1024, 2048, 4096, 8192 tokens.
- **Analysis:** Plot "Success Score" vs. "Reasoning Budget."

**Workload-Specific Budgets:**
- **Coding:** Often benefits from a larger budget (4k+) to plan complex logic.
- **Chatbot:** Benefits from a smaller budget (512-1k) to ensure low latency.
- **Agentic:** Requires the maximum budget (8k) to map out multi-step plans.

**Metrics to Graph:**
- **Efficiency Score:** `Success Score / Reasoning Tokens`.
- **Diminishing Returns Point:** The budget size where the Success Score plateaus.

**Validation Notes:**
- A model that requires 8k tokens for a simple coding task is "Reasoning Inefficient."
- A model that fails even with 8k tokens is "Reasoning Limited."

---

# 48. Hardware Thermal and Power Profiling

For a home-lab, "Performance per Watt" and "Thermal Stability" are critical metrics for long-running benchmarks.

**Power Metrics:**
- **Watts per Token (WpT):** Total power consumption divided by tokens per second.
- **Thermal Delta:** The increase in GPU temperature from the start to the end of a 20,000-token generation.

**Thermal Throttling Detection:**
- The AI Flight Recorder will flag any run where the GPU temperature exceeds 85°C.
- **Impact Analysis:** Compare the TPS of "Cool" runs vs. "Throttled" runs to quantify the performance loss.

**Checklist for Hardware Health:**
- [ ] Monitor fan speeds during long-output tests.
- [ ] Check for "VRAM Overheating" (if applicable to the R9700).
- [ ] Log "Power Spikes" that might cause system instability.

---

# 49. Final Master Summary and Roadmap

This manual concludes the benchmark campaign framework. The goal is to move from raw inference to a structured, scientific understanding of local LLM capabilities.

**The Master Roadmap:**
1.  **Phase 1: Baseline Establishment.** Run the "Gold Standard" 1,000-prompt set on all target models at Q4_K_M.
2.  **Phase 2: Workload Deep-Dives.** Execute the Coding, Agentic, RAG, and Long-Output modules.
3.  **Phase 3: Optimization.** Test Speculative Decoding and different Reasoning Budgets.
4.  **Phase 4: Final Leaderboard.** Compile the Elo ratings and generate the final report.

**Final Recommendation Logic:**
- **For Production Agents:** Choose the model with the highest "Agentic Success Score" and lowest "Sink State Frequency."
- **For Creative Writing:** Choose the model with the highest "Elo Rating" in the Creative Workload.
- **For Coding:** Choose the model with the highest "Go Compilation Pass Rate."

**Closing Note:**
The "AI Flight Recorder" is the most important tool in this manual. By capturing the "flight path" of every token, we ensure that our benchmarks are not just a collection of scores, but a deep, transparent look into the "mind" of the model. This framework allows the home-lab operator to build a truly professional-grade AI laboratory.