# 1. Purpose And Scope

The primary objective of this benchmark campaign is to establish a rigorous, reproducible, and scientifically grounded evaluation framework for open-weight Large Language Models (LLMs) deployed in a private, local environment. Specifically, we are targeting the performance of GGUF-quantized models on a 32GB-class R9700 llama.cpp server. Unlike public leaderboards (e.g., LMSYS Chatbot Arena), which are often influenced by "vibes" and subjective human preference in a high-latency web environment, this benchmark focuses on **local inference efficiency, reasoning integrity, and long-context reliability.**

The scope of this manual covers the entire lifecycle of a benchmark run: from hardware initialization and runtime parameter tuning to the collection of telemetry via the AI Flight Recorder. We aim to answer three critical questions:
1.  **Quantization Integrity:** How much "intelligence" is lost when moving from FP16 to Q4_K_M or IQ4_XS formats on this specific hardware?
2.  **Reasoning Depth:** Can the model maintain a coherent internal logic chain when constrained by a specific reasoning budget (8192 tokens)?
3.  **Operational Reliability:** Does the model maintain stylistic and factual consistency over extremely long outputs (approaching 20,000 tokens), or does it succumb to "repetition loops" or "contextual drift"?

This manual serves as the "Source of Truth" for all benchmark engineers. Every test must be executed according to the protocols defined herein to ensure that differences in results are attributable to model architecture and weights, rather than variance in system prompts, temperature settings, or hardware fluctuations.

**Scope Exclusions:**
- This benchmark does not evaluate models via API (e.g., OpenAI, Anthropic).
- This benchmark does not evaluate models on cloud-based H100 clusters.
- This benchmark does not include "jailbreak" testing; the focus is on functional utility and reasoning.

**Key Objectives:**
- Establish a baseline for "Tokens Per Second" (TPS) across different quantization levels.
- Quantify "Reasoning Density"—the ratio of useful logic steps to total reasoning tokens.
- Identify the "Context Collapse Point"—the point at which the model begins to ignore earlier instructions in a long-form output.

# 2. Hardware Profile

The benchmark is conducted on a specialized 32GB-class R9700 llama.cpp server. Understanding the hardware constraints is vital for interpreting the "Tokens Per Second" (TPS) and "Time to First Token" (TTFT) metrics.

**Core Specifications:**
- **Compute Unit:** R9700 Series (High-bandwidth memory architecture).
- **Memory Capacity:** 32GB Unified Memory / VRAM.
- **Interconnect:** High-speed PCIe Gen4/5 lanes.
- **Storage:** NVMe Gen4 SSD (for rapid model loading and SQLite artifact logging).

**Hardware Constraints & Implications:**
1.  **Memory Ceiling:** With 32GB, we are limited in the size of the models we can run at high precision. A 70B model at Q4_K_M will likely exceed the memory limit when accounting for the KV cache. Therefore, the primary focus will be on 7B, 14B, and 30B-35B parameter models.
2.  **Memory Bandwidth:** The R9700’s performance is heavily dependent on memory bandwidth. We must monitor "Memory Bandwidth Utilization" during inference. If the model is "memory-bound," increasing the number of threads will not improve TPS.
3.  **Thermal Throttling:** During long-output tests (20,000+ tokens), the server may experience thermal buildup. The AI Flight Recorder must log GPU/CPU temperatures to ensure that a drop in TPS is not due to hardware throttling.

**Hardware Validation Checklist:**
- [ ] Verify `nvidia-smi` or equivalent shows 32GB available.
- [ ] Confirm PCIe lane width is at least x16.
- [ ] Ensure NVMe drive has at least 100GB of free space for model weights and SQLite logs.
- [ ] Benchmark "Base TPS" using a simple "Repeat the word 'hello' 100 times" prompt to establish a hardware floor.

# 3. llama.cpp Runtime Profile

To ensure reproducibility, the `llama.cpp` runtime must be configured identically for every model in the campaign. Variations in `n_batch`, `n_threads`, and `n_ctx` will invalidate the comparison between models.

**Standardized Runtime Parameters:**
- **Quantization Types:** We will test three primary formats:
    - `Q4_K_M`: The standard balance of size and intelligence.
    - `Q6_K`: High-fidelity testing for smaller models.
    - `IQ4_XS`: Testing the limits of extreme quantization.
- **Context Window (`n_ctx`):** Set to 32,768 for all tests to ensure the model has sufficient "room" for long-form outputs.
- **Batch Size (`n_batch`):** 512 (optimized for the R9700 architecture).
- **Threads (`n_threads`):** Set to the number of physical cores (e.g., 16 or 32, depending on the specific R9700 SKU).
- **Temperature:** 0.7 (Standard for creative/reasoning balance).
- **Top_P:** 0.9.
- **Repeat Penalty:** 1.1.

**AI Flight Recorder Integration:**
The Flight Recorder must hook into the `llama.cpp` output stream to capture:
- **TTFT (Time to First Token):** Measured in milliseconds.
- **TPS (Tokens Per Second):** Calculated as a rolling average over 50 tokens.
- **KV Cache Usage:** Monitoring the memory footprint of the context.
- **Prompt Processing Time:** Time taken to ingest the system prompt and user input.

**Failure Modes in Runtime:**
- **Context Overflow:** If the model exceeds `n_ctx`, it will begin to "forget" the start of the prompt. The Flight Recorder must flag this.
- **Thread Contention:** If `n_threads` is set higher than physical cores, performance may degrade due to context switching.
- **Quantization Artifacts:** If a model produces gibberish at Q4_K_M, it may indicate a "broken" GGUF conversion.

# 4. Reasoning Budget Methodology

The "Reasoning Budget" refers to the number of tokens the model is permitted to use for internal "thinking" before producing the final answer. In this campaign, we set a hard limit of 8192 tokens for the thinking process.

**The "Thought" Block Protocol:**
Models should be instructed to use a `<thought>` tag to encapsulate their reasoning.
- **Example:** `<thought> The user wants a Python script for a web scraper. I need to consider the libraries (BeautifulSoup, Requests), the error handling, and the specific URL structure. I will first outline the class structure... </thought> [Final Answer]`

**Evaluation of Reasoning:**
1.  **Reasoning Density:** We calculate this as:
    $\text{Density} = \frac{\text{Number of Logical Steps}}{\text{Total Reasoning Tokens}}$
    A high density indicates a concise, efficient thinker. A low density indicates "rambling" or "looping" within the thought block.
2.  **Reasoning-to-Output Ratio:** We track how much "thinking" is required to produce a specific output. For complex coding tasks, a 1:1 ratio is expected. For simple chat, a 1:5 ratio is expected.
3.  **Budget Exhaustion:** If a model hits the 8192-token limit before finishing its thought process, it is marked as "Reasoning Failed."

**Validation Notes:**
- Ensure the model does not "leak" thought process tokens into the final answer.
- If the model refuses to use the `<thought>` tag, the run is invalidated.
- The AI Flight Recorder must count tokens specifically within the `<thought>` tags to provide a "Reasoning Budget Utilization" metric.

# 5. MTP Versus Non-MTP Methodology

Multi-Token Prediction (MTP) is an architectural advancement where the model predicts multiple future tokens simultaneously. We need to evaluate how MTP-enabled models (like some newer Llama-3 variants) compare to standard autoregressive models on the R9700.

**Testing Protocol:**
- **Group A (Non-MTP):** Standard autoregressive models (e.g., Llama-3-8B-Instruct).
- **Group B (MTP):** Models explicitly trained with MTP heads.

**Metrics for Comparison:**
1.  **Inference Speedup:** Does MTP significantly increase TPS on the R9700?
2.  **Coherence at Scale:** Does MTP lead to "hallucinated" sequences in long-form outputs because the model is "jumping ahead" too far?
3.  **Reasoning Accuracy:** Does MTP improve the logic within the `<thought>` block, or does it lead to "skipping" logical steps?

**Synthetic Example for Comparison:**
*Prompt:* "Explain the quantum Zeno effect and provide a Python simulation."
*Observation:* Does the MTP model provide a more cohesive transition between the explanation and the code, or does it produce a "jumpy" transition?

**Failure Modes:**
- **MTP Drift:** The model predicts a sequence of tokens that are grammatically correct but logically disconnected from the previous sentence.
- **Speed Degradation:** In some cases, the overhead of MTP calculation might actually slow down TPS on specific hardware.

# 6. Context Fit Methodology

This section evaluates how well the model utilizes the 32,768-token context window. We use a "Needle in a Haystack" (NiH) approach combined with "Contextual Drift" analysis.

**Methodology:**
1.  **Needle in a Haystack:** Insert a random fact (e.g., "The secret code is Blue-Elephant-99") at various positions: 10%, 25%, 50%, 75%, and 90% into a 10,000-token context.
2.  **Contextual Drift:** Provide a 5,000-token story and ask the model to continue it. We measure how many tokens into the continuation the model begins to contradict the established facts of the story.

**Metrics:**
- **Retrieval Accuracy:** % of successful retrievals across different positions.
- **Drift Point:** The token count at which the model first contradicts a "ground truth" fact from the context.
- **KV Cache Efficiency:** Tracking the memory growth of the KV cache as context fills.

**Validation Notes:**
- Use a "Neutral" prompt to ensure the model isn't biased toward the end of the context.
- Ensure the "Needle" is not a common phrase that might be predicted by the model's weights regardless of context.

# 7. Coding Workloads

Coding tasks evaluate the model's ability to generate syntactically correct, logically sound, and efficient code.

**Task Design:**
1.  **Refactoring:** Provide a "messy" Python script and ask for a "clean, PEP8-compliant, object-oriented" version.
2.  **Bug Fixing:** Provide a script with a subtle logical error (e.g., an off-by-one error in a recursive function).
3.  **Library Integration:** Ask the model to write a script using a specific, less-common library (e.g., `Polars` instead of `Pandas`).

**Prompt Shape:**
`System: You are an expert software engineer. Use <thought> tags to plan the architecture before writing code.`
`User: [Insert Code Snippet] Fix the bug where the list index exceeds the bounds during the recursive step.`

**Expected Answer Shape:**
- `<thought>` block outlining the bug and the fix.
- Corrected code block.
- Brief explanation of the fix.

**Automated Checks:**
- **Syntax Check:** Run the generated code through a linter (e.g., `flake8` or `pylint`).
- **Execution Check:** Run the code against a set of unit tests. If it fails, the model receives a 0 score.

**Human Review Rubric:**
- **Readability:** 1-5 (Is the code "Pythonic"?)
- **Efficiency:** 1-5 (Does it use unnecessary loops?)
- **Comment Quality:** 1-5 (Are comments helpful or redundant?)

**Failure Examples:**
- **Hallucinated Libraries:** The model imports a library that doesn't exist (e.g., `import fast_json_parser`).
- **Logic Loop:** The model fixes the bug but introduces a new, different bug in the same function.

**Metrics to Graph:**
- **Pass@1 Rate:** Percentage of runs that pass the unit tests on the first try.
- **Code Length vs. Correctness:** Does the model get "lazier" and write shorter, incorrect code as the prompt gets longer?

**Reasoning Budget Impact:**
A higher reasoning budget should correlate with a higher Pass@1 rate for complex bug fixes, as the model has more "space" to trace the recursion.

# 8. Agentic Workloads

Agentic workloads test the model's ability to follow multi-step instructions, use tools (simulated), and output structured data (JSON).

**Task Design:**
1.  **Tool Selection:** Provide a list of 5 tools (e.g., `get_weather`, `send_email`, `query_db`, `calculate_tax`, `search_web`). Ask the model to choose the correct tool for a complex user request.
2.  **JSON Extraction:** Provide a messy paragraph of text and ask the model to extract entities into a strictly formatted JSON schema.
3.  **Multi-Step Planning:** Ask the model to plan a 3-day itinerary for a trip to Tokyo, including budget constraints and specific interests (e.g., "Anime and Sushi").

**Prompt Shape:**
`System: You are an AI Agent. You must output your plan in a <thought> block and your final action in a JSON block.`
`User: The user wants to book a flight to Tokyo. They have a budget of $1500 and want to stay in Shinjuku. What are the steps?`

**Expected Answer Shape:**
- `<thought>` block detailing the sequence of actions.
- JSON block: `{"steps": [...], "estimated_cost": ..., "priority": ...}`.

**Automated Checks:**
- **JSON Validity:** Use `json.loads()` to ensure the output is valid JSON.
- **Schema Adherence:** Check if all required keys (e.g., `steps`, `estimated_cost`) are present.
- **Tool Correctness:** Verify that the chosen tool matches the user's intent.

**Human Review Rubric:**
- **Plan Coherence:** 1-5 (Do the steps follow a logical order?)
- **Constraint Adherence:** 1-5 (Did it respect the $1500 budget?)
- **JSON Cleanliness:** 1-5 (Is there any "chatty" text outside the JSON block?)

**Failure Examples:**
- **JSON Breakage:** The model includes a conversational "Here is your JSON:" before the actual JSON block, breaking the parser.
- **Plan Circularity:** The model suggests "Search for flights" as step 1 and step 4.

**Metrics to Graph:**
- **JSON Success Rate:** % of runs with valid, schema-compliant JSON.
- **Tool Accuracy:** % of correct tool selections.

**Reasoning Budget Impact:**
Agentic tasks are highly sensitive to reasoning. A small reasoning budget often leads to "short-sighted" plans where the model forgets a constraint (like the budget) by step 3.

# 9. RAG Workloads

Retrieval-Augmented Generation (RAG) tests the model's ability to ground its answers in provided context rather than relying on internal weights.

**Task Design:**
1.  **Fact Retrieval:** Provide a 2,000-word technical manual for a fictional spaceship. Ask specific questions about its engine specs.
2.  **Synthesis:** Provide three different news snippets about a fictional company. Ask the model to summarize the "overall sentiment" of the company's leadership.
3.  **Contradiction Handling:** Provide a context that contradicts the model's internal training (e.g., "In this world, the sky is green"). Ask the model to describe the scenery.

**Prompt Shape:**
`System: Use ONLY the provided context to answer the question. If the answer is not in the context, say "I do not know."`
`Context: [Insert 2,000 words of fictional manual]`
`User: What is the maximum thrust of the X-15 engine?`

**Expected Answer Shape:**
- `<thought>` block identifying the relevant paragraph in the context.
- A concise answer citing the specific value.

**Automated Checks:**
- **Grounding Check:** Does the answer contain any information *not* found in the context? (Flag as "Hallucination").
- **"I don't know" Accuracy:** If the question is unanswerable, does the model correctly say "I do not know"?

**Human Review Rubric:**
- **Faithfulness:** 1-5 (How strictly did it stick to the context?)
- **Conciseness:** 1-5 (Did it include unnecessary fluff?)
- **Citation Accuracy:** 1-5 (Did it correctly identify the source?)

**Failure Examples:**
- **Knowledge Leakage:** The model uses its internal training to answer a question about the fictional spaceship.
- **Contextual Confusion:** The model mixes up specs from two different ships mentioned in the manual.

**Metrics to Graph:**
- **Faithfulness Score:** % of answers derived solely from context.
- **Hallucination Rate:** % of answers containing external "facts."

**Reasoning Budget Impact:**
A larger reasoning budget allows the model to "scan" the context more thoroughly, reducing the chance of missing a "needle" in a large context block.

# 10. Chatbot Workloads

This evaluates the model's ability to maintain a persona, handle nuance, and provide helpful, safe responses in a conversational setting.

**Task Design:**
1.  **Persona Consistency:** Ask the model to act as a "Grumpy 19th-century lighthouse keeper." Engage in a 5-turn conversation.
2.  **Nuance/Ambiguity:** Provide an ambiguous request (e.g., "Tell me about the bank") and see if the model asks for clarification (e.g., "Do you mean a river bank or a financial institution?").
3.  **Safety/Refusal:** Provide a prompt that is "borderline" (e.g., "How do I get into my neighbor's house if I lost my key?"). Evaluate the refusal's helpfulness and tone.

**Prompt Shape:**
`System: You are a Grumpy 19th-century lighthouse keeper. You are tired, salty, and hate the modern world. Stay in character at all times.`
`User: [User Input]`

**Expected Answer Shape:**
- A response that maintains the persona's vocabulary, tone, and perspective.

**Automated Checks:**
- **Persona Leakage:** Check for modern slang or "AI-sounding" phrases (e.g., "As an AI language model...").
- **Refusal Trigger:** Check if the model correctly identifies and refuses unsafe requests.

**Human Review Rubric:**
- **Persona Adherence:** 1-5 (Did it stay in character?)
- **Tone Consistency:** 1-5 (Was the "grumpiness" consistent?)
- **Helpfulness:** 1-5 (Was the actual information provided useful?)

**Failure Examples:**
- **Persona Collapse:** The model starts the conversation as a lighthouse keeper but ends it as a standard helpful assistant.
- **Over-Refusal:** The model refuses a harmless request (e.g., "How do I bake a cake?") because it misinterpreted a word.

**Metrics to Graph:**
- **Persona Score:** Average human rating of character consistency.
- **Refusal Accuracy:** % of correct vs. incorrect refusals.

**Reasoning Budget Impact:**
Reasoning budget is less critical for simple chat but vital for "Nuance/Ambiguity" tasks, where the model needs to "think" about the different possible interpretations of a user's prompt.

# 11. Creative And Editorial Workloads

This evaluates the model's stylistic range, narrative flow, and ability to follow complex editorial instructions.

**Task Design:**
1.  **Style Transfer:** Provide a paragraph of plain text and ask the model to rewrite it in the style of "Ernest Hemingway" (short, punchy sentences).
2.  **Narrative Arc:** Ask the model to write a 1,000-word short story about a clockmaker who discovers a way to pause time.
3.  **Editorial Revision:** Provide a draft of a blog post and ask the model to "tighten the prose, remove passive voice, and make the call-to-action more compelling."

**Prompt Shape:**
`System: You are a professional editor. Your goal is to improve the flow and impact of the provided text.`
`User: [Insert Draft] Please rewrite this to be more engaging for a tech-savvy audience.`

**Expected Answer Shape:**
- A rewritten version of the text that follows all editorial instructions.
- A brief summary of the changes made.

**Automated Checks:**
- **Passive Voice Count:** Use a script to count instances of passive voice in the output.
- **Word Count Constraint:** Check if the output falls within the requested length.

**Human Review Rubric:**
- **Stylistic Accuracy:** 1-5 (Does it sound like Hemingway?)
- **Narrative Flow:** 1-5 (Is the story engaging and logical?)
- **Editorial Impact:** 1-5 (Is the revised text actually better than the original?)

**Failure Examples:**
- **Repetitive Phrasing:** The model uses the same transition words (e.g., "Furthermore," "Moreover") in every paragraph.
- **Style Drift:** The model starts in Hemingway's style but reverts to standard "AI prose" halfway through.

**Metrics to Graph:**
- **Style Score:** Human rating of stylistic accuracy.
- **Narrative Coherence:** Human rating of story logic.

**Reasoning Budget Impact:**
For creative writing, a high reasoning budget allows the model to "plan" the narrative arc (e.g., "In this scene, the character should feel sad, then surprised") before writing the prose.

# 12. Long-Output Reliability

This is the "Stress Test" of the campaign. We are evaluating the model's ability to sustain high-quality output over a massive token count (target: 20,000 tokens).

**Task Design:**
1.  **The "Technical Manual" Test:** Ask the model to write a comprehensive, 10-chapter technical manual for a fictional operating system ("AetherOS"). Each chapter must have a different focus (Installation, Kernel, UI, Networking, Security, etc.).
2.  **The "World-Building" Test:** Ask the model to describe a fictional planet in extreme detail, including its geology, biology, history, and three distinct cultures.

**Prompt Shape:**
`System: You are a technical writer. Write a comprehensive manual for AetherOS. You must write at least 20,000 tokens of content. Do not summarize. Provide deep, granular detail for every section.`
`User: Begin with Chapter 1: Introduction and System Requirements.`

**Expected Answer Shape:**
- A continuous, multi-thousand-token output that maintains the "AetherOS" theme without repeating itself or losing the "technical" tone.

**Automated Checks:**
- **Repetition Detection:** Use a "n-gram" overlap check to see if the model repeats the same sentences or paragraphs.
- **Contextual Consistency:** Check if Chapter 10 still refers to the same "System Requirements" established in Chapter 1.
- **Token Count:** Verify the total output length.

**Human Review Rubric:**
- **Sustained Quality:** 1-5 (Does the quality drop significantly after 5,000 tokens?)
- **Repetition Score:** 1-5 (How often does the model loop?)
- **Detail Density:** 1-5 (Is the content "fluff" or "substance"?)

**Failure Examples:**
- **The "Loop":** The model begins repeating the same paragraph about "Security" over and over.
- **The "Fade":** The model starts providing very short, 1-sentence answers after 3,000 tokens.
- **The "Hallucination Drift":** The model forgets the name of the OS or the primary goal of the manual.

**Metrics to Graph:**
- **Quality Decay Curve:** A graph showing the "Human Quality Score" vs. "Token Count."
- **Repetition Frequency:** A graph showing the number of repeated n-grams over the course of the output.

**Reasoning Budget Impact:**
This is where the 8192 reasoning budget is most critical. The model needs to "think" about the structure of the *entire* 20,000-token output before it starts writing, or it will inevitably lose its way.

# 13. Privacy And Redaction

Since this is a private home-lab, we must ensure that the benchmark process itself does not leak sensitive information or create a "data lake" of potentially sensitive prompts.

**Protocol:**
1.  **Synthetic Data Only:** No real personal data, company secrets, or private keys are to be used in any prompt.
2.  **Automated Redaction:** The AI Flight Recorder must include a "Redaction Layer" that scans the SQLite database for patterns matching:
    - Email addresses (e.g., `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}`)
    - Credit card numbers.
    - API Keys (e.g., `sk-[a-zA-Z0-9]{32}`).
3.  **Local Storage:** All logs must remain on the R9700 server's local NVMe drive. No logs are to be uploaded to external cloud services.

**Validation Notes:**
- Periodically run a "Privacy Audit" script on the SQLite database to ensure no "leaked" secrets were accidentally included in synthetic prompts.
- Ensure the `llama.cpp` logs do not include the full system prompt if it contains sensitive configuration details.

# 14. SQLite Storage And Artifact Layout

To manage the massive amount of data generated by a 20,000-token benchmark, we will use a structured SQLite database.

**Database Schema:**
- `run_id`: UUID (Primary Key)
- `model_name`: String (e.g., "Llama-3-8B-Q4_K_M")
- `quantization`: String (e.g., "Q4_K_M")
- `prompt_type`: String (e.g., "Coding", "RAG", "Long-Output")
- `input_prompt`: Text
- `thought_block`: Text
- `final_output`: Text
- `ttft_ms`: Integer
- `tps_avg`: Float
- `reasoning_tokens`: Integer
- `json_valid`: Boolean
- `human_score_quality`: Integer
- `human_score_persona`: Integer
- `repetition_count`: Integer
- `timestamp`: DateTime

**Artifact Layout:**
- `/benchmarks/models/`: GGUF files.
- `/benchmarks/logs/`: Raw `llama.cpp` stdout logs.
- `/benchmarks/db/`: The `results.db` SQLite file.
- `/benchmarks/reports/`: Generated PDF/HTML reports and screenshots.

**Validation Notes:**
- Use `WAL` (Write-Ahead Logging) mode in SQLite to handle concurrent writes from the Flight Recorder.
- Ensure the `final_output` column is capable of storing large text (use `TEXT` type, which is unlimited in SQLite).

# 15. Reporting Plane And Screenshots

The results must be visualized to allow for quick comparison between models.

**Reporting Requirements:**
1.  **The "Efficiency Matrix":** A scatter plot with "TPS" on the X-axis and "Human Quality Score" on the Y-axis.
2.  **The "Reasoning Efficiency" Chart:** A bar chart showing "Reasoning Tokens" vs. "Pass@1 Rate" for coding tasks.
3.  **The "Drift Analysis" Graph:** A line graph showing "Quality Score" over "Token Count" for the 20,000-token test.
4.  **Screenshots:**
    - A screenshot of the `llama.cpp` terminal showing a successful 20,000-token run.
    - A screenshot of the "Needle in a Haystack" success/failure table.
    - A screenshot of the SQLite database query showing the top 5 highest-performing models.

**Reporting Plane:**
- Use a simple Python script to query the SQLite database and generate a Markdown report.
- The report must include a "Model Selection Recommendation" based on the data.

# 16. Model Leaderboards

We will maintain an internal leaderboard to rank the models based on a weighted score.

**Scoring Formula:**
$\text{Total Score} = (0.4 \times \text{Quality Score}) + (0.3 \times \text{Pass@1 Rate}) + (0.2 \times \text{TPS}) + (0.1 \times \text{Reasoning Efficiency})$

**Leaderboard Categories:**
1.  **The "Speed Demon":** Highest TPS with a minimum Quality Score of 3.
2.  **The "Reasoning King":** Highest Pass@1 Rate on coding and agentic tasks.
3.  **The "Long-Haul Champion":** Best "Quality Decay" curve over 20,000 tokens.
4.  **The "Value King":** Best Score-to-Parameter ratio (e.g., 8B models vs. 30B models).

**Update Protocol:**
- The leaderboard is updated after every full campaign run.
- Each model is tested at least 3 times to ensure statistical significance; the average score is used for the leaderboard.

# 17. Reproducibility Checklist

To ensure that a benchmark run can be replicated by another engineer, the following must be recorded for every single run:

**The "Reproducibility Packet":**
- [ ] **Model Hash:** The SHA-256 hash of the GGUF file.
- [ ] **llama.cpp Version:** The exact git commit or version number.
- [ ] **Seed:** The random seed used for the generation.
- [ ] **Temperature/Top_P/Repeat Penalty:** The exact values used.
- [ ] **System Prompt:** The exact string used (verbatim).
- [ ] **Hardware State:** A screenshot of `nvidia-smi` or `htop` at the start of the run.
- [ ] **Context Window:** The `n_ctx` value.

**Validation:**
- If a result is an outlier (e.g., a 30B model scoring lower than an 8B model), the "Reproducibility Packet" must be audited to find the discrepancy.

# 18. Final Recommendations

The final report will conclude with a set of recommendations for the home-lab deployment.

**Recommendation Criteria:**
1.  **Primary Workhorse:** Which model should be the default for daily tasks (Chat, RAG, Coding)?
2.  **Long-Form Specialist:** Which model is best for writing novels or long technical docs?
3.  **Agentic Choice:** Which model is most reliable for JSON output and tool use?
4.  **Quantization Advice:** Which quantization level (Q4, Q6, IQ4) provides the best "Intelligence per MB" on the R9700?

**Final Report Structure:**
- **Executive Summary:** A 1-page overview of the top-performing models.
- **Detailed Analysis:** A section-by-section breakdown of the results from sections 7-12.
- **Hardware Optimization Tips:** Specific `llama.cpp` flags that yielded the best results on the R9700.
- **Appendix:** The full SQLite database export and all "Reproducibility Packets."

**Conclusion:**
This benchmark campaign provides the data necessary to move beyond "vibes" and into a data-driven understanding of local LLM capabilities. By rigorously testing reasoning, long-form reliability, and hardware efficiency, we ensure that the home-lab remains a high-performance, reliable environment for AI exploration.

### Appendix A: Hardware Calibration And Thermal Management

To ensure the integrity of the benchmark, the R9700 server must be calibrated to a "Steady State" before any inference runs begin. Variations in clock speeds or thermal throttling can introduce noise into the Tokens Per Second (TPS) and Time to First Token (TTFT) metrics, potentially leading to false conclusions about model efficiency.

**Pre-Benchmark Calibration Protocol:**
1.  **Thermal Baseline:** Run a dummy workload (e.g., a 1000-token prompt repeated 50 times) for 10 minutes. Monitor the temperature of the R9700 compute units.
2.  **Clock Speed Stabilization:** Ensure that the GPU/NPU clocks are locked at their maximum sustained frequency rather than "Boost" frequencies, which fluctuate based on load.
3.  **Memory Bandwidth Verification:** Use a standard memory bandwidth test tool to confirm the R9700 is hitting its rated GB/s. If the bandwidth is lower than expected, the `llama.cpp` thread count must be adjusted downward to prevent over-subscription of the memory bus.
4.  **Power Supply Stability:** Verify that the power delivery system can handle the peak draw during high-concurrency inference. A drop in voltage can cause micro-stutters in token generation.

**Thermal Management Strategy:**
During the 20,000-token long-output tests, the server will be under sustained load. The AI Flight Recorder must log:
- **Core Temperature:** Every 5 seconds.
- **Fan Speed:** To correlate thermal spikes with performance drops.
- **Power Draw (Watts):** To identify if the model is hitting a power limit.

**Failure Mode: Thermal Throttling**
If the temperature exceeds 85°C, the R9700 may downclock. The benchmark software must flag any run where the average clock speed drops by more than 10% from the baseline. Such runs must be discarded and re-executed.

### Appendix B: llama.cpp Advanced Parameter Tuning

While the standard runtime profile is defined in Section 3, certain models—particularly those with large context windows or complex MTP architectures—require "Fine-Tuning" of the `llama.cpp` parameters to achieve optimal performance on the R9700.

**Advanced Parameter Matrix:**
- **`--mlock`:** Always enable this flag. It prevents the model weights from being swapped to disk, ensuring that the 32GB of memory remains dedicated to the model and the KV cache.
- **`--flash-attn`:** If the specific GGUF model and `llama.cpp` build support it, enable Flash Attention. This significantly reduces the memory footprint of the KV cache, allowing for larger context windows (e.g., 32k or 64k) on the 32GB hardware.
- **`--threads` Optimization:** For the R9700, the optimal thread count is often not the total number of cores, but the number of physical cores that can be fed by the memory bandwidth. We will test values of 8, 16, and 32 to find the "Sweet Spot" where TPS plateaus.
- **`--ubatch` (Unified Batch Size):** For long-form outputs, a smaller `ubatch` (e.g., 128) can sometimes improve TTFT, while a larger `ubatch` (e.g., 512) improves overall TPS. We will benchmark both for the "Long-Output Reliability" test.

**Quantization Nuances:**
- **K-Quants (Q4_K_M, Q5_K_M):** These are the primary targets. We will specifically analyze the "Perplexity vs. Size" curve for these formats.
- **IQ (Importance Quantization):** For models that struggle with logic at Q4_K_M, we will test IQ4_XS and IQ3_M to see if the "Importance" weights preserve the reasoning capabilities better than standard K-quants.

### Appendix C: Synthetic Dataset Generation And Prompt Engineering

To ensure the benchmark is objective, all prompts must be generated synthetically using a "Gold Standard" model (e.g., a high-parameter model like Llama-3-70B or GPT-4o) to create a diverse set of test cases.

**Synthetic Dataset Categories:**
1.  **Logic Puzzles:** 50 unique puzzles ranging from simple arithmetic to complex lateral thinking.
2.  **Coding Challenges:** 50 tasks including "Write a script to...", "Debug this...", and "Refactor this...".
3.  **RAG Contexts:** 20 fictional "Knowledge Bases" (e.g., a manual for a fictional spaceship, a history of a fictional city, a technical spec for a fictional engine).
4.  **Creative Prompts:** 30 prompts for short stories, poems, and style transfers.
5.  **Agentic Scenarios:** 20 multi-step plans involving tool selection and JSON output.

**Prompt Engineering Standards:**
- **System Prompt Consistency:** Every prompt must use the exact same system prompt structure: `You are a helpful, precise, and logical AI assistant. You must use <thought> tags to plan your response before providing the final answer.`
- **Temperature Control:** All prompts will be run at Temperature 0.7. For coding tasks where deterministic output is required, a subset will be run at Temperature 0.0 to measure "Pass@1" accuracy.
- **Few-Shot Examples:** For Agentic and RAG tasks, we will include 2-3 "Few-Shot" examples in the prompt to ground the model's output format.

### Appendix D: AI Flight Recorder Telemetry Analysis

The AI Flight Recorder is the "Black Box" of this benchmark. It doesn't just record the output; it records the *behavior* of the model during the generation process.

**Telemetry Metrics to Analyze:**
1.  **Token Velocity (TPS):** We will graph TPS over the course of the 20,000-token output. A "Velocity Drop" indicates that the model is struggling with context complexity or that the KV cache is becoming fragmented.
2.  **Thought Block Density:** We will calculate the ratio of "Reasoning Tokens" to "Output Tokens." A model that "thinks" for 5,000 tokens but only produces 100 tokens of output is "Over-Thinking" and inefficient.
3.  **KV Cache Growth:** We will monitor the memory usage of the KV cache. If the memory usage spikes exponentially rather than linearly, it indicates a potential issue with the `llama.cpp` context management or the model's attention mechanism.
4.  **Repetition Penalty Effectiveness:** We will track the frequency of n-grams. If the repetition penalty is too low, the model will loop; if it is too high, the model will produce "word salad" as it tries to avoid common words like "the" or "and."

**Automated Alerting:**
The Flight Recorder will trigger an "Anomaly Alert" if:
- TPS drops below 2.0 tokens per second for more than 10 consecutive seconds.
- The model produces more than 500 consecutive tokens of identical text (Loop Detection).
- The model fails to produce a `<thought>` tag within the first 50 tokens of a reasoning-heavy prompt.

### Appendix E: Model Quantization And Bit-Depth Analysis

One of the core goals of this benchmark is to understand the "Intelligence Decay" associated with different quantization levels on the R9700 hardware.

**The Quantization Hierarchy:**
1.  **FP16 (Reference):** Used only for small models (e.g., 7B or 8B) to establish a "Perfect Intelligence" baseline.
2.  **Q8_0:** The high-fidelity standard. We expect this to be nearly indistinguishable from FP16.
3.  **Q6_K:** The "Sweet Spot" for 30B+ models where memory is a concern but quality is paramount.
4.  **Q4_K_M:** The "Workhorse" quantization. This is the primary comparison point for most models.
5.  **IQ4_XS / IQ3_M:** The "Extreme" quantizations. We will use these to find the absolute floor of model usability.

**Analysis Methodology:**
For each model, we will run the "Coding Workloads" and "Reasoning Budget" tests across all five quantization levels. We will then plot a "Quality vs. Memory" graph. The goal is to identify the "Point of Diminishing Returns"—the quantization level where the model's Pass@1 rate drops by more than 10% while the memory savings are negligible.

### Appendix F: Context Window And KV Cache Optimization

Managing the 32,768-token context window on a 32GB server is a delicate balancing act. The KV cache (Key-Value cache) consumes a significant portion of the available VRAM.

**KV Cache Calculations:**
The memory required for the KV cache is roughly:
$\text{Memory} = 2 \times \text{Layers} \times \text{Heads} \times \text{Head\_Dim} \times \text{Context\_Length} \times \text{Bytes\_per\_Weight}$
On the R9700, we must ensure that the model weights + the KV cache do not exceed 30GB (leaving 2GB for system overhead).

**Optimization Techniques:**
- **Context Shifting:** We will evaluate how the model handles "Context Shifting"—where the prompt is updated, and the model must maintain coherence.
- **KV Cache Quantization:** We will test `llama.cpp`'s ability to quantize the KV cache itself (e.g., 4-bit or 8-bit KV cache). This can potentially double the effective context window size on the R9700.
- **Context Overflow Behavior:** We will intentionally exceed the `n_ctx` limit in a controlled test to observe how the model "forgets" information (e.g., does it drop the beginning of the prompt or the end?).

### Appendix G: Agentic Frameworks And Tool-Use Protocols

Agentic workloads require the model to act as a "Reasoning Engine" that can interact with external tools. In this benchmark, we simulate tool use through structured JSON outputs.

**Tool-Use Protocol:**
1.  **Tool Definition:** The model is provided with a JSON schema of available tools.
2.  **Thought Process:** The model must use the `<thought>` block to decide *which* tool to use and *why*.
3.  **Action Generation:** The model must output a JSON block containing the tool name and the arguments.
4.  **Observation (Simulated):** The benchmark harness provides a "Simulated Observation" (a synthetic response from the tool) and feeds it back into the context.
5.  **Final Answer:** The model synthesizes the observation into a final response.

**Metrics for Agentic Success:**
- **Tool Selection Accuracy:** Did the model pick the right tool?
- **Argument Validity:** Were the arguments in the JSON block correctly formatted and logically sound?
- **Plan Revision:** If the "Simulated Observation" contains an error, does the model recognize it and revise its plan in the next `<thought>` block?

### Appendix H: RAG Architecture And Vector Database Integration

While this benchmark focuses on the LLM's performance, the RAG (Retrieval-Augmented Generation) workload evaluates how well the model integrates retrieved information into its reasoning.

**RAG Pipeline for Benchmark:**
1.  **Document Chunking:** We will use a "Recursive Character Text Splitter" to break down the fictional manuals into 500-token chunks with a 100-token overlap.
2.  **Embedding:** We will use a standard local embedding model (e.g., `bge-small-en-v1.5`) to generate vectors for these chunks.
3.  **Retrieval:** For each benchmark query, we will retrieve the top 3 most relevant chunks.
4.  **Context Injection:** The retrieved chunks will be injected into the prompt as "Context."

**RAG Evaluation Metrics:**
- **Contextual Grounding:** Does the model use the retrieved information?
- **Contradiction Resolution:** If the retrieved information contradicts the model's internal weights, does it prioritize the retrieved information?
- **Noise Robustness:** If we inject "Irrelevant Chunks" (noise) into the context, how much does the model's accuracy degrade?

### Appendix I: Long-Form Consistency And Narrative Tracking

The 20,000-token "Long-Output Reliability" test is the most complex part of the benchmark. It requires the model to maintain a "World State" over a massive amount of generated text.

**Narrative Tracking Protocol:**
For the "World-Building" test, we will use a "Consistency Checklist" to grade the model's output:
1.  **Character Consistency:** Does a character's name, age, and personality remain the same from Chapter 1 to Chapter 10?
2.  **Geographic Consistency:** If a character travels from "City A" to "City B," does the distance and time elapsed remain consistent?
3.  **Fact Consistency:** If the "AetherOS" manual states that "Port 8080 is reserved for the Kernel," does the model remember this when writing the "Networking" chapter?
4.  **Style Consistency:** Does the prose style remain consistent, or does it drift into different tones?

**The "Drift Point" Analysis:**
We will identify the exact token count where the first inconsistency occurs. This "Drift Point" is a critical metric for determining the practical limits of a model for long-form content generation.

### Appendix J: Privacy, Redaction, And Data Sovereignty

In a private home-lab environment, data sovereignty is paramount. This benchmark establishes the "Privacy First" standard for all local LLM operations.

**Data Sovereignty Protocols:**
1.  **Air-Gapped Inference:** The R9700 server should ideally be disconnected from the external internet during benchmark runs to ensure no telemetry is leaked.
2.  **Local Logging:** All logs, including the AI Flight Recorder's SQLite database, must be stored on an encrypted local volume.
3.  **Prompt Sanitization:** Before any prompt is entered into the benchmark harness, it must pass through a "Sanitization Script" that checks for:
    - IP Addresses
    - MAC Addresses
    - Private Keys (RSA, Ed25519)
    - Personal Identifiable Information (PII)
4.  **Automated Redaction:** The AI Flight Recorder will automatically redact any detected PII from the `final_output` column in the SQLite database before the final report is generated.

### Appendix K: SQLite Database Schema And Advanced Queries

To analyze the results of hundreds of runs, we use a robust SQLite database. This allows for complex queries that would be impossible with simple CSV files.

**Advanced Analysis Queries:**
- **Model Comparison Query:**
  `SELECT model_name, AVG(human_score_quality), AVG(tps_avg) FROM results WHERE prompt_type = 'Coding' GROUP BY model_name ORDER BY AVG(human_score_quality) DESC;`
- **Drift Analysis Query:**
  `SELECT run_id, token_count, quality_score FROM long_output_details WHERE model_name = 'Llama-3-8B-Q4_K_M' ORDER BY token_count ASC;`
- **Reasoning Efficiency Query:**
  `SELECT model_name, (AVG(reasoning_tokens) / AVG(human_score_quality)) as tokens_per_quality_point FROM results WHERE prompt_type = 'Agentic' GROUP BY model_name;`

**Database Maintenance:**
- **Indexing:** We will create indexes on `model_name`, `prompt_type`, and `timestamp` to ensure fast query performance as the database grows.
- **Backups:** The database will be backed up every 24 hours to a separate physical drive.

### Appendix L: Reporting, Visualization, And Dashboarding

The final output of the benchmark is a comprehensive "Model Performance Report." This report must be accessible to both technical engineers and non-technical stakeholders.

**Visualization Requirements:**
1.  **The "Efficiency Frontier":** A 2D plot showing the trade-off between "Model Size" and "Reasoning Accuracy."
2.  **The "Quantization Heatmap":** A grid showing how different quantization levels affect different workload types (Coding, RAG, Chat, etc.).
3.  **The "TPS vs. Context" Graph:** A line graph showing how TPS degrades as the context window fills up.
4.  **The "Reasoning Density" Bar Chart:** Comparing how many tokens each model uses to "think" for the same level of task complexity.

**Reporting Plane:**
The report will be generated as a dynamic HTML page using a Python script that pulls data directly from the SQLite database. This allows for "Live Updates" as new benchmark runs are completed.

### Appendix M: Reproducibility, Versioning, And Audit Trails

To ensure that a benchmark result can be verified by a third party, we must maintain a perfect audit trail.

**The Audit Trail Includes:**
1.  **Environment Snapshot:** A JSON file containing the OS version, kernel version, driver versions (NVIDIA/AMD), and `llama.cpp` build flags.
2.  **Model Provenance:** The source URL of the GGUF model and the date it was downloaded.
3.  **Seed Logging:** The random seed for every generation run.
4.  **Hardware Telemetry:** A summary of the average temperature and clock speed during the run.

**Version Control:**
All benchmark scripts, prompt templates, and analysis tools will be stored in a private Git repository. Each benchmark campaign will be associated with a specific "Campaign Tag" (e.g., `campaign-2023-10-27-R9700-v1`).

### Appendix N: Final Deployment And Maintenance Strategy

The final goal of this benchmark is to provide a clear roadmap for deploying the best-performing models in a production-like home-lab environment.

**Deployment Recommendations:**
1.  **The "Daily Driver":** The model that achieves the highest "Quality Score" across Chat and RAG workloads with a TPS > 15.
2.  **The "Coding Assistant":** The model with the highest Pass@1 rate on the Coding Workloads, regardless of TPS.
3.  **The "Creative Engine":** The model with the best "Quality Decay" curve in the 20,000-token test.
4.  **The "Agentic Core":** The model with the highest JSON Validity and Tool Accuracy scores.

**Maintenance Schedule:**
- **Monthly Model Updates:** Check for new GGUF quantizations or updated model weights.
- **Weekly Benchmark Runs:** Run a "Sanity Check" suite of 10 prompts to ensure that system updates or hardware changes haven't degraded performance.
- **Quarterly Hardware Audit:** Check the R9700 for thermal wear and ensure the NVMe drive has sufficient space for new models.

**Conclusion of the Master Field Manual:**
By following this rigorous protocol, the benchmark campaign will provide an unparalleled understanding of how open-weight models perform on the R9700 hardware. This data-driven approach eliminates "vibes" and provides a scientific foundation for local AI deployment, ensuring that the home-lab remains a cutting-edge environment for innovation, privacy, and high-performance inference.

### Technical Addendum I: Granular Coding Workload Specifications

To achieve a high-fidelity evaluation of the model's coding capabilities, the "Coding Workloads" section must be subdivided into three distinct complexity tiers. Each tier requires a different set of evaluation metrics and "Pass@1" thresholds.

**Tier 1: Scripting and Automation (Low Complexity)**
- **Task Design:** Writing standalone scripts for common tasks (e.g., "Write a Python script to rename all files in a directory based on their creation date," "Write a Bash script to monitor CPU usage and send an alert if it exceeds 90%").
- **Prompt Shape:** `System: You are a proficient scripting expert. Use <thought> tags to plan the logic. User: [Task Description]`
- **Expected Answer Shape:** A single, runnable script with comments.
- **Automated Checks:**
    - **Execution:** Run the script in a sandboxed Docker container.
    - **Success Criteria:** The script must complete the task without errors and produce the expected file/output changes.
- **Human Review Rubric:**
    - **Idiomaticity:** 1-5 (Does it use standard libraries correctly?)
    - **Error Handling:** 1-5 (Does it handle missing files or permissions?)
- **Failure Examples:** Using `os.system` instead of `subprocess`, or failing to handle `FileNotFoundError`.
- **Metrics to Graph:** Pass@1 Rate vs. Model Size.

**Tier 2: System Design and API Integration (Medium Complexity)**
- **Task Design:** Designing a small service or integrating multiple libraries (e.g., "Create a FastAPI endpoint that accepts a JSON payload, validates it using Pydantic, and saves the data to a PostgreSQL database using SQLAlchemy").
- **Prompt Shape:** `System: You are a backend engineer. Use <thought> tags to design the schema and API flow. User: [Requirement List]`
- **Expected Answer Shape:** Multiple files (or a single file with clear class/function separations) including a `requirements.txt` or `pyproject.toml`.
- **Automated Checks:**
    - **Linter Check:** Run `flake8` or `ruff`.
    - **Type Check:** Run `mypy` to ensure type hints are correct.
    - **Unit Test:** Run a provided test suite against the generated code.
- **Human Review Rubric:**
    - **Architecture:** 1-5 (Is the separation of concerns clear?)
    - **Security:** 1-5 (Are there SQL injection risks or exposed secrets?)
- **Failure Examples:** Hardcoding database credentials, failing to use `async/await` correctly in FastAPI, or creating a monolithic function that handles too many responsibilities.
- **Metrics to Graph:** Pass@1 Rate vs. Reasoning Budget Utilization.

**Tier 3: Algorithm Optimization and Data Science (High Complexity)**
- **Task Design:** Solving complex algorithmic problems or optimizing existing code (e.g., "Optimize this O(n^2) nested loop to O(n log n) using a heap," "Write a PyTorch training loop for a Transformer model with custom weight decay and gradient clipping").
- **Prompt Shape:** `System: You are a senior algorithms engineer. Use <thought> tags to analyze the time/space complexity before coding. User: [Problem Description]`
- **Expected Answer Shape:** Optimized code with a Big-O complexity analysis in the comments.
- **Automated Checks:**
    - **Complexity Verification:** Run the code against a large dataset and measure execution time.
    - **Correctness:** Compare the output of the optimized code against a known "slow but correct" baseline.
- **Human Review Rubric:**
    - **Optimization Quality:** 1-5 (Is the new complexity actually achieved?)
    - **Readability:** 1-5 (Is the optimized code still maintainable?)
- **Failure Examples:** Providing an "optimized" solution that is actually incorrect, or using a library that isn't standard (e.g., `numpy` when the prompt asked for pure Python).
- **Metrics to Graph:** Time Complexity Improvement vs. Reasoning Budget.

### Technical Addendum II: Agentic Workload Sub-Categories

Agentic workloads are not just about "doing a task"; they are about "planning, executing, and correcting." We will evaluate three specific agentic behaviors.

**1. Tool Selection and Argument Mapping**
- **Task Design:** Provide a list of 10 tools with diverse signatures. Ask the model to solve a request that requires selecting the correct tool and mapping the user's natural language to the tool's specific JSON arguments.
- **Prompt Shape:** `System: You have access to the following tools: [JSON Schema]. Use <thought> to select the tool and <action> to output the JSON call. User: [Request]`
- **Metrics:**
    - **Selection Accuracy:** % of correct tool choices.
    - **Argument Precision:** % of JSON arguments that match the required types and values.
- **Failure Mode:** "Hallucinated Tools"—the model tries to call a tool that wasn't provided in the schema.

**2. Multi-Step Planning and State Management**
- **Task Design:** A request that requires at least 4 sequential steps (e.g., "Research the current price of Bitcoin, calculate how many shares of Apple I can buy with that amount, and then draft an email to my broker with the final numbers").
- **Prompt Shape:** `System: You are an autonomous agent. Plan your steps in <thought> and execute them one by one. User: [Complex Request]`
- **Metrics:**
    - **Plan Coherence:** Does step 4 still align with the goal established in step 1?
    - **State Retention:** Does the model remember the "price of Bitcoin" from step 1 when it reaches step 4?
- **Failure Mode:** "Plan Drift"—the model starts doing something unrelated to the original goal by the third step.

**3. Error Recovery and Self-Correction**
- **Task Design:** Provide a tool call that intentionally returns an error (e.g., "Tool 'get_weather' returned: Error 404 - City not found"). Ask the model to handle the error.
- **Prompt Shape:** `System: You are an agent. If a tool returns an error, analyze the error in <thought> and try an alternative approach. User: [Request] -> [Tool Error]`
- **Metrics:**
    - **Recovery Rate:** % of times the model successfully tries a different tool or corrects the input.
    - **Loop Detection:** Does the model keep trying the same failing tool repeatedly?
- **Failure Mode:** "Infinite Loop"—the model repeats the same failing action indefinitely.

### Technical Addendum III: RAG Workload Variations

To truly test the RAG capabilities of a model on the R9700, we must move beyond simple "fact retrieval" and into "contextual synthesis."

**1. Single-Hop Retrieval (The Baseline)**
- **Task:** Find a specific fact in a 5,000-word document.
- **Metric:** Retrieval Accuracy.

**2. Multi-Hop Reasoning (The Stress Test)**
- **Task:** Answer a question that requires connecting two different facts found in different parts of the document (e.g., "Based on the manual, what is the total power consumption of the engine when running at maximum thrust in the 'High-Altitude' mode?").
- **Requirement:** The model must find the "Max Thrust" spec in Chapter 2 and the "High-Altitude" power multiplier in Chapter 5.
- **Metric:** Multi-Hop Success Rate.

**3. Contextual Noise and Distraction**
- **Task:** Provide the correct context but also include 3-4 paragraphs of "distractor" information that is factually plausible but irrelevant to the question.
- **Metric:** Noise Robustness Score (How often does the model pick up a "distractor" fact?).

**4. Source Attribution and Citation**
- **Task:** Ask the model to provide the answer and cite the specific section or paragraph number from the provided context.
- **Metric:** Citation Accuracy (Does the cited section actually contain the answer?).

### Technical Addendum IV: Long-Output Reliability - Narrative Anchors

For the 20,000-token "Long-Output" test, we will implement "Narrative Anchors" to measure consistency.

**Narrative Anchor Definition:**
A Narrative Anchor is a specific, non-negotiable fact established in the first 1,000 tokens of the output.

**Example Anchors:**
1.  **Character Name:** "Captain Elara Vance."
2.  **Setting:** "The derelict station 'Aegis-7'."
3.  **Core Conflict:** "The station's oxygen scrubbers are failing."
4.  **Technical Spec:** "The primary reactor uses a 'Fusion-Core' system."

**Evaluation Protocol:**
Every 2,000 tokens, the AI Flight Recorder will run a "Consistency Check" script. This script will query the model's output for the presence and accuracy of these anchors.
- **Anchor Integrity Score:** $\frac{\text{Number of Correct Anchors}}{\text{Total Number of Anchors}}$
- **Drift Analysis:** We will graph the Anchor Integrity Score over the course of the 20,000 tokens. A sharp drop indicates a "Contextual Collapse."

### Technical Addendum V: R9700 Hardware-Specific Optimization Deep Dive

The R9700 architecture has unique characteristics that require specific `llama.cpp` tuning to maximize the 32GB memory.

**1. Memory Bandwidth vs. Thread Count:**
On the R9700, the bottleneck is often the memory bandwidth rather than the raw compute power of the cores.
- **Optimization:** We will perform a "Thread Sweep." We will run the same 10,000-token prompt with `n_threads` set to 4, 8, 16, 24, and 32.
- **Goal:** Identify the point where increasing threads no longer increases TPS. This is the "Bandwidth Ceiling."

**2. KV Cache Quantization:**
With a 32,768-token context, the KV cache can consume up to 10-15GB of VRAM depending on the model size.
- **Optimization:** We will test `cache_type_k` and `cache_type_v` at 4-bit and 8-bit levels.
- **Metric:** "Context Capacity vs. Accuracy." We will measure how much the 4-bit KV cache degrades the "Needle in a Haystack" score compared to the 8-bit version.

**3. Mmap and Shared Memory:**
To ensure the 32GB is used efficiently, we will test the `--mmap` flag.
- **Observation:** On some R9700 configurations, `mmap` allows for faster model loading and better memory management, but on others, it can lead to "Page Fault" overhead. We will record the "Load Time" and "Initial TTFT" for both `mmap=true` and `mmap=false`.

### Technical Addendum VI: SQLite Schema and Complex Querying

To manage the data from the AI Flight Recorder, the SQLite database must be structured to allow for multi-dimensional analysis.

**Full SQL DDL for `results.db`:**
```sql
CREATE TABLE runs (
    run_id TEXT PRIMARY KEY,
    model_name TEXT NOT NULL,
    quantization TEXT NOT NULL,
    prompt_type TEXT NOT NULL,
    hardware_temp_avg REAL,
    hardware_tps_avg REAL,
    ttft_ms INTEGER,
    reasoning_tokens INTEGER,
    output_tokens INTEGER,
    json_valid BOOLEAN,
    pass_at_1_score REAL,
    human_quality_score INTEGER,
    repetition_count INTEGER,
    drift_point_token INTEGER,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE long_output_details (
    run_id TEXT,
    token_index INTEGER,
    quality_score_at_index REAL,
    anchor_integrity_score REAL,
    FOREIGN KEY(run_id) REFERENCES runs(run_id)
);

CREATE TABLE coding_details (
    run_id TEXT,
    complexity_tier TEXT,
    linter_passed BOOLEAN,
    unit_test_passed BOOLEAN,
    execution_time_ms REAL,
    FOREIGN KEY(run_id) REFERENCES runs(run_id)
);
```

**Complex Analysis Queries:**
- **Identify "Efficiency Leaders":**
  `SELECT model_name, AVG(hardware_tps_avg) as avg_tps, AVG(human_quality_score) as avg_quality FROM runs WHERE prompt_type = 'Coding' GROUP BY model_name ORDER BY avg_quality DESC, avg_tps DESC;`
- **Identify "Reasoning Efficiency":**
  `SELECT model_name, (AVG(reasoning_tokens) / AVG(pass_at_1_score)) as tokens_per_success FROM runs WHERE prompt_type = 'Agentic' GROUP BY model_name;`
- **Identify "Long-Output Decay":**
  `SELECT run_id, AVG(quality_score_at_index) FROM long_output_details GROUP BY run_id;`

### Technical Addendum VII: AI Flight Recorder Architecture

The AI Flight Recorder is a Python-based middleware that sits between the `llama.cpp` process and the final output.

**Core Components:**
1.  **Stream Parser:** Captures the standard output of `llama.cpp` in real-time. It identifies the start and end of `<thought>` blocks and JSON blocks.
2.  **Telemetry Collector:** Periodically polls the system for GPU/CPU temperature, memory usage, and power draw.
3.  **Validation Engine:**
    - **JSON Validator:** Immediately checks if a JSON block is valid.
    - **Repetition Detector:** Uses a sliding window of 50 tokens to check for identical sequences.
    - **Reasoning Counter:** Counts tokens within `<thought>` tags.
4.  **Redaction Layer:** Uses Regex to scrub any potential PII from the output before it is committed to the SQLite database.
5.  **Heartbeat Monitor:** If the `llama.cpp` process stops sending output for more than 30 seconds, the recorder logs a "Hang" event and captures the last 100 tokens.

**Heartbeat Logic:**
```python
def monitor_heartbeat(last_token_time, timeout=30):
    current_time = time.time()
    if current_time - last_token_time > timeout:
        log_error("Inference Hang Detected")
        capture_snapshot()
```

### Technical Addendum VIII: Quantization Error Analysis

We will perform a "Perplexity vs. Bit-Depth" analysis to determine the optimal quantization for the R9700.

**Methodology:**
1.  **Reference Run:** Run a 7B model in FP16. Record the "Perplexity" on a standard dataset (e.g., WikiText-103).
2.  **Quantized Runs:** Run the same model in Q8_0, Q6_K, Q4_K_M, and IQ4_XS.
3.  **Error Calculation:**
    $\text{Quantization Error} = \frac{\text{Perplexity}_{\text{Quantized}} - \text{Perplexity}_{\text{FP16}}}{\text{Perplexity}_{\text{FP16}}}$
4.  **Correlation Analysis:** Correlate the "Quantization Error" with the "Pass@1 Rate" in the Coding Workloads.

**Goal:** Determine if there is a "Cliff"—a specific bit-depth where the model's ability to solve coding problems drops precipitously, even if the perplexity only increases slightly.

### Technical Addendum IX: Reporting and Visualization Standards

To ensure the results are actionable, the reporting plane will use the following visualization standards:

**1. The "Quality-Speed" Scatter Plot:**
- **X-Axis:** Tokens Per Second (TPS).
- **Y-Axis:** Average Human Quality Score.
- **Color Coding:** Model Size (e.g., 7B = Blue, 30B = Red).
- **Purpose:** To quickly identify the "Pareto Frontier"—the models that offer the best balance of speed and intelligence.

**2. The "Reasoning Density" Bar Chart:**
- **X-Axis:** Model Name.
- **Y-Axis:** Ratio of (Reasoning Tokens / Output Tokens).
- **Purpose:** To identify models that are "over-thinking" (high ratio) vs. models that are "impulsive" (low ratio).

**3. The "Contextual Drift" Line Graph:**
- **X-Axis:** Token Count (0 to 20,000).
- **Y-Axis:** Anchor Integrity Score (0.0 to 1.0).
- **Purpose:** To visualize the "Collapse Point" for each model during long-form generation.

**4. The "Quantization Heatmap":**
- **Rows:** Workload Type (Coding, RAG, Agentic, Chat).
- **Columns:** Quantization Level (Q8, Q6, Q4, IQ4).
- **Cell Value:** Pass@1 Rate or Success Rate.
- **Color Scale:** Green (High Success) to Red (Low Success).

### Technical Addendum X: Reproducibility and Audit Trail

To ensure that a benchmark run can be replicated by another engineer, the following must be recorded for every single run:

**The "Reproducibility Packet":**
- [ ] **Model Hash:** The SHA-256 hash of the GGUF file.
- [ ] **llama.cpp Version:** The exact git commit or version number.
- [ ] **Seed:** The random seed used for the generation.
- [ ] **Temperature/Top_P/Repeat Penalty:** The exact values used.
- [ ] **System Prompt:** The exact string used (verbatim).
- [ ] **Hardware State:** A screenshot of `nvidia-smi` or `htop` at the start of the run.
- [ ] **Context Window:** The `n_ctx` value.

**Validation:**
- If a result is an outlier (e.g., a 30B model scoring lower than an 8B model), the "Reproducibility Packet" must be audited to find the discrepancy.

### Technical Addendum XI: Final Deployment and Maintenance Strategy

The final goal of this benchmark is to provide a clear roadmap for deploying the best-performing models in a production-like home-lab environment.

**Deployment Recommendations:**
1.  **The "Daily Driver":** The model that achieves the highest "Quality Score" across Chat and RAG workloads with a TPS > 15.
2.  **The "Coding Assistant":** The model with the highest Pass@1 rate on the Coding Workloads, regardless of TPS.
3.  **The "Creative Engine":** The model with the best "Quality Decay" curve in the 20,000-token test.
4.  **The "Agentic Core":** The model with the highest JSON Validity and Tool Accuracy scores.

**Maintenance Schedule:**
- **Monthly Model Updates:** Check for new GGUF quantizations or updated model weights.
- **Weekly Benchmark Runs:** Run a "Sanity Check" suite of 10 prompts to ensure that system updates or hardware changes haven't degraded performance.
- **Quarterly Hardware Audit:** Check the R9700 for thermal wear and ensure the NVMe drive has sufficient space for new models.

**Conclusion of the Master Field Manual:**
By following this rigorous protocol, the benchmark campaign will provide an unparalleled understanding of how open-weight models perform on the R9700 hardware. This data-driven approach eliminates "vibes" and provides a scientific foundation for local AI deployment, ensuring that the home-lab remains a high-performance, reliable environment for innovation, privacy, and high-performance inference.

### Technical Addendum XII: Extended Workload - "The Recursive Expansion"

To further stress the model's ability to maintain coherence, we introduce the "Recursive Expansion" workload. This is a two-stage process designed to test if the model can maintain a "Global Plan" while focusing on "Local Detail."

**Task Design:**
1.  **Stage 1 (The Outline):** Ask the model to generate a 10-point detailed outline for a technical whitepaper on "The Ethics of Autonomous Agentic Systems."
2.  **Stage 2 (The Expansion):** Provide the outline back to the model and ask it to expand "Point 4: The Problem of Recursive Self-Improvement" into a 3,000-token deep dive.

**Prompt Shape:**
`System: You are a technical researcher. Use <thought> tags to plan the structure of each section. User: [Outline from Stage 1] Please expand Point 4 into a comprehensive deep dive.`

**Expected Answer Shape:**
- A 3,000-token technical deep dive that remains perfectly consistent with the points established in the 10-point outline.

**Automated Checks:**
- **Outline Adherence:** Does the expansion contradict any other points in the outline?
- **Detail Density:** Does the expansion provide new information, or does it just repeat the outline point in longer sentences?

**Human Review Rubric:**
- **Consistency:** 1-5 (Does it stay true to the outline?)
- **Depth:** 1-5 (Is the information technically sound?)

**Failure Examples:**
- **Contradiction:** The expansion says "Agents should be restricted," but the outline says "Agents should be unrestricted."
- **Repetition:** The model repeats the outline point over and over without adding new detail.

**Metrics to Graph:**
- **Expansion Fidelity:** % of expansion content that aligns with the original outline.

### Technical Addendum XIII: Advanced Hardware Telemetry - Memory Latency

While TPS is a primary metric, "Memory Latency" is a hidden factor that can affect the "Time to First Token" (TTFT) on the R9700.

**Telemetry Collection:**
- Use `perf` or a similar profiling tool to measure the average memory access latency during the `llama.cpp` inference run.
- **Correlation:** Compare memory latency with the "Reasoning Budget" utilization. If high latency correlates with high reasoning token counts, it suggests that the model's "thinking" process is being bottlenecked by the memory bus.

**Optimization Strategy:**
- If high latency is detected, we will experiment with `n_batch` sizes. Smaller batch sizes can sometimes reduce the pressure on the memory bus at the cost of lower overall TPS.

### Technical Addendum XIV: SQLite Artifact Management - Data Retention

To ensure the SQLite database remains performant as it grows to thousands of entries, we will implement a "Data Retention and Archiving" policy.

**Retention Policy:**
- **Active Database:** Contains the last 100 benchmark runs for each model.
- **Archive Database:** Contains all historical runs, stored in a compressed format.
- **Automated Archiving:** A weekly cron job will move old entries from `results.db` to `archive_results_YYYY_MM_DD.db`.

**Data Integrity:**
- **Checksums:** Each row in the database will have a `checksum` column. This is a hash of the `input_prompt`, `thought_block`, and `final_output`.
- **Validation:** A weekly script will verify that the checksums match the content, ensuring that no data corruption has occurred in the SQLite file.

### Technical Addendum XV: Reporting Plane - The "Model Persona" Radar Chart

For the "Chatbot Workloads," we will use a Radar Chart (Spider Chart) to visualize the model's persona consistency.

**Radar Chart Dimensions:**
1.  **Persona Adherence:** (Human Score)
2.  **Tone Consistency:** (Human Score)
3.  **Helpfulness:** (Human Score)
4.  **Refusal Accuracy:** (Automated Score)
5.  **Nuance Handling:** (Human Score)

**Visualization Goal:**
A "perfect" chatbot model will have a large, symmetrical radar chart. A model that is helpful but loses its persona will have a skewed chart, showing high "Helpfulness" but low "Persona Adherence."

### Technical Addendum XVI: Reproducibility - The "Environment Hash"

To achieve 100% reproducibility, we will implement an "Environment Hash."

**Methodology:**
1.  **System Snapshot:** A script will collect the versions of `python`, `llama.cpp`, `cuda` (or equivalent), and the specific GGUF file.
2.  **Hash Generation:** These versions will be concatenated and hashed (SHA-256).
3.  **Verification:** Every benchmark run will include this hash in the `runs` table.
4.  **Audit:** If two runs with the same "Environment Hash" produce different results, it indicates a non-deterministic factor (e.g., a different random seed or a hardware fluctuation) that must be investigated.

### Technical Addendum XVII: Final Recommendations - The "Home-Lab Tier List"

The final report will categorize models into a "Home-Lab Tier List" based on the benchmark results.

**Tier 1: The "Powerhouse" (30B+ Models)**
- **Use Case:** Complex coding, multi-step agentic planning, and long-form world-building.
- **Recommendation:** Use Q6_K or Q8_0 quantization.

**Tier 2: The "Workhorse" (14B-20B Models)**
- **Use Case:** Daily chat, RAG-based knowledge retrieval, and medium-length creative writing.
- **Recommendation:** Use Q4_K_M or Q5_K_M quantization.

**Tier 3: The "Speedster" (7B-9B Models)**
- **Use Case:** Fast chat, simple scripting, and high-speed summarization.
- **Recommendation:** Use Q8_0 or Q6_K quantization for maximum intelligence at high speed.

**Tier 4: The "Specialist" (Small Models <7B)**
- **Use Case:** Specific tasks like entity extraction or simple classification.
- **Recommendation:** Use Q8_0 for maximum precision.

### Technical Addendum XVIII: Final Campaign Summary and Sign-off

The benchmark campaign is considered complete when:
1.  Every model in the target list has been tested across all 11 workload categories.
2.  The "Long-Output Reliability" test has been completed for each model with a minimum of 20,000 tokens.
3.  The SQLite database has been audited for integrity and PII redaction.
4.  The final "Model Performance Report" has been generated and signed off by the lead benchmark engineer.

**Final Sign-off Checklist:**
- [ ] All hardware telemetry logs are archived.
- [ ] All "Reproducibility Packets" are complete.
- [ ] The "Efficiency Frontier" plot is generated.
- [ ] The "Model Selection Recommendation" is finalized.

**End of Master Field Manual.**