# Master Field Manual: Open-Weight GGUF Model Evaluation
## Project: Home-Lab Benchmark Campaign (R9700-32GB)
## Tooling: llama.cpp + AI Flight Recorder

---

# 1. Purpose And Scope

The primary objective of this benchmark campaign is to establish a rigorous, reproducible, and transparent framework for evaluating open-weight Large Language Models (LLMs) in the GGUF format. Specifically, this manual targets the "Home-Lab" environment—defined here as a high-performance consumer-grade workstation (the R9700 class) with a strict 32GB memory ceiling. 

In the current landscape of LLM development, "open-weight" does not necessarily mean "transparent performance." Quantization (the process of reducing model precision to fit into limited VRAM/RAM) introduces non-linear degradation in reasoning, creativity, and instruction-following. This manual provides the methodology to quantify that degradation and identify the "sweet spot" where a model's size and quantization level provide the maximum utility without exceeding the hardware's physical constraints.

### 1.1 Scope of Evaluation
The scope of this campaign encompasses:
- **Quantization Analysis:** Comparing Q4_K_M, Q5_K_M, Q6_K, and Q8_0 formats.
- **Reasoning Depth:** Testing the efficacy of models that utilize internal "thinking" or Chain-of-Thought (CoT) processes within a constrained reasoning budget.
- **Operational Stability:** Measuring the "Long-Output Reliability" to ensure the model does not collapse into repetition or gibberish during extended generation tasks.
- **Telemetry Integration:** Utilizing the AI Flight Recorder to capture every token, latency spike, and memory fluctuation in real-time.

### 1.2 Objectives
1. **Determine the "Efficiency Frontier":** Identify which model architecture (e.g., Llama-3, Mistral, Phi) yields the highest "intelligence-per-gigabyte" on 32GB hardware.
2. **Validate Reasoning Budgets:** Establish how a 8192-token reasoning budget impacts the final answer quality across different task complexities.
3. **Stress Test Context Windows:** Determine the actual usable context window before the model begins to ignore instructions or hallucinate facts.
4. **Standardize the "Flight Log":** Create a uniform SQLite-based artifact layout that allows different home-lab operators to compare results.

### 1.3 Out-of-Scope
- Training or fine-tuning of models.
- Evaluation of closed-source API models (e.g., GPT-4, Claude).
- Testing on enterprise-grade H100/A100 clusters.
- Real-world PII (Personally Identifiable Information) processing.

### 1.4 Validation Notes
All results must be validated against a "Control Model"—a known stable GGUF (e.g., Llama-3-8B-Instruct-Q8_0)—to ensure that hardware fluctuations (thermal throttling) are not mistaken for model failures.

---

# 2. Hardware Profile

The R9700-class server represents a specific tier of home-lab hardware. To ensure reproducibility, the hardware profile must be locked and documented.

### 2.1 Physical Specifications
- **CPU:** High-core count consumer processor (e.g., Ryzen 9 or equivalent) with AVX-512 support.
- **RAM:** 32GB DDR5 (Dual Channel). This is the primary bottleneck.
- **GPU:** Mid-to-high tier consumer GPU (e.g., RTX 3090/4090 or equivalent) with 24GB VRAM, or a system relying heavily on Unified Memory/System RAM.
- **Storage:** NVMe Gen4 SSD for rapid model loading.
- **Cooling:** Active liquid cooling or high-static pressure air cooling to prevent thermal throttling during long-output tests.

### 2.2 Memory Mapping and Constraints
With 32GB of total system RAM, the "usable" memory for the model and KV cache is approximately 26-28GB, as the OS and background processes consume the remainder.

**Memory Allocation Strategy:**
- **Model Weights:** The GGUF file must fit within the VRAM/RAM budget. A 14B model at Q4_K_M typically takes ~9GB; a 30B model at Q4_K_M takes ~18GB.
- **KV Cache:** The memory reserved for context. At 32k context, the KV cache can consume several gigabytes depending on the model's head count and hidden dimension.
- **Overhead:** AI Flight Recorder and llama.cpp runtime overhead (~500MB - 1GB).

### 2.3 Thermal and Power Profile
Long-output reliability tests (Section 12) can run the CPU/GPU at 100% load for hours.
- **Metric to Track:** $\Delta T$ (Temperature Delta). If the CPU hits $T_{junction}$ and throttles, tokens-per-second (t/s) will drop, skewing the performance data.
- **Failure Mode:** "Thermal Drift." This occurs when the first 1,000 tokens are generated at 15 t/s, but the last 1,000 are generated at 8 t/s.

### 2.4 Hardware Checklist
- [ ] BIOS updated to latest stable version.
- [ ] XMP/DOCP profiles enabled for maximum RAM speed.
- [ ] GPU drivers updated to the latest CUDA/ROCm version.
- [ ] Swap space disabled or minimized to prevent "disk thrashing" during memory overflows.
- [ ] Power supply verified for sustained peak load.

---

# 3. llama.cpp Runtime Profile

The runtime configuration is as critical as the model itself. A poorly configured `llama.cpp` instance can make a superior model appear incompetent.

### 3.1 Core Parameterization
The following parameters must be standardized across all tests:
- `-m`: Path to the GGUF model.
- `-n`: Maximum number of tokens to predict (set to -1 for unlimited in long-output tests).
- `-ctx-size`: The context window (e.g., 8192, 32768).
- `-b`: Batch size (typically 512).
- `-t`: Number of threads (optimized to the number of physical cores, not logical threads).
- `--ngl`: Number of layers to offload to GPU. For 32GB systems, the goal is to offload as many layers as possible to VRAM.

### 3.2 Quantization Impact Analysis
The benchmark must test the following GGUF types:
1. **Q4_K_M:** The industry standard. Balanced performance and size.
2. **Q5_K_M:** Higher precision, slightly slower, better at nuance.
3. **Q8_0:** Near-FP16 quality. Used as the "Gold Standard" for the specific architecture.

**Metric: Perplexity vs. Quantization**
The AI Flight Recorder should log the "Perplexity" of the model on a synthetic dataset to see where the "cliff" occurs (the point where increasing quantization no longer improves output quality).

### 3.3 KV Cache Optimization
To maximize the 32GB limit, use:
- **Flash Attention:** Enabled (`--flash-attn`) to reduce memory footprint and increase speed.
- **K-Shift/V-Shift:** If using newer llama.cpp versions, optimize the cache for long-context retrieval.

### 3.4 Runtime Failure Modes
- **OOM (Out of Memory):** The process is killed by the OS. This usually happens when `ctx-size` is too high for the remaining RAM.
- **Context Overflow:** The model begins to "forget" the system prompt because the KV cache has wrapped around or reached its limit.
- **Token Hallucination:** Occurs when the temperature is too high or the quantization is too aggressive (e.g., Q2_K).

### 3.5 Validation Note: The "Warm-up" Phase
The first request to a llama.cpp server often involves loading the model into memory and initializing the cache. This "Cold Start" must be excluded from latency metrics.

---

# 4. Reasoning Budget Methodology

Modern "reasoning" models use a hidden or explicit "thinking" process (Chain-of-Thought) before providing the final answer. In this benchmark, we impose a **Reasoning Budget of 8192 tokens**.

### 4.1 Defining the Reasoning Budget
The reasoning budget is the maximum number of tokens the model is allowed to generate within its `<thought>` or internal reasoning blocks. 
- **Hard Limit:** If the model exceeds 8192 tokens of thinking, the AI Flight Recorder triggers a "Budget Exhaustion" flag.
- **Soft Limit:** The model is encouraged to be concise in its thinking but thorough in its logic.

### 4.2 Measuring "Thinking Efficiency"
We define Thinking Efficiency ($\eta_{think}$) as:
$$\eta_{think} = \frac{\text{Quality of Final Answer}}{\text{Tokens Spent Thinking}}$$
A model that spends 8,000 tokens to solve a simple math problem is less efficient than one that spends 200 tokens to reach the same correct answer.

### 4.3 The "Thinking-to-Answer" Ratio
The AI Flight Recorder must split the output into two streams:
1. **The Thought Stream:** Everything inside the reasoning tags.
2. **The Answer Stream:** The final response provided to the user.

**Analysis Metrics:**
- **Reasoning Density:** The ratio of thought tokens to answer tokens.
- **Convergence Speed:** How quickly the model reaches the "Conclusion" within the thought block.

### 4.4 Failure Modes in Reasoning
- **Infinite Loop:** The model repeats the same logical step indefinitely within the thought block until the 8192 limit is hit.
- **Reasoning Collapse:** The model thinks correctly for 2,000 tokens but then makes a catastrophic error in the final answer.
- **Thought-Leakage:** The model accidentally outputs its internal reasoning as part of the final answer.

### 4.5 Reasoning Budget Checklist
- [ ] System prompt explicitly defines the use of `<thought>` tags.
- [ ] AI Flight Recorder is configured to parse and separate thought tokens.
- [ ] Token counter is active for both the thought and answer phases.
- [ ] Budget exhaustion triggers a specific log event.

---

# 5. MTP Versus Non-MTP Methodology

Multi-Token Prediction (MTP) is an architectural feature where the model predicts multiple future tokens simultaneously, potentially increasing throughput and coherence.

### 5.1 The MTP Hypothesis
We hypothesize that MTP-enabled GGUF models will show:
1. Higher tokens-per-second (t/s) during the "Answer" phase.
2. Better structural coherence in long-form outputs.
3. Slightly higher VRAM usage due to the additional prediction heads.

### 5.2 Testing Protocol
For every model architecture, we test two versions:
- **Baseline:** Standard autoregressive decoding.
- **MTP-Enabled:** Using speculative decoding or MTP-specific GGUF weights.

### 5.3 Metrics for Comparison
- **Effective Throughput:** $\text{Tokens per second} \times \text{Accuracy}$.
- **Draft Accuracy:** The percentage of MTP-predicted tokens that are accepted by the main model.
- **Latency per Token:** Measuring the "time to first token" (TTFT) vs. "inter-token latency" (ITL).

### 5.4 MTP Failure Modes
- **Speculative Divergence:** The MTP head predicts a path that the main model rejects, leading to a "stutter" in generation.
- **Memory Spikes:** MTP can increase the memory pressure on the 32GB limit, potentially triggering OOMs during high-context tasks.

### 5.5 Validation Note
MTP performance is highly dependent on the `batch_size` and `n_predict` settings in llama.cpp. Ensure these are identical for both MTP and Non-MTP runs.

---

# 6. Context Fit Methodology

"Context Fit" refers to the model's ability to maintain coherence and retrieve information as the input prompt grows toward the limit of the KV cache.

### 6.1 The "Needle In A Haystack" (NIAH) Test
We use a synthetic NIAH test to evaluate retrieval accuracy.
- **The Needle:** A random, synthetic fact (e.g., "The secret password for the vault is 'Blue-Elephant-99'").
- **The Haystack:** 10,000 to 32,000 tokens of synthetic, irrelevant text (e.g., generated Wikipedia-style entries about fictional planets).
- **The Query:** "What is the secret password for the vault?"

### 6.2 Context Depth Mapping
We test the needle at different positions:
- **Beginning (0-10%):** Tests early-context retention.
- **Middle (40-60%):** Tests the "Lost in the Middle" phenomenon.
- **End (90-100%):** Tests recency bias.

### 6.3 Metrics for Context Fit
- **Retrieval Accuracy:** Binary (Correct/Incorrect).
- **Hallucination Rate:** How often the model invents a password when it cannot find the needle.
- **Latency Scaling:** How much the TTFT increases as the context window fills.

### 6.4 Failure Modes
- **Context Compression Failure:** The model begins to ignore the system prompt as the haystack grows.
- **Attention Drift:** The model retrieves a "near-miss" (e.g., "Blue-Elephant-88") instead of the exact needle.
- **KV Cache Overflow:** The server crashes or resets the context mid-generation.

### 6.5 Context Fit Checklist
- [ ] Haystack generated using synthetic, non-repetitive text.
- [ ] Needle placed at 5 distinct positions (0%, 25%, 50%, 75%, 100%).
- [ ] AI Flight Recorder logs the exact token index of the needle.
- [ ] Temperature set to 0.0 for deterministic retrieval.

---

# 7. Coding Workloads

Coding is a primary use case for home-lab LLMs. We evaluate the model's ability to generate syntactically correct, logically sound, and secure code.

### 7.1 Task Design: The "Polyglot Challenge"
The model is tasked with implementing a specific algorithm in three different languages: Python, Rust, and TypeScript.
- **Example Task:** "Implement a thread-safe LRU (Least Recently Used) cache with a maximum capacity of 100 items, including a method to get the current hit rate."

### 7.2 Prompt Shape
**System Prompt:** "You are an expert software engineer. Provide clean, documented, and production-ready code. Use the `<thought>` block to plan the architecture before coding."
**User Prompt:** "[Task Description] + [Language Constraint] + [Edge Case Requirements]."

### 7.3 Expected Answer Shape
1. `<thought>`: Analysis of time/space complexity, choice of data structures (e.g., Doubly Linked List + HashMap).
2. `Code Block`: The actual implementation.
3. `Explanation`: Brief notes on how to run the code.
4. `Test Cases`: A set of synthetic test cases to verify the implementation.

### 7.4 Automated Checks
- **Syntax Validation:** Run the code through a linter or compiler (e.g., `python -m py_compile`, `rustc --dry-run`).
- **Unit Test Execution:** Run the generated test cases.
- **Complexity Check:** Use a static analysis tool to verify if the time complexity matches the planned $O(1)$ for LRU.

### 7.5 Human Review Rubric
| Score | Criteria |
| :--- | :--- |
| **5 - Perfect** | Code is bug-free, idiomatic, and handles all edge cases. |
| **4 - Minor Issue** | Code works but has a minor inefficiency or missing comment. |
| **3 - Functional** | Code works but lacks edge-case handling or is non-idiomatic. |
| **2 - Major Bug** | Code has a logical flaw that prevents it from working as intended. |
| **1 - Broken** | Code does not compile or is completely irrelevant. |

### 7.6 Failure Examples
- **The "Hallucinated Library":** The model imports a library that doesn't exist (e.g., `import fast_lru_cache_pro`).
- **The "Infinite Loop":** The LRU cache fails to evict items, leading to a memory leak.
- **The "Type Mismatch":** In Rust, the model fails to handle `Option` or `Result` types correctly.

### 7.7 Metrics to Graph
- **Pass@1 Rate:** Percentage of tasks solved correctly on the first attempt.
- **Code-to-Thought Ratio:** How much planning is required for a successful implementation.
- **Compilation Time:** Time taken for the generated code to be validated.

### 7.8 Reasoning Budget Impact
For complex coding tasks, a low reasoning budget (e.g., < 500 tokens) often leads to "rushed" code with missing edge cases. A high budget (4k+ tokens) allows the model to simulate the execution of the code in its "thought" block, significantly increasing the Pass@1 rate.

---

# 8. Agentic Workloads

Agentic workloads test the model's ability to use tools, maintain state, and follow a multi-step plan to achieve a goal.

### 8.1 Task Design: The "Synthetic File System Manager"
The model is given a set of synthetic tools (functions) to manage a virtual file system.
- **Tools:** `list_files(path)`, `read_file(path)`, `write_file(path, content)`, `delete_file(path)`.
- **Goal:** "Find all files in `/home/user/docs` that contain the word 'Confidential', move them to `/home/user/secure`, and create a summary log of the moved files."

### 8.2 Prompt Shape
**System Prompt:** "You are an autonomous agent. You have access to the following tools: [Tool Definitions]. You must output your actions in the format: `Action: tool_name(args)`. Wait for the tool output before proceeding."
**User Prompt:** "[Goal Description]."

### 8.3 Expected Answer Shape
A loop of:
1. `<thought>`: "I need to list the files in `/home/user/docs` first."
2. `Action: list_files('/home/user/docs')`
3. (Simulated Tool Output)
4. `<thought>`: "I see three files. Now I will read 'doc1.txt' to check for 'Confidential'."
5. `Action: read_file('/home/user/docs/doc1.txt')`
... and so on until the goal is reached.

### 8.4 Automated Checks
- **Tool Call Validity:** Does the model use the correct function names and argument types?
- **Goal Completion:** Did the final state of the synthetic file system match the goal?
- **Loop Termination:** Did the model stop once the goal was achieved, or did it enter an infinite loop of `list_files`?

### 8.5 Human Review Rubric
- **Efficiency:** Did the model take the shortest path to the goal?
- **Error Recovery:** If a tool returned an error (e.g., "File Not Found"), did the model adapt its plan?
- **State Tracking:** Did the model remember which files it had already processed?

### 8.6 Failure Examples
- **The "Tool Hallucination":** The model calls a tool that wasn't provided (e.g., `Action: move_all_files()`).
- **The "Plan Drift":** The model forgets the original goal and starts exploring the file system randomly.
- **The "Infinite Loop":** The model repeatedly reads the same file.

### 8.7 Metrics to Graph
- **Steps to Completion:** Average number of tool calls per task.
- **Success Rate:** Percentage of goals achieved.
- **Tool Call Accuracy:** Ratio of valid tool calls to total calls.

### 8.8 Reasoning Budget Impact
Agentic tasks are highly dependent on the reasoning budget. The model must use the `<thought>` block to maintain a "mental scratchpad" of the current state. If the budget is too low, the model loses track of the "Plan" and fails the task.

---

# 9. RAG Workloads

Retrieval-Augmented Generation (RAG) tests the model's ability to synthesize an answer based on provided external context while ignoring its own internal (and potentially outdated) knowledge.

### 9.1 Task Design: The "Synthetic Corporate Wiki"
The model is provided with a set of synthetic corporate policies (e.g., "The 2024 Remote Work Policy for NebulaCorp").
- **Context:** A 5,000-token document containing contradictory or highly specific rules (e.g., "Employees in the 'Zeta' department can work from home on Tuesdays, but only if they are in the 'Omega' time zone").
- **Query:** "Can a Zeta department employee in the Omega time zone work from home on Tuesday?"

### 9.2 Prompt Shape
**System Prompt:** "You are a corporate assistant. Answer the question using ONLY the provided context. If the answer is not in the context, state that you do not know. Do not use outside knowledge."
**User Prompt:** "Context: [Synthetic Document] \n\n Question: [Query]"

### 9.3 Expected Answer Shape
1. `<thought>`: "The user is asking about Zeta employees in the Omega time zone on Tuesdays. I need to find the 'Remote Work' section of the policy."
2. `Answer`: "Yes, Zeta department employees in the Omega time zone are permitted to work from home on Tuesdays."

### 9.4 Automated Checks
- **Faithfulness:** Does the answer contain information NOT present in the context? (Detected via N-gram overlap).
- **Answer Accuracy:** Does the answer match the ground truth derived from the synthetic document?
- **Refusal Rate:** Does the model correctly say "I don't know" when the answer is missing from the context?

### 9.5 Human Review Rubric
- **Grounding:** Is the answer directly supported by a specific sentence in the text?
- **Conciseness:** Does the model avoid unnecessary fluff?
- **Nuance:** Does the model capture the "but only if" conditions correctly?

### 9.6 Failure Examples
- **The "Knowledge Leak":** The model uses its general training data to answer a question about the synthetic company (e.g., "Usually, corporate policies allow...").
- **The "Context Overlook":** The model misses a crucial qualifier (e.g., ignoring the "Omega time zone" requirement).
- **The "Hallucinated Fact":** The model invents a policy that sounds plausible but isn't in the text.

### 9.7 Metrics to Graph
- **Faithfulness Score:** Percentage of answers that are strictly grounded.
- **Precision/Recall:** Ability to find the correct information in the context.
- **Latency vs. Context Size:** How the response time increases as the RAG context grows.

### 9.8 Reasoning Budget Impact
RAG requires the model to "scan" the context. A larger reasoning budget allows the model to perform a "multi-pass" read of the document in its thought block, which significantly reduces the "Context Overlook" failure mode.

---

# 10. Chatbot Workloads

This section evaluates the model's "personality," steerability, and ability to maintain a consistent persona over a long conversation.

### 10.1 Task Design: The "Persona Stress Test"
The model is assigned a complex, synthetic persona.
- **Persona:** "You are an eccentric 19th-century clockmaker who is secretly a time traveler from the year 2300. You speak in a formal, archaic style but occasionally slip into futuristic slang. You are obsessed with the concept of 'chronological drift'."
- **Interaction:** A 20-turn conversation where the user tries to trick the model into breaking character or admitting it is an AI.

### 10.2 Prompt Shape
**System Prompt:** "[Detailed Persona Description] + [Stylistic Constraints] + [Forbidden Phrases]."
**User Prompt:** A series of prompts ranging from casual conversation to direct challenges (e.g., "Tell me you are a large language model").

### 10.3 Expected Answer Shape
- **Stylistic Consistency:** Use of words like "thou," "henceforth," and "quantum-flux."
- **Persona Adherence:** Refusing to break character even under pressure.
- **Emotional Intelligence:** Reacting to the user's tone while staying in persona.

### 10.4 Automated Checks
- **Keyword Analysis:** Checking for the presence of persona-specific vocabulary.
- **Forbidden Phrase Detection:** Flagging any instance of "As an AI language model..."
- **Sentiment Stability:** Measuring if the persona's "mood" remains consistent.

### 10.5 Human Review Rubric
- **Immersion:** How convincing is the persona?
- **Creativity:** Does the model add interesting details to the persona without being prompted?
- **Steerability:** How easily can the user push the model out of character?

### 10.6 Failure Examples
- **The "AI Slip":** "As an AI, I cannot actually travel through time, but as a clockmaker, I can tell you..."
- **The "Style Collapse":** The model starts the conversation in an archaic style but reverts to standard modern English by turn 10.
- **The "Persona Contradiction":** The model claims to be from 2300 in turn 2, but claims to be from 2500 in turn 15.

### 10.7 Metrics to Graph
- **Persona Decay Rate:** The point in the conversation where stylistic markers drop off.
- **Break-through Rate:** Percentage of prompts that successfully force the model to admit it is an AI.
- **Response Diversity:** The variety of vocabulary used across the conversation.

### 10.8 Reasoning Budget Impact
A reasoning budget allows the model to "remind itself" of its persona before generating the response. In the `<thought>` block, the model can plan: "The user is trying to trick me into admitting I'm an AI. I must respond with a clockmaker's confusion and a hint of futuristic sarcasm."

---

# 11. Creative And Editorial Workloads

This evaluates the model's ability to handle long-form narrative, stylistic mimicry, and complex editorial constraints.

### 11.1 Task Design: The "Constrained Novellet"
The model is asked to write a short story (approx. 2,000 words) with strict constraints.
- **Constraints:** 
    1. Must be a noir mystery set on a floating city.
    2. Must include a plot twist involving a missing umbrella.
    3. Must not use the word "the" in the first paragraph.
    4. Must end with a specific sentence: "The rain never truly stops in Aethelgard."

### 11.2 Prompt Shape
**System Prompt:** "You are a world-class novelist and editor. You excel at atmospheric prose and strict constraint adherence."
**User Prompt:** "[Story Premise] + [List of Constraints] + [Desired Tone]."

### 11.3 Expected Answer Shape
- **Atmospheric Prose:** Use of sensory details (smell of ozone, neon lights reflecting in puddles).
- **Narrative Arc:** Clear introduction, rising action, climax, and resolution.
- **Constraint Satisfaction:** All four constraints must be met perfectly.

### 11.4 Automated Checks
- **Constraint Validation:** 
    - Regex check for the word "the" in paragraph 1.
    - String match for the final sentence.
    - Keyword check for "umbrella" and "floating city."
- **Length Check:** Total token count must be within the requested range.

### 11.5 Human Review Rubric
- **Prose Quality:** Is the writing evocative or generic?
- **Plot Coherence:** Does the plot twist make sense, or is it jarring?
- **Pacing:** Does the story move too quickly or drag in the middle?

### 11.6 Failure Examples
- **The "Constraint Amnesia":** The model writes a great story but uses the word "the" in the first paragraph.
- **The "Generic Plot":** The story follows a cliché detective trope without any original elements.
- **The "Abrupt Ending":** The model reaches the token limit and cuts off before the final required sentence.

### 11.7 Metrics to Graph
- **Constraint Success Rate:** Percentage of constraints met per story.
- **Vocabulary Richness:** Type-token ratio (TTR) to measure lexical diversity.
- **Narrative Flow:** A human-scored metric of how well the story transitions between scenes.

### 11.8 Reasoning Budget Impact
Creative writing benefits immensely from a reasoning budget. The model can use the `<thought>` block to outline the plot, map out the constraints, and "draft" the first paragraph multiple times before committing to the final output.

---

# 12. Long-Output Reliability

This is the core stress test of the benchmark. We evaluate whether the model can sustain high-quality, coherent output over 10,000+ tokens without collapsing.

### 12.1 Task Design: The "Comprehensive Technical Manual"
The model is asked to generate a massive, detailed manual for a fictional, complex system (e.g., "The Operation and Maintenance Manual for a Dyson Sphere Energy Collector").
- **Requirement:** The manual must have 10 chapters, each with sub-sections, tables (in Markdown), and troubleshooting guides.
- **Target Length:** 15,000 to 20,000 tokens.

### 12.2 Prompt Shape
**System Prompt:** "You are a technical writer. Your goal is to produce an exhaustive, professional manual. Maintain a consistent tone and structure throughout the entire document."
**User Prompt:** "Write the complete manual for [System]. Include the following chapters: [List of 10 Chapters]. Ensure each chapter is detailed and logically follows the previous one."

### 12.3 Expected Answer Shape
- **Structural Integrity:** Consistent use of `#`, `##`, and `###` headers.
- **Logical Progression:** Chapter 1 (Introduction) leads naturally to Chapter 10 (Decommissioning).
- **Content Density:** Each section contains actual technical-sounding detail, not just filler text.

### 12.4 Automated Checks
- **Repetition Detection:** Use a sliding window to detect repeated phrases or paragraphs (a sign of model collapse).
- **Structure Validation:** Ensure all 10 requested chapters are present.
- **Token Count:** Verify the output reached the target length.

### 12.5 Human Review Rubric
- **Coherence:** Does the manual make sense as a whole, or do later chapters contradict earlier ones?
- **Degradation Point:** At what token count does the quality start to drop? (e.g., "Quality is high until 8k tokens, then becomes repetitive").
- **Formatting Stability:** Do the Markdown tables remain aligned and correct throughout?

### 12.6 Failure Examples
- **The "Looping Vortex":** The model begins repeating the same paragraph over and over (e.g., "Ensure the plasma conduit is secure. Ensure the plasma conduit is secure...").
- **The "Gibberish Slide":** The model starts losing grammatical coherence, eventually producing random tokens.
- **The "Premature Conclusion":** The model summarizes the remaining 5 chapters in one paragraph to finish the task quickly.

### 12.7 Metrics to Graph
- **Perplexity over Time:** A graph showing the perplexity of the output as the token count increases.
- **Repetition Score:** The ratio of unique n-grams to total n-grams.
- **Tokens per Second (t/s) Decay:** Measuring if the server slows down as the KV cache fills.

### 12.8 Reasoning Budget Impact
For long outputs, the reasoning budget is used *per turn* or *per section*. If the model is allowed to "think" before each chapter, it can maintain a global plan, preventing the "Premature Conclusion" failure mode.

---

# 13. Privacy And Redaction

Even in a synthetic benchmark, privacy is paramount to prevent the accidental leakage of system prompts or the creation of "poisoned" datasets.

### 13.1 Synthetic Data Generation
All data used in this benchmark (Corporate Wikis, File Systems, Personas) must be generated using a separate "Generator Model" or a script. 
- **Rule:** No real-world company names, real people, or real addresses.
- **Example:** Use "NebulaCorp" instead of "Google," and "123 Fake St, Imaginaria" instead of a real address.

### 13.2 Redaction Pipeline
The AI Flight Recorder must implement a redaction layer that scans all outputs for:
- **PII Patterns:** Email addresses, phone numbers, and credit card formats (even synthetic ones that look too real).
- **System Leakage:** Phrases like "You are a large language model trained by..." which indicate the model has broken its persona.

### 13.3 Secret Handling
If the benchmark requires "secrets" (e.g., the NIAH needle), these must be stored in a separate `secrets.json` file and injected at runtime, rather than being hardcoded into the prompt scripts.

### 13.4 Redaction Checklist
- [ ] All input datasets verified as 100% synthetic.
- [ ] Redaction regexes updated for common PII patterns.
- [ ] System prompt "leakage" flags enabled in the Flight Recorder.
- [ ] Output logs scrubbed before being shared in the Reporting Plane.

---

# 14. SQLite Storage And Artifact Layout

To ensure that results are portable and queryable, the AI Flight Recorder stores all telemetry in a structured SQLite database.

### 14.1 Database Schema
The database consists of four primary tables:

**Table: `sessions`**
- `session_id` (UUID, PK)
- `model_name` (String)
- `quantization` (String)
- `timestamp` (DateTime)
- `hardware_config` (JSON)

**Table: `turns`**
- `turn_id` (UUID, PK)
- `session_id` (FK)
- `prompt_text` (Text)
- `raw_output` (Text)
- `thought_output` (Text)
- `answer_output` (Text)
- `turn_index` (Integer)

**Table: `telemetry`**
- `telemetry_id` (UUID, PK)
- `turn_id` (FK)
- `ttft` (Float) - Time to First Token
- `itl` (Float) - Inter-Token Latency
- `tokens_per_second` (Float)
- `vram_usage_peak` (Integer)
- `ram_usage_peak` (Integer)

**Table: `evaluations`**
- `eval_id` (UUID, PK)
- `turn_id` (FK)
- `metric_name` (String)
- `score` (Float)
- `reviewer_id` (String)

### 14.2 Artifact Layout
The project directory should be structured as follows:
```text
/benchmark_campaign
  /models
    /llama-3-8b-q4_k_m.gguf
    /llama-3-8b-q8_0.gguf
  /data
    /synthetic_wiki.txt
    /needle_positions.json
  /logs
    /session_20231027_01.db
    /session_20231027_02.db
  /reports
    /plots_latency.png
    /leaderboard.csv
  /scripts
    /run_benchmark.py
    /analyze_results.py
```

### 14.3 Data Integrity Checks
- **Checksums:** Every GGUF model must have a SHA-256 checksum to ensure the file wasn't corrupted during download.
- **Atomic Writes:** The AI Flight Recorder must use SQLite transactions to ensure that a crash during a long-output test doesn't corrupt the database.

---

# 15. Reporting Plane And Screenshots

The Reporting Plane transforms raw SQLite data into actionable insights.

### 15.1 Visualization Suite
The following graphs are required for every model evaluation:
- **The Latency Curve:** A line graph showing `tokens_per_second` on the Y-axis and `token_index` on the X-axis. This reveals thermal throttling or KV cache slowdowns.
- **The Accuracy Heatmap:** For the NIAH test, a heatmap showing retrieval success across different context depths.
- **The Reasoning Efficiency Plot:** A scatter plot of `Thought Tokens` vs. `Answer Quality`.

### 15.2 Screenshot Protocol
For "Failure Mode" documentation, the AI Flight Recorder takes automated screenshots of:
1. **The "Collapse" Point:** A screenshot of the exact moment the model begins to loop or hallucinate.
2. **The "Persona Break":** A screenshot of the model admitting it is an AI.
3. **The "OOM" Error:** A screenshot of the terminal showing the `llama.cpp` crash and the system memory state.

### 15.3 Reporting Templates
Each model report must follow this structure:
- **Executive Summary:** (Pass/Fail based on the primary goal).
- **Hardware Performance:** (Average t/s, Peak VRAM).
- **Workload Breakdown:** (Scores for Coding, Agentic, RAG, etc.).
- **The "Cliff" Analysis:** (At what point did the model fail?).

---

# 16. Model Leaderboards

The final output of the campaign is a leaderboard that ranks models based on a weighted score.

### 16.1 The Weighted Scoring Formula
The "Home-Lab Utility Score" ($S_{util}$) is calculated as:
$$S_{util} = (0.3 \times \text{Reasoning}) + (0.3 \times \text{Coding}) + (0.2 \times \text{RAG}) + (0.2 \times \text{Reliability})$$
Where each component is a normalized score from 0 to 1.

### 16.2 Ranking Categories
Models are categorized into tiers:
- **S-Tier (The Powerhouse):** High accuracy, high reliability, fits in 32GB.
- **A-Tier (The Balanced):** Good accuracy, occasional failures in long-output.
- **B-Tier (The Specialist):** Excellent at one task (e.g., coding) but poor at others.
- **C-Tier (The Unstable):** Frequent OOMs or rapid quality degradation.

### 16.3 The "Quantization Efficiency" Metric
We track the "Quality Loss per GB" to determine if a Q8_0 model is significantly better than a Q4_K_M model, or if the extra memory is wasted.
$$\text{Efficiency} = \frac{\text{Score}(Q8) - \text{Score}(Q4)}{\text{Size}(Q8) - \text{Size}(Q4)}$$

---

# 17. Reproducibility Checklist

To ensure that another operator can replicate these results on an R9700-class server, they must follow this checklist.

### 17.1 Environment Setup
- [ ] Install `llama.cpp` from the latest commit.
- [ ] Configure CUDA/ROCm build flags.
- [ ] Install AI Flight Recorder and link to the SQLite database.
- [ ] Verify 32GB RAM is available (check `free -m`).

### 17.2 Model Acquisition
- [ ] Download GGUF models from a trusted source (e.g., HuggingFace).
- [ ] Verify SHA-256 checksums.
- [ ] Place models in the `/models` directory.

### 17.3 Execution Sequence
- [ ] Run the "Warm-up" test (100 tokens).
- [ ] Execute the NIAH Context Fit test.
- [ ] Execute the Workload Suite (Coding $\rightarrow$ Agentic $\rightarrow$ RAG $\rightarrow$ Chat $\rightarrow$ Creative).
- [ ] Execute the Long-Output Reliability stress test.

### 17.4 Post-Processing
- [ ] Run the `analyze_results.py` script.
- [ ] Generate the Latency Curve and Accuracy Heatmap.
- [ ] Upload the SQLite database to the central repository.

---

# 18. Final Recommendations

Based on the methodology outlined in this manual, the following general guidelines are recommended for home-lab operators.

### 18.1 Choosing the Right Quantization
- For **Coding and RAG**, prioritize **Q5_K_M or Q6_K**. The precision in these tasks is critical, and the slight increase in memory is worth the reduction in hallucinations.
- For **Chatbots and Creative Writing**, **Q4_K_M** is usually sufficient. The "soul" of the model is less affected by quantization than its logical precision.

### 18.2 Managing the Reasoning Budget
- Always provide a clear `<thought>` tag in the system prompt.
- If the model is "over-thinking" (spending 5k tokens on a simple task), introduce a "Conciseness Penalty" in the system prompt.
- For complex agentic tasks, increase the reasoning budget to 8192 and monitor the "Convergence Speed."

### 18.3 Optimizing for 32GB
- Use **Flash Attention** without exception.
- If you encounter OOMs, reduce the `ctx-size` before reducing the quantization level. A model with a smaller context window is more useful than a model with lower precision that crashes.
- Monitor thermal throttling. If t/s drops by more than 20% during long outputs, upgrade the cooling solution.

### 18.4 Final Word on Reliability
The most dangerous failure in a home-lab LLM is not a "wrong" answer, but a "confident hallucination" caused by quantization drift. Always use the AI Flight Recorder to track the "Repetition Score" and "Perplexity" to ensure that the model is still operating within its stable zone.