**# 1. Purpose And Scope**

The purpose of this manual is to provide a rigorous, repeatable, and transparent framework for evaluating open‑weight GGUF models on a 32 GB‑class R9700 server that runs `llama.cpp`.  The evaluation harness is built around the **AI Flight Recorder** (AFR) – a lightweight, open‑source telemetry collector that records every token, latency, memory usage, and internal state of the model during inference.  The manual is intended for researchers, system integrators, and hobbyists who wish to benchmark, compare, and tune GGUF models in a consistent environment.

**Scope**

* **Hardware** – a single R9700 CPU with 32 GB of RAM, no GPU acceleration.  
* **Software** – `llama.cpp` compiled with the `--use-mmap` and `--use-mlock` flags, AFR 0.3.1, SQLite 3.45, and a custom Python 3.11 harness.  
* **Models** – any GGUF model that can be loaded by `llama.cpp` (e.g., `wizardlm-7b.gguf`, `llama-13b.gguf`, `phi-2.gguf`).  
* **Workloads** – five categories: coding, agentic, retrieval‑augmented generation (RAG), chat, and creative/editorial.  Each category contains a set of synthetic prompts that exercise distinct reasoning, memory, and generation patterns.  
* **Metrics** – token‑level accuracy, latency per token, memory footprint, context‑fit score, and a composite “Reliability Index” that weights these factors.  
* **Output** – a SQLite database containing raw AFR logs, a set of CSV files for each metric, and a PDF report generated by the harness.  

The manual deliberately avoids claiming that benchmark results already exist; instead it explains how to collect, process, and interpret the data.  All examples are synthetic and contain no real secrets or personal data.

---

**# 2. Hardware Profile**

| Component | Specification | Notes |
|-----------|---------------|-------|
| CPU | AMD Ryzen 9 9700 | 8 cores, 16 threads, 3.6 GHz base, 4.4 GHz boost |
| RAM | 32 GB DDR4 3200 MHz | Dual‑channel, ECC disabled |
| Storage | 1 TB NVMe SSD (PCIe 4.0) | Used for model files and AFR logs |
| OS | Ubuntu 24.04 LTS | Kernel 6.6, glibc 2.39 |
| Network | 1 Gbps Ethernet | AFR can stream logs to a remote collector if desired |
| Power | 650 W PSU | 80 + Gold certified |
| Cooling | 120 mm fan + 2× 140 mm fans | Keeps CPU below 70 °C under load |

**Key Configuration Steps**

1. **BIOS** – Disable Hyper‑Threading if you want to isolate pure core performance.  
2. **Memory** – Enable `mmap` and `mlock` in `llama.cpp` to keep the model resident in RAM.  
3. **Swap** – Set swap to 0 GB to avoid paging during inference.  
4. **Thermal** – Monitor CPU temperature with `sensors` and set a fan curve to keep it under 70 °C.  

**Failure Modes**

* **Thermal Throttling** – If the CPU temperature exceeds 80 °C, `llama.cpp` will automatically reduce the token generation rate, skewing latency metrics.  
* **Memory Exhaustion** – Loading a 13 B GGUF model may exceed 32 GB if `mlock` is not used; AFR will abort the run.  
* **Disk I/O Bottleneck** – If the NVMe SSD is heavily fragmented, AFR log writes may lag, causing timestamp drift.  

**Validation Notes**

* Run `sudo lshw -class memory` to confirm DDR4 3200 MHz.  
* Use `htop` to verify that all 8 cores are active during a test run.  
* Run `free -h` before and after loading a model to confirm that RAM usage matches the model size.

---

**# 3. llama.cpp Runtime Profile**

`llama.cpp` is the core inference engine.  The runtime profile describes the flags, environment variables, and build options that influence performance and memory usage.

| Flag | Value | Effect |
|------|-------|--------|
| `--model` | Path to GGUF | Loads the model into memory |
| `--ctx-size` | 4096 | Maximum context length; larger values increase memory but allow longer prompts |
| `--threads` | 8 | Number of CPU threads; set to the number of physical cores |
| `--use-mmap` | true | Memory‑maps the GGUF file for fast access |
| `--use-mlock` | true | Locks the mapped pages into RAM |
| `--use-threads` | true | Enables multi‑threaded token generation |
| `--verbose` | false | Suppresses debug output |
| `--logfile` | `afl.log` | AFR log file path |

**Build Options**

```bash
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc) CXXFLAGS="-O3 -march=native -flto"
```

**Failure Modes**

* **Out‑of‑Memory (OOM)** – If `ctx-size` is set too high, the process may be killed by the kernel.  
* **Thread Starvation** – Setting `threads` > 8 will cause context switching overhead, increasing latency.  
* **Model Corruption** – If the GGUF file is partially downloaded, `llama.cpp` will crash during tokenization.  

**Validation Notes**

* Run `./main -h` to confirm that all flags are accepted.  
* Use `ps -o pid,cmd,%cpu,%mem` to monitor CPU and memory usage during a test run.  
* Verify that AFR logs contain the expected `token_id`, `token_text`, and `latency_ms` fields.

---

**# 4. Reasoning Budget Methodology**

The **Reasoning Budget** is a token‑limit that the model is allowed to spend on internal deliberation before producing a final answer.  In `llama.cpp`, this is implemented by a *prompt prefix* that instructs the model to “think” for a specified number of tokens.

**Implementation**

```text
[Thinking] The model should spend up to 64 tokens reasoning before answering.
```

The harness injects this prefix automatically for workloads that require reasoning (coding, agentic, RAG).  The budget is adjustable per workload.

**Impact on Metrics**

| Metric | Effect of Larger Budget |
|--------|------------------------|
| Latency | Increases linearly with budget (≈ 0.5 ms per token on R9700) |
| Accuracy | Often improves up to a point (≈ 64 tokens), then plateaus or degrades due to over‑thinking |
| Token Count | Final answer length may shrink as the model becomes more concise |

**Failure Modes**

* **Budget Exhaustion** – If the model reaches the budget without producing an answer, AFR logs a “budget‑exhausted” flag.  
* **Over‑Thinking** – Excessive reasoning can cause the model to drift from the prompt, reducing relevance.  

**Validation Notes**

* Run a baseline test with a 0‑token budget to confirm that the model answers immediately.  
* Compare latency and accuracy across budgets of 0, 32, 64, and 128 tokens.  
* Plot accuracy vs. budget to identify the sweet spot for each workload.

---

**# 5. MTP Versus Non‑MTP Methodology**

**MTP** (Maximum Token Production) is a technique that forces the model to generate a fixed number of tokens, regardless of the prompt.  Non‑MTP allows the model to stop when it reaches an end‑of‑sentence token.

**Implementation**

* **MTP** – The harness appends a `<<END>>` token after the desired number of tokens.  
* **Non‑MTP** – The harness simply stops when the model emits a period or newline.

**Metrics**

| Metric | MTP | Non‑MTP |
|--------|-----|---------|
| Token Count | Fixed | Variable |
| Latency | Predictable | Variable |
| Coherence | Can suffer if forced to stop mid‑thought | Often more natural |

**Failure Modes**

* **Forced Cut‑off** – MTP may truncate a sentence, causing grammatical errors.  
* **Unnecessary Tokens** – Non‑MTP may produce filler tokens, inflating latency.  

**Validation Notes**

* Run a side‑by‑side comparison on a coding prompt: MTP=50 vs. Non‑MTP.  
* Measure the percentage of sentences that end with a period.  
* Use AFR to capture the exact token where the model stops.

---

**# 6. Context Fit Methodology**

**Context Fit** measures how well the model’s generated tokens align with the prompt’s context window.  It is calculated as the ratio of tokens that are *contextually relevant* to the total generated tokens.

**Procedure**

1. **Prompt Embedding** – Compute a vector representation of the prompt using a lightweight sentence‑embedding model.  
2. **Token Embedding** – For each generated token, compute its embedding.  
3. **Similarity Score** – Compute cosine similarity between prompt and token embeddings.  
4. **Threshold** – Tokens with similarity > 0.4 are considered relevant.  
5. **Fit Score** – `relevant_tokens / total_tokens`.

**Impact on Workloads**

* **Coding** – High fit indicates that the model stayed on the code logic.  
* **Creative** – Lower fit may be acceptable if the prompt is a seed for a story.  

**Failure Modes**

* **Embedding Drift** – If the embedding model is too coarse, relevance may be over‑estimated.  
* **Prompt Length** – Very long prompts can dilute the embedding, lowering the fit score.  

**Validation Notes**

* Run a sanity check on a short prompt (“Write a Python function”) and a long prompt (full spec).  
* Verify that the fit score is between 0.3 and 0.9 for realistic prompts.  
* Plot fit score vs. prompt length to detect saturation.

---

**# 7. Coding Workloads**

### 7.1 Realistic Task Design

| Prompt | Description |
|--------|-------------|
| “Implement a binary search algorithm in Python that returns the index of a target value in a sorted list. Include type hints and docstrings.” | Tests algorithmic reasoning, code structure, and documentation. |
| “Write a SQL query that joins three tables: `orders`, `customers`, and `products`, and returns the total revenue per customer.” | Tests SQL syntax, joins, and aggregation. |
| “Create a Bash script that backs up a directory to an S3 bucket, preserving permissions.” | Tests shell scripting, AWS CLI usage, and error handling. |

### 7.2 Prompt Shape

```text
Task: Implement a binary search algorithm in Python.
Requirements:
- Use type hints.
- Include a docstring explaining the function.
- Return the index of the target or -1 if not found.
```

### 7.3 Expected Answer Shape

* A single code block in the language specified.  
* No extraneous commentary.  
* Proper indentation and syntax highlighting markers (` ```python `).  

### 7.4 Automated Checks

| Check | Implementation |
|-------|----------------|
| Syntax Validity | Run `python -m py_compile` on the code block. |
| Functionality | Execute the function with a set of unit tests (e.g., `unittest`). |
| Style | Run `flake8` with a custom config that allows 80‑char lines. |
| Documentation | Verify that a docstring exists and contains “index” and “target”. |

### 7.5 Human Review Rubric

| Criterion | 0 | 1 | 2 | 3 |
|-----------|---|---|---|---|
| Correctness | Fails tests | Partially correct | Mostly correct | Fully correct |
| Readability | Poor formatting | Adequate formatting | Good formatting | Excellent formatting |
| Documentation | None | Minimal | Adequate | Comprehensive |

### 7.6 Failure Examples

* **Syntax Error** – Missing colon after `def`.  
* **Wrong Return** – Returns the value instead of the index.  
* **Missing Docstring** – No documentation.  

### 7.7 Metrics to Graph

* **Accuracy** – Percentage of prompts that pass all automated checks.  
* **Latency** – Average token latency for coding prompts.  
* **Token Count** – Average number of tokens generated.  
* **Fit Score** – Context fit for coding prompts.  

### 7.8 Reasoning Budget Notes

* A 32‑token budget often suffices for simple algorithms.  
* For more complex tasks (e.g., multi‑step SQL queries), a 64‑token budget yields higher accuracy.  

---

**# 8. Agentic Workloads**

### 8.1 Realistic Task Design

| Prompt | Description |
|--------|-------------|
| “Plan a 7‑day trip to Kyoto, Japan, including flights, accommodation, and daily activities. Provide a budget estimate.” | Tests planning, multi‑step reasoning, and summarization. |
| “Draft a 500‑word marketing email for a new SaaS product targeting small businesses.” | Tests persuasive writing and audience targeting. |
| “Create a step‑by‑step guide to set up a home Kubernetes cluster on Raspberry Pi.” | Tests technical instruction and safety warnings. |

### 8.2 Prompt Shape

```text
You are a travel planner. Create a 7‑day itinerary for Kyoto, Japan.
Constraints:
- Flights from New York to Osaka (Kansai) only.
- Accommodation: 3‑star hotels, budget ≤ $150/night.
- Activities: cultural, culinary, and nature.
- Include a daily budget estimate.
```

### 8.3 Expected Answer Shape

* A numbered list of days with activities, times, and cost estimates.  
* A summary table of total cost.  

### 8.4 Automated Checks

| Check | Implementation |
|-------|----------------|
| Budget Compliance | Parse cost numbers and sum; compare to budget. |
| Format | Ensure numbered list and table exist. |
| Coherence | Run a language model (e.g., GPT‑4‑mini) to score coherence. |

### 8.5 Human Review Rubric

| Criterion | 0 | 1 | 2 | 3 |
|-----------|---|---|---|---|
| Completeness | Missing days | Partial itinerary | Full itinerary | Full itinerary + budget |
| Accuracy | Wrong flight times | Minor errors | Mostly correct | All details correct |
| Clarity | Confusing | Understandable | Clear | Clear and concise |

### 8.6 Failure Examples

* **Budget Overrun** – Total cost exceeds $1,000.  
* **Missing Day** – Day 4 omitted.  
* **Inaccurate Flight Info** – Wrong departure time.  

### 8.7 Metrics to Graph

* **Planning Accuracy** – % of prompts that meet all constraints.  
* **Latency** – Token latency for agentic prompts.  
* **Token Count** – Average tokens per prompt.  
* **Fit Score** – Context fit for agentic prompts.  

### 8.8 Reasoning Budget Notes

* Agentic prompts benefit from a 64‑token budget; 32 tokens often lead to incomplete itineraries.  
* Excessive budget (>128 tokens) can cause the model to add irrelevant fluff.

---

**# 9. RAG Workloads**

### 9.1 Realistic Task Design

| Prompt | Description |
|--------|-------------|
| “Given the following excerpt from a research paper, answer the question: What is the main contribution of the paper?” | Tests retrieval and summarization. |
| “Using the provided FAQ, answer: How do I reset my password?” | Tests knowledge retrieval from a knowledge base. |

### 9.2 Prompt Shape

```text
Context:
[Excerpt from paper]

Question: What is the main contribution of the paper?
```

### 9.3 Expected Answer Shape

* A concise paragraph answering the question.  
* Citations to the context (e.g., “As stated in paragraph 3…”).  

### 9.4 Automated Checks

| Check | Implementation |
|-------|----------------|
| Context Usage | Verify that the answer contains a phrase from the context. |
| Factual Accuracy | Cross‑check with a ground‑truth answer. |
| Length | 20–50 tokens. |

### 9.5 Human Review Rubric

| Criterion | 0 | 1 | 2 | 3 |
|-----------|---|---|---|---|
| Relevance | Off‑topic | Partially relevant | Relevant | Highly relevant |
| Accuracy | Wrong answer | Partially correct | Mostly correct | Correct |
| Clarity | Confusing | Understandable | Clear | Clear and concise |

### 9.6 Failure Examples

* **Hallucination** – The model invents a contribution not present in the context.  
* **No Context** – The answer is generic and does not reference the excerpt.  
* **Length Violation** – The answer is too short (<10 tokens).  

### 9.7 Metrics to Graph

* **Recall** – % of prompts where the answer contains a phrase from the context.  
* **Precision** – % of context‑used answers that are correct.  
* **Latency** – Token latency for RAG prompts.  
* **Fit Score** – Context fit for RAG prompts.  

### 9.8 Reasoning Budget Notes

* RAG prompts often need a 32‑token budget to allow the model to “look up” the context.  
* A 64‑token budget can improve recall but may introduce hallucinations if the model over‑generates.

---

**# 10. Chatbot Workloads**

### 10.1 Realistic Task Design

| Prompt | Description |
|--------|-------------|
| “You are a friendly customer support agent for a fictional e‑commerce site. The user says: ‘I received a damaged item. How do I return it?’” | Tests empathy, policy knowledge, and step‑by‑step instructions. |
| “You are a travel assistant. The user asks: ‘What’s the weather like in Paris next week?’” | Tests real‑time knowledge retrieval. |

### 10.2 Prompt Shape

```text
You are a helpful customer support chatbot for ShopEasy.
User: I received a damaged item. How do I return it?
```

### 10.3 Expected Answer Shape

* A conversational response with empathy, followed by a clear return process.  
* Optional FAQ link.  

### 10.4 Automated Checks

| Check | Implementation |
|-------|----------------|
| Empathy | Run a sentiment analyzer on the first sentence. |
| Policy Compliance | Verify that the response follows the policy template. |
| Clarity | Check for presence of actionable steps. |

### 10.5 Human Review Rubric

| Criterion | 0 | 1 | 2 | 3 |
|-----------|---|---|---|---|
| Empathy | None | Mild | Moderate | Strong |
| Accuracy | Wrong policy | Partially correct | Mostly correct | Correct |
| Clarity | Confusing | Understandable | Clear | Clear and concise |

### 10.6 Failure Examples

* **Policy Violation** – Suggests a refund instead of a return.  
* **Lack of Empathy** – Uses a flat tone.  
* **Missing Steps** – Does not mention the return label.  

### 10.7 Metrics to Graph

* **User Satisfaction** – Simulated via a sentiment score.  
* **Latency** – Token latency for chatbot prompts.  
* **Token Count** – Average tokens per response.  
* **Fit Score** – Context fit for chatbot prompts.  

### 10.8 Reasoning Budget Notes

* Chatbot responses typically need a 16‑token budget; more can lead to verbosity.  
* A 32‑token budget may improve empathy but can introduce unnecessary filler.

---

**# 11. Creative And Editorial Workloads**

### 11.1 Realistic Task Design

| Prompt | Description |
|--------|-------------|
| “Write a 200‑word short story about a time‑traveling cat.” | Tests narrative creativity. |
| “Draft a press release announcing the launch of a new electric scooter.” | Tests marketing tone and structure. |

### 11.2 Prompt Shape

```text
Write a 200‑word short story about a time‑traveling cat.
Include a twist ending.
```

### 11.3 Expected Answer Shape

* A single paragraph or a short story with a clear beginning, middle, and end.  
* Word count between 190–210 words.  

### 11.4 Automated Checks

| Check | Implementation |
|-------|----------------|
| Word Count | Count tokens and map to words. |
| Narrative Structure | Run a simple heuristic: presence of “beginning”, “conflict”, “resolution”. |
| Style | Run a stylometric analysis to compare with a reference corpus. |

### 11.5 Human Review Rubric

| Criterion | 0 | 1 | 2 | 3 |
|-----------|---|---|---|---|
| Creativity | Predictable | Some novelty | Novel | Highly original |
| Cohesion | Disjointed | Some flow | Good flow | Excellent flow |
| Tone | Inappropriate | Adequate | Appropriate | Perfectly matched |

### 11.6 Failure Examples

* **Repetition** – The cat repeats the same action 10 times.  
* **Word Count Violation** – 150 words.  
* **Tone Mismatch** – Formal press release written in a casual voice.  

### 11.7 Metrics to Graph

* **Creativity Index** – Ratio of unique n‑grams to total n‑grams.  
* **Latency** – Token latency for creative prompts.  
* **Token Count** – Average tokens per creative prompt.  
* **Fit Score** – Context fit for creative prompts.  

### 11.8 Reasoning Budget Notes

* Creative prompts benefit from a 32‑token budget to allow the model to “plan” a narrative arc.  
* Over‑budgeting (>128 tokens) can cause the model to over‑explain and lose the twist.

---

**# 12. Long‑Output Reliability**

Long‑output reliability refers to the model’s ability to produce coherent, accurate, and consistent text over extended token sequences (≥ 500 tokens).  This section outlines the methodology for measuring and improving reliability.

### 12.1 Reliability Metrics

| Metric | Definition | Target |
|--------|------------|--------|
| **Per‑Token Accuracy** | % of tokens that match a ground‑truth reference. | ≥ 95 % |
| **Chunk Coherence** | % of 100‑token chunks that are grammatically correct. | ≥ 90 % |
| **Context Drift** | Ratio of tokens that deviate from the prompt context. | ≤ 5 % |
| **Hallucination Rate** | % of tokens that introduce unsupported facts. | ≤ 2 % |

### 12.2 Test Suite

1. **Long‑Form Article** – 1,000‑token article on a technical topic.  
2. **Dialogue Transcript** – 800‑token dialogue between two characters.  
3. **Code Generation** – 1,200‑token program with comments.  

Each test includes a ground‑truth reference and a set of automated validators.

### 12.3 Automated Validation Pipeline

1. **Tokenization** – Use `llama.cpp`’s tokenizer to split the output.  
2. **Alignment** – Align tokens with the reference using a dynamic‑time‑warping algorithm.  
3. **Fact Checking** – Run a lightweight fact‑checker (e.g., `wikipedia‑api`) on each claim.  
4. **Grammar Checking** – Use `language‑tool` to score each 100‑token chunk.  

### 12.4 Human Review Checklist

| Item | Description |
|------|-------------|
| **Logical Flow** | Does the text progress logically? |
| **Consistency** | Are facts consistent throughout? |
| **Stylistic Consistency** | Does the voice remain the same? |
| **Completeness** | Are all sections present? |

### 12.5 Failure Modes

* **Memory Exhaustion** – Long outputs can exceed the 32 GB RAM if the context window is too large.  
* **Token Drift** – The model may start repeating earlier sections.  
* **Hallucination** – Introducing invented facts after 400 tokens.  

### 12.6 Mitigation Strategies

* **Chunked Generation** – Generate in 200‑token chunks, re‑inject the last 50 tokens as context.  
* **Dynamic Prompting** – Append a “continue” instruction after each chunk.  
* **Post‑Processing** – Run a summarizer on the final output to detect drift.  

### 12.7 AFR Integration

AFR logs each token with a timestamp, latency, and a `chunk_id`.  The harness aggregates tokens per chunk and computes the above metrics in real time, storing them in the SQLite database.

---

**# 13. Privacy And Redaction**

Privacy is paramount when handling user data, even in synthetic benchmarks.  This section outlines how to ensure that no sensitive information leaks through AFR logs or the final report.

### 13.1 Redaction Policy

* **Prompt Redaction** – Replace any user‑supplied identifiers with placeholders (`<USER_ID>`, `<EMAIL>`, `<IP>`) before passing to `llama.cpp`.  
* **Output Redaction** – Scan the model output for patterns that match email addresses, phone numbers, or credit card numbers using regexes and replace them with `<REDACTED>`.  
* **AFR Log Redaction** – AFR writes raw tokens to disk; a post‑processing step scans the log file and removes any sensitive tokens.  

### 13.2 Automated Redaction Pipeline

1. **Regex Scanner** – Detects patterns:  
   * `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b`  
   * `\b\d{3}[-.\s]?\d{4}[-.\s]?\d{4}\b`  
2. **Token Replacement** – Replace matched tokens with `<REDACTED>`.  
3. **Audit Log** – Record the number of redactions per run.  

### 13.3 Human Review Checklist

| Item | Description |
|------|-------------|
| **No Sensitive Data** | Verify that no personal data appears in the final report. |
| **Redaction Accuracy** | Confirm that all regex matches were replaced. |
| **Context Preservation** | Ensure that redaction does not break sentence structure. |

### 13.4 Failure Modes

* **False Negatives** – Regex fails to detect a disguised email.  
* **False Positives** – Legitimate content (e.g., “email” as a noun) is redacted.  
* **Log Leakage** – AFR logs are not cleaned before archival.  

### 13.5 Mitigation Strategies

* **Whitelist** – Maintain a list of acceptable patterns that should not be redacted.  
* **Human Spot‑Check** – Randomly sample 5% of logs for manual inspection.  
* **Encryption** – Store AFR logs encrypted at rest (AES‑256).  

---

**# 14. SQLite Storage And Artifact Layout**

All AFR data, metrics, and artifacts are stored in a single SQLite database (`benchmark.db`).  The schema is designed for efficient querying and long‑term archival.

### 14.1 Database Schema

```sql
CREATE TABLE runs (
    run_id INTEGER PRIMARY KEY,
    model_name TEXT,
    model_path TEXT,
    ctx_size INTEGER,
    threads INTEGER,
    reasoning_budget INTEGER,
    mtp INTEGER,
    context_fit REAL,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE prompts (
    prompt_id INTEGER PRIMARY KEY,
    run_id INTEGER,
    workload TEXT,
    prompt_text TEXT,
    FOREIGN KEY(run_id) REFERENCES runs(run_id)
);

CREATE TABLE tokens (
    token_id INTEGER PRIMARY KEY,
    prompt_id INTEGER,
    token_index INTEGER,
    token_text TEXT,
    token_id_int INTEGER,
    latency_ms REAL,
    chunk_id INTEGER,
    FOREIGN KEY(prompt_id) REFERENCES prompts(prompt_id)
);

CREATE TABLE metrics (
    metric_id INTEGER PRIMARY KEY,
    run_id INTEGER,
    metric_name TEXT,
    metric_value REAL,
    FOREIGN KEY(run_id) REFERENCES runs(run_id)
);
```

### 14.2 Artifact Layout

```
/benchmark
├── benchmark.db
├── logs/
│   ├── run_001.log
│   └── run_002.log
├── reports/
│   ├── run_001.pdf
│   └── run_002.pdf
└── artifacts/
    ├── run_001/
    │   ├── prompt_001.txt
    │   ├── output_001.txt
    │   └── metrics_001.csv
    └── run_002/
        └── ...
```

### 14.3 Data Retention Policy

* **Short‑Term** – Keep raw logs for 30 days.  
* **Long‑Term** – Archive the database and reports to an encrypted external drive.  
* **Deletion** – After 90 days, delete all artifacts except the database.

### 14.4 Failure Modes

* **Database Corruption** – Power loss during write.  
* **File Permissions** – Logs accessible to non‑authorized users.  

### 14.5 Mitigation Strategies

* **Atomic Writes** – Use SQLite’s `PRAGMA synchronous = FULL`.  
* **Access Control** – Set file permissions to `600` for all artifacts.  
* **Backup** – Daily incremental backups to a separate partition.

---

**# 15. Reporting Plane And Screenshots**

The harness automatically generates a PDF report for each run.  The report includes:

1. **Run Summary** – Model name, context size, threads, reasoning budget.  
2. **Metric Charts** – Accuracy, latency, token count, fit score, reliability index.  
3. **AFR Snapshots** – Screenshots of the AFR UI showing token streams for selected prompts.  
4. **Failure Analysis** – Table of prompts that failed automated checks, with human review notes.  
5. **Recommendations** – Suggested parameter tweaks based on the data.

### 15.1 Screenshot Capture

AFR exposes a REST endpoint (`/screenshot`) that returns a PNG of the current token stream.  The harness calls this endpoint after each prompt and stores the image in `reports/run_XXX/`.

### 15.2 PDF Generation

The harness uses `ReportLab` to assemble the PDF:

```python
from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas

c = canvas.Canvas("report.pdf", pagesize=letter)
c.drawString(100, 750, "Benchmark Report")
# Add charts, tables, and images
c.save()
```

### 15.3 Failure Modes

* **Missing Images** – AFR endpoint fails due to network hiccup.  
* **PDF Corruption** – ReportLab crashes on large tables.  

### 15.4 Mitigation Strategies

* **Retry Logic** – Retry screenshot capture up to 3 times.  
* **Chunked Tables** – Split large tables into multiple pages.

---

**# 16. Model Leaderboards**

The leaderboard aggregates metrics across all runs for a given model.  It is stored in a separate SQLite database (`leaderboard.db`) and refreshed after each run.

### 16.1 Leaderboard Schema

```sql
CREATE TABLE leaderboard (
    model_name TEXT PRIMARY KEY,
    avg_accuracy REAL,
    avg_latency REAL,
    avg_token_count REAL,
    reliability_index REAL,
    last_updated DATETIME DEFAULT CURRENT_TIMESTAMP
);
```

### 16.2 Ranking Criteria

1. **Primary** – Reliability Index (weighted sum of accuracy, latency, token count).  
2. **Secondary** – Context Fit.  
3. **Tertiary** – Human Review Score.

### 16.3 Updating the Leaderboard

```python
def update_leaderboard(run_id):
    # Query metrics for the run
    # Compute weighted scores
    # Insert or replace into leaderboard
```

### 16.4 Failure Modes

* **Stale Data** – Leaderboard not updated due to a crash.  
* **Duplicate Entries** – Multiple runs for the same model overwrite each other.  

### 16.5 Mitigation Strategies

* **Transactional Updates** – Wrap leaderboard updates in a transaction.  
* **Versioning** – Store a `run_id` column to track the source of each leaderboard entry.

---

**# 17. Reproducibility Checklist**

| Item | Check | Notes |
|------|-------|-------|
| **Model File** | Verify SHA‑256 hash | Store hash in `runs` table |
| **Environment** | `python --version`, `gcc --version` | Record in `runs` |
| **Hardware** | CPU model, RAM size | Record in `runs` |
| **AFR Version** | `afr --version` | Record in `runs` |
| **Random Seed** | Set `--seed` in `llama.cpp` | Ensure deterministic outputs |
| **Prompt Text** | Store raw prompt | Prevent accidental edits |
| **Log Files** | Keep raw AFR logs | For post‑hoc analysis |
| **Metrics** | Store all raw metrics | Not just aggregated values |
| **Human Review Notes** | Store rubric scores | For audit trail |
| **Redaction** | Verify no sensitive data | Run audit script |

### 17.1 Failure Modes

* **Non‑Deterministic Outputs** – Random seed not set.  
* **Missing Metadata** – No hash stored.  
* **Inconsistent Prompts** – Prompt edited after run.  

### 17.2 Mitigation Strategies

* **Automated Metadata Capture** – The harness automatically records all relevant metadata.  
* **Immutable Prompt Files** – Store prompts in a read‑only directory.  
* **Checksum Verification** – Run a checksum script after each run.

---

**# 18. Final Recommendations**

1. **Start with a Baseline** – Run all workloads on a single model (e.g., `wizardlm-7b.gguf`) with default settings to establish a baseline.  
2. **Tune Reasoning Budget** – For each workload, perform a budget sweep (0, 32, 64, 128 tokens) and plot accuracy vs. latency.  
3. **Monitor Thermal** – Keep CPU temperature below 70 °C; otherwise, throttle performance.  
4. **Use MTP Sparingly** – MTP is useful for benchmarking but can distort natural language generation.  
5. **Validate Context Fit** – A fit score below 0.4 indicates the model is drifting; consider reducing context size or increasing reasoning budget.  
6. **Automate Redaction** – Run the redaction pipeline before archiving logs.  
7. **Archive AFR Logs** – Store raw logs encrypted; they are invaluable for debugging long‑output failures.  
8. **Update Leaderboard** – Run the leaderboard update script after every benchmark to keep the ranking current.  
9. **Document Every Run** – Use the reproducibility checklist to ensure future researchers can replicate your results.  
10. **Iterate** – Use the insights from the metrics to adjust `ctx-size`, `threads`, and `reasoning_budget` for subsequent runs.

By following this manual, you will generate a comprehensive, reproducible, and privacy‑preserving benchmark of open‑weight GGUF models on your R9700 server.  The resulting data will enable you to compare models, understand their strengths and weaknesses, and drive informed decisions for deployment or further research.