# 1. Purpose And Scope

This master field manual establishes a standardized, reproducible methodology for evaluating open-weight GGUF models deployed on a 32GB-class R9700 llama.cpp server. The evaluation framework is designed to operate within a constrained reasoning budget of 8192 tokens, ensuring that performance metrics reflect realistic deployment conditions rather than unconstrained theoretical capacity. The manual serves as the definitive reference for benchmark engineers, home-lab operators, and inference pipeline developers who require deterministic, auditable, and hardware-aware model assessment.

The scope of this manual encompasses the complete lifecycle of a benchmark campaign: hardware readiness validation, runtime configuration, budget allocation strategies, workload-specific testing protocols, artifact storage, reporting generation, and reproducibility verification. All testing procedures are explicitly designed to collect raw telemetry via AI Flight Recorder, a structured logging and metrics aggregation framework that captures token-level timing, memory pressure, context utilization, and output fidelity. The manual does not present pre-computed results; rather, it provides the exact procedural blueprint for collecting, validating, and interpreting benchmark data.

The evaluation philosophy rests on three core principles. First, determinism: every run must be reproducible through version-controlled configurations, fixed random seeds, and explicit runtime flags. Second, constraint-awareness: the 8192-token reasoning budget is treated as a hard operational boundary, requiring careful prompt-completion partitioning and context management. Third, synthetic integrity: all test inputs, evaluation rubrics, and validation checks rely exclusively on synthetic data to eliminate leakage, bias, and privacy concerns.

This manual is organized into eighteen major sections. Sections 1 through 6 establish the foundational environment, runtime profile, and methodological constraints. Sections 7 through 12 detail workload-specific evaluation protocols, each containing task design, prompt shapes, expected outputs, automated checks, human review rubrics, failure examples, metrics, and reasoning budget considerations. Sections 13 through 18 cover privacy handling, storage architecture, reporting infrastructure, leaderboard construction, reproducibility verification, and final operational recommendations.

Operators should treat this document as a living specification. As GGUF quantization schemes evolve, llama.cpp introduces new optimization paths, and AI Flight Recorder expands its telemetry schema, the manual should be updated accordingly. All procedures assume a baseline familiarity with GGUF model loading, KV cache management, and structured logging practices. The manual explicitly avoids proprietary integrations, cloud dependencies, or external API calls, ensuring that the entire benchmark campaign can be executed in an air-gapped or isolated home-lab environment.

# 2. Hardware Profile

The evaluation server is classified as a 32GB-class R9700 platform, engineered to balance memory capacity, compute throughput, and thermal stability for sustained LLM inference. The hardware profile defines the physical and logical boundaries within which GGUF models operate, directly influencing quantization selection, context window limits, and token generation speed.

**Core Specifications:**
- CPU: 16-core / 32-thread architecture with AVX2/AVX-512 support for optimized CPU offloading paths.
- RAM: 32GB DDR5 ECC, configured in dual-channel mode to maximize memory bandwidth.
- Accelerator: Integrated or discrete GPU with 16GB VRAM, supporting CUDA 12.x or ROCm 6.x depending on deployment variant.
- Storage: NVMe Gen4 SSD (2TB) with sustained sequential read/write speeds exceeding 5000 MB/s, critical for rapid GGUF model loading and KV cache persistence.
- Network: 10GbE interface for internal telemetry streaming and artifact synchronization.
- Cooling: Active liquid or high-static-pressure air cooling, maintaining junction temperatures below 85°C under sustained load.

**Memory Architecture Considerations:**
The 32GB system RAM imposes a hard ceiling on GGUF model sizing when combined with KV cache allocation. A Q4_K_M quantized model typically consumes approximately 1.2 bytes per parameter. For a 7B parameter model, this translates to roughly 8.4GB for weights alone. The remaining memory must accommodate the KV cache, runtime overhead, AI Flight Recorder buffers, and OS processes. Operators must calculate the maximum safe context window using the formula:
`Max_Context = (Total_RAM - Model_Weights - Runtime_Overhead - OS_Reserved) / (Context_Size * Layer_Bytes * Quantization_Factor)`

**Thermal and Power Management:**
Sustained inference generates consistent thermal load. The R9700 platform must maintain stable clock speeds without thermal throttling. Operators should monitor GPU and CPU junction temperatures via `nvml` or `rocm-smi` during benchmark runs. Power delivery should be stabilized using a UPS to prevent voltage sag during peak KV cache allocation.

**Hardware Validation Checklist:**
- [ ] Run `memtest86+` for full memory integrity verification.
- [ ] Verify NVMe SMART health status and sustained throughput with `fio`.
- [ ] Confirm GPU driver compatibility with llama.cpp backend (CUDA/ROCm/SYCL).
- [ ] Validate thermal thresholds under 100% CPU/GPU load using `stress-ng`.
- [ ] Ensure BIOS settings disable aggressive power-saving states that cause clock instability.
- [ ] Verify PCIe lane allocation for GPU acceleration paths.
- [ ] Confirm system time synchronization via NTP for accurate AI Flight Recorder timestamps.

**Failure Modes and Mitigation:**
- Memory fragmentation causing KV cache allocation failures: Mitigate by using `--mlock` and pre-allocating context buffers.
- GPU VRAM overflow triggering CPU fallback: Mitigate by adjusting `--gpu-layers` and monitoring `--cache-mem` metrics.
- Thermal throttling reducing token throughput: Mitigate by optimizing airflow, adjusting fan curves, and reducing `--batch-size`.
- NVMe read latency spikes during model loading: Mitigate by using direct memory access (DMA) paths and disabling SSD sleep states.

**Validation Notes:**
Hardware profiling must be completed before any model evaluation begins. Record baseline temperatures, memory bandwidth, and PCIe throughput. These metrics serve as normalization factors when comparing token generation speeds across different GGUF quantizations. Any hardware anomaly must be logged in AI Flight Recorder as a `hardware_degradation` event, triggering automatic run suspension until resolution.

# 3. llama.cpp Runtime Profile

The llama.cpp runtime serves as the inference engine for all GGUF model evaluations. Proper configuration ensures deterministic behavior, optimal resource utilization, and accurate telemetry capture via AI Flight Recorder. The runtime profile defines the exact flags, environment variables, and architectural choices required for benchmark consistency.

**Core Runtime Flags:**
- `--model <path>`: Points to the target GGUF file. Must be verified via SHA256 hash.
- `--ctx-size <N>`: Sets the maximum context window. Must align with hardware memory constraints and the 8192-token reasoning budget.
- `--threads <N>`: CPU threads for inference. Typically set to physical core count minus 2 for OS overhead.
- `--gpu-layers <N>`: Number of layers offloaded to GPU. Must be tuned to avoid VRAM overflow.
- `--batch-size <N>`: Prompt processing batch size. Affects throughput and memory pressure.
- `--n-predict <N>`: Maximum completion tokens. Should be capped to prevent budget exhaustion.
- `--temp <F>`: Sampling temperature. Fixed at 0.7 for deterministic evaluation unless workload specifies otherwise.
- `--top-p <F>`: Nucleus sampling parameter. Fixed at 0.9.
- `--seed <N>`: Random seed for reproducibility. Must be explicitly set per run.
- `--log-prefix <S>`: Enables structured logging compatible with AI Flight Recorder ingestion.

**KV Cache and Memory Management:**
The KV cache stores attention states for generated tokens. In llama.cpp, this is managed via `--cache-type-k` and `--cache-type-v` (e.g., `f16`, `q8_0`, `q4_0`). Quantized KV caches reduce memory footprint but may introduce precision loss. Operators should benchmark cache quantization levels to determine the optimal balance between context length and generation fidelity.

**Flash Attention and Optimization Paths:**
If supported by the hardware and llama.cpp build, enable `--flash-attn` to reduce memory bandwidth pressure during attention computation. This significantly improves throughput for long-context workloads. Verify support via `llama.cpp --help` and runtime logs.

**AI Flight Recorder Integration:**
llama.cpp must be compiled with `LLAMA_LOG_AI_FR=1` to emit structured telemetry. Key metrics captured include:
- `token_generation_time_ms`
- `kv_cache_usage_bytes`
- `gpu_memory_utilization_pct`
- `context_overflow_events`
- `speculative_decoding_steps` (if MTP enabled)
- `budget_remaining_tokens`

**Runtime Validation Checklist:**
- [ ] Compile llama.cpp with AI Flight Recorder telemetry flags.
- [ ] Verify GGUF model hash matches official release.
- [ ] Test `--ctx-size` allocation without OOM errors.
- [ ] Confirm GPU offload layers do not exceed VRAM capacity.
- [ ] Validate log output format matches AI Flight Recorder schema.
- [ ] Run a 100-token smoke test to verify sampling consistency.
- [ ] Check that `--seed` produces identical outputs across runs.

**Failure Modes and Mitigation:**
- KV cache fragmentation causing context truncation: Mitigate by using `--mmap` and monitoring `cache_usage_pct`.
- GPU memory overflow triggering fallback: Mitigate by reducing `--gpu-layers` or switching to `--cache-type-v q8_0`.
- Non-deterministic sampling: Mitigate by fixing `--seed`, `--temp`, and `--top-p`.
- Log parsing failures: Mitigate by validating JSON schema against AI Flight Recorder parser before full campaign.

**Validation Notes:**
Runtime configuration must be version-controlled in YAML format. Each benchmark run should log the exact command line, environment variables, and llama.cpp commit hash. Any deviation from the baseline configuration must be flagged as a `config_drift` event. The runtime profile directly influences token throughput, latency, and memory pressure, making it a critical normalization factor for all workload metrics.

# 4. Reasoning Budget Methodology

The 8192-token reasoning budget represents the maximum token window available for prompt processing, intermediate reasoning traces, and output generation within a single inference cycle. This budget is not merely a context limit; it is a strategic resource that must be allocated to maximize task completion while preserving fidelity. The methodology defines how to partition, monitor, and optimize budget usage across all workloads.

**Budget Partitioning Strategy:**
The reasoning budget is divided into three functional zones:
1. **Prompt Zone (40-50%):** System instructions, task definitions, synthetic context chunks, and few-shot examples.
2. **Reasoning Zone (20-30%):** Intermediate thought traces, tool call logs, retrieval metadata, and planning steps.
3. **Output Zone (20-30%):** Final responses, code blocks, structured JSON, or creative text.

Operators must explicitly allocate tokens per zone based on workload requirements. For example, coding workloads may require a larger reasoning zone for step-by-step logic, while chatbot workloads may prioritize output zone efficiency.

**Budget Monitoring via AI Flight Recorder:**
AI Flight Recorder tracks budget consumption in real-time using the `reasoning_budget_remaining` metric. The telemetry schema includes:
- `prompt_tokens_consumed`
- `reasoning_tokens_consumed`
- `completion_tokens_consumed`
- `budget_exhaustion_event` (boolean)
- `graceful_degradation_flag` (boolean)

Operators should configure alerts when budget consumption exceeds 85%, triggering automatic prompt truncation or output early termination to prevent hard failures.

**Budget Optimization Techniques:**
- **Context Pruning:** Remove redundant few-shot examples or verbose system instructions.
- **Token Compression:** Use structured formats (JSON, XML) to reduce token count without losing semantic density.
- **Chunked Inference:** Split long tasks into sequential calls, resetting the budget per chunk.
- **Speculative Drafting:** Use MTP to generate reasoning traces faster, preserving budget for output generation.

**Synthetic Budget Allocation Example:**
```yaml
budget_profile:
  total_tokens: 8192
  prompt_zone: 3500
  reasoning_zone: 2200
  output_zone: 2492
  safety_margin: 0
  monitoring_interval_ms: 500
  exhaustion_action: graceful_termination
```

**Failure Modes and Mitigation:**
- Budget exhaustion mid-generation: Mitigate by implementing `--early-stop` triggers and logging `budget_cutoff` events.
- Over-allocation to prompt zone starving output: Mitigate by dynamically adjusting zone ratios based on task complexity.
- Reasoning trace bloat: Mitigate by enforcing structured thought formats and limiting trace length.
- Silent budget overflow: Mitigate by validating `budget_remaining_tokens` at each inference step.

**Validation Notes:**
Budget methodology must be documented per workload. AI Flight Recorder should log budget utilization curves, showing consumption over time. Operators should analyze the ratio of reasoning tokens to output tokens to identify inefficiencies. The 8192 limit is a hard constraint; exceeding it results in context truncation or inference failure, both of which must be captured as benchmark anomalies.

# 5. MTP Versus Non-MTP Methodology

Multi-Token Prediction (MTP) enables speculative decoding, where a smaller draft model generates multiple tokens that are verified by the target model in parallel. This methodology compares MTP-enabled runs against standard autoregressive (Non-MTP) runs to quantify throughput gains, accuracy trade-offs, and budget efficiency.

**MTP Configuration:**
- `--draft-model <path>`: Lightweight GGUF model for token speculation.
- `--mtp-n-draft <N>`: Number of draft tokens per verification step.
- `--mtp-accept-threshold <F>`: Probability threshold for accepting draft tokens.
- `--mtp-verify-batch <N>`: Batch size for verification.

**Non-MTP Baseline:**
Standard autoregressive generation with `--n-predict` set to match MTP total output length. All other flags remain identical to ensure fair comparison.

**Benchmark Protocol:**
1. Load target GGUF model with Non-MTP configuration.
2. Run synthetic workload, record `token_generation_time_ms`, `throughput_tokens_per_sec`, `accuracy_score`, and `budget_consumed`.
3. Load target GGUF model with MTP configuration.
4. Run identical synthetic workload, record same metrics.
5. Compare delta across all dimensions.

**Metrics to Graph:**
- Throughput improvement percentage
- Latency variance across workloads
- Accuracy degradation (if any)
- Budget efficiency (tokens per second per budget unit)
- Speculative acceptance rate

**Failure Modes and Mitigation:**
- Draft model mismatch causing verification failures: Mitigate by aligning draft model architecture with target model family.
- High rejection rate reducing throughput: Mitigate by adjusting `--mtp-accept-threshold` and `--mtp-n-draft`.
- Context drift during speculative steps: Mitigate by enabling `--mtp-strict-context` and monitoring `context_alignment_pct`.
- Memory pressure from dual-model loading: Mitigate by using `--cache-type-k q8_0` and monitoring `gpu_memory_utilization_pct`.

**Validation Notes:**
MTP evaluation must be conducted under identical hardware and runtime conditions. AI Flight Recorder should log `speculative_steps`, `accepted_drafts`, and `rejected_drafts` per run. Operators should analyze the acceptance rate curve to determine optimal draft parameters. MTP is highly workload-dependent; coding and agentic tasks may benefit more than creative or chatbot workloads due to structured token patterns.

# 6. Context Fit Methodology

Context fit evaluation measures how models maintain coherence, retrieval accuracy, and instruction following as context length approaches the reasoning budget limit. This methodology tests attention degradation, needle-in-a-haystack retrieval, and multi-document synthesis under constrained token windows.

**Task Design:**
- Synthetic long-context documents (5k-8k tokens) containing embedded facts, contradictions, and structural markers.
- Multi-turn queries requiring cross-reference verification.
- Instruction-following tests with nested constraints.

**Prompt Shape:**
```
[SYSTEM] You are a precision analyst. Process the following context and answer queries with exact citations.
[CONTEXT] <synthetic_document_chunk_1> ... <synthetic_document_chunk_N>
[QUERY] Extract all references to [synthetic_entity] and verify consistency across chunks.
[REASONING] Step-by-step verification trace.
[OUTPUT] Structured JSON with citations and confidence scores.
```

**Expected Answer Shape:**
Valid JSON with `citations`, `consistency_checks`, `confidence_score`, and `reasoning_trace`. All citations must map to exact synthetic context positions.

**Automated Checks:**
- Citation accuracy validation against synthetic context indices.
- Consistency score calculation across multiple queries.
- Repetition detection in reasoning traces.
- Budget utilization tracking.

**Human Review Rubric:**
- Factual grounding (0-5)
- Citation precision (0-5)
- Structural compliance (0-5)
- Reasoning clarity (0-5)

**Failure Examples:**
- Hallucinated citations not present in context.
- Contradictory statements across queries.
- Context truncation causing missing references.
- Reasoning trace bloat consuming budget prematurely.

**Metrics to Graph:**
- Retrieval accuracy vs. context length
- Consistency score decay rate
- Budget consumption per query
- Token efficiency (answers per budget unit)

**Reasoning Budget Notes:**
The 8192-token limit forces operators to balance context density with reasoning depth. MTP can accelerate retrieval verification, but Non-MTP may preserve precision for complex cross-references. AI Flight Recorder should log `context_window_utilization_pct` and `attention_decay_score` to identify optimal context partitioning strategies.

# 7. Coding Workloads

**Realistic Task Design:**
Generate, debug, and refactor synthetic codebases. Tasks include implementing data structures, fixing syntax/logic errors, writing unit tests, and optimizing algorithms. All code operates on fictional datasets and APIs.

**Prompt Shape:**
```
[SYSTEM] You are a senior software engineer. Write production-ready code following PEP-8/ESLint standards.
[TASK] Implement a binary search tree with insert, delete, and balance operations.
[CONSTRAINTS] Time complexity O(log n), space O(n), include docstrings and type hints.
[INPUT] Synthetic test cases: [10, 5, 15, 3, 7, 12, 18]
[REASONING] Step-by-step algorithm design and edge case analysis.
[OUTPUT] Complete code block with test suite.
```

**Expected Answer Shape:**
Valid code block, syntax-checked, with inline comments, type annotations, and a synthetic test runner. Reasoning trace should outline complexity analysis and boundary conditions.

**Automated Checks:**
- Syntax validation via synthetic parser.
- Complexity verification against constraints.
- Test suite execution simulation.
- Static analysis for security/style violations.

**Human Review Rubric:**
- Correctness (0-5)
- Efficiency (0-5)
- Readability (0-5)
- Security awareness (0-5)

**Failure Examples:**
- Hallucinated library functions.
- Infinite recursion or off-by-one errors.
- Missing type hints or docstrings.
- Budget exhaustion causing truncated code.

**Metrics to Graph:**
- Compilation success rate
- Test pass rate
- Latency per token
- Budget utilization vs. code length
- Reasoning trace length vs. correctness

**Reasoning Budget Notes:**
Coding tasks require substantial reasoning tokens for algorithm design and edge case analysis. The 8192 limit may force truncation of complex multi-file projects. Operators should use chunked inference for large codebases and monitor `reasoning_zone_consumption` to ensure adequate planning space. MTP can accelerate syntax generation but may reduce logical precision.

# 8. Agentic Workloads

**Realistic Task Design:**
Multi-step execution involving tool use, state management, planning, and error recovery. Tasks include scheduling synthetic events, querying mock databases, and coordinating API calls.

**Prompt Shape:**
```
[SYSTEM] You are an autonomous agent. Use provided tools to complete tasks. Log all actions.
[TOOLS] {search, schedule, query_db, notify}
[TASK] Find available slots for a meeting, book it, and notify participants.
[STATE] Current time: 2024-06-15T10:00Z, Participants: [A, B, C]
[REASONING] Plan steps, validate tool outputs, handle conflicts.
[OUTPUT] Structured action log and final confirmation.
```

**Expected Answer Shape:**
JSON action log with `tool_name`, `arguments`, `result`, `state_update`, and `reasoning_step`. Final confirmation includes booking ID and participant list.

**Automated Checks:**
- Tool call validation against schema.
- State consistency verification.
- Loop detection in action sequences.
- Budget consumption tracking.

**Human Review Rubric:**
- Planning quality (0-5)
- Tool accuracy (0-5)
- Error recovery (0-5)
- Efficiency (steps per task) (0-5)

**Failure Examples:**
- Invalid tool arguments.
- Infinite planning loops.
- State drift across steps.
- Budget overflow causing incomplete execution.

**Metrics to Graph:**
- Task completion rate
- Tool call accuracy
- Steps per task
- Budget consumption per step
- Error recovery success rate

**Reasoning Budget Notes:**
Agentic workloads consume reasoning tokens rapidly due to planning and state tracking. The 8192 limit requires careful budget partitioning. Operators should use `--early-stop` for failed branches and monitor `reasoning_zone_consumption`. MTP can accelerate tool call generation but may reduce planning depth.

# 9. RAG Workloads

**Realistic Task Design:**
Retrieval-augmented generation using synthetic document chunks. Tasks include answering queries, synthesizing cross-document insights, and verifying factual claims.

**Prompt Shape:**
```
[SYSTEM] You are a research assistant. Use provided context to answer queries accurately.
[CONTEXT] <chunk_1>, <chunk_2>, <chunk_3>
[QUERY] Summarize findings on [synthetic_topic] and cite sources.
[REASONING] Extract relevant facts, resolve contradictions, synthesize answer.
[OUTPUT] Structured response with citations and confidence score.
```

**Expected Answer Shape:**
Natural language response with inline citations `[C1]`, `[C2]`, confidence score, and reasoning trace. All claims must map to synthetic context.

**Automated Checks:**
- Citation accuracy validation.
- Hallucination detection via fact-checking engine.
- Relevance scoring against query.
- Budget utilization tracking.

**Human Review Rubric:**
- Factual grounding (0-5)
- Synthesis quality (0-5)
- Citation precision (0-5)
- Readability (0-5)

**Failure Examples:**
- Hallucinated sources.
- Irrelevant retrieval.
- Contradictory statements.
- Budget exhaustion truncating synthesis.

**Metrics to Graph:**
- Retrieval accuracy
- Answer faithfulness
- Latency per query
- Budget consumption vs. context length
- Citation precision rate

**Reasoning Budget Notes:**
RAG workloads balance context window usage with reasoning depth. The 8192 limit forces operators to prioritize high-relevance chunks. AI Flight Recorder should log `retrieval_precision` and `synthesis_efficiency`. MTP can accelerate answer generation but may reduce citation accuracy.

# 10. Chatbot Workloads

**Realistic Task Design:**
Multi-turn conversational testing with persona adherence, memory retention, and safety compliance. Tasks include roleplay, Q&A, and iterative refinement.

**Prompt Shape:**
```
[SYSTEM] You are a helpful assistant with a calm, professional tone. Maintain context across turns.
[CONVERSATION] <turn_1>, <turn_2>, <turn_3>
[QUERY] Continue conversation, address user concern, maintain persona.
[REASONING] Analyze context, extract intent, formulate response.
[OUTPUT] Natural language response matching persona and constraints.
```

**Expected Answer Shape:**
Coherent, persona-consistent response with appropriate length, tone, and contextual references. No repetition or safety violations.

**Automated Checks:**
- Coherence scoring via NLP pipeline.
- Repetition detection.
- Safety filter validation.
- Budget consumption tracking.

**Human Review Rubric:**
- Engagement (0-5)
- Consistency (0-5)
- Tone adherence (0-5)
- Helpfulness (0-5)

**Failure Examples:**
- Persona drift.
- Repetitive phrasing.
- Safety violations.
- Context forgetting.

**Metrics to Graph:**
- Coherence score
- Turn efficiency
- Safety pass rate
- Budget utilization per turn
- Response length variance

**Reasoning Budget Notes:**
Chatbot workloads require balanced budget allocation for context retention and response generation. The 8192 limit may force truncation of long histories. Operators should use context summarization and monitor `context_retention_score`. MTP can accelerate response generation but may reduce conversational nuance.

# 11. Creative And Editorial Workloads

**Realistic Task Design:**
Story generation, poetry, style transfer, and iterative editing. Tasks include writing constrained narratives, adapting tone, and refining drafts.

**Prompt Shape:**
```
[SYSTEM] You are a creative writer. Follow stylistic constraints and thematic guidelines.
[TASK] Write a short story about [synthetic_theme] in [synthetic_style].
[CONSTRAINTS] Max 500 words, include metaphor, avoid clichés.
[REASONING] Outline structure, select imagery, plan pacing.
[OUTPUT] Creative text with structural markers.
```

**Expected Answer Shape:**
Coherent narrative with stylistic fidelity, thematic consistency, and constraint compliance. No structural breakdown or cliché overuse.

**Automated Checks:**
- Originality scoring.
- Structural analysis.
- Constraint adherence validation.
- Budget consumption tracking.

**Human Review Rubric:**
- Creativity (0-5)
- Narrative flow (0-5)
- Stylistic match (0-5)
- Emotional resonance (0-5)

**Failure Examples:**
- Cliché generation.
- Structural collapse.
- Constraint violation.
- Budget exhaustion truncating draft.

**Metrics to Graph:**
- Creativity index
- Constraint compliance rate
- Latency per token
- Budget utilization vs. output length
- Iteration success rate

**Reasoning Budget Notes:**
Creative workloads benefit from reasoning tokens for planning and refinement. The 8192 limit may restrict iterative editing. Operators should use chunked generation and monitor `reasoning_zone_consumption`. MTP can accelerate drafting but may reduce stylistic precision.

# 12. Long-Output Reliability

**Realistic Task Design:**
Extended generation tasks requiring sustained coherence, thematic development, and structural integrity. Tasks include multi-chapter outlines, continuous essays, and long-form technical documentation.

**Prompt Shape:**
```
[SYSTEM] You are a technical writer. Generate structured, coherent long-form content.
[TASK] Write a 2000-word guide on [synthetic_topic] with sections, subsections, and examples.
[CONSTRAINTS] Maintain tone, avoid repetition, ensure logical flow.
[REASONING] Outline structure, plan transitions, verify consistency.
[OUTPUT] Multi-paragraph document with structural markers.
```

**Expected Answer Shape:**
Coherent long-form text with consistent themes, proper formatting, and logical progression. No topic drift or repetition loops.

**Automated Checks:**
- Length validation.
- Coherence scoring.
- Repetition detection.
- Structural integrity verification.
- Budget consumption tracking.

**Human Review Rubric:**
- Narrative consistency (0-5)
- Thematic development (0-5)
- Formatting accuracy (0-5)
- Readability (0-5)

**Failure Examples:**
- Topic drift.
- Repetition loops.
- Formatting collapse.
- Early termination.

**Metrics to Graph:**
- Output length
- Coherence decay rate
- Termination accuracy
- Budget consumption vs. length
- Structural integrity score

**Reasoning Budget Notes:**
Long-output tasks strain the 8192-token limit. Operators should use chunked inference, monitor `context_window_utilization_pct`, and implement `--early-stop` for graceful degradation. MTP can accelerate generation but may reduce long-range coherence. AI Flight Recorder should log `coherence_decay_curve` to identify optimal chunk boundaries.

# 13. Privacy And Redaction

**Realistic Task Design:**
PII detection, redaction, and compliance verification using synthetic mixed-data inputs. Tasks include cleaning datasets, masking identifiers, and generating compliance reports.

**Prompt Shape:**
```
[SYSTEM] You are a privacy engineer. Redact PII and verify compliance.
[INPUT] Synthetic dataset with mixed PII and non-PII fields.
[TASK] Identify and redact all PII, preserve context, generate report.
[REASONING] Scan fields, classify PII type, apply redaction rules.
[OUTPUT] Redacted dataset and compliance summary.
```

**Expected Answer Shape:**
Cleaned dataset with `[REDACTED]` markers, compliance report with `pii_count`, `redaction_accuracy`, and `false_positive_rate`.

**Automated Checks:**
- Regex validation for PII patterns.
- False positive/negative rate calculation.
- Context preservation verification.
- Budget consumption tracking.

**Human Review Rubric:**
- Redaction completeness (0-5)
- Context preservation (0-5)
- Compliance alignment (0-5)
- Efficiency (0-5)

**Failure Examples:**
- Missed PII.
- Over-redaction.
- Data leakage in logs.
- Budget exhaustion truncating scan.

**Metrics to Graph:**
- Redaction accuracy
- False positive rate
- Processing time
- Budget utilization vs. dataset size
- Compliance pass rate

**Reasoning Budget Notes:**
Privacy tasks require thorough scanning, consuming reasoning tokens. The 8192 limit may force batch processing. Operators should use `--cache-type-k q8_0` to preserve budget for analysis. AI Flight Recorder should log `pii_detection_latency` and `redaction_precision`.

# 14. SQLite Storage And Artifact Layout

**Database Schema Design:**
AI Flight Recorder stores benchmark telemetry in SQLite for efficient querying and archival. The schema includes:
- `runs`: `run_id`, `timestamp`, `model_hash`, `config_yaml`, `hardware_profile`, `status`
- `metrics`: `run_id`, `metric_name`, `value`, `unit`, `timestamp`
- `prompts`: `run_id`, `prompt_id`, `content_hash`, `token_count`, `zone`
- `completions`: `run_id`, `completion_id`, `content_hash`, `token_count`, `status`
- `budgets`: `run_id`, `total_tokens`, `prompt_consumed`, `reasoning_consumed`, `output_consumed`, `remaining`
- `errors`: `run_id`, `error_code`, `message`, `timestamp`, `severity`

**Indexing Strategy:**
- Composite indexes on `(run_id, timestamp)` for chronological queries.
- Hash indexes on `content_hash` for deduplication.
- Partial indexes on `status='success'` for performance filtering.

**Artifact Storage:**
- `/artifacts/models/`: GGUF files with SHA256 verification.
- `/artifacts/configs/`: YAML configuration files.
- `/artifacts/logs/`: AI Flight Recorder JSONL logs.
- `/artifacts/screenshots/`: Dashboard captures and metric visualizations.
- `/artifacts/reports/`: Generated PDF/HTML reports.

**Backup and Retention:**
- Daily incremental backups of SQLite database.
- Weekly full backups with checksum verification.
- 90-day retention for logs, 1-year for reports.
- Encryption at rest using AES-256.

**Validation Notes:**
Schema migrations must be version-controlled. Data integrity checks should run post-run using `PRAGMA integrity_check`. Operators should validate foreign key constraints and index efficiency. Synthetic examples of SQL queries for analysis should be documented in `/docs/sql_examples.md`.

# 15. Reporting Plane And Screenshots

**Dashboard Design:**
AI Flight Recorder provides a real-time dashboard displaying:
- Token throughput over time
- Budget consumption curves
- Metric heatmaps per workload
- Error rate trends
- Hardware utilization gauges

**Screenshot Capture Methodology:**
- Automated capture at 60-second intervals.
- Full-screen and component-specific views.
- Timestamp synchronization with telemetry logs.
- Metadata embedding (run_id, model_hash, config_version).

**Report Generation:**
- Automated PDF/HTML generation using Jinja2 templates.
- Executive summary with key metrics and recommendations.
- Detailed appendices with raw telemetry and screenshots.
- Version-controlled report naming convention: `report_<run_id>_<timestamp>.pdf`.

**Validation Notes:**
Dashboard metrics must align with SQLite data. Screenshot timestamps should match telemetry logs within 100ms. Report generation should be idempotent and version-controlled. Synthetic examples of report sections should be stored in `/docs/report_templates/`.

# 16. Model Leaderboards

**Scoring Methodology:**
Composite scores calculated using weighted averages across workloads:
- Coding: 25%
- Agentic: 20%
- RAG: 20%
- Chatbot: 15%
- Creative: 10%
- Long-Output: 10%

Normalization applied using min-max scaling per metric. Confidence intervals calculated via bootstrap resampling.

**Ranking Algorithm:**
Percentile-based ranking with tiered classification:
- Tier 1: Top 10%
- Tier 2: 10-30%
- Tier 3: 30-60%
- Tier 4: Bottom 40%

**Validation Notes:**
Leaderboards must be updated post-campaign with fresh data. Overfitting prevention via cross-validation. Transparency in weighting methodology. Synthetic examples of leaderboard tables should be stored in `/docs/leaderboard_examples/`.

# 17. Reproducibility Checklist

**Pre-Run Checklist:**
- [ ] Hardware validated and logged.
- [ ] llama.cpp compiled with AI Flight Recorder flags.
- [ ] GGUF model hash verified.
- [ ] Config YAML version-controlled.
- [ ] AI Flight Recorder schema validated.
- [ ] Synthetic test suite loaded.
- [ ] Budget allocation documented.

**During-Run Checklist:**
- [ ] Telemetry streaming active.
- [ ] Budget monitoring enabled.
- [ ] Error handling configured.
- [ ] Screenshot capture scheduled.
- [ ] Hardware metrics logged.
- [ ] Run ID generated and recorded.

**Post-Run Checklist:**
- [ ] SQLite data validated.
- [ ] Metrics calculated and stored.
- [ ] Artifacts archived.
- [ ] Report generated.
- [ ] Leaderboard updated.
- [ ] Reproducibility hash logged.

**Validation Notes:**
Cross-machine reproducibility requires identical hardware profiles and software versions. Deterministic seeding ensures identical outputs. Git LFS for configs, model hashes, and dependency pining. Synthetic examples of checklist items should be stored in `/docs/checklist_templates/`.

# 18. Final Recommendations

Operators should adopt a phased rollout strategy: begin with baseline Non-MTP runs, validate budget allocation, then introduce MTP and workload-specific optimizations. Hardware profiling must precede every campaign, and runtime configurations should be version-controlled. The 8192-token reasoning budget requires disciplined partitioning; prioritize reasoning zones for complex tasks and use chunked inference for long outputs. AI Flight Recorder telemetry should be analyzed for coherence decay, budget exhaustion patterns, and hardware bottlenecks. Leaderboards must be updated transparently, with clear weighting methodologies and confidence intervals. Future-proofing requires monitoring GGUF quantization advancements, llama.cpp optimization paths, and AI Flight Recorder schema evolution. This manual provides a rigorous, synthetic, and reproducible framework for home-lab benchmark campaigns. Execute with precision, log with integrity, and iterate with data.