# Answer
The AI Flight Recorder home lab operates under a comprehensive reporting and data management policy designed to balance transparency in model performance with strict privacy and security protocols. The policy is structured around three pillars: Data Retention, Reporting Standards, and Publication Security.

### Data Retention and Management
All raw data generated within the home lab is subject to a "Retain Until Deleted" mandate. Specifically, raw private prompts must be kept in local storage until a user explicitly chooses to delete them [S6]. This supersedes older policies that suggested a 14-day retention window [S1]. Furthermore, the lab maintains a strict policy regarding the preservation of unsuccessful attempts: all failed and invalid runs must be retained and clearly labeled. This is a critical requirement because these runs provide essential data regarding the reliability and failure modes of the models being tested [S8].

### Reporting and Metrics Standards
When conducting model comparisons, the lab must provide a standardized set of metrics to ensure consistency and comparability. Every comparison report must include:
1.  **Success Metrics:** The overall pass rate and the specific count of invalid runs [S5].
2.  **Performance Metrics:** The median generation Tokens Per Second (TPS) and the MTP (Maximum Token Probability) acceptance rate [S5].
3.  **Token Analysis:** A breakdown of reasoning tokens versus final tokens [S5].
4.  **Artifacts:** The inclusion of relevant output artifacts [S5].

For visual documentation, such as public screenshots, the policy requires the inclusion of specific metadata: the model name, the quantization level (quant), the context size, and the total token counts [S2].

### Publication and Privacy Protocols
The lab enforces a multi-tiered approach to sharing information:
1.  **Redaction:** Any report intended for publication must use redacted summaries rather than raw data [S6].
2.  **Screenshot Privacy:** Public screenshots are permitted to omit raw prompts, provided the required metadata (model name, quant, context size, token counts) is present [S2].
3.  **Synthetic Prompt Export:** Synthetic benchmark prompts may be exported for external use only if they have been scrubbed of all sensitive information, including real names, emails, Teams messages, or secrets [S4].
4.  **WorkDash Restrictions:** Artifacts derived from Private WorkDash are strictly prohibited from being published or shared outside of the home lab environment [S3].

# Source Priority
The following hierarchy was used to resolve conflicts and establish the policy, ordered from highest priority (newest/most specific) to lowest:
1.  **S8 (2026-06):** Governing rule for failed/invalid run retention.
2.  **S6 (2026-05):** Governing rule for raw prompt retention and report redaction.
3.  **S5 (2026-04):** Governing rule for model comparison metrics.
4.  **S4 (2026-03):** Governing rule for synthetic prompt export conditions.
5.  **S3 (2026-01):** Governing rule for WorkDash artifact restrictions.
6.  **S2 (2025-05):** Governing rule for public screenshot metadata.
7.  **S7 (2025-08):** Superseded by S8 regarding failed runs.
8.  **S1 (2024-11):** Superseded by S6 regarding artifact retention duration.

# Resolved Policy
### 1. Data Retention Policy
*   **Raw Private Prompts:** Must be retained locally until the user explicitly deletes them [S6].
*   **Failed/Invalid Runs:** Must be retained and clearly labeled to facilitate reliability analysis [S8].
*   **General Artifacts:** The 14-day retention limit is superseded by the "retain until deleted" rule for private prompts [S6].

### 2. Reporting and Metrics Policy
*   **Mandatory Metrics:** All model comparisons must report:
    *   Pass Rate [S5]
    *   Invalid-run Count [S5]
    *   Median Generation TPS [S5]
    *   MTP Acceptance [S5]
    *   Reasoning Tokens [S5]
    *   Final Tokens [S5]
    *   Output Artifacts [S5]
*   **Screenshot Requirements:** Public screenshots must include:
    *   Model Name [S2]
    *   Quantization [S2]
    *   Context Size [S2]
    *   Token Counts [S2]

### 3. Publication and Export Policy
*   **Publishable Reports:** Must use redacted summaries [S6].
*   **Public Screenshots:** May omit raw prompts [S2].
*   **Synthetic Prompt Export:** Permitted only if the prompts contain no:
    *   Real names [S4]
    *   Emails [S4]
    *   Teams messages [S4]
    *   Secrets [S4]
*   **WorkDash Artifacts:** These are strictly private and must never be published outside the home lab [S3].

# Contradictions
1.  **Retention of Failed Runs:** S7 (2025-08) suggested that all failed runs should be discarded. However, S8 (2026-06) explicitly mandates that failed and invalid runs should be retained and labeled. Because S8 is newer, the retention of failed runs is the active policy.
2.  **Artifact Retention Duration:** S1 (2024-11) stated that all benchmark artifacts should be retained for 14 days. S6 (2026-05) states that raw private prompts should be retained locally until explicitly deleted. Because S6 is newer, the "retain until deleted" rule takes precedence over the 14-day limit.

# Metrics To Report
The following metrics are required for all model comparisons [S5]:
*   **Pass Rate:** The percentage of successful completions.
*   **Invalid-run Count:** The total number of runs that failed or produced invalid output.
*   **Median Generation TPS:** The median speed of token generation.
*   **MTP Acceptance:** The rate of Maximum Token Probability acceptance.
*   **Reasoning Tokens:** The count of tokens generated during the reasoning phase.
*   **Final Tokens:** The count of tokens in the final output.
*   **Output Artifacts:** The actual files or text blocks produced by the model.

Additionally, for any public visual documentation (screenshots), the following must be included [S2]:
*   **Model Name**
*   **Quantization (Quant)**
*   **Context Size**
*   **Token Counts**

# What Must Stay Private
The following items are strictly prohibited from public disclosure or external publication:
1.  **WorkDash-derived Artifacts:** These must never leave the home lab [S3].
2.  **Raw Private Prompts:** These must be kept locally; only redacted summaries may be used in publishable reports [S6].
3.  **Personally Identifiable Information (PII) and Secrets:** Any synthetic prompts intended for export must be verified to contain no real names, emails, Teams messages, or secrets [S4].
4.  **Raw Prompts in Screenshots:** Public screenshots should omit the raw prompts [S2].

# Example Report Language
### Example 1: Model Comparison Report (Standard)
**Model Comparison Summary**
*   **Model Name:** [Model Name]
*   **Quantization:** [Quant]
*   **Context Size:** [Size]
*   **Pass Rate:** [X]%
*   **Invalid-run Count:** [Y]
*   **Median Generation TPS:** [Z]
*   **MTP Acceptance:** [A]%
*   **Reasoning Tokens:** [B]
*   **Final Tokens:** [C]
*   **Output Artifacts:** [Link/Reference]
*   **Prompt Summary:** [Redacted Summary of the prompt used for this test]

### Example 2: Public Screenshot Caption
**Screenshot Metadata:**
*   **Model:** [Name]
*   **Quant:** [Quant]
*   **Context:** [Size]
*   **Tokens:** [Count]
*(Note: Raw prompt omitted per policy S2)*

### Example 3: Synthetic Prompt Export Log
**Export Status:** Verified
**Content Check:** No real names, emails, Teams messages, or secrets detected.
**Source:** Synthetic Benchmark Suite [S4]

# Confidence
High. The policy was derived by strictly following the provided sources, resolving all chronological conflicts by preferring the newest dates (S8 > S7 and S6 > S1), and adhering to the specific constraints regarding WorkDash and redacted summaries.

# Detailed Operational Procedures for Data Lifecycle Management

To ensure the AI Flight Recorder home lab maintains a consistent and compliant data lifecycle, the following operational procedures are established based on the hierarchy of the provided synthetic sources. These procedures govern how data is ingested, stored, and eventually purged or transformed for public consumption.

### 1. Retention of Raw Private Prompts
The primary directive for raw private prompts is governed by source [S6]. Unlike the previous policy [S1] which suggested a 14-day retention window, the current policy mandates that raw private prompts must be retained locally until they are explicitly deleted by the user. This ensures that the home lab maintains a complete history of interactions for internal review and debugging. 

**Operational Steps:**
*   **Storage:** All raw prompts must be stored in a designated local directory that is not synced to public clouds.
*   **Deletion:** Data should only be removed when a user performs a manual deletion action.
*   **Conflict Resolution:** In any instance where a 14-day purge script is triggered, it must be overridden by the [S6] "retain until deleted" rule for private prompts.

### 2. Management of Failed and Invalid Runs
The handling of unsuccessful model outputs is a critical component of the lab's reliability testing. While an earlier draft [S7] suggested discarding failed runs, the current policy [S8] explicitly requires their retention. These runs are not considered "trash"; they are vital data points that reveal reliability problems and edge-case failures in the models being benchmarked.

**Operational Steps:**
*   **Labeling:** Every failed or invalid run must be clearly labeled with a "FAILURE" or "INVALID" tag.
*   **Analysis:** These runs should be grouped into a "Reliability Report" to identify patterns in model instability.
*   **Retention:** These runs are subject to the same "retain until deleted" rule as raw prompts [S6].

### 3. Artifact Management and WorkDash Restrictions
Artifacts are categorized into two types: general benchmark artifacts and Private WorkDash-derived artifacts.

*   **General Artifacts:** These are the outputs of standard benchmark runs. While [S1] suggested a 14-day retention, the [S6] rule for local retention takes precedence for private data.
*   **WorkDash Artifacts:** These are subject to the strictest privacy controls. According to [S3], any artifact derived from Private WorkDash must never be published or shared outside the home lab environment. This is a hard boundary; no amount of redaction can make a WorkDash-derived artifact "publishable" for external audiences.

# Publication Workflow and Redaction Guidelines

The transition from internal lab data to public-facing reports requires a rigorous multi-step workflow to ensure privacy and compliance with the established policies.

### 1. Redaction Standards for Reports
When preparing a report for public consumption, the "Raw Prompt" must never be included in its original form. Instead, the lab must use redacted summaries [S6]. 

**Redaction Guidelines:**
*   **Summary Content:** The summary should describe the *intent* of the prompt (e.g., "Requested a Python script for data visualization") without including the specific parameters or sensitive context of the original prompt.
*   **Verification:** A second reviewer should verify that the redacted summary contains no information that could be traced back to private user data or internal secrets.

### 2. Public Screenshot Protocols
Visual evidence is often necessary for public reports. However, screenshots must adhere to the specific metadata requirements outlined in [S2].

**Screenshot Requirements:**
*   **Omission:** Raw prompts must be omitted from the screenshot [S2].
*   **Mandatory Metadata:** Every screenshot must include a caption or overlay containing:
    *   Model Name [S2]
    *   Quantization Level (Quant) [S2]
    *   Context Size [S2]
    *   Total Token Counts [S2]

### 3. Synthetic Prompt Export Verification
The lab may export synthetic benchmark prompts for use by others, but only after a strict security audit. According to [S4], these prompts are only exportable if they are "clean."

**The Four-Point Export Check:**
Before any synthetic prompt is exported, it must be checked for the following four items:
1.  **Real Names:** Ensure no individual's name is present.
2.  **Emails:** Ensure no email addresses are present.
3.  **Teams Messages:** Ensure no snippets from internal communication platforms are present.
4.  **Secrets:** Ensure no API keys, passwords, or internal secrets are present.

If any of these four items are detected, the prompt must be scrubbed or discarded before export [S4].

# Model Comparison Reporting Framework

To ensure that model comparisons are scientifically rigorous and useful to the community, all reports must adhere to the standardized metric set defined in [S5].

### 1. Performance and Reliability Metrics
Every comparison must report the following core metrics:
*   **Pass Rate:** The percentage of prompts that were successfully completed by the model.
*   **Invalid-run Count:** The total number of runs that failed or produced non-compliant output. This provides context for the pass rate [S5].

### 2. Speed and Probability Metrics
To evaluate the efficiency of the inference engine and the model's behavior:
*   **Median Generation TPS:** The median Tokens Per Second, providing a clear picture of real-world performance.
*   **MTP Acceptance:** The Maximum Token Probability acceptance rate, which helps in understanding the model's confidence and sampling behavior [S5].

### 3. Tokenization and Output Analysis
To understand the "cost" and "logic" of the model's output:
*   **Reasoning Tokens:** The number of tokens generated during the model's internal reasoning process (if applicable).
*   **Final Tokens:** The number of tokens in the final output provided to the user.
*   **Output Artifacts:** The actual results (code, text, etc.) produced by the model [S5].

# Extended Example Reports

### Scenario A: Standard Model Comparison (Public Report)
**Model Comparison: Llama-3-70B vs. Mistral-Large**
*   **Model A:** Llama-3-70B (Q4_K_M) | **Context:** 32k | **Tokens:** 4,500
*   **Model B:** Mistral-Large (Q4_K_M) | **Context:** 32k | **Tokens:** 4,200
*   **Pass Rate (A):** 94% | **Pass Rate (B):** 91%
*   **Invalid-run Count (A):** 3 | **Invalid-run Count (B):** 4
*   **Median Generation TPS (A):** 45.2 | **Median Generation TPS (B):** 38.9
*   **MTP Acceptance (A):** 88% | **MTP Acceptance (B):** 82%
*   **Reasoning Tokens (A):** 120 | **Reasoning Tokens (B):** 0
*   **Final Tokens (A):** 4,380 | **Final Tokens (B):** 4,196
*   **Output Artifacts:** [Link to Redacted Artifacts Folder]
*   **Prompt Summary:** The model was asked to generate a complex multi-step data processing pipeline in Python. (Redacted per [S6])

### Scenario B: Failed Run Analysis (Internal Lab Log)
**Entry ID:** FR-2026-004
**Status:** FAILURE [S8]
**Model:** GPT-4o-mini
**Issue:** Model entered a repetitive loop and failed to provide a valid JSON object.
**Reasoning:** The prompt contained a highly nested logic structure that exceeded the model's coherent reasoning depth.
**Retention Status:** Retained locally until explicit deletion [S6].
**Reliability Note:** This failure highlights a limitation in handling nested logic for this specific quantization.

### Scenario C: Public Screenshot Log
**[Image: Screenshot of Model Output]**
**Metadata:**
*   **Model:** Mixtral-8x7B
*   **Quant:** Q8_0
*   **Context:** 32k
*   **Tokens:** 1,200
*(Note: Raw prompt has been omitted from this visual to comply with [S2])*

### Scenario D: Synthetic Prompt Export Verification
**Export ID:** SYN-009
**Source:** Synthetic Benchmark Suite
**Verification Checklist:**
*   Real Names: None Detected [S4]
*   Emails: None Detected [S4]
*   Teams Messages: None Detected [S4]
*   Secrets: None Detected [S4]
**Status:** APPROVED FOR EXPORT [S4]

# Compliance and Audit Checklist

Before any data is moved from the "Internal" state to the "Publishable" state, the following checklist must be completed. Failure to check all boxes constitutes a policy violation.

### Phase 1: Data Integrity (Internal)
- [ ] Have all raw private prompts been retained locally? [S6]
- [ ] Have all failed and invalid runs been labeled and retained for reliability analysis? [S8]
- [ ] Have all WorkDash-derived artifacts been excluded from the publication package? [S3]

### Phase 2: Redaction and Privacy (Pre-Publication)
- [ ] Has the raw prompt been replaced with a redacted summary? [S6]
- [ ] If screenshots are included, do they omit the raw prompt? [S2]
- [ ] Do all screenshots include Model Name, Quant, Context Size, and Token Counts? [S2]
- [ ] If synthetic prompts are being exported, have they been scrubbed of names, emails, Teams messages, and secrets? [S4]

### Phase 3: Metric Verification (Final Report)
- [ ] Does the report include the Pass Rate? [S5]
- [ ] Does the report include the Invalid-run Count? [S5]
- [ ] Does the report include the Median Generation TPS? [S5]
- [ ] Does the report include the MTP Acceptance? [S5]
- [ ] Does the report include the Reasoning Tokens? [S5]
- [ ] Does the report include the Final Tokens? [S5]
- [ ] Are the Output Artifacts included/linked? [S5]

# Policy Governance and Conflict Resolution Summary

To maintain the integrity of the AI Flight Recorder home lab, the following rules of interpretation apply to all policy conflicts:

1.  **Chronological Superiority:** In any instance where two sources provide conflicting instructions on the same topic, the source with the most recent date takes precedence. For example, the 2026-06 policy [S8] regarding the retention of failed runs overrides the 2025-08 draft [S7] which suggested discarding them. Similarly, the 2026-05 policy [S6] regarding "retain until deleted" overrides the 2024-11 policy [S1] regarding a 14-day retention window.
2.  **Specificity Superiority:** If two sources are of equal date, the more specific policy regarding the data type takes precedence.
3.  **Privacy Primacy:** In any situation where a publication rule [S2, S4, S5] conflicts with a privacy rule [S3, S6], the privacy rule must be followed. For instance, even if a model comparison requires "output artifacts" [S5], if those artifacts are WorkDash-derived, they must remain private [S3].
4.  **Redaction Requirement:** No raw private prompt may ever be published. The requirement for redacted summaries [S6] is an absolute constraint on all public-facing documentation.

# Detailed Metric Definitions for Model Comparisons [S5]

To ensure that the "Metrics To Report" section is interpreted consistently by all lab members, the following definitions are adopted:

*   **Pass Rate:** The ratio of successful completions to the total number of prompts attempted. A "success" is defined as a response that meets the predefined criteria of the benchmark prompt.
*   **Invalid-run Count:** The total number of attempts that resulted in a "Failure" status. This includes model crashes, infinite loops, refusal to answer (where not expected), or outputs that fail to meet the required format (e.g., invalid JSON).
*   **Median Generation TPS:** The median value of Tokens Per Second across all successful runs. The median is used rather than the mean to prevent outliers (extremely fast or extremely slow single runs) from skewing the performance data.
*   **MTP Acceptance:** The rate at which the model's Maximum Token Probability (MTP) was accepted during the generation process. This serves as a proxy for the model's internal confidence in its token selection.
*   **Reasoning Tokens:** The count of tokens generated by the model specifically for internal "thought" or "reasoning" blocks before the final answer is produced. This is critical for evaluating the efficiency of reasoning-heavy models.
*   **Final Tokens:** The count of tokens in the final output provided to the user, excluding any internal reasoning blocks.
*   **Output Artifacts:** The actual content produced by the model, such as code blocks, structured data, or long-form text. These should be provided in a way that respects the redaction rules of [S6].

# Data Security and Export Protocols [S3, S4]

The security of the home lab depends on the strict separation of internal data and external exports.

### WorkDash Isolation
The Private WorkDash environment is considered a high-security zone. Artifacts derived from this environment are strictly prohibited from leaving the home lab. This includes:
*   Raw logs.
*   Intermediate model outputs.
*   System prompts used within the WorkDash environment.
*   Any metadata associated with WorkDash runs.

### Synthetic Prompt Scrubbing
When exporting synthetic prompts for the purpose of benchmarking other systems, the following "Scrubbing Protocol" must be followed to satisfy [S4]:
1.  **Automated Scan:** Run a script to detect common PII patterns (emails, phone numbers).
2.  **Manual Review:** A human reviewer must check for "Teams messages" (which may not follow standard PII patterns) and "Secrets" (which may be embedded in code snippets).
3.  **Final Approval:** A "Clean" tag must be appended to the export file, certifying that the prompt contains no real names, emails, Teams messages, or secrets.

# Summary of Source Hierarchy and Application

| Source | Date | Key Policy | Status | Application |
| :--- | :--- | :--- | :--- | :--- |
| **S8** | 2026-06 | Retain/Label Failed Runs | **Active** | Overrides S7. |
| **S6** | 2026-05 | Retain until Deleted / Redact | **Active** | Overrides S1. |
| **S5** | 2026-04 | Comparison Metrics | **Active** | Standard for all reports. |
| **S4** | 2026-03 | Synthetic Export Rules | **Active** | Security check for exports. |
| **S3** | 2026-01 | WorkDash Privacy | **Active** | Hard exclusion for publication. |
| **S2** | 2025-05 | Screenshot Metadata | **Active** | Visual documentation rules. |
| **S7** | 2025-08 | Discard Failed Runs | **Superseded** | Replaced by S8. |
| **S1** | 2024-11 | 14-Day Retention | **Superseded** | Replaced by S6. |

# Final Policy Statement
The AI Flight Recorder home lab prioritizes the integrity of its data and the privacy of its users. By retaining all raw data until explicit deletion [S6], meticulously labeling and preserving failed runs for reliability analysis [S8], and enforcing strict redaction and metadata standards for public reports [S2, S6], the lab ensures a high-quality, secure, and transparent benchmarking environment. All WorkDash artifacts remain strictly internal [S3], and all exported synthetic prompts must be verified as clean of PII and secrets [S4]. All model comparisons must be reported using the standardized metric suite [S5].