# Answer
The AI Flight Recorder home lab reporting policy is designed to balance rigorous performance tracking with strict data privacy and security. The policy mandates the retention of all raw private prompts until manual deletion [S6] and requires that failed or invalid runs be preserved and explicitly labeled to facilitate the identification of reliability issues [S8]. For public-facing outputs, the policy strictly prohibits the publication of raw private prompts, requiring the use of redacted summaries instead [S6]. 

Public screenshots are permitted but must be curated to include specific technical metadata—namely model name, quantization (quant), context size, and token counts—while ensuring raw prompts are omitted [S2]. When exporting synthetic benchmark prompts, the data must be scrubbed of all personally identifiable information (PII) and secrets, including real names, emails, Teams messages, or any other secrets [S4]. Furthermore, any artifacts derived from Private WorkDash must never leave the home lab environment [S3]. 

For model comparisons, the reporting standard requires a comprehensive suite of metrics: pass rate, invalid-run count, median generation tokens per second (TPS), MTP acceptance, reasoning tokens, final tokens, and the associated output artifacts [S5]. While general benchmark artifacts are typically retained for 14 days [S1], the specific retention of raw private prompts [S6] and failed runs [S8] takes precedence.

# Source Priority
To resolve conflicts within the provided sources, the following hierarchy was applied:
1.  **Recency:** Newer policies supersede older ones. For example, S8 (2026-06) overrides S7 (2025-08) regarding the handling of failed runs.
2.  **Specificity:** More specific policies override general ones. S6 (2026-05) provides specific instructions for "raw private prompts," which takes precedence over the general "benchmark artifacts" instruction in S1 (2024-11).
3.  **Source Chronology:**
    *   S1 (2024-11)
    *   S2 (2025-05)
    *   S7 (2025-08)
    *   S3 (2026-01)
    *   S4 (2026-03)
    *   S5 (2026-04)
    *   S6 (2026-05)
    *   S8 (2026-06)

# Resolved Policy

### 1. Data Retention Standards
*   **Raw Private Prompts:** These must be retained locally in the home lab until they are explicitly deleted by the user [S6].
*   **Failed and Invalid Runs:** These must not be discarded. They must be retained and clearly labeled to ensure that reliability problems can be analyzed [S8].
*   **General Benchmark Artifacts:** Unless otherwise specified by more specific rules (like those for raw prompts or failed runs), general artifacts should be retained for a period of 14 days [S1].

### 2. Publication and Export Rules
*   **Public Reports:** All publishable reports must use redacted summaries rather than raw private prompts [S6].
*   **Synthetic Prompt Export:** Synthetic benchmark prompts may be exported only if they are verified to contain no real names, emails, Teams messages, or secrets [S4].
*   **Public Screenshots:** Screenshots intended for public viewing may omit raw prompts but must include the following four data points:
    *   Model Name [S2]
    *   Quantization (Quant) [S2]
    *   Context Size [S2]
    *   Token Counts [S2]

### 3. Privacy and Security Constraints
*   **WorkDash Artifacts:** Any artifacts derived from Private WorkDash are strictly prohibited from being published or moved outside of the home lab [S3].
*   **PII Scrubbing:** Any data exported must be checked against a blacklist of prohibited content: real names, emails, Teams messages, and secrets [S4].

### 4. Reporting Requirements
*   **Model Comparison Metrics:** Every model comparison report must include the following seven metrics:
    1.  Pass rate [S5]
    2.  Invalid-run count [S5]
    3.  Median generation TPS (Tokens Per Second) [S5]
    4.  MTP (Multi-Token Prediction) acceptance [S5]
    5.  Reasoning tokens [S5]
    6.  Final tokens [S5]
    7.  Output artifacts [S5]

# Contradictions
The following contradictions were identified and resolved based on the priority rules:

*   **Conflict: Handling of Failed Runs**
    *   *Source S7 (2025-08):* States that all failed runs should be discarded.
    *   *Source S8 (2026-06):* States that failed and invalid runs should be retained and clearly labeled.
    *   *Resolution:* S8 is newer (2026-06 vs 2025-08) and therefore takes precedence. Failed runs must be retained.

*   **Conflict: Retention of Artifacts**
    *   *Source S1 (2024-11):* States all benchmark artifacts should be retained for 14 days.
    *   *Source S6 (2026-05):* States raw private prompts should be retained locally until explicitly deleted.
    *   *Resolution:* S6 is newer and more specific regarding "raw private prompts." Therefore, raw prompts are kept until deleted, while other general artifacts follow the 14-day rule of S1.

# Metrics To Report
In accordance with S5, all model comparison reports must include:
*   **Pass Rate:** The percentage of successful completions. [S5]
*   **Invalid-Run Count:** The total number of runs that failed or produced invalid output. [S5]
*   **Median Generation TPS:** The median speed of token generation. [S5]
*   **MTP Acceptance:** The acceptance rate for Multi-Token Prediction. [S5]
*   **Reasoning Tokens:** The count of tokens generated during the reasoning phase. [S5]
*   **Final Tokens:** The count of tokens in the final output. [S5]
*   **Output Artifacts:** The actual files or data generated by the model. [S5]

Additionally, public screenshots must include:
*   **Model Name** [S2]
*   **Quant** [S2]
*   **Context Size** [S2]
*   **Token Counts** [S2]

# What Must Stay Private
The following items are strictly prohibited from public disclosure or external export:
*   **WorkDash-derived Artifacts:** These must never be published outside the home lab [S3].
*   **Raw Private Prompts:** These must not be published; only redacted summaries are allowed [S6].
*   **PII and Secrets:** Any exported synthetic prompts must be free of:
    *   Real names [S4]
    *   Emails [S4]
    *   Teams messages [S4]
    *   Secrets [S4]

# Example Report Language
*The following is a template for a publishable report based on the resolved policies:*

**Model Comparison Report: [Model Name] vs [Model Name]**

**Technical Specifications (Screenshot Data):**
*   **Model:** [Model Name]
*   **Quantization:** [Quant]
*   **Context Size:** [Context Size]
*   **Token Counts:** [Total Tokens]

**Performance Metrics:**
*   **Pass Rate:** [X]%
*   **Invalid-Run Count:** [X]
*   **Median Generation TPS:** [X]
*   **MTP Acceptance:** [X]%
*   **Reasoning Tokens:** [X]
*   **Final Tokens:** [X]
*   **Output Artifacts:** [Link/Reference to Artifacts]

**Summary of Execution:**
[Insert Redacted Summary of the prompt and response here - No raw prompts allowed] [S6]

**Reliability Notes:**
[Note: Failed runs were retained and labeled for internal review. See internal log for reliability analysis.] [S8]

**Data Integrity Statement:**
All synthetic prompts used in this benchmark have been scrubbed of real names, emails, Teams messages, and secrets. [S4] No WorkDash-derived artifacts are included in this public report. [S3]

# Confidence
High. The policy was derived by strictly following the provided sources and applying the requested conflict resolution logic (recency and specificity). All requirements for metrics, privacy, and retention were addressed using the provided source IDs.

### Detailed Operational Guidelines and Implementation Framework

#### 1. Data Classification and Governance Matrix
To ensure consistent application of the AI Flight Recorder policies, all data generated during benchmark activities must be classified into one of the following categories. Each category carries specific retention and publication rules derived from the source policies.

*   **Category A: Raw Private Prompts**
    *   **Definition:** The original, unedited input provided by a user or the system to the model.
    *   **Retention Rule:** These must be retained locally within the home lab environment until they are explicitly deleted by an authorized user [S6]. They are exempt from the standard 14-day deletion cycle [S1].
    *   **Publication Rule:** Strictly prohibited. These must never appear in public reports or screenshots. Instead, they must be replaced with redacted summaries [S6].

*   **Category B: Failed and Invalid Runs**
    *   **Definition:** Any execution where the model produced an error, crashed, timed out, or produced output that failed to meet the predefined success criteria.
    *   **Retention Rule:** These must be retained and clearly labeled [S8]. This is a priority over previous drafts that suggested discarding such data [S7].
    *   **Publication Rule:** These are primarily for internal reliability analysis. If included in a report, they must be clearly labeled as failures to highlight reliability problems [S8].

*   **Category C: WorkDash-Derived Artifacts**
    *   **Definition:** Any data, logs, outputs, or files generated by or derived from the Private WorkDash system.
    *   **Retention Rule:** These must remain within the home lab environment.
    *   **Publication Rule:** These must never be published or moved outside the home lab [S3]. This is a hard security boundary.

*   **Category D: General Benchmark Artifacts**
    *   **Definition:** Standard logs, temporary files, and non-sensitive metadata generated during the benchmarking process that do not fall into the categories above.
    *   **Retention Rule:** These should be retained for a period of 14 days [S1].
    *   **Publication Rule:** Subject to standard reporting guidelines.

*   **Category E: Synthetic Benchmark Prompts**
    *   **Definition:** Prompts generated synthetically for the purpose of benchmarking.
    *   **Retention Rule:** Standard retention applies.
    *   **Publication/Export Rule:** These may be exported only if they are verified to contain no real names, emails, Teams messages, or secrets [S4].

#### 2. Procedural Workflow for Report Generation
When a lab operator is tasked with creating a publishable report, they must follow this step-by-step workflow to ensure compliance with all sources:

**Step 1: Data Collection and Labeling**
*   Capture the raw private prompt and store it in the local "Raw_Prompts" directory [S6].
*   Execute the benchmark and capture all output artifacts [S5].
*   Identify any failed or invalid runs. Immediately move these to a "Reliability_Analysis" folder and apply a clear label describing the failure (e.g., "Invalid_JSON," "Timeout," "Hallucination") [S8].

**Step 2: Data Scrubbing and Redaction**
*   Review the raw private prompts. Create a "Redacted Summary" for each prompt that describes the intent of the prompt without revealing the specific private details [S6].
*   If the report involves synthetic prompts intended for export, run a PII scrub to ensure no real names, emails, Teams messages, or secrets are present [S4].

**Step 3: Metric Compilation**
*   Calculate and aggregate the following seven metrics for the model comparison:
    1.  Pass Rate [S5]
    2.  Invalid-Run Count [S5]
    3.  Median Generation TPS [S5]
    4.  MTP Acceptance [S5]
    5.  Reasoning Tokens [S5]
    6.  Final Tokens [S5]
    7.  Output Artifacts [S5]

**Step 4: Visual Asset Preparation**
*   Capture screenshots of the model's output.
*   Ensure the raw prompt is cropped out or omitted from the screenshot [S2].
*   Overlay or include a text block in the screenshot containing:
    *   Model Name [S2]
    *   Quantization (Quant) [S2]
    *   Context Size [S2]
    *   Token Counts [S2]

**Step 5: Final Review and Publication**
*   Verify that no WorkDash-derived artifacts are included in the final document [S3].
*   Confirm that the report uses the redacted summaries rather than raw prompts [S6].
*   Publish the report to the designated public channel.

#### 3. Detailed Metric Definitions (S5)
To ensure that "Model Comparisons" are consistent and scientifically valid, the following definitions must be used when reporting the metrics required by S5:

*   **Pass Rate:** The percentage of total runs that successfully met the success criteria. (Formula: [Successful Runs / Total Runs] * 100).
*   **Invalid-Run Count:** The total number of runs that resulted in an error, a crash, or an output that was flagged as invalid by the evaluation script.
*   **Median Generation TPS:** The median number of tokens generated per second. Using the median rather than the mean helps account for outliers caused by network latency or specific complex prompts.
*   **MTP Acceptance:** The acceptance rate of Multi-Token Prediction. This measures the model's ability to correctly predict multiple tokens in a single forward pass, which is a key indicator of inference efficiency.
*   **Reasoning Tokens:** The count of tokens generated during the model's internal reasoning phase (e.g., "thought" blocks). This helps distinguish between the "thinking" effort and the final output.
*   **Final Tokens:** The count of tokens in the final response provided to the user, excluding any reasoning or internal thought tokens.
*   **Output Artifacts:** The actual files, code blocks, or data structures produced by the model. These should be linked or included as appendices to the report.

#### 4. Security and Privacy Protocols
The AI Flight Recorder maintains a strict security posture to protect both the home lab's integrity and the privacy of the data processed within it.

**WorkDash Isolation (S3)**
The Private WorkDash system is considered a high-security zone. Any artifact derived from this system—whether it be a log file, a generated summary, or a model output—is strictly prohibited from being published outside the home lab. Operators must double-check all report components to ensure no WorkDash data has been inadvertently included.

**PII and Secret Scrubbing (S4)**
Before any synthetic benchmark prompts are exported from the home lab, they must be audited against a "Prohibited Content" list. This list includes:
*   **Real Names:** Any full names or identifying personal names.
*   **Emails:** Any email addresses or patterns resembling email addresses.
*   **Teams Messages:** Any text that appears to be a direct message or conversation from Microsoft Teams.
*   **Secrets:** Any API keys, passwords, tokens, or other sensitive credentials.
If any of these are found, the prompt must be modified or discarded before export.

**Prompt Redaction (S6)**
The policy regarding raw private prompts is designed to allow for internal debugging while maintaining public privacy. While the raw prompt is kept locally until the user decides to delete it [S6], the public-facing report must only contain a redacted summary. This summary should describe the *type* of task (e.g., "Code generation for a web scraper") without revealing the specific private parameters or data used in the original prompt.

#### 5. Retention Schedule Summary
The following table summarizes the retention requirements for different types of data produced in the home lab:

| Data Type | Retention Period | Action on Expiration | Source |
| :--- | :--- | :--- | :--- |
| **General Benchmark Artifacts** | 14 Days | Automatic Deletion | [S1] |
| **Raw Private Prompts** | Until Explicit Deletion | Manual Deletion Only | [S6] |
| **Failed/Invalid Runs** | Indefinite | Retain and Label | [S8] |
| **WorkDash Artifacts** | Indefinite | Keep in Home Lab | [S3] |
| **Synthetic Prompts** | As needed | Export if Scrubbed | [S4] |
| **Public Screenshots** | As needed | Retain for Report | [S2] |
| **Model Comparison Metrics** | As needed | Retain for Report | [S5] |

#### 6. Conflict Resolution Log
To ensure transparency in how this policy was constructed, the following conflicts were identified and resolved:

*   **Conflict 1: Failed Run Management**
    *   *Source S7 (2025-08):* Suggested discarding failed runs.
    *   *Source S8 (2026-06):* Mandates retaining and labeling failed runs.
    *   *Resolution:* S8 is newer (2026-06 vs 2025-08) and therefore takes precedence. The policy follows S8.

*   **Conflict 2: Artifact Retention Duration**
    *   *Source S1 (2024-11):* General 14-day retention for all benchmark artifacts.
    *   *Source S6 (2026-05):* Specific retention for raw private prompts (until deleted).
    *   *Resolution:* S6 is newer and more specific than S1. Therefore, raw private prompts are exempt from the 14-day rule and follow the "until deleted" rule.

*   **Conflict 3: Prompt Visibility in Reports**
    *   *Source S2 (2025-05):* Mentions omitting raw prompts in screenshots.
    *   *Source S6 (2026-05):* Mentions using redacted summaries in reports.
    *   *Resolution:* These are complementary. S2 applies to visual screenshots, while S6 applies to the text of the report. Both are required.

#### 7. Expanded Example Report Language
*The following examples demonstrate how to apply the resolved policies in different scenarios.*

**Example A: Standard Model Comparison (Public)**
"**Model Comparison: Llama-3-70B vs. Mistral-Large**
**Technical Specs:**
- Model: Llama-3-70B
- Quant: 4-bit GGUF
- Context: 32k
- Tokens: 1,240
**Metrics:**
- Pass Rate: 88%
- Invalid-Run Count: 4
- Median Generation TPS: 45.2
- MTP Acceptance: 92%
- Reasoning Tokens: 150
- Final Tokens: 1,090
- Output Artifacts: [Link to Artifact_01.py]
**Summary:** The user requested a script to automate the parsing of CSV files into a SQL database. The model successfully generated the logic, though some invalid runs occurred due to complex nested loops. [S6]
**Reliability Note:** Failed runs were retained and labeled for internal analysis to identify specific logic failures. [S8]"

**Example B: Internal Reliability Audit (Private)**
"**Internal Log: Model X Failure Analysis**
**Run ID:** 99283
**Status:** INVALID
**Label:** JSON_FORMAT_ERROR [S8]
**Raw Prompt:** [Retained in local database per S6]
**Failure Analysis:** The model failed to close the final bracket in the JSON object. This run is retained for reliability tracking to determine if the issue is related to the 4-bit quantization or the specific prompt structure. [S8]"

**Example C: Synthetic Prompt Export Verification**
"**Export Log: Synthetic Dataset_04**
**Verification Status:** PASSED
**Scrubbing Check:**
- Real Names: None detected [S4]
- Emails: None detected [S4]
- Teams Messages: None detected [S4]
- Secrets: None detected [S4]
**Action:** Dataset approved for export outside the home lab. [S4]"

#### 8. Compliance Checklist for Lab Operators
Before any data leaves the home lab or is published, the operator must check off the following:

1.  **Retention Check:** Are all general artifacts scheduled for deletion in 14 days? [S1]
2.  **Prompt Check:** Are raw private prompts stored locally and not included in the public report? [S6]
3.  **Failure Check:** Are all failed/invalid runs labeled and saved in the reliability folder? [S8]
4.  **WorkDash Check:** Is there any WorkDash-derived data in this report? (If yes, remove it). [S3]
5.  **Export Check:** Have all synthetic prompts been scrubbed of names, emails, Teams messages, and secrets? [S4]
6.  **Metric Check:** Does the comparison report include all seven required metrics (Pass Rate, Invalid-Run Count, Median TPS, MTP Acceptance, Reasoning Tokens, Final Tokens, Output Artifacts)? [S5]
7.  **Screenshot Check:** Do all screenshots omit raw prompts and include Model Name, Quant, Context Size, and Token Counts? [S2]

#### 9. Summary of Source Application
*   **S1 (2024-11):** Establishes the baseline 14-day retention for general artifacts.
*   **S2 (2025-05):** Defines the metadata requirements for public screenshots.
*   **S3 (2026-01):** Establishes the hard security boundary for WorkDash artifacts.
*   **S4 (2026-03):** Defines the safety gate for exporting synthetic prompts.
*   **S5 (2026-04):** Defines the mandatory metrics for model comparisons.
*   **S6 (2026-05):** Defines the retention and redaction rules for private prompts.
*   **S7 (2025-08):** Superseded by S8 regarding failed runs.
*   **S8 (2026-06):** Mandates the retention and labeling of failed/invalid runs for reliability.

By adhering to this policy, the AI Flight Recorder ensures that it maintains a high standard of data integrity, security, and performance transparency while protecting the privacy of the home lab environment.