# Answer

The publishable-private reporting policy for the AI Flight Recorder home lab is a tiered framework designed to maximize the utility of benchmark data while strictly safeguarding private and proprietary information. The policy mandates the retention of all run types—including failed and invalid runs—to ensure reliability analysis [S8]. It establishes a strict boundary between "Private" artifacts (such as WorkDash-derived data and raw private prompts) and "Publishable" artifacts (such as synthetic prompts and redacted summaries) [S3, S4, S6]. For public dissemination, the policy requires the omission of raw prompts in favor of redacted summaries and the inclusion of a comprehensive suite of performance metrics, including token counts, generation speed, and model specifications [S2, S5, S6].

# Source Priority

In accordance with the governing instructions, conflicts are resolved by prioritizing the newest policy first, followed by the most specific policy. The sources are prioritized as follows:

1.  **[S8] (2026-06):** Highest priority for run retention (overrides [S7]).
2.  **[S6] (2026-05):** Primary authority on raw prompt retention and report redaction.
3.  **[S5] (2026-04):** Primary authority on model comparison metrics.
4.  **[S4] (2026-03):** Primary authority on synthetic prompt exportability.
5.  **[S3] (2026-01):** Primary authority on WorkDash-derived artifacts.
6.  **[S8] (2026-06) / [S7] (2025-08):** Conflict resolution regarding failed runs (S8 wins).
7.  **[S2] (2025-05):** Authority on public screenshot requirements.
8.  **[S1] (2024-11):** Baseline for general artifact retention.

# Resolved Policy

The following resolved policy governs all activities within the AI Flight Recorder home lab.

### 1. Data Retention and Lifecycle Management
*   **General Artifacts:** All standard benchmark artifacts are subject to a baseline retention period of 14 days [S1].
*   **Raw Private Prompts:** Notwithstanding the general 14-day rule, raw private prompts must be retained locally and are not subject to automatic deletion; they must be kept until they are explicitly deleted by the user [S6].
*   **Failed and Invalid Runs:** All failed and invalid runs must be retained [S8]. These runs must be clearly labeled to ensure that reliability problems are visible and can be analyzed [S8]. This explicitly overrides any previous draft suggestions to discard failed runs [S7, S8].

### 2. Publication and Export Guidelines
*   **Strictly Private (Non-Publishable):**
    *   **WorkDash Artifacts:** Any artifacts derived from Private WorkDash must never be published outside the home lab environment [S3].
    *   **Raw Private Prompts:** Raw versions of private prompts are prohibited from publication [S6].
*   **Conditionally Publishable:**
    *   **Synthetic Prompts:** Synthetic benchmark prompts may be exported and published provided they contain no real names, no email addresses, no Teams messages, and no secrets [S4].
    *   **Redacted Summaries:** When reporting on private data, publishable reports must utilize redacted summaries rather than raw prompt text [S6].
    *   **Public Screenshots:** Screenshots intended for public view must omit raw prompts [S2].

### 3. Reporting Standards for Model Comparisons
When conducting and reporting model comparisons, the following data points must be included to ensure a complete technical profile [S2, S5]:
*   **Model Identification:** Model name and quantization level [S2].
*   **Configuration:** Context size [S2].
*   **Performance Metrics:**
    *   Pass rate [S5].
    *   Invalid-run count [S5].
    *   Median generation Tokens Per Second (TPS) [S5].
    *   MTP (Multi-Token Prediction) acceptance rate [S5].
*   **Token Analysis:**
    *   Reasoning tokens [S5].
    *   Final tokens [S5].
    *   Total token counts [S2].
*   **Supporting Evidence:** Output artifacts [S5].

# Contradictions

The following contradictions were identified and resolved based on the priority hierarchy (Newest $\rightarrow$ Most Specific):

*   **Failed Run Handling:** [S7] (2025-08) suggested that all failed runs should be discarded. However, [S8] (2026-06) explicitly mandates that failed and invalid runs be retained and labeled to reveal reliability problems. Because [S8] is the newer policy, the instruction to retain these runs is the active policy [S7, S8].
*   **Retention Duration:** [S1] (2024-11) establishes a general 14-day retention for benchmark artifacts. [S6] (2026-05) specifies that raw private prompts should be retained locally until explicitly deleted. Because [S6] is both newer and more specific to "raw private prompts," it overrides the 14-day general rule for that specific category of data [S1, S6].

# Metrics To Report

To comply with the AI Flight Recorder home lab standards, all reports must include the following metrics, categorized by their purpose:

### Model & Environment Specifications [S2]
*   **Model Name:** The specific identifier of the LLM being tested.
*   **Quantization:** The quantization method/bit-depth used (e.g., 4-bit, 8-bit).
*   **Context Size:** The maximum token window configured for the run.

### Execution & Reliability Metrics [S5, S8]
*   **Pass Rate:** The percentage of successful completions.
*   **Invalid-Run Count:** The number of runs that did not meet the validity criteria (must be retained and labeled per [S8]).
*   **Output Artifacts:** The actual generated results of the benchmark.

### Efficiency & Token Metrics [S2, S5]
*   **Median Generation TPS:** The median tokens per second during the generation phase.
*   **MTP Acceptance:** The rate at which Multi-Token Predictions were accepted.
*   **Reasoning Tokens:** The count of tokens used for internal chain-of-thought or reasoning.
*   **Final Tokens:** The count of tokens in the final output response.
*   **Total Token Counts:** The aggregate token usage for the interaction.

# What Must Stay Private

The following items are strictly prohibited from being published outside the home lab:

1.  **WorkDash-Derived Data:** Any and all artifacts originating from Private WorkDash [S3].
2.  **Raw Private Prompts:** The original, unredacted text of private prompts [S6].
3.  **Personally Identifiable Information (PII) & Secrets:** Any synthetic prompt that accidentally contains the following must be scrubbed or blocked from export [S4]:
    *   Real names.
    *   Email addresses.
    *   Teams messages.
    *   Secrets (API keys, passwords, etc.).
4.  **Raw Prompt Screenshots:** Any public-facing screenshot must have the raw prompt area removed or obscured [S2].

# Example Report Language

Below are examples of how to apply this policy in practice.

### Example 1: Public Model Comparison Table
*This table follows [S2] and [S5] for metrics and [S6] for prompt handling.*

| Metric | Model A (Q4_K_M) | Model B (Q8_0) |
| :--- | :--- | :--- |
| **Context Size** | 32k [S2] | 32k [S2] |
| **Pass Rate** | 88% [S5] | 92% [S5] |
| **Invalid-Run Count** | 12 [S5] | 5 [S5] |
| **Median Gen TPS** | 45.2 [S5] | 38.1 [S5] |
| **MTP Acceptance** | 0.72 [S5] | 0.68 [S5] |
| **Reasoning Tokens** | 450 [S5] | 610 [S5] |
| **Final Tokens** | 120 [S5] | 115 [S5] |
| **Total Token Count** | 570 [S2] | 725 [S2] |

**Prompt Summary (Redacted):**
"The model was asked to analyze a private dataset regarding internal project timelines and suggest optimizations. All specific project names and employee identifiers have been removed" [S6].

---

### Example 2: Internal Reliability Log
*This log follows [S8] and [S1/S6] for retention.*

**Run ID:** #9982-FAIL
**Status:** FAILED [S8]
**Label:** Reliability Issue - Context Overflow [S8]
**Retention Status:** Retained locally (Raw Private Prompt) [S6]
**Observation:** Model failed to terminate sequence, resulting in an invalid run. This run is retained to analyze the failure mode of the 4-bit quant [S8].

---

### Example 3: Export Request Evaluation
*This evaluation follows [S3] and [S4].*

**Request:** Export "Synthetic_Test_Set_v1.json"
**Review:**
*   Contains WorkDash artifacts? **No** [S3].
*   Contains real names? **No** [S4].
*   Contains emails? **No** [S4].
*   Contains Teams messages? **No** [S4].
*   Contains secrets? **No** [S4].
**Verdict:** APPROVED FOR EXPORT [S4].

# Confidence

**Confidence Score: 5/5**

**Reasoning:**
The policy has been constructed using only the provided synthetic sources [S1-S8]. All conflicts (S7 vs S8 and S1 vs S6) were resolved using the mandated priority logic (Newest $\rightarrow$ Most Specific). Every claim is cited to its respective source. The required sections are all present, and the content has been expanded to provide a comprehensive operational framework for the AI Flight Recorder home lab.