# Answer The AI Flight Recorder home lab operates under a comprehensive reporting and data management policy designed to balance the need for detailed performance analysis with strict privacy and security requirements. The policy establishes clear guidelines for data retention, the handling of failed runs, the specific metrics required for model comparisons, and the protocols for public versus private data dissemination. The core of the policy dictates that while raw data is preserved for internal analysis, any information shared outside the home lab must be strictly sanitized. This includes the use of redacted summaries for publishable reports and the exclusion of specific identifiers like real names or secrets. Furthermore, the policy prioritizes the retention of "failed" data, recognizing its value in diagnosing reliability issues, and mandates a specific set of technical metrics to ensure objective model comparisons. # Source Priority To resolve conflicts within the provided documentation, the following hierarchy was applied: 1. **Recency:** Newer policies supersede older ones (e.g., S8 [2026-06] overrides S7 [2025-08]). 2. **Specificity:** More specific instructions override general ones when dates are equal or when the specific instruction clarifies a general rule (e.g., S6 [2026-05] provides specific instructions for "publishable reports" which clarifies the general export rules in S4 [2026-03]). # Resolved Policy ### 1. Data Retention and Lifecycle * **Raw Private Prompts:** All raw private prompts must be retained locally within the home lab until they are explicitly deleted by the user [S6]. This overrides the older 14-day retention rule [S1]. * **Failed and Invalid Runs:** All failed and invalid runs must be retained and clearly labeled [S8]. These are preserved specifically because they provide critical insights into reliability problems [S8]. This overrides the draft instruction to discard failed runs [S7]. * **Benchmark Artifacts:** While a general rule exists for 14-day retention [S1], the more recent and specific instruction for raw private prompts [S6] takes precedence for those specific items. ### 2. Publication and Export Protocols * **Publishable Reports:** Any report intended for publication must use redacted summaries rather than raw private prompts [S6]. * **Synthetic Benchmark Prompts:** These prompts may be exported only if they are verified to contain no real names, emails, Teams messages, or secrets [S4]. * **WorkDash Artifacts:** Any artifacts derived from Private WorkDash must never be published outside of the home lab environment [S3]. * **Public Screenshots:** When sharing screenshots publicly, the following rules apply: * Raw prompts must be omitted [S2]. * The following metadata must be included: Model name, quantization (quant), context size, and token counts [S2]. ### 3. Model Comparison Standards When performing model comparisons, the following metrics are mandatory for inclusion in the report [S5]: * Pass rate * Invalid-run count * Median generation TPS (Tokens Per Second) * MTP (Multi-Token Prediction) acceptance * Reasoning tokens * Final tokens * Output artifacts # Contradictions The following contradictions were identified and resolved during the policy construction: * **Retention of Failed Runs:** S7 (2025-08) suggested that all failed runs should be discarded. However, S8 (2026-06) explicitly states that failed and invalid runs should be retained and labeled to reveal reliability problems. **Resolution:** S8 was chosen as it is the newer policy. * **Retention Period of Artifacts:** S1 (2024-11) states that all benchmark artifacts should be retained for 14 days. S6 (2026-05) states that raw private prompts should be retained locally until explicitly deleted. **Resolution:** S6 was chosen as it is the newer policy and provides a more specific instruction for private prompts. * **Prompt Exportation vs. Redaction:** S4 (2026-03) allows for the export of synthetic prompts if they are sanitized of PII/secrets. S6 (2026-05) requires redacted summaries for publishable reports. **Resolution:** These are treated as complementary but S6 is the governing rule for the final "publishable" state, while S4 governs the "exportable" state of synthetic data. # Metrics To Report For every model comparison conducted within the AI Flight Recorder home lab, the following metrics must be reported [S5]: 1. **Pass Rate:** The percentage of successful completions. 2. **Invalid-Run Count:** The total number of runs that failed or produced invalid output. 3. **Median Generation TPS:** The median tokens per second during the generation phase. 4. **MTP Acceptance:** The acceptance rate of Multi-Token Predictions. 5. **Reasoning Tokens:** The count of tokens generated during the reasoning phase. 6. **Final Tokens:** The count of tokens in the final output. 7. **Output Artifacts:** The resulting artifacts produced by the model. Additionally, for any public screenshots provided as part of the reporting, the following must be included [S2]: 1. Model Name 2. Quantization (Quant) 3. Context Size 4. Token Counts # What Must Stay Private The following items are strictly prohibited from being published outside the home lab: * **WorkDash Artifacts:** Any artifacts derived from Private WorkDash [S3]. * **Raw Private Prompts:** These must remain local and be replaced by redacted summaries in any public-facing documentation [S6]. * **Personally Identifiable Information (PII) and Secrets:** Any synthetic prompts exported must be scrubbed of: * Real names [S4] * Emails [S4] * Teams messages [S4] * Secrets [S4] # Example Report Language *The following is a template for a publishable model comparison report based on the resolved policies.* --- **Model Comparison Report: [Model Name]** **Configuration:** - **Quantization:** [Quant] - **Context Size:** [Context Size] - **Token Counts:** [Total Token Counts] **Performance Metrics:** - **Pass Rate:** [X]% - **Invalid-Run Count:** [Y] - **Median Generation TPS:** [Z] - **MTP Acceptance:** [A]% - **Reasoning Tokens:** [B] - **Final Tokens:** [C] - **Output Artifacts:** [Description of artifacts] **Reliability Analysis:** [Note: Failed and invalid runs were retained and analyzed to identify reliability problems. See labeled logs in internal lab records.] [S8] **Summary of Results:** [Insert Redacted Summary here. Do not include raw private prompts.] [S6] **Visual Evidence:** [Insert Screenshot here. Ensure raw prompts are omitted. Ensure Model Name, Quant, Context Size, and Token Counts are visible.] [S2] --- # Confidence High. The policy was derived by strictly following the provided sources and applying the requested conflict resolution logic (Newer > Specific). All requirements for metrics, privacy, and retention were addressed using the provided source IDs. **Detailed Implementation Guidelines and Standard Operating Procedures (SOPs)** ### 1. Detailed Metric Definitions and Reporting Standards [S5, S8] To ensure consistency across all model evaluations within the AI Flight Recorder home lab, the following metrics must be reported for every model comparison. Each metric serves a specific purpose in evaluating the performance, efficiency, and reliability of the models being tested. * **Pass Rate:** This metric represents the percentage of benchmark attempts that successfully met the predefined success criteria. It is the primary indicator of a model's ability to follow instructions and complete tasks accurately. * **Invalid-Run Count:** This is the total number of runs that were identified as failed or produced invalid output. In accordance with policy [S8], these runs must not be discarded. Instead, they must be retained and clearly labeled. The purpose of retaining these runs is to provide a transparent view of the model's reliability problems, allowing for a deeper analysis of where and why the model fails. * **Median Generation TPS (Tokens Per Second):** This metric measures the speed of the model's generation phase. By reporting the median rather than the mean, the lab accounts for outliers and provides a more representative view of the typical generation speed. * **MTP (Multi-Token Prediction) Acceptance:** This metric tracks the acceptance rate of Multi-Token Predictions. It is a critical technical metric for understanding the efficiency of the model's inference process. * **Reasoning Tokens:** This is the specific count of tokens generated during the model's internal reasoning phase. This allows for a comparison of how much "thought" or internal processing a model performs before producing a final answer. * **Final Tokens:** This is the count of tokens in the final output provided to the user. This is distinct from reasoning tokens and represents the actual content of the response. * **Output Artifacts:** The report must include a description or a representation of the artifacts produced by the model. This provides qualitative context to the quantitative metrics listed above. ### 2. Data Retention and Lifecycle Management [S1, S6, S8] The management of data within the home lab is governed by specific retention rules that prioritize the preservation of useful data for analysis while maintaining a manageable storage footprint. * **Raw Private Prompts:** All raw private prompts must be retained locally within the home lab environment. These prompts are to be kept until they are explicitly deleted by the user [S6]. This policy takes precedence over general artifact retention rules, ensuring that the original input data remains available for long-term internal review. * **Failed and Invalid Runs:** As established in policy [S8], failed and invalid runs are considered high-value data for reliability analysis. They must be retained and clearly labeled. This directly contradicts and supersedes the draft policy [S7] which suggested that failed runs should be discarded. * **General Benchmark Artifacts:** For all other benchmark artifacts that do not fall under the "raw private prompt" or "failed/invalid run" categories, a general retention period of 14 days applies [S1]. This ensures that the lab does not accumulate unnecessary data while still providing a window for immediate post-benchmark review. ### 3. Privacy and Security Compliance Framework [S3, S4, S6] Security and privacy are paramount when handling data within the AI Flight Recorder home lab. The following protocols must be strictly followed to prevent the leakage of sensitive information. * **WorkDash Artifact Isolation:** Any artifacts derived from Private WorkDash are strictly classified as internal. These artifacts must never be published or shared outside of the home lab environment [S3]. This is a hard boundary for data dissemination. * **Synthetic Prompt Exportation:** Synthetic benchmark prompts may be exported for use in other environments only if they undergo a rigorous sanitization check. Before export, prompts must be verified to ensure they contain no: * Real names [S4] * Emails [S4] * Teams messages [S4] * Secrets [S4] * **Publishable Report Redaction:** When preparing reports for any audience outside of the internal lab, raw private prompts must never be included. Instead, the policy requires the use of redacted summaries [S6]. These summaries should provide enough context to understand the test case without revealing the specific private content of the prompt. ### 4. Public Reporting and Visual Documentation Standards [S2, S6] When the lab produces public-facing content, such as screenshots or summary reports, the following standards apply to ensure both clarity and privacy. * **Screenshot Requirements:** Public screenshots are permitted but must adhere to strict content rules: * Raw prompts must be omitted from the screenshot [S2]. * The screenshot must clearly display the following metadata: * Model Name [S2] * Quantization (Quant) [S2] * Context Size [S2] * Token Counts [S2] * **Summary Standards:** All publishable reports must utilize redacted summaries [S6]. This ensures that the core findings of the benchmark are communicated while the private nature of the prompts is protected. ### 5. Synthetic Data Generation and Export Protocols [S4] The creation and movement of synthetic data are governed by the "No PII/Secrets" rule. * **Verification Process:** Before any synthetic benchmark prompt is cleared for export, a manual or automated check must be performed to ensure the absence of real names, emails, Teams messages, or secrets [S4]. * **Purpose:** This allows the lab to contribute to the broader AI community by sharing synthetic prompts while maintaining a zero-leakage policy regarding internal communications and personal data. ### 6. WorkDash Artifact Isolation Procedures [S3] The WorkDash environment is considered a high-security zone for artifacts. * **Publication Prohibition:** There is a total prohibition on the publication of any WorkDash-derived artifacts outside the home lab [S3]. * **Data Handling:** Any data moving from WorkDash into the reporting pipeline must be checked against the "Publishable" criteria (redaction and summary) to ensure that no WorkDash-specific artifacts are inadvertently included in public reports. ### 7. Standard Operating Procedure (SOP) for Model Comparison [S5] To ensure every model comparison is performed to the same standard, the following SOP must be followed: 1. **Execution:** Run the benchmark and capture all output, including reasoning tokens and final tokens [S5]. 2. **Data Collection:** Record the pass rate and the count of invalid runs [S5]. 3. **Performance Logging:** Calculate the median generation TPS and the MTP acceptance rate [S5]. 4. **Artifact Preservation:** Save all output artifacts and ensure that failed/invalid runs are clearly labeled for reliability analysis [S8]. 5. **Reporting:** Compile the data into a report including all metrics mandated by [S5]. ### 8. Standard Operating Procedure (SOP) for Report Publication [S2, S4, S6] To move a report from "Internal" to "Publishable," the following SOP must be followed: 1. **Content Audit:** Review the report to ensure no WorkDash-derived artifacts are present [S3]. 2. **Prompt Redaction:** Replace all raw private prompts with redacted summaries [S6]. 3. **Synthetic Check:** If synthetic prompts are included, verify they contain no names, emails, Teams messages, or secrets [S4]. 4. **Visual Prep:** If screenshots are included, crop or mask the raw prompts and ensure the model name, quant, context size, and token counts are visible [S2]. 5. **Final Approval:** Confirm that the report meets all criteria before it leaves the home lab. ### 9. Internal Audit Checklist for Data Integrity [S1, S3, S4, S6, S8] The following checklist should be used during periodic audits of the home lab's data management: * [ ] Are raw private prompts being retained locally until explicit deletion? [S6] * [ ] Are failed and invalid runs being retained and clearly labeled? [S8] * [ ] Are benchmark artifacts older than 14 days being reviewed for deletion? [S1] * [ ] Is there any WorkDash-derived data in the public-facing folders? (Should be No) [S3] * [ ] Have all exported synthetic prompts been scrubbed of PII and secrets? [S4] * [ ] Do all publishable reports use redacted summaries instead of raw prompts? [S6] ### 10. Detailed Scenario Analysis for Reliability Reporting [S8] The policy regarding failed runs [S8] is designed to facilitate "Reliability Engineering." * **Scenario A: Model Hallucination.** If a model produces a factually incorrect but structurally sound response, it should be labeled as a "Failed Run" and retained. * **Scenario B: Model Crash/Timeout.** If a model fails to produce any output, it is an "Invalid Run." These must be retained and labeled to identify hardware or software bottlenecks. * **Scenario C: Instruction Following Failure.** If a model ignores a core constraint, it is a "Failed Run." Retaining these allows the lab to track the "Reliability Problem" over time as models are updated. ### 11. Comprehensive Redaction Guidelines for Publishable Summaries [S6] When creating redacted summaries for publishable reports, the following guidelines apply: * **Context Preservation:** The summary must describe the *intent* of the prompt (e.g., "The model was asked to summarize a complex technical document") without revealing the *content* of the document. * **Privacy Shielding:** Any specific names, locations, or unique identifiers must be replaced with generic placeholders (e.g., [User A], [Company X]). * **Consistency:** Ensure that the redacted summary still allows the reader to understand why the model succeeded or failed in the given test case. ### 12. Metadata Requirements for Visual Evidence [S2] Every screenshot used in a public report must be accompanied by a metadata block or overlay. * **Model Name:** The exact identifier of the model (e.g., "Llama-3-70B"). * **Quantization:** The specific bit-depth or quantization method used (e.g., "4-bit GGUF"). * **Context Size:** The active context window used during the run (e.g., "32k"). * **Token Counts:** The total number of tokens processed or generated in the screenshot. * **Prompt Exclusion:** The actual text of the prompt must be completely obscured or removed from the frame. ### 13. Glossary of Terms * **Artifacts:** Any files, outputs, or data structures generated during a benchmark run [S1, S3, S5]. * **Failed Run:** A benchmark attempt that did not meet the success criteria [S8]. * **Invalid Run:** A benchmark attempt that resulted in an error, crash, or non-parsable output [S8]. * **Private WorkDash:** A specific internal tool/environment whose artifacts are strictly protected [S3]. * **Raw Private Prompt:** The original, unedited input provided by a user or the lab for a benchmark [S6]. * **Redacted Summary:** A high-level description of a prompt that removes all private or sensitive information [S6]. * **Synthetic Benchmark Prompt:** A prompt generated artificially for testing purposes, which may be exported if sanitized [S4]. * **TPS (Tokens Per Second):** A measure of the speed of the model's text generation [S5]. * **MTP Acceptance:** The rate at which the model's multi-token predictions are accepted during inference [S5]. ### 14. Final Compliance Statement All personnel operating within the AI Flight Recorder home lab are required to adhere to these policies. Failure to redact private prompts [S6], the publication of WorkDash artifacts [S3], or the deletion of failed runs [S8] constitutes a violation of the lab's data integrity protocols. The priority of these rules is determined by the most recent policy date and the specificity of the instruction provided in the source documentation.