# Summary On 2026‑05‑14, the AI benchmark harness executed a full run of the 27‑task evaluation suite for the new GPT‑4‑Turbo model. The harness reported a single pass out of 27, implying that the model had successfully met the acceptance criteria for only one task. However, a subsequent manual audit revealed that the model had largely produced only intermediate reasoning content (the `reasoning_content` field) and had failed to emit a final answer (`message.content`) for the majority of tasks. The token budgets allocated for each task were insufficient to accommodate the full reasoning and final answer, causing the harness to truncate the output prematurely. The harness’s reporting mechanisms did not clearly expose the token usage, the presence or absence of final content, or the “invalid‑run” warnings that were generated when the model exceeded its token budget. Consequently, the harness’s pass/fail metric was misleading, and the benchmark results were not representative of the model’s true capabilities. The incident was identified by a cross‑team review that compared the harness logs against the raw artifact files stored in the artifact repository. The review uncovered that the harness had not captured the full reasoning and final answer for 26 of the 27 tasks. The root cause was a combination of model‑side failures (the model’s inability to produce final content within the token budget) and harness‑side failures (the harness’s inadequate reporting and artifact handling). The fix involved increasing the token budgets for each task, adding artifact storage for the full reasoning and final content, and enhancing the harness’s reporting to display token usage, MTP acceptance, and invalid‑run warnings in a clear, actionable format. The postmortem below documents the impact, timeline, root causes, detection gaps, corrective actions, preventive tests, dashboard changes, remaining risks, and an owner checklist to ensure that similar incidents are prevented in the future. # Impact The immediate impact of the incident was a mis‑representation of the model’s performance: the harness reported a 3.7 % pass rate (1/27) while the model actually produced reasoning for 26 tasks but failed to produce final answers due to token budget constraints. This mis‑representation had several downstream effects: 1. **Stakeholder Confidence** – Product managers and external partners were led to believe that the model was severely under‑performing, potentially delaying product releases and affecting trust in the model’s reliability. 2. **Benchmark Integrity** – The benchmark results were used in a whitepaper and internal metrics dashboard. The inaccurate pass/fail count could have skewed the model’s ranking relative to competitors, affecting strategic decisions. 3. **Model Development** – The engineering team used the harness output to guide fine‑tuning and prompt engineering. The misleading results caused them to focus on the wrong aspects of the model, potentially wasting resources. 4. **Operational Costs** – The harness executed 27 tasks with a token budget that was too low. The truncated outputs required re‑runs, increasing compute costs and time to completion. 5. **Compliance** – The benchmark was part of a regulatory audit. The inaccurate results could have led to compliance violations if the model was deployed in regulated environments. The impact was mitigated by a manual audit and the subsequent fix, but the incident highlighted the need for robust reporting and token budgeting in the harness. # Timeline - **2026‑05‑14 09:00 UTC** – Harness initiates the 27‑task benchmark run for GPT‑4‑Turbo. - **2026‑05‑14 09:05 UTC** – Harness logs a single pass out of 27. - **2026‑05‑14 09:15 UTC** – Harness completes the run and writes the summary report to the artifact repository. - **2026‑05‑14 10:00 UTC** – Product team reviews the benchmark results and flags the low pass rate. - **2026‑05‑14 10:30 UTC** – Engineering team initiates a manual audit of the harness logs. - **2026‑05‑14 11:00 UTC** – Audit reveals that the model produced reasoning content but no final answer for 26 tasks. - **2026‑05‑14 11:30 UTC** – Root cause analysis identifies token budget constraints and reporting gaps. - **2026‑05‑14 12:00 UTC** – Fixes are implemented: token budgets increased, artifact storage added, reporting enhanced. - **2026‑05‑14 13:00 UTC** – Re‑run of the benchmark with updated harness. - **2026‑05‑14 13:30 UTC** – Re‑run shows 27 passes, confirming the fix. - **2026‑05‑14 14:00 UTC** – Postmortem drafted and shared with stakeholders. The incident was fully resolved within 5 hours of the initial run, with a comprehensive postmortem and corrective actions documented. # Root Causes The incident stemmed from a combination of model‑side and harness‑side failures. **Model‑Side Failures** 1. **Token Budget Insufficiency** – The model was allocated a token budget of 512 tokens per task. For many tasks, the model’s reasoning required 400 tokens, leaving only 112 tokens for the final answer. The model’s final answer often required 200–300 tokens, causing truncation. 2. **Reasoning‑Only Generation** – The model’s internal policy prioritized generating reasoning content over final content when token budgets were tight. This behavior was not flagged by the harness because the harness only checked for the presence of `message.content`. 3. **MTP Acceptance** – The model’s internal “Model‑Task‑Plan” (MTP) acceptance logic was not triggered because the model did not produce a final answer within the token budget, leading to an “invalid‑run” status that was not surfaced in the harness report. **Harness‑Side Failures** 1. **Reporting Gaps** – The harness’s summary report only displayed the pass/fail count and did not include token usage, MTP acceptance, or invalid‑run warnings. 2. **Artifact Storage** – The harness did not store the full reasoning and final content in the artifact repository. Instead, it truncated the output to the token budget and only stored the truncated final answer. 3. **Invalid‑Run Warning Visibility** – The harness logged “invalid‑run” warnings but did not surface them in the UI or the summary report. 4. **Pass/Fail Logic** – The harness’s pass/fail logic was based solely on the presence of `message.content`. It did not account for the possibility that the model might have produced reasoning but no final answer due to token budget constraints. The root causes can be summarized as: - **Token budget mis‑configuration** (model‑side). - **Reporting and artifact handling gaps** (harness‑side). The fix addressed both sides: increased token budgets, added artifact storage for full reasoning and final content, and enhanced reporting to display token usage, MTP acceptance, and invalid‑run warnings. # Detection Gaps The incident exposed several detection gaps in the harness and the monitoring pipeline. 1. **Missing Token Usage Metrics** – The harness did not expose token usage per task in the summary report. This prevented stakeholders from seeing that the model was hitting the token budget. 2. **Lack of Final Content Visibility** – The harness only reported the presence of `message.content` but did not show the actual content or whether it was truncated. 3. **Invalid‑Run Warning Suppression** – The harness logged “invalid‑run” warnings in the logs but did not surface them in the UI or the summary report. 4. **No MTP Acceptance Indicator** – The harness did not indicate whether the model had accepted the MTP or not. 5. **Artifact Storage Limitation** – The harness only stored the truncated final answer, not the full reasoning or final content. 6. **No Automated Alerting** – The harness did not trigger an automated alert when the model failed to produce final content or when token usage exceeded a threshold. These gaps meant that the harness could not detect that the model was generating reasoning but not final content, and that the token budget was insufficient. The corrective actions addressed these gaps by adding new metrics, enhancing reporting, and adding automated alerts. # Corrective Actions The corrective actions are split into model‑side and harness‑side fixes. **Model‑Side Fixes** 1. **Token Budget Increase** – The token budget per task was increased from 512 to 1,024 tokens. This ensures that the model has enough space for reasoning and final content. 2. **Reasoning‑Final Balance** – The model’s internal policy was updated to prioritize final content when token usage approaches the budget. 3. **MTP Acceptance Logging** – The model now logs MTP acceptance status explicitly, which the harness can capture. **Harness‑Side Fixes** 1. **Enhanced Reporting** – The harness now displays token usage, MTP acceptance, and invalid‑run warnings in the summary report. 2. **Artifact Storage** – The harness now stores the full reasoning and final content for each task in the artifact repository. 3. **Invalid‑Run Alerting** – The harness now triggers an automated alert when an invalid‑run warning is logged. 4. **Pass/Fail Logic Update** – The harness now requires both reasoning and final content to be present for a pass. 5. **UI Enhancements** – The harness UI now displays a “Token Usage” bar chart and a “Final Content Presence” indicator. **Testing and Validation** 1. **Regression Tests** – Added tests to verify that the harness correctly reports token usage and final content. 2. **Integration Tests** – Added integration tests to ensure that the harness correctly captures MTP acceptance and invalid‑run warnings. 3. **Manual Audits** – Conducted manual audits of 10 random benchmark runs to confirm that the harness now reports accurate pass/fail counts. The corrective actions were implemented and validated within 4 hours of the incident. # Preventive Tests To prevent similar incidents in the future, the following preventive tests were added to the harness pipeline: 1. **Token Usage Threshold Test** – The harness now checks that token usage per task does not exceed 90 % of the budget. 2. **Final Content Presence Test** – The harness verifies that `message.content` is present and not truncated. 3. **MTP Acceptance Test** – The harness verifies that the model has accepted the MTP before generating final content. 4. **Artifact Integrity Test** – The harness verifies that the artifact repository contains the full reasoning and final content for each task. 5. **Invalid‑Run Warning Test** – The harness verifies that any “invalid‑run” warning is surfaced in the UI and triggers an alert. 6. **Pass/Fail Logic Test** – The harness verifies that a pass requires both reasoning and final content. These tests are run automatically for every benchmark run. If any test fails, the harness will halt the run and notify the engineering team. # Dashboard Changes The benchmark dashboard was updated to provide clearer visibility into token usage, final content presence, and MTP acceptance. 1. **Token Usage Bar Chart** – A new bar chart shows token usage per task, with a red threshold line at 90 % of the budget. 2. **Final Content Indicator** – A green checkmark indicates that `message.content` is present and not truncated. 3. **MTP Acceptance Indicator** – A green checkmark indicates that the model accepted the MTP. 4. **Invalid‑Run Warning Panel** – A panel lists any “invalid‑run” warnings for each task. 5. **Artifact Link** – A link to the artifact repository for each task, allowing stakeholders to view the full reasoning and final content. 6. **Run Summary** – The run summary now includes token usage, MTP acceptance, final content presence, and invalid‑run warnings. The dashboard changes provide stakeholders with a comprehensive view of the model’s performance and the harness’s reporting. # Remaining Risks Despite the corrective actions, the following risks remain: 1. **Token Budget Drift** – Future model updates may change token usage patterns, potentially causing token budgets to become insufficient again. 2. **Reporting Regression** – The harness reporting code may regress if future changes inadvertently remove or alter the new metrics. 3. **Artifact Storage Limits** – The artifact repository may reach storage limits if token budgets are increased significantly. 4. **Alert Fatigue** – Automated alerts for invalid‑run warnings may be ignored if they occur frequently. 5. **Model Policy Drift** – The model’s internal policy may drift toward reasoning‑only generation if not monitored. Mitigation plans include continuous monitoring of token usage, automated regression tests for reporting, and periodic reviews of artifact storage usage. # Owner Checklist The following checklist is assigned to the harness owner, the model owner, and the product owner. **Harness Owner** - [ ] Verify that token usage metrics are displayed in the dashboard. - [ ] Confirm that the harness captures MTP acceptance and invalid‑run warnings. - [ ] Run the preventive tests for every benchmark run. - [ ] Monitor artifact repository usage and alert when storage approaches capacity. **Model Owner** - [ ] Ensure that the model’s policy prioritizes final content when token usage is high. - [ ] Log MTP acceptance status explicitly. - [ ] Monitor token usage patterns and adjust token budgets accordingly. **Product Owner** - [ ] Review the dashboard metrics before making product decisions. - [ ] Verify that the harness’s pass/fail logic aligns with product requirements. - [ ] Ensure that stakeholders receive alerts for any invalid‑run warnings. **All Owners** - [ ] Conduct a quarterly review of the harness and model reporting. - [ ] Update documentation to reflect the new reporting and artifact handling. - [ ] Provide training to stakeholders on interpreting the new dashboard metrics. The checklist will be reviewed and updated annually or whenever significant changes occur. # Summary On 2026‑05‑14, the AI benchmark harness executed a full run of the 27‑task evaluation suite for the new GPT‑4‑Turbo model. The harness reported a single pass out of 27, implying that the model had successfully met the acceptance criteria for only one task. However, a subsequent manual audit revealed that the model had largely produced only intermediate reasoning content (the `reasoning_content` field) and had failed to emit a final answer (`message.content`) for the majority of tasks. The token budgets allocated for each task were insufficient to accommodate the full reasoning and final answer, causing the harness to truncate the output prematurely. The harness’s reporting mechanisms did not clearly expose the token usage, the presence or absence of final content, or the “invalid‑run” warnings that were generated when the model exceeded its token budget. Consequently, the harness’s pass/fail metric was misleading, and the benchmark results were not representative of the model’s true capabilities. The incident was identified by a cross‑team review that compared the harness logs against the raw artifact files stored in the artifact repository. The review uncovered that the harness had not captured the full reasoning and final answer for 26 of the 27 tasks. The root cause was a combination of model‑side failures (the model’s inability to produce final content within the token budget) and harness‑side failures (the harness’s inadequate reporting and artifact handling). The fix involved increasing the token budgets for each task, adding artifact storage for the full reasoning and final content, and enhancing the harness’s reporting to display token usage, MTP acceptance, and invalid‑run warnings in a clear, actionable format. The postmortem below documents the impact, timeline, root causes, detection gaps, corrective actions, preventive tests, dashboard changes, remaining risks, and an owner checklist to ensure that similar incidents are prevented in the future. # Impact The immediate impact of the incident was a mis‑representation of the model’s performance: the harness reported a 3.7 % pass rate (1/27) while the model actually produced reasoning for 26 tasks but failed to produce final answers due to token budget constraints. This mis‑representation had several downstream effects: 1. **Stakeholder Confidence** – Product managers and external partners were led to believe that the model was severely under‑performing, potentially delaying product releases and affecting trust in the model’s reliability. 2. **Benchmark Integrity** – The benchmark results were used in a whitepaper and internal metrics dashboard. The inaccurate pass/fail count could have skewed the model’s ranking relative to competitors, affecting strategic decisions. 3. **Model Development** – The engineering team used the harness output to guide fine‑tuning and prompt engineering. The misleading results caused them to focus on the wrong aspects of the model, potentially wasting resources. 4. **Operational Costs** – The harness executed 27 tasks with a token budget that was too low. The truncated outputs required re‑runs, increasing compute costs and time to completion. 5. **Compliance** – The benchmark was part of a regulatory audit. The inaccurate results could have led to compliance violations if the model was deployed in regulated environments. The impact was mitigated by a manual audit and the subsequent fix, but the incident highlighted the need for robust reporting and token budgeting in the harness. # Timeline - **2026‑05‑14 09:00 UTC** – Harness initiates the 27‑task benchmark run for GPT‑4‑Turbo. - **2026‑05‑14 09:05 UTC** – Harness logs a single pass out of 27. - **2026‑05‑14 09:15 UTC** – Harness completes the run and writes the summary report to the artifact repository. - **2026‑05‑14 10:00 UTC** – Product team reviews the benchmark results and flags the low pass rate. - **2026‑05‑14 10:30 UTC** – Engineering team initiates a manual audit of the harness logs. - **2026‑05‑14 11:00 UTC** – Audit reveals that the model produced reasoning content but no final answer for 26 tasks. - **2026‑05‑14 11:30 UTC** – Root cause analysis identifies token budget constraints and reporting gaps. - **2026‑05‑14 12:00 UTC** – Fixes are implemented: token budgets increased, artifact storage added, reporting enhanced. - **2026‑05‑14 13:00 UTC** – Re‑run of the benchmark with updated harness. - **2026‑05‑14 13:30 UTC** – Re‑run shows 27 passes, confirming the fix. - **2026‑05‑14 14:00 UTC** – Postmortem drafted and shared with stakeholders. The incident was fully resolved within 5 hours of the initial run, with a comprehensive postmortem and corrective actions documented. # Root Causes The incident stemmed from a combination of model‑side and harness‑side failures. **Model‑Side Failures** 1. **Token Budget Insufficiency** – The model was allocated a token budget of 512 tokens per task. For many tasks, the model’s reasoning required 400 tokens, leaving only 112 tokens for the final answer. The model’s final answer often required 200–300 tokens, causing truncation. 2. **Reasoning‑Only Generation** – The model’s internal policy prioritized generating reasoning content over final content when token budgets were tight. This behavior was not flagged by the harness because the harness only checked for the presence of `message.content`. 3. **MTP Acceptance** – The model’s internal “Model‑Task‑Plan” (MTP) acceptance logic was not triggered because the model did not produce a final answer within the token budget, leading to an “invalid‑run” status that was not surfaced in the harness report. **Harness‑Side Failures** 1. **Reporting Gaps** – The harness’s summary report only displayed the pass/fail count and did not include token usage, MTP acceptance, or invalid‑run warnings. 2. **Artifact Storage** – The harness did not store the full reasoning and final content in the artifact repository. Instead, it truncated the output to the token budget and only stored the truncated final answer. 3. **Invalid‑Run Warning Visibility** – The harness logged “invalid‑run” warnings but did not surface them in the UI or the summary report. 4. **Pass/Fail Logic** – The harness’s pass/fail logic was based solely on the presence of `message.content`. It did not account for the possibility that the model might have produced reasoning but no final answer due to token budget constraints. The root causes can be summarized as: - **Token budget mis‑configuration** (model‑side). - **Reporting and artifact handling gaps** (harness‑side). The fix addressed both sides: increased token budgets, added artifact storage for full reasoning and final content, and enhanced reporting to display token usage, MTP acceptance, and invalid‑run warnings. # Detection Gaps The incident exposed several detection gaps in the harness and the monitoring pipeline. 1. **Missing Token Usage Metrics** – The harness did not expose token usage per task in the summary report. This prevented stakeholders from seeing that the model was hitting the token budget. 2. **Lack of Final Content Visibility** – The harness only reported the presence of `message.content` but did not show the actual content or whether it was truncated. 3. **Invalid‑Run Warning Suppression** – The harness logged “invalid‑run” warnings in the logs but did not surface them in the UI or the summary report. 4. **No MTP Acceptance Indicator** – The harness did not indicate whether the model had accepted the MTP or not. 5. **Artifact Storage Limitation** – The harness only stored the truncated final answer, not the full reasoning or final content. 6. **No Automated Alerting** – The harness did not trigger an automated alert when the model failed to produce final content or when token usage exceeded a threshold. These gaps meant that the harness could not detect that the model was generating reasoning but not final content, and that the token budget was insufficient. The corrective actions addressed these gaps by adding new metrics, enhancing reporting, and adding automated alerts. # Corrective Actions The corrective actions are split into model‑side and harness‑side fixes. **Model‑Side Fixes** 1. **Token Budget Increase** – The token budget per task was increased from 512 to 1,024 tokens. This ensures that the model has enough space for reasoning and final content. 2. **Reasoning‑Final Balance** – The model’s internal policy was updated to prioritize final content when token usage approaches the budget. 3. **MTP Acceptance Logging** – The model now logs MTP acceptance status explicitly, which the harness can capture. **Harness‑Side Fixes** 1. **Enhanced Reporting** – The harness now displays token usage, MTP acceptance, and invalid‑run warnings in the summary report. 2. **Artifact Storage** – The harness now stores the full reasoning and final content for each task in the artifact repository. 3. **Invalid‑Run Alerting** – The harness now triggers an automated alert when an invalid‑run warning is logged. 4. **Pass/Fail Logic Update** – The harness now requires both reasoning and final content to be present for a pass. 5. **UI Enhancements** – The harness UI now displays a “Token Usage” bar chart and a “Final Content Presence” indicator. **Testing and Validation** 1. **Regression Tests** – Added tests to verify that the harness correctly reports token usage and final content. 2. **Integration Tests** – Added integration tests to ensure that the harness correctly captures MTP acceptance and invalid‑run warnings. 3. **Manual Audits** – Conducted manual audits of 10 random benchmark runs to confirm that the harness now reports accurate pass/fail counts. The corrective actions were implemented and validated within 4 hours of the incident. # Preventive Tests To prevent similar incidents in the future, the following preventive tests were added to the harness pipeline: 1. **Token Usage Threshold Test** – The harness now checks that token usage per task does not exceed 90 % of the budget. 2. **Final Content Presence Test** – The harness verifies that `message.content` is present and not truncated. 3. **MTP Acceptance Test** – The harness verifies that the model has accepted the MTP before generating final content. 4. **Artifact Integrity Test** – The harness verifies that the artifact repository contains the full reasoning and final content for each task. 5. **Invalid‑Run Warning Test** – The harness verifies that any “invalid‑run” warning is surfaced in the UI and triggers an alert. 6. **Pass/Fail Logic Test** – The harness verifies that a pass requires both reasoning and final content. These tests are run automatically for every benchmark run. If any test fails, the harness will halt the run and notify the engineering team. # Dashboard Changes The benchmark dashboard was updated to provide clearer visibility into token usage, final content presence, and MTP acceptance. 1. **Token Usage Bar Chart** – A new bar chart shows token usage per task, with a red threshold line at 90 % of the budget. 2. **Final Content Indicator** – A green checkmark indicates that `message.content` is present and not truncated. 3. **MTP Acceptance Indicator** – A green checkmark indicates that the model accepted the MTP. 4. **Invalid‑Run Warning Panel** – A panel lists any “invalid‑run” warnings for each task. 5. **Artifact Link** – A link to the artifact repository for each task, allowing stakeholders to view the full reasoning and final content. 6. **Run Summary** – The run summary now includes token usage, MTP acceptance, final content presence, and invalid‑run warnings. The dashboard changes provide stakeholders with a comprehensive view of the model’s performance and the harness’s reporting. # Remaining Risks Despite the corrective actions, the following risks remain: 1. **Token Budget Drift** – Future model updates may change token usage patterns, potentially causing token budgets to become insufficient again. 2. **Reporting Regression** – The harness reporting code may regress if future changes inadvertently remove or alter the new metrics. 3. **Artifact Storage Limits** – The artifact repository may reach storage limits if token budgets are increased significantly. 4. **Alert Fatigue** – Automated alerts for invalid‑run warnings may be ignored if they occur frequently. 5. **Model Policy Drift** – The model’s internal policy may drift toward reasoning‑only generation if not monitored. Mitigation plans include continuous monitoring of token usage, automated regression tests for reporting, and periodic reviews of artifact storage usage. # Owner Checklist The following checklist is assigned to the harness owner, the model owner, and the product owner. **Harness Owner** - [ ] Verify that token usage metrics are displayed in the dashboard. - [ ] Confirm that the harness captures MTP acceptance and invalid‑run warnings. - [ ] Run the preventive tests for every benchmark run. - [ ] Monitor artifact repository usage and alert when storage approaches capacity. **Model Owner** - [ ] Ensure that the model’s policy prioritizes final content when token usage is high. - [ ] Log MTP acceptance status explicitly. - [ ] Monitor token usage patterns and adjust token budgets accordingly. **Product Owner** - [ ] Review the dashboard metrics before making product decisions. - [ ] Verify that the harness’s pass/fail logic aligns with product requirements. - [ ] Ensure that stakeholders receive alerts for any invalid‑run warnings. **All Owners** - [ ] Conduct a quarterly review of the harness and model reporting. - [ ] Update documentation to reflect the new reporting and artifact handling. - [ ] Provide training to stakeholders on interpreting the new dashboard metrics. The checklist will be reviewed and updated annually or whenever significant changes occur. # Lessons Learned 1. **Token budgets must be aligned with model behavior** – The model’s internal policy can cause it to generate more reasoning than final content when token budgets are tight. 2. **Reporting must expose all relevant metrics** – Token usage, MTP acceptance, and invalid‑run warnings are critical for accurate pass/fail determination. 3. **Artifact storage is essential for post‑run audits** – Without full artifact storage, audits cannot verify that the model produced the expected content. 4. **Automated alerts reduce human error** – Automated alerts for invalid‑run warnings help catch issues early. 5. **Clear pass/fail logic is vital** – Pass/fail logic should consider both reasoning and final content. # Future Roadmap 1. **Dynamic Token Budgeting** – Implement a dynamic token budgeting algorithm that adjusts budgets per task based on historical token usage. 2. **Model‑Side Policy Monitoring** – Add a monitoring layer that tracks the model’s internal policy decisions (e.g., reasoning vs. final content) and alerts if the policy drifts. 3. **Dashboard Enhancements** – Add a heatmap view that shows token usage across all tasks, highlighting tasks that consistently approach the budget. 4. **Artifact Compression** – Implement artifact compression to reduce storage usage while preserving full content. 5. **Automated Regression Tests** – Expand regression tests to cover all new metrics and UI elements. # Glossary - **MTP** – Model‑Task‑Plan, the internal plan the model generates before producing final content. - **Token Budget** – The maximum number of tokens the model is allowed to generate for a task. - **Invalid‑Run** – A run where the model fails to produce final content within the token budget. - **Artifact Repository** – Storage location for all raw model outputs, including reasoning and final content. # FAQ **Q: Why did the harness report only one pass?** A: The harness’s pass/fail logic only considered the presence of `message.content`. It did not check whether the content was truncated or whether the model had produced reasoning. **Q: How were the token budgets increased?** A: The harness configuration was updated to set the token budget to 1,024 tokens per task. **Q: Will the new token budget cause the model to generate longer outputs?** A: The model’s policy now prioritizes final content when token usage approaches the budget, preventing unnecessarily long reasoning. **Q: How can I view the full reasoning?** A: The artifact repository now stores the full reasoning and final content. Use the artifact link in the dashboard to download the JSON file. # Appendix: Sample Logs Below is a snippet of the harness log for a single task after the fix. ``` [2026‑05‑14 13:02:01] Task 12: GPT‑4‑Turbo [2026‑05‑14 13:02:01] Token budget: 1024 [2026‑05‑14 13:02:01] Reasoning tokens used: 480 [2026‑05‑14 13:02:01] Final content tokens used: 200 [2026‑05‑14 13:02:01] MTP accepted: true [2026‑05‑14 13:02:01] Pass: true [2026‑05‑14 13:02:01] Artifact stored at: /artifacts/task12.json ``` The log shows that the model used 480 tokens for reasoning and 200 for final content, well within the 1024 token budget. The MTP was accepted, and the task passed. --- The postmortem demonstrates that the harness and model were updated to prevent token‑budget‑related mis‑reporting. The new metrics, tests, and dashboard changes provide stakeholders with a clear view of the model’s performance.