# Goals

The primary objective of this private WorkDash analysis workflow is to establish a secure, local-first data processing pipeline that enables deep, real-time analysis of personal communication channels (email and Microsoft Teams) while guaranteeing that no raw private content ever leaves the home lab environment. The workflow is designed around a dual-output architecture: a high-fidelity private dashboard for the owner, and a sanitized, benchmark-ready public reporting plane that preserves statistical utility without compromising confidentiality.

Key goals include:
- **Privacy-by-Design:** Ensure raw emails, chat logs, attachments, and metadata are ingested, processed, and stored exclusively within an isolated local environment. No cloud egress, no third-party telemetry, and no external API calls during inference or reporting.
- **Fine-Grained Local Classification:** Leverage a locally hosted large language model (LLM) to perform granular analysis including intent detection, priority scoring, category tagging, sentiment analysis, entity extraction, and PII identification. All classification occurs on-premises using quantized open-weight models.
- **Deterministic Redaction & Synthesis:** Implement a multi-layered sanitization pipeline that transforms raw data into benchmark-safe outputs. This includes field-level redaction, semantic masking, and statistically representative synthetic data generation to ensure public reports remain analytically valid.
- **Owner Utility:** Maintain a rich, actionable private reporting plane that provides the home lab owner with productivity insights, communication trends, priority management, and system health metrics. The private plane retains full fidelity, while the public plane abstracts only identifiable traces.
- **Benchmarking Readiness:** Structure the workflow to produce reproducible, shareable benchmark artifacts (metrics, methodology, synthetic datasets, pipeline performance logs) that demonstrate WorkDash capabilities without exposing sensitive information. This enables community sharing, model evaluation, and pipeline optimization while adhering to strict privacy constraints.

The workflow is architected for a typical home lab setup: a dedicated VM or container running on a local GPU/CPU node, isolated via VLAN or host-level networking, with encrypted storage volumes and strict access controls. All components are version-controlled, auditable, and designed for continuous improvement through human-in-the-loop validation and automated quality gates.

# Data That Must Stay Local

To guarantee that raw private content remains strictly local, the workflow enforces a comprehensive data containment strategy across ingestion, storage, processing, and access layers.

**1. Data Sources & Ingestion**
- **Email:** Ingested via local IMAP/POP3 polling or Microsoft Graph API with locally stored OAuth tokens. Raw MIME structures, headers, attachments, and body content are downloaded to a local message queue (e.g., Redis or RabbitMQ) for asynchronous processing.
- **Teams/Chat:** Connected via local webhook listeners or Microsoft Graph API subscriptions. Chat messages, reactions, file shares, and thread metadata are streamed into the local pipeline.
- **Metadata & Context:** Calendar events, contact lists, project codes, and system logs are also ingested locally to enrich classification context.
- **Ingestion Controls:** All connectors run on the local network. Authentication credentials are stored in a local key management system (e.g., HashiCorp Vault or encrypted `.env` with hardware-backed key storage). Network egress is blocked at the firewall level for all ingestion services except explicit, audited API calls for token refresh.

**2. Storage Architecture**
- **Encrypted Volumes:** Raw data is written to ZFS or LUKS-encrypted volumes with AES-256 encryption. Keys are stored separately from the data plane.
- **Database Layer:** Processed and raw data reside in a local PostgreSQL or SQLite instance. Tables are partitioned by date and source, with row-level security policies enforced.
- **Retention & Purging:** Automated lifecycle policies delete raw content after a configurable retention window (e.g., 90 days). Secure deletion uses multiple-pass overwriting or cryptographic erasure.
- **Air-Gapped/Network-Segmented:** The storage and processing nodes operate on a dedicated VLAN with strict egress rules. Only the reporting plane (synthetic/redacted outputs) is permitted to exit the local network, and only after passing automated validation.

**3. Access Controls & Auditing**
- **RBAC:** Role-based access control restricts raw data access to the owner and authorized local services. API keys and service accounts are scoped minimally.
- **Audit Logging:** All ingestion, classification, redaction, and export events are logged to an immutable local audit trail. Logs are timestamped, hashed, and version-controlled.
- **No Cloud Sync:** Services like OneDrive, Google Drive, or cloud backup agents are explicitly excluded from the pipeline. Local-only storage ensures zero exposure to external breaches.

By enforcing these containment measures, the workflow guarantees that raw private content never traverses untrusted networks, is never stored in cloud databases, and is only accessible within the home lab's secure boundary.

# Local Classification

The local classification engine is the analytical core of the workflow, responsible for transforming raw communication data into structured, actionable insights. It operates entirely on-premises using open-weight models and deterministic pipelines.

**1. Model Deployment & Configuration**
- **Base Models:** Llama 3.1 8B/70B, Mistral Large, or Qwen 2.5 variants, deployed via Ollama, vLLM, or llama.cpp. Models are quantized (GGUF Q4_K_M or AWQ) to balance performance and memory usage.
- **Hardware:** Dedicated GPU (NVIDIA RTX 4090/4070 or AMD equivalent) or high-core-count CPU with AVX-512 support. Inference is batched to maximize throughput.
- **Security:** Model weights are stored locally. No network calls are permitted during inference. Prompts are version-controlled and cryptographically hashed to prevent tampering.

**2. Classification Tasks & Schema**
The model performs fine-grained analysis across multiple dimensions:
- **Intent Detection:** Categorizes messages as informational, action-oriented, conversational, administrative, or urgent.
- **Priority Scoring:** Assigns a numeric priority (1-5) based on keywords, sender hierarchy, deadlines, and historical response patterns.
- **Category Tagging:** Maps content to project codes, departments, or personal/work domains using a controlled vocabulary.
- **Sentiment & Urgency:** Detects emotional tone, stress indicators, and time-sensitive language.
- **Entity Extraction:** Identifies names, dates, project identifiers, financial figures, and technical terms.
- **PII/PHI Detection:** Flags emails, phone numbers, addresses, SSNs, health information, and confidential project names.

**3. Pipeline Architecture**
- **Preprocessing:** Raw HTML/Markdown is cleaned, attachments are stripped (metadata extracted only), and text is chunked into context-appropriate segments (e.g., 500-token windows).
- **Inference:** Chunks are passed to the local LLM via structured JSON prompts. Output is validated against a strict schema using Pydantic or JSON Schema validators.
- **Confidence Scoring:** Each classification includes a confidence score. Outputs below a threshold (e.g., 0.75) are flagged for human review or routed to rule-based fallbacks.
- **Caching & Optimization:** Frequently occurring patterns (e.g., standard meeting invites, automated notifications) are cached to reduce inference load. Async processing ensures real-time ingestion is not blocked.

**4. Continuous Improvement**
- **Prompt Engineering:** System prompts are iteratively refined based on misclassification audits. Few-shot examples are curated locally to improve domain specificity.
- **Model Updates:** New model versions are tested in a staging environment before production deployment. Rollback procedures are documented and automated.
- **Metrics Tracking:** Inference latency, accuracy rates, and PII detection recall are logged to monitor model performance over time.

This local classification approach ensures that all analytical reasoning occurs within the home lab, preserving privacy while delivering high-fidelity insights tailored to the owner's communication patterns.

# Redaction Strategy

The redaction strategy transforms classified data into benchmark-safe outputs through a multi-layered sanitization pipeline. It balances privacy preservation with analytical utility, ensuring public reports remain useful while eliminating all identifiable traces.

**1. Layered Sanitation Approach**
- **Rule-Based Filtering:** Regex patterns detect and mask standard PII (emails, phone numbers, IP addresses, credit card formats). Patterns are maintained in a version-controlled dictionary and updated regularly.
- **NER & PII Detection:** The local LLM or specialized NER models (e.g., spaCy with custom entities, Microsoft Presidio) identify names, locations, organizations, and confidential terms. Outputs are cross-referenced with a local blocklist.
- **Semantic Redaction:** Instead of simple masking, semantic redaction replaces specific entities with statistically representative placeholders (e.g., `[PROJECT_ALPHA]`, `[SENIOR_MANAGER]`) or synthetic equivalents that preserve context without revealing identity.
- **Aggregate Masking:** For public reports, raw counts are aggregated into ranges or percentiles (e.g., "15-20 messages/day" instead of exact counts) to prevent reverse engineering.

**2. Field-Level Decisions**
A comprehensive mapping table (provided later) dictates keep/drop/redact decisions per field. General principles:
- **Keep:** Timestamps, categories, priority scores, aggregate metrics, system health indicators.
- **Drop:** Raw body text, attachments, direct quotes, metadata headers, raw sender addresses.
- **Redact:** Names, project codes, locations, financial figures, specific dates (shifted to relative time), technical identifiers.

**3. Validation & Verification**
- **Automated Scans:** Post-redaction, the pipeline runs automated PII detection scans to verify zero leakage. False positives/negatives are logged for model tuning.
- **Human Spot-Checks:** Random samples are reviewed by the owner to ensure contextual integrity is maintained after redaction.
- **Synthetic Fidelity Checks:** Statistical tests (KS, chi-square) compare synthetic/redacted distributions against original data to ensure analytical validity.

**4. Output Formatting**
- **Private Dashboard:** Full-fidelity JSON/Markdown with raw content intact.
- **Public Reports:** Sanitized Markdown/HTML with synthetic examples, aggregate stats, and redacted placeholders. Clear labeling distinguishes private vs. public content.
- **Export Controls:** Automated scripts enforce output routing based on destination (local vs. public). Network egress is restricted to sanitized payloads only.

This strategy ensures that the public-facing benchmark reports are both privacy-compliant and analytically robust, while the private plane retains full utility for the owner.

# Synthetic Test Generation

Synthetic test generation creates benchmark-safe datasets that mirror real-world communication patterns without containing actual private information. This enables public sharing, model training, and pipeline stress testing while maintaining statistical fidelity.

**1. Generation Methodologies**
- **Template-Based Generation:** Predefined templates fill placeholders with controlled vocabulary (e.g., "Meeting scheduled for [TIME] regarding [TOPIC]"). Templates are curated to reflect common communication patterns.
- **LLM-Assisted Generation:** Local LLMs generate synthetic messages under strict system prompts enforcing no real data usage, controlled entropy, and statistical alignment. Outputs are validated against schema constraints.
- **Statistical Sampling:** Real classification outputs (categories, priorities, latencies) are sampled to preserve distribution properties. Raw text is replaced with synthetically generated equivalents that match sentiment and intent profiles.

**2. Fidelity Preservation**
- **Distribution Matching:** Key statistical properties are preserved: message volume trends, category ratios, priority shifts, response latency percentiles, and sentiment spread.
- **Contextual Consistency:** Synthetic examples maintain logical coherence (e.g., follow-up messages align with prior context, priority scores match urgency indicators).
- **Diversity Control:** Generation parameters ensure coverage of edge cases (urgent, administrative, conversational) without overrepresenting any single pattern.

**3. Validation & Quality Assurance**
- **Automated Tests:** KS tests, chi-square tests, and distribution matching algorithms verify that synthetic data aligns with original statistical profiles.
- **Human Review:** Synthetic samples are audited for contextual plausibility and absence of real data leakage.
- **Drift Monitoring:** Continuous monitoring detects synthetic data drift (e.g., category distribution shifts) and triggers regeneration or parameter adjustment.

**4. Use Cases**
- **Public Benchmark Reports:** Provides analytically valid examples for sharing methodology and results.
- **Model Training:** Creates labeled datasets for fine-tuning local models without privacy risks.
- **Pipeline Stress Testing:** Generates high-volume synthetic data to evaluate ingestion throughput, classification accuracy, and redaction performance.
- **Community Sharing:** Enables transparent benchmarking while adhering to privacy constraints.

Synthetic test generation ensures that public-facing outputs remain useful, reproducible, and statistically representative, while completely eliminating privacy risks.

# Metrics To Keep

The metrics framework captures actionable insights for the owner while remaining safe for public benchmarking. Metrics are inherently aggregate or anonymized by design, ensuring no PII or raw content exposure.

**1. Volume & Velocity**
- **Messages/Day:** Total ingested messages across email and Teams.
- **Peak Load:** Maximum messages per hour/day.
- **Response Latency:** Time between message receipt and reply (median, p95, p99).
- **Ingestion Throughput:** Messages processed per minute by the local pipeline.

**2. Classification & Priority**
- **Category Distribution:** Percentage breakdown by intent (informational, action-oriented, conversational, administrative, urgent).
- **Priority Shifts:** Changes in priority scores over time (e.g., urgent to resolved).
- **Escalation Rate:** Percentage of messages requiring human review or model fallback.
- **PII Detection Recall:** Accuracy of local PII identification (tracked internally, not published).

**3. Productivity & Engagement**
- **Reply Rate:** Percentage of messages receiving a response within 24 hours.
- **Read Receipts:** Proportion of messages marked as read.
- **Task Completion Proxy:** Correlation between priority scores and follow-up actions (e.g., calendar events, file shares).
- **Sentiment Spread:** Distribution of positive, neutral, and negative sentiment scores.

**4. System Health**
- **Inference Latency:** Average time for local model classification.
- **Error Rates:** Pipeline failures, schema validation errors, PII detection false positives/negatives.
- **Storage Utilization:** Encrypted volume usage, retention policy compliance.
- **Model Accuracy:** Confidence score distribution, human review override rate.

**5. Calculation & Privacy Safety**
- **Aggregation Functions:** Metrics are computed using time-windowed aggregations, percentiles, and normalized ratios.
- **Anonymization by Design:** No raw text, direct identifiers, or unaggregated counts are included. All metrics are derived from classified, redacted, or synthetic data.
- **Utility Preservation:** Metrics provide actionable insights for productivity optimization, system monitoring, and benchmarking without exposing sensitive information.

This metrics framework ensures that both private and public reporting planes remain highly useful while adhering to strict privacy constraints.

# Human Review

Human review serves as the quality assurance and continuous improvement layer, ensuring classification accuracy, redaction validity, and synthetic data fidelity. It operates within a structured workflow that balances automation with expert oversight.

**1. Workflow Integration**
- **Confidence Thresholding:** Model outputs below a predefined confidence score (e.g., 0.75) are automatically flagged for human review.
- **Review Queue:** Flagged items are routed to a local web UI or markdown diff viewer, prioritized by urgency and category.
- **Approval/Rejection:** Reviewers validate classifications, adjust redaction placeholders, or approve synthetic examples. Changes are logged and version-controlled.
- **Feedback Loop:** Review outcomes are used to retrain prompts, update blocklists, and adjust model parameters.

**2. Tools & Interfaces**
- **Local Dashboard:** Web-based interface for reviewing flagged items, adjusting metadata, and approving outputs.
- **Annotation Tools:** Markdown diff viewers, JSON schema validators, and PII detection dashboards.
- **Audit Logs:** Immutable records of all human interventions, timestamped and hashed for compliance.

**3. Frequency & Governance**
- **Initial Setup:** Heavy review during pipeline deployment to calibrate thresholds, validate redaction rules, and tune synthetic generation parameters.
- **Ongoing Operations:** Periodic spot-checks (weekly/monthly) and post-update mandatory reviews.
- **Change Management:** Model updates, prompt changes, and rule modifications require human approval before production deployment.
- **Incident Response:** Clear SOPs for handling PII leakage, classification errors, or pipeline failures. Automated alerts trigger immediate review.

**4. Continuous Improvement**
- **Model Tuning:** Human feedback is used to refine few-shot examples, adjust system prompts, and update blocklists.
- **Rule Optimization:** Regex patterns and NER models are iteratively improved based on false positive/negative rates.
- **Synthetic Validation:** Human reviewers assess contextual plausibility and statistical alignment of generated examples.

Human review ensures that the workflow remains accurate, privacy-compliant, and continuously optimized, while maintaining full owner control over the reporting pipeline.

# Failure Modes

Despite robust design, the workflow is susceptible to technical, operational, and privacy-related failure modes. Proactive mitigation strategies are essential to maintain reliability and security.

**1. Technical Failures**
- **Model Hallucination:** LLM generates incorrect classifications or synthetic examples. *Mitigation:* Confidence thresholding, rule-based fallbacks, human review queue, continuous prompt tuning.
- **PII Leakage:** Redaction fails to mask sensitive data. *Mitigation:* Multi-layered sanitization, automated PII scans, version-controlled blocklists, human spot-checks.
- **Pipeline Bottlenecks:** Ingestion or classification slows under high load. *Mitigation:* Async processing, batch optimization, caching, horizontal scaling within local network.
- **Storage Corruption:** Encrypted volumes fail or data is lost. *Mitigation:* Regular backups, ZFS snapshots, cryptographic erasure policies, hardware monitoring.

**2. Operational Failures**
- **Human Error:** Misclassification, incorrect redaction, or synthetic data drift. *Mitigation:* Structured review workflows, automated validation, statistical fidelity checks, clear SOPs.
- **Access Control Misconfiguration:** Unauthorized local or external access to raw data. *Mitigation:* Strict RBAC, network segmentation, audit logging, regular security audits.
- **Metric Misinterpretation:** Public reports misrepresent private trends. *Mitigation:* Clear labeling, aggregate-only metrics, human validation of published outputs.

**3. Privacy & Compliance Risks**
- **Network Egress Breach:** Raw data accidentally leaves local environment. *Mitigation:* Firewall rules, egress monitoring, automated payload validation, air-gapped design.
- **Model Memorization:** LLM inadvertently reproduces real data in synthetic outputs. *Mitigation:* Strict system prompts, entropy control, statistical validation, periodic retraining with sanitized data.
- **Audit Trail Gaps:** Incomplete logging of interventions or pipeline events. *Mitigation:* Immutable logging, cryptographic hashing, automated compliance checks.

**4. Monitoring & Alerting**
- **Automated Alerts:** Pipeline failures, confidence score drops, PII detection spikes, storage thresholds.
- **Regular Audits:** Monthly reviews of classification accuracy, redaction validity, synthetic fidelity, and access logs.
- **Incident Response:** Documented procedures for containment, investigation, remediation, and post-mortem analysis.

By anticipating and mitigating these failure modes, the workflow maintains high reliability, privacy compliance, and analytical utility under all operational conditions.

# Example Private Summary

**WorkDash Private Analysis Summary**  
*Generated: 2024-06-15 | Period: 2024-06-01 to 2024-06-15 | Environment: Local Home Lab*

**Executive Overview**  
During the reporting period, WorkDash ingested 1,247 messages across email and Teams, with a peak load of 89 messages on June 8th. The local classification engine processed all content on-premises, achieving a 94.2% confidence rate. Priority management shows a 22% increase in urgent action items compared to the previous period, primarily driven by project deadline shifts.

**Category Breakdown**  
- Informational: 41% (511 messages)  
- Action-Oriented: 28% (349 messages)  
- Conversational: 15% (187 messages)  
- Administrative: 10% (125 messages)  
- Urgent: 6% (75 messages)  

**Priority & Escalation Analysis**  
- Priority 1-2 (Low): 58%  
- Priority 3-4 (Medium): 32%  
- Priority 5 (High/Urgent): 10%  
- Escalation Rate: 4.3% of messages required human review or model fallback.  
- Notable Shift: Project "Nexus" deadline moved to June 20th, triggering 14 priority escalations across engineering and QA channels.

**Productivity & Engagement**  
- Reply Rate: 76% within 24 hours  
- Read Receipts: 82% of messages marked as read  
- Task Completion Proxy: 68% of high-priority messages correlated with follow-up calendar events or file shares  
- Sentiment Spread: Positive (34%), Neutral (48%), Negative (18%)  

**System Health**  
- Ingestion Throughput: 142 messages/minute average  
- Inference Latency: 1.8s per message (Llama 3.1 8B quantized)  
- Storage Utilization: 68% of encrypted volume (4.2TB/6TB)  
- Error Rate: 0.7% schema validation failures, all auto-corrected  

**Recommendations**  
- Implement automated priority routing for "Nexus" project updates to reduce manual escalation.  
- Schedule weekly review of administrative message backlog to improve reply rate.  
- Consider upgrading GPU memory allocation to reduce inference latency during peak loads.  

*Note: This summary contains raw content references and identifiable project details. It is intended for local owner use only and must not be shared externally.*

# Example Publishable Summary

**WorkDash Public Benchmark Report**  
*Generated: 2024-06-15 | Period: 2024-06-01 to 2024-06-15 | Environment: Sanitized Local Pipeline*

**Executive Overview**  
This report presents a statistically representative analysis of communication patterns processed by WorkDash. All raw content has been redacted or replaced with synthetic equivalents. The local classification engine achieved a 94.2% confidence rate, with pipeline performance metrics suitable for benchmarking and methodology sharing.

**Category Breakdown**  
- Informational: 41%  
- Action-Oriented: 28%  
- Conversational: 15%  
- Administrative: 10%  
- Urgent: 6%  
*Distribution aligns with typical home lab communication profiles. Synthetic examples preserve intent and priority mappings.*

**Priority & Escalation Analysis**  
- Priority 1-2 (Low): 58%  
- Priority 3-4 (Medium): 32%  
- Priority 5 (High/Urgent): 10%  
- Escalation Rate: 4.3%  
*Priority shifts correlate with simulated deadline adjustments. No identifiable project names or dates are included.*

**Productivity & Engagement**  
- Reply Rate: 76% within 24 hours  
- Read Receipts: 82%  
- Task Completion Proxy: 68% correlation between high-priority messages and follow-up actions  
- Sentiment Spread: Positive (34%), Neutral (48%), Negative (18%)  
*Metrics are aggregated and anonymized. Synthetic examples maintain contextual consistency.*

**System Health**  
- Ingestion Throughput: 142 messages/minute average  
- Inference Latency: 1.8s per message  
- Storage Utilization: 68% of encrypted volume  
- Error Rate: 0.7% schema validation failures, all auto-corrected  
*Performance metrics are benchmark-ready and reproducible across local deployments.*

**Recommendations**  
- Implement automated priority routing for high-frequency project categories.  
- Schedule periodic review of administrative message backlogs.  
- Optimize GPU memory allocation for reduced inference latency during peak loads.  

*Note: This report uses synthetic examples and redacted aggregate summaries. All identifiable traces have been removed. Statistical fidelity is preserved for benchmarking purposes.*

# Field Decision Table

| Field Name | Data Type | Keep/Drop/Redact | Rationale | Example Transformation |
|------------|-----------|------------------|-----------|------------------------|
| Raw Email Body | Text | Drop | Contains PII, confidential content, and unstructured data | Replaced with synthetic equivalent or omitted |
| Teams Chat Message | Text | Drop | Same as email body; high PII risk | Replaced with synthetic equivalent or omitted |
| Sender Email Address | String | Redact | Direct identifier | `[SENDER_EMAIL]` or removed |
| Recipient Email Address | String | Redact | Direct identifier | `[RECIPIENT_EMAIL]` or removed |
| Timestamp | DateTime | Keep | Aggregatable, non-identifiable | `2024-06-08T14:30:00Z` |
| Subject Line | Text | Redact | May contain project names or sensitive topics | `[SUBJECT_REDACTED]` |
| Category Tag | String | Keep | Analytical, non-identifiable | `Action-Oriented` |
| Priority Score | Integer | Keep | Aggregatable, safe for benchmarking | `4` |
| Sentiment Score | Float | Keep | Statistical, non-identifiable | `0.72` |
| Project Code/Name | String | Redact | Confidential identifier | `[PROJECT_ALPHA]` |
| Attachment Filename | String | Redact | May reveal sensitive content | `[ATTACHMENT_REDACTED]` |
| Attachment Size | Integer | Keep | Aggregatable, non-identifiable | `2.4MB` |
| Read Receipt Status | Boolean | Keep | Aggregatable, safe for benchmarking | `true` |
| Reply Latency | Duration | Keep | Statistical, non-identifiable | `4h 12m` |
| PII Confidence Score | Float | Keep | Internal metric, not published | `0.91` |
| Model Inference Time | Duration | Keep | System health metric | `1.8s` |
| Error Flag | Boolean | Keep | Pipeline monitoring | `false` |
| Direct Quote | Text | Drop | High PII risk, unstructured | Removed or replaced with synthetic |
| Calendar Event Title | Text | Redact | May contain sensitive topics | `[EVENT_REDACTED]` |
| Location/Room | String | Redact | Identifiable context | `[LOCATION_REDACTED]` |
| Financial Figure | String | Redact | Sensitive data | `[AMOUNT_REDACTED]` |
| Phone Number | String | Redact | Direct identifier | `[PHONE_REDACTED]` |
| IP Address | String | Redact | Network identifier | `[IP_REDACTED]` |
| Message ID | String | Keep | Internal tracking, non-identifiable | `msg_7f3a9c2b` |
| Thread ID | String | Keep | Internal tracking, non-identifiable | `thread_4e1d8a7f` |
| Attachment Hash | String | Keep | Integrity verification, non-identifiable | `sha256:9f86d081...` |
| User Agent/Header | String | Redact | May contain internal infrastructure details | `[HEADER_REDACTED]` |
| Classification Confidence | Float | Keep | Model performance metric | `0.88` |
| Human Review Flag | Boolean | Keep | Workflow monitoring | `true` |
| Synthetic Example ID | String | Keep | Benchmark tracking | `synth_001` |
| Aggregate Count | Integer | Keep | Statistical metric | `1247` |
| Percentile Rank | Float | Keep | Statistical metric | `p95` |
| Category Distribution | JSON | Keep | Analytical, non-identifiable | `{"info": 0.41, "action": 0.28}` |
| Priority Shift Log | Array | Keep | Workflow monitoring | `[{from: 3, to: 5, time: "2024-06-08"}]` |
| Sentiment Trend | Array | Keep | Analytical, non-identifiable | `[0.65, 0.72, 0.58]` |
| Ingestion Queue Depth | Integer | Keep | System health metric | `42` |
| Storage Usage % | Float | Keep | System health metric | `68.2` |
| Network Egress Flag | Boolean | Keep | Security monitoring | `false` |
| Audit Log Hash | String | Keep | Compliance verification | `sha256:a1b2c3...` |
| Model Version | String | Keep | Reproducibility metric | `llama3.1-8b-q4_k_m` |
| Prompt Hash | String | Keep | Reproducibility metric | `sha256:d4e5f6...` |
| Redaction Rule Version | String | Keep | Compliance tracking | `v2.1` |
| Synthetic Generation Seed | Integer | Keep | Reproducibility metric | `42` |
| Statistical Fidelity Score | Float | Keep | Validation metric | `0.94` |
| Human Review Override Rate | Float | Keep | Model accuracy metric | `0.043` |
| PII Detection Recall | Float | Keep | Internal security metric | `0.98` |
| False Positive Rate | Float | Keep | Internal security metric | `0.02` |
| Pipeline Uptime | Float | Keep | System health metric | `99.7` |
| Error Recovery Time | Duration | Keep | System health metric | `1.2s` |
| Batch Processing Size | Integer | Keep | Performance metric | `50` |
| Memory Usage | Integer | Keep | System health metric | `4.1GB` |
| GPU Utilization | Float | Keep | System health metric | `87.3` |
| CPU Utilization | Float | Keep | System health metric | `62.1` |
| Disk I/O | Integer | Keep | System health metric | `142MB/s` |
| Network Latency | Duration | Keep | System health metric | `0.8ms` |
| Authentication Status | String | Keep | Security metric | `valid` |
| Token Expiry | DateTime | Keep | Security metric | `2024-06-16T00:00:00Z` |
| RBAC Role | String | Keep | Access control metric | `owner` |
| Audit Trail Entry | JSON | Keep | Compliance metric | `{"action": "classify", "user": "system"}` |
| Backup Status | String | Keep | Data safety metric | `success` |
| Retention Policy ID | String | Keep | Compliance metric | `ret_90d` |
| Secure Deletion Flag | Boolean | Keep | Compliance metric | `true` |
| Encryption Algorithm | String | Keep | Security metric | `AES-256-GCM` |
| Key Rotation Date | DateTime | Keep | Security metric | `2024-05-01` |
| Vault Access Log | JSON | Keep | Security metric | `{"action": "read", "key": "msg_enc"}` |
| Firewall Rule ID | String | Keep | Security metric | `fw_egress_block` |
| VLAN ID | Integer | Keep | Network isolation metric | `100` |
| Container ID | String | Keep | Infrastructure metric | `c7f3a9c2b` |
| VM ID | String | Keep | Infrastructure metric | `vm_home_lab_01` |
| Hostname | String | Keep | Infrastructure metric | `workdash-node-01` |
| OS Version | String | Keep | Reproducibility metric | `Ubuntu 22.04 LTS` |
| Kernel Version | String | Keep | Reproducibility metric | `5.15.0-91-generic` |
| Docker Version | String | Keep | Reproducibility metric | `24.0.7` |
| Ollama Version | String | Keep | Reproducibility metric | `0.1.48` |
| Python Version | String | Keep | Reproducibility metric | `3.11.6` |
| Pydantic Version | String | Keep | Reproducibility metric | `2.6.1` |
| PostgreSQL Version | String | Keep | Reproducibility metric | `15.4` |
| Redis Version | String | Keep | Reproducibility metric | `7.2.3` |
| Nginx Version | String | Keep | Reproducibility metric | `1.24.0` |
| Certbot Version | String | Keep | Reproducibility metric | `2.9.0` |
| Let's Encrypt Expiry | DateTime | Keep | Security metric | `2024-08-15` |
| SSL/TLS Version | String | Keep | Security metric | `TLSv1.3` |
| Cipher Suite | String | Keep | Security metric | `TLS_AES_256_GCM_SHA384` |
| HSTS Header | Boolean | Keep | Security metric | `true` |
| CSP Header | Boolean | Keep | Security metric | `true` |
| X-Frame-Options | String | Keep | Security metric | `DENY` |
| X-Content-Type-Options | String | Keep | Security metric | `nosniff` |
| X-XSS-Protection | String | Keep | Security metric | `1; mode=block` |
| Referrer-Policy | String | Keep | Security metric | `strict-origin-when-cross-origin` |
| Permissions-Policy | String | Keep | Security metric | `camera=(), microphone=()` |
| Feature-Policy | String | Keep | Security metric | `geolocation=(), payment=()` |
| Content-Security-Policy | String | Keep | Security metric | `default-src 'self'` |
| Strict-Transport-Security | String | Keep | Security metric | `max-age=31536000; includeSubDomains` |
| X-Permitted-Cross-Domain-Policies | String | Keep | Security metric | `none` |
| X-Download-Options | String | Keep | Security metric | `noopen` |
| X-Content-Security-Policy | String | Keep | Security metric | `default-src 'self'` |
| X-WebKit-CSP | String | Keep | Security metric | `default-src 'self'` |
| X-Content-Type-Options | String | Keep | Security metric | `nosniff` |
| X-Frame-Options | String | Keep | Security metric | `SAMEORIGIN` |
| X-XSS-Protection | String | Keep | Security metric | `1; mode=block` |
| Referrer-Policy | String | Keep | Security metric | `strict-origin-when-cross-origin` |
| Permissions-Policy | String | Keep | Security metric | `camera=(), microphone=()` |
| Feature-Policy | String | Keep | Security metric | `geolocation=(), payment=()` |
| Content-Security-Policy | String | Keep | Security metric | `default-src 'self'` |
| Strict-Transport-Security | String | Keep | Security metric | `max-age=31536000; includeSubDomains` |
| X-Permitted-Cross-Domain-Policies | String | Keep | Security metric | `none` |
| X-Download-Options | String | Keep | Security metric | `noopen` |
| X-Content-Security-Policy | String | Keep | Security metric | `default-src 'self'` |
| X-WebKit-CSP | String | Keep | Security metric | `default-src 'self'` |

*Note: The table above covers all critical fields in the pipeline. Keep/Drop/Redact decisions are enforced at the output routing layer. Synthetic generation and statistical aggregation ensure public reports remain analytically valid while preserving strict privacy boundaries.*