# Goals

The primary objective of the WorkDash analysis workflow is to establish a privacy-preserving, locally-executed communication analytics pipeline that extracts actionable operational insights from real email and Microsoft Teams messages while guaranteeing zero external exposure of raw private content. The workflow is designed specifically for a home-lab environment, balancing computational constraints with enterprise-grade data handling practices. The goals are structured across four dimensions:

1. **Privacy-First Architecture**: All raw message payloads, metadata, attachments, and conversation threads must remain within the local network boundary. No plaintext or minimally transformed private data may traverse external networks, cloud APIs, or third-party telemetry endpoints. The system enforces a strict data gravity model where sensitive information never leaves the host machine or local VLAN.

2. **Actionable Owner Reporting**: Despite aggressive redaction and localization, the reporting plane must deliver high-fidelity insights to the lab owner. This includes communication bottlenecks, project velocity indicators, sentiment trends, action-item tracking, and meeting efficiency metrics. The dashboard must support drill-down capabilities, temporal filtering, and cross-channel correlation without compromising the underlying privacy guarantees.

3. **Safe Public Benchmarking**: The workflow must produce publishable benchmark reports that demonstrate analytical capability, model performance, and pipeline robustness without leaking proprietary or personal information. This is achieved through synthetic data generation, statistical aggregation, and deterministic redaction. The public-facing outputs must be reproducible, version-controlled, and suitable for community sharing or external evaluation.

4. **Local Model Utilization**: Fine-grained classification, entity extraction, and intent recognition are performed exclusively by locally-hosted models. This eliminates dependency on remote inference APIs, reduces latency, and ensures that sensitive contextual patterns are never serialized for external processing. The local model stack is optimized for the home-lab hardware profile, leveraging quantization, batching, and efficient memory management.

5. **Auditability & Reproducibility**: Every transformation, classification decision, and redaction operation is logged locally with cryptographic integrity checks. The pipeline supports deterministic replay, versioned model checkpoints, and transparent data lineage. This ensures that benchmark results can be independently verified and that the owner can trace any insight back to its source without exposing raw content.

# Data That Must Stay Local

The following data categories are classified as strictly local and must never leave the home-lab boundary under any operational condition:

- **Raw Message Payloads**: Full email bodies, Teams chat messages, channel posts, and direct messages in their original format (HTML, plain text, or rich text). This includes inline images, embedded links, and formatting metadata.
- **Conversation Threads & Context**: Message chains, reply hierarchies, quote blocks, and thread continuation markers. Contextual dependencies are critical for accurate classification and must be preserved locally.
- **Metadata & Headers**: Sender/recipient addresses, display names, timestamps, message IDs, routing paths, server fingerprints, and protocol-specific headers (e.g., X-MS-Exchange, Message-ID, In-Reply-To).
- **Attachments & Embedded Media**: PDFs, spreadsheets, code snippets, design files, voice notes, and screen recordings attached to messages or shared in Teams. Binary content is indexed locally but never transmitted.
- **User & Organizational Context**: Local directory mappings, team rosters, project assignments, role definitions, and internal naming conventions. Even if publicly available elsewhere, internal usage patterns constitute sensitive operational data.
- **Model Inference Logs**: Intermediate activations, attention weights, token probabilities, and classification confidence scores generated during local processing. These can reveal training data patterns or prompt structures if exposed.
- **Redaction Artifacts & Mapping Tables**: Pseudonymization dictionaries, entity replacement tables, and redaction rule configurations. These are cryptographic keys in disguise and must be stored with the same protection level as raw data.

Storage architecture enforces these constraints through:
- **Encrypted Local Volumes**: LUKS or BitLocker encryption at rest, with keys derived from local hardware-bound secrets.
- **Network Isolation**: Dedicated VLAN or air-gapped processing node with no outbound internet routing. Inbound feeds are pulled via authenticated, encrypted local bridges (e.g., IMAP/SMTP over TLS, Teams Graph API via local proxy).
- **Access Controls**: Role-based local authentication, session timeouts, and audit logging for all data access. No remote desktop or cloud sync is permitted for the processing environment.
- **Lifecycle Management**: Automated archival to cold storage after configurable retention periods, with cryptographic shredding upon deletion. Backup routines are encrypted and stored on local NAS devices.

# Local Classification

The local classification pipeline leverages a fine-tuned open-source language model optimized for communication analytics. The architecture is designed for home-lab hardware constraints while maintaining high accuracy on fine-grained tasks.

**Model Selection & Optimization**:
- Base model: Mistral-7B-Instruct or Llama-3-8B, quantized to 4-bit or 8-bit precision using GGUF or AWQ formats.
- Fine-tuning: LoRA adapters trained on a curated local dataset of anonymized communication samples, focusing on intent recognition, urgency detection, topic classification, and action-item extraction.
- Inference engine: llama.cpp or vLLM running on local GPU/CPU, with dynamic batching and KV-cache optimization for throughput.

**Classification Tasks**:
1. **Intent & Actionability**: Identifies requests, decisions, deadlines, and follow-up requirements. Outputs structured JSON with confidence scores.
2. **Sentiment & Tone**: Classifies emotional valence (positive, neutral, negative, urgent, frustrated) and communication style (formal, casual, directive, collaborative).
3. **Topic & Project Mapping**: Assigns messages to internal project tags, functional domains, or workflow stages using hierarchical classification.
4. **Sensitivity Scoring**: Estimates data sensitivity based on content patterns, entity density, and contextual cues. Triggers redaction rules accordingly.
5. **Entity & Relationship Extraction**: Identifies people, dates, locations, document references, and cross-message dependencies. Builds a local knowledge graph for correlation.

**Pipeline Architecture**:
- **Preprocessing**: HTML stripping, normalization, chunking (sliding window with overlap), and metadata extraction.
- **Inference**: Batched forward pass through the local model, with temperature=0.1 for deterministic outputs. Confidence thresholds filter low-probability predictions.
- **Post-processing**: Schema validation, conflict resolution (e.g., overlapping tags), and graph edge creation. Results are stored in a local SQLite/PostgreSQL database with full-text indexing.
- **Feedback Loop**: Human-reviewed corrections are logged and used for periodic LoRA fine-tuning, ensuring model adaptation to evolving communication patterns.

**Hardware & Performance**:
- Target: 16GB+ VRAM GPU or 32GB+ RAM CPU with AVX-512.
- Throughput: ~50-100 messages/second depending on quantization and batch size.
- Latency: <200ms per message for real-time dashboard updates.
- Fallback: CPU-only mode with reduced batch size if GPU resources are constrained.

# Redaction Strategy

Redaction is the critical bridge between private analysis and public benchmarking. The strategy employs a multi-layered approach to remove sensitive information while preserving analytical utility.

**Redaction Levels**:
1. **Full Removal**: Complete deletion of high-sensitivity spans (e.g., passwords, financial data, personal identifiers). Replaced with `[REDACTED]` or omitted entirely.
2. **Masking**: Character-level obfuscation (e.g., `j***@example.com`, `2024-XX-XX`). Preserves structure and length for tokenization compatibility.
3. **Generalization**: Replacing specific values with categories (e.g., `Project Alpha` → `Project [A]`, `Q3 budget` → `Quarterly financials`).
4. **Pseudonymization**: Consistent substitution using a local mapping table (e.g., `Alice` → `Person_01`, `Engineering` → `Team_X`). Maintains relational integrity for graph analysis.

**Technical Implementation**:
- **Rule-Based Layer**: Regex patterns for emails, phone numbers, IPs, dates, and common PII formats. Applied first for speed and determinism.
- **NER Layer**: Local spaCy or Hugging Face pipeline for named entity recognition. Identifies persons, organizations, locations, and custom entities.
- **LLM-Assisted Layer**: The local model evaluates context to determine redaction necessity. Prompts are structured to output JSON spans with sensitivity scores and replacement suggestions.
- **Context-Aware Replacement**: Ensures grammatical coherence post-redaction. Uses placeholder tokens that preserve part-of-speech and syntactic role.

**Validation & Safety**:
- **Automated Checks**: Post-redaction scans verify no residual PII using multiple detectors. Differential privacy noise is added to aggregate counts if k-anonymity thresholds are unmet.
- **Utility Preservation**: Structural metadata (timestamps, thread IDs, message lengths, attachment counts) is retained. Semantic relationships are preserved through pseudonymization.
- **Audit Trail**: Every redaction operation logs the original span, applied rule, confidence score, and replacement value. Logs are encrypted and stored locally.

**Workflow Integration**:
1. Raw message ingested → Preprocessed → Classified.
2. Sensitivity score triggers redaction pipeline.
3. Redacted version stored in isolated "publishable" database.
4. Original remains in "private" database, accessible only to owner dashboard.
5. Cross-references maintained via secure hash mapping for internal correlation.

# Synthetic Test Generation

Synthetic data generation enables public benchmarking without exposing real communication patterns. The process creates statistically equivalent, semantically plausible datasets that preserve analytical properties while eliminating reconstruction risk.

**Generation Architecture**:
- **Template-Based Foundation**: Structured templates define message formats, thread patterns, and metadata schemas. Ensures consistency and reproducibility.
- **LLM Synthesis**: Local model generates content conditioned on distribution parameters (topic frequency, sentiment distribution, length histograms, temporal patterns). Prompts enforce strict constraints: no real names, no proprietary terms, no identifiable contexts.
- **Distribution Matching**: Generated data is validated against real data statistics using Kolmogorov-Smirnov tests, chi-square distributions, and correlation matrices. Adjustments are made iteratively until statistical parity is achieved.
- **Adversarial Filtering**: Synthetic outputs are run through reconstruction attacks and PII detectors. Any sample that leaks structural patterns or enables inference is discarded and regenerated.

**Techniques & Controls**:
- **Topic Preservation**: Synthetic messages maintain the same topic distribution and co-occurrence patterns as real data, enabling valid benchmark comparisons.
- **Style & Length Matching**: Token counts, punctuation frequency, and formatting styles are matched to prevent distributional drift.
- **Temporal Realism**: Timestamps follow realistic work-hour patterns, meeting schedules, and response latency distributions.
- **Versioning & Reproducibility**: Each synthetic dataset is tagged with a seed, model version, and generation parameters. Enables exact reproduction for benchmark validation.

**Integration with Reporting**:
- Synthetic data feeds directly into public benchmark reports.
- Metrics computed on synthetic data are compared against redacted real-data metrics to validate pipeline accuracy.
- Public reports include methodology notes, generation parameters, and statistical validation results.
- Synthetic datasets are published alongside benchmark code for community verification.

# Metrics To Keep

The reporting plane retains a curated set of metrics that provide actionable insights while respecting privacy constraints. All metrics are derived from redacted or aggregated data, ensuring no raw content leakage.

**Core Metrics**:
1. **Communication Volume**: Messages per day/channel, peak activity hours, cross-channel distribution.
2. **Response Latency**: Median/percentile response times, bottleneck identification, SLA compliance tracking.
3. **Sentiment Distribution**: Proportions of positive/neutral/negative/urgent tones over time. Trend analysis for team morale or project stress.
4. **Topic Frequency & Evolution**: Dominant discussion themes, emerging topics, topic decay rates. Enables resource allocation insights.
5. **Action-Item Tracking**: Creation rate, completion rate, aging distribution, assignment patterns. Measures operational efficiency.
6. **Meeting Efficiency**: Duration vs. actionability ratio, overlap detection, preparation indicators (pre-read mentions, agenda references).
7. **Network Density**: Communication graph metrics (centrality, clustering coefficient, bridge nodes). Identifies key collaborators and silos.
8. **Anomaly Detection**: Statistical outliers in volume, sentiment, or latency. Flags potential issues for human review.

**Aggregation & Safety**:
- Metrics are computed over rolling windows (daily, weekly, monthly) with configurable granularity.
- Small-count suppression: Categories with <5 instances are merged or omitted to prevent identification.
- Differential privacy: Laplace or Gaussian noise added to sensitive aggregates when necessary.
- All metrics are stored in a separate analytics database, decoupled from raw message storage.

**Dashboard Utility**:
- Interactive visualizations: time series, heatmaps, network graphs, funnel charts.
- Drill-down capabilities: filter by project, channel, time range, or sentiment.
- Alerting: configurable thresholds for latency spikes, sentiment drops, or volume anomalies.
- Export: CSV/JSON for external analysis, with metadata stripped and aggregates preserved.

# Human Review

Human-in-the-loop review ensures classification accuracy, redaction safety, and model improvement. The workflow is designed for efficiency, privacy, and continuous learning.

**Review Interface**:
- Local web dashboard with role-based access.
- Side-by-side view: original message (blurred/redacted by default) vs. classified/redacted version.
- Confidence scores, highlighted spans, and suggested actions displayed prominently.
- Quick-action buttons: approve, override, flag, reclassify, add note.

**Workflow Process**:
1. **Queue Generation**: Messages with low confidence scores, high sensitivity, or conflicting classifications are prioritized.
2. **Review Session**: Owner reviews items in batches. Overrides are logged with rationale.
3. **Feedback Capture**: Corrections are stored in a structured format (original span, correct label, confidence, timestamp).
4. **Model Retraining**: Periodic LoRA fine-tuning incorporates review data. Validation ensures no performance degradation on held-out sets.
5. **Audit & Compliance**: All review actions are logged locally. Session recordings are disabled to prevent accidental leakage.

**Privacy Safeguards**:
- Review environment is air-gapped from external networks.
- No telemetry, analytics, or crash reporting leaves the local machine.
- Session auto-logout after inactivity. Screen capture prevention via OS-level flags.
- Review data is encrypted and stored with the same protection as raw messages.

**Efficiency Optimizations**:
- Smart prioritization: ML-driven queue ordering based on impact and uncertainty.
- Batch processing: Group similar messages for faster review.
- Keyboard shortcuts: Rapid navigation and action execution.
- Progress tracking: Completion metrics, backlog aging, review velocity.

# Failure Modes

Understanding potential failures is critical for maintaining system integrity. The following modes are identified with mitigation strategies:

1. **Model Hallucination & Misclassification**:
   - *Symptom*: Incorrect intent labels, false urgency flags, spurious entity extraction.
   - *Mitigation*: Confidence thresholds, human review queue, fallback to rule-based classification, periodic retraining.

2. **Redaction Leakage**:
   - *Symptom*: Residual PII, reconstructable patterns, metadata correlation attacks.
   - *Mitigation*: Multi-pass validation, adversarial testing, k-anonymity enforcement, differential privacy, strict template controls.

3. **Synthetic Data Overfitting**:
   - *Symptom*: Generated data mirrors real patterns too closely, enabling inference.
   - *Mitigation*: Distribution matching with noise injection, reconstruction attack testing, topic generalization, seed rotation.

4. **Hardware & Pipeline Bottlenecks**:
   - *Symptom*: Inference latency spikes, queue backlog, storage exhaustion.
   - *Mitigation*: Dynamic batching, priority queuing, automated archival, hardware monitoring, graceful degradation to CPU mode.

5. **False Positives/Negatives in Sensitivity Scoring**:
   - *Symptom*: Over-redaction (utility loss) or under-redaction (privacy risk).
   - *Mitigation*: Tunable sensitivity thresholds, human override, context-aware LLM evaluation, periodic threshold calibration.

6. **Backup & Log Exposure**:
   - *Symptom*: Encrypted backups compromised, audit logs contain sensitive spans.
   - *Mitigation*: Key rotation, log sanitization, secure deletion protocols, offline backup storage, integrity verification.

7. **Model Drift & Concept Shift**:
   - *Symptom*: Classification accuracy degrades over time as communication patterns evolve.
   - *Mitigation*: Continuous monitoring, drift detection metrics, scheduled retraining, versioned model rollbacks.

8. **Insider/Owner Error**:
   - *Symptom*: Accidental export, misconfigured rules, dashboard sharing.
   - *Mitigation*: Principle of least privilege, export controls, configuration validation, user training, automated safety checks.

# Example Private Summary

**Project**: Q3 Infrastructure Migration  
**Timeframe**: 2024-09-01 to 2024-09-30  
**Participants**: Sarah Chen (Lead), Marcus Rodriguez (DevOps), Priya Patel (Security), James Wu (QA)  

**Communication Volume**: 1,247 messages across Teams channels and email. Peak activity: 10:00-12:00 UTC.  
**Sentiment Distribution**: Neutral (62%), Positive (21%), Urgent (12%), Negative (5%).  
**Action Items**: 47 created, 39 completed, 8 aging >7 days.  

**Key Threads**:
- *Thread 1*: Database schema validation delays. Marcus flagged missing indexes on `user_sessions` table. Sarah assigned Priya to review security implications. Deadline: 2024-09-15. Status: Completed.
- *Thread 2*: CI/CD pipeline failures. James reported flaky tests in `auth-service`. Marcus proposed retry logic with exponential backoff. Sarah approved. Status: In Progress.
- *Thread 3*: Compliance audit prep. Priya requested documentation for data retention policies. Sarah delegated to legal team. Status: Pending.

**Metrics**:
- Median response time: 2.4 hours. P95: 8.1 hours.
- Meeting overlap: 3 instances (avg. 45 min).
- Action-item completion rate: 83%.
- Sentiment trend: Slight negative shift on 2024-09-22 due to deployment rollback.

**Raw Snippet (Internal Only)**:  
`[2024-09-14 14:32] Marcus: "The migration script failed on prod-db-03. Error: 'relation "audit_logs" does not exist'. Need to run the DDL patch before proceeding. Sarah, can you approve the change window?`  
`[2024-09-14 14:45] Sarah: "Approved. Coordinate with Priya for security sign-off. Target: 2024-09-15 02:00 UTC."`

**Classification Tags**: `#infrastructure`, `#database`, `#urgent`, `#action-required`, `#security-review`  
**Sensitivity Score**: 0.87 (High)  
**Redaction Status**: Pending (Private View)

# Example Publishable Summary

**Project**: [Project_A]  
**Timeframe**: 2024-Q3  
**Participants**: [Person_01], [Person_02], [Person_03], [Person_04]  

**Communication Volume**: 1,247 messages across [Channel_X] and email. Peak activity: 10:00-12:00 UTC.  
**Sentiment Distribution**: Neutral (62%), Positive (21%), Urgent (12%), Negative (5%).  
**Action Items**: 47 created, 39 completed, 8 aging >7 days.  

**Key Threads**:
- *Thread 1*: Schema validation delays. [Person_02] flagged missing indexes on [Table_01]. [Person_01] assigned [Person_03] to review implications. Deadline: 2024-09-15. Status: Completed.
- *Thread 2*: Pipeline failures. [Person_04] reported flaky tests in [Service_01]. [Person_02] proposed retry logic. [Person_01] approved. Status: In Progress.
- *Thread 3*: Compliance prep. [Person_03] requested documentation for retention policies. [Person_01] delegated to external team. Status: Pending.

**Metrics**:
- Median response time: 2.4 hours. P95: 8.1 hours.
- Meeting overlap: 3 instances (avg. 45 min).
- Action-item completion rate: 83%.
- Sentiment trend: Slight negative shift on 2024-09-22 due to deployment event.

**Synthetic Snippet (Benchmark Use)**:  
`[2024-09-14 14:32] [Person_02]: "The migration script failed on [Node_03]. Error: 'relation [Table_02] does not exist'. Need to run the patch before proceeding. [Person_01], can you approve the window?"`  
`[2024-09-14 14:45] [Person_01]: "Approved. Coordinate with [Person_03] for sign-off. Target: 2024-09-15 02:00 UTC."`

**Classification Tags**: `#infrastructure`, `#database`, `#urgent`, `#action-required`, `#security-review`  
**Sensitivity Score**: [REDACTED]  
**Redaction Status**: Complete (Publishable View)  
**Benchmark Validation**: Statistical parity confirmed (KS test p=0.89, χ² p=0.92). Synthetic generation seed: `synth_q3_2024_v2`.

# Field-Level Data Handling Decisions

| Field | Source | Decision | Rationale | Transformation Method |
|-------|--------|----------|-----------|------------------------|
| Sender Email | Email/Teams Header | Redact | Direct PII, enables identification | Pseudonymization (`Person_XX`) |
| Recipient List | Email/Teams Header | Redact | Reveals communication graph & roles | Pseudonymization + count preservation |
| Message Body | Payload | Redact | Contains sensitive context, quotes, decisions | NER + LLM span redaction + placeholder substitution |
| Timestamp | Metadata | Keep | Critical for temporal analysis, latency metrics | None (preserved exactly) |
| Subject Line | Email Header | Redact | Often contains project names, urgent flags | Generalization (`[Topic_XX]`) |
| Attachment Content | Binary/Text | Drop | High sensitivity, large volume, hard to redact safely | Hash-only indexing, content excluded from analysis |
| Meeting Transcript | Teams/Calendar | Redact | Verbatim speech, high PII density | Speaker diarization + pseudonymization + topic summarization |
| Sentiment Score | Model Output | Keep | Aggregate metric, no raw content leakage | None (preserved as float) |
| Urgency Flag | Model Output | Keep | Operational metric, safe for public | None (preserved as boolean) |
| Project Tag | Classification | Redact | Internal naming, reveals org structure | Pseudonymization (`Project_XX`) |
| User ID | Directory/Graph | Redact | Direct identifier | Pseudonymization + role mapping |
| IP Address | Header/Log | Drop | Network fingerprint, location inference | Excluded from pipeline |
| Device Info | Header/Log | Drop | Hardware fingerprint, privacy risk | Excluded from pipeline |
| Thread ID | Metadata | Keep | Enables correlation, no PII | None (preserved as hash) |
| Message Length | Metadata | Keep | Statistical metric, safe | None (preserved as integer) |
| Confidence Score | Model Output | Keep | Benchmark metric, no raw content | None (preserved as float) |
| Action Item Text | Extraction | Redact | Contains deadlines, names, specifics | Pseudonymization + deadline generalization |
| Meeting Duration | Calendar | Keep | Efficiency metric, safe | None (preserved as integer) |
| Response Latency | Computed | Keep | Operational metric, safe | None (preserved as float) |
| Graph Edge Weight | Computed | Keep | Network metric, safe | None (preserved as float) |

This table governs all data transformations within the WorkDash pipeline. Fields marked "Keep" are preserved in both private and publishable views. Fields marked "Redact" undergo deterministic transformation before entering the publishable database. Fields marked "Drop" are excluded from analysis entirely, with only cryptographic hashes retained for integrity verification. All transformations are logged, versioned, and auditable within the local environment.