# Goals

The primary objective of the WorkDash analysis workflow is to implement a high-fidelity productivity and communication analysis system that operates on sensitive, real-world data (Email and Microsoft Teams) while maintaining a strict "Air-Gap" philosophy regarding raw content. 

The workflow aims to achieve the following:
1.  **Privacy Preservation:** Ensure that no raw PII (Personally Identifiable Information), proprietary project details, or private conversations ever leave the local home-lab environment.
2.  **Analytical Utility:** Provide the owner with deep, fine-grained insights into their work habits, communication bottlenecks, and sentiment trends using a local Large Language Model (LLM).
3.  **Public Verifiability:** Enable the generation of benchmark reports or productivity showcases that can be shared publicly to demonstrate the efficacy of the WorkDash system without compromising the underlying data.
4.  **Automated Classification:** Shift the burden of data sorting from the human to a local model, allowing for the automatic categorization of "Noise" vs. "Signal" in communication streams.
5.  **Auditability:** Maintain a clear lineage from raw data to redacted summary, allowing the owner to verify that the redaction engine is functioning correctly.

# Data That Must Stay Local

To prevent data leakage, the system defines a "Local-Only Zone." Any data falling into these categories is prohibited from being transmitted to any external API, cloud storage, or public reporting plane.

### 1. Raw Communication Content
*   **Email Bodies:** The full text of sent and received emails.
*   **Teams Chat Logs:** Individual messages, thread histories, and direct messages.
*   **Attachments:** PDFs, spreadsheets, or documents attached to communications.

### 2. Direct Identifiers (PII)
*   **Names:** Full names of colleagues, clients, and family members.
*   **Contact Info:** Email addresses, phone numbers, and physical addresses.
*   **Handles:** User IDs, employee numbers, or internal usernames.

### 3. Proprietary and Sensitive Context
*   **Project Codenames:** Specific internal names for projects (e.g., "Project X-Ray").
*   **Financial Data:** Budget figures, salary discussions, or pricing quotes.
*   **Credentials:** API keys, passwords, or tokens accidentally pasted into chats.
*   **Legal/HR Content:** Performance reviews, disciplinary notes, or contract terms.

### 4. Metadata with High Uniqueness
*   **Exact Timestamps:** While aggregate time-of-day is okay, exact millisecond timestamps can be used for "fingerprinting" a user's activity.
*   **IP Addresses:** Local or remote IP addresses associated with the communication.

# Local Classification

The local classification layer acts as the "Intelligence Filter." Instead of sending data to a cloud provider for analysis, a local model (e.g., Llama 3, Mistral, or a fine-tuned BERT model) is deployed via Ollama or vLLM within the home lab.

### Classification Taxonomy
The local model is prompted to assign every incoming message or email to one of the following categories:
*   **Administrative:** Scheduling, logistics, "thanks" emails, and routine updates.
*   **Technical/Deep Work:** Architecture discussions, debugging, code reviews, and strategic planning.
*   **Interpersonal/Social:** Water-cooler chat, non-work related banter, and networking.
*   **Urgent/Blocker:** Messages indicating a stoppage in work or a critical deadline.
*   **Noise:** Automated notifications, newsletters, and system alerts.

### The Classification Process
1.  **Ingestion:** WorkDash pulls raw data from the API (Teams/Outlook) into a local SQLite or PostgreSQL database.
2.  **Chunking:** Long email threads are broken into logical segments to fit the model's context window.
3.  **Inference:** The local model processes the text using a structured prompt:
    *   *Prompt:* "Analyze the following message. Categorize it as [Admin, Technical, Social, Urgent, Noise]. Extract the primary intent and a sentiment score from -1 to 1. Output in JSON format."
4.  **Tagging:** The resulting JSON is stored alongside the raw data in the local database. The raw text remains, but the "Tags" now allow the owner to query the data without reading every message.

# Redaction Strategy

The redaction strategy is a multi-stage pipeline designed to transform "Private Raw Data" into "Safe Aggregate Data."

### Stage 1: Deterministic Redaction (Regex/Pattern Matching)
The system first applies a set of regular expressions to catch obvious patterns:
*   **Emails:** `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}` $\rightarrow$ `[EMAIL_REDACTED]`
*   **URLs:** `https?://\S+` $\rightarrow$ `[URL_REDACTED]`
*   **Phone Numbers:** Standard international patterns $\rightarrow$ `[PHONE_REDACTED]`

### Stage 2: Named Entity Recognition (NER)
Using a local SpaCy or HuggingFace model, the system identifies entities:
*   **PERSON:** Replaced with a consistent pseudonym (e.g., "John Doe" becomes "Person_A"). This preserves the *relationship* between people (who is talking to whom) without revealing *who* they are.
*   **ORG:** Replaced with `[ORGANIZATION]`.
*   **GPE (Location):** Replaced with `[LOCATION]`.

### Stage 3: LLM-Based Abstractive Redaction
For complex sentences where a name might be implied or a project is mentioned indirectly, the local LLM is used to "rewrite for anonymity."
*   *Raw:* "I think the budget for the New York office's Project Zephyr is too high."
*   *Abstracted:* "The user expressed concern regarding the budget of a specific regional project."

### Stage 4: Aggregation (The Final Shield)
Before any data reaches the reporting plane, it is aggregated. Instead of showing "Person_A sent 5 emails," the system reports "The 'Technical' category saw a 20% increase in volume."

## Field Decision Table

| Field Name | Action | Reason |
| :--- | :--- | :--- |
| `Message_Body` | **Redact/Abstract** | Contains PII and proprietary secrets. |
| `Sender_Name` | **Redact (Pseudonym)** | Necessary for network analysis, but identity must be hidden. |
| `Recipient_Name` | **Redact (Pseudonym)** | Necessary for network analysis, but identity must be hidden. |
| `Timestamp` | **Keep (Rounded)** | Round to nearest hour to preserve trend analysis while preventing fingerprinting. |
| `Category_Tag` | **Keep** | Generated by local model; contains no raw PII. |
| `Sentiment_Score` | **Keep** | Numerical value; safe for reporting. |
| `Project_Name` | **Drop/Replace** | High risk of leakage; replace with "Project_1", "Project_2". |
| `Email_Subject` | **Redact/Abstract** | Often contains specific client names or sensitive topics. |
| `Attachment_Name` | **Drop** | File names often reveal project specifics. |
| `Message_Length` | **Keep** | Useful for productivity metrics. |

# Synthetic Test Generation

To ensure the reporting plane is useful and the redaction is working, the system uses a "Mirror Dataset" approach.

### The Mirroring Process
1.  **Pattern Extraction:** The local model analyzes the *distribution* of the real data (e.g., "The user typically receives 10 Technical messages and 5 Admin messages on Tuesdays").
2.  **Synthetic Generation:** The local LLM is prompted to generate entirely fake conversations that mimic these patterns.
    *   *Prompt:* "Generate 5 synthetic Teams messages between two fictional engineers discussing a fake project called 'Operation Blue-Sponge'. The tone should be professional but stressed. Ensure the length and frequency match the provided distribution."
3.  **Validation:** These synthetic examples are pushed through the same redaction pipeline. If the redaction engine fails to redact the synthetic "Operation Blue-Sponge," it indicates a failure in the NER/Regex layer.
4.  **Public Benchmarking:** When publishing a report on "WorkDash Efficiency," the user provides the synthetic examples as "Sample Data" to show how the system works, while the actual metrics are derived from the redacted real data.

# Metrics To Keep

The reporting plane focuses on "Meta-Metrics"—data about the data—rather than the data itself.

### 1. Volume Metrics
*   **Messages per Day/Week:** Total count of communications.
*   **Category Distribution:** Percentage of time spent on Admin vs. Technical vs. Social.
*   **Response Latency:** Average time between receiving a message and replying (aggregated by category).

### 2. Sentiment and Health Metrics
*   **Sentiment Trend:** A rolling average of sentiment scores over a month.
*   **Urgency Spike:** Frequency of "Urgent/Blocker" tags, used to identify burnout or project crises.
*   **Communication Balance:** Ratio of sent vs. received messages.

### 3. Productivity Correlations
*   **Deep Work Blocks:** Correlation between low "Admin" message volume and high "Technical" output.
*   **Context Switching Frequency:** How often the user jumps between different categories of communication within a single hour.

# Human Review

The "Human-in-the-Loop" (HITL) stage is the final circuit breaker before any data is moved from the Local Database to the Reporting Plane.

### The Review Workflow
1.  **Sample Audit:** The system presents the owner with 10 random pairs of `[Raw Text]` $\rightarrow$ `[Redacted Text]`.
2.  **Verification:** The owner confirms that no PII leaked through the redaction.
3.  **Override:** If the owner finds a leak (e.g., a project name that the NER missed), they add that specific term to a "Local Blacklist" (a custom regex list) to ensure it is caught in all future passes.
4.  **Approval:** The owner clicks "Commit to Report," which triggers the movement of the aggregated, redacted data to the public-facing dashboard.

# Failure Modes

Despite the design, several failure modes exist that the owner must monitor:

### 1. The "Unique Identifier" Leak
*   **Scenario:** A user mentions a very specific, rare technical error code or a unique phrase that only exists in one company.
*   **Risk:** Even if the name is redacted, a savvy observer could search for that error code online and find the company or project.
*   **Mitigation:** Use the LLM to replace specific technical jargon with generic terms (e.g., "Error 0x88234" $\rightarrow$ "a specific system error").

### 2. Model Hallucination in Redaction
*   **Scenario:** The local LLM, while attempting to abstract a sentence, accidentally invents a detail or fails to remove a name because it misidentified the entity.
*   **Risk:** False sense of security; the owner believes the data is redacted when it is not.
*   **Mitigation:** Prioritize deterministic Regex and NER over LLM abstraction for the primary redaction pass.

### 3. Inference Attacks
*   **Scenario:** An observer sees that "Person_A" always messages "Person_B" at 9:00 AM and 5:00 PM.
*   **Risk:** By correlating this with public LinkedIn data (e.g., knowing who the manager is), the observer can deanonymize the pseudonyms.
*   **Mitigation:** Apply "K-anonymity" by grouping users into roles (e.g., "Manager," "Peer," "Client") rather than using individual pseudonyms.

# Example Private Summary

*This summary exists only in the local SQLite database and is never exported.*

**Date:** 2023-10-12
**User:** Alice Smith
**Total Messages:** 42
**Key Events:**
- **09:15 AM:** Received urgent email from Bob Jones regarding the "Project Zephyr" budget overrun in the London office. Sentiment: -0.8 (Stressed).
- **11:00 AM:** Teams thread with Sarah Lee and Mike Chen regarding the API migration bug in the production environment. Category: Technical.
- **02:00 PM:** 15-minute social chat with the #random channel about the new espresso machine in the breakroom.
- **04:30 PM:** Sent a follow-up to the VP of Engineering, David Vance, regarding the Q4 roadmap.

**Local Insight:** Alice is spending 40% of her morning on "Urgent" blockers, which is delaying her "Technical" deep work until after 2 PM.

# Example Publishable Summary

*This summary is generated from redacted aggregates and synthetic examples for a public benchmark report.*

**Analysis Period:** October Week 2
**User Profile:** Senior Engineer (Anonymized)
**Aggregate Metrics:**
- **Communication Volume:** 280 messages/week.
- **Work Distribution:** 
    - Technical: 55%
    - Administrative: 25%
    - Social: 15%
    - Urgent: 5%
- **Average Response Time:** 42 minutes (Technical), 110 minutes (Admin).
- **Sentiment Trend:** Stable, with a slight dip on Tuesday corresponding with a spike in "Urgent" category messages.

**Synthetic Example of Workflow (For Demonstration):**
- *Input:* "Hey [Person_A], did you see the update on [Project_1]? The budget is looking tight."
- *Processed Output:* "User inquired about the financial status of a specific project."

**Conclusion:** The WorkDash system successfully identified a pattern of "Morning Fragmentation," where administrative tasks interrupt technical flow. By shifting admin tasks to a dedicated block, the user could potentially increase deep-work efficiency by 15%.