# Goals

The primary objective of the WorkDash analysis workflow is to create a high-fidelity productivity and communication analytics engine that operates within a "Zero-Trust" privacy framework for a home lab environment. The system aims to achieve three core goals:

1.  **Actionable Intelligence:** Transform raw, noisy communication (Emails, Teams messages) into structured data to identify bottlenecks, project progress, and personal productivity trends.
2.  **Absolute Data Sovereignty:** Ensure that no PII (Personally Identifiable Information), proprietary corporate secrets, or private personal sentiments ever leave the local network or enter a public cloud-based LLM.
3.  **Benchmarkable Utility:** Create a reporting plane that allows the owner to share "success metrics" or "workflow improvements" with the public or a community without exposing the underlying sensitive data.

By separating the **Processing Plane** (Local/Private) from the **Reporting Plane** (Synthetic/Public), the owner can derive value from their data while maintaining a perfect privacy posture.

# Data That Must Stay Local

To maintain the integrity of the private home lab, the following data categories are strictly prohibited from leaving the local environment (i.e., they cannot be sent to OpenAI, Anthropic, or any external API):

*   **Raw Message Bodies:** The literal text of every email and Teams message.
*   **Sender/Recipient Identities:** Real names, email addresses, and internal handles.
*   **Project Codenames:** Internal project titles (e.g., "Project Phoenix," "Operation Stealth") that could reveal corporate strategy.
*   **Specific Dates and Times:** Precise timestamps that could be used to correlate activities with external events.
*   **Financial Data:** Any mention of budgets, salaries, pricing, or revenue.
*   **Credentials and Tokens:** Any API keys, passwords, or session tokens accidentally pasted into chat or email.
*   **Sentiment Nuance:** The raw emotional state of a specific conversation (e.g., "The boss is angry about the delay").

# Local Classification

The workflow utilizes a local LLM (e.g., Llama 3 8B or Mistral 7B running via Ollama or LocalAI) to perform "Fine-Grained Classification." This happens entirely on the home lab hardware.

**The Classification Pipeline:**
1.  **Ingestion:** A local script polls the Outlook/Teams API (using local credentials) and stores raw data in a local PostgreSQL database.
2.  **Entity Extraction (NER):** The local model identifies entities: *Person, Organization, Project, Deadline, Action Item.*
3.  **Intent Classification:** The model categorizes the message into types:
    *   *Request for Information (RFI)*
    *   *Status Update*
    *   *Urgent Escalation*
    *   *Social/Chitchat*
    *   *Task Assignment*
4.  **Priority Scoring:** A 1-10 score based on keywords (e.g., "ASAP," "Blocker," "Critical") and the sender's historical importance.
5.  **Action Item Extraction:** The model parses the text to find specific tasks (e.g., "Send the report by 5 PM") and maps them to a "To-Do" schema.

**Why Local?** By using a local model for this step, we can feed it the *entire* raw message to get the most accurate classification without risking a data leak.

# Redaction Strategy

Before any data moves toward the "Reporting Plane," it passes through a multi-stage Redaction Engine.

**Stage 1: Deterministic Scrubbing (Regex)**
*   Removes standard patterns: Emails, Phone Numbers, IP Addresses, and Credit Card numbers.

**Stage 2: Contextual NER (Local LLM)**
*   The local model scans the text and replaces specific entities with generic placeholders.
*   *Example:* "John Doe needs the Phoenix budget by Friday" $\rightarrow$ "[PERSON] needs the [PROJECT] budget by [DATE]."

**Stage 3: Semantic Generalization**
*   Instead of just redacting, the system "generalizes."
*   *Example:* "The server is down in the London office" $\rightarrow$ "Infrastructure issue in [LOCATION]."

**Stage 4: Aggregation**
*   The final step for the public report is to move from "Individual Messages" to "Aggregated Counts." Instead of showing a redacted message, the system reports: "Received 14 High-Priority Infrastructure Requests this week."

# Synthetic Test Generation

To provide a benchmark that is actually useful for others to see, we use **Template-Based Synthetic Generation**.

Instead of showing a redacted version of a real message (which can still leak context), the system identifies the *structure* of the real data and generates a completely fake message that mirrors it.

**The Process:**
1.  **Identify Structure:** The system notes that a message was a "Project Update" with a "High Priority" and a "Technical Blocker."
2.  **Generate Synthetic Seed:** The system uses a prompt: *"Generate a fake, professional Teams message about a technical blocker in a software project. Do not use real names."*
3.  **Inject Metrics:** The synthetic message is paired with the *real* metadata (e.g., "Response Time: 4 minutes," "Sentiment: Frustrated").

This allows a public benchmark to say: *"Here is a synthetic example of a high-priority blocker I handled, and here is the actual efficiency metric I achieved."*

# Metrics To Keep

The following metrics are deemed "Safe" because they are abstract enough to provide value without revealing private content. These are the primary outputs of the WorkDash reporting plane:

*   **Volume Metrics:** Total messages received per day/week/month.
*   **Velocity Metrics:** Average response time (Time from message receipt to first reply).
*   **Sentiment Trends:** Aggregated sentiment scores (e.g., "Average team sentiment: 7.2/10") without individual attribution.
*   **Category Distribution:** Pie charts showing the percentage of "Admin" vs. "Deep Work" vs. "Urgent" communications.
*   **Action Item Completion Rate:** Percentage of extracted tasks marked as "Done" within the target timeframe.
*   **Focus Time Analysis:** Correlation between high message volume and decreases in "Deep Work" blocks.

# Human Review

A "Human-in-the-Loop" (HITL) step is mandatory for the transition from the Private Plane to the Public Plane.

1.  **The Redaction Audit:** Once a week, the owner reviews a "Redaction Log." This log shows: *[Original Snippet] $\rightarrow$ [Redacted/Synthetic Version]*.
2.  **False Positive Check:** The owner ensures the local model didn't over-redact (making the data useless) or under-redact (leaving a name in).
3.  **Synthetic Validation:** The owner confirms that the synthetic examples generated for the benchmark accurately represent the *type* of work performed without being too specific.
4.  **Approval Gate:** A "Publish" button is only enabled after the owner manually signs off on the weekly aggregate.

# Failure Modes

1.  **Over-Redaction:** The model replaces so much text that the "Action Item" becomes unidentifiable, rendering the local dashboard useless for the owner.
    *   *Mitigation:* Maintain a "Raw" view that is strictly local and never touches the reporting logic.
2.  **Contextual Leakage:** A redacted message like "[PERSON] is mad about the [PROJECT] delay" still reveals that a project is delayed.
    *   *Mitigation:* Use "Aggregated Reporting" for public views rather than "Redacted Message Lists."
3.  **Model Hallucination:** The local LLM might "invent" an action item that wasn't in the message.
    *   *Mitigation:* Require the model to provide a "Quote" or "Source Snippet" for every action item it extracts.
4.  **Inference Latency:** Running a large local model for every message can slow down the dashboard.
    *   *Mitigation:* Use a small, fast model (e.g., Phi-3) for initial classification and a larger model (e.g., Llama 3 70B) for complex summarization.

# Example Private Summary

**Internal Dashboard View (Owner Only)**
> **Date:** 2023-10-27
> **Summary:** High volume of communication regarding the "Project Phoenix" migration.
> **Key Action Items:**
> 1. [URGENT] John Doe requested the final API keys by 4 PM today. (Status: Pending)
> 2. [TASK] Update the documentation for the London server cluster. (Status: In Progress)
> **Sentiment Analysis:**
> - Project Phoenix: 4/10 (High stress detected in messages from Sarah Miller).
> - General Admin: 8/10 (Routine).
> **Productivity Insight:** You spent 45% of your morning responding to "Urgent" pings, delaying the "Deep Work" block on the database schema.

# Example Publishable Summary

**Public Benchmark Report (Synthetic/Aggregated)**
> **Date:** 2023-10-27
> **Workflow Performance:**
> - **Total Messages Processed:** 142
> - **Avg. Response Time:** 12 minutes
> - **Action Item Completion:** 88%
>
> **Workload Distribution:**
> - Infrastructure: 40%
> - Project Management: 30%
> - General Admin: 30%
>
> **Synthetic Example of High-Priority Handling:**
> *Synthetic Message:* "We are seeing a critical failure in the production environment. Need a status update on the rollback procedure immediately."
> *Actual Metric:* Handled in 4 minutes; 100% resolution rate.
>
> **Owner Insight:** By using WorkDash to categorize "Urgent" pings, I was able to identify that 60% of "Urgent" messages were actually non-critical, allowing me to batch-process them and reclaim 2 hours of deep work time per week.

***

# Data Field Decision Matrix

| Field | Decision | Reason |
| :--- | :--- | :--- |
| Raw Email Body | **Drop** | Contains PII and proprietary secrets; never leaves local. |
| Sender Name | **Redact** | Replace with [PERSON] or [ROLE] in reporting. |
| Project Name | **Redact** | Replace with [PROJECT_TYPE] (e.g., "Cloud Migration"). |
| Exact Timestamp | **Drop** | Replace with "Morning," "Afternoon," or "Relative Time." |
| Sentiment Score | **Keep** | Abstract numerical value (e.g., 0.8) is safe for benchmarking. |
| Action Item Text | **Redact** | Replace with "Task: [Technical/Admin/Meeting]." |
| Response Time | **Keep** | Purely a performance metric; no private context. |
| Message Count | **Keep** | Aggregate volume is safe for benchmarking. |
| Specific Error Codes | **Redact** | Could reveal internal infrastructure vulnerabilities. |
| Completion Status | **Keep** | Binary (Done/Not Done) is safe for productivity tracking. |
| Location Data | **Redact** | Replace with [REGION] or [OFFICE]. |
| Budget Figures | **Drop** | Strictly prohibited from the reporting plane. |

# System Architecture & Infrastructure

To support the WorkDash workflow while maintaining the "Zero-Trust" privacy model, the home lab infrastructure must be segmented into three distinct functional zones: the **Ingestion Zone**, the **Processing Zone**, and the **Reporting Zone**.

### 1. The Ingestion Zone (Data Collection)
This zone is responsible for pulling data from external sources (Outlook, Microsoft Teams, Slack, etc.) into the local environment.
*   **Hardware:** A low-power Raspberry Pi or a dedicated VM on a home server.
*   **Software:** A Python-based "Collector" service running in a Docker container.
*   **Security:** This service uses OAuth2 tokens stored in a local encrypted vault (e.g., HashiCorp Vault). It only has "Read" permissions.
*   **Function:** It polls the APIs every 5 minutes, fetches new messages, and pushes them immediately into the local PostgreSQL database. It does not perform any analysis; it is a "dumb" pipe to ensure no logic (and thus no risk of leakage) occurs during the initial fetch.

### 2. The Processing Zone (The "Brain")
This is where the heavy lifting happens. This zone is isolated from the internet.
*   **Hardware:** A high-performance workstation or a server with a dedicated GPU (e.g., NVIDIA RTX 3090/4090) to run local LLMs.
*   **Software:** 
    *   **Ollama or LocalAI:** To serve the Llama 3 or Mistral models.
    *   **PostgreSQL:** The primary database for raw and classified data.
    *   **Redis:** For task queuing (e.g., managing the sequence of messages to be classified).
    *   **Celery/RabbitMQ:** To handle asynchronous processing of large batches of emails.
*   **Function:** This zone pulls raw data from the database, runs the NER (Named Entity Recognition) and Intent Classification, extracts action items, and then triggers the Redaction Engine.

### 3. The Reporting Zone (The "Public" Face)
This zone handles the visualization and the generation of the synthetic reports.
*   **Software:** 
    *   **Grafana or Streamlit:** For the owner's private dashboard.
    *   **Static Site Generator (e.g., Hugo or Jekyll):** To host the public-facing benchmark reports.
*   **Function:** It only has access to the "Redacted" and "Synthetic" tables in the database. It is physically and logically incapable of querying the "Raw" data tables.

# Database Schema & Data Modeling

A robust relational schema is required to ensure that data flows correctly through the pipeline without mixing private and public attributes.

### Table: `raw_communications` (Private)
*Only accessible by the Processing Zone.*
*   `id`: UUID (Primary Key)
*   `source_platform`: String (e.g., "Teams", "Outlook")
*   `raw_body`: Text (The actual message)
*   `sender_id`: String (Internal ID)
*   `recipient_id`: String (Internal ID)
*   `received_at`: Timestamp
*   `processed_status`: Boolean (True/False)

### Table: `classified_metadata` (Private)
*Populated by the Local LLM.*
*   `id`: UUID
*   `raw_id`: UUID (Foreign Key to `raw_communications`)
*   `intent_category`: Enum (RFI, Status_Update, Urgent, Social, Task)
*   `priority_score`: Integer (1-10)
*   `extracted_action_items`: JSONB (List of tasks, deadlines, and owners)
*   `sentiment_score`: Float (-1.0 to 1.0)
*   `project_tag`: String (Internal project name)

### Table: `redacted_records` (Semi-Private)
*The output of the Redaction Engine.*
*   `id`: UUID
*   `raw_id`: UUID (Foreign Key)
*   `redacted_body`: Text (e.g., "[PERSON] requested [PROJECT] update")
*   `generalized_category`: String (e.g., "Infrastructure Request")
*   `is_synthetic_ready`: Boolean

### Table: `public_benchmarks` (Public)
*Only contains aggregated or synthetic data.*
*   `id`: UUID
*   `report_date`: Date
*   `total_volume`: Integer
*   `avg_response_time`: Float
*   `completion_rate`: Float
*   `synthetic_example_body`: Text (The fake message generated for the benchmark)
*   `metric_snapshot`: JSONB (Aggregated counts of categories)

# LLM Prompt Engineering for Classification

To ensure high-fidelity classification without "hallucinating" data, the local LLM requires highly structured system prompts.

### Prompt 1: Intent & Priority Classification
> **System Prompt:** You are a professional project management assistant. Your task is to analyze the following communication and output a JSON object. Do not include any conversational filler.
> 
> **Input Text:** {{raw_body}}
> 
> **Required JSON Schema:**
> {
>   "intent": "RFI" | "Status_Update" | "Urgent_Escalation" | "Social" | "Task_Assignment",
>   "priority": 1-10,
>   "summary": "A one-sentence summary of the request",
>   "action_items": [
>     {"task": "description", "deadline": "YYYY-MM-DD or null", "owner": "name"}
>   ],
>   "sentiment": -1.0 to 1.0
> }

### Prompt 2: Contextual Redaction & Generalization
> **System Prompt:** You are a data privacy officer. Your task is to rewrite the following message to remove all PII (Names, Specific Projects, Locations) while preserving the *functional intent* of the message. Replace specific entities with generic placeholders like [PERSON], [PROJECT], [LOCATION], or [DATE].
> 
> **Input Text:** {{raw_body}}
> 
> **Output Format:** Only provide the redacted text.

### Prompt 3: Synthetic Example Generation
> **System Prompt:** You are a technical writer. Based on the following metadata, generate a completely fictional, professional message that represents this type of interaction. Do not use any real names or real project details.
> 
> **Metadata:**
> - Category: {{intent_category}}
> - Priority: {{priority_score}}
> - Summary: {{summary}}
> 
> **Output Format:** A single paragraph of text.

# The Redaction Pipeline Logic

The transition from the Processing Zone to the Reporting Zone follows a strict state machine to prevent "leakage by omission."

1.  **Trigger:** A message is marked as `processed_status = True` in the `raw_communications` table.
2.  **Extraction:** The system pulls the `raw_body` and the `classified_metadata`.
3.  **Scrubbing:** A Python script runs a regex pass to strip out phone numbers, emails, and IP addresses.
4.  **LLM Redaction:** The scrubbed text is sent to the local LLM (Prompt 2). The output is saved to `redacted_records`.
5.  **Validation:** A secondary script checks the `redacted_body` against a "Blacklist" of known sensitive keywords (e.g., the owner's actual company name, specific client names). If a keyword is found, the record is flagged for **Human Review**.
6.  **Synthetic Mapping:** If the owner wants to publish a benchmark, the system takes the `redacted_records` entry and uses Prompt 3 to generate a "Synthetic Example."
7.  **Finalization:** The `public_benchmarks` table is updated with the synthetic example and the aggregated metrics from the `classified_metadata` table.

# Automation & Orchestration

To ensure the system runs autonomously in the home lab, the following orchestration is used:

### Docker Compose Setup
The entire stack is containerized to ensure environment parity and easy backups.
*   **Service: `workdash-db`** (PostgreSQL 15)
*   **Service: `workdash-llm`** (Ollama running Llama 3)
*   **Service: `workdash-collector`** (Python script with Cron-like interval)
*   **Service: `workdash-processor`** (Celery worker for LLM tasks)
*   **Service: `workdash-dashboard`** (Streamlit app)

### Cron Job Schedule
*   **00:00:** Run "Daily Aggregation" script. This script calculates the daily totals (Volume, Velocity, Sentiment) and moves them from the `classified_metadata` table to the `public_benchmarks` table.
*   **00:05:** Run "Privacy Audit" script. This script scans the `redacted_records` table for any strings that match the "Blacklist" and alerts the owner via a local notification.
*   **00:10:** Generate the "Public Report" static file.

# Privacy Audit & Compliance

Because this is a private home lab, "compliance" is self-imposed but must be rigorous.

### The "Leakage" Test
Once a month, the owner performs a "Leakage Test." They select 10 random entries from the `public_benchmarks` table and attempt to "reverse-engineer" the original message. If they can identify a specific project, person, or internal secret from the synthetic example or the aggregate metrics, the Redaction Engine's prompts are adjusted to be more aggressive.

### Differential Privacy (Conceptual)
To further protect the data, the system can implement a "K-Anonymity" rule for the public reports. For example, if a specific category (e.g., "Urgent Infrastructure Requests") has fewer than 3 occurrences in a week, the system will not report the specific count. Instead, it will report "Low Volume" to prevent someone from deducing that a specific event occurred.

# Visualization & Dashboarding

The owner interacts with two distinct dashboards.

### 1. The Private Command Center (Streamlit/Grafana)
*   **Real-time Feed:** A list of all incoming messages with their "Intent" and "Priority" badges.
*   **Action Item Tracker:** A Kanban-style view of all extracted tasks, showing "To-Do," "In Progress," and "Done."
*   **Sentiment Heatmap:** A calendar view showing the owner's stress levels (derived from sentiment scores) over the month.
*   **Bottleneck Analysis:** A chart showing which "Intent Categories" take the longest to respond to.

### 2. The Public Benchmark Portal (Static Site)
*   **Efficiency Scorecard:** A high-level summary of "Response Velocity" and "Task Completion."
*   **Workload Distribution:** A donut chart showing the split between "Deep Work" (low-volume, high-complexity) and "Reactive Work" (high-volume, low-complexity).
*   **Synthetic Gallery:** A carousel of synthetic messages that demonstrate the *types* of problems solved, paired with the actual time-to-resolution metrics.

# Future Roadmap & Scalability

1.  **Multi-Model Routing:** Implement a router that sends simple "Social" messages to a tiny model (Phi-3) and "Urgent/Technical" messages to a larger model (Llama 3 70B) to save on compute.
2.  **Voice Integration:** Add a "Voice-to-WorkDash" feature where the owner can dictate an action item into a local Whisper instance, which then gets classified and added to the database.
3.  **Cross-Platform Correlation:** Correlate Teams messages with Calendar events to see if "Urgent" pings are causing "Meeting Overrun" or "Focus Time" interruptions.
4.  **Automated Redaction Training:** Use the "Human Review" feedback to fine-tune a small, dedicated "Redaction Model" using LoRA (Low-Rank Adaptation) to improve accuracy over time.

# Technical Appendix: Sample SQL for Aggregation

To generate the public metrics, the following SQL query is run by the automation script:

```sql
INSERT INTO public_benchmarks (report_date, total_volume, avg_response_time, completion_rate, metric_snapshot)
SELECT 
    CURRENT_DATE,
    COUNT(id),
    AVG(EXTRACT(EPOCH FROM (response_time - received_at))),
    (COUNT(CASE WHEN status = 'Done' THEN 1 END)::float / COUNT(id)) * 100,
    json_build_object(
        'infrastructure', COUNT(CASE WHEN intent_category = 'Infrastructure' THEN 1 END),
        'project_mgmt', COUNT(CASE WHEN intent_category = 'Project_Management' THEN 1 END),
        'admin', COUNT(CASE WHEN intent_category = 'Admin' THEN 1 END)
    )
FROM classified_metadata
WHERE received_at >= CURRENT_DATE - INTERVAL '1 day';
```

This query ensures that the public report is a mathematical derivative of the private data, with no raw text ever being included in the `INSERT` statement. By strictly controlling the SQL queries that can access the `public_benchmarks` table, the owner ensures a physical "air gap" in the data logic.