# Goals

The primary objective of the WorkDash analysis workflow is to provide a high-fidelity productivity and workflow analysis system that operates entirely within a private home-lab environment. The system aims to bridge the gap between "Raw Data Utility" (the owner needs to see exactly what they did) and "Public Accountability" (the owner wants to share benchmarks or progress without compromising corporate secrets or personal privacy).

Key objectives include:
1.  **Privacy Preservation:** Ensuring that no PII (Personally Identifiable Information), proprietary corporate data, or private communications ever leave the local network.
2.  **Granular Analysis:** Utilizing local Large Language Models (LLMs) to perform nuanced classification of tasks, sentiment, and "Deep Work" vs. "Shallow Work" metrics.
3.  **Automated Redaction:** Creating a pipeline that automatically transforms private logs into anonymized, aggregate, or synthetic data for public reporting.
4.  **Actionable Insights:** Providing the owner with a dashboard that identifies bottlenecks, context-switching costs, and time-sink patterns.
5.  **Benchmark Integrity:** Allowing the owner to share "WorkDash" performance metrics (e.g., "My productivity increased by 15% using this workflow") without revealing the underlying data that produced those numbers.

# Data That Must Stay Local

To maintain a "Zero-Leak" posture, the following data categories are strictly confined to the local home-lab infrastructure (e.g., a local PostgreSQL instance and a local vector database):

*   **Raw Communication Logs:** Full bodies of emails, Teams chat transcripts, and Slack messages.
*   **Identity Data:** Real names of colleagues, clients, managers, and external vendors.
*   **Project Codenames:** Internal project names (e.g., "Project Phoenix," "Q4 Infrastructure Overhaul") that could be sensitive to competitors.
*   **Credentials & Tokens:** API keys, OAuth tokens, and internal server addresses used to fetch the data.
*   **Specific Technical Details:** Snippets of proprietary code, internal IP addresses, specific database schemas, or unique error logs.
*   **Sensitive Context:** Mentions of salaries, HR issues, health information, or private personal matters discussed in professional channels.

# Local Classification

The workflow utilizes a local LLM (e.g., Llama 3 8B or Mistral 7B running via Ollama) to process the raw data. This ensures that the "intelligence" layer of the analysis is also private.

**Classification Pipeline:**
1.  **Ingestion:** Data is pulled via local scripts into a staging table.
2.  **Entity Extraction:** The local model identifies and tags entities (e.g., "Person," "Project," "Action Item"). These tags are stored locally.
3.  **Task Categorization:** The model classifies messages into categories:
    *   *Deep Work:* Coding, writing, architectural planning.
    *   *Shallow Work:* Email triaging, status updates, scheduling.
    *   *Meetings:* Synchronous communication.
    *   *Blockers:* Identifying messages that indicate a delay in progress.
4.  **Sentiment & Urgency:** Scoring the "stress level" or "urgency" of incoming communications to map out peak periods of high-pressure interaction.
5.  **Vector Embedding:** Generating embeddings for all messages to allow for local RAG (Retrieval-Augmented Generation) queries, such as "When did I last discuss the database migration?"

# Redaction Strategy

The redaction engine acts as a "Gateway" between the private database and the public reporting plane. It employs a multi-stage approach:

1.  **Hard Redaction (Deletion):** Complete removal of PII (Phone numbers, emails, specific names).
2.  **Soft Redaction (Generalization):** Replacing specific entities with generic descriptors.
    *   *Example:* "Meeting with John Doe about Project X" $\rightarrow$ "Meeting with [Colleague] about [Engineering Project]."
3.  **Aggregation:** Instead of reporting on individual messages, the system sums the data.
    *   *Example:* Instead of "Spent 2 hours on a bug fix for the login page," the report shows "Spent 2 hours on [Bug Fixes]."
4.  **Contextual Scrubbing:** A final pass by a local LLM to ensure that the "Summary" of a conversation doesn't accidentally include a specific detail that identifies the project (e.g., "Discussed the 50% discount for Client Y" $\rightarrow$ "Discussed pricing strategy").

# Synthetic Test Generation

To provide a public-facing benchmark that is actually useful, the system generates "Synthetic Twin" data. This allows the owner to demonstrate the *accuracy* of the WorkDash analysis without using real data.

**The Process:**
1.  **Statistical Profiling:** The system analyzes the *distribution* of the real data (e.g., "The user spends 30% of their time in meetings, 50% on coding, and 20% on admin").
2.  **Synthetic Seed Generation:** A local LLM is prompted to generate 100 "Fake" emails and messages that follow the same distribution and complexity as the real data but are entirely fictional.
3.  **Benchmark Execution:** The WorkDash analysis is run on this synthetic dataset.
4.  **Public Reporting:** The owner publishes the results of the analysis on the synthetic data. This proves that the WorkDash logic works, the classification is accurate, and the metrics are meaningful, all while the real data remains untouched.

# Metrics To Keep

The following metrics are deemed "Safe" for public reporting because they represent trends and behaviors rather than specific content:

*   **Time Distribution %:** Percentage of day spent in different categories (Deep Work vs. Shallow Work).
*   **Context Switching Frequency:** Number of times the user switched between different project categories in an hour.
*   **Response Latency:** Average time taken to respond to different types of messages (e.g., "Urgent" vs. "General").
*   **Focus Score:** A proprietary metric calculated by the local model based on the duration of uninterrupted "Deep Work" blocks.
*   **Trend Lines:** Weekly/Monthly growth in "Deep Work" hours or reduction in "Shallow Work" overhead.
*   **Blocker Density:** The frequency of "Blocker" tags appearing in the workflow over time.

# Human Review

To ensure the highest level of safety, a "Human-in-the-Loop" (HITL) step is required before any data is moved to the public reporting plane.

1.  **The "Export Preview":** The system generates a draft of the public report.
2.  **The "Redaction Audit":** The owner reviews the draft to ensure no "leaky" summaries exist.
3.  **Manual Override:** The owner can manually flag a summary as "Too Specific" and force the system to re-generate it with higher abstraction.
4.  **Final Approval:** A physical "Publish" button in the local dashboard triggers the export of the redacted/synthetic report to the public-facing site.

# Failure Modes

1.  **LLM Hallucination in Redaction:** The local model might summarize a private conversation and accidentally include a specific name or secret in the summary.
    *   *Mitigation:* Use a deterministic regex-based PII scrubber *after* the LLM summary is generated.
2.  **Over-Redaction:** The redaction is so aggressive that the public report becomes useless (e.g., "User did work for 8 hours").
    *   *Mitigation:* Use "Soft Redaction" (Generalization) instead of "Hard Redaction" for project types.
3.  **Inference Latency:** Processing thousands of emails through a local LLM can be slow.
    *   *Mitigation:* Use a queue system (e.g., Celery/Redis) to process messages in the background.
4.  **Data Drift:** The "Synthetic Twin" data might diverge from real-world behavior over time.
    *   *Mitigation:* Periodically re-generate the synthetic seed based on the latest 30 days of real-world statistical distributions.

# Example Private Summary

**Internal Dashboard View (Owner Only):**
> **Date:** 2023-10-27
> **Summary:** Spent 4.5 hours on "Project Phoenix" (Database Migration). Had a high-friction meeting with Sarah Jenkins regarding the API latency issues in the production environment. Identified a blocker: The DevOps team hasn't provided the staging credentials for the new cluster.
> **Deep Work Score:** 82%
> **Context Switches:** 14
> **Key Action Items:** 1. Follow up with Sarah on the latency logs. 2. Ping Dave for the staging keys.

# Example Publishable Summary

**Public Benchmark View (Publicly Accessible):**
> **Date:** 2023-10-27
> **Summary:** Focused heavily on a major infrastructure migration project. Participated in a technical sync regarding system performance and identified a dependency blocker related to environment access.
> **Deep Work Score:** 82%
> **Context Switches:** 14
> **Key Action Items:** 2
> **Note:** *This report is based on a synthetic dataset generated to mirror the statistical distribution of the user's actual workflow.*

---

### Data Handling Decision Matrix

| Field / Data Point | Action | Reason |
| :--- | :--- | :--- |
| Raw Email Body | **Drop** | Contains PII and proprietary secrets. |
| Sender Name | **Redact** | Replace with "[Colleague]" or "[Client]". |
| Project Name | **Redact** | Replace with "[Project Category]" (e.g., "Cloud Infra"). |
| Time Spent (Minutes) | **Keep** | Purely numerical; no privacy risk. |
| Deep Work Score | **Keep** | Derived metric; no privacy risk. |
| Context Switch Count | **Keep** | Behavioral metric; no privacy risk. |
| Specific Bug IDs | **Drop** | Could reveal internal system vulnerabilities. |
| Meeting Sentiment | **Keep** | Aggregate sentiment is safe; individual sentiment is private. |
| Action Item Text | **Redact** | Generalize to "Follow up on [Task Type]". |
| Response Latency | **Keep** | Performance metric; no privacy risk. |
| Internal IP Addresses | **Drop** | Security risk. |
| Synthetic Twin Data | **Keep** | Purpose-built for public consumption. |

## Technical Infrastructure Stack

To ensure the "Zero-Leak" posture, the infrastructure must be physically and logically isolated from the public internet, except for the final reporting plane which acts as a one-way data diode.

**Hardware Requirements:**
*   **Compute:** A dedicated workstation or server with at least one NVIDIA RTX 3090/4090 (24GB VRAM) or an enterprise-grade A6000. This is necessary to run 7B to 13B parameter models (like Llama 3 or Mistral) with high-speed inference and enough context window to process long email threads.
*   **Storage:** NVMe SSD for the vector database and PostgreSQL instance to ensure low-latency retrieval during RAG operations.
*   **Networking:** A dedicated VLAN for the "WorkDash" server, isolated from the main home network.

**Software Stack:**
*   **Orchestration:** Docker and Docker Compose for containerizing the entire pipeline.
*   **LLM Engine:** Ollama or LocalAI for serving the models via a local API.
*   **Database:** 
    *   *PostgreSQL:* For structured data (timestamps, categories, scores, redacted summaries).
    *   *Qdrant or ChromaDB:* For vector embeddings of the raw (local-only) messages.
*   **Data Ingestion:** Python-based scripts using `imaplib` for emails and `Microsoft Graph API` (via a local proxy) or `Teams Webhooks` for message ingestion.
*   **Reporting Plane:** A Streamlit or Grafana dashboard hosted on a separate, public-facing VPS that only receives the final, redacted JSON payloads.

## Data Pipeline Architecture

The data flows through a multi-stage pipeline where the "Privacy Perimeter" is strictly enforced at each transition.

1.  **Ingestion Layer:** Scripts pull raw data into a `raw_staging` table in PostgreSQL. This table is encrypted at rest and has no external access.
2.  **Processing Layer (Local LLM):**
    *   A worker process pulls a batch of raw messages.
    *   The LLM performs **Entity Extraction** (identifying names, dates, and projects).
    *   The LLM performs **Task Classification** (assigning a "Work Type" and "Urgency Score").
    *   The LLM generates a **Private Summary** (a 2-3 sentence description of the interaction).
3.  **Vectorization Layer:** The raw text is converted into embeddings and stored in the local vector database. This allows the owner to ask, "What did I do last Tuesday?" without the data ever leaving the machine.
4.  **Redaction & Aggregation Layer:**
    *   The system identifies the "Public Reporting" requirements.
    *   It passes the *Private Summary* through a **Redaction Prompt**.
    *   It aggregates individual tasks into daily/weekly buckets (e.g., "Total Deep Work: 4 hours").
5.  **Synthetic Generation Layer:**
    *   The system calculates the statistical distribution of the week's work (e.g., 40% Coding, 30% Meetings, 30% Admin).
    *   It generates a "Synthetic Twin" dataset—a set of fictional logs that mirror these percentages.
6.  **Export Layer:** The redacted summaries and the synthetic twin data are bundled into a JSON object and pushed to the public reporting plane via a secure `POST` request.

## Database Schema (Simplified)

To maintain the distinction between private and public data, the schema is split into two primary logical zones.

**Zone A: Private (Local Only)**
```sql
CREATE TABLE raw_communications (
    id UUID PRIMARY KEY,
    source_type VARCHAR(50), -- 'email', 'teams', 'slack'
    raw_content TEXT,
    sender_identity TEXT,
    received_at TIMESTAMP,
    vector_id UUID -- Link to vector database
);

CREATE TABLE classified_events (
    id UUID PRIMARY KEY,
    communication_id UUID REFERENCES raw_communications(id),
    work_category VARCHAR(50), -- 'Deep Work', 'Shallow Work', 'Meeting'
    urgency_score INT,
    private_summary TEXT,
    context_switch_cost FLOAT,
    project_tag TEXT
);
```

**Zone B: Public (Redacted/Synthetic)**
```sql
CREATE TABLE public_benchmarks (
    id SERIAL PRIMARY KEY,
    report_date DATE,
    total_deep_work_hours FLOAT,
    context_switch_count INT,
    focus_score FLOAT,
    redacted_summary TEXT, -- "Focused on infrastructure migration"
    is_synthetic BOOLEAN DEFAULT TRUE,
    synthetic_twin_id UUID
);
```

## Prompt Engineering for Local LLMs

The quality of the analysis depends on the precision of the prompts. Using a "System Prompt" to define the persona is critical.

**Classification Prompt:**
> "You are a professional productivity analyst. Analyze the following communication and categorize it into one of these categories: [Deep Work, Shallow Work, Meeting, Blocked, Social]. 
> 
> Provide the output in JSON format:
> {
>   "category": "...",
>   "urgency": 1-10,
>   "summary": "A one-sentence summary of the core task.",
>   "entities": ["list of names/projects"]
> }
> 
> Communication: [INSERT RAW TEXT]"

**Redaction Prompt (The "Privacy Filter"):**
> "You are a data privacy officer. Take the following private summary and rewrite it for a public benchmark report. 
> 
> Rules:
> 1. Remove all specific names of people or companies.
> 2. Replace specific project names with generic categories (e.g., 'Project Phoenix' becomes 'Internal Infrastructure').
> 3. Remove any specific technical details (e.g., 'fixing a SQL injection' becomes 'addressing security vulnerabilities').
> 4. Maintain the same sentiment and general scope of work.
> 
> Private Summary: [INSERT PRIVATE SUMMARY]"

## Synthetic Twin Generation Algorithm

To ensure the public benchmark is statistically valid without being personally identifiable, the system uses a "Distributional Mirroring" algorithm.

1.  **Step 1: Statistical Profiling.** The system calculates the mean, standard deviation, and mode for:
    *   Time spent per category.
    *   Frequency of context switches per hour.
    *   Average response latency.
2.  **Step 2: Seed Generation.** The system generates a "Seed" of 50 random events.
3.  **Step 3: Stochastic Expansion.** Using a local LLM, the system expands the seed into a full day's worth of "Synthetic Logs."
    *   *Prompt:* "Generate a fictional work log for a software engineer. The log must contain 4 hours of 'Deep Work' on 'Cloud Infrastructure', 2 hours of 'Meetings', and 2 hours of 'Administrative Tasks'. Ensure the tone is professional and the tasks are varied."
4.  **Step 4: Validation.** The system compares the "Synthetic Twin" distribution against the "Real Data" distribution. If the variance is > 10%, it regenerates the twin.
5.  **Step 5: Publication.** The public report displays the metrics from the Synthetic Twin, ensuring that the "WorkDash" logic is proven to work on a realistic dataset that contains zero real data.

## Advanced Metrics & Formulas

To provide the owner with high-value insights, the system calculates several proprietary metrics:

**1. Context Switching Cost (CSC):**
Calculated every time the `work_category` changes within a 60-minute window.
*   *Formula:* $CSC = \sum (T_{switch} \times \text{Complexity\_Weight})$
*   *Complexity Weight:* A value (1-5) assigned by the LLM based on how different the two tasks are (e.g., switching from "Coding" to "Coding" is weight 1; switching from "Coding" to "HR Meeting" is weight 5).

**2. Focus Decay Rate (FDR):**
Measures how quickly "Deep Work" blocks are interrupted.
*   *Formula:* $FDR = \frac{\text{Total Interruptions}}{\text{Total Deep Work Minutes}}$
*   *Goal:* A lower FDR indicates a more successful "Flow State" environment.

**3. Response Latency Variance (RLV):**
Analyzes the gap between message receipt and response.
*   *Insight:* High variance in response latency for "High Urgency" messages suggests a breakdown in triage processes.

## Security & Network Isolation

To prevent any possibility of data leakage, the following security measures are implemented:

**1. The Data Diode Pattern:**
The server that processes the raw data (The "Processor") and the server that hosts the public dashboard (The "Publisher") are separate. The Processor can send a `POST` request to the Publisher, but the Publisher has no way to query the Processor's database.

**2. Local-Only Inference:**
By using Ollama or LocalAI, the system never sends a single byte of raw text to an external API (like OpenAI or Anthropic). All "thinking" happens on the local GPU.

**3. Automated Purge Policy:**
The `raw_communications` table is set to auto-delete entries older than 30 days. Only the aggregated, redacted metrics are kept in the long-term database.

## Human Review & Override Workflow

Before the "Publish" button becomes active, the owner must complete a "Sanity Check" in the local dashboard.

*   **The "Redaction Heatmap":** The dashboard highlights any summary that the LLM flagged as "High Risk" (e.g., if the LLM detected a name that it failed to redact).
*   **Manual Edit:** The owner can click on any redacted summary and manually type a replacement if the LLM's generalization was too vague.
*   **Synthetic Verification:** The owner can toggle a view to see the "Real Distribution" vs. the "Synthetic Distribution" to ensure the twin is an accurate representation of their productivity.

## Failure Modes & Mitigation

| Failure Mode | Impact | Mitigation Strategy |
| :--- | :--- | :--- |
| **LLM Hallucination** | A summary might include a secret. | Implement a secondary "Regex Scrubber" that looks for known patterns (emails, IPs, phone numbers) in the final summary. |
| **Context Window Overflow** | Long email threads might be cut off. | Implement a "Chunking" strategy where threads are split into 1000-token segments and summarized individually before being merged. |
| **Inference Bottleneck** | Processing 500 emails takes hours. | Use a priority queue. Process "Urgent" messages first and run the rest as a background cron job overnight. |
| **Synthetic Drift** | The twin data doesn't look like the real work. | Implement a "Distribution Check" script that fails the build if the synthetic data deviates too far from the real data's mean. |
| **API Key Exposure** | The script fetching Teams data is compromised. | Use a "Local Proxy" that handles the OAuth flow. The WorkDash script only talks to the local proxy, never the internet directly. |

## Example Private Summary (Detailed)

**Internal Dashboard View (Owner Only):**
> **Date:** 2023-10-27
> **Raw Source:** Teams Message from "Sarah Jenkins" (Engineering Lead)
> **Raw Content:** "Hey, we're seeing a 500 error on the /auth endpoint. It seems to be related to the new Redis cluster we spun up this morning. Can you take a look at the connection pool settings?"
> **LLM Classification:** 
> - Category: Deep Work (Bug Fix)
> - Urgency: 9/10
> - Project: Project Phoenix (Infrastructure)
> - Context Switch Cost: 4.2 (Switching from 'Documentation' to 'Emergency Fix')
> **Private Summary:** Sarah reported a 500 error on the /auth endpoint linked to the new Redis cluster. I need to investigate the connection pool settings immediately.
> **Action Item:** Check Redis config and update connection pool limits.

## Example Publishable Summary (Detailed)

**Public Benchmark View (Publicly Accessible):**
> **Date:** 2023-10-27
> **Summary:** Addressed a critical production error related to database connectivity. Investigated infrastructure configuration to ensure system stability.
> **Deep Work Score:** 82%
> **Context Switches:** 14
> **Key Action Items:** 2
> **Note:** *This report is based on a synthetic dataset generated to mirror the statistical distribution of the user's actual workflow. No private data is included.*

## Scaling for High-Volume Environments

For users who handle hundreds of messages a day, the "Batch Processing" model is essential.

**1. Vectorized Batching:**
Instead of processing every message individually, the system groups messages by "Thread ID." The LLM summarizes the entire thread once, rather than summarizing every individual reply. This reduces LLM tokens by up to 70%.

**2. Model Distillation:**
For simple tasks like "Category Classification," a smaller, faster model (e.g., Phi-3 or a quantized 3B model) can be used. The larger, more capable model (e.g., Llama 3 70B) is reserved for "Summary Generation" and "Redaction," where nuance is required.

**3. Distributed Workers:**
If the home lab has multiple machines, the "Ingestion" and "Vectorization" can happen on a low-power Raspberry Pi or NUC, while the "LLM Inference" is offloaded to the GPU workstation. This prevents the ingestion scripts from being interrupted by heavy GPU compute tasks.

## Final Data Integrity Check

To ensure the system remains "Private," a final "Data Diode" check is performed. Before the JSON payload is sent to the public server, a script runs a "Negative Lookahead" check. It scans the JSON for:
1.  Any string longer than 50 characters that contains a digit (potential IP or ID).
2.  Any string containing "@" (potential email).
3.  Any string containing "Project [Name]" (potential project leak).

If any of these are found, the export is blocked, and an alert is sent to the local dashboard for manual review. This provides a multi-layered defense-in-depth strategy, combining LLM intelligence, statistical synthesis, and deterministic security checks.