WorkScreen (Gilbreth)

A desktop app that records what you do — clicks, keystrokes, focus changes, clipboard — through OS accessibility APIs, into a local SQLite database that you (and your LLM tools) can query later.

Status

Early. macOS and Windows both have working capture backends (macOS via CGEventTap + the Accessibility API; Windows via UI Automation + event hooks). Signed installers (macOS .dmg, Windows NSIS) with auto-update are published via GitHub Releases. The storage schema is not yet stable — migrations are additive, but queries written against it may need updating.

What exists is capture, meeting recording, on-device transcription and real-time suggestions. Pattern mining over the captured stream — the therbligs the name refers to — is the next layer and is not implemented.

What it captures

One row per atomic action in actions:

• click — mouse button down, enriched with the AX element (role, name, value) under the cursor when available
• text — typed text after a 300 ms debounce (one row = one "burst")
• key — non-printable navigation/editing keys (Enter, Tab, arrows, Backspace, …)
• scroll — wheel events
• clipboard — clipboard text
• focus_change — frontmost app changed

Every row carries the foreground app context (app_name, app_bundle_id, window_title) and a session ID. Rows captured while a password field had focus are masked at the SQL layer (text_content and element_value become '[masked]', password_flag = true).

Meetings are captured in parallel: when a video-conference app starts a call, WorkScreen records the meeting (start/end, app, audio/video paths) into a meetings table and links subsequent actions to it via actions.meeting_id. After the call ends, the audio is transcribed fully on-device with whisper.cpp into meeting_transcripts.

Real-time suggestions (optional)

While a call is running, WorkScreen can transcribe it live and show short suggestions in a small always-on-top panel — powered by an agent on your own machine, talked to over ACP. Nothing leaves the machine except what that agent itself sends.

It works with the coding CLI you already have — Claude Code or Gemini CLI. Nothing else to install: the interactive CLIs do not speak ACP themselves, so WorkScreen runs the adapter for you, fetching it with npx the first time if it is not already there. Point GILB_ASSIST_AGENT at something else if you would rather use your own.

It is off by default at three levels: a Cargo feature, a switch in the app, and a ~570 MB speech model downloaded only if you turn it on. The prompt is a plain file you can edit — ~/Documents/Gilb/prompts/realtime_assist.md — not a string baked into the binary. See docs/assist.md for how the pipeline fits together.

Requirements

• macOS (Apple Silicon or Intel) or Windows. Linux is out of scope.
• Rust toolchain (stable, edition 2021).
• Node + npm (only for the Tauri shell).
• On macOS: Accessibility and Input Monitoring permissions, granted in System Settings → Privacy & Security. The app exposes an open_privacy_pane command that jumps you to the relevant pane.

Build and run

# Tauri shell with the recorder UI.
cd apps/gilb-app-tauri
npm install
npm run tauri dev

# Read-only MCP server over ~/Documents/Gilb/db.sqlite (stdio transport).
# Point an MCP client (e.g. Claude Code) at this binary to query
# recorded activity. See apps/gilb-mcp/help.md for the tool catalog.
cargo run -p gilb-mcp

Build options live in RecordingSettings::from_env:

Env var	Default	Effect
CAPTURE_EVENTS	true	Toggle the entire capture pipeline
CAPTURE_MOUSE_MOVE	false	Record raw mouse-move events (noisy)
CAPTURE_CLIPBOARD	true	Record clipboard text
CAPTURE_TREE_SNAPSHOTS	true	Periodic full AX tree dumps
RUST_LOG	varies	Standard tracing filter

Querying recorded activity from Claude Code

If you installed WorkScreen from a release (the macOS .dmg), the read-only MCP server ships inside the app bundle — no build step needed. The binary lives at:

/Applications/WorkScreen.app/Contents/MacOS/gilb-mcp

Register it with Claude Code:

claude mcp add gilb --scope user /Applications/WorkScreen.app/Contents/MacOS/gilb-mcp

Use --scope user so the server is available in every project, since WorkScreen records activity regardless of which repo you're working in. Drop the flag to register it for the current project only. Confirm it registered and connected:

claude mcp list

Inside a Claude Code session the gilb_* tools are now available (see apps/gilb-mcp/help.md for the full catalog). The server reads ~/Documents/Gilb/db.sqlite over stdio; WorkScreen.app itself does not need to be running. If you built WorkScreen from source, point the same command at the built binary instead (cargo run -p gilb-mcp, or target/release/gilb-mcp). See INSTALL.md for the end-user install and permissions guide.

Architecture

Cargo workspace with three runnable apps and fourteen library crates under crates/:

• apps/gilb-app-tauri — the desktop app (tray + one window).
• apps/gilb-mcp — read-only MCP server over the recorded database.
• apps/gilb-analyzer — runs prompt-jobs against your own recorded activity and pushes findings to a server. Requires credentials most users will not have; nothing else depends on it.

The capture pipeline is platform-gated behind a CapturePlatform trait; macOS uses CGEventTap + the Accessibility API, Windows uses UI Automation

• event hooks. A no-op backend keeps the workspace compiling elsewhere, which is what CI builds on Linux.

See CLAUDE.md for the full crate graph, capture → DB data flow, and macOS-specific notes (entitlements, signing, permission prompts).

Privacy

Capture, storage and transcription are entirely local: the database is a file in ~/Documents/Gilb/, whisper.cpp runs on-device, and nothing is uploaded. The capture pipeline drops events from a fixed block-list of password managers — 1Password, Bitwarden, KeePassXC, and macOS Keychain Access — at the source, so those apps never produce rows. For everything else, rows captured while a password field had focus are masked in-place as described above.

Two optional parts do leave the machine, and only if you enable them:

• Real-time suggestions send the transcribed conversation to the agent you configured. Where that goes is that agent's business — a cloud model if it is Claude or Gemini, nowhere if it is a local one.
• gilb-analyzer posts its findings to a server, and needs credentials you must supply. Without them it does nothing.

Meeting recording captures screen and audio to disk while a call is running. That is the point of the feature, but it is worth saying plainly: those files are as sensitive as the calls themselves.

License

MIT — see LICENSE.