A desktop assistant you talk to. It looks at your screen, answers out loud, and draws on top of the application you are actually using — a circle on the button you asked about, an arrow to the next step, a number on each thing to click.
The point is the last part. Asking an AI "what does this knob do?" normally means alt-tabbing away, typing a description of what you are looking at, reading an answer, then hunting for the control yourself. deskagent skips all of it: you ask without leaving the app, and the answer is pointed at.
It works with your provider. Any OpenAI-compatible endpoint will do — OpenAI, a self-hosted gateway, vLLM, llama.cpp, LM Studio, Ollama — or Anthropic's Messages API. Nothing is hardcoded to a vendor, and no telemetry goes anywhere.
Early. Being built in phases, each of which produces something runnable.
| Phase | What it adds | State |
|---|---|---|
| P0 | Config, layered profiles, coordinate maths, logging, CI | done |
| P1 | Screen capture, per-monitor click-through overlay, floating orb | done |
| P2 | Provider client, tool-use loop, text questions with annotations | done |
| P3 | Microphone, VAD, speech-to-text, spoken answers | done |
| P4 | Wake phrase ("hey deskagent", editable) | done |
| P5 | OCR / accessibility-tree snapping, privacy zones, consent screen | next |
| P6 | Opt-in control mode, packaging, Anthropic backend | planned |
So a full spoken turn works today:
deskagent doctor # what is installed, and what each extra unlocks
deskagent demo-annotate --shape all # check the shapes land on the right pixel
deskagent ask "what am I looking at?" # one text question, answered with annotations
deskagent listen # say "hey deskagent", or hold the orb and talkTwo things in this README are design, not yet code, and are marked where they appear: snapping annotations onto detected controls (P5) and control mode (P6). Until P5 lands, a circle is drawn where the model said, which is exactly the weakness snapping exists to fix — so treat pointing as approximate for now.
Requires Python 3.10 or newer.
git clone https://github.com/ribdsp/deskagent
cd deskagent
python -m venv .venv && . .venv/Scripts/activate # macOS/Linux: . .venv/bin/activate
pip install -e ".[dev]"Optional extras, installed only if you want the feature:
| Extra | Gives you |
|---|---|
voice |
microphone, playback, free edge-tts voices |
wakeword |
wake phrase and offline speech-to-text |
ocr |
snapping annotations onto on-screen text (P5, not wired up yet) |
uia |
element rectangles from the OS accessibility tree (P5, not wired up yet) |
control |
global hotkeys, and control mode if you enable it (P6) |
pip install -e ".[voice,wakeword]"deskagent doctor lists what is missing and what each extra unlocks.
Three layers, each overriding the one before: config/default.yaml, then a
profile from profiles/, then the environment.
cp .env.example .env # put your key here, never in YAML
cp profiles/example.yaml profiles/mine.yaml
$EDITOR profiles/mine.yaml
deskagent --profile mine doctorA profile only names what differs from the defaults:
provider:
kind: openai_compat
base_url: https://api.example.com
api_key_env: DESKAGENT_API_KEY # the variable name, not the value
models:
vision: gpt-4o # turns that include a screenshot
chat: gpt-4o-mini # turns that do not
stt: whisper-1
tts: tts-1Everything under profiles/ except example.yaml is gitignored, so your own
endpoint and model names stay on your machine. A test in CI fails the build if a
private profile or a .env ever becomes committable.
Fully local, nothing leaving the machine:
provider: { base_url: http://127.0.0.1:11434 }
models: { vision: qwen2.5vl, chat: qwen2.5 }
speech:
stt: { backend: local, local_model: base }
tts: { backend: os }
privacy: { local_only: true }local_only: true is enforced, not advisory: startup fails if any part of the
configuration would still send data off the machine.
Two things make on-screen pointing hard, and both are handled deliberately.
Models are bad at coordinates. Asked for the centre of an image, a
frontier model will happily answer (0, 0) — measured, not assumed. The fix
(P5, not yet built) is that a model estimate is never drawn directly: it is
snapped onto the nearest real control detected by OCR or the OS accessibility
tree, within a configurable radius, and targets can be named by visible text or
element id to skip estimation entirely. When nothing can be resolved the
assistant will explain in words rather than draw a circle somewhere plausible — a
wrong circle is worse than none, because you trust the circle. Today, without
snapping, the estimate is what you get.
Display scaling. Physical pixels, Qt's logical points, image pixels and the
model's 0..1 coordinates are four different spaces, and per-monitor DPI means
the conversion is per screen, not global. Every conversion lives in one module,
deskagent/vision/geometry.py, covered by tests
that use real 125%-scaling numbers and monitors at negative coordinates. This
part is done and is why a shape lands where the geometry says it should.
A typical spoken question is one screenshot plus a short answer. Downscaled to
1280 px, a full-HD screen is roughly 1.2k image tokens, and screenshots per turn
are capped in code (agent.max_captures_per_turn, default 2) rather than merely
requested in the prompt. Follow-ups that need no picture use the cheaper chat
model. Expect single-digit cents per question on a mid-tier hosted model, and
nothing at all on a local one.
Working today:
- Screenshots are held in memory and not written to disk unless you set
vision.save_captures. - Logs redact API keys and token-shaped strings, on every handler.
privacy.local_onlyrefuses any remote endpoint outright, at startup.- The overlay is kept out of every screenshot deskagent takes, so your annotations are never uploaded back to the model.
Configured but not yet enforced (P5):
privacy.process_blocklist— windows belonging to those processes (password managers) will never be captured. The setting is read and validated today; the capture path does not consult it yet.- Rectangles you mark as private will be blurred before anything is uploaded.
deskagent is designed to be able to click and type for you, off by default — and as of today it is not implemented at all (P6). The design it will be built to: turning it on requires typed consent in Settings that spells out what it means; even then, every action is proposed, drawn on screen, and confirmed by you before it runs. Some limits will not be configurable away: never typing into a password field, a process blocklist, refusal of destructive key combinations, one action per confirmation, a rate limit, a panic hotkey, and an append-only action log.
If you never want any of this, you will be able to leave it off. Everything else works without it.
Honest matrix. The weak spots on Wayland are OS policy, not a toolkit choice — they apply equally to any framework.
| Capability | Windows | macOS | Linux X11 | Wayland |
|---|---|---|---|---|
| click-through always-on-top overlay | yes | yes | yes | degrades to a normal window |
| screen capture | yes | needs Screen Recording permission | yes | needs the pipewire portal |
| overlay hidden from capture | native | hide-then-capture | hide-then-capture | limited |
| active window title and process | yes | yes (pyobjc) | yes (xlib) | no API |
| precise UI element tree | UIA | AX, needs permission | AT-SPI | limited |
| global hotkeys | yes | needs permission | yes | via the compositor |
| control mode | yes | needs permission | yes | no |
The last two rows describe what each platform allows; deskagent's own support for them is P5 and P6 above. Windows is first-class and the only platform tested on real hardware so far. macOS and Linux/X11 are supported by design and covered by CI for imports and unit tests; treat them as beta and please report what breaks.
pytest -q # unit tests, no network, no display needed
ruff check .
mypy deskagent
pytest -m integration # opt in; needs a real provider keyTests focus on the logic that can fail quietly: coordinate transforms,
configuration precedence, the capture budget, audio frame arithmetic, and wake
phrase scoring. GUI layers are smoke-tested rather than chased for coverage, and
the 80% bar is enforced on core/, vision/, llm/, speech/ and control/
instead of being claimed globally — it currently sits around 96%.
See CONTRIBUTING.md before opening a pull request, and SECURITY.md to report a vulnerability.
MIT. Qt is used through PySide6 under LGPLv3.