Skip to content

feat(desktop): expose native Driver calls and screenshot URLs - #26

Open
jmgilman wants to merge 7 commits into
masterfrom
feat/desktop-server
Open

jmgilman wants to merge 7 commits into
masterfrom
feat/desktop-server

Conversation

@jmgilman

@jmgilman jmgilman commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Change

  • Implement desktop.info, desktop.enable, desktop.call, and desktop.screenshot over the persistent guest Cua Driver daemon. Preserve native tool names, schemas, tokens, and results; use one-shot CLI requests through Incus exec.
  • Execute as the automation user in its X11 session, include desktop readiness in create/wait, and report the actual VNC endpoint.
  • Fetch PNGs through the binary Incus file API. Return short-lived HTTP URLs only, with bounded image/store bytes, expiry and sandbox-reaper cleanup, safe PNG dimensions, and optional resizing.
  • Support the HTTP MCP handler and a separate screenshot listener for stdio.
  • Authenticate private native Incus image pulls with image-access secrets. Resolve VNC forwards through DHCP leases when rebooted guests temporarily lack observed NIC addresses.
  • Qualify catalog desktop VMs by their declared capability: X11, the active automation-user Driver service, and native list_apps. Preserve router/runner checks and apply the desktop catalog CPU/memory defaults.

Verification

  • mise exec -- moon run root:check passed: 13 tasks, including build, tests, lint, formatting, docs, and offline CLI/MCP smoke; two tasks used Moon cache.
  • Reproduced the old catalog bug against the published image: initialization failed at systemctl is-active incus-gh-runner-guest.path with exit 4 (68.06 s).
  • The corrected TestDesktopAcceptance passed with the complete catalog from feat(images): promote qualified lab image digests #27 and no GHCR credentials: 182.26 s overall, 26.822 s for the representative program. It exercised real stdio MCP, private OVN networking/router NAT, Incus exec, and HTTP PNG fetching.
  • One native foreground token click changed one document into two document tabs, verified independently by a fresh accessibility tree and fetched before/after PNGs. The Driver reports effect: unverifiable; the server does not invent success or retry ineffective background delivery.
  • Reboot restored Driver readiness and the advertised VNC endpoint (10.10.40.67:5900, RFB 003.008). The running reaper made the screenshot URL return 404 21.773 s after sandbox expiry.
  • Disposable probe programs were removed. Contract regression tests and pinned native fixtures remain. Detailed build measurements, native timings, earlier local runs, and published acceptance are recorded in spikes/desktop/README.md.

Published image and related work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant