Skip to content

[Feat] Add bounded computer use to Linux sandboxes - #2972

Draft
roomote-roomote[bot] wants to merge 2 commits into
feature/shared-desktop-2p7nb2sim988efrom
feature/cua-computer-use-prototype
Draft

roomote-roomote[bot] wants to merge 2 commits into
feature/shared-desktop-2p7nb2sim988efrom
feature/cua-computer-use-prototype

Conversation

@roomote-roomote

Copy link
Copy Markdown
Contributor

​Opened on behalf of @mrubens. Follow up by mentioning @roomote-roomote, in the web UI, or in Slack.

What changed

  • Add opt-in computer_use environment configuration with an exact HTTP(S) browser-origin allowlist.
  • Pin Cua Driver 0.28.2 and its Linux archive checksums in the worker image, then generate a short-lived bounded manifest for Roomote's existing shared Chrome profile.
  • Mount Cua through a task-local MCP proxy that preserves Shared Desktop's human_driving handoff, keeps observations available, and fail-closes mutations when a person is driving or handoff state is unavailable.
  • Document the Linux X11 browser-only scope and explicitly exclude personal-computer control, broad native desktop input, Jev credentials, and the draft perception extension.

Why this change was made

Cua Driver now provides a released Linux X11 and Chromium DOM surface that can complement Roomote's shared sandbox desktop. This prototype evaluates that path without conflating the separate jev-use chooser recipe with an agent loop or adopting the draft AGPL perception artifacts.

Impact

Environment owners can enable bounded browser computer use for selected origins; existing environments remain unchanged. The prototype depends on the Shared Desktop work in #2791, so this is a stacked PR targeting that feature branch.

A production-like live check ran Cua Driver 0.28.2 through the bundled proxy as the unprivileged sandbox user under Xvfb. It attached to the baked Chrome-for-Testing profile, observed DOM refs, typed and submitted a unique fixture value, verified the result semantically, and matched the fixture's independent state endpoint. Native AT-SPI remains unavailable without a desktop session bus, so this slice claims Chromium DOM support only, not general Linux application control. Browser screenshots were not applicable because the shipped change is runtime configuration and mediation rather than a rendered Roomote UI.

@roomote-community

roomote-community Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

1 issue outstanding. See task

  • apps/worker/src/mcp/cua-driver-proxy/guard.ts:63 Cua input can be dispatched after a person takes control because the handoff state is only sampled from /metrics.

Reviewed fe771e7

return { forward: true };
}

const state = await readState();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a time-of-check/time-of-use gap rather than a handoff guard. A person can send input after readState() returns agent but before the proxy writes the request to the driver, so the Cua action still reaches the shared browser while that person is driving. Route agent actions through a desktop-side reservation/serialization point (or otherwise revalidate at the injection boundary) so the advertised fail-closed handoff is enforceable.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant