fix(browser): improve snapshots, native controls and action reliability

[waleedlatif1]

Companion PRs

• https://github.com/simstudioai/mothership/pull/502

Merge together into staging so the browser tool contracts and desktop implementation stay aligned.

Summary

• Preserve reachable click targets and retain obstruction/stale-target guards; use bounded native Electron capture with hidden-window preservation and no overlapping stalled captures.
• Preserve mixed inline snapshot text within the existing text budget and expose interactive headings as actionable references, retaining later controls on text-heavy pages.
• Support validated native structured inputs and atomic complete-set multi-select. Verify all selected values/labels after events, retain credential/focus checks, and never replay uncertain writes.
• Report unconfirmed effects clearly and remove obsolete capture decoding and nullable-image branches.

Uses Playwright text-node traversal, explicit-role precedence, and native fill/complete-set selection. Sim adds prevalidation before mutation and full post-event readback. Native capture keeps the existing five-second deadline and cancellation semantics. Canonical tool descriptions and additive schemas are in Mothership #502; deploy with the updated desktop for the new values argument.

Type of Change

• Bug fix

Testing

• Real Electron against 22 unchanged MiniWoB++ tasks: 91/110 before → 109/110 after; fresh holdout 218/220. Snapshot instruction coverage 90/110 → 110/110. These are deterministic driver scripts, not an LLM benchmark score; episode timeout is 120 seconds. Multi-select uses the new complete-set API.
• Remaining failures are a bundled jQuery autocomplete defect: Playwright locator.click reproduces the same wrong selection in 10/10 controlled trials. No widget-specific workaround added.
• 1,644 desktop unit tests, 13 real Electron scenarios, all workspace type checks, lint, 46 audits and generated-artifact checks passed. Regression tests include atomic refusal, duplicate-value swaps, focus changes, malformed native values and uncertain-write handling.
• Full repository run passed all 19 test tasks: 51,810 web-app tests passed; dedicated database suites remain skipped without their fixture configuration. A separate 31-case generated-schema suite includes the eight new selection-contract cases. All seven structured input types are exercised in Electron.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 16, 2026 1:07am UTC

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding findings or newly established correctness, security, or repository-rule violations.

Summary

This PR improves the Electron browser agent’s screenshot capture, snapshot traversal, click reliability, structured native input handling, and complete-set multi-select support.

• Replaces CDP screenshot decoding with bounded native Electron capture while preserving hidden-window state and preventing overlapping stalled captures.
• Preserves mixed inline snapshot text under a shared budget while retaining references for later controls.
• Avoids unnecessary scrolling for reachable click targets and revalidates targets after scrolling.
• Adds validated native writes for structured input types and atomic multi-select mutation with post-event readback.
• Propagates cancellation into screenshot capture and improves uncertain-effect reporting.
• Updates generated browser-tool contracts and expands unit and real-Electron regression coverage.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Request[Browser tool request] --> Validate[Validate arguments and current target]
  Validate --> Action{Action type}
  Action -->|Snapshot| Traverse[Traverse semantic page tree]
  Traverse --> Budget[Bound text output while retaining controls]
  Action -->|Click| Reachable{Target currently reachable?}
  Reachable -->|Yes| Dispatch[Dispatch guarded action]
  Reachable -->|No| Scroll[Scroll once and revalidate identity]
  Scroll --> Dispatch
  Action -->|Structured input| Prevalidate[Validate native value before mutation]
  Prevalidate --> Write[Use native setter and dispatch events]
  Action -->|Multi-select| Resolve[Resolve complete requested option set]
  Resolve --> Atomic[Apply complete selection atomically]
  Write --> Readback[Read post-event state]
  Atomic --> Readback
  Action -->|Screenshot| Capture[Bound native capture with hidden-window preservation]
  Capture --> Verify[Verify viewport and image dimensions]
  Dispatch --> Result[Return confirmed or uncertain effect]
  Readback --> Result
  Verify --> Result

Loading

_{Reviews (3) · Last reviewed commit: "fix(browser): share snapshot text budget..."}

[cubic-dev-ai]

All reported issues were addressed across 9 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 12 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 12 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[github-actions]

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

• ❌ simstudioai/mothership#502 — OPEN, not merged (targets staging) — fix(browser): align structured input and multiple selection contracts