Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 39 additions & 19 deletions docs/specs/dor-browser.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ size landed**, so a resize transient is not read as an external override.

| From -> To | Behavior |
| --- | --- |
| `iframe` -> `ab-screencast` / `ab-popout` | Host spawns a fresh `gui-<hex>` session at the current URL via `agentBrowserOpen`; `ab-popout` spawns headed in one shot, so the surface mounts already popped out. Hidden/inert without that capability. |
| `iframe` -> `ab-screencast` / `ab-popout` | **The pane swaps at once** to a session-less agent-browser pane — inert, so it cannot race the boot (rationale) — while the host spawns a fresh `gui-<hex>` session at the current URL via `agentBrowserOpen` and hands over `{session, wsPort, binaryPath}` as **one** params refresh. `ab-popout` spawns headed in one shot, so the surface mounts already popped out. A spawn that rejects or yields no session restores the iframe; a Surface minimized meanwhile receives either outcome through its Door, while one killed meanwhile closes a spawned session. Hidden/inert without that capability. |
| `ab-screencast` <-> `ab-popout` | Same session, headed/headless relaunch in `AgentBrowserPanel`; preserves only the active URL. |
| `ab-*` -> `iframe` | Uses canonical `params.url`; with multiple tabs, requires the user to press `c` in the warning overlay, because only the active tab survives. |

Expand Down Expand Up @@ -281,9 +281,19 @@ its own because clients trigger it. Rules park and recovery must not break:
- **Never set `AGENT_BROWSER_IDLE_TIMEOUT_MS`** for Dormouse-managed sessions —
daemon self-exit when idle would defeat "alive while parked".
- **Never query the daemon mid-relaunch** — see [Pop-Out](#pop-out).

The stream WebSocket carries frame pulses + status, tab snapshots, and native
`input_mouse` / `input_keyboard` input.
- **A relaunch in flight drops the stream and CDP observer at once**, shows a
relaunching placeholder, and reconnects only to the port the host hands back
(rationale). **One relaunch at a time**: a pop-out or pop-in issued during
one is ignored; a session-less pane has nothing to relaunch.
- **A `{session, wsPort, binaryPath}` refresh reconciles its session even at the
live port**, with no `stream status`.

The stream carries frames, status, tab snapshots, `url`, and native
`input_mouse` / `input_keyboard` input. **Control envelopes dispatch at any
size.** **`url` names the active tab at navigation commit; `tabs`
refreshes only when the driving command completes**
(for a slow page, after the load; rationale), so every commit clears the title
until `tabs` refreshes, even at the same URL.

**Two-stage paint.** A changed stream JPEG paints at once as a **provisional
frame** — the first image, and 250ms after pointer input (continuous movement
Expand Down Expand Up @@ -336,23 +346,33 @@ shared by the stream and `tab list --json`).

`ab-popout` relaunches the same session headed, because Chrome fixes
headed/headless at daemon launch. The pane becomes a stub with Pop back in, plus
Bring to front where a host implements `agentBrowserBringToFront`. **State
carried in v1 is only the active non-blank URL**: other tabs, DOM state, scroll,
form inputs, session storage, cookies/logins do not survive.
Bring to front where a host implements `agentBrowserBringToFront`; while the
window is still opening (a relaunch in flight, or an eager pane without its
session) the stub offers neither. **State carried in v1 is only the active
non-blank URL**: other tabs, DOM state, scroll, form inputs, session storage,
cookies/logins do not survive.

Host sequence: run `close`, **then terminate the daemon by its pid file**
(`$AGENT_BROWSER_SOCKET_DIR/<session>.pid`, default `~/.agent-browser`) **and
wait for it to exit** (rationale), then reopen and read a new stream port.
**Never query the daemon during the close/reopen gap** (rationale), host and
controller park/recovery paths alike, so **Dormouse supplies the active-tab URL
and the host trusts it**. After reopening, the host best-effort closes any stray
`about:blank` tab the close+reopen race left behind, **but only while a real page
is open**, so it never closes the sole tab (rationale).

While popped out, Dormouse keeps a stream/CDP observer so URL/header state
follows same-tab navigation and a headed window close can auto-revert to
headless. **Hosts must close tracked popped-out sessions on shutdown** to avoid
orphan headed windows.
wait for it to exit** (rationale), then reopen. **Never wait for the page to
load** (rationale): every launch — pop-out, pop-in, `agentBrowserOpen` —
resolves once the *relaunched* daemon is up (a pid file naming a pid other than
the killed one, and a `<session>.stream` file naming a port that accepts a
connection), asking `stream status` only after `open` returns. **A non-zero
`open` exit with the daemon up is a page still loading, not a failed launch**;
only a launch without a published port fails, including after a zero exit;
`agentBrowserOpen` then closes its spawn. **A headed session is tracked for
shutdown before its launch**, so a window whose page never loads is still
closed. **Never query the daemon during the close/reopen gap** (rationale), host and controller
park/recovery paths alike, so **Dormouse supplies the active-tab URL and the
host trusts it**. Once `open` returns, only a still-current relaunch best-effort
closes stray `about:blank` tabs, **and only while a real page is open**, so it
never closes the sole tab (rationale).

While popped out, Dormouse keeps a stream/CDP observer for same-tab URL/header
updates and headed-window close auto-revert. **Hosts must cancel pending
relaunch sweeps, then close tracked popped-out sessions on shutdown** so
quitting orphans no headed window.

Source of truth: `lib/src/components/wall/agent-browser-surface-controller.ts` (pop-out state, CDP
observer, auto-revert), `lib/src/host/agent-browser-host.ts` (`popOut`, `popIn`,
Expand All @@ -371,7 +391,7 @@ sidecar/Rust adapter.
| `agentBrowserStreamStatus` | Current stream port, for stale-`wsPort` recovery. |
| `agentBrowserEdit` | select-all/copy/cut via fixed host-owned JS plus an OS clipboard write. |
| `getAgentBrowserStreamUrl` | Direct stream URL, or the VS Code relay URL. |
| `agentBrowserOpen` | Spawn a GUI-owned session for iframe -> agent-browser. |
| `agentBrowserOpen` | Spawn a GUI-owned session for iframe -> agent-browser; resolves when the daemon is up, not when the page loads ([Pop-Out](#pop-out)). |
| `agentBrowserPopOut` / `agentBrowserPopIn` | Headed/headless relaunch. |
| `agentBrowserBringToFront` | Optional; no host implements it today. |

Expand Down
16 changes: 16 additions & 0 deletions docs/specs/dor-browser.rationale.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@

**Why the handover is a single params refresh.** Setting `session` is what reconciles the controller and connects it, so landing it ahead of `wsPort`/`binaryPath` — or before `agent-browser open` has returned — connects against a daemon that is not up. Handing it over even after a failed `open` lets the placeholder name what it is waiting for; the menu that would have reported the error closed long ago.

The persisted `wsPort` mirror can lag the controller's already-live port after a buffered write, so a simultaneous session change still reconciles when setting that port itself is a no-op.

## Display Modal And Render Swaps

**Why the iframe swap is eager.** The same 1–3s daemon boot as the context-menu connect, behind a modal that has already closed; and while the swap awaited `open`, a slow page held the iframe on screen for the whole load and a timed-out `open` dropped the swap silently — leaving an orphan `gui-<hex>` browser nobody could see or close.

## Agent-Browser Renderer

**Why one-session-one-surface is not an invariant.** `dor ab` forwards the user's command and then runs `stream status` before it asks the host for a surface, so a surface killed or render-swapped inside that window is gone by the time the trailing request arrives — and the session behind it is still live and needs somewhere to render.
Expand All @@ -38,14 +44,24 @@

**Why every non-crisp painter must bump the draw generation.** The byte-dedup compares an incoming capture against the last crisp draw. A resting page whose crisp bytes match that draw dedups to a no-op and strands the pane on the blurry provisional frame; a freshly re-attached canvas mounts blank and has the same problem.

**Why the connection is dropped at relaunch start rather than left to fail.** The host closes the browser and kills the daemon, so the old socket's close is certain. Left connected, its three reconnect failures flagged the pane "ended" about three seconds into a pop-in that a slow page could hold open for 25s, and the popped-out CDP observer's `get cdp-url` — issued the moment `poppedOut` flipped — landed in the close→reopen gap, where a daemon command spawns a competing headless daemon that the headed relaunch then reattaches to.

**Why `url` is tracked separately from `tabs`.** Measured against agent-browser 0.31.1 (2026-09): on `open`, the stream sends `tabs` (about:blank), then `url` naming the target at navigation commit, and refreshes `tabs` only when the CLI command completes — after `load`. During a slow load the tab list still named the previous page, so a pop-out issued then relaunched the page before the one being loaded.

**Whose limitation the CSS-resolution provisional frame is.** Chromium's `Page.startScreencast` captures in DIP and exposes no DPR knob, so the stream is CSS-resolution whatever the client asks for — upstream Chromium, not something agent-browser chose or could fix.

## Pop-Out

**The symptom when the daemon is not killed first.** `agent-browser --headed open` against a live headless daemon reattaches to it and exits 0, so the host logs a successful headed open and the mode never changes. The user presses Pop out, gets the pane stub with no OS window anywhere, and nothing in the logs says why.

**Why the host does not wait for `open`.** Measured against agent-browser 0.31.1 (2026-09): `open <url>` blocks until the page's `load` event, up to the CLI's 25s default action timeout, then exits 1 with "Operation timed out" — with the daemon up, the tab on the URL, and `stream status` answering. Every other daemon command queues behind it: a `stream status` issued mid-`open` returned after 22s. Meanwhile the daemon writes `<session>.pid` and `<session>.stream` within ~100ms of launch, and the stream serves status, tabs and frames from then on. Awaiting `open` therefore made a slow page cost the whole load before the pane showed anything, and turned the timeout into a "failed" relaunch — one whose headed window was never tracked for shutdown, because tracking followed a zero exit.

**Why the stale state files need the replaced pid.** SIGTERM leaves the dead daemon's `.pid` and `.stream` files in place for the new daemon to overwrite. A port read from the stale file is probed against nothing — unless some other process has since taken it — so the launch also waits for a pid other than the one it killed before it trusts the stream file.

**Why nothing may query the daemon during the close/reopen gap.** With the old daemon dead and the new one not yet up, a `stream status` or tab query spawns a *competing* daemon at `about:blank` — agent-browser's CLI starts one on demand — and the relaunch then races two daemons for the same session.

A post-open blank-tab sweep can become such a query when a later relaunch, explicit Surface close, or host shutdown starts before the earlier page finishes loading, so the host invalidates the sweep before any close can release that pending launch.

**Why the stray-`about:blank` sweep is guarded.** The close/reopen pair can leave an extra blank tab beside the navigated one. Sweeping blanks unconditionally is the obvious fix and is wrong: a session whose only tab is legitimately blank would lose it, leaving the pane with nothing to show.

## Agent-Browser Host Capabilities
Expand Down
124 changes: 124 additions & 0 deletions lib/src/components/Wall.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { SURFACE_CONTROL_METHODS } from 'dor/protocol';
import { sessionForKey } from 'dor-lib-common/agent-browser';
import { Wall } from './Wall';
import { getAgentBrowserScreenController } from './wall/agent-browser-screen';
import { setPlatform } from '../lib/platform';
import { FakePtyAdapter } from '../lib/platform/fake-adapter';
import type { PlatformAdapter } from '../lib/platform/types';
Expand Down Expand Up @@ -525,6 +526,129 @@ describe('Wall on the Lath engine', () => {
}
});

it('hands an eager render-swap session to a Surface minimized during launch', async () => {
const untouchedSpy = vi.spyOn(terminalRegistry, 'isUntouched').mockReturnValue(true);
let resolveOpen!: (result: { ok: boolean; session?: string; wsPort?: number; binaryPath?: string }) => void;
const openResult = new Promise<{ ok: boolean; session?: string; wsPort?: number; binaryPath?: string }>((resolve) => {
resolveOpen = resolve;
});
(fake as PlatformAdapter).agentBrowserOpen = vi.fn(() => openResult);
const agentBrowserCommand = vi.fn(async () => ({ exitCode: 0, stdout: '', stderr: '' }));
(fake as PlatformAdapter).agentBrowserCommand = agentBrowserCommand;

try {
await act(async () => {
root.render(<Wall initialPaneIds={['pane-a']} initialMode="command" showBaseboard />);
});
await flush();
const iframeId = (await dispatchIframe('http://localhost:5173/')).id;

await act(async () => {
getAgentBrowserScreenController(iframeId)?.actions.setRenderMode?.('ab-screencast');
});
await flush();
const eagerLeaf = container.querySelector<HTMLElement>('[data-lath-leaf]')!;
const eagerId = eagerLeaf.dataset.lathLeaf!;
expect(eagerId).not.toBe(iframeId);

await act(async () => {
eagerLeaf.querySelector<HTMLButtonElement>('[aria-label="Minimize"]')!.click();
});
await flush();
expect(container.querySelector(`[data-door-id="${eagerId}"]`)).not.toBeNull();

await act(async () => {
resolveOpen({ ok: true, session: 'dormouse.1.gui-minimized', wsPort: 4321, binaryPath: '/usr/bin/agent-browser' });
await openResult;
});
await flush();

expect(await dispatchResolveAgentBrowser(eagerId)).toEqual({
ok: true,
result: { surfaceId: eagerId, surfaceRef: 'surface:1', session: 'dormouse.1.gui-minimized' },
});
expect(agentBrowserCommand).not.toHaveBeenCalledWith(
'dormouse.1.gui-minimized',
['close'],
'/usr/bin/agent-browser',
);
} finally {
untouchedSpy.mockRestore();
}
});

it('restores a minimized eager render swap to iframe when launch returns no session', async () => {
const untouchedSpy = vi.spyOn(terminalRegistry, 'isUntouched').mockReturnValue(true);
let resolveOpen!: (result: { ok: boolean; error?: string }) => void;
const openResult = new Promise<{ ok: boolean; error?: string }>((resolve) => {
resolveOpen = resolve;
});
(fake as PlatformAdapter).agentBrowserOpen = vi.fn(() => openResult);

try {
await act(async () => {
root.render(<Wall initialPaneIds={['pane-a']} initialMode="command" showBaseboard />);
});
await flush();
const iframeId = (await dispatchIframe('http://localhost:5173/')).id;

await act(async () => {
getAgentBrowserScreenController(iframeId)?.actions.setRenderMode?.('ab-screencast');
});
await flush();
const eagerLeaf = container.querySelector<HTMLElement>('[data-lath-leaf]')!;
const eagerId = eagerLeaf.dataset.lathLeaf!;
await act(async () => {
eagerLeaf.querySelector<HTMLButtonElement>('[aria-label="Minimize"]')!.click();
});
await flush();

await act(async () => {
resolveOpen({ ok: false, error: 'launch failed' });
await openResult;
});
await flush();

expect(container.querySelector(`[data-door-id="${eagerId}"]`)).not.toBeNull();
expect(await dispatchResolveAgentBrowser(eagerId)).toEqual({
ok: false,
error: "surface 'surface:1' is not agent-browser rendered (render_mode: iframe)",
});
} finally {
untouchedSpy.mockRestore();
}
});

it('restores an eager render swap to iframe when launch rejects', async () => {
const untouchedSpy = vi.spyOn(terminalRegistry, 'isUntouched').mockReturnValue(true);
(fake as PlatformAdapter).agentBrowserOpen = vi.fn(async () => {
throw new Error('transport failed');
});

try {
await act(async () => {
root.render(<Wall initialPaneIds={['pane-a']} initialMode="command" showBaseboard />);
});
await flush();
const iframeId = (await dispatchIframe('http://localhost:5173/')).id;

await act(async () => {
getAgentBrowserScreenController(iframeId)?.actions.setRenderMode?.('ab-screencast');
});
await flush();

const restoredId = container.querySelector<HTMLElement>('[data-lath-leaf]')!.dataset.lathLeaf!;
expect(restoredId).not.toBe(iframeId);
expect(getAgentBrowserScreenController(restoredId)?.snapshot().renderMode).toBe('iframe');
expect(await dispatchResolveAgentBrowser('surface:1')).toEqual({
ok: false,
error: "surface 'surface:1' is not agent-browser rendered (render_mode: iframe)",
});
} finally {
untouchedSpy.mockRestore();
}
});

it('resolves a browser surface handle to its agent-browser session, and gates the rest', async () => {
const untouchedSpy = vi.spyOn(terminalRegistry, 'isUntouched').mockReturnValue(false);
(fake as PlatformAdapter).agentBrowserCommand = vi.fn(async () => ({ exitCode: 0, stdout: '', stderr: '' }));
Expand Down
Loading