Skip to content

Commit 1a82a17

Browse files
authored
feat(desktop): browser pages as resource tabs, agent works in a background tab (#7745)
* feat(desktop): browser pages as resource tabs, agent works in the background Each live desktop browser page is now its own resource tab in the Chat panel, derived from the desktop app's tab list instead of a persisted singleton "Browser" resource with a second tab strip inside it. The inner browser tab strip, tab pinning, duplicate, reorder, the native tab context menu, and the auto-replacement of a closed last tab are removed. The agent drives its own tab (the desktop's automation cursor) without moving the user's visible page: the renderer no longer force-activates the browser on every tool call nor auto-switches the panel to the agent's tab. The agent's tab is announced as resource activity, so the existing view policy shows it or badges it depending on whether the user has taken over selection. Mentions and attachments point at exact tabs; the chat row never stores browser resources, and legacy rows are dropped on read. * fix(desktop): settle browser tab sync edge cases and ease the tab width cap Review and in-app verification of the resource-tab browser model found five problems, all fixed here with tests: - The strip's switch and follow effects could trade `switch-tab` calls forever after a native switch; the switch effect is now keyed on the selection alone and ignores the switch it requested itself. - Migrating a pending chat onto its durable id briefly removed and re-added every browser tab; a missing store bucket is no longer read as closed. - Chat hydration could replace the resource list underneath a fresh add, dropping restored tabs on return; a tab stays projected until its resource is seen. Hydration also falls back to the last server-held resource instead of whichever tab happened to land first. - Strip-driven `switch-tab` no longer claims the page for the user (`claim: false`), so the agent can still close or adopt it; a user closing the agent's tab leaves the agent cursor unset rather than announcing the neighbour as agent activity. - The driver marked every restored scope as material, refusing pending chat migration after an empty restore; only a restore that yields pages does now, matching the session layer. The bridge snapshot is regenerated with the documented `'new-tab'` fallback. Resource tab titles ellipsize at 200px for one or two tabs, 180px for three, and 160px from four on, one small step at a time. * fix(desktop): mirror strip order into native tabs and drag live tab titles Reordering a browser tab in the resource strip now reorders the desktop's native tab list too, so restore and the agent's tab list keep the strip's order (`reorderTab` returns as an optional bridge method). Dragging a tab into the chat carries the strip's live title rather than the title captured when the tab was added. The fork route test follows the new policy that stored browser rows are dropped. * fix(emcn): drive the loader spin period from a CSS variable instead of an inline style
1 parent 8f6ccf7 commit 1a82a17

56 files changed

Lines changed: 1297 additions & 1562 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/desktop/src/main/browser-agent/driver.test.ts

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1277,6 +1277,18 @@ describe('executeTool', () => {
12771277
expect(driver.migrateBrowserScope('pending:other-chat', 'chat-occupied')).toBe(false)
12781278
})
12791279

1280+
it('keeps a durable destination adoptable after an empty restore', async () => {
1281+
await driver.executeTool('pending:new-chat', 'browser_open_tab', {})
1282+
driver.activateBrowserScope('chat-real')
1283+
expect(driver.restoreBrowserScope('chat-real')).toMatchObject({ tabs: [] })
1284+
1285+
expect(driver.migrateBrowserScope('pending:new-chat', 'chat-real')).toBe(true)
1286+
await expect(driver.executeTool('chat-real', 'browser_list_tabs', {})).resolves.toMatchObject({
1287+
ok: true,
1288+
result: { scopeId: 'chat-real', tabs: [{ tabId: '1' }] },
1289+
})
1290+
})
1291+
12801292
it('cancels only the replaced destination authorizations during migration', async () => {
12811293
await driver.executeTool('pending:new-chat', 'browser_open_tab', {})
12821294
driver.activateBrowserScope('chat-real')
@@ -1395,7 +1407,7 @@ describe('executeTool', () => {
13951407
it('keeps activation lazy, then restores and disposes through the driver API', async () => {
13961408
const snapshot: BrowserSessionSnapshot = {
13971409
v: 1,
1398-
tabs: [{ url: 'https://restored.example/', pinned: false }],
1410+
tabs: [{ url: 'https://restored.example/' }],
13991411
activeIndex: 0,
14001412
downloads: [],
14011413
}

apps/desktop/src/main/browser-agent/driver.ts

Lines changed: 8 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -794,10 +794,13 @@ export function restoreBrowserScope(scopeId: string): BrowserTabsState {
794794
return session.withBrowserScope(resolved, () => session.peekTabsState())
795795
}
796796
const state = driverScopeState(resolved)
797-
state.activationOnly = false
798797
return session.withBrowserScope(resolved, () => {
799798
session.restoreBrowserSession()
800-
return session.peekTabsState()
799+
const tabs = session.peekTabsState()
800+
// Only a scope that actually holds pages is material; one restored empty
801+
// stays adoptable by a pending chat migrating onto its id.
802+
if (tabs.tabs.length > 0) state.activationOnly = false
803+
return tabs
801804
})
802805
}
803806

@@ -905,7 +908,7 @@ export async function clearBrowserProfile(
905908
retireAllDriverScopeStates()
906909
const settingsCleared = knownSessions?.clear() !== false
907910
const outcomes = await Promise.allSettled([session.clearProfileStorage(), clearCredentials()])
908-
// Last, covering the pinned-tab list `clearProfileStorage` just emptied.
911+
// Last, covering the saved tab list `clearProfileStorage` just emptied.
909912
// Settings writes coalesce, and an erasure that is still sitting in that
910913
// window when the process dies leaves the previous account's data on disk
911914
// after sign-out already told the user it was gone.
@@ -2368,8 +2371,7 @@ async function executeToolInner(
23682371
}
23692372
}
23702373
assertCurrentExecution()
2371-
// The agent chose to open this page to work in, so the panel follows it.
2372-
const tab = session.addAutomationTab({ reveal: true })
2374+
const tab = session.addAutomationTab()
23732375
const contents = tab.view.webContents
23742376
if (url) {
23752377
assertCurrentExecution()
@@ -4782,19 +4784,9 @@ export async function handlePanelAction(
47824784
}
47834785
return
47844786
}
4785-
if (action.action === 'new-tab') {
4786-
session.addTab()
4787-
return
4788-
}
4789-
if (action.action === 'duplicate-tab') {
4790-
if (typeof action.tabId === 'string') {
4791-
session.duplicateTab(action.tabId)
4792-
}
4793-
return
4794-
}
47954787
if (action.action === 'switch-tab') {
47964788
if (typeof action.tabId === 'string') {
4797-
session.switchTab(action.tabId)
4789+
session.switchTab(action.tabId, { claim: action.claim !== false })
47984790
}
47994791
return
48004792
}

apps/desktop/src/main/browser-agent/panel.test.ts

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ function freshPanel(): PanelModule {
2020
getMainWindow: () => null,
2121
activeTab: () => null,
2222
backgroundColor: () => '#ffffff',
23-
ensureInitialTab: () => {},
23+
restoreActiveScope: () => {},
2424
onViewDetached: () => {},
2525
})
2626
panelModule.activatePanelScope('chat-test')
@@ -33,12 +33,12 @@ const PANEL_RECT = { x: 400, y: 64, width: 600, height: 800 }
3333
function showPanel(panel: PanelModule) {
3434
const win = new BrowserWindow()
3535
const view = new WebContentsView()
36-
const active = { id: 'tab-1', scopeId: 'chat-test', view, pinned: false }
36+
const active = { id: 'tab-1', scopeId: 'chat-test', view }
3737
panel.initPanel({
3838
getMainWindow: () => win,
3939
activeTab: () => active,
4040
backgroundColor: () => '#0c0c0c',
41-
ensureInitialTab: () => {},
41+
restoreActiveScope: () => {},
4242
onViewDetached: () => {},
4343
})
4444
panel.activatePanelScope('chat-test')
@@ -56,13 +56,13 @@ describe('panel chat scope', () => {
5656
it('returns keyboard focus to the renderer when attaching a view steals it mid-typing', () => {
5757
const win = new BrowserWindow()
5858
const view = new WebContentsView()
59-
const active = { id: 'tab-1', scopeId: 'chat-test', view, pinned: false }
59+
const active = { id: 'tab-1', scopeId: 'chat-test', view }
6060
vi.mocked(win.webContents.isFocused).mockReturnValue(true)
6161
panel.initPanel({
6262
getMainWindow: () => win,
6363
activeTab: () => active,
6464
backgroundColor: () => '#0c0c0c',
65-
ensureInitialTab: () => {},
65+
restoreActiveScope: () => {},
6666
onViewDetached: () => {},
6767
})
6868
panel.activatePanelScope('chat-test')
@@ -327,12 +327,12 @@ describe('panel chat scope', () => {
327327
it('applies a forced hide before the panel reports its first bounds', () => {
328328
const win = new BrowserWindow()
329329
const view = new WebContentsView()
330-
const active = { id: 'tab-1', scopeId: 'chat-test', view, pinned: false }
330+
const active = { id: 'tab-1', scopeId: 'chat-test', view }
331331
panel.initPanel({
332332
getMainWindow: () => win,
333333
activeTab: () => active,
334334
backgroundColor: () => '#0c0c0c',
335-
ensureInitialTab: () => {},
335+
restoreActiveScope: () => {},
336336
onViewDetached: () => {},
337337
})
338338
panel.activatePanelScope('chat-test')

apps/desktop/src/main/browser-agent/panel.ts

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -41,12 +41,8 @@ export interface PanelHost {
4141
activeTab: () => AgentTab | null
4242
/** Native backdrop used by a blank tab before its first page paint. */
4343
backgroundColor: () => string
44-
/**
45-
* Materializes the initial tab when the panel first becomes visible: a
46-
* visible browser resource always represents one open browser window, and
47-
* the tab strip, omnibox, and native session must not disagree about that.
48-
*/
49-
ensureInitialTab: () => void
44+
/** Hydrates the active scope's saved pages when the panel first becomes visible. */
45+
restoreActiveScope: () => void
5046
/** Lets the session drop focus tracking for a view that is no longer attached. */
5147
onViewDetached: (view: WebContentsView | null) => void
5248
}
@@ -55,7 +51,7 @@ let host: PanelHost = {
5551
getMainWindow: () => null,
5652
activeTab: () => null,
5753
backgroundColor: () => '#ffffff',
58-
ensureInitialTab: () => {},
54+
restoreActiveScope: () => {},
5955
onViewDetached: () => {},
6056
}
6157

@@ -834,7 +830,7 @@ export function setPanelBounds(
834830
panelBounds = bounds
835831
panelAnchor = bounds === null ? null : (anchor ?? null)
836832
if (bounds !== null) {
837-
host.ensureInitialTab()
833+
host.restoreActiveScope()
838834
} else {
839835
resetOcclusion()
840836
}

0 commit comments

Comments
 (0)