diff --git a/docs/docs/configuration.md b/docs/docs/configuration.md index a458100..210958d 100644 --- a/docs/docs/configuration.md +++ b/docs/docs/configuration.md @@ -41,6 +41,10 @@ sandbox: max_ttl_minutes: 1440 default_network_kind: bridge images_file: images/catalog.yaml +screenshots: + base_url: http://127.0.0.1:8081 + listen: 127.0.0.1:8081 + dir: screenshots ``` | Field | Contract | @@ -53,12 +57,26 @@ images_file: images/catalog.yaml | `incus.pool` | Required storage pool for instance root disks. | | `sandbox.default_ttl_minutes` | Positive creation default, 240 minutes if omitted. | | `sandbox.max_ttl_minutes` | Positive upper bound, 1440 minutes if omitted; must be at least the default. | -| `sandbox.default_network_kind` | `bridge` (default) or `ovn`. OVN requires an existing central, configured chassis, and physical uplink. | +| `sandbox.default_network_kind` | `ovn` (default) or `bridge`. OVN requires an existing central, configured chassis, and physical uplink. | | `images_file` | Schema-version-1 catalog path; defaults to `images/catalog.yaml`. | -| `screenshots.dir`, `screenshots.base_url` | Accepted schema fields reserved for desktop integration; the screenshot service is not implemented. | +| `screenshots.base_url` | Required absolute HTTP(S) URL reachable from the agent host, without credentials, query, or fragment. Never derived from the request's `Host` header. | +| `screenshots.dir` | Scratch parent directory; defaults to `screenshots` beside the configuration file. Each process locks the parent and clears only its owned child directory. | +| `screenshots.listen` | Separate screenshot listener in STDIO mode; defaults to `127.0.0.1:8081`. HTTP mode instead mounts screenshots beside MCP on `--addr`. | `sandbox.extend` replaces the expiry with **now + TTL**, rather than adding time to the old expiry. Explicit deletion first expires the project so a partial failure is retried by the reaper. The reaper scans at startup and every 30 seconds. +The example above targets STDIO. For `agentcompute http --addr localhost:8080`, +set `screenshots.base_url` to the agent-reachable URL for port 8080 instead. +Configure a reverse proxy consistently if the public URL has a path prefix. +An unreachable URL is a network configuration error; there is no base64 fallback. + +Screenshot URLs are bearer capabilities with 128-bit random identifiers. They +do not require the MCP bearer token. Only GET and HEAD are accepted; responses +use `image/png`, `Cache-Control: no-store`, and `X-Content-Type-Options: nosniff`. +Retention is five minutes or sandbox expiry, whichever is earlier. Sandbox +deletion and reaping purge associated screenshots. A PNG may use at most 16 MiB; +the store rejects new publications at 128 MiB rather than evicting live images. + In OVN mode, sandbox projects own their logical networks and NICs use managed networks only. Networks span members. A `nat=false` network is isolated, has no external allocation or direct outside path, and requires peering or a dual-NIC router for reachability. External forwards require NAT-enabled networks. Automatic instance placement prefers free RAM, then one-minute load, then member name; explicit online hosts take precedence. In bridge mode, the sandbox's default bridge enables IPv4 DHCP and NAT. Additional bridge networks require `kind="bridge"`. Physical network names are opaque `ac` plus eight lowercase hex characters; `user.agentcompute.sandbox`, `.name`, and `.version` metadata resolve their logical names. Pending creations also reserve the physical name on the project for cleanup. diff --git a/go.mod b/go.mod index 0745b1e..cd19f9a 100644 --- a/go.mod +++ b/go.mod @@ -13,6 +13,7 @@ require ( github.com/spf13/viper v1.21.0 github.com/stretchr/testify v1.12.1 go.yaml.in/yaml/v3 v3.0.5 + golang.org/x/image v0.31.0 ) require ( diff --git a/go.sum b/go.sum index 71d9c86..9e37de9 100644 --- a/go.sum +++ b/go.sum @@ -323,6 +323,8 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M= golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis= +golang.org/x/image v0.31.0 h1:mLChjE2MV6g1S7oqbXC0/UcKijjm5fnJLUYKIYrLESA= +golang.org/x/image v0.31.0/go.mod h1:R9ec5Lcp96v9FTF+ajwaH3uGxPH4fKfHHAVbUILxghA= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= diff --git a/images/README.md b/images/README.md index 12af9b2..9fb615f 100644 --- a/images/README.md +++ b/images/README.md @@ -97,6 +97,12 @@ in the restricted `image-build` project, publishes to the existing `ghcr.io/gilmanlab/agentcompute/` namespace, and fetches each digest back independently before opening a public catalog PR. +A newly created GHCR package starts private. For this public catalog, its +owner must explicitly make the package public in GitHub's package settings +after the first publication. This change cannot be reversed to private. +Verify anonymous access before merging its catalog entry: authenticated +fetch-back by the publisher does not establish public availability. + The immutable tag hashes definition inputs under `images/`, excluding `catalog.yaml` and Markdown. An existing release skips assembly and publication but is still fetched and boot-qualified. Registry or @@ -108,6 +114,9 @@ attestations; new private bakes must not be described as carrying that provenance. Digest verification and boot tests establish different properties. The server's startup reconciler imports catalog digest references into `image-build`, verifies bytes, smoke-launches the image, and moves the alias only after success. It records the imgoci digest in image properties; the Incus fingerprint is derived, not a stable identity across rebuilds. +Desktop VM qualification waits for X11, the automation user's Driver +service, and a native `list_apps` call through that user's session. It uses +the catalog's CPU and memory defaults and does not require the GitHub runner. For a verified download without import: @@ -217,12 +226,13 @@ Scratch is sampled every 100 ms, so short peaks between samples are missed. ### Desktop Phase 6 -Two local desktop builds used fresh work and output directories: +The two local builds and protected publication produced these measurements: | Build evidence | Download | Compile | Assemble | Peak RSS | Scratch high-water | `incus.tar.xz` | `disk.qcow2` | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | First image | 5.849 s | 31.670 s | 842.017 s | 507,180 KiB | 7,625,043,968 B | 656 B | 744,611,840 B | | Corrected networkd rebuild | 5.824 s | 31.834 s | 392.512 s | 517,712 KiB | 7,621,808,128 B | 640 B | 745,013,248 B | +| Protected published build | 6.624 s | 33.283 s | 403.364 s | 505,216 KiB | 7,620,071,424 B | 628 B | 742,923,264 B | Download includes verified retrieval of Go, vendored distrobuilder source, Ubuntu base, the snapshot CA package, and the full Cua Driver archive. Compile @@ -234,18 +244,28 @@ subsequently passed `images/ubuntu-24.04-desktop/smoke.py` in `image-build`: X11, the graphical user's active Driver service, and 234 native `list_apps` entries. The smoke removed its own VM and imported image. -Protected bootstrap PR #23 and private bake run 34854323245 completed -successfully. The desktop-aware publisher must be deployed before the -four-image bake. Desktop publication, verified fetch-back, and the public -catalog update remain pending; no desktop GHCR digest is claimed. +The desktop-aware publisher is deployed. Image PR #25 merged as +`e4333f245b4e81c8d7753038f0ddf04a620bd0a2`, and +[protected bake 34872818589](https://github.com/GilmanLab/agentcompute-images/actions/runs/34872818589) +built, boot-qualified, published, and fetched back all four images. +The desktop pipeline took 674.806 s and produced: + +```text +ghcr.io/gilmanlab/agentcompute/ubuntu-24.04-desktop@sha256:5dc4e120a79dd06ad6784e69474f0617387f74cb98685af8844170b7165ea8e2 +``` + +[Catalog PR #27](https://github.com/GilmanLab/agentcompute/pull/27) records +this digest and remains unmerged by request. Published artifact hashes and +the independent publisher rollout evidence are in the spike report. The corrected image passed the complete production-stdio MCP acceptance run after OVN recovery: private-only client, desktop readiness, native `list_apps`, -PNG URL fetch/decode, one foreground token click changing one editor tab to -two, reboot recovery, and the VNC endpoint reported by `desktop.info`. -The representative program took 23.624 s; the full run took 158.45 s. -The running reaper returned 404 for the original screenshot 27.120 s after -the shortened sandbox expiry. See the spike report for the native background +PNG URL fetch/decode, one foreground token click changing a single document +to two document tabs, reboot recovery, and the VNC endpoint reported by +`desktop.info`. The post-refactor repeat's representative program took +22.764 s; full acceptance took 160.32 s. The running reaper returned 404 for +the original screenshot 26.804 s after the shortened sandbox expiry. +See the spike report for the native background delivery limitation and [fleet PR #20](https://github.com/GilmanLab/fleet/pull/20) for the separately recovered stale-CA reconnect storm and active log limits. diff --git a/internal/cli/dependencies_test.go b/internal/cli/dependencies_test.go index 1292245..8950332 100644 --- a/internal/cli/dependencies_test.go +++ b/internal/cli/dependencies_test.go @@ -20,6 +20,6 @@ func testDependencies(t *testing.T) *mcpserver.Dependencies { require.NoError(t, err) service, err := compute.New(mocks.NewMockBackend(t), catalog, compute.Options{Host: "lab01"}) require.NoError(t, err) - deps := mcpserver.NewDependencies(service) + deps := mcpserver.NewDependencies(service, nil) return &deps } diff --git a/internal/cli/desktop_integration_test.go b/internal/cli/desktop_integration_test.go new file mode 100644 index 0000000..58624f5 --- /dev/null +++ b/internal/cli/desktop_integration_test.go @@ -0,0 +1,218 @@ +//go:build integration + +package cli + +import ( + "bytes" + "context" + "fmt" + "image" + _ "image/png" + "io" + "net" + "net/http" + "os" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/GilmanLab/agentcompute/internal/incus" +) + +// TestDesktopAcceptance drives real Driver tools through the production stdio +// MCP server. The representative client never receives an uplink-facing NIC; +// a separate viewer guest verifies the human VNC endpoint and reboot recovery. +func TestDesktopAcceptance(t *testing.T) { + fx := loadOVNFixture(t) + ctx, cancel := context.WithTimeout(t.Context(), 12*time.Minute) + defer cancel() + root, dir := integrationRoot(t), t.TempDir() + binary := buildAgentcompute(ctx, t, root, dir) + config := writeOVNConfig(t, dir, root, fx) + imageName := envOr("AGENTCOMPUTE_TEST_DESKTOP_IMAGE", "ubuntu/24.04/desktop") + name := fmt.Sprintf("desktop-%x", time.Now().UnixNano()) + backend, err := incus.New( + ctx, + incus.Options{Remote: fx.Remote, Pool: "data", OVNUplink: fx.Uplink, OVNRanges: fx.Ranges}, + ) + require.NoError(t, err) + t.Cleanup(func() { assert.NoError(t, backend.Close()) }) + t.Cleanup(func() { + cleanup, stop := context.WithTimeout(context.Background(), 2*time.Minute) + defer stop() + if _, err := backend.GetSandbox(cleanup, name); err == nil { + assert.NoError(t, backend.DeleteSandbox(cleanup, name)) + } + }) + session, _ := startClusterClient(ctx, t, binary, config, + "sandbox.create", "sandbox.extend", "instance.create", "instance.get", "instance.exec", + "instance.wait", "instance.restart", "net.create", "net.attach", "net.forward", + "desktop.info", "desktop.enable", "desktop.call", "desktop.screenshot") + + started := time.Now() + representative := integrationExecute(ctx, t, session, fmt.Sprintf(`def main(): + name = %q + sandbox.create(name=name, ttl_minutes=10) + net.create(sandbox=name, name="lan", cidr="192.168.50.0/24", nat=False) + net.create(sandbox=name, name="wan", cidr="10.99.0.0/24", nat=True) + instance.create(sandbox=name, name="rtr", image="router", network="lan") + net.attach(sandbox=name, instance="rtr", network="wan") + nat = instance.exec(sandbox=name, name="rtr", command="/opt/router/nat --mode port-restricted --inside eth0 --outside eth1") + if nat["exit_code"] != 0: + fail(nat["stderr"]) + instance.create(sandbox=name, name="client", image=%q, network="lan") + waited = instance.wait(sandbox=name, name="client", until="desktop", timeout_seconds=300) + shot = desktop.screenshot(sandbox=name, instance="client") + called = desktop.call(sandbox=name, instance="client", tool="list_apps") + if not called["ok"]: + fail(called["summary"]) + return {"router": instance.get(sandbox=name, name="rtr"), "client": instance.get(sandbox=name, name="client"), "screenshot": shot, "apps": json.decode(called["result"]), "waited": waited} +`, name, imageName)) + t.Logf("representative desktop program: %s", time.Since(started)) + clientNICs := asSlice(t, asMap(t, representative["client"])["nics"]) + require.Len(t, clientNICs, 1, "the representative client must remain private-only") + assert.Equal(t, "lan", asMap(t, clientNICs[0])["network"]) + apps := asSlice(t, asMap(t, representative["apps"])["apps"]) + var editorInstalled bool + for _, app := range apps { + if asMap(t, app)["bundle_id"] == "org.gnome.TextEditor" { + editorInstalled = true + } + } + require.True(t, editorInstalled, "native list_apps must expose the installed editor") + shot := asMap(t, representative["screenshot"]) + originalURL := asString(t, shot["url"]) + _, dimensions := fetchDesktopPNG(t, originalURL) + assert.Equal(t, jsonInt(t, shot["width"]), int64(dimensions.Width)) + assert.Equal(t, jsonInt(t, shot["height"]), int64(dimensions.Height)) + + interaction := integrationExecute(ctx, t, session, fmt.Sprintf(`def main(): + name = %q + launched = desktop.call(sandbox=name, instance="client", tool="launch_app", args=json.encode({"name":"gnome-text-editor"})) + if not launched["ok"]: + fail(launched["summary"]) + app = json.decode(launched["result"]) + pid = app["pid"] + window = app["windows"][0]["window_id"] + args = json.encode({"pid":pid,"window_id":window,"include_accessibility_tree":True}) + before_call = desktop.call(sandbox=name, instance="client", tool="get_window_state", args=args) + if not before_call["ok"]: + fail(before_call["summary"]) + before = json.decode(before_call["result"]) + token = "" + for element in before["elements"]: + if element.get("label") == "New tab" and "click" in element.get("actions", []): + token = element["element_token"] + break + if not token: + fail("editor snapshot has no actionable New tab element") + clicked = desktop.call(sandbox=name, instance="client", tool="click", args=json.encode({"pid":pid,"window_id":window,"element_token":token,"delivery_mode":"foreground"})) + if not clicked["ok"]: + fail(clicked["summary"]) + after_call = desktop.call(sandbox=name, instance="client", tool="get_window_state", args=args) + if not after_call["ok"]: + fail(after_call["summary"]) + after = json.decode(after_call["result"]) + return {"before":before["tree_markdown"], "after":after["tree_markdown"], "before_url":before_call["screenshot_url"], "after_url":after_call["screenshot_url"], "click":json.decode(clicked["result"]), "scaled":desktop.screenshot(sandbox=name, instance="client", max_dimension=640), "window_scaled":desktop.screenshot(sandbox=name, instance="client", pid=pid, window_id=window, max_dimension=640), "window_width":before["window_bounds"]["width"]} +`, name)) + assert.Equal(t, 1, strings.Count(asString(t, interaction["before"]), `tab panel = "New Document"`)) + assert.Equal(t, 2, strings.Count(asString(t, interaction["after"]), `tab panel = "New Document"`)) + beforePNG, _ := fetchDesktopPNG(t, asString(t, interaction["before_url"])) + afterPNG, _ := fetchDesktopPNG(t, asString(t, interaction["after_url"])) + assert.False(t, bytes.Equal(beforePNG, afterPNG), "the verified second tab must also change the screenshot") + scaled := asMap(t, interaction["scaled"]) + _, scaledDimensions := fetchDesktopPNG(t, asString(t, scaled["url"])) + assert.LessOrEqual(t, scaledDimensions.Width, 640) + assert.LessOrEqual(t, scaledDimensions.Height, 640) + windowScaled := asMap(t, interaction["window_scaled"]) + _, windowDimensions := fetchDesktopPNG(t, asString(t, windowScaled["url"])) + assert.LessOrEqual(t, max(windowDimensions.Width, windowDimensions.Height), 640) + assert.InDelta( + t, + float64(jsonInt(t, interaction["window_width"]))/float64(windowDimensions.Width), + windowScaled["scale"], + 1e-9, + ) + t.Logf("token click native outcome (state verified separately): %v", interaction["click"]) + if evidence := os.Getenv("AGENTCOMPUTE_TEST_DESKTOP_EVIDENCE"); evidence != "" { + require.NoError(t, os.MkdirAll(evidence, 0o700)) + require.NoError(t, os.WriteFile(evidence+"/before.png", beforePNG, 0o600)) + require.NoError(t, os.WriteFile(evidence+"/after.png", afterPNG, 0o600)) + } + + viewer := integrationExecute(ctx, t, session, fmt.Sprintf(`def main(): + name = %q + instance.create(sandbox=name, name="viewer", image=%q) + enabled = desktop.enable(sandbox=name, instance="viewer") + forward = net.forward(sandbox=name, network="default", instance="viewer", port=5900, listen_port=5900, protocol="tcp") + before = desktop.info(sandbox=name, instance="viewer") + instance.restart(sandbox=name, name="viewer") + waited = instance.wait(sandbox=name, name="viewer", until="desktop", timeout_seconds=300) + after = desktop.info(sandbox=name, instance="viewer") + return {"enabled":enabled,"forward":forward,"before":before,"after":after,"waited":waited,"private_client":instance.get(sandbox=name, name="client")} +`, name, imageName)) + afterInfo := asMap(t, viewer["after"]) + assert.True(t, jsonBool(t, asMap(t, viewer["enabled"])["ready"])) + assert.True(t, jsonBool(t, afterInfo["ready"])) + assert.Equal(t, "0.28.1", afterInfo["driver_version"]) + assert.Contains(t, asSlice(t, afterInfo["tools"]), "get_window_state") + assert.Len(t, asSlice(t, asMap(t, viewer["private_client"])["nics"]), 1) + forward := asMap(t, viewer["forward"]) + vncAddress := asString(t, afterInfo["vnc"]) + assert.Equal( + t, + net.JoinHostPort(asString(t, forward["address"]), fmt.Sprint(jsonInt(t, forward["port"]))), + vncAddress, + ) + vnc, err := net.DialTimeout("tcp", vncAddress, 10*time.Second) + require.NoError(t, err) + defer vnc.Close() + require.NoError(t, vnc.SetReadDeadline(time.Now().Add(10*time.Second))) + banner := make([]byte, 12) + _, err = io.ReadFull(vnc, banner) + require.NoError(t, err) + assert.Equal(t, "RFB 003.008\n", string(banner)) + t.Logf("reboot ready and VNC reachable: %s", vncAddress) + + // Shorten the sandbox lifetime after publication. Its first screenshot was + // initially retained for five minutes, so only lifecycle purge can make it + // disappear at the newly shortened expiry. + extended := integrationExecute(ctx, t, session, fmt.Sprintf(`def main(): + return sandbox.extend(name=%q, ttl_minutes=1) +`, name)) + expires, err := time.Parse(time.RFC3339, asString(t, extended["expires_at"])) + require.NoError(t, err) + httpClient := &http.Client{Timeout: 5 * time.Second} + require.Eventually(t, func() bool { + response, err := httpClient.Get(originalURL) + if err != nil { + return false + } + defer response.Body.Close() + return response.StatusCode == http.StatusNotFound + }, 100*time.Second, time.Second, "expired sandbox screenshot must be purged by the running reaper") + t.Logf("screenshot 404 observed %s after shortened sandbox expiry", time.Since(expires)) +} + +func fetchDesktopPNG(t *testing.T, url string) ([]byte, image.Config) { + t.Helper() + client := &http.Client{Timeout: 15 * time.Second} + response, err := client.Get(url) + require.NoError(t, err) + defer response.Body.Close() + require.Equal(t, http.StatusOK, response.StatusCode) + assert.Equal(t, "image/png", response.Header.Get("Content-Type")) + assert.Equal(t, "nosniff", response.Header.Get("X-Content-Type-Options")) + assert.Equal(t, "no-store", response.Header.Get("Cache-Control")) + data, err := io.ReadAll(io.LimitReader(response.Body, (16<<20)+1)) + require.NoError(t, err) + require.LessOrEqual(t, len(data), 16<<20) + decoded, format, err := image.Decode(bytes.NewReader(data)) + require.NoError(t, err) + require.Equal(t, "png", format) + bounds := decoded.Bounds() + return data, image.Config{Width: bounds.Dx(), Height: bounds.Dy()} +} diff --git a/internal/cli/http.go b/internal/cli/http.go index bf280a8..db30d09 100644 --- a/internal/cli/http.go +++ b/internal/cli/http.go @@ -18,6 +18,7 @@ import ( "github.com/meigma/codemode/authz" hostmcp "github.com/meigma/codemode/mcpserver" + "github.com/GilmanLab/agentcompute/internal/desktop" "github.com/GilmanLab/agentcompute/internal/mcpserver" "github.com/GilmanLab/agentcompute/internal/templateinfo" ) @@ -69,7 +70,8 @@ type httpConfig struct { // keeping logs on stderr stays consistent with the stdio transport. logger *slog.Logger // deps are constructed once before serving any HTTP session. - deps mcpserver.Dependencies + deps mcpserver.Dependencies + screenshots *desktop.Store } // newHTTPCommand builds the "http" subcommand, which serves the MCP server over @@ -103,12 +105,13 @@ func newHTTPCommand(options Options) *cobra.Command { return err } runErr := runHTTP(cmd.Context(), httpConfig{ - build: options.Build, - addr: options.Viper.GetString(addrFlag), - authToken: options.Viper.GetString(authTokenFlag), - insecure: options.Viper.GetBool(insecureFlag), - logger: logger, - deps: rt.deps, + build: options.Build, + addr: options.Viper.GetString(addrFlag), + authToken: options.Viper.GetString(authTokenFlag), + insecure: options.Viper.GetBool(insecureFlag), + logger: logger, + deps: rt.deps, + screenshots: rt.screenshots, }) return errors.Join(runErr, rt.close()) }, @@ -202,6 +205,12 @@ func serveHTTP(ctx context.Context, ln net.Listener, cfg httpConfig) error { if cfg.authToken != "" { rootHandler = requireBearerToken(cfg.authToken, cfg.addr)(rootHandler) } + if cfg.screenshots != nil { + mux := http.NewServeMux() + mux.Handle(desktop.ScreenshotPath, cfg.screenshots) + mux.Handle("/", rootHandler) + rootHandler = mux + } srv := &http.Server{ Handler: rootHandler, diff --git a/internal/cli/integration_test.go b/internal/cli/integration_test.go index 126f775..1fabafe 100644 --- a/internal/cli/integration_test.go +++ b/internal/cli/integration_test.go @@ -6,6 +6,7 @@ import ( "context" "encoding/json" "fmt" + "net" "os" "os/exec" "path/filepath" @@ -46,6 +47,7 @@ func TestClusterLifecycle(t *testing.T) { host, filepath.Join(root, "images", "catalog.yaml"), ) + text += integrationScreenshotConfig(t) require.NoError(t, os.WriteFile(config, []byte(text), 0o600)) backend, err := incus.New(ctx, incus.Options{Remote: remote, Host: host, Pool: "data"}) require.NoError(t, err) @@ -305,3 +307,12 @@ func jsonBool(t *testing.T, value any) bool { require.True(t, ok, "expected bool, got %T", value) return out } + +func integrationScreenshotConfig(t *testing.T) string { + t.Helper() + listener, err := net.Listen("tcp", "127.0.0.1:0") + require.NoError(t, err) + address := listener.Addr().String() + require.NoError(t, listener.Close()) + return fmt.Sprintf("screenshots:\n listen: %q\n base_url: %q\n", address, "http://"+address) +} diff --git a/internal/cli/ovn_integration_test.go b/internal/cli/ovn_integration_test.go index 38b6055..755ca15 100644 --- a/internal/cli/ovn_integration_test.go +++ b/internal/cli/ovn_integration_test.go @@ -522,7 +522,12 @@ func writeOVNConfig(t *testing.T, dir, root string, fx ovnFixture) string { fmt.Fprintf(&b, " ovn_ranges: %q\n", fx.Ranges) } b.WriteString("sandbox:\n default_network_kind: ovn\n default_ttl_minutes: 30\n") - fmt.Fprintf(&b, "images_file: %q\n", filepath.Join(root, "images", "catalog.yaml")) + fmt.Fprintf( + &b, + "images_file: %q\n", + envOr("AGENTCOMPUTE_TEST_CATALOG", filepath.Join(root, "images", "catalog.yaml")), + ) + b.WriteString(integrationScreenshotConfig(t)) path := filepath.Join(dir, "config.yaml") require.NoError(t, os.WriteFile(path, []byte(b.String()), 0o600)) return path diff --git a/internal/cli/runtime.go b/internal/cli/runtime.go index 0ddb04c..afcf7b7 100644 --- a/internal/cli/runtime.go +++ b/internal/cli/runtime.go @@ -19,6 +19,7 @@ import ( "go.yaml.in/yaml/v3" "github.com/GilmanLab/agentcompute/internal/compute" + "github.com/GilmanLab/agentcompute/internal/desktop" "github.com/GilmanLab/agentcompute/internal/incus" "github.com/GilmanLab/agentcompute/internal/mcpserver" ) @@ -28,12 +29,15 @@ const ( defaultTTLMinutes = 240 maxTTLMinutes = 1440 yamlExtension = ".yaml" + tomlExtension = ".toml" ) // runtime owns backend connections and the reaper, not individual MCP sessions. type runtime struct { - deps mcpserver.Dependencies - close func() error + deps mcpserver.Dependencies + close func() error + screenshots *desktop.Store + screenshotListen string } type runtimeConfig struct { @@ -64,6 +68,7 @@ type sandboxConfig struct { type screenshotConfig struct { Dir string `yaml:"dir" toml:"dir"` BaseURL string `yaml:"base_url" toml:"base_url"` + Listen string `yaml:"listen" toml:"listen"` } func loadRuntimeConfig(path string) (runtimeConfig, error) { @@ -73,7 +78,8 @@ func loadRuntimeConfig(path string) (runtimeConfig, error) { MaxTTLMinutes: maxTTLMinutes, DefaultNetworkKind: "ovn", }, - ImagesFile: "images/catalog.yaml", + Screenshots: screenshotConfig{Dir: "screenshots", Listen: "127.0.0.1:8081"}, + ImagesFile: "images/catalog.yaml", } if path == "" { return cfg, errors.New("configuration is required: use --config or AGENTCOMPUTE_CONFIG") @@ -95,7 +101,7 @@ func decodeRuntimeConfig(path string, cfg *runtimeConfig) error { } defer file.Close() switch strings.ToLower(filepath.Ext(path)) { - case ".toml": + case tomlExtension: err = toml.NewDecoder(file).DisallowUnknownFields().Decode(cfg) case yamlExtension, ".yml": err = decodeYAMLConfig(file, cfg) @@ -142,6 +148,9 @@ func validateRuntimeConfig(cfg runtimeConfig) error { if cfg.ImagesFile == "" { return errors.New("images_file must not be empty") } + if cfg.Screenshots.BaseURL == "" { + return errors.New("screenshots.base_url is required") + } return nil } @@ -182,16 +191,27 @@ func newRuntime(ctx context.Context, path string, logger *slog.Logger) (*runtime if err != nil { return nil, errors.Join(err, client.Close()) } + screenshots, err := desktop.NewStore(cfg.Screenshots.Dir, cfg.Screenshots.BaseURL) + if err != nil { + return nil, errors.Join(err, client.Close()) + } + var driver *desktop.Driver service, err := compute.New(client, catalog, compute.Options{ Host: cfg.Incus.Host, DefaultNetworkKind: cfg.Sandbox.DefaultNetworkKind, DefaultTTL: time.Duration(cfg.Sandbox.DefaultTTLMinutes) * time.Minute, MaxTTL: time.Duration(cfg.Sandbox.MaxTTLMinutes) * time.Minute, Logger: logger, + OnSandboxExpired: screenshots.PurgeSandbox, + OnReap: screenshots.Sweep, + DesktopReady: func(ctx context.Context, ref compute.Ref) (bool, error) { + return driver.Ready(ctx, ref) + }, }) if err != nil { - return nil, errors.Join(err, client.Close()) + return nil, errors.Join(err, screenshots.Close(), client.Close()) } + driver = desktop.NewDriver(service, screenshots) lifecycle, cancel := context.WithCancel(ctx) done := make(chan struct{}) go func() { @@ -200,11 +220,16 @@ func newRuntime(ctx context.Context, path string, logger *slog.Logger) (*runtime logger.ErrorContext(lifecycle, "reaper stopped", "err", reapErr) } }() - return &runtime{deps: mcpserver.NewDependencies(service), close: func() error { - cancel() - <-done - return client.Close() - }}, nil + return &runtime{ + deps: mcpserver.NewDependencies(service, driver), + screenshots: screenshots, + screenshotListen: cfg.Screenshots.Listen, + close: func() error { + cancel() + <-done + return errors.Join(screenshots.Close(), client.Close()) + }, + }, nil } func (o Options) openRuntime(ctx context.Context, logger *slog.Logger) (*runtime, error) { diff --git a/internal/cli/runtime_test.go b/internal/cli/runtime_test.go index ac07176..5681199 100644 --- a/internal/cli/runtime_test.go +++ b/internal/cli/runtime_test.go @@ -14,13 +14,18 @@ func TestRuntimeConfigurationRejectsInvalidDocuments(t *testing.T) { for _, tc := range []struct{ name, extension, body string }{ {"unknown YAML key", yamlExtension, "incus:\n remote: nas01\n host: lab01\n pool: data\n typo: true\n"}, {"second YAML document", yamlExtension, "incus:\n remote: nas01\n host: lab01\n pool: data\n---\n{}\n"}, - {"unknown TOML key", ".toml", "[incus]\nremote='nas01'\nhost='lab01'\npool='data'\ntypo=true\n"}, + {"unknown TOML key", tomlExtension, "[incus]\nremote='nas01'\nhost='lab01'\npool='data'\ntypo=true\n"}, {"ambiguous endpoint", yamlExtension, "incus:\n remote: nas01\n url: https://example.invalid\n host: lab01\n pool: data\n"}, {"overflowing TTL", yamlExtension, "incus:\n remote: nas01\n host: lab01\n pool: data\nsandbox:\n max_ttl_minutes: 9223372036854775807\n"}, } { t.Run(tc.name, func(t *testing.T) { t.Parallel() path := filepath.Join(t.TempDir(), "config"+tc.extension) + if tc.extension == tomlExtension { + tc.body += "\n[screenshots]\nbase_url='http://127.0.0.1:8081'\n" + } else { + tc.body += "\nscreenshots:\n base_url: http://127.0.0.1:8081\n" + } require.NoError(t, os.WriteFile(path, []byte(tc.body), 0o600)) _, err := loadRuntimeConfig(path) require.Error(t, err) @@ -37,7 +42,7 @@ func TestRuntimeConfigurationResolvesPathsRelativeToFile(t *testing.T) { os.WriteFile( path, []byte( - "images_file='catalog.yaml'\n[incus]\nurl='https://example.invalid'\nclient_cert='client.crt'\nclient_key='client.key'\nhost='lab01'\npool='data'\n", + "images_file='catalog.yaml'\n[incus]\nurl='https://example.invalid'\nclient_cert='client.crt'\nclient_key='client.key'\nhost='lab01'\npool='data'\n[screenshots]\nbase_url='http://127.0.0.1:8081'\n", ), 0o600, ), diff --git a/internal/cli/screenshots.go b/internal/cli/screenshots.go new file mode 100644 index 0000000..3291152 --- /dev/null +++ b/internal/cli/screenshots.go @@ -0,0 +1,49 @@ +package cli + +import ( + "context" + "errors" + "fmt" + "io" + "log/slog" + "net" + "net/http" +) + +func (rt *runtime) runStdio( + ctx context.Context, + logger *slog.Logger, + build BuildInfo, + in io.Reader, + out io.Writer, +) error { + if rt.screenshots == nil { + return runStdio(ctx, logger, build, in, out, rt.deps) + } + lifecycle, cancel := context.WithCancel(ctx) + defer cancel() + var lc net.ListenConfig + listener, err := lc.Listen(lifecycle, "tcp", rt.screenshotListen) + if err != nil { + return fmt.Errorf("listen for screenshots on %s: %w", rt.screenshotListen, err) + } + server := &http.Server{Handler: rt.screenshots, ReadHeaderTimeout: httpReadHeaderTimeout} + served := make(chan error, 1) + go func() { + err := server.Serve(listener) + if errors.Is(err, http.ErrServerClosed) { + err = nil + } + served <- err + cancel() + }() + logger.InfoContext(ctx, "serving screenshots", "addr", listener.Addr().String()) + runErr := runStdio(lifecycle, logger, build, in, out, rt.deps) + shutdownCtx, stop := context.WithTimeout(context.WithoutCancel(ctx), httpShutdownTimeout) + defer stop() + shutdownErr := server.Shutdown(shutdownCtx) + if shutdownErr != nil { + shutdownErr = errors.Join(shutdownErr, server.Close()) + } + return errors.Join(runErr, shutdownErr, <-served) +} diff --git a/internal/cli/stdio.go b/internal/cli/stdio.go index e026aed..3650638 100644 --- a/internal/cli/stdio.go +++ b/internal/cli/stdio.go @@ -48,7 +48,7 @@ func newStdioCommand(options Options) *cobra.Command { if err != nil { return err } - runErr := runStdio(cmd.Context(), logger, options.Build, cmd.InOrStdin(), cmd.OutOrStdout(), rt.deps) + runErr := rt.runStdio(cmd.Context(), logger, options.Build, cmd.InOrStdin(), cmd.OutOrStdout()) return errors.Join(runErr, rt.close()) }, } diff --git a/internal/compute/desktop.go b/internal/compute/desktop.go new file mode 100644 index 0000000..5af67b4 --- /dev/null +++ b/internal/compute/desktop.go @@ -0,0 +1,90 @@ +package compute + +import ( + "context" + "errors" + "io" + "os" + "time" +) + +const ( + desktopJSONLimit = 4 << 20 + desktopPollInterval = 500 * time.Millisecond +) + +// ExecJSON runs a guest argv with a 4 MiB output bound and rejects truncation. +// It uses the same draining, deadline, and cancellation path as Exec. +func (s *Service) ExecJSON(ctx context.Context, req ExecRequest) (ExecResult, error) { + result, err := s.exec(ctx, req, desktopJSONLimit) + if err != nil { + return result, err + } + if result.StdoutTruncated || result.StderrTruncated { + return ExecResult{}, agentError("Driver output exceeds the 4 MiB JSON limit") + } + return result, nil +} + +// ReadBinaryFile opens a guest file for bounded streaming by an internal consumer. +// Missing files retain [os.ErrNotExist] so optional screenshots need no text parsing. +func (s *Service) ReadBinaryFile(ctx context.Context, ref Ref, path string) (io.ReadCloser, error) { + if err := validateRef(ref); err != nil { + return nil, err + } + if err := validateFilePath(path); err != nil { + return nil, err + } + if _, err := s.SandboxExpiry(ctx, ref.Sandbox); err != nil { + return nil, err + } + body, err := s.backend.ReadBinaryFile(ctx, ref, path) + if err != nil && !errors.Is(err, os.ErrNotExist) { + return nil, s.mapBackend(ctx, "read binary file", err) + } + return body, err +} + +// SandboxExpiry reads live metadata without taking the control-plane mutation gate. +func (s *Service) SandboxExpiry(ctx context.Context, name string) (time.Time, error) { + if err := validateName(name); err != nil { + return time.Time{}, err + } + box, err := s.backend.GetSandbox(ctx, name) + if err != nil { + if errors.Is(err, ErrNotFound) { + return time.Time{}, sandboxNotFound(name) + } + return time.Time{}, s.backendError(ctx, "get sandbox", err) + } + if !box.ExpiresAt.After(time.Now()) { + return time.Time{}, sandboxExpired(name) + } + return box.ExpiresAt, nil +} + +func (s *Service) waitDesktop(ctx context.Context, ref Ref) (WaitResult, error) { + if s.desktopReady == nil { + return WaitResult{}, agentError("desktop readiness is not configured") + } + result, err := s.backend.WaitInstance(ctx, WaitRequest{Ref: ref, Until: WaitUntilAgent}) + if err != nil { + return result, err + } + ticker := time.NewTicker(desktopPollInterval) + defer ticker.Stop() + for { + ready, err := s.desktopReady(ctx, ref) + if err != nil { + return result, err + } + if ready { + return result, nil + } + select { + case <-ctx.Done(): + return result, ctx.Err() + case <-ticker.C: + } + } +} diff --git a/internal/compute/exec_test.go b/internal/compute/exec_test.go index 36e0eb2..89beb8d 100644 --- a/internal/compute/exec_test.go +++ b/internal/compute/exec_test.go @@ -20,7 +20,9 @@ func TestExecTimeoutSetsFlagNotCancellation(t *testing.T) { tc := newTestContext(t) ref := compute.Ref{Sandbox: "demo", Name: "web"} - tc.backend.EXPECT().GetInstance(mock.Anything, ref).Return(runningInstance(), nil) + inst := runningInstance() + inst.Status = "Ready" + tc.backend.EXPECT().GetInstance(mock.Anything, ref).Return(inst, nil) tc.backend.EXPECT().Exec(mock.Anything, mock.Anything, mock.Anything, mock.Anything). RunAndReturn(func(ctx context.Context, _ compute.ExecRequest, stdout, stderr io.Writer) (int64, error) { _, _ = stdout.Write([]byte("out")) diff --git a/internal/compute/lifecycle.go b/internal/compute/lifecycle.go index a9fcda5..9db40b0 100644 --- a/internal/compute/lifecycle.go +++ b/internal/compute/lifecycle.go @@ -17,7 +17,7 @@ const ( WaitUntilAgent = "agent" // WaitUntilNetwork means at least one NIC has a non-link address. WaitUntilNetwork = "network" - // WaitUntilDesktop is reserved for Phase 6. + // WaitUntilDesktop means the Driver answers in the guest graphical session. WaitUntilDesktop = "desktop" // WaitUntilStopped means the guest has reached Stopped. WaitUntilStopped = "stopped" @@ -122,7 +122,13 @@ func (s *Service) WaitInstance(ctx context.Context, req WaitRequest) (WaitResult waitCtx, cancel := execContext(ctx, req.Timeout) defer cancel() started := time.Now() - result, err := s.backend.WaitInstance(waitCtx, req) + var result WaitResult + var err error + if req.Until == WaitUntilDesktop { + result, err = s.waitDesktop(waitCtx, req.Ref) + } else { + result, err = s.backend.WaitInstance(waitCtx, req) + } result.Elapsed = time.Since(started) if err == nil { return result, nil @@ -365,10 +371,8 @@ func (s *Service) mapBackend(ctx context.Context, op string, err error) error { func validateWaitUntil(until string) error { switch until { - case WaitUntilRunning, WaitUntilAgent, WaitUntilNetwork, WaitUntilStopped: + case WaitUntilRunning, WaitUntilAgent, WaitUntilNetwork, WaitUntilStopped, WaitUntilDesktop: return nil - case WaitUntilDesktop: - return agentErrorf("until %q is not available yet", until) default: return agentErrorf("until %q is not available yet", until) } diff --git a/internal/compute/lifecycle_test.go b/internal/compute/lifecycle_test.go index e739ef6..8426899 100644 --- a/internal/compute/lifecycle_test.go +++ b/internal/compute/lifecycle_test.go @@ -14,30 +14,6 @@ import ( "github.com/GilmanLab/agentcompute/internal/compute" ) -func TestWaitInstanceRejectsDesktopAndUnknownUntil(t *testing.T) { - t.Parallel() - - tests := []struct { - name string - until string - }{ - {name: "desktop is reserved", until: compute.WaitUntilDesktop}, - {name: "unknown stage", until: "booted"}, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - t.Parallel() - tc := newTestContext(t) - _, err := tc.service.WaitInstance(t.Context(), compute.WaitRequest{ - Ref: compute.Ref{Sandbox: "demo", Name: "web"}, - Until: tt.until, - }) - requireAgentContains(t, err, "not available yet") - }) - } -} - func TestWriteFileRejectsOversizedContentAndRelativePath(t *testing.T) { t.Parallel() diff --git a/internal/compute/mocks/mock_backend.go b/internal/compute/mocks/mock_backend.go index 04fbcf7..05ed980 100644 --- a/internal/compute/mocks/mock_backend.go +++ b/internal/compute/mocks/mock_backend.go @@ -77,7 +77,7 @@ type MockBackend_AddACLRule_Call struct { // - s string // - s1 string // - aCLRule compute.ACLRule -func (_e *MockBackend_Expecter) AddACLRule(context1 interface{}, s interface{}, s1 interface{}, aCLRule interface{}) *MockBackend_AddACLRule_Call { +func (_e *MockBackend_Expecter) AddACLRule(context1 any, s any, s1 any, aCLRule any) *MockBackend_AddACLRule_Call { return &MockBackend_AddACLRule_Call{Call: _e.mock.On("AddACLRule", context1, s, s1, aCLRule)} } @@ -157,7 +157,7 @@ type MockBackend_AttachNIC_Call struct { // - s1 string // - s2 string // - s3 string -func (_e *MockBackend_Expecter) AttachNIC(context1 interface{}, ref interface{}, s interface{}, s1 interface{}, s2 interface{}, s3 interface{}) *MockBackend_AttachNIC_Call { +func (_e *MockBackend_Expecter) AttachNIC(context1 any, ref any, s any, s1 any, s2 any, s3 any) *MockBackend_AttachNIC_Call { return &MockBackend_AttachNIC_Call{Call: _e.mock.On("AttachNIC", context1, ref, s, s1, s2, s3)} } @@ -245,7 +245,7 @@ type MockBackend_BeginCreateInstance_Call struct { // BeginCreateInstance is a helper method to define mock.On call // - context1 context.Context // - createInstance compute.CreateInstance -func (_e *MockBackend_Expecter) BeginCreateInstance(context1 interface{}, createInstance interface{}) *MockBackend_BeginCreateInstance_Call { +func (_e *MockBackend_Expecter) BeginCreateInstance(context1 any, createInstance any) *MockBackend_BeginCreateInstance_Call { return &MockBackend_BeginCreateInstance_Call{Call: _e.mock.On("BeginCreateInstance", context1, createInstance)} } @@ -316,7 +316,7 @@ type MockBackend_CreateForward_Call struct { // - n int64 // - n1 int64 // - s2 string -func (_e *MockBackend_Expecter) CreateForward(context1 interface{}, s interface{}, s1 interface{}, ref interface{}, n interface{}, n1 interface{}, s2 interface{}) *MockBackend_CreateForward_Call { +func (_e *MockBackend_Expecter) CreateForward(context1 any, s any, s1 any, ref any, n any, n1 any, s2 any) *MockBackend_CreateForward_Call { return &MockBackend_CreateForward_Call{Call: _e.mock.On("CreateForward", context1, s, s1, ref, n, n1, s2)} } @@ -408,7 +408,7 @@ type MockBackend_CreateNetwork_Call struct { // - context1 context.Context // - s string // - network compute.Network -func (_e *MockBackend_Expecter) CreateNetwork(context1 interface{}, s interface{}, network interface{}) *MockBackend_CreateNetwork_Call { +func (_e *MockBackend_Expecter) CreateNetwork(context1 any, s any, network any) *MockBackend_CreateNetwork_Call { return &MockBackend_CreateNetwork_Call{Call: _e.mock.On("CreateNetwork", context1, s, network)} } @@ -470,7 +470,7 @@ type MockBackend_CreateSandbox_Call struct { // CreateSandbox is a helper method to define mock.On call // - context1 context.Context // - sandbox compute.Sandbox -func (_e *MockBackend_Expecter) CreateSandbox(context1 interface{}, sandbox interface{}) *MockBackend_CreateSandbox_Call { +func (_e *MockBackend_Expecter) CreateSandbox(context1 any, sandbox any) *MockBackend_CreateSandbox_Call { return &MockBackend_CreateSandbox_Call{Call: _e.mock.On("CreateSandbox", context1, sandbox)} } @@ -528,7 +528,7 @@ type MockBackend_CreateSnapshot_Call struct { // - context1 context.Context // - ref compute.Ref // - s string -func (_e *MockBackend_Expecter) CreateSnapshot(context1 interface{}, ref interface{}, s interface{}) *MockBackend_CreateSnapshot_Call { +func (_e *MockBackend_Expecter) CreateSnapshot(context1 any, ref any, s any) *MockBackend_CreateSnapshot_Call { return &MockBackend_CreateSnapshot_Call{Call: _e.mock.On("CreateSnapshot", context1, ref, s)} } @@ -590,7 +590,7 @@ type MockBackend_DeleteInstance_Call struct { // DeleteInstance is a helper method to define mock.On call // - context1 context.Context // - ref compute.Ref -func (_e *MockBackend_Expecter) DeleteInstance(context1 interface{}, ref interface{}) *MockBackend_DeleteInstance_Call { +func (_e *MockBackend_Expecter) DeleteInstance(context1 any, ref any) *MockBackend_DeleteInstance_Call { return &MockBackend_DeleteInstance_Call{Call: _e.mock.On("DeleteInstance", context1, ref)} } @@ -648,7 +648,7 @@ type MockBackend_DeleteNetwork_Call struct { // - context1 context.Context // - s string // - s1 string -func (_e *MockBackend_Expecter) DeleteNetwork(context1 interface{}, s interface{}, s1 interface{}) *MockBackend_DeleteNetwork_Call { +func (_e *MockBackend_Expecter) DeleteNetwork(context1 any, s any, s1 any) *MockBackend_DeleteNetwork_Call { return &MockBackend_DeleteNetwork_Call{Call: _e.mock.On("DeleteNetwork", context1, s, s1)} } @@ -710,7 +710,7 @@ type MockBackend_DeleteSandbox_Call struct { // DeleteSandbox is a helper method to define mock.On call // - context1 context.Context // - s string -func (_e *MockBackend_Expecter) DeleteSandbox(context1 interface{}, s interface{}) *MockBackend_DeleteSandbox_Call { +func (_e *MockBackend_Expecter) DeleteSandbox(context1 any, s any) *MockBackend_DeleteSandbox_Call { return &MockBackend_DeleteSandbox_Call{Call: _e.mock.On("DeleteSandbox", context1, s)} } @@ -768,7 +768,7 @@ type MockBackend_DeleteSnapshot_Call struct { // - context1 context.Context // - ref compute.Ref // - s string -func (_e *MockBackend_Expecter) DeleteSnapshot(context1 interface{}, ref interface{}, s interface{}) *MockBackend_DeleteSnapshot_Call { +func (_e *MockBackend_Expecter) DeleteSnapshot(context1 any, ref any, s any) *MockBackend_DeleteSnapshot_Call { return &MockBackend_DeleteSnapshot_Call{Call: _e.mock.On("DeleteSnapshot", context1, ref, s)} } @@ -831,7 +831,7 @@ type MockBackend_DetachNIC_Call struct { // - context1 context.Context // - ref compute.Ref // - s string -func (_e *MockBackend_Expecter) DetachNIC(context1 interface{}, ref interface{}, s interface{}) *MockBackend_DetachNIC_Call { +func (_e *MockBackend_Expecter) DetachNIC(context1 any, ref any, s any) *MockBackend_DetachNIC_Call { return &MockBackend_DetachNIC_Call{Call: _e.mock.On("DetachNIC", context1, ref, s)} } @@ -904,7 +904,7 @@ type MockBackend_Exec_Call struct { // - execRequest compute.ExecRequest // - writer io.Writer // - writer1 io.Writer -func (_e *MockBackend_Expecter) Exec(context1 interface{}, execRequest interface{}, writer interface{}, writer1 interface{}) *MockBackend_Exec_Call { +func (_e *MockBackend_Expecter) Exec(context1 any, execRequest any, writer any, writer1 any) *MockBackend_Exec_Call { return &MockBackend_Exec_Call{Call: _e.mock.On("Exec", context1, execRequest, writer, writer1)} } @@ -981,7 +981,7 @@ type MockBackend_ExtendSandbox_Call struct { // - context1 context.Context // - s string // - time1 time.Time -func (_e *MockBackend_Expecter) ExtendSandbox(context1 interface{}, s interface{}, time1 interface{}) *MockBackend_ExtendSandbox_Call { +func (_e *MockBackend_Expecter) ExtendSandbox(context1 any, s any, time1 any) *MockBackend_ExtendSandbox_Call { return &MockBackend_ExtendSandbox_Call{Call: _e.mock.On("ExtendSandbox", context1, s, time1)} } @@ -1052,7 +1052,7 @@ type MockBackend_GetInstance_Call struct { // GetInstance is a helper method to define mock.On call // - context1 context.Context // - ref compute.Ref -func (_e *MockBackend_Expecter) GetInstance(context1 interface{}, ref interface{}) *MockBackend_GetInstance_Call { +func (_e *MockBackend_Expecter) GetInstance(context1 any, ref any) *MockBackend_GetInstance_Call { return &MockBackend_GetInstance_Call{Call: _e.mock.On("GetInstance", context1, ref)} } @@ -1119,7 +1119,7 @@ type MockBackend_GetNetwork_Call struct { // - context1 context.Context // - s string // - s1 string -func (_e *MockBackend_Expecter) GetNetwork(context1 interface{}, s interface{}, s1 interface{}) *MockBackend_GetNetwork_Call { +func (_e *MockBackend_Expecter) GetNetwork(context1 any, s any, s1 any) *MockBackend_GetNetwork_Call { return &MockBackend_GetNetwork_Call{Call: _e.mock.On("GetNetwork", context1, s, s1)} } @@ -1190,7 +1190,7 @@ type MockBackend_GetSandbox_Call struct { // GetSandbox is a helper method to define mock.On call // - context1 context.Context // - s string -func (_e *MockBackend_Expecter) GetSandbox(context1 interface{}, s interface{}) *MockBackend_GetSandbox_Call { +func (_e *MockBackend_Expecter) GetSandbox(context1 any, s any) *MockBackend_GetSandbox_Call { return &MockBackend_GetSandbox_Call{Call: _e.mock.On("GetSandbox", context1, s)} } @@ -1257,7 +1257,7 @@ type MockBackend_GetSandboxImage_Call struct { // - context1 context.Context // - s string // - s1 string -func (_e *MockBackend_Expecter) GetSandboxImage(context1 interface{}, s interface{}, s1 interface{}) *MockBackend_GetSandboxImage_Call { +func (_e *MockBackend_Expecter) GetSandboxImage(context1 any, s any, s1 any) *MockBackend_GetSandboxImage_Call { return &MockBackend_GetSandboxImage_Call{Call: _e.mock.On("GetSandboxImage", context1, s, s1)} } @@ -1294,6 +1294,84 @@ func (_c *MockBackend_GetSandboxImage_Call) RunAndReturn(run func(context1 conte return _c } +// InstanceForward provides a mock function for the type MockBackend +func (_mock *MockBackend) InstanceForward(context1 context.Context, ref compute.Ref, n int64, s string) (compute.Forward, error) { + ret := _mock.Called(context1, ref, n, s) + + if len(ret) == 0 { + panic("no return value specified for InstanceForward") + } + + var r0 compute.Forward + var r1 error + if returnFunc, ok := ret.Get(0).(func(context.Context, compute.Ref, int64, string) (compute.Forward, error)); ok { + return returnFunc(context1, ref, n, s) + } + if returnFunc, ok := ret.Get(0).(func(context.Context, compute.Ref, int64, string) compute.Forward); ok { + r0 = returnFunc(context1, ref, n, s) + } else { + r0 = ret.Get(0).(compute.Forward) + } + if returnFunc, ok := ret.Get(1).(func(context.Context, compute.Ref, int64, string) error); ok { + r1 = returnFunc(context1, ref, n, s) + } else { + r1 = ret.Error(1) + } + return r0, r1 +} + +// MockBackend_InstanceForward_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'InstanceForward' +type MockBackend_InstanceForward_Call struct { + *mock.Call +} + +// InstanceForward is a helper method to define mock.On call +// - context1 context.Context +// - ref compute.Ref +// - n int64 +// - s string +func (_e *MockBackend_Expecter) InstanceForward(context1 any, ref any, n any, s any) *MockBackend_InstanceForward_Call { + return &MockBackend_InstanceForward_Call{Call: _e.mock.On("InstanceForward", context1, ref, n, s)} +} + +func (_c *MockBackend_InstanceForward_Call) Run(run func(context1 context.Context, ref compute.Ref, n int64, s string)) *MockBackend_InstanceForward_Call { + _c.Call.Run(func(args mock.Arguments) { + var arg0 context.Context + if args[0] != nil { + arg0 = args[0].(context.Context) + } + var arg1 compute.Ref + if args[1] != nil { + arg1 = args[1].(compute.Ref) + } + var arg2 int64 + if args[2] != nil { + arg2 = args[2].(int64) + } + var arg3 string + if args[3] != nil { + arg3 = args[3].(string) + } + run( + arg0, + arg1, + arg2, + arg3, + ) + }) + return _c +} + +func (_c *MockBackend_InstanceForward_Call) Return(forward compute.Forward, err error) *MockBackend_InstanceForward_Call { + _c.Call.Return(forward, err) + return _c +} + +func (_c *MockBackend_InstanceForward_Call) RunAndReturn(run func(context1 context.Context, ref compute.Ref, n int64, s string) (compute.Forward, error)) *MockBackend_InstanceForward_Call { + _c.Call.Return(run) + return _c +} + // ListInstances provides a mock function for the type MockBackend func (_mock *MockBackend) ListInstances(context1 context.Context, s string) ([]compute.Instance, error) { ret := _mock.Called(context1, s) @@ -1330,7 +1408,7 @@ type MockBackend_ListInstances_Call struct { // ListInstances is a helper method to define mock.On call // - context1 context.Context // - s string -func (_e *MockBackend_Expecter) ListInstances(context1 interface{}, s interface{}) *MockBackend_ListInstances_Call { +func (_e *MockBackend_Expecter) ListInstances(context1 any, s any) *MockBackend_ListInstances_Call { return &MockBackend_ListInstances_Call{Call: _e.mock.On("ListInstances", context1, s)} } @@ -1398,7 +1476,7 @@ type MockBackend_ListNetworks_Call struct { // ListNetworks is a helper method to define mock.On call // - context1 context.Context // - s string -func (_e *MockBackend_Expecter) ListNetworks(context1 interface{}, s interface{}) *MockBackend_ListNetworks_Call { +func (_e *MockBackend_Expecter) ListNetworks(context1 any, s any) *MockBackend_ListNetworks_Call { return &MockBackend_ListNetworks_Call{Call: _e.mock.On("ListNetworks", context1, s)} } @@ -1465,7 +1543,7 @@ type MockBackend_ListSandboxes_Call struct { // ListSandboxes is a helper method to define mock.On call // - context1 context.Context -func (_e *MockBackend_Expecter) ListSandboxes(context1 interface{}) *MockBackend_ListSandboxes_Call { +func (_e *MockBackend_Expecter) ListSandboxes(context1 any) *MockBackend_ListSandboxes_Call { return &MockBackend_ListSandboxes_Call{Call: _e.mock.On("ListSandboxes", context1)} } @@ -1528,7 +1606,7 @@ type MockBackend_ListSnapshots_Call struct { // ListSnapshots is a helper method to define mock.On call // - context1 context.Context // - ref compute.Ref -func (_e *MockBackend_Expecter) ListSnapshots(context1 interface{}, ref interface{}) *MockBackend_ListSnapshots_Call { +func (_e *MockBackend_Expecter) ListSnapshots(context1 any, ref any) *MockBackend_ListSnapshots_Call { return &MockBackend_ListSnapshots_Call{Call: _e.mock.On("ListSnapshots", context1, ref)} } @@ -1587,7 +1665,7 @@ type MockBackend_PeerNetworks_Call struct { // - s string // - s1 string // - s2 string -func (_e *MockBackend_Expecter) PeerNetworks(context1 interface{}, s interface{}, s1 interface{}, s2 interface{}) *MockBackend_PeerNetworks_Call { +func (_e *MockBackend_Expecter) PeerNetworks(context1 any, s any, s1 any, s2 any) *MockBackend_PeerNetworks_Call { return &MockBackend_PeerNetworks_Call{Call: _e.mock.On("PeerNetworks", context1, s, s1, s2)} } @@ -1664,7 +1742,7 @@ type MockBackend_PublishInstance_Call struct { // - context1 context.Context // - ref compute.Ref // - s string -func (_e *MockBackend_Expecter) PublishInstance(context1 interface{}, ref interface{}, s interface{}) *MockBackend_PublishInstance_Call { +func (_e *MockBackend_Expecter) PublishInstance(context1 any, ref any, s any) *MockBackend_PublishInstance_Call { return &MockBackend_PublishInstance_Call{Call: _e.mock.On("PublishInstance", context1, ref, s)} } @@ -1701,6 +1779,80 @@ func (_c *MockBackend_PublishInstance_Call) RunAndReturn(run func(context1 conte return _c } +// ReadBinaryFile provides a mock function for the type MockBackend +func (_mock *MockBackend) ReadBinaryFile(context1 context.Context, ref compute.Ref, s string) (io.ReadCloser, error) { + ret := _mock.Called(context1, ref, s) + + if len(ret) == 0 { + panic("no return value specified for ReadBinaryFile") + } + + var r0 io.ReadCloser + var r1 error + if returnFunc, ok := ret.Get(0).(func(context.Context, compute.Ref, string) (io.ReadCloser, error)); ok { + return returnFunc(context1, ref, s) + } + if returnFunc, ok := ret.Get(0).(func(context.Context, compute.Ref, string) io.ReadCloser); ok { + r0 = returnFunc(context1, ref, s) + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(io.ReadCloser) + } + } + if returnFunc, ok := ret.Get(1).(func(context.Context, compute.Ref, string) error); ok { + r1 = returnFunc(context1, ref, s) + } else { + r1 = ret.Error(1) + } + return r0, r1 +} + +// MockBackend_ReadBinaryFile_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ReadBinaryFile' +type MockBackend_ReadBinaryFile_Call struct { + *mock.Call +} + +// ReadBinaryFile is a helper method to define mock.On call +// - context1 context.Context +// - ref compute.Ref +// - s string +func (_e *MockBackend_Expecter) ReadBinaryFile(context1 any, ref any, s any) *MockBackend_ReadBinaryFile_Call { + return &MockBackend_ReadBinaryFile_Call{Call: _e.mock.On("ReadBinaryFile", context1, ref, s)} +} + +func (_c *MockBackend_ReadBinaryFile_Call) Run(run func(context1 context.Context, ref compute.Ref, s string)) *MockBackend_ReadBinaryFile_Call { + _c.Call.Run(func(args mock.Arguments) { + var arg0 context.Context + if args[0] != nil { + arg0 = args[0].(context.Context) + } + var arg1 compute.Ref + if args[1] != nil { + arg1 = args[1].(compute.Ref) + } + var arg2 string + if args[2] != nil { + arg2 = args[2].(string) + } + run( + arg0, + arg1, + arg2, + ) + }) + return _c +} + +func (_c *MockBackend_ReadBinaryFile_Call) Return(readCloser io.ReadCloser, err error) *MockBackend_ReadBinaryFile_Call { + _c.Call.Return(readCloser, err) + return _c +} + +func (_c *MockBackend_ReadBinaryFile_Call) RunAndReturn(run func(context1 context.Context, ref compute.Ref, s string) (io.ReadCloser, error)) *MockBackend_ReadBinaryFile_Call { + _c.Call.Return(run) + return _c +} + // ReadFile provides a mock function for the type MockBackend func (_mock *MockBackend) ReadFile(context1 context.Context, fileReadRequest compute.FileReadRequest) (compute.FileReadResult, error) { ret := _mock.Called(context1, fileReadRequest) @@ -1735,7 +1887,7 @@ type MockBackend_ReadFile_Call struct { // ReadFile is a helper method to define mock.On call // - context1 context.Context // - fileReadRequest compute.FileReadRequest -func (_e *MockBackend_Expecter) ReadFile(context1 interface{}, fileReadRequest interface{}) *MockBackend_ReadFile_Call { +func (_e *MockBackend_Expecter) ReadFile(context1 any, fileReadRequest any) *MockBackend_ReadFile_Call { return &MockBackend_ReadFile_Call{Call: _e.mock.On("ReadFile", context1, fileReadRequest)} } @@ -1794,7 +1946,7 @@ type MockBackend_RemoveACLRule_Call struct { // - s string // - s1 string // - s2 string -func (_e *MockBackend_Expecter) RemoveACLRule(context1 interface{}, s interface{}, s1 interface{}, s2 interface{}) *MockBackend_RemoveACLRule_Call { +func (_e *MockBackend_Expecter) RemoveACLRule(context1 any, s any, s1 any, s2 any) *MockBackend_RemoveACLRule_Call { return &MockBackend_RemoveACLRule_Call{Call: _e.mock.On("RemoveACLRule", context1, s, s1, s2)} } @@ -1871,7 +2023,7 @@ type MockBackend_RestartInstance_Call struct { // - context1 context.Context // - ref compute.Ref // - b bool -func (_e *MockBackend_Expecter) RestartInstance(context1 interface{}, ref interface{}, b interface{}) *MockBackend_RestartInstance_Call { +func (_e *MockBackend_Expecter) RestartInstance(context1 any, ref any, b any) *MockBackend_RestartInstance_Call { return &MockBackend_RestartInstance_Call{Call: _e.mock.On("RestartInstance", context1, ref, b)} } @@ -1934,7 +2086,7 @@ type MockBackend_RestoreSnapshot_Call struct { // - context1 context.Context // - ref compute.Ref // - s string -func (_e *MockBackend_Expecter) RestoreSnapshot(context1 interface{}, ref interface{}, s interface{}) *MockBackend_RestoreSnapshot_Call { +func (_e *MockBackend_Expecter) RestoreSnapshot(context1 any, ref any, s any) *MockBackend_RestoreSnapshot_Call { return &MockBackend_RestoreSnapshot_Call{Call: _e.mock.On("RestoreSnapshot", context1, ref, s)} } @@ -2006,7 +2158,7 @@ type MockBackend_StartInstance_Call struct { // - context1 context.Context // - ref compute.Ref // - b bool -func (_e *MockBackend_Expecter) StartInstance(context1 interface{}, ref interface{}, b interface{}) *MockBackend_StartInstance_Call { +func (_e *MockBackend_Expecter) StartInstance(context1 any, ref any, b any) *MockBackend_StartInstance_Call { return &MockBackend_StartInstance_Call{Call: _e.mock.On("StartInstance", context1, ref, b)} } @@ -2078,7 +2230,7 @@ type MockBackend_StopInstance_Call struct { // - context1 context.Context // - ref compute.Ref // - b bool -func (_e *MockBackend_Expecter) StopInstance(context1 interface{}, ref interface{}, b interface{}) *MockBackend_StopInstance_Call { +func (_e *MockBackend_Expecter) StopInstance(context1 any, ref any, b any) *MockBackend_StopInstance_Call { return &MockBackend_StopInstance_Call{Call: _e.mock.On("StopInstance", context1, ref, b)} } @@ -2149,7 +2301,7 @@ type MockBackend_WaitInstance_Call struct { // WaitInstance is a helper method to define mock.On call // - context1 context.Context // - waitRequest compute.WaitRequest -func (_e *MockBackend_Expecter) WaitInstance(context1 interface{}, waitRequest interface{}) *MockBackend_WaitInstance_Call { +func (_e *MockBackend_Expecter) WaitInstance(context1 any, waitRequest any) *MockBackend_WaitInstance_Call { return &MockBackend_WaitInstance_Call{Call: _e.mock.On("WaitInstance", context1, waitRequest)} } @@ -2215,7 +2367,7 @@ type MockBackend_WriteFile_Call struct { // WriteFile is a helper method to define mock.On call // - context1 context.Context // - fileWriteRequest compute.FileWriteRequest -func (_e *MockBackend_Expecter) WriteFile(context1 interface{}, fileWriteRequest interface{}) *MockBackend_WriteFile_Call { +func (_e *MockBackend_Expecter) WriteFile(context1 any, fileWriteRequest any) *MockBackend_WriteFile_Call { return &MockBackend_WriteFile_Call{Call: _e.mock.On("WriteFile", context1, fileWriteRequest)} } diff --git a/internal/compute/mocks/mock_pending_instance.go b/internal/compute/mocks/mock_pending_instance.go index b787889..548bf5d 100644 --- a/internal/compute/mocks/mock_pending_instance.go +++ b/internal/compute/mocks/mock_pending_instance.go @@ -72,7 +72,7 @@ type MockPendingInstance_Wait_Call struct { // Wait is a helper method to define mock.On call // - context1 context.Context -func (_e *MockPendingInstance_Expecter) Wait(context1 interface{}) *MockPendingInstance_Wait_Call { +func (_e *MockPendingInstance_Expecter) Wait(context1 any) *MockPendingInstance_Wait_Call { return &MockPendingInstance_Wait_Call{Call: _e.mock.On("Wait", context1)} } diff --git a/internal/compute/network.go b/internal/compute/network.go index 5eee933..2b88143 100644 --- a/internal/compute/network.go +++ b/internal/compute/network.go @@ -309,6 +309,25 @@ func (s *Service) CreateForward( return created, nil } +// InstanceForward finds an existing scalar port forward to a guest. +// An empty Address means no matching forward; this method never exposes a port. +func (s *Service) InstanceForward(ctx context.Context, ref Ref, targetPort int64, protocol string) (Forward, error) { + if err := validateRef(ref); err != nil { + return Forward{}, err + } + if targetPort < 1 || targetPort > 65535 || (protocol != "tcp" && protocol != "udp") { + return Forward{}, agentError("forward lookup requires a valid port and tcp or udp protocol") + } + if _, err := s.SandboxExpiry(ctx, ref.Sandbox); err != nil { + return Forward{}, err + } + forward, err := s.backend.InstanceForward(ctx, ref, targetPort, protocol) + if err != nil { + return Forward{}, s.mapBackend(ctx, "find instance forward", err) + } + return forward, nil +} + // ImpairNIC applies Linux-only tc netem settings inside a guest. It is not gated. func (s *Service) ImpairNIC(ctx context.Context, ref Ref, nic string, impairment Impairment) error { if err := validateRef(ref); err != nil { diff --git a/internal/compute/reaper.go b/internal/compute/reaper.go index 381674a..db83d70 100644 --- a/internal/compute/reaper.go +++ b/internal/compute/reaper.go @@ -11,6 +11,9 @@ const reaperInterval = 30 * time.Second // Reap deletes expired sandboxes. A failed delete leaves the project in place // for the next scan. func (s *Service) Reap(ctx context.Context) error { + if s.onReap != nil { + s.onReap() + } boxes, err := s.backend.ListSandboxes(ctx) if err != nil { return s.backendError(ctx, "list sandboxes", err) @@ -68,6 +71,9 @@ func (s *Service) reapSandbox(ctx context.Context, name string) error { if box.ExpiresAt.After(time.Now()) { return nil } + if s.onSandboxExpired != nil { + s.onSandboxExpired(name) + } if err := s.backend.DeleteSandbox(ctx, name); err != nil { if errors.Is(err, ErrNotFound) { return nil diff --git a/internal/compute/service.go b/internal/compute/service.go index cdfbd37..b1861d4 100644 --- a/internal/compute/service.go +++ b/internal/compute/service.go @@ -36,6 +36,12 @@ type Options struct { DefaultNetworkKind string // Logger receives operational logs. Nil selects a no-op logger. Logger *slog.Logger + // OnSandboxExpired purges transient artifacts once a sandbox is expired. + OnSandboxExpired func(string) + // OnReap expires transient artifacts before each backend reaper scan. + OnReap func() + // DesktopReady probes the guest session after Incus agent readiness. + DesktopReady func(context.Context, Ref) (bool, error) } // Service orchestrates sandboxes against a Backend and an immutable catalog. @@ -48,6 +54,9 @@ type Service struct { defaultTTL time.Duration maxTTL time.Duration defaultNetworkKind string + onSandboxExpired func(string) + onReap func() + desktopReady func(context.Context, Ref) (bool, error) } // New constructs a Service. Bridge defaults require Host; zero TTLs select the documented defaults. @@ -92,6 +101,9 @@ func New(backend Backend, catalog *Catalog, opts Options) (*Service, error) { defaultTTL: resolvedDefault, maxTTL: resolvedMax, defaultNetworkKind: kind, + onSandboxExpired: opts.OnSandboxExpired, + onReap: opts.OnReap, + desktopReady: opts.DesktopReady, }, nil } @@ -217,6 +229,9 @@ func (s *Service) DeleteSandbox(ctx context.Context, name string) error { } return s.backendError(ctx, "expire sandbox", err) } + if s.onSandboxExpired != nil { + s.onSandboxExpired(name) + } if err := s.backend.DeleteSandbox(ctx, name); err != nil { return s.backendError(ctx, "delete sandbox", err) } @@ -245,6 +260,12 @@ func (s *Service) CreateInstance(ctx context.Context, req CreateInstance) (Insta } return Instance{}, s.backendError(ctx, "wait instance", err) } + if inst.Desktop && req.Start { + if _, err := s.WaitInstance(createCtx, WaitRequest{Ref: req.Ref, Until: WaitUntilDesktop}); err != nil { + return Instance{}, err + } + return s.GetInstance(createCtx, req.Ref) + } return inst, nil } @@ -303,6 +324,10 @@ func (s *Service) DeleteInstance(ctx context.Context, ref Ref) error { // Exec runs a bounded command without holding the mutation gate. func (s *Service) Exec(ctx context.Context, req ExecRequest) (ExecResult, error) { + return s.exec(ctx, req, execOutputLimit) +} + +func (s *Service) exec(ctx context.Context, req ExecRequest, outputLimit int) (ExecResult, error) { if err := validateRef(req.Ref); err != nil { return ExecResult{}, err } @@ -313,12 +338,12 @@ func (s *Service) Exec(ctx context.Context, req ExecRequest) (ExecResult, error) if err != nil { return ExecResult{}, err } - if inst.Status != statusRunning { + if inst.Status != statusRunning && inst.Status != "Ready" { return ExecResult{}, agentErrorf("instance %q in sandbox %q is not running", req.Ref.Name, req.Ref.Sandbox) } - stdout := newDrainingWriter(execOutputLimit) - stderr := newDrainingWriter(execOutputLimit) + stdout := newDrainingWriter(outputLimit) + stderr := newDrainingWriter(outputLimit) execCtx, cancel := execContext(ctx, req.Timeout) defer cancel() diff --git a/internal/compute/types.go b/internal/compute/types.go index 95f37cb..1da5bd7 100644 --- a/internal/compute/types.go +++ b/internal/compute/types.go @@ -209,11 +209,13 @@ type Backend interface { AddACLRule(context.Context, string, string, ACLRule) (ACLRule, error) RemoveACLRule(context.Context, string, string, string) error CreateForward(context.Context, string, string, Ref, int64, int64, string) (Forward, error) + InstanceForward(context.Context, Ref, int64, string) (Forward, error) StartInstance(context.Context, Ref, bool) (Instance, error) StopInstance(context.Context, Ref, bool) (Instance, error) RestartInstance(context.Context, Ref, bool) (Instance, error) WaitInstance(context.Context, WaitRequest) (WaitResult, error) ReadFile(context.Context, FileReadRequest) (FileReadResult, error) + ReadBinaryFile(context.Context, Ref, string) (io.ReadCloser, error) WriteFile(context.Context, FileWriteRequest) (FileWriteResult, error) CreateSnapshot(context.Context, Ref, string) error RestoreSnapshot(context.Context, Ref, string) error diff --git a/internal/desktop/driver.go b/internal/desktop/driver.go new file mode 100644 index 0000000..ffadf31 --- /dev/null +++ b/internal/desktop/driver.go @@ -0,0 +1,668 @@ +package desktop + +import ( + "bytes" + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "image" + "image/png" + "io" + "net" + "os" + "strconv" + "strings" + "time" + "unicode" + + "golang.org/x/image/draw" + + "github.com/meigma/codemode" + + "github.com/GilmanLab/agentcompute/internal/compute" +) + +const ( + driverBin = "/usr/local/bin/cua-driver" + driverSocket = "/run/user/1000/cua-driver.sock" + driverHome = "/home/automation" + driverRuntime = "/run/user/1000" + driverUID = "1000" + vncPort = "5900" + vncTargetPort = int64(5900) + nativeOKText = "[OK]" + jsonNull = "null" + emptyJSONObject = "{}" + dataImagePrefix = "data:image/" + guestCleanupTimeout = 10 * time.Second + minImagePayloadLen = 512 + imagePayloadSampleLen = 1024 +) + +// Driver proxies native Cua Driver CLI calls inside a Linux desktop guest. +type Driver struct { + compute *compute.Service + store *Store +} + +// Info reports Driver catalog metadata, readiness, and a human VNC endpoint. +type Info struct { + // Ready is true when the guest Driver daemon answers status. + Ready bool + // OS is the catalog operating system family for the instance image. + OS string + // DriverVersion is the version string from guest dump-docs. + DriverVersion string + // Tools are native tool names from guest dump-docs, in dump-docs order. + Tools []string + // VNC is an existing TCP forward to port 5900, or a guest address if none exists. + VNC string +} + +// CallResult is a native tool response with screenshots reduced to URLs. +type CallResult struct { + // OK is true only for structured JSON or native [OK] text. It does not mean an effect occurred. + OK bool + // Summary is a short classification or the exact native diagnostic. + Summary string + // Result is stripped native structured content as a JSON string. + Result string + // ScreenshotURL is a published PNG URL when the guest wrote a screenshot file. + ScreenshotURL string +} + +// NewDriver returns a Driver that execs through compute and publishes PNGs to store. +func NewDriver(service *compute.Service, store *Store) *Driver { + return &Driver{compute: service, store: store} +} + +// Info discovers dump-docs, probes daemon status, and reports an existing VNC endpoint. +func (d *Driver) Info(ctx context.Context, ref compute.Ref) (Info, error) { + inst, err := d.compute.GetInstance(ctx, ref) + if err != nil { + return Info{}, err + } + if !strings.EqualFold(inst.Status, "Running") && !strings.EqualFold(inst.Status, "Ready") { + osName, osErr := d.imageOS(ctx, ref, inst.Image) + return Info{OS: osName, Tools: []string{}}, osErr + } + version, tools, err := d.discoverDocs(ctx, ref) + if err != nil { + return Info{}, err + } + ready, err := d.Ready(ctx, ref) + if err != nil { + return Info{}, err + } + osName, err := d.imageOS(ctx, ref, inst.Image) + if err != nil { + return Info{}, err + } + vnc, err := d.vncEndpoint(ctx, ref, inst) + if err != nil { + return Info{}, err + } + return Info{ + Ready: ready, + OS: osName, + DriverVersion: version, + Tools: tools, + VNC: vnc, + }, nil +} + +// Enable discovers dump-docs then reports whether the guest daemon answers. +func (d *Driver) Enable(ctx context.Context, ref compute.Ref) (bool, error) { + if _, _, err := d.discoverDocs(ctx, ref); err != nil { + return false, err + } + return d.Ready(ctx, ref) +} + +// Ready reports whether the guest Driver daemon answers status. +// +// A stopped or absent daemon returns false and a nil error so waiters can poll. +// Missing or expired sandboxes, missing instances, and context cancellation +// are returned as errors. +func (d *Driver) Ready(ctx context.Context, ref compute.Ref) (bool, error) { + if err := ctx.Err(); err != nil { + return false, err + } + if _, err := d.compute.SandboxExpiry(ctx, ref.Sandbox); err != nil { + return false, err + } + inst, err := d.compute.GetInstance(ctx, ref) + if err != nil { + return false, err + } + if !strings.EqualFold(inst.Status, "Running") && !strings.EqualFold(inst.Status, "Ready") { + return false, nil + } + result, err := d.execDriver(ctx, ref, []string{driverBin, "status", "--socket", driverSocket}) + if err != nil { + return false, err + } + if result.TimedOut { + return false, nil + } + return result.ExitCode == 0, nil +} + +// Call invokes a native Driver tool with a JSON object payload. +func (d *Driver) Call(ctx context.Context, ref compute.Ref, tool, args string) (CallResult, error) { + if strings.TrimSpace(tool) == "" || strings.HasPrefix(tool, "-") { + return CallResult{}, agentError("tool must be a native Driver tool name, not a CLI option") + } + payload, err := normalizeArgs(args) + if err != nil { + return CallResult{}, err + } + path, err := randomGuestPNG() + if err != nil { + return CallResult{}, err + } + missing := false + defer func() { + if !missing { + d.removeGuestFile(ctx, ref, path) + } + }() + + result, err := d.execDriver(ctx, ref, callArgv(tool, payload, path)) + if err != nil { + return CallResult{}, err + } + if result.TimedOut { + return CallResult{}, agentError("Driver call timed out") + } + classified := classifyCall(result.ExitCode, result.Stdout, result.Stderr) + shot, shotErr := d.pullScreenshot(ctx, ref, path, 0) + switch { + case shotErr == nil: + classified.ScreenshotURL = shot.URL + case errors.Is(shotErr, os.ErrNotExist): + missing = true + case classified.OK: + return classified, shotErr + } + return classified, nil +} + +// Screenshot captures the desktop or a pid/window_id pair without an accessibility tree. +// +// Zero pid and window_id select get_desktop_state. get_desktop_state has no +// max_dimension input, so requested bounds are applied by host-side resize +// only when the guest PNG's long edge exceeds max_dimension. Window captures +// pass max_dimension through to get_window_state and still honor the bound +// after the file is pulled. Scale maps returned pixels back to the native +// window or screen coordinate width, including any resize performed in-guest. +func (d *Driver) Screenshot( + ctx context.Context, + ref compute.Ref, + pid, windowID, maxDimension int64, +) (Screenshot, error) { + if err := validateScreenshotArgs(pid, windowID, maxDimension); err != nil { + return Screenshot{}, err + } + tool := "get_desktop_state" + payload := emptyJSONObject + if pid != 0 { + tool = "get_window_state" + body := map[string]any{ + "pid": pid, + "window_id": windowID, + "include_accessibility_tree": false, + } + if maxDimension > 0 { + body["max_dimension"] = maxDimension + } + encoded, err := marshalJSON(body) + if err != nil { + return Screenshot{}, err + } + payload = encoded + } + path, err := randomGuestPNG() + if err != nil { + return Screenshot{}, err + } + defer d.removeGuestFile(ctx, ref, path) + + result, err := d.execDriver(ctx, ref, callArgv(tool, payload, path)) + if err != nil { + return Screenshot{}, err + } + if result.TimedOut { + return Screenshot{}, agentError("Driver screenshot timed out") + } + classified := classifyCall(result.ExitCode, result.Stdout, result.Stderr) + if !classified.OK { + if classified.Summary == "" { + return Screenshot{}, agentError("Driver screenshot failed") + } + return Screenshot{}, agentError(classified.Summary) + } + var dimensions struct { + ScreenWidth float64 `json:"screen_width"` + WindowBounds struct { + Width float64 `json:"width"` + } `json:"window_bounds"` + } + if err = json.Unmarshal([]byte(classified.Result), &dimensions); err != nil { + return Screenshot{}, agentError("Driver screenshot metadata is not JSON") + } + originalWidth := dimensions.ScreenWidth + if pid != 0 { + originalWidth = dimensions.WindowBounds.Width + } + if originalWidth <= 0 { + return Screenshot{}, agentError("Driver screenshot omitted its coordinate width") + } + shot, err := d.pullScreenshot(ctx, ref, path, maxDimension) + if errors.Is(err, os.ErrNotExist) { + return Screenshot{}, agentError("Driver did not write a screenshot") + } + if err != nil { + return Screenshot{}, err + } + shot.Scale = originalWidth / float64(shot.Width) + return shot, nil +} + +func (d *Driver) discoverDocs(ctx context.Context, ref compute.Ref) (string, []string, error) { + result, err := d.execDriver(ctx, ref, []string{driverBin, "dump-docs", "--type", "mcp"}) + if err != nil { + return "", nil, err + } + if result.TimedOut { + return "", nil, agentError("Driver dump-docs timed out") + } + if result.ExitCode != 0 { + return "", nil, agentError(nonzeroDiagnostic(result.ExitCode, result.Stdout, result.Stderr, "Driver dump-docs")) + } + return parseDumpDocs([]byte(result.Stdout)) +} + +func (d *Driver) imageOS(ctx context.Context, ref compute.Ref, imageName string) (string, error) { + image, err := d.compute.ResolveImage(ctx, ref.Sandbox, imageName) + if err != nil { + if imageMissing(err) { + return "", nil + } + return "", err + } + return image.OS, nil +} + +func (d *Driver) vncEndpoint(ctx context.Context, ref compute.Ref, inst compute.Instance) (string, error) { + fwd, err := d.compute.InstanceForward(ctx, ref, vncTargetPort, "tcp") + if err != nil { + return "", err + } + if fwd.Address != "" { + port := fwd.Port + if port <= 0 { + port = vncTargetPort + } + return net.JoinHostPort(fwd.Address, strconv.FormatInt(port, 10)), nil + } + for _, nic := range inst.NICs { + for _, addr := range nic.Addresses { + if addr != "" { + return net.JoinHostPort(addr, vncPort), nil + } + } + } + return "", nil +} + +func (d *Driver) execDriver(ctx context.Context, ref compute.Ref, argv []string) (compute.ExecResult, error) { + if _, err := d.compute.SandboxExpiry(ctx, ref.Sandbox); err != nil { + return compute.ExecResult{}, err + } + return d.compute.ExecJSON(ctx, compute.ExecRequest{ + Ref: ref, + Argv: argv, + User: driverUID, + Cwd: driverHome, + Env: driverEnv(), + }) +} + +func (d *Driver) removeGuestFile(ctx context.Context, ref compute.Ref, path string) { + if path == "" { + return + } + rmCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), guestCleanupTimeout) + defer cancel() + _, _ = d.compute.ExecJSON(rmCtx, compute.ExecRequest{ + Ref: ref, + Argv: []string{"/bin/rm", "-f", "--", path}, + User: driverUID, + Cwd: driverHome, + Env: driverEnv(), + }) +} + +func (d *Driver) pullScreenshot( + ctx context.Context, + ref compute.Ref, + path string, + maxDimension int64, +) (Screenshot, error) { + body, err := d.compute.ReadBinaryFile(ctx, ref, path) + if err != nil { + return Screenshot{}, err + } + defer body.Close() + raw, err := io.ReadAll(io.LimitReader(body, MaxImageBytes+1)) + if err != nil { + return Screenshot{}, fmt.Errorf("read screenshot: %w", err) + } + if int64(len(raw)) > MaxImageBytes { + return Screenshot{}, agentError("screenshot exceeds 16 MiB") + } + expiry, err := d.compute.SandboxExpiry(ctx, ref.Sandbox) + if err != nil { + return Screenshot{}, err + } + if maxDimension <= 0 { + return d.store.Publish(ref.Sandbox, expiry, bytes.NewReader(raw)) + } + return publishBoundedPNG(d.store, ref.Sandbox, expiry, raw, maxDimension) +} + +func callArgv(tool, payload, screenshotPath string) []string { + return []string{ + driverBin, "call", + "--socket", driverSocket, + "--screenshot-out-file", screenshotPath, + tool, payload, + } +} + +func driverEnv() map[string]string { + return map[string]string{ + "HOME": driverHome, + "XDG_RUNTIME_DIR": driverRuntime, + } +} + +func randomGuestPNG() (string, error) { + var token [16]byte + if _, err := rand.Read(token[:]); err != nil { + return "", fmt.Errorf("generate screenshot path: %w", err) + } + return "/tmp/cua-" + hex.EncodeToString(token[:]) + ".png", nil +} + +func validateScreenshotArgs(pid, windowID, maxDimension int64) error { + if pid < 0 || windowID < 0 { + return agentError("pid and window_id must be non-negative") + } + if (pid == 0) != (windowID == 0) { + return agentError("pid and window_id must both be set or both omitted") + } + if maxDimension < 0 { + return agentError("max_dimension must be a positive integer") + } + return nil +} + +func normalizeArgs(args string) (string, error) { + trimmed := strings.TrimSpace(args) + if trimmed == "" { + return emptyJSONObject, nil + } + if trimmed[0] != '{' || !json.Valid([]byte(trimmed)) { + return "", agentError("args must be a JSON object") + } + return trimmed, nil +} + +func parseDumpDocs(raw []byte) (string, []string, error) { + raw = bytes.TrimSpace(raw) + var docs struct { + Version string `json:"version"` + Tools []struct { + Name string `json:"name"` + } `json:"tools"` + } + if err := json.Unmarshal(raw, &docs); err != nil { + return "", nil, agentError("Driver dump-docs did not return JSON") + } + tools := make([]string, 0, len(docs.Tools)) + for _, tool := range docs.Tools { + if tool.Name != "" { + tools = append(tools, tool.Name) + } + } + if docs.Version == "" || len(tools) == 0 { + return "", nil, agentError("Driver dump-docs must include version and tools") + } + return docs.Version, tools, nil +} + +// classifyCall interprets one-shot CLI stdout. Native run_call drops MCP isError +// and prints structuredContent when present, otherwise text, so exit 0 is not an +// effect claim. Ambiguous non-JSON text fails closed except native [OK]. +func classifyCall(exitCode int64, stdout, stderr string) CallResult { + out := strings.TrimSpace(stdout) + errText := strings.TrimSpace(stderr) + if exitCode != 0 { + summary := errText + if summary == "" { + summary = out + } + return CallResult{OK: false, Summary: summary, Result: jsonNull} + } + if out == "" { + return CallResult{OK: false, Summary: errText, Result: jsonNull} + } + if isNativeOK(out) { + return CallResult{OK: true, Summary: out, Result: jsonNull} + } + decoder := json.NewDecoder(strings.NewReader(out)) + decoder.UseNumber() + var value any + if err := decoder.Decode(&value); err != nil { + return CallResult{OK: false, Summary: out, Result: jsonNull} + } + if err := decoder.Decode(&struct{}{}); err != io.EOF { + return CallResult{OK: false, Summary: out, Result: jsonNull} + } + switch value.(type) { + case map[string]any, []any: + encoded, err := marshalJSON(stripImages(value)) + if err != nil { + return CallResult{OK: false, Summary: out, Result: jsonNull} + } + return CallResult{OK: true, Summary: "structured content", Result: encoded} + default: + return CallResult{OK: false, Summary: out, Result: jsonNull} + } +} + +func isNativeOK(text string) bool { + if text == nativeOKText { + return true + } + if strings.HasPrefix(text, nativeOKText) && len(text) > len(nativeOKText) { + return unicode.IsSpace(rune(text[len(nativeOKText)])) + } + return false +} + +func stripImages(v any) any { + switch t := v.(type) { + case map[string]any: + stripImageMap(t) + return t + case []any: + for i, child := range t { + t[i] = stripImages(child) + } + return t + case string: + if strings.HasPrefix(t, dataImagePrefix) { + return "" + } + return t + default: + return v + } +} + +func stripImageMap(m map[string]any) { + if typeName, _ := m["type"].(string); strings.EqualFold(typeName, "image") { + delete(m, "data") + if src, ok := m["source"].(map[string]any); ok { + delete(src, "data") + } + } + for key, child := range m { + if shouldStripImageValue(key, child) { + delete(m, key) + continue + } + m[key] = stripImages(child) + } +} + +func shouldStripImageValue(key string, child any) bool { + s, ok := child.(string) + if !ok { + return false + } + return strings.HasPrefix(s, dataImagePrefix) || (stripImageKey(key) && looksLikeImagePayload(s)) +} + +func stripImageKey(key string) bool { + switch strings.ToLower(key) { + case "screenshot", "image", "image_data", pngFormat, "png_base64", + "jpeg", "jpeg_base64", "screenshot_png_b64", "screenshot_base64", + "image_base64", "screenshot_data": + return true + default: + return false + } +} + +func looksLikeImagePayload(s string) bool { + if strings.HasPrefix(s, dataImagePrefix) { + return true + } + if len(s) < minImagePayloadLen { + return false + } + sample := s + if len(sample) > imagePayloadSampleLen { + sample = sample[:imagePayloadSampleLen] + } + for _, r := range sample { + if unicode.IsLetter(r) || unicode.IsDigit(r) || r == '+' || r == '/' || r == '=' || r == '\n' || r == '\r' { + continue + } + return false + } + return true +} + +func publishBoundedPNG( + store *Store, + sandbox string, + expiry time.Time, + raw []byte, + maxDimension int64, +) (Screenshot, error) { + config, format, err := image.DecodeConfig(bytes.NewReader(raw)) + if err != nil || format != pngFormat { + return store.Publish(sandbox, expiry, bytes.NewReader(raw)) + } + if config.Width <= 0 || config.Height <= 0 || int64(config.Width) > maxImagePixels/int64(config.Height) { + return Screenshot{}, agentError("screenshot exceeds pixel bounds") + } + if maxDimension <= 0 || int64(max(config.Width, config.Height)) <= maxDimension { + return store.Publish(sandbox, expiry, bytes.NewReader(raw)) + } + src, _, err := image.Decode(bytes.NewReader(raw)) + if err != nil { + return store.Publish(sandbox, expiry, bytes.NewReader(raw)) + } + origW := src.Bounds().Dx() + bounded := boundImage(src, int(maxDimension)) + payload := raw + if bounded != src { + var buf bytes.Buffer + if err = png.Encode(&buf, bounded); err != nil { + return Screenshot{}, agentError("encode screenshot PNG") + } + payload = buf.Bytes() + } + shot, err := store.Publish(sandbox, expiry, bytes.NewReader(payload)) + if err != nil { + return Screenshot{}, err + } + if shot.Width > 0 && origW > 0 { + shot.Scale = float64(origW) / float64(shot.Width) + } + return shot, nil +} + +func boundImage(src image.Image, maxDimension int) image.Image { + b := src.Bounds() + w, h := boundsForDimension(b.Dx(), b.Dy(), maxDimension) + if w == b.Dx() && h == b.Dy() { + return src + } + dst := image.NewNRGBA(image.Rect(0, 0, w, h)) + draw.CatmullRom.Scale(dst, dst.Bounds(), src, b, draw.Src, nil) + return dst +} + +func boundsForDimension(width, height, maxDimension int) (int, int) { + if maxDimension <= 0 || width <= 0 || height <= 0 { + return width, height + } + long := max(width, height) + if long <= maxDimension { + return width, height + } + return max(width*maxDimension/long, 1), max(height*maxDimension/long, 1) +} + +func marshalJSON(v any) (string, error) { + var buf bytes.Buffer + enc := json.NewEncoder(&buf) + enc.SetEscapeHTML(false) + if err := enc.Encode(v); err != nil { + return "", err + } + return strings.TrimSpace(buf.String()), nil +} + +func nonzeroDiagnostic(code int64, stdout, stderr, op string) string { + summary := strings.TrimSpace(stderr) + if summary == "" { + summary = strings.TrimSpace(stdout) + } + if summary == "" { + return fmt.Sprintf("%s exited %d", op, code) + } + return summary +} + +func imageMissing(err error) bool { + var agent *codemode.AgentError + if !errors.As(err, &agent) || agent == nil { + return false + } + return strings.HasPrefix(agent.Message, "image ") && strings.HasSuffix(agent.Message, " not found") +} + +func agentError(message string) error { + return &codemode.AgentError{Message: message} +} diff --git a/internal/desktop/driver_test.go b/internal/desktop/driver_test.go new file mode 100644 index 0000000..fdf9d49 --- /dev/null +++ b/internal/desktop/driver_test.go @@ -0,0 +1,292 @@ +package desktop + +import ( + "bytes" + "encoding/json" + "image" + "image/png" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/meigma/codemode" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func fixture(t *testing.T, name string) string { + t.Helper() + data, err := os.ReadFile(filepath.Join("testdata", name)) + require.NoError(t, err) + return string(data) +} + +func TestClassifyCallPinnedFixtures(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + file string + exitCode int64 + wantOK bool + wantExact string + assertResult func(t *testing.T, got CallResult) + }{ + { + name: "exit-zero plain-text missing pid fails closed", + file: "click-missing-pid.stdout", + exitCode: 0, + wantOK: false, + wantExact: "Missing required integer field: pid", + }, + { + name: "valid click preserves native fields without claiming effect", + file: "click-with-pid.stdout", + exitCode: 0, + wantOK: true, + assertResult: func(t *testing.T, got CallResult) { + t.Helper() + var body map[string]any + require.NoError(t, json.Unmarshal([]byte(got.Result), &body)) + assert.Equal(t, "unverifiable", body["effect"]) + assert.Equal(t, "accessibility", body["route"]) + delivery, ok := body["delivery"].(map[string]any) + require.True(t, ok) + assert.Equal(t, "background", delivery["mode"]) + }, + }, + { + name: "snapshot preserves element tokens", + file: "snapshot-tree.stdout", + exitCode: 0, + wantOK: true, + assertResult: func(t *testing.T, got CallResult) { + t.Helper() + var body map[string]any + require.NoError(t, json.Unmarshal([]byte(got.Result), &body)) + assert.Equal(t, "s00000001", body["snapshot_id"]) + elements, ok := body["elements"].([]any) + require.True(t, ok) + require.NotEmpty(t, elements) + first, ok := elements[0].(map[string]any) + require.True(t, ok) + assert.Equal(t, "s00000001:0", first["element_token"]) + var token string + for _, raw := range elements { + el, ok := raw.(map[string]any) + require.True(t, ok) + if el["label"] == "New tab" { + token, _ = el["element_token"].(string) + } + } + assert.Equal(t, "s00000001:5", token) + }, + }, + { + name: "desktop state is structured content", + file: "desktop.stdout", + exitCode: 0, + wantOK: true, + assertResult: func(t *testing.T, got CallResult) { + t.Helper() + var body map[string]any + require.NoError(t, json.Unmarshal([]byte(got.Result), &body)) + assert.Equal(t, "linux", body["platform"]) + assert.EqualValues(t, 1280, body["screen_width"]) + assert.EqualValues(t, 800, body["screen_height"]) + assert.NotContains(t, got.Result, "screenshot_png_b64") + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got := classifyCall(tt.exitCode, fixture(t, tt.file), "") + assert.Equal(t, tt.wantOK, got.OK) + if tt.wantExact != "" { + assert.Equal(t, tt.wantExact, got.Summary) + assert.JSONEq(t, "null", got.Result) + } + if tt.assertResult != nil { + tt.assertResult(t, got) + } + }) + } +} + +func TestClassifyCallPreservesIntegerPrecision(t *testing.T) { + t.Parallel() + got := classifyCall(0, `{"window_id":18446744073709551615}`, "") + require.True(t, got.OK) + var result struct { + WindowID uint64 `json:"window_id"` + } + require.NoError(t, json.Unmarshal([]byte(got.Result), &result)) + assert.Equal(t, ^uint64(0), result.WindowID) +} + +func TestClassifyCallSafetyAndAmbiguity(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + exitCode int64 + stdout string + stderr string + wantOK bool + wantSum string + }{ + { + name: "native [OK] text is success", + exitCode: 0, + stdout: "[OK]\n", + wantOK: true, + wantSum: "[OK]", + }, + { + name: "native [OK] with trailing text is success", + exitCode: 0, + stdout: "[OK] done", + wantOK: true, + wantSum: "[OK] done", + }, + { + name: "ambiguous non-JSON text fails closed with exact diagnostic", + exitCode: 0, + stdout: "something went wrong maybe", + wantOK: false, + wantSum: "something went wrong maybe", + }, + { + name: "JSON string is not structured content", + exitCode: 0, + stdout: `"Missing required integer field: pid"`, + wantOK: false, + wantSum: `"Missing required integer field: pid"`, + }, + { + name: "nonzero exit prefers stderr diagnostic", + exitCode: 1, + stdout: "ignored", + stderr: "Cua Driver daemon is not running on /run/user/1000/cua-driver.sock.", + wantOK: false, + wantSum: "Cua Driver daemon is not running on /run/user/1000/cua-driver.sock.", + }, + { + name: "empty exit-zero output fails closed", + exitCode: 0, + wantOK: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + got := classifyCall(tt.exitCode, tt.stdout, tt.stderr) + assert.Equal(t, tt.wantOK, got.OK) + assert.Equal(t, tt.wantSum, got.Summary) + if !tt.wantOK { + assert.JSONEq(t, "null", got.Result) + } + }) + } +} + +func TestStripImagesRemovesKnownBase64Fields(t *testing.T) { + t.Parallel() + + payload := strings.Repeat("A", 600) + raw := map[string]any{ + "element_token": "s00000001:5", + "snapshot_id": "s00000001", + "screenshot_png_b64": payload, + "screenshot_width": float64(822), + "content": []any{ + map[string]any{"type": "image", "data": payload, "mimeType": "image/png"}, + map[string]any{"type": "text", "text": "hello"}, + }, + "note": "data:image/png;base64," + payload, + } + + got := stripImages(raw).(map[string]any) + assert.Equal(t, "s00000001:5", got["element_token"]) + assert.Equal(t, "s00000001", got["snapshot_id"]) + assert.EqualValues(t, 822, got["screenshot_width"]) + assert.NotContains(t, got, "screenshot_png_b64") + content := got["content"].([]any) + imageBlock := content[0].(map[string]any) + assert.NotContains(t, imageBlock, "data") + assert.Equal(t, "image/png", imageBlock["mimeType"]) + assert.NotContains(t, got, "note") + + encoded, err := marshalJSON(got) + require.NoError(t, err) + assert.NotContains(t, encoded, payload) + assert.NotContains(t, encoded, "screenshot_png_b64") +} + +func TestParseDumpDocsUsesPinnedCatalog(t *testing.T) { + t.Parallel() + + version, tools, err := parseDumpDocs([]byte(fixture(t, "dump-docs.json"))) + require.NoError(t, err) + assert.Equal(t, "0.28.1", version) + assert.Contains(t, tools, "get_desktop_state") + assert.Contains(t, tools, "get_window_state") + assert.Contains(t, tools, "click") + assert.Contains(t, tools, "list_apps") + assert.Equal(t, "list_apps", tools[0]) + assert.NotContains(t, tools, "") +} + +func TestInvalidDesktopArguments(t *testing.T) { + t.Parallel() + + for _, args := range []string{`["click"]`, "not-json"} { + _, err := normalizeArgs(args) + var agent *codemode.AgentError + require.ErrorAs(t, err, &agent) + } + for _, args := range []struct{ pid, windowID, maxDimension int64 }{ + {1220, 0, 0}, + {0, 31457284, 0}, + {0, 0, -1}, + } { + var agent *codemode.AgentError + require.ErrorAs(t, validateScreenshotArgs(args.pid, args.windowID, args.maxDimension), &agent) + } +} + +func TestBoundImageHonorsMaxDimension(t *testing.T) { + t.Parallel() + + src := image.NewNRGBA(image.Rect(0, 0, 1280, 800)) + unchanged := boundImage(src, 0) + assert.Equal(t, src, unchanged) + unchanged = boundImage(src, 1280) + assert.Equal(t, src, unchanged) + + got := boundImage(src, 640) + assert.Equal(t, 640, got.Bounds().Dx()) + assert.Equal(t, 400, got.Bounds().Dy()) +} + +func TestPublishBoundedPNGSetsCoordinateScale(t *testing.T) { + t.Parallel() + + store, err := NewStore(t.TempDir(), "https://shots.example") + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, store.Close()) }) + + var buf bytes.Buffer + require.NoError(t, png.Encode(&buf, image.NewNRGBA(image.Rect(0, 0, 100, 50)))) + shot, err := publishBoundedPNG(store, "demo", time.Now().Add(time.Hour), buf.Bytes(), 40) + require.NoError(t, err) + assert.Equal(t, 40, shot.Width) + assert.Equal(t, 20, shot.Height) + assert.InDelta(t, 2.5, shot.Scale, 1e-9) + assert.NotEmpty(t, shot.URL) +} diff --git a/internal/desktop/store.go b/internal/desktop/store.go new file mode 100644 index 0000000..6dacee7 --- /dev/null +++ b/internal/desktop/store.go @@ -0,0 +1,244 @@ +// Package desktop proxies the guest Driver and serves transient screenshots. +package desktop + +import ( + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "image" + _ "image/png" // Register the only supported screenshot format. + "io" + "net/http" + "net/url" + "os" + "path/filepath" + "strings" + "sync" + "syscall" + "time" +) + +const ( + // MaxImageBytes bounds a single encoded screenshot. + MaxImageBytes = 16 << 20 + maxStoreBytes = 128 << 20 + maxImagePixels = 32 << 20 + screenshotRetention = 5 * time.Minute + pngFormat = "png" + // ScreenshotPath is the HTTP route shared by both transports. + ScreenshotPath = "/screenshots/" +) + +// Screenshot describes a published PNG without carrying its bytes. +type Screenshot struct { + // URL is built only from the configured public base URL. + URL string + // Width is the encoded image width in pixels. + Width int + // Height is the encoded image height in pixels. + Height int + // Scale maps image pixels to the Driver's coordinate frame. + Scale float64 +} + +type storedImage struct { + path string + sandbox string + expires time.Time + bytes int64 +} + +// Store keeps bounded PNG files and an ephemeral in-memory index. +type Store struct { + mu sync.Mutex + dir string + lock *os.File + baseURL string + images map[string]storedImage + bytes int64 + closed bool + now func() time.Time +} + +// NewStore exclusively owns and clears a dedicated scratch child of dir. +// Unrelated contents of dir are preserved; another live store is refused. +func NewStore(dir, baseURL string) (*Store, error) { + u, err := url.Parse(baseURL) + if err != nil || u.Host == "" || (u.Scheme != "http" && u.Scheme != "https") || u.User != nil || u.RawQuery != "" || + u.Fragment != "" { + return nil, errors.New( + "screenshots.base_url must be an absolute HTTP or HTTPS URL without credentials, query, or fragment", + ) + } + if dir == "" { + return nil, errors.New("screenshots.dir is required") + } + if err = os.MkdirAll(dir, 0o700); err != nil { + return nil, fmt.Errorf("create screenshot directory: %w", err) + } + lock, err := os.OpenFile(filepath.Join(dir, ".agentcompute-screenshots.lock"), os.O_CREATE|os.O_RDWR, 0o600) + if err != nil { + return nil, fmt.Errorf("open screenshot lock: %w", err) + } + if err = syscall.Flock(int(lock.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil { + _ = lock.Close() + return nil, fmt.Errorf("screenshot directory is already in use: %w", err) + } + scratch := filepath.Join(dir, ".agentcompute-screenshots") + if err = os.RemoveAll(scratch); err != nil { + _ = lock.Close() + return nil, fmt.Errorf("clear screenshot scratch: %w", err) + } + if err = os.Mkdir(scratch, 0o700); err != nil { + _ = lock.Close() + return nil, fmt.Errorf("create screenshot scratch: %w", err) + } + return &Store{ + dir: scratch, + lock: lock, + baseURL: strings.TrimRight(baseURL, "/"), + images: make(map[string]storedImage), + now: time.Now, + }, nil +} + +// Publish stores one PNG until five minutes or sandbox expiry, whichever is sooner. +// Oversized images and exhausted capacity fail without evicting live screenshots. +func (s *Store) Publish(sandbox string, sandboxExpiry time.Time, source io.Reader) (Screenshot, error) { + s.mu.Lock() + defer s.mu.Unlock() + if s.closed { + return Screenshot{}, errors.New("screenshot store is closed") + } + now := s.now() + s.expireLocked(now) + expires := now.Add(screenshotRetention) + if sandboxExpiry.Before(expires) { + expires = sandboxExpiry + } + if !expires.After(now) { + return Screenshot{}, errors.New("sandbox expired before screenshot publication") + } + file, err := os.CreateTemp(s.dir, ".pending-") + if err != nil { + return Screenshot{}, fmt.Errorf("create screenshot: %w", err) + } + pending := file.Name() + defer func() { _ = file.Close(); _ = os.Remove(pending) }() + n, err := io.Copy(file, io.LimitReader(source, MaxImageBytes+1)) + if err != nil { + return Screenshot{}, fmt.Errorf("copy screenshot: %w", err) + } + if n > MaxImageBytes { + return Screenshot{}, errors.New("screenshot exceeds 16 MiB") + } + if n > maxStoreBytes-s.bytes { + return Screenshot{}, errors.New("screenshot store exceeds 128 MiB") + } + if _, err = file.Seek(0, io.SeekStart); err != nil { + return Screenshot{}, fmt.Errorf("rewind screenshot: %w", err) + } + config, format, err := image.DecodeConfig(file) + if err != nil || format != pngFormat { + return Screenshot{}, errors.New("screenshot is not a valid PNG header") + } + if config.Width <= 0 || config.Height <= 0 || int64(config.Width) > maxImagePixels/int64(config.Height) { + return Screenshot{}, errors.New("screenshot exceeds pixel bounds") + } + if err = file.Close(); err != nil { + return Screenshot{}, fmt.Errorf("close screenshot: %w", err) + } + var token [16]byte + if _, err = rand.Read(token[:]); err != nil { + return Screenshot{}, fmt.Errorf("generate screenshot identifier: %w", err) + } + id := hex.EncodeToString(token[:]) + path := filepath.Join(s.dir, id) + if err = os.Rename(pending, path); err != nil { + return Screenshot{}, fmt.Errorf("publish screenshot: %w", err) + } + s.images[id] = storedImage{path: path, sandbox: sandbox, expires: expires, bytes: n} + s.bytes += n + return Screenshot{URL: s.baseURL + ScreenshotPath + id, Width: config.Width, Height: config.Height, Scale: 1}, nil +} + +// PurgeSandbox removes every screenshot owned by the named sandbox. +func (s *Store) PurgeSandbox(sandbox string) { + s.mu.Lock() + defer s.mu.Unlock() + for id, entry := range s.images { + if entry.sandbox == sandbox { + s.removeLocked(id, entry) + } + } +} + +// Sweep removes expired screenshots and releases their capacity. +func (s *Store) Sweep() { + s.mu.Lock() + defer s.mu.Unlock() + s.expireLocked(s.now()) +} + +func (s *Store) expireLocked(now time.Time) { + for id, entry := range s.images { + if !entry.expires.After(now) { + s.removeLocked(id, entry) + } + } +} + +func (s *Store) removeLocked(id string, entry storedImage) { + _ = os.Remove(entry.path) + delete(s.images, id) + s.bytes -= entry.bytes +} + +// ServeHTTP serves GET and HEAD only; screenshot identifiers are bearer URLs. +func (s *Store) ServeHTTP(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Content-Type-Options", "nosniff") + w.Header().Set("Cache-Control", "no-store") + if r.Method != http.MethodGet && r.Method != http.MethodHead { + w.Header().Set("Allow", "GET, HEAD") + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + id := strings.TrimPrefix(r.URL.Path, ScreenshotPath) + if !strings.HasPrefix(r.URL.Path, ScreenshotPath) || len(id) != 32 { + http.NotFound(w, r) + return + } + if _, err := hex.DecodeString(id); err != nil { + http.NotFound(w, r) + return + } + s.mu.Lock() + s.expireLocked(s.now()) + entry, exists := s.images[id] + var file *os.File + if exists && !s.closed { + file, _ = os.Open(entry.path) + } + s.mu.Unlock() + if file == nil { + http.NotFound(w, r) + return + } + defer file.Close() + w.Header().Set("Content-Type", "image/png") + http.ServeContent(w, r, "screenshot.png", time.Time{}, file) +} + +// Close drops the ephemeral index and removes this store's scratch directory. +func (s *Store) Close() error { + s.mu.Lock() + defer s.mu.Unlock() + if s.closed { + return nil + } + s.closed = true + clear(s.images) + s.bytes = 0 + return errors.Join(os.RemoveAll(s.dir), s.lock.Close()) +} diff --git a/internal/desktop/store_test.go b/internal/desktop/store_test.go new file mode 100644 index 0000000..63cde71 --- /dev/null +++ b/internal/desktop/store_test.go @@ -0,0 +1,156 @@ +package desktop + +import ( + "bytes" + "image" + "image/png" + "io" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func testPNG(t *testing.T) []byte { + t.Helper() + var data bytes.Buffer + require.NoError(t, png.Encode(&data, image.NewNRGBA(image.Rect(0, 0, 24, 16)))) + return data.Bytes() +} + +func testStore(t *testing.T) (*Store, *time.Time) { + t.Helper() + store, err := NewStore(t.TempDir(), "https://shots.example") + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, store.Close()) }) + now := time.Now() + store.now = func() time.Time { return now } + return store, &now +} + +func requestShot(t *testing.T, store *Store, method, target string) *httptest.ResponseRecorder { + t.Helper() + u, err := url.Parse(target) + require.NoError(t, err) + r := httptest.NewRequest(method, u.RequestURI(), nil) + r.Host = "attacker.example" + w := httptest.NewRecorder() + store.ServeHTTP(w, r) + return w +} + +func TestScreenshotHTTP(t *testing.T) { + store, now := testStore(t) + data := testPNG(t) + shot, err := store.Publish("sandbox", now.Add(time.Hour), bytes.NewReader(data)) + require.NoError(t, err) + assert.Equal(t, 24, shot.Width) + assert.Equal(t, 16, shot.Height) + assert.Regexp(t, `^https://shots\.example/screenshots/[a-f0-9]{32}$`, shot.URL) + for _, method := range []string{http.MethodGet, http.MethodHead, http.MethodPost, http.MethodDelete, http.MethodPut} { + t.Run(method, func(t *testing.T) { + w := requestShot(t, store, method, shot.URL) + assert.Equal(t, "nosniff", w.Header().Get("X-Content-Type-Options")) + assert.Equal(t, "no-store", w.Header().Get("Cache-Control")) + if method != http.MethodGet && method != http.MethodHead { + assert.Equal(t, http.StatusMethodNotAllowed, w.Code) + assert.Equal(t, "GET, HEAD", w.Header().Get("Allow")) + return + } + assert.Equal(t, http.StatusOK, w.Code) + assert.Equal(t, "image/png", w.Header().Get("Content-Type")) + if method == http.MethodHead { + assert.Empty(t, w.Body.Bytes()) + } else { + assert.Equal(t, data, w.Body.Bytes()) + _, err := png.Decode(w.Body) + require.NoError(t, err) + } + }) + } +} + +func TestScreenshotExpiryAndSandboxPurge(t *testing.T) { + for _, tc := range []struct { + name string + ttl time.Duration + advance time.Duration + }{ + {"retention", time.Hour, 5 * time.Minute}, + {"sandbox expiry", time.Minute, time.Minute}, + } { + t.Run(tc.name, func(t *testing.T) { + store, now := testStore(t) + shot, err := store.Publish("sandbox", now.Add(tc.ttl), bytes.NewReader(testPNG(t))) + require.NoError(t, err) + *now = now.Add(tc.advance) + store.Sweep() + assert.Equal(t, http.StatusNotFound, requestShot(t, store, http.MethodGet, shot.URL).Code) + }) + } + store, now := testStore(t) + first, err := store.Publish("one", now.Add(time.Hour), bytes.NewReader(testPNG(t))) + require.NoError(t, err) + second, err := store.Publish("two", now.Add(time.Hour), bytes.NewReader(testPNG(t))) + require.NoError(t, err) + store.PurgeSandbox("one") + assert.Equal(t, http.StatusNotFound, requestShot(t, store, http.MethodGet, first.URL).Code) + assert.Equal(t, http.StatusOK, requestShot(t, store, http.MethodGet, second.URL).Code) + _, err = store.Publish("expired", *now, bytes.NewReader(testPNG(t))) + require.Error(t, err) +} + +func TestScreenshotSizeAndCapacityRejectWithoutEviction(t *testing.T) { + store, now := testStore(t) + pngData := testPNG(t) + padded := make([]byte, MaxImageBytes) + copy(padded, pngData) + _, err := store.Publish( + "one", + now.Add(time.Hour), + io.MultiReader(bytes.NewReader(padded), bytes.NewReader([]byte{0})), + ) + require.ErrorContains(t, err, "16 MiB") + var urls []string + for range 8 { + var shot Screenshot + shot, err = store.Publish("one", now.Add(time.Hour), bytes.NewReader(padded)) + require.NoError(t, err) + urls = append(urls, shot.URL) + } + _, err = store.Publish("two", now.Add(time.Hour), bytes.NewReader(pngData)) + require.ErrorContains(t, err, "128 MiB") + for _, target := range urls { + assert.Equal(t, http.StatusOK, requestShot(t, store, http.MethodHead, target).Code) + } + *now = now.Add(5 * time.Minute) + _, err = store.Publish("two", now.Add(time.Hour), bytes.NewReader(pngData)) + require.NoError(t, err, "expired images release capacity before rejecting publication") + _, err = store.Publish("two", now.Add(time.Hour), bytes.NewReader([]byte("not a PNG"))) + require.Error(t, err) +} + +func TestScreenshotStartupClearsOnlyOwnedScratch(t *testing.T) { + dir := t.TempDir() + keep := filepath.Join(dir, "unrelated") + require.NoError(t, os.WriteFile(keep, []byte("keep"), 0o600)) + old := filepath.Join(dir, ".agentcompute-screenshots") + require.NoError(t, os.Mkdir(old, 0o700)) + require.NoError(t, os.WriteFile(filepath.Join(old, "old-image"), []byte("expired"), 0o600)) + store, err := NewStore(dir, "https://shots.example") + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, store.Close()) }) + _, err = os.Stat(filepath.Join(old, "old-image")) + require.ErrorIs(t, err, os.ErrNotExist) + data, err := os.ReadFile(keep) + require.NoError(t, err) + assert.Equal(t, "keep", string(data)) + _, err = NewStore(dir, "https://shots.example") + require.Error(t, err, "a second process must not erase live screenshots") +} diff --git a/internal/desktop/testdata/click-missing-pid.stdout b/internal/desktop/testdata/click-missing-pid.stdout new file mode 100644 index 0000000..790fa1c --- /dev/null +++ b/internal/desktop/testdata/click-missing-pid.stdout @@ -0,0 +1 @@ +Missing required integer field: pid diff --git a/internal/desktop/testdata/click-with-pid.stdout b/internal/desktop/testdata/click-with-pid.stdout new file mode 100644 index 0000000..f419e28 --- /dev/null +++ b/internal/desktop/testdata/click-with-pid.stdout @@ -0,0 +1,7 @@ +{ + "delivery": { + "mode": "background" + }, + "effect": "unverifiable", + "route": "accessibility" +} diff --git a/internal/desktop/testdata/desktop.stdout b/internal/desktop/testdata/desktop.stdout new file mode 100644 index 0000000..b28df1a --- /dev/null +++ b/internal/desktop/testdata/desktop.stdout @@ -0,0 +1,10 @@ +{ + "display": "primary", + "platform": "linux", + "scale_factor": 1, + "screen_height": 800, + "screen_width": 1280, + "screenshot_height": 800, + "screenshot_mime_type": "image/png", + "screenshot_width": 1280 +} diff --git a/internal/desktop/testdata/dump-docs.json b/internal/desktop/testdata/dump-docs.json new file mode 100644 index 0000000..8b0db3f --- /dev/null +++ b/internal/desktop/testdata/dump-docs.json @@ -0,0 +1 @@ +{"tools":[{"description":"List Linux apps — both currently running and installed-but-not-running — with per-app state flags:\n\n- running: is a process for this app live? (pid is 0 when false)\n- active: reserved (Linux X11/Wayland focus model differs from frontmost-app); always false.\n- kind: `\"desktop\"` for XDG `.desktop` launcher entries.\n- launch_path: the launcher command from `Exec=` (field codes stripped). Pass to `launch_app(launch_path=...)`.\n- bundle_id: the XDG \"desktop file id\" — the `.desktop` file's path relative to its XDG `applications/` root with the `.desktop` suffix stripped and path separators replaced with `-` (e.g. `kde4/konqbrowser.desktop` → `kde4-konqbrowser`).\n- last_used: RFC3339 mtime of the `.desktop` file, when readable.\n\nRunning apps come from `/proc`. Installed apps come from XDG Desktop Entry files in $XDG_DATA_HOME/applications and each $XDG_DATA_DIRS entry's applications/ subdir. Entries with `NoDisplay=true` or `Hidden=true` are filtered. A `.desktop` file whose launcher matches a running process (by basename) is merged into a single entry with `running: true`.\n\nUse this for \"is X installed?\" as well as \"is X running?\". For per-window state — visibility, geometry, titles — call list_windows instead.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{},"type":"object"},"name":"list_apps","read_only":true},{"description":"List top-level windows. Each record includes z_index (integer or null; higher values are closer to the front; null means stacking order is unavailable and callers must not infer one). To select a frontmost candidate, take the maximum integer z_index; if every value is null, use an explicit fallback instead of relying on array order.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"on_screen_only":{"description":"When true, filter to visible windows only. Default false.","type":"boolean"},"pid":{"type":"integer"}},"type":"object"},"name":"list_windows","read_only":true},{"description":"Walk a running app's AT-SPI tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array.\n\nPREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of AT-SPI actions exposed by the element, omitted when empty), `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side. Set `query` to project BOTH representations to matching rows plus their ancestor chain while preserving original indices. `total_element_count` reports the complete snapshot and `returned_element_count` reports the projection.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_index/element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail.\n\nOptional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"capture_mode":{"description":"DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old \"som\"/\"screenshot\" aliases) is accepted but has no effect.","enum":["ax","vision"],"type":"string"},"include_accessibility_tree":{"description":"Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).","type":"boolean"},"include_screenshot":{"description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action).","type":"boolean"},"max_depth":{"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps.","minimum":1,"type":"integer"},"max_dimension":{"description":"Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview. Applied on top of the configured max_image_dimension ceiling; the tighter wins. Omit for the configured default.","minimum":1,"type":"integer"},"max_elements":{"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees.","minimum":1,"type":"integer"},"pid":{"type":"integer"},"query":{"description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count.","type":"string"},"screenshot_out_file":{"description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"window_id":{"description":"Native window identifier from list_windows.","type":"integer"}},"required":["pid","window_id"],"type":"object"},"name":"get_window_state","read_only":true},{"description":"Deterministically verify bounded predicates against one exact window. The driver evaluates structured window/accessibility state and may return the final screenshot as uninterpreted visual evidence for a multimodal caller. Predicate results are satisfied, unsatisfied, or unknown; unknown never implies success. Accessibility projections are conservative: absence remains unknown unless the observed search domain is proven exhaustive.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"expect":{"description":"One to eight predicates, combined with logical AND.","items":{"additionalProperties":false,"properties":{"element":{"additionalProperties":false,"properties":{"enabled":{"type":["boolean","null"]},"exists":{"description":"Assert that at least one trusted element matches the selector.\n\nElement walks are not yet exhaustive on every platform, so absence\ncannot be proven. `false` is rejected instead of returning an\nindefinitely-unknown predicate.","enum":[true],"type":"boolean"},"selected":{"type":["boolean","null"]},"selector":{"additionalProperties":false,"properties":{"label_contains":{"minLength":1,"type":"string"},"role":{"minLength":1,"type":"string"}},"required":[],"type":"object"},"value_equals":{"type":["string","null"]}},"required":["selector"],"type":["object","null"]},"window":{"additionalProperties":false,"properties":{"bounds":{"additionalProperties":false,"properties":{"height":{"type":"number"},"tolerance_px":{"maximum":100,"minimum":0,"type":"number"},"width":{"type":"number"},"x":{"type":"number"},"y":{"type":"number"}},"required":["x","y","width","height"],"type":["object","null"]},"exists":{"type":["boolean","null"]}},"type":["object","null"]}},"required":[],"type":"object"},"maxItems":8,"minItems":1,"type":"array"},"include_screenshot":{"description":"Return the final window screenshot as image content for a multimodal\ncaller. The driver does not interpret that image.","type":["boolean","null"]},"pid":{"description":"Exact process whose window may be observed.","minimum":1,"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session. This\nfield never selects capture modality or authorization.","type":"string"},"stable_samples":{"default":2,"description":"Consecutive satisfied samples required before returning success.","maximum":5,"minimum":1,"type":"integer"},"timeout_ms":{"default":5000,"description":"Bounded wait. Zero performs one sample.","maximum":10000,"minimum":0,"type":"integer"},"window_id":{"description":"Exact native window identifier.","type":"integer"}},"required":["pid","window_id","expect"],"type":"object"},"name":"verify_state","read_only":true},{"description":"Launch a Linux app in the background. Provide launch_path (preferred — round-trip the value from list_apps), name (tried as a direct command, then matched against installed .desktop applications, then handed to xdg-open if it is a URL or existing file path), bundle_id (ignored on Linux), or urls (list of URLs to open). Resolution precedence: launch_path > name > bundle_id. Errors when the name resolves to nothing launchable.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"additional_arguments":{"description":"Extra command-line arguments passed to the launched process.","items":{"type":"string"},"type":"array"},"bundle_id":{"description":"Ignored on Linux (macOS/Windows concept).","type":"string"},"launch_path":{"description":"Round-trip the `launch_path` returned by `list_apps` — the Exec= command from the .desktop file with XDG field codes already stripped. Highest precedence on Linux; spawned directly via the system shell.","type":"string"},"name":{"description":"App name or command to launch. Tried as a direct command first, then matched against installed .desktop applications (exact display name, desktop-file id, or Exec basename; else an unambiguous display-name substring).","type":"string"},"urls":{"description":"URLs to open via xdg-open.","items":{"type":"string"},"type":"array"}},"type":"object"},"name":"launch_app","read_only":false},{"description":"Force-terminate a process by pid (kill -9 equivalent on Linux). Use as escalation when the cooperative close path failed to make the process exit. Unsaved state is lost — prefer the cooperative path first.","destructive":true,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"pid":{"description":"PID of the process to terminate.","type":"integer"}},"required":["pid"],"type":"object"},"name":"kill_app","read_only":false},{"description":"Persistently activate a window so subsequent input lands on it. This deliberately breaks the no-foreground contract and is not part of the normal input ladder. For an ordinary `background_unavailable` response, retry only the refused action with `delivery_mode:\"foreground\"`; the input tool performs its own activate, act, and restore sequence. Use `bring_to_front` only for a focus-proxy surface that must remain foreground across multiple calls, such as a remote desktop session, or when repeated action-scoped activation prevents the remote surface from accepting input. X11: EWMH _NET_ACTIVE_WINDOW activation (the `wmctrl -a` equivalent, with proper timestamp handling to beat focus-stealing prevention). Wayland: activates through a target-addressable compositor adapter (wlroots foreign-toplevel or the GNOME Shell helper) and refuses when the compositor offers no safe adapter. Matches the macOS / Windows bring_to_front rung.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"pid":{"type":"integer"},"window_id":{"description":"X11 window id (xid) to activate. If omitted, the first window of `pid` is used.","type":"integer"}},"required":["pid"],"type":"object"},"name":"bring_to_front","read_only":false},{"description":"Set one exact top-level window's frame in the desktop-coordinate space reported by list_windows and verify the resulting geometry through an independent readback.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"height":{"minimum":1,"type":"number"},"pid":{"minimum":1,"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"width":{"minimum":1,"type":"number"},"window_id":{"minimum":1,"type":"integer"},"x":{"type":"number"},"y":{"type":"number"}},"required":["pid","window_id","x","y","width","height"],"type":"object"},"name":"set_window_frame","read_only":false},{"description":"Resolve an exact application-menu path one live native level at a time and invoke its final item through accessibility APIs. Missing, ambiguous, disabled, or structurally mismatched segments fail closed; this tool never falls back to pixels.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"description":"Exact, immediate-child application menu path to resolve and invoke through\nthe operating system's accessibility API. Path labels are matched after\ntrimming surrounding whitespace and otherwise remain case-sensitive.","properties":{"path":{"items":{"maxLength":200,"minLength":1,"type":"string"},"maxItems":16,"minItems":1,"type":"array"},"pid":{"minimum":1,"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"window_id":{"minimum":1,"type":"integer"}},"required":["pid","window_id","path"],"type":"object"},"name":"invoke_menu","read_only":false},{"description":"Click against a target pid. **Prefer `element_index` over pixel coordinates** — element_index works on backgrounded / hidden windows, surfaces a stable handle, and tells you what you're clicking via the cached AT-SPI element's role + label. Reach for `x, y` only when the target is a canvas / custom-drawn surface that doesn't appear in the AT-SPI tree.\n\nProvide either (window_id + x/y) or (pid + element_index). Routes via XSendEvent (no focus steal). element_index cache is scoped per (pid, window_id) and is replaced by the next get_window_state of the same window — re-snapshot every turn before clicking.\n\nAfter a zoom call, pass from_zoom=true to auto-translate zoom-image coords back to full-window space.\n\nbutton: \"left\" (default), \"right\", or \"middle\". Defaults to left so the field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release with the matching button code. Native Wayland: only left-button is supported via the virtual-pointer protocol — right/middle return an error rather than silently degrading to left. `modifier` holds ctrl/shift/alt/super for the click on X11. Native Wayland refuses modified pointer clicks until its input protocol can carry keyboard modifier state.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"button":{"description":"Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error.","enum":["left","right","middle"],"type":"string"},"count":{"type":"integer"},"cursor_id":{"description":"Optional multi-cursor instance id. Default: 'default'.","type":"string"},"delivery_mode":{"default":"background","description":"Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.","enum":["background","foreground"],"type":"string"},"element_index":{"description":"Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.","type":"integer"},"element_token":{"description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.","type":"string"},"from_zoom":{"description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.","type":"boolean"},"modifier":{"description":"Modifier keys held during the action: cmd, shift, option/alt, ctrl.","items":{"type":"string"},"type":"array"},"pid":{"type":"integer"},"scope":{"default":"window","enum":["window","desktop"],"type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"snapshot_id":{"description":"Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.","pattern":"^s[0-9a-f]{8}$","type":"string"},"target":{"description":"Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.","oneOf":[{"additionalProperties":true,"properties":{"kind":{"const":"window","type":"string"},"pid":{"format":"uint32","minimum":0,"type":"integer"},"window_id":{"format":"uint64","minimum":0,"type":"integer"}},"required":["kind","pid","window_id"],"type":"object"},{"additionalProperties":true,"properties":{"display_id":{"type":"string"},"kind":{"const":"desktop","type":"string"}},"required":["kind","display_id"],"type":"object"}]},"window_id":{"type":"integer"},"x":{"type":"number"},"y":{"type":"number"}},"required":[],"type":"object"},"name":"click","read_only":false},{"description":"Double-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No focus steal. Provide either (window_id + x/y) or (pid + element_index). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"cursor_id":{"description":"Optional multi-cursor instance id. Default: 'default'.","type":"string"},"delivery_mode":{"default":"background","description":"Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.","enum":["background","foreground"],"type":"string"},"element_index":{"description":"Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.","type":"integer"},"element_token":{"description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.","type":"string"},"from_zoom":{"description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.","type":"boolean"},"pid":{"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"snapshot_id":{"description":"Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.","pattern":"^s[0-9a-f]{8}$","type":"string"},"window_id":{"type":"integer"},"x":{"type":"number"},"y":{"type":"number"}},"required":["pid"],"type":"object"},"name":"double_click","read_only":false},{"description":"Right-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No focus steal. Provide either (window_id + x/y) or (pid + element_index). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"cursor_id":{"description":"Optional multi-cursor instance id. Default: 'default'.","type":"string"},"delivery_mode":{"default":"background","description":"Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.","enum":["background","foreground"],"type":"string"},"element_index":{"description":"Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.","type":"integer"},"element_token":{"description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.","type":"string"},"from_zoom":{"description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.","type":"boolean"},"modifier":{"description":"Modifier keys held during the action: cmd, shift, option/alt, ctrl.","items":{"type":"string"},"type":"array"},"pid":{"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"snapshot_id":{"description":"Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.","pattern":"^s[0-9a-f]{8}$","type":"string"},"window_id":{"type":"integer"},"x":{"type":"number"},"y":{"type":"number"}},"required":["pid"],"type":"object"},"name":"right_click","read_only":false},{"description":"Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels via XSendEvent (ButtonPress + MotionNotify × steps + ButtonRelease). duration_ms (default 500), steps (default 20). No focus steal.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"button":{"description":"Mouse button. Default \"left\".","enum":["left","right","middle"],"type":"string"},"cursor_id":{"description":"Optional multi-cursor instance id. Default: 'default'.","type":"string"},"delivery_mode":{"default":"background","description":"Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.","enum":["background","foreground"],"type":"string"},"duration_ms":{"description":"Total drag duration. Default: 500.","maximum":10000,"minimum":0,"type":"integer"},"from_x":{"type":"number"},"from_y":{"type":"number"},"from_zoom":{"type":"boolean"},"modifier":{"description":"Modifier keys held during the action: cmd, shift, option/alt, ctrl.","items":{"type":"string"},"type":"array"},"pid":{"type":"integer"},"scope":{"default":"window","enum":["window","desktop"],"type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"steps":{"description":"Intermediate MotionNotify events. Default: 20.","maximum":200,"minimum":1,"type":"integer"},"target":{"anyOf":[{"description":"Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.","oneOf":[{"additionalProperties":true,"properties":{"kind":{"const":"window","type":"string"},"pid":{"format":"uint32","minimum":0,"type":"integer"},"window_id":{"format":"uint64","minimum":0,"type":"integer"}},"required":["kind","pid","window_id"],"type":"object"},{"additionalProperties":true,"properties":{"display_id":{"type":"string"},"kind":{"const":"desktop","type":"string"}},"required":["kind","display_id"],"type":"object"}]},{"type":"null"}]},"to_x":{"type":"number"},"to_y":{"type":"number"},"window_id":{"description":"Target window XID. Required.","type":"integer"}},"required":["from_x","from_y","to_x","to_y"],"type":"object"},"name":"drag","read_only":false},{"description":"Press and hold a mouse button at (x,y) via background X11 delivery. Does not release the button; pair with mouse_drag / mouse_button_up. Returns the current held-button state.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"button":{"description":"Mouse button. Default \"left\".","enum":["left","right","middle"],"type":"string"},"cursor_id":{"description":"Optional multi-cursor instance id. Default: 'default'.","type":"string"},"from_zoom":{"description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.","type":"boolean"},"pid":{"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.","type":"string"},"window_id":{"type":"integer"},"x":{"type":"number"},"y":{"type":"number"}},"required":["pid","window_id","x","y"],"type":"object"},"name":"mouse_button_down","read_only":false},{"description":"Move a previously-held mouse button to a new point via background X11 delivery. Requires an active mouse_button_down state; does not release the button. Returns the updated held-button state.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"cursor_id":{"description":"Optional multi-cursor instance id. Default: 'default'.","type":"string"},"duration_ms":{"description":"Total drag duration. Default: 500.","maximum":10000,"minimum":0,"type":"integer"},"from_zoom":{"description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.","type":"boolean"},"pid":{"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.","type":"string"},"steps":{"description":"Intermediate MotionNotify events. Default: 20.","maximum":200,"minimum":1,"type":"integer"},"window_id":{"type":"integer"},"x":{"type":"number"},"y":{"type":"number"}},"required":["x","y"],"type":"object"},"name":"mouse_drag","read_only":false},{"description":"Release a previously-held mouse button via background X11 delivery. If x/y are omitted, releases at the last held position. Returns the current held-button state.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"cursor_id":{"description":"Optional multi-cursor instance id. Default: 'default'.","type":"string"},"from_zoom":{"description":"Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.","type":"boolean"},"pid":{"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.","type":"string"},"window_id":{"type":"integer"},"x":{"type":"number"},"y":{"type":"number"}},"type":"object"},"name":"mouse_button_up","read_only":false},{"description":"Run multiple mouse drag gestures concurrently via Linux MPX/XI2 virtual master pointers. Each drag item runs on its own session-scoped master pointer (true same-window concurrent draws on X11). Each item presses once, glides continuously through its whole path, and releases once — one smooth held drag, not a chain of clicks. A path is given either as a straight segment (from_x/from_y → to_x/to_y) or as a function `fn` = y(x) sampled over [x_from, x_to] in window-local pixels (e.g. fn:\"x\" is a diagonal, fn:\"300+120*sin(x/40)\" a sine wave). Functions support + - * / ^, sin/cos/tan, sqrt, abs, exp, ln, pi, e.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"drags":{"items":{"additionalProperties":false,"properties":{"button":{"description":"Mouse button. Default \"left\".","enum":["left","right","middle"],"type":"string"},"duration_ms":{"description":"Default: 1500 for fn paths, 500 for straight.","maximum":10000,"minimum":0,"type":"integer"},"fn":{"description":"Expression y(x) in window-local pixels; sampled over [x_from,x_to]. Mutually exclusive with from_x/to_x.","type":"string"},"from_x":{"type":"number"},"from_y":{"type":"number"},"path":{"description":"Explicit window-local waypoints [[x,y],...] (>=2); pressed once, glided through, released once. Takes precedence over fn/from-to.","items":{"items":{"type":"number"},"type":"array"},"type":"array"},"samples":{"description":"Waypoints sampled along `fn`. Default: 80.","maximum":400,"minimum":2,"type":"integer"},"session":{"description":"Session/cursor id; also keys the virtual master pointer.","type":"string"},"steps":{"description":"Motion sub-steps along the whole path. Default: scaled to path length.","maximum":300,"minimum":1,"type":"integer"},"to_x":{"type":"number"},"to_y":{"type":"number"},"window_id":{"type":"integer"},"x_from":{"description":"Domain start (window-local x) when `fn` is used.","type":"number"},"x_to":{"description":"Domain end (window-local x) when `fn` is used.","type":"number"}},"required":["session","window_id"],"type":"object"},"minItems":2,"type":"array"}},"required":["drags"],"type":"object"},"name":"parallel_mouse_drag","read_only":false},{"description":"Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"delivery_mode":{"default":"background","description":"Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.","enum":["background","foreground"],"type":"string"},"element_index":{"description":"Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.","type":"integer"},"element_token":{"description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.","type":"string"},"pid":{"type":"integer"},"scope":{"default":"window","enum":["window","desktop"],"type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"snapshot_id":{"description":"Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.","pattern":"^s[0-9a-f]{8}$","type":"string"},"target":{"anyOf":[{"description":"Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.","oneOf":[{"additionalProperties":true,"properties":{"kind":{"const":"window","type":"string"},"pid":{"format":"uint32","minimum":0,"type":"integer"},"window_id":{"format":"uint64","minimum":0,"type":"integer"}},"required":["kind","pid","window_id"],"type":"object"},{"additionalProperties":true,"properties":{"display_id":{"type":"string"},"kind":{"const":"desktop","type":"string"}},"required":["kind","display_id"],"type":"object"}]},{"type":"null"}]},"text":{"type":"string"},"window_id":{"type":"integer"},"x":{"description":"Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click.","type":"number"},"y":{"description":"Screenshot-pixel Y of the field (see x).","type":"number"}},"required":["text"],"type":"object"},"name":"type_text","read_only":false},{"description":"Press a key via XSendEvent to a window. No focus steal.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"delivery_mode":{"default":"background","description":"Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.","enum":["background","foreground"],"type":"string"},"element_index":{"description":"Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.","type":"integer"},"element_token":{"description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.","type":"string"},"key":{"type":"string"},"modifiers":{"items":{"type":"string"},"type":"array"},"pid":{"type":"integer"},"scope":{"default":"window","enum":["window","desktop"],"type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"snapshot_id":{"description":"Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.","pattern":"^s[0-9a-f]{8}$","type":"string"},"target":{"anyOf":[{"description":"Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.","oneOf":[{"additionalProperties":true,"properties":{"kind":{"const":"window","type":"string"},"pid":{"format":"uint32","minimum":0,"type":"integer"},"window_id":{"format":"uint64","minimum":0,"type":"integer"}},"required":["kind","pid","window_id"],"type":"object"},{"additionalProperties":true,"properties":{"display_id":{"type":"string"},"kind":{"const":"desktop","type":"string"}},"required":["kind","display_id"],"type":"object"}]},{"type":"null"}]},"window_id":{"type":"integer"},"x":{"description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index.","type":"number"},"y":{"description":"Screenshot-pixel Y (see x).","type":"number"}},"required":["key"],"type":"object"},"name":"press_key","read_only":false},{"description":"Press a combination of keys simultaneously, e.g. [\"ctrl\",\"c\"] for Copy. Sent via XSendEvent directly to the target pid; target does NOT need to be frontmost.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"delivery_mode":{"default":"background","description":"Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.","enum":["background","foreground"],"type":"string"},"element_index":{"description":"Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.","type":"integer"},"element_token":{"description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.","type":"string"},"keys":{"description":"Modifier(s) + one non-modifier key, e.g. [\"ctrl\",\"c\"].","items":{"type":"string"},"minItems":2,"type":"array"},"pid":{"type":"integer"},"scope":{"default":"window","enum":["window","desktop"],"type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"snapshot_id":{"description":"Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.","pattern":"^s[0-9a-f]{8}$","type":"string"},"target":{"anyOf":[{"description":"Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.","oneOf":[{"additionalProperties":true,"properties":{"kind":{"const":"window","type":"string"},"pid":{"format":"uint32","minimum":0,"type":"integer"},"window_id":{"format":"uint64","minimum":0,"type":"integer"}},"required":["kind","pid","window_id"],"type":"object"},{"additionalProperties":true,"properties":{"display_id":{"type":"string"},"kind":{"const":"desktop","type":"string"}},"required":["kind","display_id"],"type":"object"}]},{"type":"null"}]},"window_id":{"type":"integer"},"x":{"description":"Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach.","type":"number"},"y":{"description":"Screenshot-pixel Y (see x).","type":"number"}},"required":["keys"],"type":"object"},"name":"hotkey","read_only":false},{"description":"Set value of an AT-SPI element via SetValue action.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"element_index":{"description":"Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.","type":"integer"},"element_token":{"description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.","type":"string"},"pid":{"type":"integer"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"snapshot_id":{"description":"Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.","pattern":"^s[0-9a-f]{8}$","type":"string"},"value":{"type":"string"},"window_id":{"description":"Required when element_index is used; optional when element_token is supplied (the token carries it).","type":"integer"}},"required":["pid","value"],"type":"object"},"name":"set_value","read_only":false},{"description":"Scroll the target pid's focused region via XSendEvent Button4/5. direction required; by defaults to line, amount defaults to 3.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"amount":{"maximum":50,"minimum":1,"type":"integer"},"by":{"enum":["line","page"],"type":"string"},"cursor_id":{"description":"Optional multi-cursor instance id. Default: 'default'.","type":"string"},"delivery_mode":{"default":"background","description":"Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.","enum":["background","foreground"],"type":"string"},"direction":{"enum":["up","down","left","right"],"type":"string"},"element_index":{"description":"Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values.","type":"integer"},"element_token":{"description":"Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it.","type":"string"},"pid":{"type":"integer"},"scope":{"default":"window","enum":["window","desktop"],"type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"snapshot_id":{"description":"Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed.","pattern":"^s[0-9a-f]{8}$","type":"string"},"target":{"anyOf":[{"description":"Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.","oneOf":[{"additionalProperties":true,"properties":{"kind":{"const":"window","type":"string"},"pid":{"format":"uint32","minimum":0,"type":"integer"},"window_id":{"format":"uint64","minimum":0,"type":"integer"}},"required":["kind","pid","window_id"],"type":"object"},{"additionalProperties":true,"properties":{"display_id":{"type":"string"},"kind":{"const":"desktop","type":"string"}},"required":["kind","display_id"],"type":"object"}]},{"type":"null"}]},"window_id":{"type":"integer"},"x":{"description":"Window-local screenshot-pixel X of the scroll target. Pass with y and without element_index.","type":"number"},"y":{"description":"Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_index.","type":"number"}},"required":["direction"],"type":"object"},"name":"scroll","read_only":false},{"description":"List available system clipboard types and optionally return privacy-sensitive plain text. Clipboard content is never retained in telemetry.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"include_text":{"default":false,"description":"Return plain-text clipboard content in addition to the available types.\nClipboard content is privacy-sensitive and is never retained in telemetry.","type":"boolean"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"}},"required":[],"type":"object"},"name":"clipboard_read","read_only":true},{"description":"Replace the system clipboard with exactly one value: plain text, an image from an absolute local path, or a file URL from an absolute local path. Returns the available types for read-back before paste.","destructive":true,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"file_path":{"description":"Absolute path to a local file to place on the clipboard as a file URL.","type":"string"},"image_path":{"description":"Absolute path to a local image to place on the clipboard.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"text":{"description":"Plain text to place on the clipboard.","type":"string"}},"required":[],"type":"object"},"name":"clipboard_write","read_only":false},{"description":"Return the logical size of the main display in points plus its backing scale factor. Agents click in points; Retina displays have scale_factor 2.0. Requires no TCC permissions.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"}},"type":"object"},"name":"get_screen_size","read_only":true},{"description":"Capture the full display in the desktop action coordinate frame. Use the returned PNG directly as the coordinate source for actions whose target is {kind:\"desktop\",display_id:\"primary\"}. No AT-SPI walk.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"screenshot_out_file":{"description":"Write PNG here instead of base64.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"}},"type":"object"},"name":"get_desktop_state","read_only":true},{"description":"Return the current mouse cursor position in screen points (origin top-left).","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"}},"type":"object"},"name":"get_cursor_position","read_only":true},{"description":"Move the synthetic agent cursor without changing the user's pointer. Only an explicit scope=desktop request moves the real OS pointer in get_desktop_state coordinates.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"cursor_id":{"type":"string"},"scope":{"default":"window","enum":["window","desktop"],"type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.","type":"string"},"target":{"anyOf":[{"description":"Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.","oneOf":[{"additionalProperties":true,"properties":{"kind":{"const":"window","type":"string"},"pid":{"format":"uint32","minimum":0,"type":"integer"},"window_id":{"format":"uint64","minimum":0,"type":"integer"}},"required":["kind","pid","window_id"],"type":"object"},{"additionalProperties":true,"properties":{"display_id":{"type":"string"},"kind":{"const":"desktop","type":"string"}},"required":["kind","display_id"],"type":"object"}]},{"type":"null"}],"description":"Preferred per-call target. New callers should set this field."},"x":{"type":"number"},"y":{"type":"number"}},"required":["x","y"],"type":"object"},"name":"move_cursor","read_only":false},{"description":"Show or hide the agent cursor owned by a session.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"enabled":{"type":"boolean"},"session":{"type":"string"}},"required":["session","enabled"],"type":"object"},"name":"set_agent_cursor_enabled","read_only":false},{"description":"Configure only movement physics and visibility timing for a session cursor.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"arc_flow":{"format":"double","type":["number","null"]},"arc_size":{"format":"double","type":["number","null"]},"dwell_after_click_ms":{"format":"double","type":["number","null"]},"end_handle":{"format":"double","type":["number","null"]},"glide_duration_ms":{"format":"double","type":["number","null"]},"idle_hide_ms":{"format":"double","type":["number","null"]},"session":{"type":"string"},"spring":{"format":"double","type":["number","null"]},"start_handle":{"format":"double","type":["number","null"]},"turn_radius":{"format":"double","type":["number","null"]}},"required":["session"],"type":"object"},"name":"set_agent_cursor_motion","read_only":false},{"description":"Return the session cursor's theme, semantic playback, position, visibility, and motion.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"session":{"type":"string"}},"required":["session"],"type":"object"},"name":"get_agent_cursor_state","read_only":true},{"description":"Select an already-installed cursor theme for a session.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"reduced_motion":{"default":"auto","enum":["auto","on","off"],"type":"string"},"session":{"type":"string"},"theme_id":{"maxLength":200,"minLength":1,"type":"string"}},"required":["session","theme_id"],"type":"object"},"name":"set_agent_cursor_theme","read_only":false},{"description":"Check required permissions for cua-driver-rs on Linux.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{},"type":"object"},"name":"check_permissions","read_only":true},{"description":"Single-call end-to-end driver diagnostics. Designed to let downstream consumers ship one stable call instead of stitching together check_permissions, doctor, version, bundle attribution, and platform capability status. On macOS, prompt-capable direct capture is deliberately skipped; use `cua-driver permissions grant` to verify it explicitly. cua-driver owns the health model; consumers stay thin.\n\nInput — all optional:\n {\n \"include\": [\"\", ...], // run only these\n \"skip\": [\"\", ...] // skip these\n }\nIf both are given, `include` wins.\n\nCanonical check names:\n macOS : binary_version, platform_supported, session_active,\n bundle_identity, tcc_accessibility, tcc_screen_recording,\n ax_capability, screen_capture_capability\n Windows: binary_version, platform_supported, session_active,\n ax_capability (via UIA), screen_capture_capability (via DXGI)\n Linux : binary_version, platform_supported, session_active,\n ax_capability (via AT-SPI), screen_capture_capability (via X11)\n\nOutput — stable contract, schema_version=\"1\":\n {\n \"schema_version\": \"1\",\n \"platform\": \"darwin\" | \"win32\" | \"linux\",\n \"driver_version\": \"\",\n \"overall\": \"ok\" | \"degraded\" | \"failed\",\n \"checks\": [\n {\n \"name\": \"\",\n \"status\": \"pass\" | \"fail\" | \"skip\",\n \"message\": \"\",\n \"hint\": \"\",\n \"data\": { /* check-specific structured fields */ }\n },\n ...\n ]\n }\n\n`overall` rules:\n - `ok` — every non-skipped check passes\n - `degraded` — at least one non-core check fails (binary is still usable)\n - `failed` — any core check fails (binary_version, platform_supported, session_active)\n\nStability: schema_version=\"1\" is the contract. Future breaking changes will be `\"2\"`. Adding new check names under the same schema_version is non-breaking; consumers must tolerate unknown check names.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"include":{"description":"Only run these checks (canonical names). Wins over `skip`.","items":{"type":"string"},"type":"array"},"skip":{"description":"Skip these checks (canonical names). Ignored when `include` is set.","items":{"type":"string"},"type":"array"}},"type":"object"},"name":"health_report","read_only":true},{"description":"Return current cua-driver-rs configuration.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{},"type":"object"},"name":"get_config","read_only":true},{"description":"Update cua-driver-rs configuration. capture_mode / max_image_dimension take effect immediately.\n\nTwo input shapes (both accepted, matching Windows/Swift):\n- **{key, value}** (preferred): `{\"key\": \"max_image_dimension\", \"value\": 800}` — single leaf write.\n- **Legacy per-field**: `{\"capture_mode\": \"som\", \"max_image_dimension\": 0}`.\n\nThe experimental_pip keys persist to ~/.cua-driver/config.json and apply on next daemon restart (the PiP backend is initialised once at startup; Linux ships only the trait stub today — see issue #1729).","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"capture_mode":{"description":"Legacy per-field shape. Default capture mode for get_window_state. (\"som\"/\"screenshot\" still decode as deprecated aliases.)","enum":["ax","vision"],"type":"string"},"experimental_pip":{"description":"Enable the experimental PiP preview window (applies next restart; Linux backend stubbed).","type":"boolean"},"experimental_pip_geometry":{"description":"PiP window size + optional position in `WxH` or `WxH+X+Y` form.","type":"string"},"key":{"description":"Name of a single config field to write ({key, value} shape). Pair with `value`.","type":"string"},"max_image_dimension":{"description":"Legacy per-field shape. Max dimension for screenshot resizing (0 = no limit).","type":"integer"},"value":{"description":"New value for `key`. JSON type depends on the key."}},"type":"object"},"name":"set_config","read_only":false},{"description":"Return a lightweight snapshot of the desktop: running processes and on-screen visible X11 windows with their bounds and owner pid.\n\nFor the full AT-SPI subtree of a single window (with interactive element indices you can click by), use get_window_state instead — this is a fast discovery read.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{},"type":"object"},"name":"get_accessibility_tree","read_only":true},{"description":"Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixels, with 20% padding. Output is at most 500 px wide.\n\nAfter a zoom, pass from_zoom=true to click/type_text to auto-translate coordinates back to full-window space.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"pid":{"description":"Target pid — required for from_zoom click/type translation.","type":"integer"},"window_id":{"type":"integer"},"x1":{"type":"number"},"x2":{"type":"number"},"y1":{"type":"number"},"y2":{"type":"number"}},"required":["window_id","x1","y1","x2","y2"],"type":"object"},"name":"zoom","read_only":true},{"description":"Legacy browser compatibility tool. Prefer get_browser_state and the typed browser_* tools for exact targeting, endpoint ownership, and consent. Read-only get_text and query_dom remain available by default. Mutating actions require the daemon operator to set CUA_DRIVER_ENABLE_LEGACY_PAGE_MUTATIONS=1 before daemon startup (restart the daemon after changing it); this escape hatch does not provide the typed browser surface's exact binding or existing-profile grant guarantees. Supports Chrome, Brave, Edge, Safari (via AppleScript on macOS), Electron apps (via CDP), Chromium/Firefox on Windows (via UIA for read; CDP for execute_javascript when --remote-debugging-port is set), and WKWebView/Tauri/AT-SPI fallbacks.\n\nActions:\n- execute_javascript: Run JS and return the result.\n- get_text: Extract visible text from the page.\n- query_dom: Find elements matching a CSS selector.\n- click_element: Click a CSS-selected element AND animate the agent cursor to its on-screen center first (so the user sees what the agent is doing). Prefer over `execute_javascript('el.click()')` whenever you want visible cursor feedback.\n- insert_text: Insert `text` at whatever currently holds DOM focus in one native operation (CDP Input.insertText) — no synthesized key events, but more durable than a one-shot execute_javascript write since rich-text editors already have to treat it like an IME commit. Try this before type_keystrokes on a contenteditable that discarded an execute_javascript write. Click/focus the target field first.\n- type_keystrokes: Type `text` via real per-character keystroke events into whatever currently holds DOM focus. Slower than insert_text but the most durable rung — use it when insert_text also gets discarded, or the editor's own keydown/keyup handlers need to see real keys. Click/focus the target field first.\n- enable_javascript_apple_events: macOS-only — patch the browser's Preferences to allow JS from Apple Events (Chrome/Brave/Edge, requires user confirmation and a browser restart).","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"action":{"description":"Action to perform.","enum":["execute_javascript","get_text","query_dom","click_element","insert_text","type_keystrokes","enable_javascript_apple_events"],"type":"string"},"attributes":{"description":"Element attributes to include in query_dom results.","items":{"type":"string"},"type":"array"},"bundle_id":{"description":"Bundle ID of the browser. Required for enable_javascript_apple_events (macOS only).","type":"string"},"cdp_port":{"description":"Optional, for execute_javascript/insert_text/type_keystrokes: use this exact CDP port instead of auto-discovering one from pid. Needed when the port was opened via the browser's own remote-debugging toggle rather than a launch-time flag, since that path may not answer the auto-discovery probe.","maximum":65535,"minimum":1,"type":"integer"},"css_selector":{"description":"CSS selector for query_dom (e.g. 'a', 'button', 'input', 'h1'-'h6', 'p', 'img', 'select', '*').","type":"string"},"javascript":{"description":"JavaScript to execute. Required for execute_javascript.","type":"string"},"pid":{"description":"Target process ID.","type":"integer"},"selector":{"description":"CSS selector for click_element (e.g. 'button.submit', '#login a').","type":"string"},"target_url_contains":{"description":"Optional, for execute_javascript/insert_text/type_keystrokes: require exactly one browser tab whose URL contains this substring. Use this on a multi-tab browser — there's no built-in link between window_id and which tab a CDP call reaches.","type":"string"},"text":{"description":"Text to insert or type. Required for insert_text and type_keystrokes. The target field must already have DOM focus (click/focus it first).","type":"string"},"user_has_confirmed_enabling":{"description":"Must be true to proceed with enable_javascript_apple_events. This will quit and relaunch the browser.","type":"boolean"},"window_id":{"description":"Target window ID from list_windows.","type":"integer"}},"required":["action"],"type":"object"},"name":"page","read_only":false},{"description":"Read-only browser inspection. Mode 1 (bind): pass pid + window_id of a native browser window to classify it, correlate it to a CDP target (exact-or-refuse), and mint a session-scoped target id plus tab ids. Mode 2 (snapshot): pass target_id + tab_id. The dom_refs_v1 compatibility format returns composed DOM refs. semantic_v2 joins accessibility, DOM, layout, and viewport state; ranks visible content before retained/offscreen state; and returns a semantic outline, typed action refs, content refs, scoped reads, and opaque continuation. Never performs setup — a missing endpoint is a structured browser_requires_setup refusal pointing at browser_prepare.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":true,"properties":{"continuation":{"description":"Opaque continuation minted by an earlier semantic_v2 response.","type":"string"},"include_screenshot":{"default":false,"description":"Capture the exact tab viewport as PNG through CDP without selecting the tab or foregrounding its native window. The request refuses if capture cannot be completed.","type":"boolean"},"pid":{"description":"Native browser process id (bind mode).","type":"integer"},"query":{"description":"Read-only semantic match over role, accessible name, and visible text.","type":"string"},"scope_ref":{"description":"Current semantic/content ref whose subtree should be observed.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.","type":"string"},"snapshot_format":{"description":"Versioned snapshot contract. dom_refs_v1 remains the compatibility default.","enum":["dom_refs_v1","semantic_v2"],"type":"string"},"tab_id":{"description":"Opaque tab id from get_browser_state (session-scoped).","type":"string"},"target_id":{"description":"Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).","type":"string"},"window_id":{"description":"Native window id owned by pid (bind mode).","type":"integer"}},"type":"object"},"name":"get_browser_state","read_only":true},{"description":"Explicitly prepare an owned DevTools endpoint for a browser. pid is required for an existing process or existing-profile attachment, and optional only for allow_launch=true with an isolated profile. Existing endpoints are detected without side effects. Acting setup for an isolated profile follows the runtime permission mode and optional capability manifest. It requires allow_launch=true, launches a separate browser, and never copies, modifies, or terminates the requested user profile. Without pid, only a platform-attested system Chrome/Edge installation (or a root-owned package payload on Linux) is eligible; redirects and user-controlled locations fail closed. Existing-profile attachment is explicit and follows the runtime's immutable permission mode: standard requires an explicit --grant existing-profile launch grant or an embedding authorization host, bounded requires a launch-approved exact resource manifest, and unrestricted requires explicit trusted startup risk acceptance. Ordinary MCP transport approval never proves profile authorization. On proven platforms, an authorized request also permits one bounded exact-window setup: open the recognized browser product's fixed remote-debugging page, toggle its uniquely matched per-instance checkbox, prove the PID-owned loopback endpoint, and close the temporary tab. Every visible effect is reported; ambiguity is refused.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"allow_launch":{"description":"Allow a separate driver-owned isolated Chromium process to be launched (default false).","type":"boolean"},"pid":{"description":"Browser process id to prepare. Required except for a driver-owned isolated_new/isolated_named launch with allow_launch=true.","type":"integer"},"profile":{"additionalProperties":false,"properties":{"mode":{"enum":["isolated_new","isolated_named"],"type":"string"},"name":{"description":"Required only for isolated_named; 1-64 path-safe ASCII characters.","type":"string"}},"required":["mode"],"type":"object"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.","type":"string"},"strategy":{"additionalProperties":false,"properties":{"kind":{"enum":["existing_profile"],"type":"string"}},"required":["kind"],"type":"object"},"window_id":{"description":"Exact native window approval anchor; required for strategy.kind=existing_profile.","type":"integer"}},"required":[],"type":"object"},"name":"browser_prepare","read_only":false},{"description":"Navigate one tab of an exactly-bound browser target to a new URL (http/https/about only). Refused for heuristic bindings. Navigation invalidates all p: refs for the tab.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.","type":"string"},"tab_id":{"description":"Opaque tab id from get_browser_state (session-scoped).","type":"string"},"target_id":{"description":"Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).","type":"string"},"url":{"description":"Destination URL (http:, https:, or about:).","type":"string"}},"required":["target_id","tab_id","url"],"type":"object"},"name":"browser_navigate","read_only":false},{"description":"Click a page element (by ref) or viewport coordinates in an exactly-bound tab. Default route is trusted hardware-like input (Input.dispatchMouseEvent), and refuses where that route cannot preserve standalone-browser background posture. input_route=\"dom_event\" (synthetic el.click(), ref required) is used only when explicitly requested; it proves dispatch, not control activation, because trust-gated controls may ignore synthetic events. Refused for heuristic bindings.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"input_route":{"description":"\"trusted\" (default): Input.dispatchMouseEvent. It refuses rather than foregrounding a standalone browser. \"dom_event\": synthetic full-background DOM click, only when explicitly requested. Dispatch does not prove the control activated; refresh page state and verify the expected postcondition.","enum":["trusted","dom_event"],"type":"string"},"ref":{"description":"Page element ref in the p: namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.","type":"string"},"tab_id":{"description":"Opaque tab id from get_browser_state (session-scoped).","type":"string"},"target_id":{"description":"Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).","type":"string"},"x":{"description":"Viewport x (CSS px) — alternative to ref.","type":"number"},"y":{"description":"Viewport y (CSS px) — alternative to ref.","type":"number"}},"required":["target_id","tab_id"],"type":"object"},"name":"browser_click","read_only":false},{"description":"Type text into an exactly-bound tab via the Input domain. mode=\"insert_text\" (default) uses Input.insertText; mode=\"keystrokes\" dispatches per-character key events. Both insert at the caret, so typing into a field that already holds text appends to it; pass replace=true to set the field instead, or to clear it by typing an empty string. Pass a ref to an editable element from the latest snapshot. A ref is required; heuristic bindings are refused.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"mode":{"description":"insert_text (default): bulk Input.insertText. keystrokes: per-character Input.dispatchKeyEvent.","enum":["insert_text","keystrokes"],"type":"string"},"ref":{"description":"Page element ref in the p: namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.","type":"string"},"replace":{"description":"false (default): insert at the caret, appending to whatever the field already holds. true: select the element's whole content first so the text replaces it — with an empty text this clears the field. Replacement goes through the selection, so beforeinput/input still fire and framework state stays consistent.","type":"boolean"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.","type":"string"},"tab_id":{"description":"Opaque tab id from get_browser_state (session-scoped).","type":"string"},"target_id":{"description":"Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).","type":"string"},"text":{"description":"Text to type.","type":"string"}},"required":["target_id","tab_id","ref","text"],"type":"object"},"name":"browser_type","read_only":false},{"description":"Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"action":{"enum":["inspect","accept","dismiss"],"type":"string"},"delivery_mode":{"default":"background","description":"Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only.","enum":["background","foreground"],"type":"string"},"dialog_id":{"description":"Opaque current dialog generation returned by action=inspect.","type":"string"},"prompt_text":{"description":"Sensitive response text, valid only when accepting a prompt dialog.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.","type":"string"},"tab_id":{"description":"Opaque tab id from get_browser_state (session-scoped).","type":"string"},"target_id":{"description":"Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).","type":"string"}},"required":["target_id","tab_id","action"],"type":"object"},"name":"browser_dialog","read_only":false},{"description":"Assign one or more explicit absolute local files to an exact live ref through CDP. This bypasses native file pickers, rejects symlinks and non-regular files, and never returns local paths.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"files":{"items":{"description":"Absolute path to one local regular file.","type":"string"},"maxItems":32,"minItems":1,"type":"array"},"ref":{"description":"Page element ref in the p: namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.","type":"string"},"tab_id":{"description":"Opaque tab id from get_browser_state (session-scoped).","type":"string"},"target_id":{"description":"Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).","type":"string"}},"required":["target_id","tab_id","ref","files"],"type":"object"},"name":"browser_set_input_files","read_only":false},{"description":"Trigger one download through an exact live browser ref and save it inside an explicitly approved directory. Requires MCP-host destructive-tool approval, refuses ambiguous or stale capabilities, and never returns the source URL, filename, or destination path.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"destination_root":{"description":"Absolute, existing, canonical directory approved to receive the download.","type":"string"},"ref":{"description":"Live page ref whose activation initiates the download.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. This tool requires the label that owns its browser target, tab, and refs.","type":"string"},"tab_id":{"description":"Opaque exact tab id from get_browser_state.","type":"string"},"target_id":{"description":"Opaque exact browser target id from get_browser_state.","type":"string"}},"required":["session","target_id","tab_id","ref","destination_root"],"type":"object"},"name":"browser_download","read_only":false},{"description":"Perform hover, right-click, double-click, scroll, or drag in an exactly-bound browser tab. Semantic refs must declare pointer for hover, right-click, double-click, and drag; scroll accepts a scroll or pointer capability. The trusted route uses CDP Input events and refuses if standalone background posture cannot be preserved. The explicit dom_event route requires a page ref and synthesizes full-background DOM events. Never activates or brings a tab to the foreground.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"action":{"enum":["hover","right_click","double_click","scroll","drag"],"type":"string"},"delta_x":{"description":"Horizontal scroll delta in CSS pixels.","type":"number"},"delta_y":{"description":"Vertical scroll delta in CSS pixels.","type":"number"},"destination_ref":{"description":"Drag destination page ref in the exact same frame.","type":"string"},"input_route":{"default":"trusted","enum":["trusted","dom_event"],"type":"string"},"ref":{"description":"Origin page ref. Alternative to x/y.","type":"string"},"session":{"description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. This tool requires the label that owns its browser target, tab, and refs.","type":"string"},"tab_id":{"description":"Opaque tab id minted by get_browser_state.","type":"string"},"target_id":{"description":"Opaque target id minted by get_browser_state.","type":"string"},"to_x":{"description":"Drag destination viewport x in CSS pixels.","type":"number"},"to_y":{"description":"Drag destination viewport y in CSS pixels.","type":"number"},"x":{"description":"Origin viewport x in CSS pixels.","type":"number"},"y":{"description":"Origin viewport y in CSS pixels.","type":"number"}},"required":["target_id","tab_id","session","action"],"type":"object"},"name":"browser_pointer","read_only":false},{"description":"Start trajectory recording. Every subsequent action-tool invocation (click, right_click, scroll, type_text, press_key, hotkey, set_value) writes a turn folder under `output_dir`:\n\n- `before_state.json` / `after_state.json` — application AX/UIA/AT-SPI state immediately before and after the action.\n- `before.png` / `after.png` — target-window screenshots immediately before and after the action.\n- `evidence.json` — capture status and a stable classification when an expected artifact could not be captured.\n- `app_state.json` — post-action AX/UIA snapshot for the target pid.\n- `screenshot.png` — compatibility alias of `after.png`.\n- `action.json` — tool name, full input arguments, result summary, result-error flag, pid, click point (when applicable), ISO-8601 timestamp.\n- `click.png` — for dispatched click-family actions only, `before.png` with a red marker at the click point. A call refused before target resolution is explicitly not applicable instead.\n\nTurn folders are named `turn-00001/`, `turn-00002/`, etc. Turn numbering restarts at 1 each time recording is (re-)started.\n\n**Video is off by default.** Pass `record_video: true` to also capture the main display to `/recording.mp4` (H.264 / 30 fps) for the lifetime of the session. The recording is torn down automatically when the MCP client disconnects.\n\n**macOS uses native ScreenCaptureKit** (daemon-owned SCStream + SCRecordingOutput) so video inherits the daemon's Screen Recording grant — no extra TCC prompt, no ffmpeg subprocess. Requires macOS 15.0+.\n\n**Windows + Linux use an ffmpeg subprocess** (`gdigrab` / `x11grab` + libx264). Requires ffmpeg on PATH (winget install Gyan.FFmpeg / apt install ffmpeg); when ffmpeg is missing or fails on startup the per-turn capture (screenshots + action.json) still runs and the session's `last_error` field carries the diagnostic.\n\nState persists for the life of the daemon; a restart resets to disabled with no on-disk state. Call `stop_recording` to disable + finalize the mp4.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{"output_dir":{"description":"Absolute or ~-rooted directory where turn folders and (when enabled) the video file are written.","type":"string"},"record_video":{"description":"Capture the main display to /recording.mp4. Default: false. Set to true to also capture the main display to recording.mp4 (otherwise only the per-turn screenshots + JSON are recorded). On macOS this uses native ScreenCaptureKit (no extra TCC prompt, macOS 15.0+); on Windows + Linux it requires ffmpeg on PATH.","type":"boolean"}},"required":["output_dir"],"type":"object"},"name":"start_recording","read_only":false},{"description":"Stop trajectory recording. Disables further per-turn capture and, when video was enabled, gracefully terminates the ffmpeg subprocess so the mp4's moov atom is finalized (the file is playable). Calling stop on an already-stopped session is a no-op. The response carries `last_video_path` pointing at the finalized mp4 (when video was on).\n\nA manual `stop_recording` is **unconditional** — it stops whatever recording is active regardless of which session started it. Ownership-scoped teardown (so one client disconnecting can't stop a recording a later client started) is handled by the registry's `session_end` lifecycle hook, not by this tool.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{},"type":"object"},"name":"stop_recording","read_only":false},{"description":"Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder that will be written. Counter increments on every recorded action tool call and resets to 1 each time recording is (re-)enabled.\n\nPure read-only.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{},"type":"object"},"name":"get_recording_state","read_only":true},{"description":"Replay a recorded trajectory by re-invoking every turn's tool call in lexical order. `dir` must point at a directory previously written by `start_recording`. Each `turn-NNNNN/` is parsed for `action.json`, and the recorded tool is called with its recorded `arguments` via the same dispatch path an MCP / CLI call uses.\n\nCaveats:\n- Element-indexed actions (`click({pid, element_index})` etc.) will fail because element indices are per-snapshot and don't survive across sessions. Pixel clicks (`click({pid, x, y})`) and all keyboard tools replay cleanly. Failures are reported but don't stop replay unless `stop_on_error` is true.\n- `get_window_state` and other read-only tools are NOT currently recorded, so replays do not re-populate the per-(pid, window_id) element cache.\n- If recording is ENABLED while replay runs, the replay itself is recorded into the currently configured output directory. That's deliberate: recording a replay against a new build and diffing the two trajectories is the regression-test workflow.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"delay_ms":{"description":"Milliseconds to sleep between turns, for human-observable pacing. Default 500.","maximum":10000,"minimum":0,"type":"integer"},"dir":{"description":"Trajectory directory previously written by `start_recording`. Absolute or ~-rooted.","type":"string"},"stop_on_error":{"description":"Stop replay on the first tool-call error. Default true — set false to best-effort through the full trajectory.","type":"boolean"}},"required":["dir"],"type":"object"},"name":"replay_trajectory","read_only":false},{"description":"Install the ffmpeg binary used by start_recording's video capture (Linux/Windows; macOS records natively and needs no ffmpeg). Two-step and confirmed: called without `confirm` it only REPORTS the exact install command for this platform's package manager; pass `confirm: true` to actually run it. No-op if ffmpeg is already on PATH. ffmpeg is run as a separate process, never linked into the driver.","destructive":true,"idempotent":false,"input_schema":{"additionalProperties":false,"properties":{"confirm":{"description":"Run the install command. Without it, only the planned command is reported.","type":"boolean"}},"type":"object"},"name":"install_ffmpeg","read_only":false},{"description":"Optionally create or return a lifecycle session before acting. For multi-call work, prefer a short public `session` label and repeat it on every call that accepts it; an omitted value uses the authenticated transport lease's implicit session instead. This tool is optional because an ordinary action can create or reuse a named run directly. Use it to set the initial cursor theme before acting or to revive a public name after it has ended; ordinary actions never revive ended names. `capture_scope` is deprecated compatibility input; new callers select window or desktop modality per action. Idempotent.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":true,"properties":{"capture_scope":{"description":"Deprecated compatibility policy. New callers select window or desktop\nmodality on each action instead of storing it on the session.","enum":["auto","window","desktop"],"type":"string"},"cursor_theme":{"description":"Optional initial cursor theme. The host applies it before the cursor is\nfirst made visible, avoiding a flash of the default theme.","properties":{"reduced_motion":{"default":"auto","enum":["auto","on","off"],"type":"string"},"theme_id":{"type":"string"}},"required":["theme_id"],"type":["object","null"]},"session":{"description":"Optional stable public label for this run (e.g. \"research-run-1\").\nWhen omitted, the authenticated transport lease's implicit session is\ncreated or returned.","type":"string"}},"required":[],"type":"object"},"name":"start_session","read_only":false},{"description":"Deprecated compatibility tool for legacy capture-scope sessions. New callers select window or desktop modality on each action. No deescalate_session tool exists.","destructive":false,"idempotent":false,"input_schema":{"additionalProperties":true,"properties":{"detail":{"description":"Optional bounded diagnostic detail. Never use secrets or page content.","maxLength":200,"type":"string"},"reason":{"enum":["ax_tree_pixel_mismatch","background_delivery_failed","foreground_ineffective","no_window_target","other"],"type":"string"},"session":{"type":"string"}},"required":["session","reason"],"type":"object"},"name":"escalate_session","read_only":false},{"description":"Read content-free lifecycle, cursor, recording, and idle status for one session visible to this authenticated transport. Omit `session` to inspect its implicit session.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":true,"properties":{"session":{"description":"Optional public label. When omitted, inspect the caller's attached\nimplicit session.","type":"string"}},"required":[],"type":"object"},"name":"get_session","read_only":true},{"description":"List content-free lifecycle summaries attached to this authenticated transport lease. It does not enumerate other callers' sessions.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":true,"properties":{"cursor":{"description":"Opaque continuation cursor returned by a previous call.","type":"string"},"limit":{"description":"Maximum number of content-free summaries to return (default 50, max\n100). Ordinary agent transports are scoped to their own lease.","format":"uint32","minimum":0,"type":["integer","null"]}},"required":[],"type":"object"},"name":"list_sessions","read_only":true},{"description":"Deprecated compatibility alias that reads a live legacy session's capture policy. Use get_session for lifecycle state.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":true,"properties":{"session":{"description":"Optional public label. When omitted, inspect the caller's attached\nimplicit session.","type":"string"}},"required":[],"type":"object"},"name":"get_session_state","read_only":true},{"description":"End one visible lifecycle session and run its cursor, recording, configuration, and other cleanup hooks exactly once. Omit `session` to end the authenticated transport's implicit session. Idempotent.","destructive":true,"idempotent":true,"input_schema":{"additionalProperties":true,"properties":{"session":{"description":"Optional public label to end. When omitted, end the caller's attached\nimplicit session.","type":"string"}},"required":[],"type":"object"},"name":"end_session","read_only":false},{"description":"Check the saved stable/nightly Cua Driver channel for a release on GitHub. Returns current and selected channels, current and latest versions, an `update_available` boolean, the install one-liner, and the release notes URL. Read-only — never installs. Pacman-owned Linux executables return package-manager guidance without checking GitHub. Mirror of `cua-driver check-update --json`.","destructive":false,"idempotent":true,"input_schema":{"additionalProperties":false,"properties":{},"type":"object"},"name":"check_for_update","read_only":true}],"version":"0.28.1"} diff --git a/internal/desktop/testdata/snapshot-tree.stdout b/internal/desktop/testdata/snapshot-tree.stdout new file mode 100644 index 0000000..03acb57 --- /dev/null +++ b/internal/desktop/testdata/snapshot-tree.stdout @@ -0,0 +1,296 @@ +{ + "_note": "Prefer `elements` — `tree_markdown` will continue to work but new fields will only be added to the structured side. Use `max_elements` / `max_depth` to bound the AT-SPI walk on apps with very large trees.", + "app_name": "gnome-text-editor", + "element_count": 13, + "elements": [ + { + "actions": [ + "page.move-right", + "settings.show-right-margin", + "page.save-as", + "win.show-preferences", + "win.close-current-page", + "page.move-to-new-window", + "settings.spellcheck", + "win.close-other-pages", + "session.new-draft", + "win.focus-search", + "default.activate", + "page.zoom-one", + "settings.discover-settings", + "window.minimize", + "win.open", + "page.begin-search", + "page.confirm-discard-changes", + "win.close-page-or-window", + "page.copy-all", + "win.alternate-help-overlay", + "page.zoom-in", + "window.close", + "page.properties", + "page.print", + "page.move-left", + "settings.wrap-text", + "page.save", + "page.zoom-out", + "page.confirm-save", + "win.show-primary-menu", + "page.begin-replace", + "window.toggle-maximized", + "page.change-language", + "settings.auto-indent", + "settings.show-line-numbers" + ], + "depth": 0, + "element_index": 0, + "element_token": "s00000001:0", + "enabled": true, + "frame": { + "h": 520, + "w": 700, + "x": 322, + "y": 114 + }, + "label": "New Document (Draft) - Text Editor", + "role": "application" + }, + { + "actions": [ + "search.hide", + "page.goto-line", + "view.auto-indent", + "page.show-goto-line" + ], + "depth": 10, + "element_index": 1, + "element_token": "s00000001:1", + "enabled": true, + "frame": { + "h": 474, + "w": 700, + "x": 322, + "y": 160 + }, + "parent_index": 0, + "role": "generic" + }, + { + "actions": [ + "spelling.enabled", + "buffer.delete-line", + "buffer.select-line", + "misc.insert-emoji", + "page.zoom-in", + "selection.select-all", + "page.zoom-one", + "clipboard.paste", + "menu.popup", + "clipboard.copy", + "page.zoom-out", + "selection.delete", + "buffer.duplicate-line", + "clipboard.cut" + ], + "depth": 15, + "element_index": 2, + "element_token": "s00000001:2", + "enabled": true, + "frame": { + "h": 474, + "w": 700, + "x": 322, + "y": 160 + }, + "parent_index": 1, + "role": "text box" + }, + { + "depth": 14, + "element_index": 3, + "element_token": "s00000001:3", + "enabled": true, + "frame": { + "h": 34, + "w": 80, + "x": 328, + "y": 120 + }, + "label": "Open", + "parent_index": 1, + "role": "button" + }, + { + "actions": [ + "click" + ], + "depth": 15, + "element_index": 4, + "element_token": "s00000001:4", + "enabled": true, + "frame": { + "h": 34, + "w": 80, + "x": 337, + "y": 125 + }, + "label": "Open", + "parent_index": 3, + "role": "toggle button" + }, + { + "actions": [ + "click" + ], + "depth": 14, + "element_index": 5, + "element_token": "s00000001:5", + "enabled": true, + "frame": { + "h": 34, + "w": 34, + "x": 419, + "y": 125 + }, + "label": "New tab", + "parent_index": 1, + "role": "button" + }, + { + "depth": 14, + "element_index": 6, + "element_token": "s00000001:6", + "enabled": true, + "frame": { + "h": 34, + "w": 34, + "x": 828, + "y": 120 + }, + "label": "View", + "parent_index": 1, + "role": "button" + }, + { + "actions": [ + "click" + ], + "depth": 15, + "element_index": 7, + "element_token": "s00000001:7", + "enabled": true, + "frame": { + "h": 34, + "w": 34, + "x": 833, + "y": 125 + }, + "label": "View", + "parent_index": 6, + "role": "toggle button" + }, + { + "depth": 14, + "element_index": 8, + "element_token": "s00000001:8", + "enabled": true, + "frame": { + "h": 34, + "w": 34, + "x": 868, + "y": 120 + }, + "label": "Main menu", + "parent_index": 1, + "role": "button" + }, + { + "actions": [ + "click" + ], + "depth": 15, + "element_index": 9, + "element_token": "s00000001:9", + "enabled": true, + "frame": { + "h": 34, + "w": 34, + "x": 873, + "y": 125 + }, + "label": "Main menu", + "parent_index": 8, + "role": "toggle button" + }, + { + "actions": [ + "click" + ], + "depth": 14, + "element_index": 10, + "element_token": "s00000001:10", + "enabled": true, + "frame": { + "h": 34, + "w": 34, + "x": 913, + "y": 125 + }, + "label": "Minimize", + "parent_index": 1, + "role": "button" + }, + { + "actions": [ + "click" + ], + "depth": 14, + "element_index": 11, + "element_token": "s00000001:11", + "enabled": true, + "frame": { + "h": 34, + "w": 34, + "x": 950, + "y": 125 + }, + "label": "Maximize", + "parent_index": 1, + "role": "button" + }, + { + "actions": [ + "click" + ], + "depth": 14, + "element_index": 12, + "element_token": "s00000001:12", + "enabled": true, + "frame": { + "h": 34, + "w": 34, + "x": 987, + "y": 125 + }, + "label": "Close", + "parent_index": 1, + "role": "button" + } + ], + "elements_complete": false, + "pid": 1220, + "returned_element_count": 13, + "screenshot_height": 642, + "screenshot_mime_type": "image/png", + "screenshot_width": 822, + "snapshot_id": "s00000001", + "total_element_count": 13, + "tree_markdown": "- [0] application \"New Document (Draft) - Text Editor\" [actions=[page.move-right,settings.show-right-margin,page.save-as,win.show-preferences,win.close-current-page,page.move-to-new-window,settings.spellcheck,win.close-other-pages,session.new-draft,win.focus-search,default.activate,page.zoom-one,settings.discover-settings,window.minimize,win.open,page.begin-search,page.confirm-discard-changes,win.close-page-or-window,page.copy-all,win.alternate-help-overlay,page.zoom-in,window.close,page.properties,page.print,page.move-left,settings.wrap-text,page.save,page.zoom-out,page.confirm-save,win.show-primary-menu,page.begin-replace,window.toggle-maximized,page.change-language,settings.auto-indent,settings.show-line-numbers]]\n - tab panel = \"New Document\"\n - [1] generic \"\" [actions=[search.hide,page.goto-line,view.auto-indent,page.show-goto-line]]\n - [2] text box \"\" [actions=[spelling.enabled,buffer.delete-line,buffer.select-line,misc.insert-emoji,page.zoom-in,selection.select-all,page.zoom-one,clipboard.paste,menu.popup,clipboard.copy,page.zoom-out,selection.delete,buffer.duplicate-line,clipboard.cut]]\n - [3] button \"Open\" [actions=[]]\n - [4] toggle button \"Open\" [actions=[click]]\n - label = \"Open\"\n - [5] button \"New tab\" [actions=[click]]\n - label = \"New Document\"\n - [6] button \"View\" [actions=[]]\n - [7] toggle button \"View\" [actions=[click]]\n - [8] button \"Main menu\" [actions=[]]\n - [9] toggle button \"Main menu\" [actions=[click]]\n - [10] button \"Minimize\" [actions=[click]]\n - [11] button \"Maximize\" [actions=[click]]\n - [12] button \"Close\" [actions=[click]]\n", + "window_bounds": { + "height": 642, + "width": 822, + "x": 261, + "y": 59 + }, + "window_id": 31457284, + "window_title": "New Document (Draft) - Text Editor" +} diff --git a/internal/incus/instance.go b/internal/incus/instance.go index 0115cde..039acac 100644 --- a/internal/incus/instance.go +++ b/internal/incus/instance.go @@ -55,7 +55,11 @@ func (c *Client) BeginCreateInstance(ctx context.Context, req compute.CreateInst return nil, err } - source, err := c.instanceSource(ctx, projectName(req.Ref.Sandbox), req.Image) + kind := api.InstanceTypeContainer + if req.Kind == kindVM { + kind = api.InstanceTypeVM + } + source, err := c.instanceSource(ctx, projectName(req.Ref.Sandbox), req.Image, kind) if err != nil { return nil, err } @@ -84,11 +88,6 @@ func (c *Client) BeginCreateInstance(ctx context.Context, req compute.CreateInst } } - kind := api.InstanceTypeContainer - if req.Kind == kindVM { - kind = api.InstanceTypeVM - } - op, err := c.Scoped(ctx, projectName(req.Ref.Sandbox), host).CreateInstance(api.InstancesPost{ Name: req.Ref.Name, Type: kind, @@ -368,6 +367,7 @@ func (c *Client) instanceSource( ctx context.Context, project string, image compute.CatalogImage, + kind api.InstanceType, ) (api.InstanceSource, error) { if isSandboxImage(image) { return api.InstanceSource{ @@ -376,27 +376,7 @@ func (c *Client) instanceSource( }, nil } if isUpstreamRef(image.Reference) { - remote, alias, _ := splitRemoteAlias(image.Reference) - server, err := c.RemoteImage(ctx, remote) - if err != nil { - return api.InstanceSource{}, err - } - info, err := server.GetConnectionInfo() - if err != nil { - return api.InstanceSource{}, mapError(err) - } - source := api.InstanceSource{ - Type: sourceTypeImage, - Alias: alias, - Server: info.URL, - Protocol: info.Protocol, - Certificate: info.Certificate, - } - if image.Fingerprint != "" { - source.Fingerprint = image.Fingerprint - source.Alias = "" - } - return source, nil + return c.upstreamInstanceSource(ctx, image, kind) } fingerprint, err := c.copyImage(ctx, project, image) @@ -409,6 +389,87 @@ func (c *Client) instanceSource( }, nil } +func (c *Client) upstreamInstanceSource( + ctx context.Context, + image compute.CatalogImage, + kind api.InstanceType, +) (api.InstanceSource, error) { + remote, alias, _ := splitRemoteAlias(image.Reference) + server, err := c.RemoteImage(ctx, remote) + if err != nil { + return api.InstanceSource{}, err + } + info, err := server.GetConnectionInfo() + if err != nil { + return api.InstanceSource{}, mapError(err) + } + source := api.InstanceSource{ + Type: sourceTypeImage, + Alias: alias, + Server: info.URL, + Protocol: info.Protocol, + Certificate: info.Certificate, + } + if image.Fingerprint != "" { + source.Fingerprint = image.Fingerprint + source.Alias = "" + } + if _, native := server.(*incusclient.ProtocolIncus); !native { + return source, nil + } + return nativeInstanceSource(server, source, kind, info.Project) +} + +func nativeInstanceSource( + server incusclient.ImageServer, + source api.InstanceSource, + kind api.InstanceType, + project string, +) (api.InstanceSource, error) { + target := source.Alias + if source.Fingerprint != "" { + target = source.Fingerprint + } + resolved, err := resolveNativeImage(server, target, source.Alias, kind) + if err != nil { + return api.InstanceSource{}, err + } + source.Fingerprint = resolved.Fingerprint + source.Alias = "" + source.Project = project + if resolved.Public { + return source, nil + } + source.Secret, err = server.GetImageSecret(resolved.Fingerprint) + if err != nil { + return api.InstanceSource{}, mapError(err) + } + return source, nil +} + +func resolveNativeImage( + server incusclient.ImageServer, + target, alias string, + kind api.InstanceType, +) (*api.Image, error) { + resolved, _, err := server.GetImage(target) + if err == nil { + return resolved, nil + } + if !errors.Is(mapError(err), compute.ErrNotFound) || alias == "" { + return nil, mapError(err) + } + entry, _, aliasErr := server.GetImageAliasType(string(kind), alias) + if aliasErr != nil { + return nil, mapError(aliasErr) + } + resolved, _, err = server.GetImage(entry.Target) + if err != nil { + return nil, mapError(err) + } + return resolved, nil +} + func isSandboxImage(image compute.CatalogImage) bool { return image.Fingerprint != "" && image.Reference == "" } diff --git a/internal/incus/instance_file.go b/internal/incus/instance_file.go index 9ac1840..8a41ce2 100644 --- a/internal/incus/instance_file.go +++ b/internal/incus/instance_file.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "io" + "os" "strconv" "strings" @@ -18,33 +19,16 @@ const defaultFileReadLimit = 64 * 1024 // ReadFile pulls a bounded guest file through the Incus agent. func (c *Client) ReadFile(ctx context.Context, req compute.FileReadRequest) (compute.FileReadResult, error) { - if err := c.requireInstance(ctx, req.Ref); err != nil { - return compute.FileReadResult{}, err - } - srv := c.Scoped(ctx, projectName(req.Ref.Sandbox), "") - body, info, err := srv.GetInstanceFile(req.Ref.Name, req.Path) + body, err := c.ReadBinaryFile(ctx, req.Ref, req.Path) if err != nil { - if errors.Is(mapError(err), compute.ErrNotFound) { + if errors.Is(err, os.ErrNotExist) { return compute.FileReadResult{}, fileErrorf( "file not found on instance %q in sandbox %q", req.Ref.Name, req.Ref.Sandbox, ) } - return compute.FileReadResult{}, mapError(err) - } - if info != nil && info.Type == "directory" { - if body != nil { - _ = body.Close() - } - return compute.FileReadResult{}, fileErrorf( - "path is a directory on instance %q in sandbox %q", - req.Ref.Name, - req.Ref.Sandbox, - ) - } - if body == nil { - return compute.FileReadResult{}, errors.New("file read returned no content") + return compute.FileReadResult{}, err } defer body.Close() @@ -63,6 +47,32 @@ func (c *Client) ReadFile(ctx context.Context, req compute.FileReadRequest) (com return compute.FileReadResult{Content: string(buf), Truncated: truncated}, nil } +// ReadBinaryFile opens a guest file without passing bytes through exec or text conversion. +// The caller must close the stream and enforce its own byte bound. +func (c *Client) ReadBinaryFile(ctx context.Context, ref compute.Ref, path string) (io.ReadCloser, error) { + if err := c.requireInstance(ctx, ref); err != nil { + return nil, err + } + srv := c.Scoped(ctx, projectName(ref.Sandbox), "") + body, info, err := srv.GetInstanceFile(ref.Name, path) + if err != nil { + if errors.Is(mapError(err), compute.ErrNotFound) { + return nil, fmt.Errorf("guest file missing: %w", os.ErrNotExist) + } + return nil, mapError(err) + } + if info != nil && info.Type != "file" { + if body != nil { + _ = body.Close() + } + return nil, fileErrorf("path is not a regular file on instance %q in sandbox %q", ref.Name, ref.Sandbox) + } + if body == nil { + return nil, errors.New("file read returned no content") + } + return body, nil +} + // WriteFile pushes a bounded guest file through the Incus agent. func (c *Client) WriteFile(ctx context.Context, req compute.FileWriteRequest) (compute.FileWriteResult, error) { if err := c.requireInstance(ctx, req.Ref); err != nil { diff --git a/internal/incus/ovn_forward.go b/internal/incus/ovn_forward.go index d82b3d1..26bdada 100644 --- a/internal/incus/ovn_forward.go +++ b/internal/incus/ovn_forward.go @@ -10,6 +10,7 @@ import ( "strconv" "strings" + incusclient "github.com/lxc/incus/v7/client" "github.com/lxc/incus/v7/shared/api" "github.com/GilmanLab/agentcompute/internal/compute" @@ -27,16 +28,18 @@ func (c *Client) CreateForward( if _, err := c.ownedProjectNetwork(ctx, sandbox, network); err != nil { return compute.Forward{}, err } - target, err := c.forwardTargetAddress(ctx, sandbox, network, ref) + instance, err := c.GetInstance(ctx, ref) + if err != nil { + return compute.Forward{}, err + } + target, err := c.forwardTargetAddress(ctx, sandbox, network, instance) if err != nil { return compute.Forward{}, err } - listen := strconv.FormatInt(listenPort, 10) - targetPort := strconv.FormatInt(port, 10) portSpec := api.NetworkForwardPort{ Protocol: protocol, - ListenPort: listen, - TargetPort: targetPort, + ListenPort: strconv.FormatInt(listenPort, 10), + TargetPort: strconv.FormatInt(port, 10), TargetAddress: target, } @@ -46,29 +49,50 @@ func (c *Client) CreateForward( return compute.Forward{}, mapOVNError(err) } if len(forwards) > 0 { - existing := forwards[0] - if forwardHasPort(existing, protocol, listen) { - return compute.Forward{}, fmt.Errorf( - "listen port %s/%s is already forwarded on %s", - listen, - protocol, - existing.ListenAddress, - ) - } - writable := existing.Writable() - writable.Ports = append(writable.Ports, portSpec) - if err := srv.UpdateNetworkForward(network, existing.ListenAddress, writable, ""); err != nil { - return compute.Forward{}, mapOVNError(err) - } - return compute.Forward{ - Address: existing.ListenAddress, - Port: listenPort, - Protocol: protocol, - Network: network, - Instance: ref.Name, - }, nil + return addForwardPort(srv, network, forwards[0], portSpec, listenPort, protocol, ref.Name) + } + return c.createAllocatedForward(ctx, srv, sandbox, network, ref, portSpec, listenPort, protocol) +} + +func addForwardPort( + srv incusclient.InstanceServer, + network string, + existing api.NetworkForward, + portSpec api.NetworkForwardPort, + listenPort int64, + protocol, instance string, +) (compute.Forward, error) { + if forwardHasPort(existing, protocol, portSpec.ListenPort) { + return compute.Forward{}, fmt.Errorf( + "listen port %s/%s is already forwarded on %s", + portSpec.ListenPort, + protocol, + existing.ListenAddress, + ) + } + writable := existing.Writable() + writable.Ports = append(writable.Ports, portSpec) + if err := srv.UpdateNetworkForward(network, existing.ListenAddress, writable, ""); err != nil { + return compute.Forward{}, mapOVNError(err) } + return compute.Forward{ + Address: existing.ListenAddress, + Port: listenPort, + Protocol: protocol, + Network: network, + Instance: instance, + }, nil +} +func (c *Client) createAllocatedForward( + ctx context.Context, + srv incusclient.InstanceServer, + sandbox, network string, + ref compute.Ref, + portSpec api.NetworkForwardPort, + listenPort int64, + protocol string, +) (compute.Forward, error) { for { address, err := c.allocateForwardAddress(ctx) if err != nil { @@ -93,22 +117,30 @@ func (c *Client) CreateForward( } // Retry only a confirmed allocation race. A forward already on this // network may be an uncertain commit; never create a second one. - current, inspectErr := srv.GetNetworkForwards(network) - if inspectErr != nil || len(current) != 0 { - return compute.Forward{}, mapOVNError(err) - } - used, inspectErr := c.usedOVNAddresses(ctx) - if inspectErr != nil || !used[address] { + if !c.forwardAllocationRace(ctx, srv, network, address) { return compute.Forward{}, mapOVNError(err) } } } -func (c *Client) forwardTargetAddress(ctx context.Context, sandbox, network string, ref compute.Ref) (string, error) { - instance, err := c.GetInstance(ctx, ref) - if err != nil { - return "", err +func (c *Client) forwardAllocationRace( + ctx context.Context, + srv incusclient.InstanceServer, + network, address string, +) bool { + current, inspectErr := srv.GetNetworkForwards(network) + if inspectErr != nil || len(current) != 0 { + return false } + used, inspectErr := c.usedOVNAddresses(ctx) + return inspectErr == nil && used[address] +} + +func (c *Client) forwardTargetAddress( + ctx context.Context, + sandbox, network string, + instance compute.Instance, +) (string, error) { for _, nic := range instance.NICs { if nic.Network != network { continue @@ -125,7 +157,7 @@ func (c *Client) forwardTargetAddress(ctx context.Context, sandbox, network stri return "", mapError(err) } for _, lease := range leases { - if lease.Hostname != ref.Name { + if lease.Hostname != instance.Ref.Name { continue } ip := net.ParseIP(lease.Address) @@ -133,7 +165,7 @@ func (c *Client) forwardTargetAddress(ctx context.Context, sandbox, network stri return ip.String(), nil } } - return "", fmt.Errorf("instance %q has no address on network %q", ref.Name, network) + return "", fmt.Errorf("instance %q has no address on network %q", instance.Ref.Name, network) } func (c *Client) allocateForwardAddress(ctx context.Context) (string, error) { @@ -185,6 +217,106 @@ func (c *Client) usedOVNAddresses(ctx context.Context) (map[string]bool, error) return used, nil } +// InstanceForward observes the scalar forward shape created by net.forward. +func (c *Client) InstanceForward( + ctx context.Context, + ref compute.Ref, + targetPort int64, + protocol string, +) (compute.Forward, error) { + inst, err := c.GetInstance(ctx, ref) + if err != nil { + return compute.Forward{}, err + } + networks, err := c.ListNetworks(ctx, ref.Sandbox) + if err != nil { + return compute.Forward{}, err + } + srv := c.Scoped(ctx, projectName(ref.Sandbox), "") + target := strconv.FormatInt(targetPort, 10) + for _, network := range networks { + forward, found, err := c.instanceForwardOnNetwork(ctx, srv, ref, inst, network, target, protocol) + if err != nil || found { + return forward, err + } + } + return compute.Forward{}, nil +} + +func (c *Client) instanceForwardOnNetwork( + ctx context.Context, + srv incusclient.InstanceServer, + ref compute.Ref, + inst compute.Instance, + network compute.Network, + target, protocol string, +) (compute.Forward, bool, error) { + if network.Kind != networkKindOVN { + return compute.Forward{}, false, nil + } + for _, nic := range inst.NICs { + if nic.Network != network.Name { + continue + } + forwards, err := srv.GetNetworkForwards(network.Name) + if err != nil { + return compute.Forward{}, false, mapOVNError(err) + } + if len(forwards) == 0 { + continue + } + targetAddress, err := c.forwardTargetAddress(ctx, ref.Sandbox, network.Name, inst) + if err != nil { + return compute.Forward{}, false, err + } + if forward, ok := matchingForward(forwards, protocol, target, targetAddress, network.Name, ref.Name); ok { + return forward, true, nil + } + } + return compute.Forward{}, false, nil +} + +func matchingForward( + forwards []api.NetworkForward, + protocol, target, targetAddress, network, instance string, +) (compute.Forward, bool) { + for _, forward := range forwards { + for _, port := range forward.Ports { + if !forwardPortMatches(forward, port, protocol, target, targetAddress) { + continue + } + listen, err := strconv.ParseInt(port.ListenPort, 10, 64) + if err != nil { + continue + } + return compute.Forward{ + Address: forward.ListenAddress, + Port: listen, + Protocol: protocol, + Network: network, + Instance: instance, + }, true + } + } + return compute.Forward{}, false +} + +func forwardPortMatches( + forward api.NetworkForward, + port api.NetworkForwardPort, + protocol, target, targetAddress string, +) bool { + address := port.TargetAddress + if address == "" { + address = forward.Config["target_address"] + } + mapped := port.TargetPort + if mapped == "" { + mapped = port.ListenPort + } + return port.Protocol == protocol && mapped == target && address == targetAddress +} + func (c *Client) deleteForwardsInProject(ctx context.Context, project, network string) []error { ctx, cancel := c.ovnContext(ctx) defer cancel() diff --git a/internal/incus/reconcile.go b/internal/incus/reconcile.go index f3f513d..545849e 100644 --- a/internal/incus/reconcile.go +++ b/internal/incus/reconcile.go @@ -13,6 +13,7 @@ import ( "os" "path/filepath" "slices" + "strconv" "strings" "time" "unicode" @@ -45,6 +46,7 @@ const ( startAction = "start" stopAction = "stop" trueCommand = "/bin/true" + desktopAutomationID = 1000 ) const ( @@ -122,7 +124,7 @@ func (c *Client) ensureDigestImage(ctx context.Context, image compute.CatalogIma if err != nil { return "", fmt.Errorf("catalog image %q: %w", image.Name, err) } - if err := smokeLaunch(ctx, server, image.Name, fingerprint, image.Kind == kindVM); err != nil { + if err := smokeLaunch(ctx, server, image, fingerprint); err != nil { return "", fmt.Errorf("catalog image %q: %w", image.Name, err) } if err := recordDigest(server, fingerprint, digest); err != nil { @@ -343,10 +345,10 @@ func recordDigest(server incusclient.InstanceServer, fingerprint, digest string) func smokeLaunch( ctx context.Context, server incusclient.InstanceServer, - imageName, fingerprint string, - vm bool, + image compute.CatalogImage, + fingerprint string, ) (err error) { - name, err := smokeInstanceName(imageName) + name, err := smokeInstanceName(image.Name) if err != nil { return err } @@ -356,13 +358,17 @@ func smokeLaunch( Source: api.InstanceSource{Type: sourceTypeImage, Fingerprint: fingerprint}, } req.Profiles = []string{imageBuildProfile} - checks := routerChecks() - if vm { + if image.Kind == kindVM { req.Type = api.InstanceTypeVM req.Profiles = []string{"runner-smoke"} - checks = [][]string{ - {"systemctl", "is-active", "incus-gh-runner-guest.path"}, - {"test", "-x", "/opt/actions-runner/bin/Runner.Listener"}, + } + if image.Desktop { + req.Config = api.ConfigMap{} + if image.CPUs > 0 { + req.Config["limits.cpu"] = strconv.FormatInt(image.CPUs, 10) + } + if image.MemoryMB > 0 { + req.Config["limits.memory"] = fmt.Sprintf("%dMiB", image.MemoryMB) } } op, err := server.CreateInstance(req) @@ -388,17 +394,58 @@ func smokeLaunch( if err = startOp.WaitContext(smokeCtx); err != nil { return err } - if err = waitGuestReady(smokeCtx, server, name); err != nil { + if err = waitGuestCommand( + smokeCtx, + server, + name, + api.InstanceExecPost{Command: []string{trueCommand}}, + ); err != nil { return err } + if image.Desktop { + return smokeDesktop(smokeCtx, server, name) + } + var checks [][]string + if image.Kind == kindVM { + checks = [][]string{ + {"systemctl", "is-active", "incus-gh-runner-guest.path"}, + {"test", "-x", "/opt/actions-runner/bin/Runner.Listener"}, + } + } else { + checks = routerChecks() + } for _, command := range checks { - if err = execCommand(smokeCtx, server, name, command); err != nil { + if err = execCommand(smokeCtx, server, name, api.InstanceExecPost{Command: command}); err != nil { return fmt.Errorf("smoke check %s: %w", strings.Join(command, " "), err) } } return nil } +func smokeDesktop(ctx context.Context, server incusclient.InstanceServer, name string) error { + checks := []api.InstanceExecPost{ + {Command: []string{"test", "-S", "/tmp/.X11-unix/X0"}}, + {Command: []string{"systemctl", "--user", "--machine=automation@", "is-active", "cua-driver.service"}}, + { + Command: []string{ + "/usr/local/bin/cua-driver", "call", "--socket", "/run/user/1000/cua-driver.sock", "list_apps", "{}", + }, + User: desktopAutomationID, Group: desktopAutomationID, Cwd: "/home/automation", + Environment: map[string]string{ + "HOME": "/home/automation", + "XDG_RUNTIME_DIR": "/run/user/1000", + "DBUS_SESSION_BUS_ADDRESS": "unix:path=/run/user/1000/bus", + }, + }, + } + for _, check := range checks { + if err := waitGuestCommand(ctx, server, name, check); err != nil { + return fmt.Errorf("desktop smoke check %s: %w", strings.Join(check.Command, " "), err) + } + } + return nil +} + func promoteAlias(server incusclient.InstanceServer, name, fingerprint string) error { alias, etag, err := server.GetImageAlias(name) if isNotFound(err) { @@ -457,8 +504,13 @@ func imageWithDigest(server incusclient.InstanceServer, digest string) (string, return "", false, nil } -func waitGuestReady(ctx context.Context, server incusclient.InstanceServer, name string) error { - if err := execCommand(ctx, server, name, []string{trueCommand}); err == nil { +func waitGuestCommand( + ctx context.Context, + server incusclient.InstanceServer, + name string, + request api.InstanceExecPost, +) error { + if err := execCommand(ctx, server, name, request); err == nil { return nil } ticker := time.NewTicker(readyPollInterval) @@ -466,16 +518,21 @@ func waitGuestReady(ctx context.Context, server incusclient.InstanceServer, name for { select { case <-ctx.Done(): - return fmt.Errorf("instance %s was not reachable: %w", name, ctx.Err()) + return fmt.Errorf("instance %s did not pass guest check: %w", name, ctx.Err()) case <-ticker.C: - if err := execCommand(ctx, server, name, []string{trueCommand}); err == nil { + if err := execCommand(ctx, server, name, request); err == nil { return nil } } } } -func execCommand(ctx context.Context, server incusclient.InstanceServer, name string, command []string) error { +func execCommand( + ctx context.Context, + server incusclient.InstanceServer, + name string, + request api.InstanceExecPost, +) error { done := make(chan bool) args := &incusclient.InstanceExecArgs{ Stdin: bytes.NewReader(nil), @@ -483,10 +540,8 @@ func execCommand(ctx context.Context, server incusclient.InstanceServer, name st Stderr: io.Discard, DataDone: done, } - op, err := server.ExecInstance(name, api.InstanceExecPost{ - Command: command, - WaitForWS: true, - }, args) + request.WaitForWS = true + op, err := server.ExecInstance(name, request, args) if err != nil { return err } diff --git a/internal/mcpserver/contract_test.go b/internal/mcpserver/contract_test.go index c12ca52..a3ed68b 100644 --- a/internal/mcpserver/contract_test.go +++ b/internal/mcpserver/contract_test.go @@ -485,6 +485,63 @@ func capabilityContracts() []capabilityContract { }, output: []fieldShape{}, }, + { + name: capabilityDesktopInfo, + signature: "desktop.info(*, sandbox: str, instance: str)", + input: []fieldShape{ + {Name: "sandbox", Type: "str", Required: true}, + {Name: "instance", Type: "str", Required: true}, + }, + output: []fieldShape{ + {Name: "ready", Type: "bool", Required: true}, + {Name: "os", Type: "str", Required: true}, + {Name: "driver_version", Type: "str", Required: true}, + {Name: "tools", Type: "list[str]", Required: true}, + {Name: "vnc", Type: "str"}, + }, + }, + { + name: capabilityDesktopEnable, + signature: "desktop.enable(*, sandbox: str, instance: str)", + input: []fieldShape{ + {Name: "sandbox", Type: "str", Required: true}, + {Name: "instance", Type: "str", Required: true}, + }, + output: []fieldShape{{Name: "ready", Type: "bool", Required: true}}, + }, + { + name: capabilityDesktopCall, + signature: "desktop.call(*, sandbox: str, instance: str, tool: str, args: str | None)", + input: []fieldShape{ + {Name: "sandbox", Type: "str", Required: true}, + {Name: "instance", Type: "str", Required: true}, + {Name: "tool", Type: "str", Required: true}, + {Name: "args", Type: "str | None"}, + }, + output: []fieldShape{ + {Name: "ok", Type: "bool", Required: true}, + {Name: "summary", Type: "str", Required: true}, + {Name: "result", Type: "str", Required: true}, + {Name: "screenshot_url", Type: "str"}, + }, + }, + { + name: capabilityDesktopScreenshot, + signature: "desktop.screenshot(*, sandbox: str, instance: str, pid: int | None, window_id: int | None, max_dimension: int | None)", + input: []fieldShape{ + {Name: "sandbox", Type: "str", Required: true}, + {Name: "instance", Type: "str", Required: true}, + {Name: "pid", Type: "int | None"}, + {Name: "window_id", Type: "int | None"}, + {Name: "max_dimension", Type: "int | None"}, + }, + output: []fieldShape{ + {Name: "url", Type: "str", Required: true}, + {Name: "width", Type: "int", Required: true}, + {Name: "height", Type: "int", Required: true}, + {Name: "scale", Type: "float", Required: true}, + }, + }, } } diff --git a/internal/mcpserver/desktop.go b/internal/mcpserver/desktop.go new file mode 100644 index 0000000..398e702 --- /dev/null +++ b/internal/mcpserver/desktop.go @@ -0,0 +1,158 @@ +package mcpserver + +import ( + "context" + + "github.com/meigma/codemode" + "github.com/meigma/codemode/authz" + + "github.com/GilmanLab/agentcompute/internal/compute" + "github.com/GilmanLab/agentcompute/internal/desktop" +) + +const ( + capabilityDesktopInfo = "desktop.info" + capabilityDesktopEnable = "desktop.enable" + capabilityDesktopCall = "desktop.call" + capabilityDesktopScreenshot = "desktop.screenshot" +) + +type desktopService interface { + Info(context.Context, compute.Ref) (desktop.Info, error) + Enable(context.Context, compute.Ref) (bool, error) + Call(context.Context, compute.Ref, string, string) (desktop.CallResult, error) + Screenshot(context.Context, compute.Ref, int64, int64, int64) (desktop.Screenshot, error) +} + +type desktopInstanceIn struct { + Sandbox string `json:"sandbox"` + Instance string `json:"instance"` +} + +type desktopInfoOut struct { + Ready bool `json:"ready"` + OS string `json:"os"` + DriverVersion string `json:"driver_version"` + Tools []string `json:"tools"` + VNC *string `json:"vnc,omitempty"` +} + +type desktopEnableOut struct { + Ready bool `json:"ready"` +} + +type desktopCallIn struct { + Sandbox string `json:"sandbox"` + Instance string `json:"instance"` + Tool string `json:"tool"` + Args *string `json:"args,omitempty"` +} + +type desktopCallOut struct { + OK bool `json:"ok"` + Summary string `json:"summary"` + Result string `json:"result"` + ScreenshotURL *string `json:"screenshot_url,omitempty"` +} + +type desktopScreenshotIn struct { + Sandbox string `json:"sandbox"` + Instance string `json:"instance"` + PID *int64 `json:"pid,omitempty"` + WindowID *int64 `json:"window_id,omitempty"` + MaxDimension *int64 `json:"max_dimension,omitempty"` +} + +type desktopScreenshotOut struct { + URL string `json:"url"` + Width int64 `json:"width"` + Height int64 `json:"height"` + Scale float64 `json:"scale"` +} + +type desktopAPI struct { + driver desktopService +} + +func registerDesktop(builder *codemode.Builder, deps Dependencies) { + api := desktopAPI{driver: deps.Desktop} + codemode.Register(builder, codemode.Capability[desktopInstanceIn, desktopInfoOut]{ + ID: capabilityDesktopInfo, Name: capabilityDesktopInfo, + Summary: "Inspect Driver readiness, version, native tool names, and the human VNC endpoint.", + Handler: api.info, + }) + codemode.Register(builder, codemode.Capability[desktopInstanceIn, desktopEnableOut]{ + ID: capabilityDesktopEnable, Name: capabilityDesktopEnable, + Summary: "Verify the installed Driver daemon answers in the guest graphical session.", + Handler: api.enable, + }) + codemode.Register(builder, codemode.Capability[desktopCallIn, desktopCallOut]{ + ID: capabilityDesktopCall, + Name: capabilityDesktopCall, + Summary: "Call a native Driver tool with a JSON object string; decode result with json.decode. Screenshots return URLs only. Linux token clicks also need pid; snapshot again to verify effects.", + Handler: api.call, + }) + codemode.Register(builder, codemode.Capability[desktopScreenshotIn, desktopScreenshotOut]{ + ID: capabilityDesktopScreenshot, + Name: capabilityDesktopScreenshot, + Summary: "Capture the desktop or a pid/window_id pair without an accessibility tree. Return a PNG URL, dimensions, and image-to-Driver coordinate scale.", + Handler: api.screenshot, + }) +} + +func (api desktopAPI) info(ctx context.Context, _ authz.Subject, in desktopInstanceIn) (desktopInfoOut, error) { + info, err := api.driver.Info(ctx, compute.Ref{Sandbox: in.Sandbox, Name: in.Instance}) + if err != nil { + return desktopInfoOut{}, err + } + out := desktopInfoOut{Ready: info.Ready, OS: info.OS, DriverVersion: info.DriverVersion, Tools: info.Tools} + if info.VNC != "" { + out.VNC = &info.VNC + } + return out, nil +} + +func (api desktopAPI) enable(ctx context.Context, _ authz.Subject, in desktopInstanceIn) (desktopEnableOut, error) { + ready, err := api.driver.Enable(ctx, compute.Ref{Sandbox: in.Sandbox, Name: in.Instance}) + return desktopEnableOut{Ready: ready}, err +} + +func (api desktopAPI) call(ctx context.Context, _ authz.Subject, in desktopCallIn) (desktopCallOut, error) { + result, err := api.driver.Call( + ctx, + compute.Ref{Sandbox: in.Sandbox, Name: in.Instance}, + in.Tool, + deref(in.Args, "{}"), + ) + if err != nil { + return desktopCallOut{}, err + } + out := desktopCallOut{OK: result.OK, Summary: result.Summary, Result: result.Result} + if result.ScreenshotURL != "" { + out.ScreenshotURL = &result.ScreenshotURL + } + return out, nil +} + +func (api desktopAPI) screenshot( + ctx context.Context, + _ authz.Subject, + in desktopScreenshotIn, +) (desktopScreenshotOut, error) { + shot, err := api.driver.Screenshot( + ctx, + compute.Ref{Sandbox: in.Sandbox, Name: in.Instance}, + deref(in.PID, 0), + deref(in.WindowID, 0), + deref(in.MaxDimension, 0), + ) + if err != nil { + return desktopScreenshotOut{}, err + } + return desktopScreenshotOut{ + URL: shot.URL, + Width: int64(shot.Width), + Height: int64(shot.Height), + Scale: shot.Scale, + }, nil +} diff --git a/internal/mcpserver/server.go b/internal/mcpserver/server.go index b2b7390..4e97438 100644 --- a/internal/mcpserver/server.go +++ b/internal/mcpserver/server.go @@ -118,6 +118,9 @@ type Dependencies struct { // Image is the catalog service consumed by image.list and instance.create. Image imageService + + // Desktop proxies the guest Driver and publishes screenshot URLs. + Desktop desktopService } // Options configures the agentcompute MCP server. @@ -152,12 +155,13 @@ type Options struct { } // NewDependencies adapts a compute service to the handler consumer interfaces. -func NewDependencies(svc *compute.Service) Dependencies { +func NewDependencies(svc *compute.Service, driver desktopService) Dependencies { return Dependencies{ Sandbox: svc, Instance: svc, Network: svc, Image: svc, + Desktop: driver, } } @@ -180,6 +184,7 @@ func New(options Options) (*mcp.Server, error) { registerImage(builder, options.Deps) registerInstance(builder, options.Deps) registerNet(builder, options.Deps) + registerDesktop(builder, options.Deps) service, err := builder.Build() if err != nil { return nil, fmt.Errorf("build CodeMode runtime: %w", err) diff --git a/spikes/desktop/README.md b/spikes/desktop/README.md index 3f356ac..cfd5da8 100644 --- a/spikes/desktop/README.md +++ b/spikes/desktop/README.md @@ -14,8 +14,12 @@ The results use these evidence sources: | `internal/desktop/testdata/` | Captured Driver catalog and result fixtures used by the host adapter, including the missing-`pid` diagnostic and successful click response. | | `/tmp/agentcompute-desktop-corrected-smoke/` | Corrected image boot: X11, active Driver user service, 234 apps from `list_apps`, and cleanup of smoke-owned resources. | | `/tmp/agentcompute-desktop-mcp-evidence/` | Before/after PNGs from the corrected image's real MCP token interaction. The full acceptance run passed in 158.45 s after OVN recovery. | +| `/tmp/agentcompute-desktop-mcp-final-evidence/` | Post-refactor repeat: fetched PNGs show a single document before the native token click and two document tabs afterward. Full acceptance passed in 160.32 s. | +| `/tmp/agentcompute-publisher-rollout/result.json` | Approved controller replacement, matching recovered identities, new publisher standby, and no-drift Terraform plan. | +| `/tmp/agentcompute-desktop-published-evidence/` | Protected bake 34872818589: four immutable releases, desktop build measurements, X11/Driver boot qualification, and verified fetch-back. | +| `/tmp/agentcompute-desktop-published-mcp-evidence/` | Exact catalog from PR #27, without GHCR credentials: full MCP acceptance passed in 182.26 s. Fetched PNGs show one document before the native token click and two document tabs afterward. | -The first guest did not acquire its network until `/etc/netplan/10-incus.yaml` was repaired with `renderer: networkd`. The current recipe embeds that renderer and enables `systemd-networkd`. The corrected image passed the dedicated boot smoke with fingerprint `9d0da557210766289d59823a540050c3e7f328af66380974cc1ebd08e877bd11`; the token, screenshot, timing, and restart measurements below remain attributed to the repaired first guest. +The first guest did not acquire its network until `/etc/netplan/10-incus.yaml` was repaired with `renderer: networkd`. The current recipe embeds that renderer and enables `systemd-networkd`. The corrected image passed the dedicated boot smoke with fingerprint `9d0da557210766289d59823a540050c3e7f328af66380974cc1ebd08e877bd11`. Direct native measurements use the repaired first guest; the later MCP runs identify their image separately. ## Image and session contract @@ -100,6 +104,16 @@ The direct spike reached the management-network VNC endpoint at `10.10.40.65:590 The corrected-image MCP acceptance created a separate viewer on the default OVN network, created a TCP forward, restarted the guest, and observed `desktop.info.ready == true` again. The address reported by `desktop.info.vnc`, `10.10.40.67:5900`, answered with `RFB 003.008\n`. The representative client retained only its private LAN NIC. Its complete create/wait/screenshot/list-apps program took 23.624 s. After shortening sandbox lifetime, the running reaper made the original screenshot URL return 404 at 27.120 s after expiry. +The post-refactor repeat passed the same full scenario in 160.32 s. Its representative program took 22.764 s, and screenshot expiry returned 404 at 26.804 s after sandbox expiry. The reported VNC endpoint again answered at `10.10.40.67:5900`. + +The published-image run used the complete catalog from PR #27 without GHCR +credentials and passed in 182.26 s. Its representative program took +26.822 s. One native foreground token click changed the editor from a +single document to two document tabs, confirmed by a fresh accessibility +tree and the fetched before/after PNGs. Reboot restored Driver readiness +and the reported VNC endpoint at `10.10.40.67:5900`; the running reaper +made the screenshot URL return 404 at 21.773 s after sandbox expiry. + ## Build measurements and release status Both local builds used fresh work and output directories. Download time includes the Go toolchain, vendored distrobuilder source, Ubuntu base, snapshot CA package, and full Cua Driver archive. Compile time excludes downloads and assembly. Scratch usage was sampled every 100 ms, so an interval peak can be missed. @@ -108,11 +122,30 @@ Both local builds used fresh work and output directories. Download time includes | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | First image | 5.849 s | 31.670 s | 842.017 s | 507,180 KiB | 7,625,043,968 B | 656 B | 744,611,840 B | | Corrected rebuild | 5.824 s | 31.834 s | 392.512 s | 517,712 KiB | 7,621,808,128 B | 640 B | 745,013,248 B | +| Protected published build | 6.624 s | 33.283 s | 403.364 s | 505,216 KiB | 7,620,071,424 B | 628 B | 742,923,264 B | The first image artifacts were `incus.tar.xz` SHA-256 `3174b0a6e76d6e1b3e601a7ffc761589615205fed0124bd1f300ed52aa612f54` and `disk.qcow2` SHA-256 `ed3cfd45045f459b5b17ac5782a2e8f17f10e06b5805719ccbdf27a04bd22f37`. The corrected artifacts were `incus.tar.xz` SHA-256 `d7e5e20009425f2d2164770797e84e5d99611d0fe0d5a0b0e719032aa5c9cd9d` and `disk.qcow2` SHA-256 `42c8bdd02f8991822ecdb2093d02b760f781ab21e301a15ed1466830c2d36c2c`. Both qcow2 files report a 17,179,869,184-byte virtual size. -Protected bootstrap PR #23 and private bake run 34854323245 completed successfully. The desktop-aware publisher awaits deployment before the four-image bake. Desktop publication and catalog promotion remain pending; no desktop GHCR digest is claimed. +Protected bootstrap PR #23 and private bake run 34854323245 completed successfully. The approved publisher rollout replaced `ghrunner01` through its existing Terraform module, restored both escrowed identities, and qualified a standby with fingerprint `c6b3815e002e101d256980a1a80e55bfd8053ea3dea324383cd0f8fd43ddc60e`. Squid and scheduling recovered, and the post-apply plan reported no changes. + +Image PR #25 merged as `e4333f245b4e81c8d7753038f0ddf04a620bd0a2`. [Protected bake 34872818589](https://github.com/GilmanLab/agentcompute-images/actions/runs/34872818589) built, boot-qualified, published, and fetched back all four images. The desktop pipeline took 674.806 s, including those stages. Its immutable reference is: + +```text +ghcr.io/gilmanlab/agentcompute/ubuntu-24.04-desktop@sha256:5dc4e120a79dd06ad6784e69474f0617387f74cb98685af8844170b7165ea8e2 +``` + +The published desktop artifacts are `incus.tar.xz` SHA-256 `c673c40c973731471f404bd59a8cfebfeb043bdd9a20c2076894b13e48fa75d6` and `disk.qcow2` SHA-256 `40abe7aea48ebe8bb989cbe71afc4b4376a0ecb5dd23ed2630ba73985de7ed0d`; the split-image Incus fingerprint is `cc9ed27aa5cde44aaf075cc78474b038011172c979549ae43cf538fc5ece9a93`. Qualification observed X11, the active automation-user Driver service, and 232 apps including `gnome-text-editor`, then removed its owned VM and image. [Catalog PR #27](https://github.com/GilmanLab/agentcompute/pull/27) contains the real digest and remains unmerged by request. + +The first anonymous catalog-backed attempt could not import the new desktop +package because GitHub created it private. The owner made it public. The +unfixed server then downloaded the published image but rejected it with +`smoke check systemctl is-active incus-gh-runner-guest.path: exit 4`; the +reproduction failed in 68.06 s. Catalog qualification had treated every VM +as a GitHub runner. The reconciler now selects desktop qualification from +the catalog capability: X11, the active automation-user Driver unit, and +native `list_apps` executed as UID/GID 1000 with that user's session +environment. Router and runner checks are unchanged. -The original MCP blocker was a Phase 5 OVN outage: 19,307,134,976 bytes of logs filled central's 20 GiB root. The approved fleet recovery preserved complete signature counts and log boundaries before truncation, then recycled only the three Incus daemons retaining stale CA trust. No central database or northd process restarted. Logging limits are now active; details are in [fleet PR #20](https://github.com/GilmanLab/fleet/pull/20) and the central OVN runbook. +The original MCP blocker was a Phase 5 OVN outage: 19,307,134,976 bytes of logs filled central's 20 GiB root. The approved fleet recovery preserved complete signature counts and log boundaries before truncation, then recycled only the three Incus daemons retaining stale CA trust. No central database or northd process restarted. Logging limits are now active; details are in [fleet PR #20](https://github.com/GilmanLab/fleet/pull/20) and [the central OVN recovery runbook PR](https://github.com/GilmanLab/root/pull/34). Live acceptance also corrected two adapter boundaries. Native private Incus images need an image-access secret in the create request; the image existed even though the unauthenticated pull reported it missing. VNC forward discovery must use the same DHCP-lease fallback as forward creation because guest NIC state can briefly lack addresses after reboot. The passing run exercised both corrections without publishing the temporary image or delaying Driver readiness for networking.