Skip to content

Commit 3ba105a

Browse files
rgarciaclaude
andauthored
ai: tighten public surface and document CuaProvider + supported models (#7)
* ai: tighten public surface and document CuaProvider + supported models - Un-export parseCuaModelRef/formatCuaModelRef (internal helpers). - Add docs/supported-models.md enumerating CUA-supported models per provider with source citations, linked from README. - Add a CuaProvider section to the README explaining the type, its relationship to pi-ai's Provider, and the gemini/google rename. - Drop the now-redundant "See examples/quickstart.ts" line from the README quick start. * ai: rename gemini -> google, gate public API via named re-exports - Rename CuaProvider key from "gemini" to "google" so it matches pi-ai's Model.provider exactly. providerForModel becomes a thin isCuaProvider guard. Drops the rename map and the dead piProviderFor switch. - Switch packages/ai/src/index.ts from `export *` to named re-exports for models.ts and providers/common.ts. Keeps the public surface to getCuaModel/listCuaModels/providerForModel/isCuaProvider, the action types/input types, CUA tool name constants, CUA_ACTION_TYPES, and createComputerToolDefinitions. Internal exports (parseCuaModelRef, formatCuaModelRef, findCuaAnnotation, CUA_PROVIDERS, CUA_MODEL_ANNOTATIONS, schemas) stay reachable from tests via ../src/models.js but are no longer part of the package interface. - Restore parse/format unit tests, plus update annotation tests to the new google key. - Trim README copy: drop the registry/override caveat, drop "with source citations", and rewrite the CuaProvider section now that the rename is gone. Reword the action-vocabulary section to talk about types instead of dropped schemas. * ai: update integration test gemini ref to google * ai: stabilize tzafon batch tool integration expectations Tzafon's model is non-deterministic about emitting tool calls under our test prompts; mirror the requireToolCalls guard from #8 so a no-tool-call response no longer fails CI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: rgarcia <72655+rgarcia@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 3135a5d commit 3ba105a

9 files changed

Lines changed: 181 additions & 80 deletions

File tree

packages/agent/src/agent.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ function defaultSystemPrompt(provider: ReturnType<typeof providerForModel>): str
5454
switch (provider) {
5555
case "anthropic":
5656
return anthropic.buildAnthropicSystemPrompt();
57-
case "gemini":
57+
case "google":
5858
return gemini.buildGeminiSystemPrompt();
5959
case "tzafon":
6060
return tzafon.buildTzafonSystemPrompt();

packages/agent/src/tools.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ export function createCuaComputerTools(args: CuaComputerToolsOptions): AgentTool
3737
return createOpenAIComputerTools(args);
3838
case "anthropic":
3939
return createAnthropicComputerTools(args);
40-
case "gemini":
40+
case "google":
4141
return createGeminiComputerTools(args);
4242
case "tzafon":
4343
return createTzafonComputerTools(args);

packages/ai/README.md

Lines changed: 33 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,6 @@ npm install @onkernel/cua-ai
1212

1313
## Quick Start
1414

15-
See [`examples/quickstart.ts`](./examples/quickstart.ts) for a runnable version
16-
that reads `examples/screenshot.png` and uses `OPENAI_API_KEY`.
17-
1815
```ts
1916
import { readFile } from "node:fs/promises";
2017
import { complete, getCuaModel, openai } from "@onkernel/cua-ai";
@@ -56,20 +53,38 @@ computer-use model catalog and provider/tool metadata.
5653

5754
### Model Refs
5855

59-
`getCuaModel()` accepts only provider-qualified model refs:
56+
`getCuaModel()` accepts only provider-qualified model refs of the form
57+
`<provider>:<model-id>`:
6058

6159
```ts
6260
getCuaModel("openai:gpt-5.5");
6361
getCuaModel("anthropic:claude-opus-4-7");
64-
getCuaModel("gemini:gemini-2.5-computer-use-preview-10-2025");
62+
getCuaModel("google:gemini-2.5-computer-use-preview-10-2025");
6563
getCuaModel("tzafon:tzafon.northstar-cua-fast");
6664
getCuaModel("yutori:n1.5-latest");
6765
```
6866

69-
`getCuaModel(ref)` returns a pi-ai `Model<Api>` object. You pass that model to
70-
pi-ai functions like `complete(model, context)` or `stream(model, context)`.
67+
`getCuaModel(ref)` returns a pi-ai `Model<Api>` you can pass to `complete()`
68+
or `stream()`.
69+
70+
See [`docs/supported-models.md`](./docs/supported-models.md) for the current
71+
list of CUA-supporting models per provider.
72+
73+
### CuaProvider
74+
75+
`CuaProvider` is the string union of provider IDs this package targets:
76+
77+
```ts
78+
type CuaProvider = "openai" | "anthropic" | "google" | "tzafon" | "yutori";
79+
```
80+
81+
The IDs match pi-ai's `Model.provider` values exactly. `providerForModel(model)`
82+
narrows a pi-ai `Model<Api>` to a `CuaProvider`.
7183

72-
`listCuaModels(provider?)` returns:
84+
### Listing Models
85+
86+
`listCuaModels(provider?)` returns every CUA-supporting model, optionally
87+
filtered to one provider:
7388

7489
```ts
7590
interface CuaModelInfo {
@@ -86,12 +101,8 @@ Top-level exports:
86101

87102
- `getCuaModel(ref: CuaModelRef): Model<Api>`
88103
- `listCuaModels(provider?: CuaProvider): CuaModelInfo[]`
89-
- `parseCuaModelRef(ref: string): { provider: CuaProvider; model: string }`
90-
- `formatCuaModelRef(provider: CuaProvider, model: string): CuaModelRef`
91104
- `providerForModel(model: Model<Api>): CuaProvider`
92-
- `CUA_PROVIDERS: readonly CuaProvider[]`
93-
- `CuaBatchSchema`, `CuaActionSchema`, `CuaNavigationSchema` TypeBox schemas
94-
- `createCuaActionSchema(actions?)`, `createCuaBatchSchema(actions?)`
105+
- `isCuaProvider(value: string): value is CuaProvider`
95106

96107
Provider namespaces expose `createComputerToolDefinitions({ actions? })` for
97108
building model-facing pi-ai `Tool[]` definitions. Omit `actions` for the
@@ -130,9 +141,8 @@ Current coordinate contracts:
130141
- `yutori`: normalized coordinates in the 0-1000 range ([source](https://docs.yutori.com/reference/navigator), [SDK helper](https://github.com/yutori-ai/yutori-sdk-python/blob/main/yutori/navigator/coordinates.py))
131142
- `tzafon`: normalized coordinates in the 0-999 range ([source](https://docs.lightcone.ai/guides/coordinates/), [model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast))
132143

133-
`CuaActionSchema` validates one normalized computer action. The action
134-
vocabulary is intentionally provider-neutral and OpenAI-shaped because it maps
135-
cleanly to most browser computer-use APIs:
144+
The action vocabulary is intentionally provider-neutral and OpenAI-shaped
145+
because it maps cleanly to most browser computer-use APIs:
136146

137147
```ts
138148
type CuaAction =
@@ -171,21 +181,21 @@ type CuaActionGoto = {
171181
};
172182
```
173183

174-
`CuaBatchSchema` validates the input for a batched computer tool:
184+
The provider namespace `createComputerToolDefinitions()` emits a
185+
`batch_computer_actions` tool whose input is:
175186

176187
```ts
177188
type CuaBatchInput = {
178189
actions: CuaAction[];
179190
};
180191
```
181192

182-
Use it for a tool like `batch_computer_actions`, where the model can plan
183-
several writes and reads in one call. Read actions such as `screenshot`, `url`,
184-
and `cursor_position` can be interleaved with writes so your executor can return
185-
fresh state in the same order.
193+
The model can plan several writes and reads in one call. Read actions such as
194+
`screenshot`, `url`, and `cursor_position` can be interleaved with writes so
195+
your executor can return fresh state in the same order.
186196

187-
`CuaNavigationSchema` validates a smaller convenience tool for high-level
188-
navigation:
197+
When `actions` is omitted, the OpenAI namespace also emits a `computer_use_extra`
198+
navigation tool whose input is:
189199

190200
```ts
191201
type CuaNavigationInput = {
@@ -194,9 +204,6 @@ type CuaNavigationInput = {
194204
};
195205
```
196206

197-
Use it for a simple `computer_use_extra`-style tool when you want navigation
198-
available without exposing the full batch action surface.
199-
200207
Provider namespaces:
201208

202209
- `openai`: `createComputerToolDefinitions`, `COMPUTER_TOOL_COORDINATES`, OpenAI CUA action schemas, and `OPENAI_BATCH_INSTRUCTIONS`
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Supported CUA Models
2+
3+
`@onkernel/cua-ai` accepts any pi-ai model whose ID is annotated as
4+
CUA-supporting in `CUA_MODEL_ANNOTATIONS` (see
5+
[`src/models.ts`](../src/models.ts)). Annotations are either a `family`
6+
match (root + dated snapshots) or an `exact` ID match. Each annotation
7+
cites the provider's CUA docs.
8+
9+
The list below is the current snapshot. Run
10+
`listCuaModels(provider?)` for the live list — it merges pi-ai's registry
11+
with CUA-only entries that pi-ai does not ship yet.
12+
13+
## `openai`
14+
15+
API: `openai-responses` · coordinates: pixel
16+
17+
Family matches (all dated snapshots accepted):
18+
19+
- `gpt-5.4` ([docs](https://developers.openai.com/api/docs/models/gpt-5.4))
20+
- `gpt-5.5` ([docs](https://developers.openai.com/api/docs/models/gpt-5.5))
21+
22+
## `anthropic`
23+
24+
API: `anthropic-messages` · coordinates: pixel
25+
26+
Family matches (all dated snapshots accepted):
27+
28+
- `claude-3-7-sonnet`
29+
- `claude-opus-4`
30+
- `claude-sonnet-4`
31+
- `claude-haiku-4`
32+
33+
Source: [Anthropic computer use docs](https://docs.anthropic.com/en/docs/build-with-claude/computer-use).
34+
35+
## `google`
36+
37+
API: `google-generative-ai` · coordinates: normalized 0–999
38+
39+
Exact IDs:
40+
41+
- `gemini-3-flash-preview`
42+
- `gemini-2.5-computer-use-preview-10-2025`
43+
44+
Source: [Gemini computer use docs](https://ai.google.dev/gemini-api/docs/computer-use).
45+
46+
## `tzafon`
47+
48+
API: `tzafon-responses` · coordinates: normalized 0–999
49+
50+
Exact IDs:
51+
52+
- `tzafon.northstar-cua-fast` ([model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast))
53+
54+
## `yutori`
55+
56+
API: `yutori-chat-completions` · coordinates: normalized 0–1000
57+
58+
Exact IDs:
59+
60+
- `n1-latest`
61+
- `n1-20260203`
62+
- `n1.5-latest`
63+
- `n1.5-20260428`
64+
65+
Source: [Yutori Navigator reference](https://docs.yutori.com/reference/navigator).

packages/ai/examples/quickstart.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ const screenshot = await readFile(screenshotPath);
1919
// const tools = anthropic.createComputerToolDefinitions({ actions: ["click"] });
2020
//
2121
// const apiKey = process.env.GOOGLE_API_KEY;
22-
// const modelRef = "gemini:gemini-2.5-computer-use-preview-10-2025";
22+
// const modelRef = "google:gemini-2.5-computer-use-preview-10-2025";
2323
// const model = getCuaModel(modelRef);
2424
// const tools = gemini.createComputerToolDefinitions({ actions: ["click"] });
2525

packages/ai/src/index.ts

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,45 @@ import { registerCuaProviders } from "./providers.js";
22

33
export * from "@earendil-works/pi-ai";
44

5-
export * from "./models.js";
6-
export * from "./providers/common.js";
5+
export {
6+
getCuaModel,
7+
isCuaProvider,
8+
listCuaModels,
9+
providerForModel,
10+
} from "./models.js";
11+
export type { CuaModelInfo, CuaModelRef, CuaProvider } from "./models.js";
12+
13+
export {
14+
CUA_ACTION_TYPES,
15+
CUA_BATCH_TOOL_NAME,
16+
CUA_NAVIGATION_TOOL_NAME,
17+
createComputerToolDefinitions,
18+
} from "./providers/common.js";
19+
export type {
20+
ComputerToolCoordinateSystem,
21+
CreateComputerToolDefinitionsOptions,
22+
CuaAction,
23+
CuaActionBack,
24+
CuaActionClick,
25+
CuaActionCursorPosition,
26+
CuaActionDoubleClick,
27+
CuaActionDrag,
28+
CuaActionForward,
29+
CuaActionGoto,
30+
CuaActionKeypress,
31+
CuaActionMouseDown,
32+
CuaActionMouseUp,
33+
CuaActionMove,
34+
CuaActionScreenshot,
35+
CuaActionScroll,
36+
CuaActionType,
37+
CuaActionTypeText,
38+
CuaActionUrl,
39+
CuaActionWait,
40+
CuaBatchInput,
41+
CuaNavigationInput,
42+
} from "./providers/common.js";
43+
744
export * as anthropic from "./providers/anthropic/index.js";
845
export * as gemini from "./providers/gemini/index.js";
946
export * as openai from "./providers/openai/index.js";

packages/ai/src/models.ts

Lines changed: 13 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ import {
55
getModels,
66
} from "@earendil-works/pi-ai";
77

8-
export type CuaProvider = "openai" | "anthropic" | "gemini" | "tzafon" | "yutori";
8+
export type CuaProvider = "openai" | "anthropic" | "google" | "tzafon" | "yutori";
99
export type CuaModelRef = `${CuaProvider}:${string}`;
1010

1111
export interface CuaModelInfo {
@@ -15,7 +15,7 @@ export interface CuaModelInfo {
1515
name: string;
1616
}
1717

18-
export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "gemini", "tzafon", "yutori"];
18+
export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "tzafon", "yutori"];
1919

2020
// CUA support annotations.
2121
//
@@ -52,7 +52,7 @@ export const CUA_MODEL_ANNOTATIONS: Record<CuaProvider, readonly CuaModelAnnotat
5252
{ match: { kind: "family", family: "claude-sonnet-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" },
5353
{ match: { kind: "family", family: "claude-haiku-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" },
5454
],
55-
gemini: [
55+
google: [
5656
{ match: { kind: "exact", id: "gemini-3-flash-preview" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" },
5757
{ match: { kind: "exact", id: "gemini-2.5-computer-use-preview-10-2025" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" },
5858
],
@@ -78,8 +78,8 @@ const CUA_MODEL_OVERRIDES: Record<CuaProvider, readonly Model<Api>[]> = {
7878
cuaModel("openai", "gpt-5.5-2026-04-23", "GPT-5.5 (2026-04-23)"),
7979
],
8080
anthropic: [],
81-
gemini: [
82-
cuaModel("gemini", "gemini-2.5-computer-use-preview-10-2025", "Gemini 2.5 Computer Use Preview"),
81+
google: [
82+
cuaModel("google", "gemini-2.5-computer-use-preview-10-2025", "Gemini 2.5 Computer Use Preview"),
8383
],
8484
tzafon: [
8585
cuaModel("tzafon", "tzafon.northstar-cua-fast", "Tzafon Northstar CUA Fast"),
@@ -106,7 +106,6 @@ export function parseCuaModelRef(ref: string): { provider: CuaProvider; model: s
106106
}
107107

108108
export function formatCuaModelRef(provider: CuaProvider, model: string): CuaModelRef {
109-
if (!model.trim()) throw new Error("model id is empty");
110109
return `${provider}:${model}` as CuaModelRef;
111110
}
112111

@@ -119,7 +118,7 @@ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] {
119118
const ref = formatCuaModelRef(p, model.id);
120119
byRef.set(ref, { ref, provider: p, model: model.id, name: model.name });
121120
}
122-
for (const model of getModels(piProviderFor(p) as never) as Model<Api>[]) {
121+
for (const model of getModels(p as never) as Model<Api>[]) {
123122
if (!supportsCuaProvider(p, model.id)) continue;
124123
const ref = formatCuaModelRef(p, model.id);
125124
if (byRef.has(ref)) continue;
@@ -140,49 +139,24 @@ export function getCuaModel(ref: CuaModelRef): Model<Api> {
140139
if (!supportsCuaProvider(provider, modelId)) {
141140
throw new Error(`unsupported CUA model "${ref}"`);
142141
}
143-
const fromRegistry = getModel(piProviderFor(provider) as never, modelId as never) as Model<Api> | undefined;
142+
const fromRegistry = getModel(provider as never, modelId as never) as Model<Api> | undefined;
144143
if (fromRegistry) return fromRegistry;
145144
const override = CUA_MODEL_OVERRIDES[provider].find((m) => m.id === modelId);
146145
if (override) return override;
147146
throw new Error(`CUA model "${ref}" is supported but not registered. Add it to pi-ai (models.dev) or CUA_MODEL_OVERRIDES.`);
148147
}
149148

150149
export function providerForModel(model: Model<Api>): CuaProvider {
151-
switch (model.provider) {
152-
case "openai":
153-
return "openai";
154-
case "anthropic":
155-
return "anthropic";
156-
case "google":
157-
return "gemini";
158-
case "tzafon":
159-
return "tzafon";
160-
case "yutori":
161-
return "yutori";
162-
default:
163-
throw new Error(`unsupported CUA model provider "${model.provider}"`);
150+
if (!isCuaProvider(model.provider)) {
151+
throw new Error(`unsupported CUA model provider "${model.provider}"`);
164152
}
153+
return model.provider;
165154
}
166155

167156
export function isCuaProvider(value: string): value is CuaProvider {
168157
return (CUA_PROVIDERS as readonly string[]).includes(value);
169158
}
170159

171-
function piProviderFor(provider: CuaProvider): string {
172-
switch (provider) {
173-
case "openai":
174-
return "openai";
175-
case "anthropic":
176-
return "anthropic";
177-
case "gemini":
178-
return "google";
179-
case "tzafon":
180-
return "tzafon";
181-
case "yutori":
182-
return "yutori";
183-
}
184-
}
185-
186160
function supportsCuaProvider(provider: CuaProvider, modelId: string): boolean {
187161
return findCuaAnnotation(provider, modelId) !== undefined;
188162
}
@@ -204,8 +178,8 @@ function cuaModel(provider: CuaProvider, id: string, name: string): Model<Api> {
204178
const base = {
205179
id,
206180
name,
207-
provider: piProviderFor(provider),
208-
reasoning: provider === "openai" || provider === "anthropic" || provider === "gemini",
181+
provider,
182+
reasoning: provider === "openai" || provider === "anthropic" || provider === "google",
209183
input: ["text", "image"],
210184
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
211185
} satisfies Partial<Model<Api>>;
@@ -215,7 +189,7 @@ function cuaModel(provider: CuaProvider, id: string, name: string): Model<Api> {
215189
return { ...base, api: "openai-responses", baseUrl: "https://api.openai.com/v1", contextWindow: 400_000, maxTokens: 32_768 } as Model<Api>;
216190
case "anthropic":
217191
return { ...base, api: "anthropic-messages", baseUrl: "https://api.anthropic.com", contextWindow: 200_000, maxTokens: 64_000 } as Model<Api>;
218-
case "gemini":
192+
case "google":
219193
return { ...base, api: "google-generative-ai", baseUrl: "https://generativelanguage.googleapis.com/v1beta", contextWindow: 1_048_576, maxTokens: 65_536 } as Model<Api>;
220194
case "tzafon":
221195
return { ...base, api: "tzafon-responses", baseUrl: "https://api.lightcone.ai", contextWindow: 128_000, maxTokens: 4_096 } as Model<Api>;

0 commit comments

Comments
 (0)