Skip to content

Commit 92f4dbf

Browse files
BrainerViruscursoragent
BrainerVirus
and
cursoragent
committed
docs: spec and plan for catalog modalities from models.dev
Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 529b93f commit 92f4dbf

2 files changed

Lines changed: 375 additions & 0 deletions

File tree

‎docs/2026-08-28-catalog-modalities/plan.md‎

Lines changed: 313 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
1+
# Catalog Modalities Implementation Plan
2+
3+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4+
5+
**Spec:** `docs/2026-08-28-catalog-modalities/spec.md`
6+
**Branch:** `feature/2026-08-28-catalog-modalities`
7+
8+
**Goal:** Bundled catalog models expose OpenCode `attachment` and `modalities` copied from models.dev, with unmatched SKUs defaulting to text-only.
9+
10+
**Architecture:** Extend the existing models.dev parser and matching index. A new `applyModelsDevModalities` runs on every catalog model (unlike the cost skip set). `generateOpencodeModels` always emits the fields. Sync uses one models.dev fetch for both costs and modalities.
11+
12+
**Tech Stack:** TypeScript, Bun (`bun test tests/unit/`), existing `src/costs-models-dev.ts` + `src/catalog.ts`.
13+
14+
## Global Constraints
15+
16+
- No vision allowlist. No session-prompt injection. No overwriting `limit` / `reasoning` / `tool_call`.
17+
- Runtime does not fetch models.dev.
18+
- Tests: `bun test tests/unit/` from `/home/cristhofer-pincetti/Documents/projects/personal/opencode-commandcode-provider`.
19+
- Plugin imports stay `.js`; tests import `.ts`.
20+
- Do not fold this package into workit. PRs against `BrainerVirus/opencode-commandcode` base `main`.
21+
- Implementation on `feature/2026-08-28-catalog-modalities` (in-place checkout, no worktrees).
22+
23+
---
24+
25+
### Task 1: Parse and apply modalities from models.dev
26+
27+
**Files:**
28+
- Modify: `src/costs-models-dev.ts`
29+
- Modify: `tests/fixtures/models-dev/api.subset.json`
30+
- Test: `tests/unit/costs-models-dev.test.ts`
31+
32+
**Interfaces:**
33+
- Consumes: existing `parseModelsDev`, `findRow` / `indexRows` matching
34+
- Produces:
35+
- `TEXT_ONLY_MODALITIES = { input: ["text"], output: ["text"] }`
36+
- `ModelsDevRow` gains optional `attachment?: boolean` and `modalities?: { input: string[]; output: string[] }`
37+
- `applyModelsDevModalities(models: ModelEntry[], rows: ModelsDevRow[]): number` — returns how many models matched a models.dev row with explicit attachment/modalities; every model is assigned fields
38+
39+
- [ ] **Step 1: Write the failing test**
40+
41+
Add vision + text-only rows to `tests/fixtures/models-dev/api.subset.json`:
42+
43+
```json
44+
{
45+
"google": {
46+
"models": {
47+
"google/gemini-3.5-flash": {
48+
"id": "google/gemini-3.5-flash",
49+
"name": "Gemini 3.5 Flash",
50+
"attachment": true,
51+
"modalities": {
52+
"input": ["text", "image", "video", "audio", "pdf"],
53+
"output": ["text"]
54+
},
55+
"cost": { "input": 1.5, "output": 9, "cache_read": 0.15 }
56+
}
57+
}
58+
},
59+
"tencent": {
60+
"models": {
61+
"tencent/hy4-preview": {
62+
"id": "tencent/hy4-preview",
63+
"name": "Tencent Hy4 Preview",
64+
"attachment": false,
65+
"modalities": { "input": ["text"], "output": ["text"] },
66+
"cost": { "input": 0.834, "output": 2.501, "cache_read": 0.042 }
67+
}
68+
}
69+
}
70+
}
71+
```
72+
73+
Keep the existing `zai`, `vercel`, and `qwen` objects. Qwen stays cost-only (no modalities in the fixture) so the apply function must default it to text-only.
74+
75+
In `tests/unit/costs-models-dev.test.ts`, add:
76+
77+
```ts
78+
import { applyModelsDevModalities, TEXT_ONLY_MODALITIES } from "../../src/costs-models-dev.ts";
79+
80+
describe("applyModelsDevModalities", () => {
81+
const rows = parseModelsDev(
82+
readFileSync(join(import.meta.dir, "../fixtures/models-dev/api.subset.json"), "utf-8"),
83+
);
84+
85+
test("copies vision modalities and defaults unmatched plus cost-only rows to text-only", () => {
86+
const models = [
87+
model({ id: "google/gemini-3.5-flash", name: "Gemini 3.5 Flash" }),
88+
model({ id: "tencent/hy4-preview", name: "Tencent Hy4 Preview" }),
89+
model({ id: "Qwen/Qwen3.6-Plus", name: "Qwen 3.6 Plus" }),
90+
model({ id: "unknown/not-on-models-dev", name: "Unknown" }),
91+
model({
92+
id: "inclusionai/ling-3.0-flash-free",
93+
name: "Ling 3.0 Flash Free",
94+
cost: { input: 0, output: 0 },
95+
}),
96+
];
97+
const n = applyModelsDevModalities(models, rows);
98+
expect(n).toBe(2);
99+
expect(models[0].attachment).toBe(true);
100+
expect(models[0].modalities).toEqual({
101+
input: ["text", "image", "video", "audio", "pdf"],
102+
output: ["text"],
103+
});
104+
expect(models[1].attachment).toBe(false);
105+
expect(models[1].modalities).toEqual(TEXT_ONLY_MODALITIES);
106+
expect(models[2].attachment).toBe(false);
107+
expect(models[2].modalities).toEqual(TEXT_ONLY_MODALITIES);
108+
expect(models[3].attachment).toBe(false);
109+
expect(models[3].modalities).toEqual(TEXT_ONLY_MODALITIES);
110+
expect(models[4].attachment).toBe(false);
111+
expect(models[4].modalities).toEqual(TEXT_ONLY_MODALITIES);
112+
});
113+
});
114+
```
115+
116+
- [ ] **Step 2: Run test to verify it fails**
117+
118+
Run: `bun test tests/unit/costs-models-dev.test.ts`
119+
120+
Expected: FAIL — `applyModelsDevModalities` / `TEXT_ONLY_MODALITIES` not exported.
121+
122+
- [ ] **Step 3: Write minimal implementation**
123+
124+
In `src/costs-models-dev.ts`:
125+
126+
```ts
127+
export const TEXT_ONLY_MODALITIES = { input: ["text"], output: ["text"] } as const;
128+
129+
export type ModelsDevRow = {
130+
id: string;
131+
name: string;
132+
cost: { input: number; output: number; cache_read?: number; cache_write?: number };
133+
attachment?: boolean;
134+
modalities?: { input: string[]; output: string[] };
135+
};
136+
```
137+
138+
Extend `ModelsDevModel` with optional `attachment?: boolean` and `modalities?: { input?: string[]; output?: string[] }`.
139+
140+
In `parseModelsDev`, after building `cost`, copy:
141+
142+
```ts
143+
if (typeof model.attachment === "boolean") row.attachment = model.attachment;
144+
const input = model.modalities?.input?.filter((x) => typeof x === "string");
145+
const output = model.modalities?.output?.filter((x) => typeof x === "string");
146+
if (input?.length || output?.length) {
147+
row.modalities = {
148+
input: input?.length ? input : ["text"],
149+
output: output?.length ? output : ["text"],
150+
};
151+
}
152+
```
153+
154+
Add `applyModelsDevModalities` using the existing `indexRows` / `findRow`. For each model: if the row has `modalities` or `attachment !== undefined`, copy them (`attachment` defaults to `modalities.input.includes("image")` when omitted); otherwise assign `attachment: false` and `{ ...TEXT_ONLY_MODALITIES }` (spread into a mutable `{ input: string[]; output: string[] }`). Return the count of models that used an explicit models.dev attachment/modalities (not the default).
155+
156+
Extend `ModelEntry` in `src/catalog.ts`:
157+
158+
```ts
159+
attachment?: boolean;
160+
modalities?: { input: string[]; output: string[] };
161+
```
162+
163+
- [ ] **Step 4: Run test to verify it passes**
164+
165+
Run: `bun test tests/unit/costs-models-dev.test.ts`
166+
167+
Expected: PASS
168+
169+
- [ ] **Step 5: Commit** (only if the user asked to commit)
170+
171+
```bash
172+
git add src/costs-models-dev.ts src/catalog.ts tests/unit/costs-models-dev.test.ts tests/fixtures/models-dev/api.subset.json
173+
git commit -m "feat: copy vision and text-only modalities from models.dev"
174+
```
175+
176+
---
177+
178+
### Task 2: Emit attachment and modalities on OpenCode models
179+
180+
**Files:**
181+
- Modify: `src/catalog.ts` (`generateOpencodeModels`)
182+
- Test: `tests/unit/catalog.test.ts`
183+
184+
**Interfaces:**
185+
- Consumes: `ModelEntry.attachment` / `ModelEntry.modalities` from Task 1
186+
- Produces: each OpenCode model object includes `attachment: boolean` and `modalities: { input: string[]; output: string[] }`; missing fields emit text-only
187+
188+
- [ ] **Step 1: Write the failing test**
189+
190+
In `describe("generateOpencodeModels")` add:
191+
192+
```ts
193+
test("emits attachment and modalities, defaulting to text-only", () => {
194+
const models = generateOpencodeModels([
195+
{
196+
id: "google/gemini-3.5-flash",
197+
name: "Gemini 3.5 Flash",
198+
tier: "open-source",
199+
reasoning: false,
200+
tool_call: true,
201+
cost: { input: 1.5, output: 9 },
202+
limit: { context: 1048576, output: 65536 },
203+
attachment: true,
204+
modalities: { input: ["text", "image"], output: ["text"] },
205+
},
206+
{
207+
id: "tencent/hy4-preview",
208+
name: "Tencent Hy4 Preview",
209+
tier: "open-source",
210+
reasoning: true,
211+
tool_call: true,
212+
cost: { input: 0.834, output: 2.501 },
213+
limit: { context: 1048576, output: 64000 },
214+
},
215+
]);
216+
const gemini = models["gemini-3.5-flash"] as Record<string, unknown>;
217+
const hy4 = models["hy4-preview"] as Record<string, unknown>;
218+
expect(gemini.attachment).toBe(true);
219+
expect(gemini.modalities).toEqual({ input: ["text", "image"], output: ["text"] });
220+
expect(hy4.attachment).toBe(false);
221+
expect(hy4.modalities).toEqual({ input: ["text"], output: ["text"] });
222+
});
223+
```
224+
225+
- [ ] **Step 2: Run test to verify it fails**
226+
227+
Run: `bun test tests/unit/catalog.test.ts`
228+
229+
Expected: FAIL — `attachment` / `modalities` undefined on emitted models.
230+
231+
- [ ] **Step 3: Write minimal implementation**
232+
233+
In `generateOpencodeModels`, after building `model`:
234+
235+
```ts
236+
model.attachment = entry.attachment ?? false;
237+
model.modalities = entry.modalities ?? { input: ["text"], output: ["text"] };
238+
```
239+
240+
- [ ] **Step 4: Run test to verify it passes**
241+
242+
Run: `bun test tests/unit/catalog.test.ts`
243+
244+
Expected: PASS
245+
246+
- [ ] **Step 5: Commit** (only if the user asked to commit)
247+
248+
```bash
249+
git add src/catalog.ts tests/unit/catalog.test.ts
250+
git commit -m "feat: emit OpenCode attachment and modalities"
251+
```
252+
253+
---
254+
255+
### Task 3: Wire sync, refresh catalog, document
256+
257+
**Files:**
258+
- Modify: `scripts/sync-models.ts`
259+
- Modify: `README.md` (What this package adds)
260+
- Modify: `CHANGELOG.md` via `workit_changelog_apply`
261+
- Modify: `models.json` (from `bun run sync`)
262+
263+
**Interfaces:**
264+
- Consumes: `parseModelsDev`, `applyModelsDevCosts`, `applyModelsDevModalities`
265+
- Produces: one models.dev parse shared by costs and modalities; if fetch throws, still `applyModelsDevModalities(entries, [])` so every written model is text-only tagged
266+
267+
- [ ] **Step 1: Write the failing test**
268+
269+
No new unit test for the script. Assert the sync wiring by reading `scripts/sync-models.ts` in the step 4 check: `applyModelsDevModalities` is called with the same parsed rows, and the catch path still applies empty rows.
270+
271+
- [ ] **Step 2: Run existing tests (still pass; wiring not covered)**
272+
273+
Run: `bun test tests/unit/`
274+
275+
Expected: PASS (Task 1–2 tests).
276+
277+
- [ ] **Step 3: Write minimal implementation**
278+
279+
In `scripts/sync-models.ts`, import `applyModelsDevModalities`. After the models.dev try/catch, always apply modalities:
280+
281+
```ts
282+
let modelsDevRows: ReturnType<typeof parseModelsDev> = [];
283+
try {
284+
const modelsDevJson = await fetchModelsDevJson();
285+
modelsDevRows = parseModelsDev(modelsDevJson);
286+
const filled = applyModelsDevCosts(entries, modelsDevRows, skipAfterFree, thirdPartyIds);
287+
console.log(` Applied models.dev costs to ${filled} models`);
288+
} catch (err) {
289+
const message = err instanceof Error ? err.message : String(err);
290+
console.log(` models.dev fill skipped: ${message}`);
291+
}
292+
const modalityFilled = applyModelsDevModalities(entries, modelsDevRows);
293+
console.log(` Applied models.dev modalities to ${modalityFilled} models`);
294+
```
295+
296+
README, under “What this package adds”, add: vision vs text-only (`attachment` / `modalities`) is copied from models.dev during catalog sync; unmatched models are text-only.
297+
298+
Changelog Unreleased / Added: catalog models include vision and text-only metadata from models.dev.
299+
300+
Run: `bun run sync` from the repo root (needs network). Then `bun run check`.
301+
302+
- [ ] **Step 4: Verify**
303+
304+
- `bun run check` passes
305+
- `models.json` contains `"attachment"` and `"modalities"` on `tencent/hy4-preview` with `attachment: false` and input `["text"]`
306+
- A known vision id (e.g. `google/gemini-3.5-flash`) has `attachment: true` and `"image"` in `modalities.input`
307+
308+
- [ ] **Step 5: Commit** (only if the user asked to commit)
309+
310+
```bash
311+
git add scripts/sync-models.ts README.md CHANGELOG.md models.json manifest.json
312+
git commit -m "feat: sync OpenCode modalities from models.dev"
313+
```

‎docs/2026-08-28-catalog-modalities/spec.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Spec: Catalog modalities from models.dev
2+
3+
Status: approved (2026-08-28)
4+
**Branch:** `feature/2026-08-28-catalog-modalities`
5+
6+
Parent: [docs/2026-08-28-ci-catalog/spec.md](../2026-08-28-ci-catalog/spec.md)
7+
8+
## Goals
9+
10+
- G1: Every bundled catalog model carries OpenCode `attachment` and `modalities` copied from [models.dev](https://models.dev).
11+
- G2: Unmatched models default to text-only (no vision guessing, no frozen allowlist).
12+
- G3: Cost waterfall stays independent; modalities apply to all matching models including free SKUs.
13+
14+
## Locked (do not reopen)
15+
16+
- Source of truth is models.dev `attachment` + `modalities` (same fetch already used for costs). No human allowlist. No guessing vision.
17+
- Cost waterfall stays independent. Modalities apply to **all** matching models, including free SKUs and models that already have CLI/docs prices.
18+
- Unmatched models default to text-only: `attachment: false`, `modalities: { input: ["text"], output: ["text"] }`.
19+
- Runtime still does not fetch models.dev. Sync writes the fields into `models.json`; `generateOpencodeModels` emits them (and the text-only default if a field is missing).
20+
- Do not overwrite CLI `limit`, `reasoning`, or `tool_call` from models.dev.
21+
- Do not inject capability text into the session prompt. Asking the chat model “what can you do?” is out of scope.
22+
23+
## Context
24+
25+
`generateOpencodeModels` currently emits `id`, `name`, `reasoning`, `tool_call`, `cost`, `limit`, and optional `reasoningEfforts`/`variants`. OpenCode uses `attachment` and `modalities.input` for vision. Hy4 Preview (`tencent/hy4-preview`) is text-only on models.dev with a 1,048,576 context window already present on `limit`; the model itself does not read that metadata.
26+
27+
## Non-goals
28+
29+
- Session/system-prompt injection of context window or vision flags
30+
- Filling `limit` from models.dev
31+
- Local OpenCode overlay `commandcode-modalities.ts` (becomes redundant after this ships; not deleted in this repo)
32+
- Changing hybrid Provider API transport
33+
34+
## Architecture
35+
36+
```mermaid
37+
flowchart TD
38+
sync["bun run sync"] --> fetch[Fetch models.dev api.json]
39+
fetch --> costs[applyModelsDevCosts skip priced/free]
40+
fetch --> mods[applyModelsDevModalities all models]
41+
mods --> unmatched[No match: text-only default]
42+
costs --> json[models.json]
43+
unmatched --> json
44+
json --> emit[generateOpencodeModels]
45+
emit --> oc["OpenCode attachment + modalities"]
46+
```
47+
48+
Matching reuses the existing models.dev index: exact id, then last path segment, then display name (case-insensitive). Copy `modalities` arrays as-is (Gemini may include `video` / `audio` / `pdf`). `attachment` comes from models.dev when present, otherwise `modalities.input` includes `"image"`.
49+
50+
## Acceptance criteria
51+
52+
- CA-01: `parseModelsDev` retains `attachment` and `modalities` on rows that already have costs.
53+
- CA-02: `applyModelsDevModalities` sets Gemini-style vision on a matching catalog id and text-only on Hy4 Preview / unmatched ids.
54+
- CA-03: Free SKUs and CLI-priced models still receive modalities (not skipped the way costs are).
55+
- CA-04: `generateOpencodeModels` always emits `attachment` and `modalities`; missing fields become text-only.
56+
- CA-05: `bun run sync` applies modalities from the same models.dev JSON used for costs; if that fetch fails, every model still gets the text-only default before write.
57+
- CA-06: README states that vision/text-only comes from models.dev.
58+
59+
## Decisions
60+
61+
- D-01: Conservative default is text-only, not “omit the field and let OpenCode guess”.
62+
- D-02: Extra modality strings from models.dev are copied, not filtered to `text`/`image`.

0 commit comments

Comments
 (0)