[TheGreatAxios]

Summary

• Document runEvaluation({ name, data, task, scorers }), expectedOutput, systemOne, and judgeModel().
• Note the breaks: saved experiments are format 2, old JSONL keys throw, and ranSteps([]) fails on an empty step list.
• Docs only. No runtime change.

Verification

• bun run check exited 0 (2 unit tests, 72 e2e tests).
• Docs-only diff. No separate code review.

Stacked on #29.

Fixes CL-11514

[linear-code]

CL-11514

[corbits-triage]

Triage: Pass

No changes needed from you. A maintainer will review this pull request.

Stacked on #29: only this pull request's own changes were checked.

Waiting on a maintainer to approve: approve, request review.

[TheGreatAxios]

Review · Request changes

Documents the current eval API. One new sentence tells readers to run the prebuilt rubrics through judgeModel, and that call throws.

Findings

• README.md:204 — “Pair them with judgeModel when you want those rubrics run.” judgeModel({ model: faithfulness({ provider: "ollama", model: "pinned" }).definition, invokeStep }) throws judgeModel: step must not set input. judgeWorkflow always sets input: { from: "trigger.payload" } (src/prebuilt.ts:86) and assertJudgeStep rejects that (src/workflow.ts:107). Passing the Prebuilt object as a scorer throws check "faith" is not a scorer. Those factories return a one-step definition plus payload / failure, and rubric JSON is read with parsePrebuilt. judgeModel expects { decisions } or { fallback } and is not a runner for these rubrics.

The other new claims match this branch: runEvaluation({ name, data, task, scorers }), JSONL expect / expected rejected, jev as an alias of systemOne, stored format 2, ranSteps([]) on an empty step list, CLI exits 0/1/2, and the four prebuilt ids.

[TheGreatAxios]

Review · Follow-up

The prebuilt section no longer tells readers to run those rubrics through judgeModel.

It now says each factory returns { definition, payload, failure }, to run definition with workflow using payload(run) as the trigger, and to read the step JSON with parsePrebuilt. That matches Prebuilt and workflow(definition, options) on this branch. AGENTS.md did not repeat the old sentence.

Pushed as 318fc91 (docs: correct how prebuilt rubrics are run). README only, so this did not get a second code review.

[TheGreatAxios]

Superseded by the 0.2.0 release: #32

Do not merge this pull request separately. The release branch cherry-picks the feature stack (except the 0.1.3 bump on #22) and sets @corbits/evals to 0.2.0 with @corbits/system-one ^0.4.0.

[TheGreatAxios]

Superseded by the 0.2.0 release: #32