From ac5f042c4c2e2add74b04f39c8d8bd90d051ee19 Mon Sep 17 00:00:00 2001 From: "user.email" Date: Mon, 26 Oct 2026 02:29:14 -0700 Subject: [PATCH 1/2] docs: describe the current eval API State format 2 experiments, expectedOutput JSONL rows, and ranSteps([]) on an empty step list. --- AGENTS.md | 16 ++++++++++++++++ README.md | 49 +++++++++++++++++++++++++++++++++---------------- 2 files changed, 49 insertions(+), 16 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6bc8f7c..8ed0d6c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,6 +4,22 @@ `@corbits/evals` package. +Public call: `runEvaluation({ name, data, task, scorers })`. There is no +`defineEval`, no `evaluate(name, options)`, and no `runEval`. Ground truth +is `expectedOutput`. JSONL keys `expect` and `expected` throw. + +The judge model is `judgeModel()` from `@corbits/evals/workflow`: one step, +inference source pinned on the agent. Thresholds are `systemOne`. `jev` is +an alias of `systemOne`, not an endpoint. Do not document `jev({ endpoint })`. + +Saved experiments are format 2 (`{ format: 2, name, cases }`). Format 1 is +rejected. `ranSteps([])` fails when `steps` is empty; an observed model-only +run still passes. CLI is `corbits-eval run ` and exits 0, 1, or 2. + +`includes` and the four prebuilt workflow scorers +(`faithfulness`, `answerRelevancy`, `correctness`, `trajectoryAccuracy`) exist. +Do not document toxicity, bias, or pii. + ## Conventions - Read `README.md` before changing public behavior. diff --git a/README.md b/README.md index 8fc45a3..e4a5d99 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ console.log(report.markdown()); ## Checks -`checks` maps a name to a check. Without `checks`, one `matches` check +`scorers` maps a name to a check. Without `scorers`, one `matches` check compares output with `expectedOutput`. Every check gets the run `r = { input, output, expectedOutput, steps, ms }`; `output` is whatever the run returned. @@ -47,19 +47,19 @@ returned. `unsure (p=…)` in between. `{ question, yes, no }` sets its own bars. - **Helpers**: -| helper | passes when | -| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `matches(field?)` | output deep-equals expectedOutput, or the same field (key or accessor) of each | -| `includes()` | both strings, and output contains expectedOutput after NFKC, trim, and whitespace collapse; value is 1 or 0 | -| `fitsSchema(schema)` | output satisfies a Standard Schema (arktype, zod, valibot) | -| `mentions(words)` | output contains every word, case-insensitive; value is the fraction found | -| `avoids(words)` | output contains none of the words | -| `ranSteps(names)` | `tool` and `step` names match `names`, order ignored; value is their F1. Agent trajectory comes from `stream()`. A reply-only send without `stream` does not satisfy `ranSteps([])` | -| `stepsAtMost(n)` | the run took at most `n` steps | -| `under({ ms })` | the run finished within `ms` | -| `ask(q, { options, pass })` | Jev picks an option named in `pass` | -| `ask(q, { options, passAt })` | Jev picks an option; its credit (0..1) is the value and passes at `passAt` | -| `ask(q, { levels, pass })` | Jev grades at level `pass` or above (levels lowest first) | +| helper | passes when | +| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `matches(field?)` | output deep-equals expectedOutput, or the same field (key or accessor) of each | +| `includes()` | both strings, and output contains expectedOutput after NFKC, trim, and whitespace collapse; value is 1 or 0 | +| `fitsSchema(schema)` | output satisfies a Standard Schema (arktype, zod, valibot) | +| `mentions(words)` | output contains every word, case-insensitive; value is the fraction found | +| `avoids(words)` | output contains none of the words | +| `ranSteps(names)` | `tool` and `step` names match `names`, order ignored; value is their F1. Agent trajectory comes from `stream()`. `ranSteps([])` fails when `steps` is empty. An observed model-only run (llm steps, no tool or step names) still passes | +| `stepsAtMost(n)` | the run took at most `n` steps | +| `under({ ms })` | the run finished within `ms` | +| `ask(q, { options, pass })` | Jev picks an option named in `pass` | +| `ask(q, { options, passAt })` | Jev picks an option; its credit (0..1) is the value and passes at `passAt` | +| `ask(q, { levels, pass })` | Jev grades at level `pass` or above (levels lowest first) | `mentions` and `avoids` also take `(r) => words` for per-case words. `mentions` and `ranSteps` take `{ atLeast }` to pass on partial credit. @@ -104,7 +104,7 @@ space cannot absorb — those fail closed by design. as a message; output is the reply. When the agent has `stream()`, steps are recorded from that stream (`inference.done` as `llm`, `tool.start` / `tool.done` as `tool`). A reply-only send without `stream` leaves `steps` - empty, which does not satisfy `ranSteps([])`. + empty, which fails `ranSteps([])`. An observed model-only run still passes. - **Workflow**: `workflow(definition, { invokeStep })` from `@corbits/evals/workflow`. A definition cannot run locally without a step invoker, and the root never loads `@intx/workflow`, so a bare @@ -121,7 +121,8 @@ attempt passes. `concurrency` (default 4) bounds cases in flight; ## Cases An array of `{ name?, input, expectedOutput?, must?, skip? }`, or the path of a -JSONL file of the same objects. `name` keys the case in reports and +JSONL file of the same objects. The ground-truth key is `expectedOutput`. +`expect` and `expected` throw. `name` keys the case in reports and compared runs and defaults to the input. ## Passing @@ -157,6 +158,8 @@ cases; `.report` holds every result. An experiment is each case × scorer pass and value — plus a Jev scorer's probability distribution and confidence — and `about`, no times. +Saved files are format 2: `{ format: 2, name, cases }`. Any other `format`, +including format 1, is rejected. `compare` only reads; a missing file skips drops. `save` only writes, on every run. One path for both compares with the last experiment, then replaces it. `compare` also takes an experiment from an earlier `runEvaluation` of the same @@ -193,6 +196,20 @@ Exit codes: `--format json` (and `--json`) come from incur and print the result as JSON. +## Workflow scorers + +`@corbits/evals/workflow` also exports four prebuilt one-step judges. Each +takes a pinned `{ provider, model }` and does not call a model by itself: +`faithfulness`, `answerRelevancy`, `correctness`, and `trajectoryAccuracy`. +Pair them with `judgeModel` when you want those rubrics run. `includes` stays +a code check on the root package. + +## Breaking changes + +- Saved experiments must be format 2 (`{ format: 2, name, cases }`). Format 1 is rejected. +- JSONL rows use `expectedOutput`. `expect` and `expected` throw. +- `ranSteps([])` fails when `steps` is empty. An observed model-only run still passes. + ## Calibrating Thresholds are per-workload starting points, not constants. `calibrate` From 318fc91e19ac8ba8dfd6134ddef2031912712ea1 Mon Sep 17 00:00:00 2001 From: Sawyer Cutler Date: Mon, 26 Oct 2026 20:42:26 -0700 Subject: [PATCH 2/2] docs: correct how prebuilt rubrics are run --- README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e4a5d99..af23692 100644 --- a/README.md +++ b/README.md @@ -201,7 +201,12 @@ Exit codes: `@corbits/evals/workflow` also exports four prebuilt one-step judges. Each takes a pinned `{ provider, model }` and does not call a model by itself: `faithfulness`, `answerRelevancy`, `correctness`, and `trajectoryAccuracy`. -Pair them with `judgeModel` when you want those rubrics run. `includes` stays +Each returns `{ definition, payload, failure }`. Run `definition` with +`workflow`, using `payload(run)` as the trigger. An outcome from `payload` +is the result; `failure` is the closed miss when the run cannot be judged. +`parsePrebuilt(id, output)` reads the step's category or score JSON. +`judgeModel` expects `{ decisions }` or `{ fallback }` and does not run +these rubrics. `includes` stays a code check on the root package. ## Breaking changes