Skip to content

feat: record agent steps from stream() #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
TheGreatAxios wants to merge 3 commits into from
Closed

feat: record agent steps from stream() #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
10ef921
feat: record agent steps from stream()
TheGreatAxios Oct 26, 2026
a5fd4a1
fix: isolate aborted turns and name denied tools
TheGreatAxios Oct 27, 2026
8b2795f
fix: hold an aborted turn on the raw send rejection
TheGreatAxios Oct 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 19 additions & 14 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,19 +47,19 @@ returned.
`unsure (p=…)` in between. `{ question, yes, no }` sets its own bars.
- **Helpers**:

| helper | passes when |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `matches(field?)` | output deep-equals expectedOutput, or the same field (key or accessor) of each |
| `includes()` | both strings, and output contains expectedOutput after NFKC, trim, and whitespace collapse; value is 1 or 0 |
| `fitsSchema(schema)` | output satisfies a Standard Schema (arktype, zod, valibot) |
| `mentions(words)` | output contains every word, case-insensitive; value is the fraction found |
| `avoids(words)` | output contains none of the words |
| `ranSteps(names)` | `tool` and `step` names match `names`, order ignored; value is their F1 |
| `stepsAtMost(n)` | the run took at most `n` steps |
| `under({ ms })` | the run finished within `ms` |
| `ask(q, { options, pass })` | Jev picks an option named in `pass` |
| `ask(q, { options, passAt })` | Jev picks an option; its credit (0..1) is the value and passes at `passAt` |
| `ask(q, { levels, pass })` | Jev grades at level `pass` or above (levels lowest first) |
| helper | passes when |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `matches(field?)` | output deep-equals expectedOutput, or the same field (key or accessor) of each |
| `includes()` | both strings, and output contains expectedOutput after NFKC, trim, and whitespace collapse; value is 1 or 0 |
| `fitsSchema(schema)` | output satisfies a Standard Schema (arktype, zod, valibot) |
| `mentions(words)` | output contains every word, case-insensitive; value is the fraction found |
| `avoids(words)` | output contains none of the words |
| `ranSteps(names)` | `tool` and `step` names match `names`, order ignored; value is their F1. Agent trajectory comes from `stream()`. A reply-only send without `stream` does not satisfy `ranSteps([])` |
| `stepsAtMost(n)` | the run took at most `n` steps |
| `under({ ms })` | the run finished within `ms` |
| `ask(q, { options, pass })` | Jev picks an option named in `pass` |
| `ask(q, { options, passAt })` | Jev picks an option; its credit (0..1) is the value and passes at `passAt` |
| `ask(q, { levels, pass })` | Jev grades at level `pass` or above (levels lowest first) |

`mentions` and `avoids` also take `(r) => words` for per-case words.
`mentions` and `ranSteps` take `{ atLeast }` to pass on partial credit.
Expand Down Expand Up @@ -100,7 +100,12 @@ space cannot absorb — those fail closed by design.

- **Function**: `(input, signal) => output`. `signal` aborts on `timeout`.
- **Agent**: an `@intx/agent` Agent, detected by `send`. Input is sent
as a message; output is the reply.
as a message; output is the reply. When the agent has `stream()`, steps
are recorded from that stream (`inference.done` as `llm`, `tool.start` /
`tool.done` as `tool`). A policy deny emits `tool.done` without
`tool.start`; the tool is still recorded when `inference.tool_call.end`
or the assistant `tool_call` block names it. A reply-only send without
`stream` leaves `steps` empty, which does not satisfy `ranSteps([])`.
- **Workflow**: `workflow(definition, { invokeStep })` from
`@corbits/evals/workflow`. A definition cannot run locally without a
step invoker, and the root never loads `@intx/workflow`, so a bare
Expand Down
Loading
Loading