diff --git a/.abcd/development/brief/04-surfaces/01-ahoy.md b/.abcd/development/brief/04-surfaces/01-ahoy.md index 99f2f934cb..361c799e3a 100644 --- a/.abcd/development/brief/04-surfaces/01-ahoy.md +++ b/.abcd/development/brief/04-surfaces/01-ahoy.md @@ -1322,6 +1322,7 @@ Sub-verbs: none. | `--key` | string | | `--model` | stringArray | | `--resume` | string | +| `--stream-usage` | bool | ### `abcd ahoy credential` diff --git a/.abcd/development/brief/04-surfaces/05-intent.md b/.abcd/development/brief/04-surfaces/05-intent.md index af84b5ee6e..4223741505 100644 --- a/.abcd/development/brief/04-surfaces/05-intent.md +++ b/.abcd/development/brief/04-surfaces/05-intent.md @@ -682,12 +682,15 @@ agent and refuses the override. A step no configured provider can serve at its tier goes to the harness with the tier named in its request, and one stderr line says so. The receipt is a `route` member in the JSON and a `route:` line in the text, carrying `tier_asked`, `connection_tried`, `connection_used`, -`fallback_reason`, `override`, `settings_sent`, `model_reported` and -`provider_call`. `model_reported` is read from the payload's own `model` field (a -reading's `instrument.model`) and empty when the payload names none; -`provider_call` names the provider, the model asked for and the model it -reported when a provider adapter answered the step, and is null on the harness -leg. A routing table that cannot be read, an +`fallback_reason`, `override`, `settings_sent`, `model_reported`, +`provider_call` and `unpinned`. `model_reported` is read from the payload's own +`model` field (a reading's `instrument.model`) and empty when the payload names +none; `provider_call` names the provider, the model asked for and the model it +reported when a provider adapter answered the step, with both sides of the call +(the internals' adapters chapter lists them), and is null on the harness leg; +`unpinned` is the harness leg's list of the fields it could not pin, the +payload's model among them as self-reported, and null on a provider leg, whose +call record carries its own. A routing table that cannot be read, an override naming an agent this invocation does not dispatch, a tier outside `local`, `economy`, `frontier` and `host-decides`, or a connection this machine has not configured exits 2 before anything is written. With no table accepted diff --git a/.abcd/development/brief/04-surfaces/34-build.md b/.abcd/development/brief/04-surfaces/34-build.md index ee69af5749..ce7ca603fc 100644 --- a/.abcd/development/brief/04-surfaces/34-build.md +++ b/.abcd/development/brief/04-surfaces/34-build.md @@ -267,7 +267,17 @@ repository abcd manages has one, so a run is managed-only by construction. Each run directory is created one level at a time and proved real, the state file is replaced atomically inside an `os.Root`, and the reader decodes strictly, refusing an unknown field, a schema version it does not know, or a file stored -under a run id it does not name. The state is schema version 13. Version 13 +under a run id it does not name. The state is schema version 14. Version 14 +added both sides of a run (itd-166): a runner's `route` on a verified receipt or +a validator's return carries, beside the route asked, the route that ran and the +served model, the model the runner asked for (`model_asked`), the SHA-256 of +the prompt it handed the harness, the served models, provider, finish reason +and harness version the harness's events name, the token counts and the cost, +and `unpinned`, each field of the fixed asked and served lists the run could +not pin with its reason. Version 13 is its strict subset, read as a run whose +routes recorded neither side; a file of version 13 or lower carrying either is +refused, and an abcd that writes version 13 refuses a version-14 file as a +newer abcd's. Version 13 added a lane's `refused` (spc-2610161044461448 step 5): an issue run's lane logs its run of refused steps, the stage, the check, the reason, a count of the same refusal in a row and since when it has refused at that stage, which a @@ -326,7 +336,7 @@ added the fix-round cap (ruling DR1): the pace's `fix_rounds` and a lane's `validation`). Each earlier version is the next one's strict subset, read as a run that predates the addition (a version-5 run runs on the bundled cap, a version-7 run is one the host ran every agent of) and -written back at version 13 by its next mutation; an earlier version carrying what +written back at version 14 by its next mutation; an earlier version carrying what only a later one writes is refused. Version 4 renamed the lane's stage (BU1, iss-2609291313276243): a lane's and a record line's `step` became `stage`, so "step" names only the spec's steps (`spec_step`, @@ -480,9 +490,11 @@ transcript lands in abcd's history store, keyed on the repository's root commit, and its receipt is handed back through the same receipt verb and verified by the stage's own verifier, so a verified one completes the stage in the same call. The verified receipt, or the validator's recorded return, names -the route that ran it (asked, ran, the model the runner reported), which is the -only field a runner-run review's record differs in from a host-run one's; the -record's receipt or verdict line names the runner that ran it. A runner that is absent, refuses, fails, runs past +the route that ran it (asked, ran, the model the runner reported, and both +sides of the run: the model it asked for, the token counts, the cost and the +fields it could not pin), which is the only field a runner-run review's record differs in from a host-run one's; the +record's receipt or verdict line names the runner that ran it, with the model +it asked for and the cost when the route carries them. A runner that is absent, refuses, fails, runs past its time, answers unparsably or writes a receipt the verifier refuses leaves the lane awaiting: the call records one fallback receipt (the role, the runner asked for, the reason and the route that runs it) in the state and the record, and diff --git a/.abcd/development/brief/05-internals/02-adapters.md b/.abcd/development/brief/05-internals/02-adapters.md index fc642283cf..6fae7a8c3b 100644 --- a/.abcd/development/brief/05-internals/02-adapters.md +++ b/.abcd/development/brief/05-internals/02-adapters.md @@ -155,7 +155,28 @@ prompt, and `reading ingest --dispatch ` sends a parked run's bundle with the position's definition. The verb runs the answer through its own ingest, and the receipt names the provider as `connection_used` with its call record (`provider_call`: provider, model asked, model reported, credential name), so a -host reading such a receipt knows the step already ran. DR5 is checked before a +host reading such a receipt knows the step already ran. The call record carries +both sides of the call (itd-166), built by the dispatch from the request it sent +and the response it read, never by asking the model about itself: the asked side +beside the receipt's `settings_sent` (the SHA-256 of the rendered messages, +never their text, and `adapter_added`, the request field the adapter sent on +its own, `stream_options.include_usage`), and the served side as the response +names it (`served_provider`, the upstream provider an aggregator names; +`generation_id`; `finish_reason`; `usage`, with prompt, completion, reasoning, +cache-read and cache-write tokens; and `cost`, in the provider's credits). Every +string the provider supplied is scrubbed of the key, cleaned and bounded as the +reported model is, and a usage member of an unexpected shape is a detail not +reported, never a refused answer: each count it does not carry readably is +listed as not reported on its own, and a later usage chunk that reads fewer +fields than an earlier one does not replace it. `unpinned` lists each field of the fixed asked +and served lists the call could not pin, with its reason: `not_reported` (the +response does not carry it), `self_reported`, `harness_computed` or +`not_configured` (no setting or pin names it; a route to this machine lists the +weight file's fingerprint, its quantisation and the runtime's version so until a +local route can be pinned). Recording adds no request to the ones the call +makes. On the harness leg the receipt's own `unpinned` lists the payload's +`model_reported` as self-reported and every other field as not configured or not +reported, since the host reports none of it. DR5 is checked before a verb writes anything (`APIConfig.Admitted`), and again by the dispatch. An ingest handed a payload the host produced while its agent is routed to a provider refuses it, since the receipt would name work the provider never did; @@ -181,7 +202,22 @@ review is skipped. contract grants the shell carries the guard as the harness's permission prompt tool, `abcd guard approve` over MCP on stdio, set on that launch's command line ([`17-guard.md`](../04-surfaces/17-guard.md#answering-a-headless-agents-prompts)). - Besides the reviewers and the implementer, the drain's judge, `drain-judge`, takes the same route + A run records both sides + (itd-166) in the same vocabulary as the provider adapter's call record: the + model the runner passed and the prompt's SHA-256, beside what the harness's + events report. The claude CLI's result event gives the token counts, the + stop reason, the cost and the models its per-model usage names, and its init + event the served model and the harness's version; opencode's run mode names + no model, so its served model is empty and listed as not reported, never the + asked one, and its step-finish events' tokens and cost are summed, a floor, + since a sub-agent's steps are not among them. Either harness computes the + cost from its own price table, so it is listed as `harness_computed`. A + count is read only when present and not negative, and a cost only when it + is zero or more and finite, the rule the provider adapter holds its usage + member to; a summed count that would overflow, or a summed cost that stops + being finite, is listed as not reported rather than recorded. + Besides the reviewers and the + implementer, the drain's judge, `drain-judge`, takes the same route ([`35-drain.md`](../04-surfaces/35-drain.md#the-judgement-before-a-lane-opens)), to the claude runner alone, in a private directory of its own outside every working tree, granted reading that directory and writing its answer file diff --git a/.abcd/development/brief/05-internals/03-configuration.md b/.abcd/development/brief/05-internals/03-configuration.md index aaddb6ad7d..6ae7592fd4 100644 --- a/.abcd/development/brief/05-internals/03-configuration.md +++ b/.abcd/development/brief/05-internals/03-configuration.md @@ -179,7 +179,9 @@ rather than skipped: "base_url": "https://openrouter.ai/api/v1", // https, or http to this machine "key": "openrouter", // a credential NAME, resolved through the credential // source; omitted for a server that takes no key - "models": ["typesafe/jev-1.13"] // the allowlist: the only models it may serve + "models": ["typesafe/jev-1.13"], // the allowlist: the only models it may serve + "stream_usage": false // optional, default true: false stops asking for + // usage in the stream, for a server that refuses it } }, "denylist": ["openai/*"], // optional, none bundled; repo or machine @@ -192,8 +194,22 @@ rather than skipped: - **A provider block sits on the machine alone.** It names the address a key is sent to, so a repository's `.abcd/config.json` declaring `oracle.api` is refused: a checkout must never be able to aim the person's key at a server of - its choosing. `abcd ahoy connect` writes the block, after one verification - call, and it is the one write abcd makes to `~/.abcd.noindex/config.json`. + its choosing. `abcd ahoy connect` writes the block once its verification + succeeds (asked at most twice, below), and it is the one write abcd makes to `~/.abcd.noindex/config.json`. +- **A call asks for its usage unless the block says not to.** The adapter sends + `stream_options.include_usage` with every call, so the streamed answer ends + with the token counts and the cost the call record carries (itd-166). A + server that refuses the field fails the call with HTTP 400 or 422; when the + refusal's body names `stream_options` or `include_usage`, it names + `oracle.api..stream_usage`, and set to `false`, calls go without + the field and a call whose answer carries no usage records it as not + reported. No ordinary call is retried without the field, since a retry is a + second request. The setup is the one exception: `abcd ahoy connect` asks + its verification call once more without the field when the first is + refused with 400 or 422, and when that succeeds writes `stream_usage: false` + into the block it creates and says so (`stream_usage_note` in its JSON). + `--stream-usage=false` (or `=true`) sets the block's value outright, and + the verification is then asked once, as given. - **The allowlist alone decides; the denylist is optional and a union.** abcd bundles no vendor denylist (adr-2609300107513982), so a model a provider lists is served whichever vendor made it. `oracle.denylist` is the configuration's diff --git a/.abcd/development/intents/planned/itd-166-each-run-should-record-the-model.md b/.abcd/development/intents/planned/itd-166-each-run-should-record-the-model.md deleted file mode 100644 index aab1c2ba79..0000000000 --- a/.abcd/development/intents/planned/itd-166-each-run-should-record-the-model.md +++ /dev/null @@ -1,88 +0,0 @@ ---- -id: itd-166 -slug: each-run-should-record-the-model -spec_id: spc-2610170652111963 -kind: standalone -suggested_kind: null -reclassification_history: [] -builds_on: [] -severity: minor -related_issues: [iss-2608290811463906, iss-2610161709506207, iss-2610161734413178, iss-2610161734428291] -impact: additive ---- - -# A run records what it was actually run with - -## Press Release - -> **Every model call abcd makes now records two sides: what abcd asked for, and what actually answered.** A role run through the host, a command-line runner or a provider connection names the model it requested and the settings it sent, then the model, provider, token counts and cost the answer reported. Whatever the route cannot show is written down as unpinned or self-reported, never left out and never guessed. -> -> "A review came back weaker than last week's and I spent an hour blaming the diff," said a technical facilitator. "The record now shows the provider had served a different model under the same name. It took one line to see." - -## Why This Matters - -A model identifier does not determine behaviour, so a record that names only the identifier does not make a run interpretable or reproducible. Whatever the model exposes belongs alongside it: reasoning or thinking depth, sampling parameters, a seed where one exists, the context-window variant, and any speed or quality mode. - -Two runs stamped with the same identifier can differ on all of these, so a record naming only the identifier invites a false comparison between them. An audit verdict, a benchmark, or a regression blamed on a code change may in fact be a settings difference. That matters most where a verdict is treated as evidence. - -This is facilitator-tier diagnostic data and it never surfaces to the product thinker: its use is the trace backwards when a report arrives that the product does not match what was expected. That also settles where it lives, since a product-thinker-facing record carries none of it. - -The hard constraint is that a model cannot reliably introspect the settings it runs under, so the dispatching layer has to supply them. Any design that asks the agent to state its own settings inherits the truthfulness problem the attribution gate already has. - -Widened by the decomposition of itd-2610161733316518 (2026-10-16, routing confirmed by the product thinker): the record has two sides. The **asked** side is what the dispatching layer requested: model and snapshot, provider and any provider pin, sampling, seed, effort, context, and the prompt and tool definitions by hash. The **served** side is what actually answered: the model and upstream provider the response names, any fallback taken, token counts including reasoning and cached, and cost where the provider reports it. A local model adds the weight file's fingerprint, its quantisation, the runtime and its version, and the served context. Whatever a route cannot expose is written down as unpinned or self-reported rather than left out (see the principle that a model run is re-runnable, not reproducible). - -Today (checked 2026-10-17): the provider adapter records the model asked and the model the API reported, and drops the usage it already parses; the claude runner reads the served model from its init event and ignores the usage and cost in its result event; the opencode runner stamps the asked model as the served one (iss-2610161709506207); no route records a prompt hash, an upstream provider, a generation id or a cost. - -## Mechanism - -We expect a call record built by the dispatching layer, from the request it sent and the response it read, to show a settings or model difference that a record naming only the asked model hides, because the layer that sends the request and reads the answer is the only one that sees both without asking the model about itself. - -## Scope Conditions - -- Holds for the routes abcd dispatches through today: the claude and opencode command-line runners, the OpenAI-compatible provider adapter, and the oracle's hand-off to the host harness. A loop role the host runs as its own sub-agent keeps the receipt it has today, which carries no route. -- Holds only for what each route's own output carries: a field the response or event stream does not name is recorded as unpinned or self-reported, and no extra network request is made to fill it. -- Holds for local models only as far as the asked side configured them: the weight fingerprint, quantisation and runtime version are recorded once a local route can be pinned (iss-2610161734428291), and are marked unpinned until then. - -## Acceptance Criteria - -- Given a role routed to the provider adapter, when the call returns, then its route receipt carries the asked side (model, every setting sent, and a sha256 of the rendered messages) and the served side (the model the response names, the upstream provider when the response names one, the generation id, the finish reason, prompt, completion, reasoning and cached token counts, and the cost when the response reports it). -- Given a role run by the claude runner, when the run ends, then its route record carries the asked model it passed, the served models the result event's per-model usage names, the token counts and the cost from the result event, and the harness version from the init event. -- Given a role run by the opencode runner, when the run ends, then the served model is empty and listed as not reported (its run mode names no model), never a copy of the asked model, and the token counts and cost are summed across its step-finish events. -- Given any route, when a field of the fixed asked or served list cannot be read from what the route returned, then the record lists that field by name with its reason (not reported, self-reported, computed by the harness, or not configured) rather than omitting it. -- Given a role the oracle hands to the host harness, when its receipt is written, then the served model is the payload's own declaration listed as self-reported. -- Given a provider whose server refuses the usage-in-stream request, when the person turns that request off for the provider, then calls succeed with usage listed as not reported, and the refusal's error names the setting. -- Given the run state, a lane receipt and the consistency answer's route file, when one is written, then the two sides travel with the existing route records, every string a provider or harness supplied is cleaned and bounded, and no new file is written. -- Given a call, when its record is built, then building it adds no request to the ones the call already makes. - -## Decisions - -Recorded by the technical facilitator on the product thinker's delegation (2026-10-17: "build autonomously"), each with its reason, for review on return. A design review pair (Opus and Sonnet, 2026-10-17) returned NEEDS-FIXES on the first plan; every finding is applied below (import direction, opencode's run mode, the schema bump, masking of route files, the opt-out, the request-count test, the fixed lists, harness-computed cost). - -1. **Who reads it (the open question).** The record stays facilitator-tier. The model-matrix report shows the product thinker a drift flag and the names of the fields that drifted, never their values; the values are read from the run record. Reason: the draft's own constraint, and itd-2610161733316518's decision that a re-run reports drift field by field. -2. **Extend, do not add, a receipt.** The two sides extend `oracle.CallRecord` and `oracle.ReceiptRoute` (the override's receipt) and `runner.RouteRecord` (the runner's). The shared vocabulary lives in `oracle`, which the runner package already imports, so neither imports the other the wrong way. Reason: itd-2610161733316518 decision 14 says the sides extend the route receipt the override already writes; a second receipt would be a second place to disagree. -3. **No extra request.** OpenRouter's generation record is not fetched here; what the response itself carries (its `id`, `provider`, `usage` with cost and token details) is recorded, and the readback belongs to the OpenRouter pin (iss-2610161734413178) inside a consented run. Reason: itd-2610161733316518 decision 12 (every network call inside the run the person started) and this record's scope condition 2. -4. **Usage in the stream, with an opt-out.** The adapter asks for usage in the stream with `stream_options.include_usage`, which OpenAI, vLLM, OpenRouter and Ollama accept and llama.cpp and LM Studio accept or ignore. A provider whose server refuses it is given an opt-out on its connection (default on), and a refusal names that setting. No automatic retry, since a retry is a second request. No provider-specific request field is added. Reason: a server that refused the field would otherwise turn a working route into a failure. -5. **The prompt by hash.** The rendered messages are recorded as a sha256, never their text. Reason: the run's transcripts already hold the text where the person has kept them, and a receipt that carries prompts carries whatever they quoted. -6. **The opencode fix lands here, as its remedy's second branch.** opencode 1.18.31's run mode names no model in its JSON events (checked in the installed binary by the review of 2026-10-17), so the runner stops copying the asked model, leaves the served model empty and lists it as not reported. That resolves iss-2610161709506207. -7. **Local fingerprints wait on the local pin.** The asked-side weight fingerprint, quantisation and runtime version need a configured pin that does not exist; until iss-2610161734428291 lands, a local route lists them as not configured. Reason: nothing can be recorded that was never configured, and guessing it from the server would be self-reported at best. -8. **Four reasons for a missing field.** Not reported (the route's output does not carry it), self-reported (the model or payload declared it), computed by the harness (claude's and opencode's cost come from the harness's own price table, not the provider), not configured (no pin exists). Reason: the review found that harness-computed cost is neither reported nor self-reported, and a reader must be able to tell the four apart. -9. **Run state moves to schema 13.** Older binaries read run state strictly, so the new route fields bump the version under the existing rule: a version-12 file reads as not recorded and is written back at 13. Reason: the run-state reader refuses unknown fields by design. -10. **Fixed field lists.** Asked: model, provider pin, each sampling setting, seed, effort, context, prompt sha256, harness version. Served: model, provider, generation id, finish reason, prompt, completion, reasoning, cache-read and cache-write tokens, cost. Each route lists exactly the fields it lacks. Reason: criterion 4 is testable only against a fixed list. - -## Open Questions - -None. The one open question was decided on delegation (decision 1). - -## Typed Links - -- **blocks `itd-2610161733316518`** (one feature built across a model matrix): the run's manifest is this record, and nothing it compares is interpretable without it. -- **resolves `iss-2610161709506207`** (the opencode runner stamps the asked model): decision 6. -- **refined later by `iss-2610161734413178` and `iss-2610161734428291`**: the OpenRouter and local pins fill fields this record marks unpinned. - -## Audit Notes - -_Empty. Populated by intent-auditor when intent moves to shipped/._ - -## Grounds - -- pursued: we expect a record built by the dispatching layer from the request it sent and the response it read to expose a model or settings difference that the asked model alone hides; a matrix re-run whose drift report names no field that the receipts show changed would show it wrong diff --git a/.abcd/development/intents/shipped/itd-166-each-run-should-record-the-model.md b/.abcd/development/intents/shipped/itd-166-each-run-should-record-the-model.md new file mode 100644 index 0000000000..cdee1755b1 --- /dev/null +++ b/.abcd/development/intents/shipped/itd-166-each-run-should-record-the-model.md @@ -0,0 +1,197 @@ +--- +id: itd-166 +slug: each-run-should-record-the-model +spec_id: spc-2610170652111963 +kind: standalone +suggested_kind: null +reclassification_history: [] +builds_on: [] +severity: minor +related_issues: [iss-2608290811463906, iss-2610161709506207] +impact: additive +--- + +# A run records what it was actually run with + +## Press Release + +> **Every model call abcd makes now records two sides: what abcd asked for, and what actually answered.** A role run through the host, a command-line runner or a provider connection names the model it requested and the settings it sent, then the model, provider, token counts and cost the answer reported. Whatever the route cannot show is written down as unpinned or self-reported, never left out and never guessed. +> +> "A review came back weaker than last week's and I spent an hour blaming the diff," said a technical facilitator. "The record now shows the provider had served a different model under the same name. It took one line to see." + +## Why This Matters + +A model identifier does not determine behaviour, so a record that names only the identifier does not make a run interpretable or reproducible. Whatever the model exposes belongs alongside it: reasoning or thinking depth, sampling parameters, a seed where one exists, the context-window variant, and any speed or quality mode. + +Two runs stamped with the same identifier can differ on all of these, so a record naming only the identifier invites a false comparison between them. An audit verdict, a benchmark, or a regression blamed on a code change may in fact be a settings difference. That matters most where a verdict is treated as evidence. + +This is facilitator-tier diagnostic data and it never surfaces to the product thinker: its use is the trace backwards when a report arrives that the product does not match what was expected. That also settles where it lives, since a product-thinker-facing record carries none of it. + +The hard constraint is that a model cannot reliably introspect the settings it runs under, so the dispatching layer has to supply them. Any design that asks the agent to state its own settings inherits the truthfulness problem the attribution gate already has. + +Widened by the decomposition of itd-2610161733316518 (2026-10-16, routing confirmed by the product thinker): the record has two sides. The **asked** side is what the dispatching layer requested: model and snapshot, provider and any provider pin, sampling, seed, effort, context, and the prompt and tool definitions by hash. The **served** side is what actually answered: the model and upstream provider the response names, any fallback taken, token counts including reasoning and cached, and cost where the provider reports it. A local model adds the weight file's fingerprint, its quantisation, the runtime and its version, and the served context. Whatever a route cannot expose is written down as unpinned or self-reported rather than left out (see the principle that a model run is re-runnable, not reproducible). + +Today (checked 2026-10-17): the provider adapter records the model asked and the model the API reported, and drops the usage it already parses; the claude runner reads the served model from its init event and ignores the usage and cost in its result event; the opencode runner stamps the asked model as the served one (iss-2610161709506207); no route records a prompt hash, an upstream provider, a generation id or a cost. + +## Mechanism + +We expect a call record built by the dispatching layer, from the request it sent and the response it read, to show a settings or model difference that a record naming only the asked model hides, because the layer that sends the request and reads the answer is the only one that sees both without asking the model about itself. + +## Scope Conditions + +- Holds for the routes abcd dispatches through today: the claude and opencode command-line runners, the OpenAI-compatible provider adapter, and the oracle's hand-off to the host harness. A loop role the host runs as its own sub-agent keeps the receipt it has today, which carries no route. +- Holds only for what each route's own output carries: a field the response or event stream does not name is recorded as unpinned or self-reported, and no extra network request is made to fill it. +- Holds for local models only as far as the asked side configured them: the weight fingerprint, quantisation and runtime version are recorded once a local route can be pinned (iss-2610161734428291), and are marked unpinned until then. + +## Acceptance Criteria + +- Given a role routed to the provider adapter, when the call returns, then its route receipt carries the asked side (model, every setting sent, and a sha256 of the rendered messages) and the served side (the model the response names, the upstream provider when the response names one, the generation id, the finish reason, prompt, completion, reasoning and cached token counts, and the cost when the response reports it). +- Given a role run by the claude runner, when the run ends, then its route record carries the asked model it passed, the served models the result event's per-model usage names, the token counts and the cost from the result event, and the harness version from the init event. +- Given a role run by the opencode runner, when the run ends, then the served model is empty and listed as not reported (its run mode names no model), never a copy of the asked model, and the token counts and cost are summed across its step-finish events. +- Given any route, when a field of the fixed asked or served list cannot be read from what the route returned, then the record lists that field by name with its reason (not reported, self-reported, computed by the harness, or not configured) rather than omitting it. +- Given a role the oracle hands to the host harness, when its receipt is written, then the served model is the payload's own declaration listed as self-reported. +- Given a provider whose server refuses the usage-in-stream request, when the person turns that request off for the provider, then calls succeed with usage listed as not reported, and the refusal's error names the setting. +- Given the run state, a lane receipt and the consistency answer's route file, when one is written, then the two sides travel with the existing route records, every string a provider or harness supplied is cleaned and bounded, and no new file is written. +- Given a call, when its record is built, then building it adds no request to the ones the call already makes. + +## Decisions + +Recorded by the technical facilitator on the product thinker's delegation (2026-10-17: "build autonomously"), each with its reason, for review on return. A design review pair (Opus and Sonnet, 2026-10-17) returned NEEDS-FIXES on the first plan; every finding is applied below (import direction, opencode's run mode, the schema bump, masking of route files, the opt-out, the request-count test, the fixed lists, harness-computed cost). + +1. **Who reads it (the open question).** The record stays facilitator-tier. The model-matrix report shows the product thinker a drift flag and the names of the fields that drifted, never their values; the values are read from the run record. Reason: the draft's own constraint, and itd-2610161733316518's decision that a re-run reports drift field by field. +2. **Extend, do not add, a receipt.** The two sides extend `oracle.CallRecord` and `oracle.ReceiptRoute` (the override's receipt) and `runner.RouteRecord` (the runner's). The shared vocabulary lives in `oracle`, which the runner package already imports, so neither imports the other the wrong way. Reason: itd-2610161733316518 decision 14 says the sides extend the route receipt the override already writes; a second receipt would be a second place to disagree. +3. **No extra request.** OpenRouter's generation record is not fetched here; what the response itself carries (its `id`, `provider`, `usage` with cost and token details) is recorded, and the readback belongs to the OpenRouter pin (iss-2610161734413178) inside a consented run. Reason: itd-2610161733316518 decision 12 (every network call inside the run the person started) and this record's scope condition 2. +4. **Usage in the stream, with an opt-out.** The adapter asks for usage in the stream with `stream_options.include_usage`, which OpenAI, vLLM, OpenRouter and Ollama accept and llama.cpp and LM Studio accept or ignore. A provider whose server refuses it is given an opt-out on its connection (default on), and a refusal that names the field names that setting. An ordinary call is never retried, since a retry is a second request; the one exception, decided in the build's review (2026-10-17), is `ahoy connect`, whose verification is asked once more without the field after a 400 or 422 and, when that succeeds, writes the opt-out, because a setup that cannot get past the refusal leaves the person nothing to turn off. No provider-specific request field is added. Reason: a server that refused the field would otherwise turn a working route into a failure. +5. **The prompt by hash.** The rendered messages are recorded as a sha256, never their text. Reason: the run's transcripts already hold the text where the person has kept them, and a receipt that carries prompts carries whatever they quoted. +6. **The opencode fix lands here, as its remedy's second branch.** opencode 1.18.31's run mode names no model in its JSON events (checked in the installed binary by the review of 2026-10-17), so the runner stops copying the asked model, leaves the served model empty and lists it as not reported. That resolves iss-2610161709506207. +7. **Local fingerprints wait on the local pin.** The asked-side weight fingerprint, quantisation and runtime version need a configured pin that does not exist; until iss-2610161734428291 lands, a local route lists them as not configured. Reason: nothing can be recorded that was never configured, and guessing it from the server would be self-reported at best. +8. **Four reasons for a missing field.** Not reported (the route's output does not carry it), self-reported (the model or payload declared it), computed by the harness (claude's and opencode's cost come from the harness's own price table, not the provider), not configured (no pin exists). Reason: the review found that harness-computed cost is neither reported nor self-reported, and a reader must be able to tell the four apart. +9. **Run state moves to schema 14.** Older binaries read run state strictly, so the new route fields bump the version under the existing rule: a version-13 file reads as not recorded and is written back at 14. Reason: the run-state reader refuses unknown fields by design. (Built as 13; renumbered to 14 when release v0.15.1 took 13 for a drain lane's `refused`.) +10. **Fixed field lists.** Asked: model, provider pin, each sampling setting, seed, effort, context, prompt sha256, harness version. Served: model, provider, generation id, finish reason, prompt, completion, reasoning, cache-read and cache-write tokens, cost. Each route lists exactly the fields it lacks. Reason: criterion 4 is testable only against a fixed list. + +## Open Questions + +None. The one open question was decided on delegation (decision 1). + +## Typed Links + +- **blocks `itd-2610161733316518`** (one feature built across a model matrix): the run's manifest is this record, and nothing it compares is interpretable without it. +- **resolves `iss-2610161709506207`** (the opencode runner stamps the asked model): decision 6. +- **refined later by `iss-2610161734413178` and `iss-2610161734428291`**: the OpenRouter and local pins fill fields this record marks unpinned. + +## Audit Notes + + +Fidelity review — receipt rcp-f25a69346de1 (verifier intent-auditor claude-opus-5-5). + +Provenance: intent-auditor@claude-opus-5-5 · rubric_hash sha256:effa65b3e9e88ff29433b443ec2be159522a8b0b71cf1434526514aa61edb13e · prompt_hash sha256:beafc9d930e3df84cf97e1f5dd434a99d9edc2c29b38c1df4d1b066c7e38d00d +Input attestations: diff:7ae8fdde8..f5a25ce84@sha256:9ad6cfb8962de4c4e72d23e030a86520334354f4f25506c6a29eab57ce603037; tree:f5a25ce84@-; + +Acceptance rollup: MET 6 · MET_WITH_CONCERNS 2 · NOT_MET 0 · INCONCLUSIVE 0 + +Per-criterion verdicts: +- ac-1 — MET: callRecord builds the asked side (model asked, prompt sha256 of the rendered messages, adapter-added field) beside the receipt's existing settings_sent, and the served side (reported model, upstream provider, generation id, finish reason, prompt/completion/reasoning/cache-read/cache-write counts, cost in credits) from the adapter's Result; the dispatch attaches it to the route receipt and a test reads an OpenRouter-shaped answer through to the encoded receipt + evidence: internal/core/oracle/call.go:168 — "func callRecord(p Provider, brief openaiapi.Brief, settings Settings, res openaiapi.Result) CallRecord {" + evidence: internal/core/oracle/call.go:170 — "PromptSHA256: openaiapi.PromptSHA256(brief), AdapterAdded: []string{}," + evidence: internal/adapter/openaiapi/client.go:655 — "func readUsage(raw json.RawMessage) *Usage {" + evidence: internal/core/oracle/dispatch.go:57 — "return payload, r.Receipt(ModelReported(payload)).WithCall(rec), nil" + evidence: internal/core/oracle/receipt.go:72 — "SettingsSent Settings `json:"settings_sent"`" + evidence: internal/core/oracle/sides_test.go:138 — "func TestTheCallRecordTravelsInTheReceipt(t *testing.T) {" + evidence: internal/core/oracle/sides_test.go:61 — "func TestAProviderCallRecordsBothSides(t *testing.T) {" +- ac-2 — MET: the claude runner sets the asked model it passes (after the provider prefix), reads the result event's modelUsage keys as served models, its usage counts (thinking tokens as reasoning) and total_cost_usd, and the init event's claude_code_version; a test over a recorded 2.1.302 stream asserts each, with cost listed harness_computed + evidence: internal/core/runner/claude.go:99 — "ans.ModelAsked, ans.PromptSHA256 = c.askedModel(), promptSHA256(req)" + evidence: internal/core/runner/claude.go:193 — "func meterClaude(ans *Answer, line []byte) {" + evidence: internal/core/runner/claude.go:214 — "ans.Cost = &oracle.Cost{Amount: *m.TotalCostUSD, Unit: oracle.CostUSD}" + evidence: internal/core/runner/claude.go:236 — "ans.ServedModels = append(ans.ServedModels, cleanHarness(k))" + evidence: internal/core/runner/claude.go:272 — "ans.HarnessVersion = cleanHarness(ev.Version)" + evidence: internal/core/runner/sides_test.go:83 — "func TestClaudeRecordsBothSidesFromARealStream(t *testing.T) {" +- ac-3 — MET: the opencode runner no longer copies o.model into ans.Model: it records o.model as ModelAsked only, leaves the served model empty so servedUnpinned lists served.model not_reported, and meterOpenCode sums tokens and cost across step_finish events; a test over a 1.18.31 fixture asserts Model empty, ServedModels empty and the summed usage and cost + evidence: internal/core/runner/opencode.go:96 — "ans.ModelAsked, ans.PromptSHA256 = o.model, promptSHA256(req)" + evidence: internal/core/runner/opencode.go:99 — "servedUnpinned(&un, ans.Model, ans.Sides, ans.counted)" + evidence: internal/core/runner/sides.go:109 — "un.Unless(model != "", oracle.ServedModel, oracle.NotReported)" + evidence: internal/core/runner/opencode.go:137 — "func meterOpenCode(ans *Answer, line []byte) {" + evidence: internal/core/runner/opencode.go:152 — "sum += ans.Cost.Amount" + evidence: internal/core/runner/sides_test.go:153 — "func TestOpenCodeNeverStampsTheAskedModelAsServed(t *testing.T) {" +- ac-4 — MET_WITH_CONCERNS: every route builds an Unpinned list over the fixed asked and served lists with the four reasons (adapter callRecord, runner askedUnpinned/servedUnpinned, host HandOffUnpinned), and tests hold each list to the fixed fields; concern: the local fields (weights_sha256, quantisation, runtime_version) are listed not_configured only on the provider adapter when its base URL is loopback, while a runner routed to a local model (the opencode test itself uses local/qwen3-coder) omits them, and its test helper excludes LocalFields from the known list + evidence: internal/core/oracle/call.go:191 — "if openaiapi.IsLocal(p.BaseURL) {" + evidence: internal/core/runner/sides.go:91 — "func askedUnpinned(un *oracle.Unpin, s Sides) {" + evidence: internal/core/runner/sides_test.go:49 — "func assertFixedLists(t *testing.T, us []oracle.Unpinned) {" + evidence: internal/core/oracle/receipt.go:117 — "func HandOffUnpinned(model string) []Unpinned {" + evidence: internal/core/oracle/sides_test.go:97 — "func TestACallNamingNoServedDetailListsEachMissing(t *testing.T) {" +- ac-5 — MET: on the harness leg Route.Receipt attaches HandOffUnpinned(model), which lists served.model self_reported when the payload declared one (not_reported when it declared none) and every other fixed field with its reason; the model passed is ModelReported(payload); a test asserts the self_reported entry in the encoded receipt + evidence: internal/core/oracle/receipt.go:108 — "rr.Unpinned = HandOffUnpinned(model)" + evidence: internal/core/oracle/receipt.go:129 — "un.Add(f, SelfReported)" + evidence: internal/core/oracle/sides_test.go:220 — "func TestTheHostHandOffMarksTheServedModelSelfReported(t *testing.T) {" +- ac-6 — MET_WITH_CONCERNS: a provider block's stream_usage false drops stream_options from the request and the call succeeds with counts listed not_reported, and a 400/422 refusal whose body names stream_options or include_usage gets the oracle.api.< provider>.stream_usage setting named; concerns: a refusal whose body does not name the field carries no hint on an ordinary call, so the setting is named only for self-describing refusals; and setup (ahoy connect) retries the verification once without the field on any 400/422 and writes stream_usage false, which the intent's decision 4 (no automatic retry) did not provide for + evidence: internal/adapter/openaiapi/client.go:548 — "fields["stream_options"] = map[string]bool{"include_usage": true}" + evidence: internal/adapter/openaiapi/client.go:492 — "if named {" + evidence: internal/core/oracle/call.go:115 — "return fmt.Errorf("%w; if this server does not take %s, set %s.%s.%s to false in %s, "+" + evidence: internal/core/oracle/sides_test.go:165 — "func TestAConnectionThatTurnsStreamUsageOffSendsWithoutIt(t *testing.T) {" + evidence: internal/core/oracle/sides_test.go:194 — "func TestARefusedUsageRequestNamesTheSetting(t *testing.T) {" + evidence: internal/adapter/openaiapi/served_test.go:300 — "func TestARefusalNotNamingTheUsageFieldCarriesNoHint(t *testing.T) {" + evidence: internal/core/oracle/connect.go:526 — "case p.StreamUsage == nil && errors.Is(err, openaiapi.ErrUsageRefused):" +- ac-7 — MET: the runner's Sides are embedded in the existing runner.RouteRecord that the dispatch and the drive already store on lane receipts and the run state (bumped to schema 14 with version-13 files read as not recorded), the provider call record rides the existing ReceiptRoute the consistency route file holds, maskRoute masks every new string, adapter strings go through scrub and cleanModel and harness strings through cleanHarness, and a test asserts the reviews directory gains no new kind of file + evidence: internal/core/runner/receipt.go:24 — "Sides" + evidence: internal/core/implement/loop/drive.go:141 — "rec := runner.RecordOf(route.Runner, ran, ans)" + evidence: internal/core/implement/loop/state.go:194 — "const SchemaVersion = 14" + evidence: internal/core/implement/loop/state_mask.go:177 — "func maskRoute(r runner.RouteRecord) runner.RouteRecord {" + evidence: internal/adapter/openaiapi/client.go:747 — "Provider: cleanModel(c.scrub(a.provider)), Usage: a.usage}" + evidence: internal/core/runner/sides.go:75 — "func cleanHarness(s string) string {" + evidence: internal/surface/cli/sides_test.go:60 — "func TestTheConsistencyRouteFileCarriesBothSides(t *testing.T) {" + evidence: internal/core/implement/loop/sides_test.go:112 — "func TestTheDrivePassesBothSidesThrough(t *testing.T) {" +- ac-8 — MET: callRecord and the runners' meters are pure functions of the request sent and the response or event stream read, and a test counts requests on a sized vLLM-shaped server with and without the usage request and served detail and finds them equal; the setup retry in Connect is a second verification request, not part of building any record (recorded under ac-6 and the gap audit) + evidence: internal/adapter/openaiapi/served_test.go:204 — "func TestRecordingTheServedSideAddsNoRequest(t *testing.T) {" + evidence: internal/core/oracle/call.go:168 — "func callRecord(p Provider, brief openaiapi.Brief, settings Settings, res openaiapi.Result) CallRecord {" + evidence: internal/core/oracle/sides_test.go:194 — "func TestARefusedUsageRequestNamesTheSetting(t *testing.T) {" + +Gap audit: +- honoured: + - every model call records what abcd asked for and what answered, on the provider adapter, both command-line runners and the host hand-off + evidence: internal/core/oracle/sides.go:56 — "AskedFields = []Field{AskedModel, AskedProviderPin, AskedTemperature, AskedTopP, AskedSeed," + evidence: internal/core/oracle/call.go:168 — "func callRecord(p Provider, brief openaiapi.Brief, settings Settings, res openaiapi.Result) CallRecord {" + evidence: internal/core/runner/claude.go:193 — "func meterClaude(ans *Answer, line []byte) {" + evidence: internal/core/oracle/receipt.go:117 — "func HandOffUnpinned(model string) []Unpinned {" + - whatever a route cannot show is written down with a reason (not reported, self-reported, harness-computed, not configured), never left out or guessed + evidence: internal/core/oracle/sides.go:67 — "const (" + evidence: internal/core/runner/sides.go:121 — "un.Add(oracle.ServedCost, oracle.HarnessComputed)" + - the opencode runner stops stamping the asked model as served (iss-2610161709506207) + evidence: internal/core/runner/opencode.go:96 — "ans.ModelAsked, ans.PromptSHA256 = o.model, promptSHA256(req)" + evidence: internal/core/runner/sides_test.go:153 — "func TestOpenCodeNeverStampsTheAskedModelAsServed(t *testing.T) {" + - a served model differing from the asked one under the same name is visible on the route line, with upstream provider and cost + evidence: internal/surface/cli/dispatch.go:115 — "line += fmt.Sprintf(", model asked %s, model reported %s", pc.ModelAsked, pc.ModelReported)" + evidence: internal/surface/cli/sides_test.go:26 — "func TestTheRouteLineNamesTheAskedModelAndCost(t *testing.T) {" + - the prompt is recorded by sha256, never its text (decision 5) + evidence: internal/core/oracle/call.go:170 — "PromptSHA256: openaiapi.PromptSHA256(brief), AdapterAdded: []string{}," +- diverged: + - decision 4 says no automatic retry, since a retry is a second request; delivered: ahoy connect's verification is asked once more without stream_options on any HTTP 400/422 (whatever the body names) and the block is written with stream_usage false (iss-2610170934491794), while the intent's decision text was left unchanged + evidence: .abcd/development/intents/shipped/itd-166-each-run-should-record-the-model.md:64 — "No automatic retry, since a retry is a second request." + evidence: internal/core/oracle/connect.go:530 — "if err := verify(without); err != nil {" + evidence: internal/core/oracle/connect_test.go:602 — "func TestConnectAsksAgainWithoutUsageWhenItIsRefused(t *testing.T) {" + - a refusal names the setting (decision 4) only when the refusal body names stream_options or include_usage; a bare 400/422 on an ordinary call carries no hint + evidence: internal/adapter/openaiapi/client.go:183 — "func namesStreamUsage(body string) bool {" + evidence: internal/adapter/openaiapi/served_test.go:300 — "func TestARefusalNotNamingTheUsageFieldCarriesNoHint(t *testing.T) {" + - a local model's weight fingerprint, quantisation and runtime are listed not configured on a local route (decision 7); delivered only on the provider adapter when its base URL is loopback, not on a runner routed to a local model + evidence: internal/adapter/openaiapi/client.go:240 — "return err == nil && u.Host != "" && loopback(u.Hostname())" + evidence: internal/core/runner/sides.go:91 — "func askedUnpinned(un *oracle.Unpin, s Sides) {" +- missing: + - a generation id on the runner routes: both runners always list served.generation_id as not reported rather than recording one + evidence: internal/core/runner/sides.go:111 — "// Neither harness names a generation id for the run: a run is many" + +Scope-condition dispositions: +- cond-2610170652113675 — survived: the sides were added to exactly the four named routes (claude and opencode runners, the OpenAI-compatible adapter, the oracle hand-off), and a host-run loop role's RouteRecord keeps its old shape because Sides is embedded with every field omitempty and set only when a runner ran + evidence: internal/core/runner/receipt.go:24 — "Sides" + evidence: internal/core/runner/dispatch.go:156 — "o.Receipt.Route = RecordOf(o.Receipt.Route.Asked, route, ans)" + evidence: internal/core/oracle/receipt.go:108 — "rr.Unpinned = HandOffUnpinned(model)" +- cond-2610170652112593 — survived: every served value is read from the response or the harness event stream, absent ones are listed unpinned, no generation readback is fetched, and the request count is equal with and without the record; the setup retry is a second verification request, not a fetch to fill a field + evidence: internal/adapter/openaiapi/served_test.go:204 — "func TestRecordingTheServedSideAddsNoRequest(t *testing.T) {" + evidence: internal/core/oracle/call.go:207 — "un.Unless(u != nil && u.Cost != nil, ServedCost, NotReported)" +- cond-2610170652113516 — narrowed: the local fields are marked not_configured until a pin exists, but only where the provider adapter's base URL is a loopback address; the runners never list them, even for a local model (the opencode test uses local/qwen3-coder), and a local server on a non-loopback address is not recognised as local + narrowing: holds for the provider adapter route whose base URL is localhost or a loopback address only; command-line runner routes to local models and local servers reached on non-loopback addresses record no local-field entries at all + evidence: internal/core/oracle/call.go:191 — "if openaiapi.IsLocal(p.BaseURL) {" + evidence: internal/adapter/openaiapi/client.go:240 — "return err == nil && u.Host != "" && loopback(u.Hostname())" + evidence: internal/core/runner/sides_test.go:49 — "func assertFixedLists(t *testing.T, us []oracle.Unpinned) {" + + +## Grounds + +- pursued: we expect a record built by the dispatching layer from the request it sent and the response it read to expose a model or settings difference that the asked model alone hides; a matrix re-run whose drift report names no field that the receipts show changed would show it wrong diff --git a/.abcd/development/release/surface.json b/.abcd/development/release/surface.json index d3c575abde..92c5316af3 100644 --- a/.abcd/development/release/surface.json +++ b/.abcd/development/release/surface.json @@ -90,7 +90,7 @@ { "path": "abcd ahoy connect", "hidden": false, - "sentence": "Verify a provider with one call and configure it, or guide setup: Writes ~/.abcd.noindex/ and the key's home, nothing guided; refuses no --model off a terminal.", + "sentence": "Verify a provider and configure it, or guide setup: Writes ~/.abcd.noindex/ and the key's home, nothing guided; refuses no --model off a terminal.", "flags": [ { "name": "answer", @@ -161,6 +161,13 @@ "type": "string", "required": false, "hidden": false + }, + { + "name": "stream-usage", + "shorthand": "", + "type": "bool", + "required": false, + "hidden": false } ] }, diff --git a/.abcd/development/specs/open/spc-2610170652111963-each-run-should-record-the-model.md b/.abcd/development/specs/closed/spc-2610170652111963-each-run-should-record-the-model.md similarity index 91% rename from .abcd/development/specs/open/spc-2610170652111963-each-run-should-record-the-model.md rename to .abcd/development/specs/closed/spc-2610170652111963-each-run-should-record-the-model.md index 4083df7bbb..f2df95be52 100644 --- a/.abcd/development/specs/open/spc-2610170652111963-each-run-should-record-the-model.md +++ b/.abcd/development/specs/closed/spc-2610170652111963-each-run-should-record-the-model.md @@ -17,14 +17,14 @@ Delivers itd-166: every model call records the side abcd asked for and the side - **Shared vocabulary in `internal/core/oracle`** (the runner package already imports oracle; oracle imports nothing from runner): `Usage` (prompt, completion, reasoning, cache-read, cache-write tokens), `Cost` (a pointer, so zero is not confused with unreported, with its unit fixed per route), and `Unpinned` (field name and reason: `not_reported`, `self_reported`, `harness_computed`, `not_configured`) checked against the fixed asked and served lists of decision 10. - **Provider adapter path** (`internal/adapter/openaiapi`, `internal/core/oracle`): the request adds `stream_options.include_usage` unless the provider's connection turns it off (`stream_usage: false`, default on), and a refusal's error names that setting; it is recorded as adapter-added, not as a setting sent. `Result`, `Usage` and the stream `chunk` gain the generation `id`, a top-level `provider`, `prompt_tokens_details.cached_tokens`, `completion_tokens_details.reasoning_tokens` and `cost`; the parser keeps reading past the finish reason to `[DONE]` for the usage chunk. Every provider-supplied string is cleaned and bounded in `admit`, as the reported model is. `complete()` hashes the rendered messages. `CallRecord` gains prompt sha256, provider, generation id, finish reason, usage, cost and the unpinned list; the harness hand-off marks `ReceiptRoute`'s served model self-reported. No request is added. - **Runner path** (`internal/core/runner`): `Answer` gains usage, cost, served models and harness version, and its comment stops saying "or the one it was asked for". The claude runner reads the result event's `usage` and `total_cost_usd` (harness-computed), the served models from `modelUsage`'s keys (cross-checked against the init event's model), and `claude_code_version`; reasoning tokens are listed not reported. The opencode runner stops stamping the asked model (resolves iss-2610161709506207): served model empty and not reported, tokens and cost summed across `step_finish` events (cost harness-computed; sub-session events are already filtered out, so the totals are a floor and say so). `RouteRecord` gains the asked model, usage, cost, harness version and unpinned list; `Route.Model` is the canonical served model and `ReceiptRecord.Model` stays as the legacy copy. -- **Persistence**: no new file. Run state moves to schema 13 (a version-12 file reads as not recorded and is written back at 13; a file at 12 or lower carrying the new fields is refused). `maskRoute` names every new string and slice. The consistency answer's `route.json` carries the oracle receipt, cleaned at the adapter. +- **Persistence**: no new file. Run state moves to schema 14 (a version-13 file reads as not recorded and is written back at 14; a file at 13 or lower carrying the new fields is refused). Built as 13; renumbered when release v0.15.1 took 13 for a drain lane's `refused`. `maskRoute` names every new string and slice. The consistency answer's `route.json` carries the oracle receipt, cleaned at the adapter. **How each acceptance criterion is met.** (1) adapter both sides: step 1; (2) claude: step 2; (3) opencode: step 2; (4) reasons against fixed lists: steps 1 and 2; (5) host hand-off self-reported: step 1; (6) usage opt-out: step 1; (7) receipts, cleaning, no new file: steps 1 and 3; (8) no added request: step 1. ## Footprint - packages: internal/adapter/openaiapi, internal/core/oracle, internal/core/runner, internal/core/implement/loop, internal/core/intent, internal/surface/cli -- tests: OpenRouter-shaped streamed and plain responses fill provider, generation id, finish reason, usage details and cost; a usage chunk after the finish reason is read; absent fields are listed with their reason; the request carries `stream_options.include_usage` unless the connection turns it off, and a refusing server's error names the setting; the request count is the same with and without the record; the prompt hash is stable and sensitive to one byte; control characters in provider, generation id and model are cleaned; a claude result event's usage, cost, `modelUsage` keys and version are read from a fixture taken from a real stream; an opencode stream's step-finish tokens and cost are summed and its served model is empty and not reported; the oracle's host hand-off marks the served model self-reported; state.json at 13 round-trips, at 12 reads as not recorded, and an older file carrying the new fields is refused; the mask covers each new field; drive passes the asked model and usage through +- tests: OpenRouter-shaped streamed and plain responses fill provider, generation id, finish reason, usage details and cost; a usage chunk after the finish reason is read; absent fields are listed with their reason; the request carries `stream_options.include_usage` unless the connection turns it off, and a refusing server's error names the setting; the request count is the same with and without the record; the prompt hash is stable and sensitive to one byte; control characters in provider, generation id and model are cleaned; a claude result event's usage, cost, `modelUsage` keys and version are read from a fixture taken from a real stream; an opencode stream's step-finish tokens and cost are summed and its served model is empty and not reported; the oracle's host hand-off marks the served model self-reported; state.json at 14 round-trips, at 13 reads as not recorded, and an older file carrying the new fields is refused; the mask covers each new field; drive passes the asked model and usage through ## Steps @@ -36,4 +36,4 @@ Delivers itd-166: every model call records the side abcd asked for and the side - tests: claude result usage, harness-computed cost, `modelUsage` served models and version read from a real-stream fixture, reasoning listed not reported; opencode step-finish tokens and cost summed, served model empty and not reported, never the asked model; the asked model kept beside the served one 3. The sides travel with the existing receipts - packages: internal/core/implement/loop, internal/core/intent, internal/surface/cli - - tests: run state schema 13 with the version-12 rule; the lane receipt and the consistency route file round-trip the new fields with no new file; `maskRoute` covers each new string and slice; drive passes the asked model and usage through; the route line render names the served model and cost when present + - tests: run state schema 14 with the version-13 rule; the lane receipt and the consistency route file round-trip the new fields with no new file; `maskRoute` covers each new string and slice; drive passes the asked model and usage through; the route line render names the served model and cost when present diff --git a/.abcd/work/issues/open/iss-2610161734413178-abcd-cannot-hold-an-openrouter-call-to.md b/.abcd/work/issues/open/iss-2610161734413178-abcd-cannot-hold-an-openrouter-call-to.md index 0055ee5770..490845fc06 100644 --- a/.abcd/work/issues/open/iss-2610161734413178-abcd-cannot-hold-an-openrouter-call-to.md +++ b/.abcd/work/issues/open/iss-2610161734413178-abcd-cannot-hold-an-openrouter-call-to.md @@ -10,7 +10,7 @@ origin: researcher-authored production_mode: hand-written found_at: "internal/core/runner/opencode.go" remedy: "Two seams, one for each path a call takes. For a tool-using role run through the opencode runner, abcd writes a sealed per-run opencode provider configuration carrying the pin (provider order, fallbacks off, required parameters, quantisations, data collection denied, the canonical model slug), records its hash and opencode's version with the route, and reads OpenRouter's generation record for each call where the generation id is available. For a self-contained call through the OpenAI-compatible adapter, the listed route carries the same pin and the adapter reads the same record into the route receipt. Waits on the planning of itd-2610161733316518, which decides the fallback policy." -related_intents: [itd-2610161733316518, itd-166] +related_intents: [itd-2610161733316518] --- abcd cannot hold an OpenRouter call to one named upstream provider, and it records only part of what served it. diff --git a/.abcd/work/issues/open/iss-2610161734428291-no-local-model-can-be-pinned-for-a.md b/.abcd/work/issues/open/iss-2610161734428291-no-local-model-can-be-pinned-for-a.md index d0eded0db1..1231465269 100644 --- a/.abcd/work/issues/open/iss-2610161734428291-no-local-model-can-be-pinned-for-a.md +++ b/.abcd/work/issues/open/iss-2610161734428291-no-local-model-can-be-pinned-for-a.md @@ -10,7 +10,7 @@ origin: researcher-authored production_mode: hand-written found_at: "internal/core/runner/opencode.go" remedy: "Give a local route a pin abcd records with every call: runtime and version, weight file sha256 or model digest, quantisation, served context length and KV-cache type, written into the sealed per-run opencode configuration for a tool-using role and onto the connection for a self-contained call. Probe the server once before use for tool-call support and its served context, and refuse a build-role route to a server that fails the probe, naming the runtime and the setting that would fix it." -related_intents: [itd-2610161733316518, itd-166] +related_intents: [itd-2610161733316518] --- No local model can be pinned for a tool-using build role. Nothing records the weight file's sha256 or Hugging Face revision, the quantisation, the runtime and its version, or the served context length, and the known traps go unchecked: diff --git a/.abcd/work/issues/open/iss-2610171545099084-a-runner-pointed-at-a-local-model-leaves.md b/.abcd/work/issues/open/iss-2610171545099084-a-runner-pointed-at-a-local-model-leaves.md new file mode 100644 index 0000000000..79520d51b1 --- /dev/null +++ b/.abcd/work/issues/open/iss-2610171545099084-a-runner-pointed-at-a-local-model-leaves.md @@ -0,0 +1,15 @@ +--- +schema_version: 1 +id: "iss-2610171545099084" +slug: "a-runner-pointed-at-a-local-model-leaves" +severity: "minor" +category: "bug" +source: "impl-review" +found_during: "intent audit of itd-166 (rcp-f25a69346de1)" +origin: researcher-authored +production_mode: dictated-and-formatted +found_at: "internal/core/runner/sides.go" +remedy: "When the local pin (iss-2610161734428291) lands, a runner route whose provider it declares local lists the three local fields as not configured until the pin fills them, as the adapter's loopback route does; tested with the opencode runner on a local provider." +--- + +A runner pointed at a local model leaves the weight fingerprint, quantisation and runtime version out of its record entirely, where the provider adapter's loopback route lists them as not configured; the intent audit of itd-166 (ac-4) found it with the opencode runner on local/qwen3-coder. A runner cannot tell from its own output that a model is served locally, so the fix needs the local pin's notion of which providers are local. diff --git a/.abcd/work/issues/open/iss-2610180052110870-a-record-declined-in-wontfix-as-a.md b/.abcd/work/issues/open/iss-2610180052110870-a-record-declined-in-wontfix-as-a.md new file mode 100644 index 0000000000..87a29e6152 --- /dev/null +++ b/.abcd/work/issues/open/iss-2610180052110870-a-record-declined-in-wontfix-as-a.md @@ -0,0 +1,15 @@ +--- +schema_version: 1 +id: "iss-2610180052110870" +slug: "a-record-declined-in-wontfix-as-a" +severity: "minor" +category: "bug" +source: "agent-finding" +found_during: "shipping itd-166 (issue-drift shipped_unresolved)" +origin: researcher-authored +production_mode: dictated-and-formatted +found_at: "internal/core/capture/drift.go" +remedy: "Let capture resolve take a wontfix record whose related_intents name the intent being shipped, moving it to resolved/ with the resolve verb's fields and dropping wontfix_reason, or have spec close do it for every such record of the intent it ships; tested by shipping an intent with a duplicate declined in its favour." +--- + +A record declined in wontfix/ as a duplicate of an intent cannot be moved to resolved/ when that intent ships: capture resolve refuses it as already in wontfix, yet the issue-drift gate refuses a shipped intent whose related_issues name a wontfix record, so the only way through is a hand move of the record (done for iss-2608290811463906 when itd-166 shipped). diff --git a/.abcd/work/issues/wontfix/iss-2608290811463906-each-run-should-record-the-model.md b/.abcd/work/issues/resolved/iss-2608290811463906-each-run-should-record-the-model.md similarity index 83% rename from .abcd/work/issues/wontfix/iss-2608290811463906-each-run-should-record-the-model.md rename to .abcd/work/issues/resolved/iss-2608290811463906-each-run-should-record-the-model.md index 6326c4e762..27839709ea 100644 --- a/.abcd/work/issues/wontfix/iss-2608290811463906-each-run-should-record-the-model.md +++ b/.abcd/work/issues/resolved/iss-2608290811463906-each-run-should-record-the-model.md @@ -10,7 +10,10 @@ found_at: "internal/core/history" related_intents: [itd-166] deferred_after: "v0.10.0" deferral_reason: "ruling owed to the product thinker (away; run A 2026-09-25): Plan itd-166 (a run records its model settings) together with the run-launch requirements of iss-2609240648533996?" -wontfix_reason: "Duplicate of itd-166 (draft), the intent this record was promoted into (itd-166 related_issues and its press-release seed both name it): recording a run's model settings beside the model id is carried whole there, together with the constraint that the harness, not the agent, supplies them. Planning itd-166 is the owed act, and it is the product thinker's." +resolution: "Delivered by itd-166, the intent this record was promoted into and first declined as a duplicate of: every model call now records the side abcd asked for (model, settings sent, prompt hash, harness version) and the side that answered (served model, provider, generation id, finish reason, token counts, cost), with each field it cannot read named with its reason." +impact: additive +resolved_by: + intent: "itd-166" --- Each run should record the model SETTINGS alongside the model id, because the id alone does not determine behaviour and so does not make a run interpretable or reproducible. Whatever the current model exposes belongs in the record: reasoning or thinking depth, temperature and other sampling parameters, a seed where one exists, the context-window variant, and any speed or quality mode the harness offers. Two runs stamped with the same model id can differ substantially on all of these, so a record that names only the id invites a false comparison between them: an audit verdict, a benchmark, or a regression blamed on a code change may in fact be a settings difference. This matters most where a verdict is treated as evidence. The intent-audit verdict already carries a verifier block of id and version, which is the natural place to widen; the per-run transcript store is the natural home for the settings of an ordinary session; and the Assisted-by trailer is NOT the place, since its accepted shape is a vendor and model pair and widening it would break the gate that reads it. The hard constraint is the same one iss-2608290810032799 records: a model cannot reliably introspect the settings it is running under, so the harness has to supply them. That makes this a capability of the dispatching layer rather than something an agent can be instructed to self-report, and any design that asks the agent to state its own settings inherits exactly the truthfulness problem that the attribution gate already has. diff --git a/.abcd/work/issues/open/iss-2610161709506207-the-opencode-runner-stamps-the-model-it.md b/.abcd/work/issues/resolved/iss-2610161709506207-the-opencode-runner-stamps-the-model-it.md similarity index 75% rename from .abcd/work/issues/open/iss-2610161709506207-the-opencode-runner-stamps-the-model-it.md rename to .abcd/work/issues/resolved/iss-2610161709506207-the-opencode-runner-stamps-the-model-it.md index c867d10043..c5bfb1886b 100644 --- a/.abcd/work/issues/open/iss-2610161709506207-the-opencode-runner-stamps-the-model-it.md +++ b/.abcd/work/issues/resolved/iss-2610161709506207-the-opencode-runner-stamps-the-model-it.md @@ -7,10 +7,15 @@ category: "bug" source: "user-observation" found_during: "manual-capture" origin: researcher-authored -production_mode: hand-written +production_mode: dictated-and-formatted found_at: "internal/core/runner/opencode.go" remedy: "Read the served model from opencode's run-mode JSON events (or the provider's response) and stamp that; when the events carry none, leave the reported model empty and say so in the receipt rather than copying the asked model." related_intents: [itd-2610161733316518, itd-166] +resolution: "The opencode runner no longer copies the asked model into the served side: its run mode names no model, so the served model is left empty and listed as not reported, beside the asked model and the summed step-finish tokens and cost (itd-166)." +impact: fix +resolved_by: + intent: "itd-166" + commit: "7a305e062" --- The opencode runner stamps the model it was asked for as the model that answered: internal/core/runner/opencode.go:85 sets ans.Model = o.model after parsing, so the route receipt's model field can never differ from the request, and a run routed through opencode to OpenRouter or a local server cannot show that a different model, provider or fallback served it. diff --git a/.abcd/work/issues/resolved/iss-2610170934491794-abcd-ahoy-connect-runs-its-verification.md b/.abcd/work/issues/resolved/iss-2610170934491794-abcd-ahoy-connect-runs-its-verification.md new file mode 100644 index 0000000000..ab833ab856 --- /dev/null +++ b/.abcd/work/issues/resolved/iss-2610170934491794-abcd-ahoy-connect-runs-its-verification.md @@ -0,0 +1,19 @@ +--- +schema_version: 1 +id: "iss-2610170934491794" +slug: "abcd-ahoy-connect-runs-its-verification" +severity: "minor" +category: "bug" +source: "user-observation" +found_during: "itd-166 build, 2026-10-17" +origin: researcher-authored +production_mode: dictated-and-formatted +found_at: "internal/core/oracle/config.go" +remedy: "Have ahoy connect retry its one verification call without the usage request when the refusal names it, and write stream_usage: false into the provider block it creates, saying so on the setup screen; the call stays inside the setup the person started." +resolution: "ahoy connect asks its verification call once more without stream_options.include_usage when the first is refused with HTTP 400 or 422, writes stream_usage false into the block it creates and says so in its text and JSON output; --stream-usage sets the value outright" +impact: fix +resolved_by: + intent: "itd-166" +--- + +abcd ahoy connect runs its verification call before any provider block exists, so a server that refuses the usage-in-stream request (stream_options.include_usage, added by itd-166) fails setup, and the refusal names the stream_usage setting on a block the person can only add by hand. diff --git a/commands/ahoy.md b/commands/ahoy.md index e6f8fd4002..c2618c0c36 100644 --- a/commands/ahoy.md +++ b/commands/ahoy.md @@ -1225,7 +1225,7 @@ command for the person.** ### By hand The setup is `abcd ahoy connect --base-url --model -[--model …] --home [--key ]`. **This writes, under +[--model …] --home [--key ] [--stream-usage=false]`. **This writes, under `~/.abcd.noindex/` and, for the keychain home, into the platform keychain.** Ask the technical facilitator which home through your question tool, under a `Tech` chip, after relaying `key_homes`, and offer the three without @@ -1235,11 +1235,18 @@ is); `abcd` and `keychain` take the key piped in on stdin from a file or a variable, or, when the person runs the command in a terminal, pasted on hidden input: abcd prints one line on stderr, "Paste the key for and press Enter. It is not shown.", and reads the key with echo off. An empty paste is -refused as an empty pipe is. It verifies the provider with one call to the first model listed, -and only when that call succeeds keeps the key in that home and writes the +refused as an empty pipe is. It verifies the provider with a call to the first model listed, +asked once more on the usage refusal below, and only when the verification succeeds keeps the key in that home and writes the provider block (the base URL, the key's name and the models, the allowlist) into `~/.abcd.noindex/config.json`. Nothing goes into the repository or the harness's -settings, and a failed verification writes nothing. A `~/.abcd.noindex` that is a +settings, and a failed verification writes nothing. The verification call +asks the server for usage in the stream (`stream_options.include_usage`); +a server that refuses that request with HTTP 400 or 422 is asked once more +without it, and when that succeeds the block is written with +`stream_usage: false` (its calls then go without the field) and the result +says so in `stream_usage_note`: relay that line. `--stream-usage=false` +writes that setting outright, for a server known to refuse the field, and +the verification is then asked once, as given. A `~/.abcd.noindex` that is a symlink (into a dotfiles checkout, say) is refused with nothing written, in any home, naming the link: the key would otherwise land wherever it points. A `--file` pointer is refused, naming the link, when any directory between the @@ -1266,7 +1273,7 @@ and nothing written. **Never ask the person for the key and never pass it yourself**: it would enter this conversation. Give them the command to run in their own terminal, where they paste the key on hidden input or pipe it in from a file or a variable they hold, and relay the result — `verified` (the provider, the model asked for, the model -it reported and the credential's name), each `wrote` path, and `dispatch`. A +it reported and the credential's name), `stream_usage` and `stream_usage_note` when present, each `wrote` path, and `dispatch`. A route the configuration read skips is named on stderr, in the text and the JSON form alike, and the setup stands: relay that line too. diff --git a/commands/disembark.md b/commands/disembark.md index b42eb64d47..e5ac881ee4 100644 --- a/commands/disembark.md +++ b/commands/disembark.md @@ -265,8 +265,8 @@ that route as `--route` to the ingest, which records it as an override. Without its payload flag a synthesis verb runs its deterministic mode, dispatches no agent, and refuses `--route`. The ingest's `--json` result carries a `route` receipt (`tier_asked`, `connection_tried`, `connection_used`, `fallback_reason`, -`override`, `settings_sent`, `model_reported`, and `provider_call`, null until a provider -adapter answers the step) and its text a `route:` line; +`override`, `settings_sent`, `model_reported`, `provider_call`, null until a provider adapter answers the step, and +`unpinned`, the fields the receipt could not pin, each with its reason) and its text a `route:` line; relay it with the result. When no configured provider can serve the tier, one stderr line says the step goes through the harness instead. A `--route` naming an agent this invocation does not dispatch, a tier outside the set, a connection diff --git a/commands/intent.md b/commands/intent.md index 0ea3032c13..2f1dd9539a 100644 --- a/commands/intent.md +++ b/commands/intent.md @@ -952,8 +952,8 @@ without one, one stderr warning names `intent audit ` as the re-emit that adds it, and the close stands. The bare listing and `--issue-drift` dispatch no agent and refuse `--route`. The ingest's `--json` result carries a `route` receipt (`tier_asked`, `connection_tried`, `connection_used`, `fallback_reason`, `override`, -`settings_sent`, `model_reported`, and `provider_call`, null until a provider -adapter answers the step) and its text a `route:` line; relay it with +`settings_sent`, `model_reported`, `provider_call`, null until a provider adapter answers the step, and +`unpinned`, the fields the receipt could not pin, each with its reason) and its text a `route:` line; relay it with the result. When no configured provider can serve the tier, one stderr line says the step goes through the harness instead. A `--route` naming an agent this invocation does not dispatch, a tier outside the set, a connection this machine diff --git a/commands/launch.md b/commands/launch.md index edd3b8c7b4..58abcdc12b 100644 --- a/commands/launch.md +++ b/commands/launch.md @@ -663,8 +663,8 @@ ready cut's `--json` result carries the request block as a `routing` member the harness lets you choose one, and pass the same `--route` to the ingest step so its receipt records the override. The ingest's `--json` result carries a `route` receipt (`tier_asked`, `connection_tried`, `connection_used`, -`fallback_reason`, `override`, `settings_sent`, `model_reported`, and `provider_call`, null until a provider -adapter answers the step) and its text a +`fallback_reason`, `override`, `settings_sent`, `model_reported`, `provider_call`, null until a provider adapter answers the step, and +`unpinned`, the fields the receipt could not pin, each with its reason) and its text a `route:` line; relay it with the result. When no configured provider can serve the tier, one stderr line says the step goes through the harness instead. A `--route` naming an agent this invocation does not dispatch, a tier outside the diff --git a/commands/reading.md b/commands/reading.md index b7b0ba8b7e..059ce5dcfe 100644 --- a/commands/reading.md +++ b/commands/reading.md @@ -290,8 +290,8 @@ nothing else, so run the reading at the tier you mean to and pass that route to the ingest, which records it as an override. The receipt's `model_reported` is the output's `instrument.model`. The ingest's `--json` result carries a `route` receipt (`tier_asked`, `connection_tried`, `connection_used`, `fallback_reason`, -`override`, `settings_sent`, `model_reported`, and `provider_call`, null until a provider -adapter answers the step) and its text a `route:` line; +`override`, `settings_sent`, `model_reported`, `provider_call`, null until a provider adapter answers the step, and +`unpinned`, the fields the receipt could not pin, each with its reason) and its text a `route:` line; relay it with the result. When no configured provider can serve the tier, one stderr line says the step goes through the harness instead. A `--route` naming an agent this invocation does not dispatch, a tier outside the set, a connection diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index 7431f7e688..01e9609da7 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -60,7 +60,7 @@ Detect abcd's install state and list its gaps, or report one mode a flag names: #### `abcd ahoy connect` -Verify a provider with one call and configure it, or guide setup: Writes ~/.abcd.noindex/ and the key's home, nothing guided; refuses no --model off a terminal. +Verify a provider and configure it, or guide setup: Writes ~/.abcd.noindex/ and the key's home, nothing guided; refuses no --model off a terminal. **Usage:** `abcd ahoy connect [] [flags]` @@ -77,6 +77,7 @@ Verify a provider with one call and configure it, or guide setup: Writes ~/.abcd --key string the credential's name (default: the provider's name) --model stringArray a model the provider may serve, repeated for each (the first allowlist; the verification call asks for the first); omitted at a terminal, the service's models are listed with the key and you pick one --resume string with --guide: the last turn's resume object, or - to read it from stdin + --stream-usage whether calls ask the server for usage in the stream (stream_options.include_usage), written into the block as stream_usage; omitted, the verification asks, and a server that refuses it is asked once more without it and the block is written with false (default true) ``` **Example:** diff --git a/internal/adapter/openaiapi/client.go b/internal/adapter/openaiapi/client.go index 98737351d1..8823bd9823 100644 --- a/internal/adapter/openaiapi/client.go +++ b/internal/adapter/openaiapi/client.go @@ -43,11 +43,14 @@ package openaiapi import ( "bytes" "context" + "crypto/sha256" + "encoding/hex" "encoding/json" "errors" "fmt" "html" "io" + "math" "net" "net/http" "net/url" @@ -139,6 +142,46 @@ type Request struct { Model string Brief Brief Settings map[string]json.RawMessage + // NoStreamUsage leaves stream_options.include_usage out of the request. + // The adapter asks for usage in the stream by default (itd-166 decision + // 4); a provider connection whose server refuses the field turns it off. + NoStreamUsage bool +} + +// StreamUsageField is the request field the adapter adds to ask for usage in +// the stream. It is the adapter's own, never a setting, and a record names it +// as added by the adapter rather than as a setting sent. +const StreamUsageField = "stream_options.include_usage" + +// ErrUsageRefused is what a refusal is when the server refused, with HTTP +// 400 or 422, a request that asked for usage in the stream, whatever its body +// says: the field may be what it refused. Only a setup acts on it, by asking +// once more without the field (oracle.Connect); an ordinary call is never +// sent twice. +var ErrUsageRefused = errors.New("openaiapi: the server refused a request that asked for usage in the stream") + +// ErrStreamUsage is the ErrUsageRefused whose refusal names the field +// (stream_options or include_usage): the field is likely what it refused, so +// the caller names the setting that stops asking. +var ErrStreamUsage = errors.New("openaiapi: the server refused the request's usage in the stream") + +// streamUsageError is a scrubbed refusal of a request that asked for usage +// in the stream: always ErrUsageRefused, and ErrStreamUsage when its body +// names the field. +type streamUsageError struct { + msg string + named bool +} + +func (e *streamUsageError) Error() string { return e.msg } +func (e *streamUsageError) Is(target error) bool { + return target == ErrUsageRefused || (e.named && target == ErrStreamUsage) +} + +// namesStreamUsage reports whether a refusal's body, scrubbed, names the +// field the adapter added. +func namesStreamUsage(body string) bool { + return strings.Contains(body, "stream_options") || strings.Contains(body, "include_usage") } // Result is one validated answer: the document the output contract admitted, @@ -151,15 +194,50 @@ type Result struct { // FinishReason is why the provider stopped ("stop", "length"), bounded // and cleaned like the model; empty when it gave none. FinishReason string + // GenerationID is the answer's own id (OpenRouter's gen-..., OpenAI's + // chatcmpl-...), and Provider the upstream provider an aggregator names + // as the one that served it. Each is the provider's own text, bounded and + // cleaned like the model; empty when the answer names none. + GenerationID string + Provider string // Usage is the token count the provider reported, nil when it sent none. Usage *Usage } -// Usage is the token count a provider reports with its answer. +// Usage is the token count a provider reports with its answer. Every count +// is a pointer: nil is a count the provider did not report, or reported in a +// shape the adapter could not read, never a zero it did. type Usage struct { - PromptTokens int `json:"prompt_tokens"` - CompletionTokens int `json:"completion_tokens"` - TotalTokens int `json:"total_tokens"` + PromptTokens *int `json:"prompt_tokens,omitempty"` + CompletionTokens *int `json:"completion_tokens,omitempty"` + TotalTokens *int `json:"total_tokens,omitempty"` + // CachedTokens is prompt_tokens_details.cached_tokens, the prompt tokens + // read from the provider's cache, and CacheWriteTokens + // prompt_tokens_details.cache_write_tokens, the ones written to it. + CachedTokens *int `json:"cached_tokens,omitempty"` + CacheWriteTokens *int `json:"cache_write_tokens,omitempty"` + // ReasoningTokens is completion_tokens_details.reasoning_tokens. + ReasoningTokens *int `json:"reasoning_tokens,omitempty"` + // Cost is the usage member's cost, in the provider's own unit + // (OpenRouter's credits); nil when it reports none. + Cost *float64 `json:"cost,omitempty"` +} + +// PromptSHA256 is the SHA-256, in hex, of the brief's messages as the call +// renders them (itd-166 decision 5): what was sent is recorded by its hash, +// never its text. +func PromptSHA256(b Brief) string { + raw, _ := json.Marshal(messages(b)) + sum := sha256.Sum256(raw) + return hex.EncodeToString(sum[:]) +} + +// IsLocal reports whether baseURL reaches a server on this machine (localhost +// or a loopback address): a local route, whose weights and runtime the +// configuration does not pin yet. +func IsLocal(baseURL string) bool { + u, err := url.Parse(baseURL) + return err == nil && u.Host != "" && loopback(u.Hostname()) } // Client is one provider connection. It holds the key; it never prints it. @@ -403,7 +481,21 @@ func (c *Client) Complete(ctx context.Context, req Request, contract func([]byte live := &liveBody{r: resp.Body, l: l} if resp.StatusCode != http.StatusOK { - return Result{}, c.failedStatus(l, resp) + body, err := c.failedStatus(l, resp) + if !req.NoStreamUsage && (resp.StatusCode == http.StatusBadRequest || resp.StatusCode == http.StatusUnprocessableEntity) { + // The status refuses the request, and the field the adapter added + // may be what it refused. Only a body that names the field says + // so; any other keeps its own words. No second request is made + // here without the field. + named := namesStreamUsage(body) + msg := err.Error() + if named { + msg += " (the request asked for usage in the stream, " + StreamUsageField + + ", which a server that does not take the field refuses)" + } + return Result{}, &streamUsageError{msg: msg, named: named} + } + return Result{}, err } if eventStream(resp.Header.Get("Content-Type")) { return c.readStream(l, live, req.Model, contract) @@ -425,18 +517,20 @@ func (c *Client) Complete(ctx context.Context, req Request, contract func([]byte // status is reported. A body cut off, by the wait, the size bound or a read // error, is not quoted at all: the cut can end inside a key the provider // echoed, and the scrub matches whole key forms only (iss-2610031210435016). -func (c *Client) failedStatus(l *limits, resp *http.Response) error { +// The whole body, scrubbed, comes back beside the error, "" when it was cut +// off, so the caller can read what the refusal names without quoting more. +func (c *Client) failedStatus(l *limits, resp *http.Response) (string, error) { wait := time.AfterFunc(c.errorWait, func() { l.cancel(errErrorBody) }) raw, err := io.ReadAll(io.LimitReader(resp.Body, maxErrorBodyBytes+1)) wait.Stop() msg := fmt.Sprintf("%s answered HTTP %d", c.host, resp.StatusCode) if err != nil || len(raw) > maxErrorBodyBytes { - return c.fail(msg + " (its body was cut off, so it is not quoted)") + return "", c.fail(msg + " (its body was cut off, so it is not quoted)") } if said := c.providerSaid(raw); said != "" { msg += ": " + said } - return c.fail(msg) + return c.scrub(string(raw)), c.fail(msg) } // render builds the request body: the adapter's own fields, then the @@ -450,6 +544,9 @@ func (c *Client) render(req Request) ([]byte, error) { "messages": messages(req.Brief), "stream": true, } + if !req.NoStreamUsage { + fields["stream_options"] = map[string]bool{"include_usage": true} + } keys := make([]string, 0, len(req.Settings)) for k := range req.Settings { keys = append(keys, k) @@ -487,7 +584,10 @@ func accepted(k string) bool { return false } -// completion is the part of a chat completion the adapter reads. +// completion is the part of a chat completion the adapter reads. The id, the +// provider and the usage are read leniently (servedBy): a member of another +// type than the protocol's is a detail not reported, never a refusal of the +// answer. type completion struct { Model string `json:"model"` Choices []struct { @@ -496,10 +596,107 @@ type completion struct { } `json:"message"` FinishReason *string `json:"finish_reason"` } `json:"choices"` - Usage *Usage `json:"usage"` + servedBy Error json.RawMessage `json:"error"` } +// servedBy is the members of a chat completion or a chunk that name what +// served the answer: its id, the upstream provider and the usage. +type servedBy struct { + ID json.RawMessage `json:"id"` + Provider json.RawMessage `json:"provider"` + Usage json.RawMessage `json:"usage"` +} + +// readInto copies what s names onto a, a value already read kept: an id and +// a provider from the first chunk that names them, the usage from the last +// that reads at least as many fields as the one already read, so a later, +// thinner usage member never replaces a fuller one. +func (s servedBy) readInto(a *assembled) { + if a.id == "" { + a.id = lenientString(s.ID) + } + if a.provider == "" { + a.provider = lenientString(s.Provider) + } + if u := readUsage(s.Usage); u != nil && (a.usage == nil || u.readable() >= a.usage.readable()) { + a.usage = u + } +} + +// readable is how many of u's fields were read. +func (u *Usage) readable() int { + n := 0 + for _, p := range []*int{u.PromptTokens, u.CompletionTokens, u.TotalTokens, u.CachedTokens, u.CacheWriteTokens, u.ReasoningTokens} { + if p != nil { + n++ + } + } + if u.Cost != nil { + n++ + } + return n +} + +// lenientString is raw as a JSON string, "" when it is absent or not one. +func lenientString(raw json.RawMessage) string { + var s string + if len(raw) == 0 || json.Unmarshal(raw, &s) != nil { + return "" + } + return s +} + +// readUsage reads a usage member field by field: a field of another type +// than the protocol's, a negative count, or a cost that is negative or not +// finite (ReadableCost) is left unread, so a provider's variation never +// refuses an answer. nil when the member is absent, not an object, or has no +// field that reads. +func readUsage(raw json.RawMessage) *Usage { + var m map[string]json.RawMessage + if len(raw) == 0 || json.Unmarshal(raw, &m) != nil || m == nil { + return nil + } + // A null is nil, never zero: encoding/json reads a null into a number as + // 0 with no error, and a provider answers null for a count it has none of. + null := func(raw json.RawMessage) bool { return string(bytes.TrimSpace(raw)) == "null" } + count := func(raw json.RawMessage) *int { + var n int + if len(raw) == 0 || null(raw) || json.Unmarshal(raw, &n) != nil || n < 0 { + return nil + } + return &n + } + detail := func(member, field string) *int { + var d map[string]json.RawMessage + if json.Unmarshal(m[member], &d) != nil { + return nil + } + return count(d[field]) + } + u := &Usage{ + PromptTokens: count(m["prompt_tokens"]), CompletionTokens: count(m["completion_tokens"]), + TotalTokens: count(m["total_tokens"]), + CachedTokens: detail("prompt_tokens_details", "cached_tokens"), + CacheWriteTokens: detail("prompt_tokens_details", "cache_write_tokens"), + ReasoningTokens: detail("completion_tokens_details", "reasoning_tokens"), + } + var cost float64 + if raw := m["cost"]; len(raw) > 0 && !null(raw) && json.Unmarshal(raw, &cost) == nil && ReadableCost(cost) { + u.Cost = &cost + } + if u.readable() == 0 { + return nil + } + return u +} + +// ReadableCost reports whether a cost a route reported is one a record +// keeps: zero or more, and finite. It is the one policy every route's cost +// is held to (the adapter's usage member, and the runners' harness costs +// through oracle.ReadableCost); a cost that fails it is not reported. +func ReadableCost(c float64) bool { return c >= 0 && !math.IsInf(c, 0) && !math.IsNaN(c) } + // decode reads one chat-completion body: a server's answer to a call that // asked for a stream and got one body instead. func (c *Client) decode(raw []byte, asked string, contract func([]byte) error) (Result, error) { @@ -514,7 +711,8 @@ func (c *Client) decode(raw []byte, asked string, contract func([]byte) error) ( if len(cc.Choices) == 0 { return Result{}, c.noChoice() } - a := assembled{model: cc.Model, usage: cc.Usage} + a := assembled{model: cc.Model} + cc.servedBy.readInto(&a) if p := cc.Choices[0].Message.Content; p != nil { a.content.WriteString(*p) } @@ -526,10 +724,12 @@ func (c *Client) decode(raw []byte, asked string, contract func([]byte) error) ( // assembled is an answer as read, from one body or from a stream's chunks. type assembled struct { - model string - content strings.Builder - finish string - usage *Usage + model string + content strings.Builder + finish string + id string + provider string + usage *Usage } // admit holds an answer as read to the adapter's rules and the caller's @@ -541,11 +741,13 @@ func (c *Client) admit(a *assembled, asked string, contract func([]byte) error) if strings.TrimSpace(a.model) == "" { return Result{}, c.fail(c.host + " reported no model with its answer, so what answered cannot be shown; the answer is refused rather than used") } - // The reported model and the finish reason are the provider's own text, - // recorded and quoted in a denylist refusal, so they are scrubbed like any - // other, bounded and cleaned of hidden runes. + // The reported model, the finish reason, the generation id and the + // upstream provider are the provider's own text, recorded and the model + // quoted in a denylist refusal, so they are scrubbed like any other, + // bounded and cleaned of hidden runes. res := Result{ModelAsked: asked, ModelReported: cleanModel(c.scrub(a.model)), - FinishReason: cleanModel(c.scrub(a.finish)), Usage: a.usage} + FinishReason: cleanModel(c.scrub(a.finish)), GenerationID: cleanModel(c.scrub(a.id)), + Provider: cleanModel(c.scrub(a.provider)), Usage: a.usage} // The answer is the payload a verb records, so it is scrubbed like every // other text the provider sends, before the contract judges it. res.Content = []byte(c.scrubAnswer(unfence(a.content.String()))) @@ -964,8 +1166,10 @@ func asciiEscape(s string) string { return b.String() } -// cleanModel bounds a provider-reported model and percent-encodes any hidden -// or control rune, so it is recorded but cannot reorder or escape the record. +// cleanModel bounds a provider-reported model, or any other short string the +// provider names (a finish reason, a generation id, an upstream provider), and +// percent-encodes any hidden or control rune, so it is recorded but cannot +// reorder or escape the record. func cleanModel(m string) string { m = termsafe.EncodeHiddenRunes(m) if len(m) > MaxModelBytes { diff --git a/internal/adapter/openaiapi/served_test.go b/internal/adapter/openaiapi/served_test.go new file mode 100644 index 0000000000..2f45d13b3d --- /dev/null +++ b/internal/adapter/openaiapi/served_test.go @@ -0,0 +1,398 @@ +package openaiapi + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "strings" + "testing" +) + +// The tests in this file are itd-166's adapter half: a call records what +// answered as the response names it (the generation id, the upstream +// provider, the finish reason, the token counts and the cost), asks for usage +// in the stream unless the connection turns that off, and adds no request to +// the ones the call already makes. + +// openRouterUsage is the usage member OpenRouter sends: its cost in credits +// and the token details beside the counts. +const openRouterUsage = `{"prompt_tokens":120,"completion_tokens":30,"total_tokens":150,` + + `"prompt_tokens_details":{"cached_tokens":100,"cache_write_tokens":20},` + + `"completion_tokens_details":{"reasoning_tokens":12},"cost":0.00042}` + +// openRouterChunk is one OpenRouter-shaped stream event: the generation id +// and the upstream provider ride on every chunk. +func openRouterChunk(content, finish, usage string) string { + delta := map[string]any{} + if content != "" { + delta["content"] = content + } + var fr any + if finish != "" { + fr = finish + } + m := map[string]any{ + "id": "gen-1760000000-abcdef", "provider": "Fireworks", "object": "chat.completion.chunk", "model": "typesafe/jev-1.13-20260915", + "choices": []any{map[string]any{"index": 0, "delta": delta, "finish_reason": fr}}, + } + if usage != "" { + m["choices"] = []any{} + m["usage"] = json.RawMessage(usage) + } + b, _ := json.Marshal(m) + return "data: " + string(b) +} + +// assertServed holds a result to the OpenRouter-shaped answer's served side. +func assertServed(t *testing.T, res Result) { + t.Helper() + if res.GenerationID != "gen-1760000000-abcdef" || res.Provider != "Fireworks" || res.FinishReason != "stop" { + t.Fatalf("served: id %q, provider %q, finish %q", res.GenerationID, res.Provider, res.FinishReason) + } + u := res.Usage + if u == nil || !readCounts(u, 120, 30, 150) { + t.Fatalf("usage = %+v", u) + } + if u.CachedTokens == nil || *u.CachedTokens != 100 || u.CacheWriteTokens == nil || *u.CacheWriteTokens != 20 { + t.Fatalf("cache details = %v / %v", u.CachedTokens, u.CacheWriteTokens) + } + if u.ReasoningTokens == nil || *u.ReasoningTokens != 12 { + t.Fatalf("reasoning = %v", u.ReasoningTokens) + } + if u.Cost == nil || *u.Cost != 0.00042 { + t.Fatalf("cost = %v", u.Cost) + } +} + +// readCounts reports whether u read the prompt, completion and total counts +// given, each present. +func readCounts(u *Usage, prompt, completion, total int) bool { + is := func(p *int, n int) bool { return p != nil && *p == n } + return is(u.PromptTokens, prompt) && is(u.CompletionTokens, completion) && is(u.TotalTokens, total) +} + +// TestAnOpenRouterStreamFillsTheServedSide: a streamed answer whose usage +// arrives in a chunk after the finish reason is read to [DONE], and every +// served field the stream names is on the result. +func TestAnOpenRouterStreamFillsTheServedSide(t *testing.T) { + f := newFake(t, streams( + openRouterChunk(`{"verdict":"yes"}`, "", ""), "", + openRouterChunk("", "stop", ""), "", + openRouterChunk("", "", openRouterUsage), "", + "data: [DONE]", "", + )) + res, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err != nil { + t.Fatalf("Complete: %v", err) + } + assertServed(t, res) +} + +// TestAnOpenRouterPlainBodyFillsTheServedSide: the same answer as one +// chat-completion body fills the same fields. +func TestAnOpenRouterPlainBodyFillsTheServedSide(t *testing.T) { + f := newFake(t, func(w http.ResponseWriter, _ *http.Request, _ map[string]json.RawMessage) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"id":"gen-1760000000-abcdef","provider":"Fireworks","model":"typesafe/jev-1.13-20260915",`+ + `"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"{\"verdict\":\"yes\"}"}}],`+ + `"usage":`+openRouterUsage+`}`) + }) + res, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err != nil { + t.Fatalf("Complete: %v", err) + } + assertServed(t, res) +} + +// TestAnAnswerNamingNoServedDetailLeavesThemUnset: an answer with no id, no +// provider and no usage leaves each unset, never zero standing in for a value. +func TestAnAnswerNamingNoServedDetailLeavesThemUnset(t *testing.T) { + f := newFake(t, func(w http.ResponseWriter, _ *http.Request, _ map[string]json.RawMessage) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"model":"m","choices":[{"index":0,"finish_reason":"stop","message":{"content":"{\"verdict\":\"yes\"}"}}],`+ + `"usage":{"prompt_tokens":3,"completion_tokens":2,"total_tokens":5}}`) + }) + res, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err != nil { + t.Fatalf("Complete: %v", err) + } + if res.GenerationID != "" || res.Provider != "" { + t.Fatalf("id %q, provider %q, want both empty", res.GenerationID, res.Provider) + } + if u := res.Usage; u == nil || u.CachedTokens != nil || u.CacheWriteTokens != nil || u.ReasoningTokens != nil || u.Cost != nil { + t.Fatalf("usage = %+v, want counts alone", res.Usage) + } +} + +// TestAMalformedUsageMemberIsNotAFailure: a usage member whose fields are of +// another type (a cost as text, details as a list) is read for what it holds +// and never turns an answer into a refusal. +func TestAMalformedUsageMemberIsNotAFailure(t *testing.T) { + f := newFake(t, streams( + chunkLine("m", `{"verdict":"yes"}`, "stop"), "", + `data: {"id":7,"provider":{"name":"x"},"model":"m","choices":[],"usage":{"prompt_tokens":4,"completion_tokens":"2","cost":"0.1","prompt_tokens_details":[1]}}`, "", + "data: [DONE]", "", + )) + res, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err != nil { + t.Fatalf("Complete refused an answer over a malformed usage member: %v", err) + } + u := res.Usage + if u == nil || u.PromptTokens == nil || *u.PromptTokens != 4 || u.Cost != nil || u.CachedTokens != nil { + t.Fatalf("usage = %+v", u) + } + // The completion count, sent as text, is unread rather than zero: the + // record lists it as not reported (oracle's + // TestAMalformedUsageMemberListsEachUnreadCount holds the list). + if u.CompletionTokens != nil || u.TotalTokens != nil { + t.Fatalf("completion %v, total %v: want both unread", u.CompletionTokens, u.TotalTokens) + } + if res.Provider != "" || res.GenerationID != "gen-1" { + t.Fatalf("provider %q, id %q", res.Provider, res.GenerationID) + } +} + +// TestTheRequestAsksForUsageInTheStreamUnlessTurnedOff is decision 4: the +// request carries stream_options.include_usage, and a connection that turns +// it off sends a request without it. +func TestTheRequestAsksForUsageInTheStreamUnlessTurnedOff(t *testing.T) { + f := newFake(t, ok("m", `{"verdict":"yes"}`)) + if _, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject); err != nil { + t.Fatal(err) + } + if got := string(f.last.Load().body["stream_options"]); got != `{"include_usage":true}` { + t.Fatalf("stream_options = %s, want include_usage", got) + } + req := request() + req.NoStreamUsage = true + if _, err := mustClient(t, f.base(), testKey).Complete(context.Background(), req, jsonObject); err != nil { + t.Fatal(err) + } + if got, present := f.last.Load().body["stream_options"]; present { + t.Fatalf("stream_options = %s, want it absent", got) + } +} + +// TestARefusalOfTheUsageRequestNamesIt: a server that refuses the request +// with HTTP 400 while it asked for usage in the stream is told apart +// (ErrStreamUsage) and the refusal names the field; the same refusal of a +// request that did not ask is not. +func TestARefusalOfTheUsageRequestNamesIt(t *testing.T) { + f := newFake(t, func(w http.ResponseWriter, _ *http.Request, _ map[string]json.RawMessage) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusBadRequest) + _, _ = io.WriteString(w, `{"error":{"message":"Unrecognized request argument supplied: stream_options"}}`) + }) + _, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if !errors.Is(err, ErrStreamUsage) || !strings.Contains(err.Error(), "stream_options.include_usage") { + t.Fatalf("err = %v, want ErrStreamUsage naming stream_options.include_usage", err) + } + req := request() + req.NoStreamUsage = true + _, err = mustClient(t, f.base(), testKey).Complete(context.Background(), req, jsonObject) + if err == nil || errors.Is(err, ErrStreamUsage) { + t.Fatalf("err = %v, want a refusal that is not ErrStreamUsage", err) + } +} + +// TestRecordingTheServedSideAddsNoRequest is criterion 8: on a server the +// size check reaches (vLLM's listing and /tokenize), a call makes the same +// requests whether it asks for usage or not and whether the answer carries +// the served detail or not. +func TestRecordingTheServedSideAddsNoRequest(t *testing.T) { + count := func(noUsage bool, chat func(http.ResponseWriter, *http.Request, map[string]json.RawMessage)) int32 { + s := newSizedService(t, []map[string]any{{"id": "typesafe/jev-1.13", "max_model_len": 100000}}, chat) + s.tokenize = counts(map[string]int{"typesafe/jev-1.13": 50}) + req := request() + req.NoStreamUsage = noUsage + if _, err := mustClient(t, s.base(), "").Complete(context.Background(), req, jsonObject); err != nil { + t.Fatal(err) + } + return s.calls.Load() + } + bare := count(true, ok("typesafe/jev-1.13", `{"verdict":"yes"}`)) + recorded := count(false, streams( + openRouterChunk(`{"verdict":"yes"}`, "stop", ""), "", + openRouterChunk("", "", openRouterUsage), "", + "data: [DONE]", "", + )) + if bare != recorded || bare < 2 { + t.Fatalf("requests: %d without the record, %d with it; want the same (and the size check's among them)", bare, recorded) + } +} + +// TestThePromptHashIsStableAndSensitive is decision 5: the rendered messages +// are recorded by their SHA-256, the same for the same brief and different +// for a brief one byte apart. +func TestThePromptHashIsStableAndSensitive(t *testing.T) { + b := request().Brief + h := PromptSHA256(b) + if len(h) != 64 || strings.Trim(h, "0123456789abcdef") != "" { + t.Fatalf("hash = %q, want 64 hex digits", h) + } + if PromptSHA256(b) != h { + t.Fatal("the same brief hashed twice differs") + } + b.Input += "." + if PromptSHA256(b) == h { + t.Fatal("a brief one byte longer hashes the same") + } + b = request().Brief + b.Instructions, b.Input = b.Instructions+b.Input, "" + if PromptSHA256(b) == h { + t.Fatal("moving text between the messages hashes the same") + } +} + +// TestServedStringsAreScrubbedCleanedAndBounded: the provider, the +// generation id and the finish reason are the provider's own text, so each +// is scrubbed of the key, has its hidden and control runes encoded and is +// bounded, as the reported model is. +func TestServedStringsAreScrubbedCleanedAndBounded(t *testing.T) { + long := strings.Repeat("p", 300) + f := newFake(t, func(w http.ResponseWriter, _ *http.Request, _ map[string]json.RawMessage) { + w.Header().Set("Content-Type", "application/json") + b, _ := json.Marshal(map[string]any{ + "id": "gen-\x1b[31m-‮-" + testKey, "provider": long, "model": "m\x07", + "choices": []any{map[string]any{"index": 0, "finish_reason": "stop​", "message": map[string]any{"content": `{"verdict":"yes"}`}}}, + }) + _, _ = w.Write(b) + }) + res, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err != nil { + t.Fatal(err) + } + for name, v := range map[string]string{"id": res.GenerationID, "provider": res.Provider, "model": res.ModelReported, "finish": res.FinishReason} { + if strings.Contains(v, testKey) { + t.Errorf("%s carries the key: %q", name, v) + } + for _, r := range v { + if r < 0x20 || r == 0x7f || r == '‮' || r == '​' { + t.Errorf("%s carries rune %U: %q", name, r, v) + } + } + if len(v) > MaxModelBytes+len("...") { + t.Errorf("%s is %d bytes, over the bound", name, len(v)) + } + } +} + +// TestIsLocalNamesThisMachine: a base URL on this machine is a local route, +// whose weights and runtime no pin names yet (decision 7). +func TestIsLocalNamesThisMachine(t *testing.T) { + for base, want := range map[string]bool{ + "http://localhost:8000/v1": true, "http://127.0.0.1:11434/v1": true, "http://[::1]:8080/v1": true, + "https://openrouter.ai/api/v1": false, "not a url": false, + } { + if got := IsLocal(base); got != want { + t.Errorf("IsLocal(%q) = %v, want %v", base, got, want) + } + } +} + +// TestARefusalNotNamingTheUsageFieldCarriesNoHint: a 400 whose body does not +// name stream_options or include_usage, to a request that asked for usage, +// reads as it did before the adapter asked: no hint that the field is to +// blame, and not ErrStreamUsage. It is still told apart as a refusal of a +// request that asked (ErrUsageRefused), which only a setup acts on. +func TestARefusalNotNamingTheUsageFieldCarriesNoHint(t *testing.T) { + for _, code := range []int{http.StatusBadRequest, http.StatusUnprocessableEntity} { + f := newFake(t, status(code, `{"error":{"message":"max_tokens is too large"}}`)) + _, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err == nil || errors.Is(err, ErrStreamUsage) || strings.Contains(err.Error(), StreamUsageField) { + t.Fatalf("HTTP %d: err = %v, want a refusal with no usage hint", code, err) + } + if !errors.Is(err, ErrUsageRefused) { + t.Fatalf("HTTP %d: err = %v, want it told apart as ErrUsageRefused", code, err) + } + req := request() + req.NoStreamUsage = true + if _, err := mustClient(t, f.base(), testKey).Complete(context.Background(), req, jsonObject); errors.Is(err, ErrUsageRefused) { + t.Fatalf("HTTP %d: a request that did not ask is ErrUsageRefused: %v", code, err) + } + } + // A body that names include_usage alone carries the hint. + f := newFake(t, status(http.StatusUnprocessableEntity, `{"error":{"message":"extra field: include_usage"}}`)) + _, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if !errors.Is(err, ErrStreamUsage) || !errors.Is(err, ErrUsageRefused) || !strings.Contains(err.Error(), StreamUsageField) { + t.Fatalf("err = %v, want ErrStreamUsage naming the field", err) + } +} + +// TestAMissingCountIsNotReportedAsZero: a usage member that carries no +// readable prompt or completion count leaves that count unset, never a zero +// the provider did not report. +func TestAMissingCountIsNotReportedAsZero(t *testing.T) { + for usage, want := range map[string][2]bool{ + `{"cost":0.1}`: {false, false}, + `{"prompt_tokens":4}`: {true, false}, + `{"completion_tokens":"2"}`: {false, false}, + `{"prompt_tokens":-1,"completion_tokens":3}`: {false, true}, + } { + f := newFake(t, streams( + chunkLine("m", `{"verdict":"yes"}`, "stop"), "", + `data: {"id":"gen-1","model":"m","choices":[],"usage":`+usage+`}`, "", + "data: [DONE]", "", + )) + res, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err != nil { + t.Fatalf("%s: %v", usage, err) + } + if res.Usage == nil { + if want[0] || want[1] { + t.Fatalf("%s: usage = nil", usage) + } + continue + } + if (res.Usage.PromptTokens != nil) != want[0] || (res.Usage.CompletionTokens != nil) != want[1] { + t.Fatalf("%s: prompt %v, completion %v; want present %v", usage, res.Usage.PromptTokens, res.Usage.CompletionTokens, want) + } + } +} + +// TestALaterThinnerUsageKeepsTheFullerOne: a stream whose later usage chunk +// reads fewer fields than an earlier one keeps the earlier, fuller usage. +func TestALaterThinnerUsageKeepsTheFullerOne(t *testing.T) { + f := newFake(t, streams( + openRouterChunk(`{"verdict":"yes"}`, "", ""), "", + openRouterChunk("", "stop", ""), "", + openRouterChunk("", "", openRouterUsage), "", + openRouterChunk("", "", `{"prompt_tokens":1}`), "", + "data: [DONE]", "", + )) + res, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err != nil { + t.Fatalf("Complete: %v", err) + } + assertServed(t, res) +} + +// TestANullCountOrCostIsNotReportedAsZero: encoding/json reads a null into a +// number as 0 with no error, so a usage member answering null for a count or +// the cost (a provider's way of saying it has none) must leave it unset. +func TestANullCountOrCostIsNotReportedAsZero(t *testing.T) { + f := newFake(t, streams( + chunkLine("m", `{"verdict":"yes"}`, "stop"), "", + `data: {"id":"gen-1","model":"m","choices":[],"usage":{"prompt_tokens":4,"completion_tokens":null,`+ + `"completion_tokens_details":{"reasoning_tokens":null},"prompt_tokens_details":{"cached_tokens":null},"cost":null}}`, "", + "data: [DONE]", "", + )) + res, err := mustClient(t, f.base(), testKey).Complete(context.Background(), request(), jsonObject) + if err != nil { + t.Fatalf("Complete: %v", err) + } + u := res.Usage + if u == nil || u.PromptTokens == nil || *u.PromptTokens != 4 { + t.Fatalf("usage = %+v, want the prompt count it carries", u) + } + for name, p := range map[string]*int{"completion": u.CompletionTokens, "reasoning": u.ReasoningTokens, "cached": u.CachedTokens} { + if p != nil { + t.Errorf("a null %s count read as %d", name, *p) + } + } + if u.Cost != nil { + t.Errorf("a null cost read as %v", *u.Cost) + } +} diff --git a/internal/adapter/openaiapi/stream.go b/internal/adapter/openaiapi/stream.go index aaa2fa0a06..d233d11ef2 100644 --- a/internal/adapter/openaiapi/stream.go +++ b/internal/adapter/openaiapi/stream.go @@ -174,7 +174,7 @@ type chunk struct { } `json:"delta"` FinishReason *string `json:"finish_reason"` } `json:"choices"` - Usage *Usage `json:"usage"` + servedBy Error json.RawMessage `json:"error"` } @@ -222,9 +222,10 @@ func (c *Client) readStream(l *limits, body io.Reader, asked string, contract fu // models cannot show which one answered. return Result{}, c.fail(c.host + " reported more than one model in one answer, so what answered cannot be shown; the answer is refused rather than used") } - if ch.Usage != nil { - a.usage = ch.Usage - } + // The usage arrives in a chunk of its own after the finish reason + // when the request asked for it, which is why the stream is read to + // [DONE] rather than to the finish reason. + ch.servedBy.readInto(&a) for _, choice := range ch.Choices { if choice.Index != 0 { continue diff --git a/internal/adapter/openaiapi/stream_test.go b/internal/adapter/openaiapi/stream_test.go index 976f9c2278..46f33f9441 100644 --- a/internal/adapter/openaiapi/stream_test.go +++ b/internal/adapter/openaiapi/stream_test.go @@ -151,7 +151,7 @@ func TestAPlainJSONAnswerToAStreamRequestIsAccepted(t *testing.T) { if string(res.Content) != `{"verdict":"yes"}` || res.FinishReason != "stop" { t.Fatalf("content %q, finish %q", res.Content, res.FinishReason) } - if res.Usage == nil || res.Usage.TotalTokens != 16 || res.Usage.PromptTokens != 11 || res.Usage.CompletionTokens != 5 { + if u := res.Usage; u == nil || !readCounts(u, 11, 5, 16) { t.Fatalf("usage = %+v", res.Usage) } } @@ -173,7 +173,7 @@ func TestUsageInTheLastChunkIsRead(t *testing.T) { if string(res.Content) != `{"verdict":"no"}` { t.Fatalf("content = %q", res.Content) } - if res.Usage == nil || res.Usage.PromptTokens != 40 || res.Usage.CompletionTokens != 7 || res.Usage.TotalTokens != 47 { + if u := res.Usage; u == nil || !readCounts(u, 40, 7, 47) { t.Fatalf("usage = %+v", res.Usage) } } diff --git a/internal/core/implement/loop/drain_stuck_test.go b/internal/core/implement/loop/drain_stuck_test.go index c47aa6a6b4..dd0571354b 100644 --- a/internal/core/implement/loop/drain_stuck_test.go +++ b/internal/core/implement/loop/drain_stuck_test.go @@ -484,8 +484,8 @@ func TestALaneRefusedIsVersion13s(t *testing.T) { } }) } - if SchemaVersion != schemaVersionUnrefused+1 { - t.Fatalf("the current version is the one after the version before `refused`: %d, %d", SchemaVersion, schemaVersionUnrefused) + if schemaVersionUnmetered != schemaVersionUnrefused+1 { + t.Fatalf("`refused` is the version after the version before it: %d, %d", schemaVersionUnmetered, schemaVersionUnrefused) } for _, version := range []int{schemaVersionUnrefused, schemaVersionUnanswered} { as(version, true) @@ -500,8 +500,8 @@ func TestALaneRefusedIsVersion13s(t *testing.T) { if err != nil { t.Fatalf("a version-12 file reads: %v", err) } - if st.SchemaVersion != 13 || st.Lanes[0].Refused != nil || st.Lanes[0].Stage != StageBrief { - t.Fatalf("a version-12 file reads as version 13, its lane as it was: version %d, %+v", st.SchemaVersion, st.Lanes[0]) + if st.SchemaVersion != SchemaVersion || st.Lanes[0].Refused != nil || st.Lanes[0].Stage != StageBrief { + t.Fatalf("a version-12 file reads as the current version, its lane as it was: version %d, %+v", st.SchemaVersion, st.Lanes[0]) } as(13, true) if st, err := ReadState(repo.Root(), runID); err != nil || st.Lanes[0].Refused == nil { diff --git a/internal/core/implement/loop/drive.go b/internal/core/implement/loop/drive.go index 25b3782be8..9490f072fb 100644 --- a/internal/core/implement/loop/drive.go +++ b/internal/core/implement/loop/drive.go @@ -141,8 +141,8 @@ func Drive(ctx context.Context, repoRoot, runID string, steps Stages, o Options, HostSession: true, Transcripts: rs.Transcripts, Validate: func(ran string, _ runner.Request, ans runner.Answer) error { - r, err := receipt(repoRoot, runID, aw.Receipt, steps, o, - &runner.RouteRecord{Asked: route.Runner, Ran: ran, Model: ans.Model}) + rec := runner.RecordOf(route.Runner, ran, ans) + r, err := receipt(repoRoot, runID, aw.Receipt, steps, o, &rec) if err != nil { return err } diff --git a/internal/core/implement/loop/drive_test.go b/internal/core/implement/loop/drive_test.go index f4bd88d084..408394fdf6 100644 --- a/internal/core/implement/loop/drive_test.go +++ b/internal/core/implement/loop/drive_test.go @@ -145,7 +145,7 @@ func loopFakeHarness(mode string) int { body, _ := json.Marshal(a) _ = os.WriteFile(field("Receipt: "), body, 0o600) } - if mode == "ok" || strings.HasPrefix(mode, "model-") { + if mode == "ok" || mode == "metered" || strings.HasPrefix(mode, "model-") { body := "{}\n" if role := field("You are the "); strings.HasPrefix(role, RoleRuthless) { body = reviewReturn @@ -166,6 +166,15 @@ func loopFakeHarness(mode string) int { model = "fake\x1b[2Jmodel\r\n" } init, _ := json.Marshal(map[string]string{"type": "system", "subtype": "init", "session_id": "fake-session-2", "model": model}) + if mode == "metered" { + // The claude CLI 2.1's init and result events as they meter a run + // (internal/core/runner's real-stream fixture, reduced). + fmt.Println(`{"type":"system","subtype":"init","session_id":"fake-session-2","model":"claude-haiku-5-5","claude_code_version":"2.1.302"}`) + fmt.Println(`{"type":"result","subtype":"success","is_error":false,"result":"done","session_id":"fake-session-2","stop_reason":"end_turn",` + + `"total_cost_usd":0.0038,"usage":{"input_tokens":2,"output_tokens":76,"cache_read_input_tokens":11228,"cache_creation_input_tokens":18650},` + + `"modelUsage":{"claude-haiku-5-5":{"provider":"firstParty"}}}`) + return 0 + } fmt.Println(string(init)) fmt.Println(`{"type":"result","subtype":"success","is_error":false,"result":"done","session_id":"fake-session-2"}`) return 0 diff --git a/internal/core/implement/loop/sides_test.go b/internal/core/implement/loop/sides_test.go new file mode 100644 index 0000000000..63da1a4241 --- /dev/null +++ b/internal/core/implement/loop/sides_test.go @@ -0,0 +1,137 @@ +package loop + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/intentdriven/abcd/internal/core/oracle" + "github.com/intentdriven/abcd/internal/core/runner" +) + +// The tests in this file are itd-166's loop half: both sides of a run travel +// with the route record the run state already stores on a verified receipt +// and a validator's return, at schema version 14, and the drive passes them +// through from the runner's answer. + +// sidedRoute is a runner's route record carrying both sides. +func sidedRoute() runner.RouteRecord { + return runner.RouteRecord{Asked: runner.OpenCode, Ran: runner.OpenCode, Model: "", + Sides: runner.Sides{ModelAsked: "local/qwen3-coder", FinishReason: "stop", PromptSHA256: strings.Repeat("a", 64), + Usage: &oracle.Usage{PromptTokens: 1600, CompletionTokens: 300}, + Cost: &oracle.Cost{Amount: 0.002, Unit: oracle.CostUSD}, + Unpinned: []oracle.Unpinned{{Field: oracle.ServedModel, Reason: oracle.NotReported}}}} +} + +// sidedRun is a run whose implement stage a runner ran: its verified receipt +// carries a route with both sides. It returns the root, the run and the +// state file's path. +func sidedRun(t *testing.T) (string, string, string) { + t.Helper() + root, id, steps, o := startToImplement(t) + newDriveEnv(t, "ok", "claude") + cfg := runnerConfig(t, `{"roles":{"implementer":{"runner":"claude"}},"runner":{"claude":{}}}`, "") + if _, err := Drive(context.Background(), root, id, steps, o, Runners{Config: cfg, Transcripts: &memTranscripts{}}); err != nil { + t.Fatal(err) + } + r := sidedRoute() + editState(t, root, id, func(doc map[string]any) { + var route map[string]any + raw, _ := json.Marshal(r) + _ = json.Unmarshal(raw, &route) + doc["lanes"].([]any)[0].(map[string]any)["receipts"].([]any)[0].(map[string]any)["route"] = route + }) + return root, id, filepath.Join(root, filepath.FromSlash(StateRelPath(id))) +} + +// TestASidedRouteRoundTripsAtVersion14 is criterion 7's run-state half: a +// route record carrying both sides is read back whole at the current +// version, 14, which is the version that added them. +func TestASidedRouteRoundTripsAtVersion14(t *testing.T) { + if SchemaVersion != 14 || schemaVersionUnmetered != 13 || schemaVersionUnrefused != 12 { + t.Fatalf("both sides are version 14's, after a lane's refusal's 13: %d %d %d", SchemaVersion, schemaVersionUnmetered, schemaVersionUnrefused) + } + root, id, _ := sidedRun(t) + st, err := ReadState(root, id) + if err != nil { + t.Fatal(err) + } + got := st.Lanes[0].Receipts[0].Route + want := sidedRoute() + if got == nil || got.ModelAsked != want.ModelAsked || got.Usage == nil || *got.Usage != *want.Usage || + got.Cost == nil || *got.Cost != *want.Cost || len(got.Unpinned) != 1 || got.Unpinned[0] != want.Unpinned[0] { + t.Fatalf("route read back = %+v, want %+v", got, want) + } +} + +// TestAVersion13FileReadsAsNotRecorded is the version-13 rule: version 13 is +// version 14's strict subset, so a version-13 file without either side reads +// as the current version, its routes not recorded, and one of version 13 or +// lower carrying a side is not a file its version wrote and is refused, +// naming the version. +func TestAVersion13FileReadsAsNotRecorded(t *testing.T) { + root, id, path := sidedRun(t) + sided, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + for _, version := range []int{schemaVersionUnmetered, schemaVersionUnrefused, schemaVersionUnanswered, schemaVersionUnrouted + 1} { + if err := os.WriteFile(path, sided, 0o600); err != nil { + t.Fatal(err) + } + editState(t, root, id, func(doc map[string]any) { doc["schema_version"] = version }) + _, err := ReadState(root, id) + r := mustRefusal(t, err) + if !strings.Contains(r.Reason, fmt.Sprintf("schema version %d", version)) || !strings.Contains(r.Reason, "both sides") { + t.Errorf("version %d carrying a route's sides is refused naming the version: %+v", version, r) + } + } + if err := os.WriteFile(path, sided, 0o600); err != nil { + t.Fatal(err) + } + editState(t, root, id, func(doc map[string]any) { + doc["schema_version"] = schemaVersionUnmetered + doc["lanes"].([]any)[0].(map[string]any)["receipts"].([]any)[0].(map[string]any)["route"] = + map[string]any{"asked": "claude", "ran": "claude", "model": "fake-model"} + }) + st, err := ReadState(root, id) + if err != nil || st.SchemaVersion != SchemaVersion { + t.Fatalf("a version-13 file without sides reads as the current version: %v %v", st.SchemaVersion, err) + } + if r := st.Lanes[0].Receipts[0].Route; r == nil || r.Usage != nil || r.Unpinned != nil || r.ModelAsked != "" { + t.Fatalf("a version-13 route reads as not recorded: %+v", r) + } +} + +// TestTheDrivePassesBothSidesThrough: a role the loop runs through a runner +// has the runner's asked model, token counts and cost on the route its +// verified receipt and its result name, the same route on both. +func TestTheDrivePassesBothSidesThrough(t *testing.T) { + root, id, steps, o := startToImplement(t) + newDriveEnv(t, "metered", "claude") + cfg := runnerConfig(t, `{"roles":{"implementer":{"runner":"claude"}},"runner":{"claude":{}}}`, "") + res, err := Drive(context.Background(), root, id, steps, o, Runners{Config: cfg, Transcripts: &memTranscripts{}}) + if err != nil { + t.Fatal(err) + } + r := res.Route + if r == nil || r.HarnessVersion != "2.1.302" || r.Usage == nil || r.Usage.CompletionTokens != 76 || + r.Cost == nil || r.Cost.Unit != oracle.CostUSD || len(r.Unpinned) == 0 { + t.Fatalf("the result's route carries both sides: %+v", r) + } + st, err := ReadState(root, id) + if err != nil { + t.Fatal(err) + } + stored := st.Lanes[0].Receipts[0].Route + if stored == nil || stored.Usage == nil || *stored.Usage != *r.Usage || stored.HarnessVersion != r.HarnessVersion { + t.Fatalf("the verified receipt's route carries the same sides: %+v", stored) + } + if st.SchemaVersion != SchemaVersion { + t.Fatalf("written at version %d", st.SchemaVersion) + } +} diff --git a/internal/core/implement/loop/state.go b/internal/core/implement/loop/state.go index 6028877eb0..947bc3f808 100644 --- a/internal/core/implement/loop/state.go +++ b/internal/core/implement/loop/state.go @@ -48,6 +48,7 @@ import ( "fmt" "io/fs" "os" + "reflect" "regexp" "slices" "strings" @@ -175,10 +176,26 @@ const lockFileName = ".lock" // Version 13 added a lane's `refused` (spc-2610161044461448 step 5): an issue // run's lane logs its run of refused steps, which a drain reads to hand a // stuck lane back. Version 12 is its strict subset, read as a run whose lanes -// have refused nothing and written back at version 13; a file of version 12 or -// lower carrying `refused` is not one its version wrote, and is refused. -// Releases that write version 12 refuse a version-13 file as a newer abcd's. -const SchemaVersion = 13 +// have refused nothing and written back at the current version; a file of +// version 12 or lower carrying `refused` is not one its version wrote, and is +// refused. Releases that write version 12 refuse a version-13 file as a newer +// abcd's. +// +// Version 14 added both sides of a run (itd-166): a runner's route record on +// a verified receipt or a validator's return carries the model the runner +// asked for, the prompt's hash, the served models, provider, finish reason +// and harness version the harness reported, the token counts and cost, and +// the fields it could not pin (runner.Sides). Version 13 is its strict +// subset, read as a run whose routes recorded neither side, its lanes' +// `refused` kept, and written back at version 14; a file of version 13 or +// lower carrying either side is not one its version wrote, and is refused. +// Releases that write version 13 (v0.15.1) refuse a version-14 file as a +// newer abcd's before any strict decode. +const SchemaVersion = 14 + +// schemaVersionUnmetered is the version before a route's both sides: read, +// never written. +const schemaVersionUnmetered = 13 // schemaVersionUnrefused is the version before a lane's `refused`: read, // never written. @@ -717,7 +734,7 @@ func (s State) kept() string { } // answered reports the first await, in a state, that carries `answered`, -// which only version 12 writes, or "". +// which only version 12 and later write, or "". func (s State) answered() string { for _, l := range s.Lanes { for _, a := range l.Awaits { @@ -729,8 +746,31 @@ func (s State) answered() string { return "" } +// metered reports the first route, in a state, that carries either side of +// its run, which only version 14 writes, or "". +func (s State) metered() string { + sided := func(r *runner.RouteRecord) bool { + return r != nil && !reflect.DeepEqual(r.Sides, runner.Sides{}) + } + for _, l := range s.Lanes { + for i, r := range l.Receipts { + if sided(r.Route) { + return fmt.Sprintf("%s's receipt %d's route", l.ID, i+1) + } + } + for _, v := range l.Validation { + for i, vr := range v.Validators { + if sided(vr.Route) { + return fmt.Sprintf("%s's round %d validator %d's route", l.ID, v.Round, i+1) + } + } + } + } + return "" +} + // refused reports the first lane, in a state, that carries `refused`, which -// only version 13 writes, or "". +// only version 13 and later write, or "". func (s State) refused() string { for _, l := range s.Lanes { if l.Refused != nil { @@ -885,7 +925,10 @@ func readStateIn(root *os.Root, runID string) (State, error) { case st.SchemaVersion <= schemaVersionUnrefused && st.refused() != "": return State{}, refuse("state", "", "", fmt.Sprintf("%s is schema version %d but carries %s, which version %d never wrote", rel, st.SchemaVersion, st.refused(), st.SchemaVersion), "the loop is the file's only writer; restore it or remove the run directory "+runRel(runID)) - case st.SchemaVersion >= schemaVersionUnpaced && st.SchemaVersion <= schemaVersionUnrefused: + case st.SchemaVersion <= schemaVersionUnmetered && st.metered() != "": + return State{}, refuse("state", "", "", fmt.Sprintf("%s is schema version %d but carries both sides of %s, which version %d never wrote", rel, st.SchemaVersion, st.metered(), st.SchemaVersion), + "the loop is the file's only writer; restore it or remove the run directory "+runRel(runID)) + case st.SchemaVersion >= schemaVersionUnpaced && st.SchemaVersion <= schemaVersionUnmetered: // Read as the current version, its stages already carried over by // decodeState when it named them `step`; the next write carries it, and // this read writes nothing. diff --git a/internal/core/implement/loop/state_mask.go b/internal/core/implement/loop/state_mask.go index 3de19cbaab..6e5f7eb3ed 100644 --- a/internal/core/implement/loop/state_mask.go +++ b/internal/core/implement/loop/state_mask.go @@ -4,6 +4,7 @@ import ( "strings" "github.com/intentdriven/abcd/internal/core/intent" + "github.com/intentdriven/abcd/internal/core/oracle" "github.com/intentdriven/abcd/internal/core/runner" "github.com/intentdriven/abcd/internal/fsutil" "github.com/intentdriven/abcd/internal/termsafe" @@ -175,9 +176,22 @@ func (t Transcript) masked() Transcript { return t } -// maskRoute masks the route a runner reported, its model above all. +// maskRoute masks the route a runner reported, its model above all, and both +// sides of its run (itd-166), each string the harness supplied included. func maskRoute(r runner.RouteRecord) runner.RouteRecord { r.Asked, r.Ran, r.Model = line(r.Asked), line(r.Ran), line(r.Model) + s := &r.Sides + s.ModelAsked, s.ServedProvider, s.FinishReason = line(s.ModelAsked), line(s.ServedProvider), line(s.FinishReason) + s.HarnessVersion, s.PromptSHA256 = line(s.HarnessVersion), line(s.PromptSHA256) + s.ServedModels = maskEach(s.ServedModels, line) + s.Cost = maskPtr(s.Cost, func(c oracle.Cost) oracle.Cost { + c.Unit = line(c.Unit) + return c + }) + s.Unpinned = maskEach(s.Unpinned, func(u oracle.Unpinned) oracle.Unpinned { + u.Field, u.Reason = oracle.Field(line(string(u.Field))), oracle.UnpinnedReason(line(string(u.Reason))) + return u + }) return r } diff --git a/internal/core/implement/loop/validate_review_test.go b/internal/core/implement/loop/validate_review_test.go index 2a83ec41b4..3d38275e23 100644 --- a/internal/core/implement/loop/validate_review_test.go +++ b/internal/core/implement/loop/validate_review_test.go @@ -98,7 +98,7 @@ func TestTheReviewsGoToRepoPromptOverMCPAndTheLoopRecordsTheirVerdicts(t *testin if err != nil { t.Fatalf("the %s over the route: %v", role, err) } - if got.Route == nil || *got.Route != (runner.RouteRecord{Asked: "rp", Ran: "rp"}) || got.Awaiting != nil || got.Stage != StageValidate { + if got.Route == nil || !reflect.DeepEqual(*got.Route, runner.RouteRecord{Asked: "rp", Ran: "rp"}) || got.Awaiting != nil || got.Stage != StageValidate { t.Fatalf("the route ran the %s and the lane stays at validate: %+v", role, got) } sent := sentTo(srv) @@ -684,7 +684,7 @@ func TestAHandBackAfterABusyLockRecordsTheRoute(t *testing.T) { return } want := runner.RouteRecord{Asked: "rp", Ran: "rp"} - if v.Route == nil || *v.Route != want || got.Route == nil || *got.Route != want || v.Verdict != "SHIP" { + if v.Route == nil || !reflect.DeepEqual(*v.Route, want) || got.Route == nil || !reflect.DeepEqual(*got.Route, want) || v.Verdict != "SHIP" { t.Fatalf("the hand-back records the route that ran the review: %+v %+v", v, got.Route) } var record strings.Builder diff --git a/internal/core/oracle/call.go b/internal/core/oracle/call.go index c61f43ad90..565d81a7c9 100644 --- a/internal/core/oracle/call.go +++ b/internal/core/oracle/call.go @@ -19,6 +19,7 @@ import ( "github.com/intentdriven/abcd/internal/adapter/openaiapi" "github.com/intentdriven/abcd/internal/core/credential" + "github.com/intentdriven/abcd/internal/core/layered" ) // CallRecord is the per-call record the run record carries (criterion 5, @@ -26,6 +27,14 @@ import ( // model the provider reported, side by side, so a substitution is visible, and // the name of the credential the call used. It never carries a key or the // brief. +// +// It carries both sides of the call (itd-166): the asked side beside the +// settings the receipt records as sent (the prompt's SHA-256 and the request +// fields the adapter added on its own), and the served side as the response +// named it (the upstream provider, the generation id, the finish reason, the +// token counts and the cost). Every string the provider supplied is scrubbed, +// cleaned and bounded by the adapter (openaiapi's admit). Unpinned lists each +// field of the fixed lists the call could not pin, with its reason. type CallRecord struct { Provider string `json:"provider"` ModelAsked string `json:"model_asked"` @@ -34,6 +43,26 @@ type CallRecord struct { // value; empty for a provider that takes no key (itd-2609221017023290 // criterion 5). Credential string `json:"credential,omitempty"` + // PromptSHA256 is the SHA-256 of the rendered messages, never their text + // (decision 5). + PromptSHA256 string `json:"prompt_sha256"` + // AdapterAdded names the request fields the adapter sent on its own + // (openaiapi.StreamUsageField), which are not settings sent. + AdapterAdded []string `json:"adapter_added"` + // ServedProvider is the upstream provider an aggregator names as the one + // that served the call; "" when the response names none. + ServedProvider string `json:"served_provider"` + // GenerationID is the response's own id; "" when it names none. + GenerationID string `json:"generation_id"` + // FinishReason is why the provider stopped; "" when it gave none. + FinishReason string `json:"finish_reason"` + // Usage is the token counts the response reported, nil when it reported + // none; a count it did not report is listed in Unpinned. + Usage *Usage `json:"usage"` + // Cost is the usage member's cost, in credits; nil when not reported. + Cost *Cost `json:"cost"` + // Unpinned lists every field of the fixed lists the call could not pin. + Unpinned []Unpinned `json:"unpinned"` } // CallRequest is one call: where it goes, the brief, the settings as sent and @@ -65,11 +94,27 @@ func (c *APIConfig) Call(ctx context.Context, creds credential.Source, req CallR if err != nil { return nil, CallRecord{}, err } - payload, rec, err := complete(ctx, p.Name, p.BaseURL, key, t.Model, req.Brief, req.Settings, req.Contract, c.denylist, opts...) - if err == nil { - rec.Credential = p.Key + payload, rec, err := complete(ctx, p, key, t.Model, req.Brief, req.Settings, req.Contract, c.denylist, opts...) + if err != nil { + // Decision 4: an ordinary call is never sent again without the + // field; the refusal names the setting that stops asking. + return nil, CallRecord{}, withStreamUsageSetting(err, p.Name) } - return payload, rec, err + rec.Credential = p.Key + return payload, rec, nil +} + +// withStreamUsageSetting adds to err, when the server refused a request's +// usage in the stream by name (openaiapi.ErrStreamUsage), the setting in the +// configured block of provider that stops asking for it. Any other error is +// returned as it is. +func withStreamUsageSetting(err error, provider string) error { + if !errors.Is(err, openaiapi.ErrStreamUsage) { + return err + } + return fmt.Errorf("%w; if this server does not take %s, set %s.%s.%s to false in %s, "+ + "and its calls are sent without it, their usage recorded as not reported when the answer carries none", + err, openaiapi.StreamUsageField, apiKey, provider, streamUsageKey, layered.Config.MachineOrigin()) } // resolveKey resolves a provider's key by name; a keyless block resolves to @@ -94,13 +139,19 @@ func resolveKey(creds credential.Source, p Provider) (string, error) { // complete is the one place a call is made: the client built on the pinned // base URL, the answer read, and the reported model held to oracle.denylist. -func complete(ctx context.Context, provider, baseURL, key, model string, brief openaiapi.Brief, settings Settings, +// The request asks for usage in the stream unless p's block turns that off, +// and a refusal of it is returned as the adapter gave it: Call names the +// setting that stops asking, and the setup's verification (Connect) asks +// once more without it. complete itself never sends a second request. +func complete(ctx context.Context, p Provider, key, model string, brief openaiapi.Brief, settings Settings, contract func([]byte) error, denylist []DenyEntry, opts ...openaiapi.Option) ([]byte, CallRecord, error) { - client, err := openaiapi.New(baseURL, key, opts...) + provider := p.Name + client, err := openaiapi.New(p.BaseURL, key, opts...) if err != nil { return nil, CallRecord{}, fmt.Errorf("oracle adapter: provider %s: %w", provider, err) } - res, err := client.Complete(ctx, openaiapi.Request{Model: model, Brief: brief, Settings: settings}, contract) + req := openaiapi.Request{Model: model, Brief: brief, Settings: settings, NoStreamUsage: !p.AsksStreamUsage()} + res, err := client.Complete(ctx, req, contract) if err != nil { return nil, CallRecord{}, fmt.Errorf("oracle adapter: provider %s, model %s: %w", provider, model, err) } @@ -108,5 +159,71 @@ func complete(ctx context.Context, provider, baseURL, key, model string, brief o return nil, CallRecord{}, fmt.Errorf("oracle adapter: provider %s was asked for %s and reported answering with %s, "+ "which %s refuses (%s, from %s); the answer is discarded", provider, model, res.ModelReported, denylistKey, e.Pattern, e.Origin) } - return res.Content, CallRecord{Provider: provider, ModelAsked: res.ModelAsked, ModelReported: res.ModelReported}, nil + return res.Content, callRecord(p, brief, settings, res), nil +} + +// callRecord is the call's both sides, built from the request this layer sent +// and the response the adapter read: nothing in it was asked of the model, +// and building it sends nothing (criterion 8). +func callRecord(p Provider, brief openaiapi.Brief, settings Settings, res openaiapi.Result) CallRecord { + rec := CallRecord{Provider: p.Name, ModelAsked: res.ModelAsked, ModelReported: res.ModelReported, + PromptSHA256: openaiapi.PromptSHA256(brief), AdapterAdded: []string{}, + ServedProvider: res.Provider, GenerationID: res.GenerationID, FinishReason: res.FinishReason} + if p.AsksStreamUsage() { + rec.AdapterAdded = append(rec.AdapterAdded, openaiapi.StreamUsageField) + } + var un Unpin + // The asked side: the model and the prompt are always the call's own. No + // provider pin exists yet (iss-2610161734413178), no setting names an + // effort or a context variant, and a sampling setting not sent leaves the + // server's default, whatever it is. + un.Add(AskedProviderPin, NotConfigured) + for _, s := range []struct { + f Field + name string + }{{AskedTemperature, "temperature"}, {AskedTopP, "top_p"}, {AskedSeed, "seed"}} { + _, sent := settings[s.name] + un.Unless(sent, s.f, NotConfigured) + } + un.Add(AskedEffort, NotConfigured) + un.Add(AskedContext, NotConfigured) + un.Add(AskedHarnessVersion, NotReported) + if openaiapi.IsLocal(p.BaseURL) { + for _, f := range LocalFields { + un.Add(f, NotConfigured) + } + } + // The served side: the adapter refuses an answer that names no model, so + // the model is always the provider's own report. + un.Unless(res.Provider != "", ServedProvider, NotReported) + un.Unless(res.GenerationID != "", ServedGenerationID, NotReported) + un.Unless(res.FinishReason != "", ServedFinishReason, NotReported) + u := res.Usage + un.Unless(u != nil && u.PromptTokens != nil, ServedPromptTokens, NotReported) + un.Unless(u != nil && u.CompletionTokens != nil, ServedCompletionTokens, NotReported) + un.Unless(u != nil && u.ReasoningTokens != nil, ServedReasoningTokens, NotReported) + un.Unless(u != nil && u.CachedTokens != nil, ServedCacheReadTokens, NotReported) + un.Unless(u != nil && u.CacheWriteTokens != nil, ServedCacheWriteTokens, NotReported) + un.Unless(u != nil && u.Cost != nil, ServedCost, NotReported) + if u == nil { + rec.Unpinned = un.List() + return rec + } + deref := func(p *int) int { + if p == nil { + return 0 + } + return *p + } + // The counts are recorded when any of them read; a usage member that + // carried only a cost leaves Usage nil, as one that carried nothing does. + if u.PromptTokens != nil || u.CompletionTokens != nil || u.ReasoningTokens != nil || u.CachedTokens != nil || u.CacheWriteTokens != nil { + rec.Usage = &Usage{PromptTokens: deref(u.PromptTokens), CompletionTokens: deref(u.CompletionTokens), + ReasoningTokens: deref(u.ReasoningTokens), CacheReadTokens: deref(u.CachedTokens), CacheWriteTokens: deref(u.CacheWriteTokens)} + } + if u.Cost != nil { + rec.Cost = &Cost{Amount: *u.Cost, Unit: CostCredits} + } + rec.Unpinned = un.List() + return rec } diff --git a/internal/core/oracle/call_test.go b/internal/core/oracle/call_test.go index 7a08094791..3f2e9913ea 100644 --- a/internal/core/oracle/call_test.go +++ b/internal/core/oracle/call_test.go @@ -7,6 +7,7 @@ import ( "io" "net/http" "net/http/httptest" + "reflect" "strings" "sync/atomic" "testing" @@ -80,6 +81,13 @@ func configured(t *testing.T, base, key string) (*fx, *APIConfig) { return f, f.loadAPI() } +// who is a call record's identity alone: the provider, the models and the +// credential, which a test compares whole; the two sides itd-166 added are +// held by their own tests (sides_test.go). +func who(c CallRecord) CallRecord { + return CallRecord{Provider: c.Provider, ModelAsked: c.ModelAsked, ModelReported: c.ModelReported, Credential: c.Credential} +} + func verdictContract(b []byte) error { var v struct { Verdict string `json:"verdict"` @@ -110,7 +118,7 @@ func TestCallSendsTheBriefToThePointedModelWithTheKeyByName(t *testing.T) { t.Fatalf("payload = %q", payload) } want := CallRecord{Provider: "openrouter", ModelAsked: "typesafe/jev-1.13", ModelReported: "typesafe/jev-1.13-20260915", Credential: "openrouter"} - if rec != want { + if !reflect.DeepEqual(who(rec), want) { t.Fatalf("record = %+v, want %+v", rec, want) } if p.auth.Load() != "Bearer "+callKey { @@ -250,7 +258,7 @@ func TestTheReceiptCarriesTheProviderCall(t *testing.T) { enc, _ = json.Marshal(r.Receipt("").WithCall(rec)) // The run's record names the credential the call used, and never a value // (itd-2609221017023290 criterion 5). - for _, want := range []string{`"provider_call":{"provider":"openrouter","model_asked":"typesafe/jev-1.13","model_reported":"typesafe/jev-1.13-20260915","credential":"openrouter"}`} { + for _, want := range []string{`"provider_call":{"provider":"openrouter","model_asked":"typesafe/jev-1.13","model_reported":"typesafe/jev-1.13-20260915","credential":"openrouter",`} { if !strings.Contains(string(enc), want) { t.Fatalf("receipt = %s, want %s", enc, want) } diff --git a/internal/core/oracle/config.go b/internal/core/oracle/config.go index af88633463..c8889ee325 100644 --- a/internal/core/oracle/config.go +++ b/internal/core/oracle/config.go @@ -64,6 +64,8 @@ const ( denylistKey = "oracle.denylist" rolesKey = "oracle.roles" judgementsKey = "oracle.judgements" + // streamUsageKey is a provider block's opt-out of usage in the stream. + streamUsageKey = "stream_usage" ) // Bounds on what the configuration may carry. @@ -98,8 +100,16 @@ type Provider struct { Key string `json:"key,omitempty"` Models []string `json:"models"` Origin string `json:"origin"` + // StreamUsage is the block's stream_usage: false stops a call asking for + // usage in the stream (openaiapi.StreamUsageField), for a server that + // refuses the field. nil, the default, asks (itd-166 decision 4). + StreamUsage *bool `json:"stream_usage,omitempty"` } +// AsksStreamUsage reports whether a call through p asks for usage in the +// stream: unless its block sets stream_usage to false. +func (p Provider) AsksStreamUsage() bool { return p.StreamUsage == nil || *p.StreamUsage } + // Target is where a role or a judgement type is pointed: a configured // provider and a model on its list. type Target struct { @@ -138,9 +148,10 @@ type APIConfig struct { // providerFile is a provider block as the file spells it, decoded strictly. type providerFile struct { - BaseURL string `json:"base_url"` - Key *string `json:"key"` - Models []string `json:"models"` + BaseURL string `json:"base_url"` + Key *string `json:"key"` + Models []string `json:"models"` + StreamUsage *bool `json:"stream_usage"` } // LoadAPI reads the provider configuration through the layered resolver and @@ -234,7 +245,7 @@ func (c *APIConfig) readProviders(s *layered.Stack) error { origin, apiKey, name, Harness) } } - if err := s.Claim(apiKey+".*", "base_url", "key", "models"); err != nil { + if err := s.Claim(apiKey+".*", "base_url", "key", "models", streamUsageKey); err != nil { return fmt.Errorf("oracle adapter: %w", err) } for _, name := range names { @@ -265,7 +276,7 @@ func (c *APIConfig) provider(name string, fd layered.Found) (Provider, error) { if err := openaiapi.ValidateBaseURL(pf.BaseURL); err != nil { return Provider{}, err } - p := Provider{Name: name, BaseURL: pf.BaseURL, Origin: fd.Origin} + p := Provider{Name: name, BaseURL: pf.BaseURL, Origin: fd.Origin, StreamUsage: pf.StreamUsage} if pf.Key != nil { if !credential.ValidName(*pf.Key) { return Provider{}, fmt.Errorf("key %q is not a plain credential name (lower case letters, digits, '.', '_' and '-'); "+ diff --git a/internal/core/oracle/connect.go b/internal/core/oracle/connect.go index 4e7faa15a2..7c7307f489 100644 --- a/internal/core/oracle/connect.go +++ b/internal/core/oracle/connect.go @@ -66,6 +66,12 @@ type ConnectRequest struct { Pointer credential.Pointer // Timeout bounds the verification call; 0 keeps its own short bound. Timeout time.Duration + // StreamUsage is the block's stream_usage, written as given: false sends + // the verification call, and every call after it, without asking for + // usage in the stream (openaiapi.StreamUsageField). nil asks, and a + // refusal of that (HTTP 400 or 422) has the verification asked once more + // without it, writing false when that succeeds. + StreamUsage *bool // Pick, when Models is empty, chooses one model from the listed ids. // Connect calls it after listing with the key it holds; nil refuses as today. Pick func(ctx context.Context, listed []string) (string, error) @@ -80,6 +86,13 @@ type ConnectResult struct { KeyName string `json:"key_name,omitempty"` // Verified is the verification call's record. Verified CallRecord `json:"verified"` + // StreamUsage is the stream_usage written into the block, nil when none + // was (the block then asks for usage in the stream). + StreamUsage *bool `json:"stream_usage,omitempty"` + // StreamUsageNote says why stream_usage false was written when the + // setup chose it itself, after the server refused the field; "" when it + // did not. + StreamUsageNote string `json:"stream_usage_note,omitempty"` // Wrote names each file written, in the tilde form. Wrote []string `json:"wrote"` // Diagnostics are the configuration read's non-fatal reports (APIConfig's), @@ -140,8 +153,11 @@ func Connect(ctx context.Context, req ConnectRequest) (ConnectResult, error) { } res := ConnectResult{Provider: req.Provider, BaseURL: req.BaseURL, Models: append([]string(nil), req.Models...), KeyHome: req.Home, Diagnostics: append([]string(nil), cfg.Diagnostics...)} - svc := providerService(Provider{Name: req.Provider, BaseURL: req.BaseURL, Key: req.KeyName, Models: req.Models}, - cfg.denylist, &res.Verified, opts...) + // fellBack is set when the server refused the verification's usage + // request and answered it asked again without (providerService). + var fellBack bool + svc := providerService(Provider{Name: req.Provider, BaseURL: req.BaseURL, Key: req.KeyName, Models: req.Models, StreamUsage: req.StreamUsage}, + cfg.denylist, &res.Verified, &fellBack, opts...) block := map[string]any{"base_url": req.BaseURL, "models": req.Models} if req.Home == KeyHomeNone { if err := svc.Verify(ctx, ""); err != nil { @@ -164,6 +180,18 @@ func Connect(ctx context.Context, req ConnectRequest) (ConnectResult, error) { res.Wrote = append(res.Wrote, credential.WritesFor(req.Home, req.KeyName)...) } } + res.StreamUsage = req.StreamUsage + if fellBack { + off := false + res.StreamUsage = &off + res.StreamUsageNote = fmt.Sprintf("the server refused the verification call while it asked for usage in the stream (%s) "+ + "and answered when asked again without it, so the block sets %s to false: its calls are sent without the field, "+ + "and a call whose answer carries no usage records its token counts and cost as not reported", + openaiapi.StreamUsageField, streamUsageKey) + } + if res.StreamUsage != nil { + block[streamUsageKey] = *res.StreamUsage + } if err := writeProviderBlock(req.Roots.Home, req.Provider, block); err != nil { if len(res.Wrote) > 0 { return ConnectResult{}, fmt.Errorf("%w; the key was stored in the %s home under %s, and the provider block was not written", @@ -463,7 +491,15 @@ var KeyHomesProse = credential.HomesProse // key: what it unlocks, what works without it, and the adapter's own // verification call, one short exchange with the first model listed. The // call's record is written to rec when rec is non-nil. -func providerService(p Provider, denylist []DenyEntry, rec *CallRecord, opts ...openaiapi.Option) credential.Service { +// +// setup is non-nil for a setup's verification (Connect), the one place a +// call is asked twice: when p's block does not say whether to ask for usage +// in the stream and the server refuses the request that asked (HTTP 400 or +// 422, openaiapi.ErrUsageRefused), the call is asked once more without it, +// and *setup is set when that succeeds, for the block to say so. With setup +// nil (a configured provider's key verified again) a refusal is never +// retried and names the block's setting, as an ordinary call's does. +func providerService(p Provider, denylist []DenyEntry, rec *CallRecord, setup *bool, opts ...openaiapi.Option) credential.Service { return credential.Service{ Name: p.Key, Unlocks: "calls to the model provider " + p.Name + " at " + p.BaseURL + ", for the models its allowlist names", @@ -472,11 +508,36 @@ func providerService(p Provider, denylist []DenyEntry, rec *CallRecord, opts ... // The verification's own bounds come first, so a caller's option // replaces them. bounded := append([]openaiapi.Option{openaiapi.WithTimeout(verifyTimeout), openaiapi.WithFirstByteTimeout(verifyTimeout)}, opts...) - _, r, err := complete(ctx, p.Name, p.BaseURL, key, p.Models[0], verifyBrief, - Settings{"max_tokens": json.RawMessage(`16`)}, nil, denylist, bounded...) - r.Credential = p.Key - if rec != nil { - *rec = r + verify := func(p Provider) error { + _, r, err := complete(ctx, p, key, p.Models[0], verifyBrief, + Settings{"max_tokens": json.RawMessage(`16`)}, nil, denylist, bounded...) + r.Credential = p.Key + if rec != nil { + *rec = r + } + return err + } + err := verify(p) + switch { + case err == nil: + return nil + case setup == nil: + return withStreamUsageSetting(err, p.Name) + case p.StreamUsage == nil && errors.Is(err, openaiapi.ErrUsageRefused): + off := false + without := p + without.StreamUsage = &off + if err := verify(without); err != nil { + return fmt.Errorf("%w; the verification was asked once more without %s after a first refusal, and was refused again", + err, openaiapi.StreamUsageField) + } + *setup = true + return nil + case errors.Is(err, openaiapi.ErrStreamUsage): + // The setup was told to ask, and the server refused the + // field by name. + return fmt.Errorf("%w; if this server does not take %s, run the setup again with --stream-usage=false, "+ + "and the block it writes sends its calls without it", err, openaiapi.StreamUsageField) } return err }, @@ -495,7 +556,7 @@ func CredentialService(roots layered.Roots, name string) (credential.Service, bo } for _, p := range cfg.Providers() { if p.Key == name && len(p.Models) > 0 { - return providerService(p, cfg.denylist, nil), true, cfg.Diagnostics, nil + return providerService(p, cfg.denylist, nil, nil), true, cfg.Diagnostics, nil } } return credential.Service{}, false, cfg.Diagnostics, nil diff --git a/internal/core/oracle/connect_test.go b/internal/core/oracle/connect_test.go index 1598868fa0..61a2fd7fd5 100644 --- a/internal/core/oracle/connect_test.go +++ b/internal/core/oracle/connect_test.go @@ -5,6 +5,9 @@ import ( "encoding/json" "errors" "fmt" + "io" + "net/http" + "net/http/httptest" "os" "path/filepath" "reflect" @@ -15,6 +18,7 @@ import ( "time" "github.com/intentdriven/abcd/internal/abcdhome" + "github.com/intentdriven/abcd/internal/adapter/openaiapi" "github.com/intentdriven/abcd/internal/core/credential" "github.com/intentdriven/abcd/internal/fsutil" ) @@ -52,7 +56,7 @@ func TestConnectVerifiesThenWritesTheBlockAndTheKey(t *testing.T) { t.Fatal("the verification call did not carry the key") } want := CallRecord{Provider: "openrouter", ModelAsked: "typesafe/jev-1.13", ModelReported: "typesafe/jev-1.13-20260915", Credential: "openrouter"} - if res.Verified != want || res.KeyName != "openrouter" || res.KeyHome != KeyHomeABCD { + if !reflect.DeepEqual(who(res.Verified), want) || res.KeyName != "openrouter" || res.KeyHome != KeyHomeABCD { t.Fatalf("result = %+v", res) } if !reflect.DeepEqual(res.Wrote, []string{credential.StorePath, abcdhome.Display("config.json")}) { @@ -565,3 +569,114 @@ func TestTheVerificationCallKeepsItsOwnShortBound(t *testing.T) { t.Fatalf("a failed verification wrote the configuration: %v", err) } } + +// usageRefuser is a provider that refuses, with code and refusal, every chat +// request asking for usage in the stream and answers ok to one that does not, +// or refuses it too when always is set. It counts the chat requests. +func usageRefuser(t *testing.T, code int, refusal string, always bool) (base string, calls *atomic.Int32) { + t.Helper() + calls = &atomic.Int32{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet { + http.NotFound(w, r) + return + } + calls.Add(1) + raw, _ := io.ReadAll(r.Body) + w.Header().Set("Content-Type", "application/json") + if always || strings.Contains(string(raw), "stream_options") { + w.WriteHeader(code) + _, _ = io.WriteString(w, refusal) + return + } + _, _ = io.WriteString(w, chat("typesafe/jev-1.13-20260915", "ok")) + })) + t.Cleanup(srv.Close) + return srv.URL + "/v1", calls +} + +// TestConnectAsksAgainWithoutUsageWhenItIsRefused is iss-2610170934491794: +// a server that refuses the verification call's usage request, with 400 or +// 422 and whatever its body says, is asked once more without it; the setup +// then writes stream_usage false into the block it creates and says so. +func TestConnectAsksAgainWithoutUsageWhenItIsRefused(t *testing.T) { + for _, tc := range []struct { + code int + body string + }{ + {400, `{"error":{"message":"Unrecognized request argument supplied: stream_options"}}`}, + {422, `{"error":{"message":"invalid request"}}`}, + } { + base, calls := usageRefuser(t, tc.code, tc.body, false) + f := newFx(t) + res, err := Connect(context.Background(), connectReq(f, base)) + if err != nil { + t.Fatalf("HTTP %d: Connect: %v", tc.code, err) + } + if n := calls.Load(); n != 2 { + t.Fatalf("HTTP %d: %d calls, want the refused one and one without the field", tc.code, n) + } + if res.StreamUsage == nil || *res.StreamUsage || !strings.Contains(res.StreamUsageNote, "stream_usage") { + t.Fatalf("HTTP %d: result stream_usage %v, note %q", tc.code, res.StreamUsage, res.StreamUsageNote) + } + if len(res.Verified.AdapterAdded) != 0 { + t.Fatalf("HTTP %d: the record names the field as sent: %v", tc.code, res.Verified.AdapterAdded) + } + got, ok := f.loadAPI().Provider("openrouter") + if !ok || got.StreamUsage == nil || *got.StreamUsage || got.AsksStreamUsage() { + t.Fatalf("HTTP %d: block read back = %+v", tc.code, got) + } + } +} + +// TestConnectRefusedAgainMakesNoThirdCall: a refusal of a request that +// did not ask for usage, or a second refusal of the one asked again, is the +// setup's failure: nothing is written, and no third call is made. +func TestConnectRefusedAgainMakesNoThirdCall(t *testing.T) { + base, calls := usageRefuser(t, 400, `{"error":{"message":"bad model"}}`, true) + f := newFx(t) + _, err := Connect(context.Background(), connectReq(f, base)) + if err == nil || !strings.Contains(err.Error(), "nothing was written") || !strings.Contains(err.Error(), "without "+openaiapi.StreamUsageField) { + t.Fatalf("Connect: %v", err) + } + if n := calls.Load(); n != 2 { + t.Fatalf("%d calls, want two", n) + } + if _, err := os.Stat(machineFile(f, "config.json")); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("a failed verification wrote the configuration: %v", err) + } +} + +// TestConnectWritesTheStreamUsageItIsGiven: a setup told the server's +// stream_usage verifies with it, writes it into the block, and asks nothing +// again; told true and refused, its refusal names what the person can do. +func TestConnectWritesTheStreamUsageItIsGiven(t *testing.T) { + base, calls := usageRefuser(t, 400, `{"error":{"message":"Unrecognized request argument supplied: stream_options"}}`, false) + f := newFx(t) + req := connectReq(f, base) + off := false + req.StreamUsage = &off + res, err := Connect(context.Background(), req) + if err != nil || calls.Load() != 1 { + t.Fatalf("Connect: %v after %d calls", err, calls.Load()) + } + if res.StreamUsage == nil || *res.StreamUsage || res.StreamUsageNote != "" { + t.Fatalf("result stream_usage %v, note %q", res.StreamUsage, res.StreamUsageNote) + } + if got, _ := f.loadAPI().Provider("openrouter"); got.AsksStreamUsage() { + t.Fatalf("block read back = %+v", got) + } + + calls.Store(0) + f = newFx(t) + req = connectReq(f, base) + on := true + req.StreamUsage = &on + _, err = Connect(context.Background(), req) + if err == nil || calls.Load() != 1 || !strings.Contains(err.Error(), "--stream-usage=false") || strings.Contains(err.Error(), "config.json") { + t.Fatalf("Connect told true: %v after %d calls", err, calls.Load()) + } + if _, err := os.Stat(machineFile(f, "config.json")); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("a failed verification wrote the configuration: %v", err) + } +} diff --git a/internal/core/oracle/dispatch_test.go b/internal/core/oracle/dispatch_test.go index 6ccaabb0bb..5d308587de 100644 --- a/internal/core/oracle/dispatch_test.go +++ b/internal/core/oracle/dispatch_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json" "errors" + "reflect" "strings" "testing" @@ -118,7 +119,7 @@ func TestDispatchSendsTheStepThroughThePointedProvider(t *testing.T) { if rc.ConnectionUsed != "openrouter" || rc.ConnectionTried != "openrouter" || rc.FallbackReason != "" { t.Fatalf("receipt = %+v; want openrouter tried and used", rc) } - if rc.ProviderCall == nil || *rc.ProviderCall != (CallRecord{Provider: "openrouter", ModelAsked: "typesafe/jev-1.13", + if rc.ProviderCall == nil || !reflect.DeepEqual(who(*rc.ProviderCall), CallRecord{Provider: "openrouter", ModelAsked: "typesafe/jev-1.13", ModelReported: "typesafe/jev-1.13-20260915", Credential: "openrouter"}) { t.Fatalf("provider call = %+v", rc.ProviderCall) } diff --git a/internal/core/oracle/receipt.go b/internal/core/oracle/receipt.go index a9912ae672..ced92f9385 100644 --- a/internal/core/oracle/receipt.go +++ b/internal/core/oracle/receipt.go @@ -77,6 +77,11 @@ type ReceiptRoute struct { // payload: the provider, the model asked for and the model the provider // reported (itd-2609081951381895 criterion 5). null on the harness leg. ProviderCall *CallRecord `json:"provider_call"` + // Unpinned is the harness leg's list of the fixed fields it could not pin + // (itd-166): the model is the payload's own declaration, so it is + // self-reported, and nothing else of either side reaches abcd from the + // host. null on a provider leg, whose call record carries its own list. + Unpinned []Unpinned `json:"unpinned"` } // WithCall returns the receipt carrying the provider call that produced its @@ -93,12 +98,44 @@ func (r Route) Receipt(model string) ReceiptRoute { if sent == nil { sent = Settings{} } - return ReceiptRoute{ + rr := ReceiptRoute{ Agent: r.Agent, TierAsked: r.Row.Tier, ConnectionTried: r.ConnectionTried, ConnectionUsed: r.ConnectionUsed, FallbackReason: r.Fallback, Override: r.Override, SettingsSent: sent, ModelReported: model, } + if !r.OnProvider() { + rr.Unpinned = handOffUnpinned(model) + } + return rr +} + +// HandOffUnpinnedForTest is handOffUnpinned, for the surface tests that +// build the receipt a harness leg writes. +func HandOffUnpinnedForTest(model string) []Unpinned { return handOffUnpinned(model) } + +// handOffUnpinned is the harness leg's list: the host chooses the model and +// its settings and renders the prompt, and reports none of it, so every asked +// field is not configured by abcd or not reported to it, and every served +// field but the payload's own model is not reported. +func handOffUnpinned(model string) []Unpinned { + var un Unpin + for _, f := range AskedFields { + switch f { + case AskedPromptSHA256, AskedHarnessVersion: + un.Add(f, NotReported) + default: + un.Add(f, NotConfigured) + } + } + for _, f := range ServedFields { + if f == ServedModel && model != "" { + un.Add(f, SelfReported) + continue + } + un.Add(f, NotReported) + } + return un.List() } // RenderRequestSection renders the routing section a request document carries, diff --git a/internal/core/oracle/sides.go b/internal/core/oracle/sides.go new file mode 100644 index 0000000000..19fe555d3e --- /dev/null +++ b/internal/core/oracle/sides.go @@ -0,0 +1,138 @@ +package oracle + +// sides.go is the vocabulary every model call's record shares (itd-166): the +// side abcd asked for and the side that answered, on whichever route the call +// took. The provider adapter's record (CallRecord), the harness hand-off's +// receipt (ReceiptRoute) and the command-line runners' route record +// (runner.RouteRecord) each carry the same token counts, the same cost and the +// same list of the fields they could not pin, so a reader compares two calls +// field by field whatever routes they took. The runner package imports this +// one; nothing here imports the runner. + +import "github.com/intentdriven/abcd/internal/adapter/openaiapi" + +// Field is one field of the fixed lists a call's record is held to (itd-166 +// decision 10). Every route records each field or lists it in its Unpinned +// with the reason, so a field is never silently absent. +type Field string + +// The asked side: what the dispatching layer requested. +const ( + AskedModel Field = "asked.model" + AskedProviderPin Field = "asked.provider_pin" + AskedTemperature Field = "asked.temperature" + AskedTopP Field = "asked.top_p" + AskedSeed Field = "asked.seed" + AskedEffort Field = "asked.effort" + AskedContext Field = "asked.context" + AskedPromptSHA256 Field = "asked.prompt_sha256" + AskedHarnessVersion Field = "asked.harness_version" + // The local fields are a local model's (decision 7): the weight file's + // fingerprint, its quantisation and the runtime's version, which a local + // route lists as not configured until a local route can be pinned + // (iss-2610161734428291). + AskedWeightsSHA256 Field = "asked.weights_sha256" + AskedQuantisation Field = "asked.quantisation" + AskedRuntimeVersion Field = "asked.runtime_version" +) + +// The served side: what actually answered. +const ( + ServedModel Field = "served.model" + ServedProvider Field = "served.provider" + ServedGenerationID Field = "served.generation_id" + ServedFinishReason Field = "served.finish_reason" + ServedPromptTokens Field = "served.prompt_tokens" + ServedCompletionTokens Field = "served.completion_tokens" + ServedReasoningTokens Field = "served.reasoning_tokens" + ServedCacheReadTokens Field = "served.cache_read_tokens" + ServedCacheWriteTokens Field = "served.cache_write_tokens" + ServedCost Field = "served.cost" +) + +// AskedFields and ServedFields are the fixed lists, in the order a record +// lists them. LocalFields extends the asked list on a local route. +var ( + AskedFields = []Field{AskedModel, AskedProviderPin, AskedTemperature, AskedTopP, AskedSeed, + AskedEffort, AskedContext, AskedPromptSHA256, AskedHarnessVersion} + LocalFields = []Field{AskedWeightsSHA256, AskedQuantisation, AskedRuntimeVersion} + ServedFields = []Field{ServedModel, ServedProvider, ServedGenerationID, ServedFinishReason, + ServedPromptTokens, ServedCompletionTokens, ServedReasoningTokens, + ServedCacheReadTokens, ServedCacheWriteTokens, ServedCost} +) + +// UnpinnedReason is why a field is not pinned (decision 8). +type UnpinnedReason string + +const ( + // NotReported: what the route returned does not carry the field. + NotReported UnpinnedReason = "not_reported" + // SelfReported: the model, or the payload it wrote, declared the value, + // which nothing outside it confirms. + SelfReported UnpinnedReason = "self_reported" + // HarnessComputed: the harness computed the value (a cost from its own + // price table), not the provider that served the call. + HarnessComputed UnpinnedReason = "harness_computed" + // NotConfigured: no setting or pin names the value, so the server's + // default applied, whatever it was. + NotConfigured UnpinnedReason = "not_configured" +) + +// Unpinned is one field a record could not pin, and why. +type Unpinned struct { + Field Field `json:"field"` + Reason UnpinnedReason `json:"reason"` +} + +// Usage is the token counts a call's record carries. A count the route did +// not report is zero here and listed in the record's Unpinned. +type Usage struct { + PromptTokens int `json:"prompt_tokens"` + CompletionTokens int `json:"completion_tokens"` + ReasoningTokens int `json:"reasoning_tokens"` + CacheReadTokens int `json:"cache_read_tokens"` + CacheWriteTokens int `json:"cache_write_tokens"` +} + +// Cost is what a call cost, in the unit its route reports: CostUSD for the +// runners' harnesses, CostCredits for a provider's usage member. A record +// carries a *Cost, so a cost of zero is never confused with one not reported. +type Cost struct { + Amount float64 `json:"amount"` + Unit string `json:"unit"` +} + +// ReadableCost reports whether a cost a route reported, or a sum of them, is +// one a record keeps: zero or more, and finite. It is the adapter's policy +// (openaiapi.ReadableCost), the one every route's cost is held to: a cost +// that fails it is not reported. +func ReadableCost(c float64) bool { return openaiapi.ReadableCost(c) } + +// The units a Cost is reported in. +const ( + CostUSD = "usd" + CostCredits = "credits" +) + +// Unpin is a record's list of the fields it could not pin, built in the +// fixed lists' order. +type Unpin []Unpinned + +// Add lists f with reason r. +func (u *Unpin) Add(f Field, r UnpinnedReason) { *u = append(*u, Unpinned{Field: f, Reason: r}) } + +// Unless lists f with reason r when have is false. +func (u *Unpin) Unless(have bool, f Field, r UnpinnedReason) { + if !have { + u.Add(f, r) + } +} + +// List is the list as a record stores it: never nil, so a record that pins +// every field says so with an empty list. +func (u Unpin) List() []Unpinned { + if u == nil { + return []Unpinned{} + } + return []Unpinned(u) +} diff --git a/internal/core/oracle/sides_test.go b/internal/core/oracle/sides_test.go new file mode 100644 index 0000000000..82d6ff97f6 --- /dev/null +++ b/internal/core/oracle/sides_test.go @@ -0,0 +1,285 @@ +package oracle + +import ( + "context" + "encoding/json" + "reflect" + "strings" + "testing" + + "github.com/intentdriven/abcd/internal/adapter/openaiapi" + "github.com/intentdriven/abcd/internal/core/credential" +) + +// The tests in this file are itd-166's oracle half: a call through the +// provider adapter records the side abcd asked for and the side that +// answered, and lists every field of the fixed lists it could not pin with +// its reason; the harness hand-off's receipt lists the payload's model as +// self-reported. + +// openRouterBody is an OpenRouter-shaped chat completion: its generation id, +// the upstream provider, the finish reason and a usage member carrying the +// token details and the cost. +const openRouterBody = `{"id":"gen-1760000000-abcdef","provider":"Fireworks","model":"typesafe/jev-1.13-20260915",` + + `"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"{\"verdict\":\"keep\"}"}}],` + + `"usage":{"prompt_tokens":120,"completion_tokens":30,"total_tokens":150,` + + `"prompt_tokens_details":{"cached_tokens":100,"cache_write_tokens":20},` + + `"completion_tokens_details":{"reasoning_tokens":12},"cost":0.00042}}` + +var sidesBrief = openaiapi.Brief{Instructions: "the scribe's contract", Input: "the request document"} + +// assertListed holds every entry of a record's list to the fixed lists and +// the four reasons, each field at most once. +func assertListed(t *testing.T, us []Unpinned) { + t.Helper() + known := map[Field]bool{} + for _, f := range append(append(append([]Field{}, AskedFields...), LocalFields...), ServedFields...) { + known[f] = true + } + seen := map[Field]bool{} + for _, u := range us { + if !known[u.Field] { + t.Errorf("listed field %q is not on the fixed lists", u.Field) + } + switch u.Reason { + case NotReported, SelfReported, HarnessComputed, NotConfigured: + default: + t.Errorf("field %s has reason %q, outside the four", u.Field, u.Reason) + } + if seen[u.Field] { + t.Errorf("field %s is listed twice", u.Field) + } + seen[u.Field] = true + } +} + +// TestAProviderCallRecordsBothSides is criterion 1: the record carries the +// asked side (the model, the prompt's SHA-256 and the field the adapter added) +// and the served side (the model, upstream provider, generation id, finish +// reason, every token count and the cost) the response names, and lists +// exactly the fixed fields neither carries. +func TestAProviderCallRecordsBothSides(t *testing.T) { + p := Provider{Name: "openrouter", BaseURL: "https://openrouter.ai/api/v1"} + res := openaiapi.Result{ModelAsked: "typesafe/jev-1.13", ModelReported: "typesafe/jev-1.13-20260915", + FinishReason: "stop", GenerationID: "gen-1", Provider: "Fireworks"} + n := func(v int) *int { return &v } + cost := 0.00042 + res.Usage = &openaiapi.Usage{PromptTokens: n(120), CompletionTokens: n(30), TotalTokens: n(150), + CachedTokens: n(100), CacheWriteTokens: n(20), ReasoningTokens: n(12), Cost: &cost} + rec := callRecord(p, sidesBrief, Settings{"temperature": raw("0")}, res) + if rec.PromptSHA256 != openaiapi.PromptSHA256(sidesBrief) || rec.ServedProvider != "Fireworks" || + rec.GenerationID != "gen-1" || rec.FinishReason != "stop" { + t.Fatalf("record = %+v", rec) + } + if rec.Usage == nil || *rec.Usage != (Usage{PromptTokens: 120, CompletionTokens: 30, ReasoningTokens: 12, CacheReadTokens: 100, CacheWriteTokens: 20}) { + t.Fatalf("usage = %+v", rec.Usage) + } + if rec.Cost == nil || *rec.Cost != (Cost{Amount: 0.00042, Unit: CostCredits}) { + t.Fatalf("cost = %+v", rec.Cost) + } + if !reflect.DeepEqual(rec.AdapterAdded, []string{openaiapi.StreamUsageField}) { + t.Fatalf("adapter added = %v", rec.AdapterAdded) + } + want := []Unpinned{ + {AskedProviderPin, NotConfigured}, {AskedTopP, NotConfigured}, {AskedSeed, NotConfigured}, + {AskedEffort, NotConfigured}, {AskedContext, NotConfigured}, {AskedHarnessVersion, NotReported}, + } + if !reflect.DeepEqual(rec.Unpinned, want) { + t.Fatalf("unpinned = %v, want %v", rec.Unpinned, want) + } + assertListed(t, rec.Unpinned) +} + +// TestACallNamingNoServedDetailListsEachMissing is criterion 4 on the +// adapter: an answer that names only its model leaves every other served +// field listed as not reported, and a local route lists its weights, +// quantisation and runtime as not configured. +func TestACallNamingNoServedDetailListsEachMissing(t *testing.T) { + prov := newProvFake(t, 200, chat("typesafe/jev-1.13-20260915", `{"verdict":"keep"}`)) + f, c := configured(t, prov.base(), callKey) + tgt, _ := c.Role("scribe") + _, rec, err := c.Call(context.Background(), credential.Machine(f.roots.Home), CallRequest{Target: tgt, Brief: sidesBrief}) + if err != nil { + t.Fatal(err) + } + if rec.Usage != nil || rec.Cost != nil { + t.Fatalf("usage %+v, cost %+v: want neither", rec.Usage, rec.Cost) + } + got := map[Field]UnpinnedReason{} + for _, u := range rec.Unpinned { + got[u.Field] = u.Reason + } + for _, f := range ServedFields { + if f == ServedModel { + if _, listed := got[f]; listed { + t.Errorf("the served model the provider reported is listed: %v", rec.Unpinned) + } + continue + } + if got[f] != NotReported { + t.Errorf("%s = %q, want not_reported", f, got[f]) + } + } + for _, f := range LocalFields { + if got[f] != NotConfigured { + t.Errorf("%s on a local route = %q, want not_configured", f, got[f]) + } + } + for _, f := range []Field{AskedTemperature, AskedTopP, AskedSeed} { + if got[f] != NotConfigured { + t.Errorf("%s with no setting sent = %q, want not_configured", f, got[f]) + } + } + assertListed(t, rec.Unpinned) +} + +// TestTheCallRecordTravelsInTheReceipt: a dispatched step's receipt carries +// the call's both sides, read from an OpenRouter-shaped answer. +func TestTheCallRecordTravelsInTheReceipt(t *testing.T) { + prov := newProvFake(t, 200, openRouterBody) + f, c := configured(t, prov.base(), callKey) + tgt, _ := c.Role("scribe") + _, rec, err := c.Call(context.Background(), credential.Machine(f.roots.Home), CallRequest{Target: tgt, Brief: sidesBrief}) + if err != nil { + t.Fatal(err) + } + if rec.ServedProvider != "Fireworks" || rec.GenerationID != "gen-1760000000-abcdef" || rec.Usage == nil || rec.Cost == nil { + t.Fatalf("record = %+v", rec) + } + if !strings.Contains(prov.body.Load().(string), `"stream_options":{"include_usage":true}`) { + t.Fatalf("the request did not ask for usage in the stream: %s", prov.body.Load()) + } + enc, _ := json.Marshal(ReceiptRoute{}.WithCall(rec)) + for _, k := range []string{`"prompt_sha256":"`, `"served_provider":"Fireworks"`, `"generation_id":"gen-1760000000-abcdef"`, + `"finish_reason":"stop"`, `"cost":{"amount":0.00042,"unit":"credits"}`, `"reasoning_tokens":12`, `"adapter_added":["stream_options.include_usage"]`} { + if !strings.Contains(string(enc), k) { + t.Errorf("receipt lacks %s: %s", k, enc) + } + } +} + +// TestAConnectionThatTurnsStreamUsageOffSendsWithoutIt is criterion 6: a +// provider block with stream_usage false sends no stream_options, the call +// succeeds with usage listed as not reported, and nothing is recorded as +// added by the adapter. +func TestAConnectionThatTurnsStreamUsageOffSendsWithoutIt(t *testing.T) { + prov := newProvFake(t, 200, chat("typesafe/jev-1.13-20260915", `{"verdict":"keep"}`)) + f := newFx(t) + f.machineConfig(`{"oracle":{"api":{"local":{"base_url":"` + prov.base() + `","stream_usage":false,` + + `"models":["typesafe/jev-1.13"]}},"roles":{"scribe":"local/typesafe/jev-1.13"}}}`) + c := f.loadAPI() + tgt, _ := c.Role("scribe") + _, rec, err := c.Call(context.Background(), nil, CallRequest{Target: tgt, Brief: sidesBrief}) + if err != nil { + t.Fatalf("Call: %v", err) + } + if strings.Contains(prov.body.Load().(string), "stream_options") { + t.Fatalf("a connection with stream_usage false asked for it: %s", prov.body.Load()) + } + if len(rec.AdapterAdded) != 0 { + t.Fatalf("adapter added = %v, want none", rec.AdapterAdded) + } + listed := false + for _, u := range rec.Unpinned { + listed = listed || (u.Field == ServedPromptTokens && u.Reason == NotReported) + } + if !listed { + t.Fatalf("usage is not listed as not reported: %v", rec.Unpinned) + } +} + +// TestARefusedUsageRequestNamesTheSetting is criterion 6's refusal: a server +// that refuses the request while it asked for usage in the stream is refused +// with the setting that stops asking named. +func TestARefusedUsageRequestNamesTheSetting(t *testing.T) { + prov := newProvFake(t, 400, `{"error":{"message":"Unrecognized request argument supplied: stream_options"}}`) + f, c := configured(t, prov.base(), callKey) + tgt, _ := c.Role("scribe") + _, _, err := c.Call(context.Background(), credential.Machine(f.roots.Home), CallRequest{Target: tgt, Brief: sidesBrief}) + if err == nil || !strings.Contains(err.Error(), "oracle.api.openrouter.stream_usage") { + t.Fatalf("err = %v, want it to name oracle.api.openrouter.stream_usage", err) + } + if prov.calls.Load() != 1 { + t.Fatalf("the provider was called %d times; a refusal is never retried", prov.calls.Load()) + } +} + +// TestStreamUsageMustBeABoolean: the block's stream_usage is read strictly. +func TestStreamUsageMustBeABoolean(t *testing.T) { + f := newFx(t) + f.machineConfig(`{"oracle":{"api":{"local":{"base_url":"http://localhost:8000/v1","stream_usage":"no","models":["m"]}}}}`) + if err := f.loadAPIErr(); !strings.Contains(err.Error(), "stream_usage") { + t.Fatalf("err = %v, want it to name stream_usage", err) + } +} + +// TestTheHostHandOffMarksTheServedModelSelfReported is criterion 5: on the +// harness leg the receipt's model is the payload's own declaration, listed +// as self-reported, and every other field of the fixed lists is listed with +// its reason; a payload naming no model lists it as not reported. +func TestTheHostHandOffMarksTheServedModelSelfReported(t *testing.T) { + r, err := Resolve("scribe", newFx(t).load(), NoConnections{}) + if err != nil { + t.Fatal(err) + } + rc := r.Receipt("claude-example-1") + assertListed(t, rc.Unpinned) + got := map[Field]UnpinnedReason{} + for _, u := range rc.Unpinned { + got[u.Field] = u.Reason + } + if got[ServedModel] != SelfReported { + t.Fatalf("served model = %q, want self_reported: %v", got[ServedModel], rc.Unpinned) + } + if len(rc.Unpinned) != len(AskedFields)+len(ServedFields) { + t.Fatalf("listed %d fields, want all %d of the fixed lists: %v", len(rc.Unpinned), len(AskedFields)+len(ServedFields), rc.Unpinned) + } + enc, _ := json.Marshal(rc) + if !strings.Contains(string(enc), `{"field":"served.model","reason":"self_reported"}`) { + t.Fatalf("receipt = %s", enc) + } + for _, u := range r.Receipt("").Unpinned { + if u.Field == ServedModel && u.Reason != NotReported { + t.Fatalf("a payload naming no model lists it as %q", u.Reason) + } + } +} + +// TestAMalformedUsageMemberListsEachUnreadCount: a usage member whose prompt +// or completion count is missing or of another type than the protocol's +// lists that count as not reported, each on its own, never a zero standing in +// for it; a usage member with no readable count lists both. +func TestAMalformedUsageMemberListsEachUnreadCount(t *testing.T) { + for _, tc := range []struct { + usage string + prompt, complete bool // whether each count is pinned + cost bool + }{ + {`{"prompt_tokens":4,"completion_tokens":"2","cost":"0.1","prompt_tokens_details":[1]}`, true, false, false}, + {`{"cost":0.1}`, false, false, true}, + {`{"completion_tokens":2}`, false, true, false}, + } { + body := `{"model":"typesafe/jev-1.13-20260915","choices":[{"index":0,"finish_reason":"stop",` + + `"message":{"role":"assistant","content":"{\"verdict\":\"keep\"}"}}],"usage":` + tc.usage + `}` + prov := newProvFake(t, 200, body) + f, c := configured(t, prov.base(), callKey) + tgt, _ := c.Role("scribe") + _, rec, err := c.Call(context.Background(), credential.Machine(f.roots.Home), CallRequest{Target: tgt, Brief: sidesBrief}) + if err != nil { + t.Fatalf("%s: %v", tc.usage, err) + } + got := map[Field]UnpinnedReason{} + for _, u := range rec.Unpinned { + got[u.Field] = u.Reason + } + for f, pinned := range map[Field]bool{ServedPromptTokens: tc.prompt, ServedCompletionTokens: tc.complete, ServedCost: tc.cost} { + if r, listed := got[f]; listed == pinned || (listed && r != NotReported) { + t.Errorf("%s: %s listed %v (%q), want pinned %v", tc.usage, f, listed, r, pinned) + } + } + if tc.prompt && (rec.Usage == nil || rec.Usage.PromptTokens != 4) { + t.Errorf("%s: usage = %+v", tc.usage, rec.Usage) + } + assertListed(t, rec.Unpinned) + } +} diff --git a/internal/core/runner/adapter_test.go b/internal/core/runner/adapter_test.go index f74d7467a2..9581cf1fb4 100644 --- a/internal/core/runner/adapter_test.go +++ b/internal/core/runner/adapter_test.go @@ -78,7 +78,9 @@ func TestOpenCodeLaunch(t *testing.T) { if argv[len(argv)-2] != "--" { t.Errorf("the prompt is not behind the end-of-options marker: %q", argv) } - if ans.SessionID != "ses_fake1" || ans.Text != "done" || ans.Model != "openrouter/qwen/qwen3-coder" { + // Run mode names no model, so the served model is empty and the one the + // runner passed is the asked side's (iss-2610161709506207). + if ans.SessionID != "ses_fake1" || ans.Text != "done" || ans.Model != "" || ans.ModelAsked != "openrouter/qwen/qwen3-coder" { t.Errorf("answer = %+v", ans) } } diff --git a/internal/core/runner/claude.go b/internal/core/runner/claude.go index 4196cec739..7b1a0acc60 100644 --- a/internal/core/runner/claude.go +++ b/internal/core/runner/claude.go @@ -38,7 +38,10 @@ import ( "os" "path/filepath" "slices" + "sort" "strings" + + "github.com/intentdriven/abcd/internal/core/oracle" ) // ClaudeCLI is the claude CLI runner. @@ -102,6 +105,13 @@ func (c *ClaudeCLI) approverConfig(dir, base string, runDirs []string) string { // Name is the route's name. func (*ClaudeCLI) Name() string { return Claude } +// askedModel is the model the runner passes the harness: the route's model +// after its provider, "" for the harness's own default. +func (c *ClaudeCLI) askedModel() string { + _, m, _ := strings.Cut(c.model, "/") + return m +} + // args is the launch's argv after the binary. func (c *ClaudeCLI) args(req Request) []string { shell := grantsShell(req.Tools) @@ -140,7 +150,7 @@ func (c *ClaudeCLI) args(req Request) []string { for _, d := range dirs { a = append(a, "--add-dir="+d) } - if _, m, ok := strings.Cut(c.model, "/"); ok { + if m := c.askedModel(); m != "" { a = append(a, "--model="+m) } return append(a, "--", prompt(req)) @@ -178,7 +188,12 @@ func (c *ClaudeCLI) Run(ctx context.Context, req Request) (Answer, []byte, error return Answer{}, transcript, err } ans, err := parseClaude(res.stdout) - return ans, transcript, err + if err != nil { + return ans, transcript, err + } + ans.ModelAsked, ans.PromptSHA256 = c.askedModel(), promptSHA256(req) + ans.Unpinned = claudeUnpinned(ans) + return ans, transcript, nil } // rateLimited is the claude runner's rate-limit failure, what naming the @@ -229,6 +244,101 @@ type claudeEvent struct { Model string `json:"model"` IsError bool `json:"is_error"` Result string `json:"result"` + // Version is the init event's claude_code_version. + Version string `json:"claude_code_version"` +} + +// claudeMeter is the part of the result event that meters the run (the +// claude CLI 2.1): its usage, its cost, why it stopped and the usage per +// model. It is read leniently and apart from claudeEvent, so a metering field +// of an unexpected shape is a detail not reported, never an unparsable run. +type claudeMeter struct { + StopReason string `json:"stop_reason"` + TotalCostUSD *float64 `json:"total_cost_usd"` + Usage struct { + InputTokens *int `json:"input_tokens"` + OutputTokens *int `json:"output_tokens"` + CacheReadInputTokens *int `json:"cache_read_input_tokens"` + CacheCreationInputTokens *int `json:"cache_creation_input_tokens"` + OutputTokensDetails struct { + ThinkingTokens *int `json:"thinking_tokens"` + } `json:"output_tokens_details"` + } `json:"usage"` + ModelUsage map[string]struct { + Provider string `json:"provider"` + } `json:"modelUsage"` +} + +// claudeCounted reports which of the counts a result event carries readably: +// present and not negative, as the provider adapter reads a count. A count +// below zero is unreadable, listed as not reported. +func claudeCounted(m claudeMeter) counted { + u := m.Usage + ok := func(p *int) bool { return p != nil && *p >= 0 } + return counted{prompt: ok(u.InputTokens), completion: ok(u.OutputTokens), + reasoning: ok(u.OutputTokensDetails.ThinkingTokens), + cacheRead: ok(u.CacheReadInputTokens), cacheWrite: ok(u.CacheCreationInputTokens)} +} + +// meterClaude reads the result event's metering onto ans: the token counts +// and the cost the harness computed, the stop reason, the models its +// per-model usage names (the init event's model stays the canonical one, and +// stands in only when the init event named none and one model served), and +// the provider it names for the canonical model. +func meterClaude(ans *Answer, line []byte) { + var m claudeMeter + if json.Unmarshal(line, &m) != nil { + return + } + ans.FinishReason = cleanHarness(m.StopReason) + u := m.Usage + c := claudeCounted(m) + ans.counted = c + if c.prompt || c.completion || c.reasoning || c.cacheRead || c.cacheWrite { + n := func(p *int) int { + if p == nil || *p < 0 { + return 0 + } + return *p + } + ans.Usage = &oracle.Usage{PromptTokens: n(u.InputTokens), CompletionTokens: n(u.OutputTokens), + ReasoningTokens: n(u.OutputTokensDetails.ThinkingTokens), + CacheReadTokens: n(u.CacheReadInputTokens), CacheWriteTokens: n(u.CacheCreationInputTokens)} + } + if m.TotalCostUSD != nil && oracle.ReadableCost(*m.TotalCostUSD) { + ans.Cost = &oracle.Cost{Amount: *m.TotalCostUSD, Unit: oracle.CostUSD} + } + keys := make([]string, 0, len(m.ModelUsage)) + for k := range m.ModelUsage { + keys = append(keys, k) + } + sort.Strings(keys) + if ans.Model == "" && len(keys) == 1 && modelRe.MatchString(keys[0]) { + ans.Model = keys[0] + } + provider := "" + if mu, ok := m.ModelUsage[ans.Model]; ok { + provider = mu.Provider + } else if len(keys) == 1 { + provider = m.ModelUsage[keys[0]].Provider + } + ans.ServedProvider = cleanHarness(provider) + ans.ServedModels = nil + for _, k := range keys { + if len(ans.ServedModels) == maxServedModels { + break + } + ans.ServedModels = append(ans.ServedModels, cleanHarness(k)) + } +} + +// claudeUnpinned lists the fields a claude run could not pin. The cost is +// the harness's own computation, from its price table. +func claudeUnpinned(ans Answer) []oracle.Unpinned { + var un oracle.Unpin + askedUnpinned(&un, ans.Sides) + servedUnpinned(&un, ans.Model, ans.Sides, ans.counted) + return un.List() } // parseClaude reads the event stream: one JSON object per line, the init @@ -254,6 +364,7 @@ func parseClaude(out []byte) (Answer, error) { return Answer{}, fail(Claude, ReasonUnparsable, "its init event reports a model that is not a plain model id of at most 128 characters") } ans.Model = ev.Model + ans.HarnessVersion = cleanHarness(ev.Version) } if ans.SessionID == "" && ev.SessionID != "" { ans.SessionID = ev.SessionID @@ -261,6 +372,7 @@ func parseClaude(out []byte) (Answer, error) { if ev.Type == "result" { e := ev result = &e + meterClaude(&ans, line) } } if result == nil || result.IsError || result.Subtype != "success" { diff --git a/internal/core/runner/dispatch.go b/internal/core/runner/dispatch.go index 2cf95aee7f..f99c038838 100644 --- a/internal/core/runner/dispatch.go +++ b/internal/core/runner/dispatch.go @@ -153,8 +153,7 @@ const none = "none" func (o Outcome) ran(route string, ans Answer) Outcome { o.Answer = &ans - o.Receipt.Route.Ran = route - o.Receipt.Route.Model = ans.Model + o.Receipt.Route = RecordOf(o.Receipt.Route.Asked, route, ans) return o } diff --git a/internal/core/runner/dispatch_test.go b/internal/core/runner/dispatch_test.go index deb0a50a60..3bad628edc 100644 --- a/internal/core/runner/dispatch_test.go +++ b/internal/core/runner/dispatch_test.go @@ -324,7 +324,7 @@ func TestReceiptsDifferOnlyInRoute(t *testing.T) { t.Fatal(err) } a, b := viaRunner.Receipt, viaHost.Receipt - if a.Route == b.Route { + if reflect.DeepEqual(a.Route, b.Route) { t.Fatal("the two receipts do not name different routes") } a.Route, b.Route = RouteRecord{}, RouteRecord{} diff --git a/internal/core/runner/main_test.go b/internal/core/runner/main_test.go index 5fa3787a87..aa86233915 100644 --- a/internal/core/runner/main_test.go +++ b/internal/core/runner/main_test.go @@ -28,6 +28,8 @@ const ( // the test sets it to a value built at run time and asserts it never // reaches an argument, an error or a receipt. fakeCredEnv = "ANTHROPIC_API_KEY" + // fakeStreamEnv names the fixture the stream mode answers with. + fakeStreamEnv = "ABCD_RUNNER_FAKE_STREAM" ) func TestMain(m *testing.M) { @@ -128,6 +130,18 @@ func fakeHarness(mode string) int { fmt.Fprintln(os.Stdout, string(init)) fmt.Fprintln(os.Stdout, `{"type":"result","subtype":"success","is_error":false,"result":"done","session_id":"fake-session-1"}`) return 0 + case "stream": + // A harness that answers with a recorded event stream: the file + // fakeStreamEnv names, a fixture under testdata/. + if rec := promptField(prompt, "Receipt: "); rec != "" { + _ = os.WriteFile(rec, []byte(`{"ok":true}`), 0o600) + } + raw, err := os.ReadFile(os.Getenv(fakeStreamEnv)) + if err != nil { + return 3 + } + _, _ = os.Stdout.Write(raw) + return 0 case "approve": // A harness that puts two permission prompts to the prompt tool its // launch names, through the MCP server its --mcp-config starts, and diff --git a/internal/core/runner/opencode.go b/internal/core/runner/opencode.go index 223e1dcb95..89d9eaa770 100644 --- a/internal/core/runner/opencode.go +++ b/internal/core/runner/opencode.go @@ -24,11 +24,22 @@ package runner // // The runner reaches opencode through run mode only; its server's session // endpoints, where a server is already up, are a later adapter. +// +// Run mode's JSON events name no model (opencode 1.18.31: tool_use, +// step_start, step_finish, text, reasoning and error), so a run's served model +// is not reported and is recorded so, never as the model the runner asked for +// (itd-166 decision 6, iss-2610161709506207). Its step-finish events carry +// each step's tokens and the cost opencode computed, which a run sums. Run +// mode emits the run's own session's events only, so a sub-agent's steps are +// not among them and the sums are a floor. import ( "bytes" "context" "encoding/json" + "math" + + "github.com/intentdriven/abcd/internal/core/oracle" ) // openCodeSeal is the environment every opencode launch is sealed with, set @@ -82,7 +93,11 @@ func (o *OpenCodeCLI) Run(ctx context.Context, req Request) (Answer, []byte, err if err != nil { return Answer{}, transcript, err } - ans.Model = o.model + ans.ModelAsked, ans.PromptSHA256 = o.model, promptSHA256(req) + var un oracle.Unpin + askedUnpinned(&un, ans.Sides) + servedUnpinned(&un, ans.Model, ans.Sides, ans.counted) + ans.Unpinned = un.List() return ans, transcript, nil } @@ -96,6 +111,79 @@ type openCodeEvent struct { } `json:"part"` } +// openCodeStep is the metering of a step_finish event's part: why the step +// stopped, the cost opencode computed and the step's tokens. It is read +// leniently and apart from openCodeEvent, so a field of an unexpected shape +// is a detail not reported, never an unparsable run. +type openCodeStep struct { + Part struct { + Reason string `json:"reason"` + Cost *float64 `json:"cost"` + Tokens *struct { + Input *int `json:"input"` + Output *int `json:"output"` + Reasoning *int `json:"reasoning"` + Cache struct { + Read *int `json:"read"` + Write *int `json:"write"` + } `json:"cache"` + } `json:"tokens"` + } `json:"part"` +} + +// meterOpenCode adds one step_finish event's metering to ans: its tokens and +// cost summed onto the run's, its reason the run's finish reason until a +// later step's replaces it. +func meterOpenCode(ans *Answer, line []byte) { + var st openCodeStep + if json.Unmarshal(line, &st) != nil { + return + } + p := st.Part + if p.Reason != "" { + ans.FinishReason = cleanHarness(p.Reason) + } + // A step's cost is summed when it is readable (oracle.ReadableCost); a + // sum that stops being finite is dropped, not reported for the rest of + // the run, since +Inf is no cost and no record can write it. + if p.Cost != nil && !ans.costLost && oracle.ReadableCost(*p.Cost) { + sum := *p.Cost + if ans.Cost != nil { + sum += ans.Cost.Amount + } + if oracle.ReadableCost(sum) { + ans.Cost = &oracle.Cost{Amount: sum, Unit: oracle.CostUSD} + } else { + ans.Cost, ans.costLost = nil, true + } + } + t := p.Tokens + if t == nil { + return + } + if ans.Usage == nil { + ans.Usage = &oracle.Usage{} + } + // A count is summed when present and not negative; a sum that would + // overflow is unreadable, zeroed and not reported for the rest of the run. + add := func(sum *int, n *int, seen, lost *bool) { + switch { + case *lost || n == nil || *n < 0: + case *sum > math.MaxInt-*n: + *sum, *seen, *lost = 0, false, true + default: + *sum += *n + *seen = true + } + } + c, l := &ans.counted, &ans.lost + add(&ans.Usage.PromptTokens, t.Input, &c.prompt, &l.prompt) + add(&ans.Usage.CompletionTokens, t.Output, &c.completion, &l.completion) + add(&ans.Usage.ReasoningTokens, t.Reasoning, &c.reasoning, &l.reasoning) + add(&ans.Usage.CacheReadTokens, t.Cache.Read, &c.cacheRead, &l.cacheRead) + add(&ans.Usage.CacheWriteTokens, t.Cache.Write, &c.cacheWrite, &l.cacheWrite) +} + // parseOpenCode reads the event stream: one JSON object per line. An error // event is a refusal; a line that is not an event, or a stream with no text // and no finished step, is unparsable. The last text part is the final @@ -123,6 +211,7 @@ func parseOpenCode(out []byte) (Answer, error) { seen = true case "step_finish": seen = true + meterOpenCode(&ans, line) } } if !seen { diff --git a/internal/core/runner/receipt.go b/internal/core/runner/receipt.go index 7f869b620b..1a16a121a9 100644 --- a/internal/core/runner/receipt.go +++ b/internal/core/runner/receipt.go @@ -7,15 +7,21 @@ package runner import "time" -// RouteRecord is the route a role asked for and the one that ran it. +// RouteRecord is the route a role asked for and the one that ran it, and, +// when a runner ran it, both sides of the run (itd-166). type RouteRecord struct { // Asked is the route the configuration named: host or a runner. Asked string `json:"asked"` // Ran is the route that ran the role: host, a runner, or none. Ran string `json:"ran"` - // Model is the model the runner that ran it reported; "" on the host, - // whose own receipt reports it. + // Model is the model the runner that ran it reported, the canonical + // served model; "" on the host, whose own receipt reports it, and when + // the runner reported none. Model string `json:"model"` + // Sides is both sides of the run, present when a runner ran it; its + // fields are omitted when empty, so a host's or an older record reads as + // it always has. + Sides } // RoleReceipt is a role run's receipt block. Everything but Route is the diff --git a/internal/core/runner/runner.go b/internal/core/runner/runner.go index 19f04b5e94..32a59aa514 100644 --- a/internal/core/runner/runner.go +++ b/internal/core/runner/runner.go @@ -95,11 +95,23 @@ type Request struct { type Answer struct { // Text is the harness's final message. Text string - // Model is the model the harness reports, or the one it was asked for when - // it reports none. + // Model is the model the harness reports, "" when it reports none: the + // model the runner asked for is Sides.ModelAsked, and never stands in for + // the one that answered (itd-166, iss-2610161709506207). Model string // SessionID is the harness's own session id, "" when it reports none. SessionID string + // Sides is both sides of the run: what the runner asked for, and the + // token counts, cost and served models the harness's events report. + Sides + // counted is which token counts the harness's events carried, read + // with them, for the record's list of what it could not pin. + counted counted + // lost is which summed counts stopped being readable (a sum that would + // overflow), and costLost whether the summed cost did (a sum that is no + // longer finite): each stays unread for the rest of the run. + lost counted + costLost bool } // Runner is one route a role can run through. diff --git a/internal/core/runner/sides.go b/internal/core/runner/sides.go new file mode 100644 index 0000000000..de0365fd6b --- /dev/null +++ b/internal/core/runner/sides.go @@ -0,0 +1,125 @@ +package runner + +// sides.go is what a runner records of both sides of a run (itd-166): the +// model it passed the harness and the prompt it handed it, beside what the +// harness's events say answered, with the token counts and the cost they +// carry. The vocabulary (the fixed field lists, the four reasons, the usage +// and the cost) is internal/core/oracle's, shared with the provider adapter's +// record, so two runs compare field by field whatever routes they took. +// +// A harness computes the cost it reports from its own price table, so a +// runner's cost is listed as computed by the harness, never as the +// provider's. Nothing here asks the model about itself: every value is the +// runner's own or read from the harness's event stream. + +import ( + "crypto/sha256" + "encoding/hex" + + "github.com/intentdriven/abcd/internal/core/oracle" + "github.com/intentdriven/abcd/internal/termsafe" +) + +// maxHarnessText bounds one string a harness supplied to a record: a served +// model, provider, finish reason or version. +const maxHarnessText = 128 + +// maxServedModels bounds the served models one run records. +const maxServedModels = 16 + +// Sides is both sides of a run as a runner records them. Every string the +// harness supplied is cleaned and bounded (cleanHarness) before it is set. +type Sides struct { + // ModelAsked is the model the runner passed the harness, "" when it + // passed none and the harness chose. + ModelAsked string `json:"model_asked,omitempty"` + // ServedModels is every model the harness says served the run (claude's + // per-model usage), sorted; empty when it names none. + ServedModels []string `json:"served_models,omitempty"` + // ServedProvider is the provider the harness names as serving the + // canonical model, "" when it names none. + ServedProvider string `json:"served_provider,omitempty"` + // FinishReason is why the run's last turn stopped, as the harness says. + FinishReason string `json:"finish_reason,omitempty"` + // HarnessVersion is the harness's own version, as it reports it. + HarnessVersion string `json:"harness_version,omitempty"` + // PromptSHA256 is the SHA-256 of the prompt the runner handed the harness + // (prompt), never its text; the brief it names is the run's own file. + PromptSHA256 string `json:"prompt_sha256,omitempty"` + // Usage and Cost are the run's token counts and cost, nil when the + // harness reported none. Each count is the harness's own: claude's + // input_tokens and opencode's input exclude the cached prompt tokens, + // which are counted apart. + Usage *oracle.Usage `json:"usage,omitempty"` + Cost *oracle.Cost `json:"cost,omitempty"` + // Unpinned lists every field of the fixed lists the run could not pin. + Unpinned []oracle.Unpinned `json:"unpinned,omitempty"` +} + +// RecordOf is the route record of a role ran ran, routed to asked, whose +// harness answered ans: the served model as the canonical one, and both sides +// beside it. +func RecordOf(asked, ran string, ans Answer) RouteRecord { + return RouteRecord{Asked: asked, Ran: ran, Model: ans.Model, Sides: ans.Sides} +} + +// promptSHA256 is the hash of the prompt a route hands req's role. +func promptSHA256(req Request) string { + sum := sha256.Sum256([]byte(prompt(req))) + return hex.EncodeToString(sum[:]) +} + +// cleanHarness bounds a string a harness supplied and percent-encodes any +// hidden or control rune, so it is recorded but cannot reorder or escape the +// record it lands in. +func cleanHarness(s string) string { + s = termsafe.EncodeHiddenRunes(s) + if len(s) > maxHarnessText { + cut := maxHarnessText + for cut > 0 && s[cut]&0xC0 == 0x80 { + cut-- + } + s = s[:cut] + "..." + } + return s +} + +// askedUnpinned lists the asked side's fields neither command-line runner +// can pin: no provider pin, sampling setting, seed, effort or context variant +// is passed to a harness (each is the harness's or the owner's configuration), +// nor, when the runner passed none, a model. +func askedUnpinned(un *oracle.Unpin, s Sides) { + un.Unless(s.ModelAsked != "", oracle.AskedModel, oracle.NotConfigured) + for _, f := range []oracle.Field{oracle.AskedProviderPin, oracle.AskedTemperature, oracle.AskedTopP, + oracle.AskedSeed, oracle.AskedEffort, oracle.AskedContext} { + un.Add(f, oracle.NotConfigured) + } + un.Unless(s.PromptSHA256 != "", oracle.AskedPromptSHA256, oracle.NotReported) + un.Unless(s.HarnessVersion != "", oracle.AskedHarnessVersion, oracle.NotReported) +} + +// counted is which of a usage's counts the harness reported. +type counted struct { + prompt, completion, reasoning, cacheRead, cacheWrite bool +} + +// servedUnpinned lists the served side's fields the run could not pin, for a +// harness that reports the counts have names and computes the cost itself. +func servedUnpinned(un *oracle.Unpin, model string, s Sides, c counted) { + un.Unless(model != "", oracle.ServedModel, oracle.NotReported) + un.Unless(s.ServedProvider != "", oracle.ServedProvider, oracle.NotReported) + // Neither harness names a generation id for the run: a run is many + // generations, and their ids are not in the events read. + un.Add(oracle.ServedGenerationID, oracle.NotReported) + un.Unless(s.FinishReason != "", oracle.ServedFinishReason, oracle.NotReported) + un.Unless(c.prompt, oracle.ServedPromptTokens, oracle.NotReported) + un.Unless(c.completion, oracle.ServedCompletionTokens, oracle.NotReported) + un.Unless(c.reasoning, oracle.ServedReasoningTokens, oracle.NotReported) + un.Unless(c.cacheRead, oracle.ServedCacheReadTokens, oracle.NotReported) + un.Unless(c.cacheWrite, oracle.ServedCacheWriteTokens, oracle.NotReported) + if s.Cost != nil { + un.Add(oracle.ServedCost, oracle.HarnessComputed) + } else { + un.Add(oracle.ServedCost, oracle.NotReported) + } +} diff --git a/internal/core/runner/sides_test.go b/internal/core/runner/sides_test.go new file mode 100644 index 0000000000..ff5e7750cd --- /dev/null +++ b/internal/core/runner/sides_test.go @@ -0,0 +1,325 @@ +package runner + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "math" + "os" + "path/filepath" + "reflect" + "strconv" + "strings" + "testing" + + "github.com/intentdriven/abcd/internal/core/oracle" +) + +// The tests in this file are itd-166's runner half: a run records the model +// the runner passed beside what the harness says answered, with the token +// counts and cost its events carry, and lists every field of the fixed lists +// it could not pin with its reason. +// +// testdata/claude-stream-2.1.302.jsonl is a real stream: one call of the +// claude CLI 2.1.302 in print mode with --output-format stream-json +// --verbose, asking haiku to reply "ok". It was trimmed of what identifies the +// machine or the account (the hook events, the init event's paths, tools and +// plugins, the rate-limit details and the thinking signature) and its session +// id replaced; the usage, cost, model and version are as the harness sent +// them. +// +// testdata/opencode-run-1.18.31.jsonl is not a real stream: no opencode call +// was made for it. It is built from the shape opencode 1.18.31's run mode +// documents for `run --format json` (step_start, tool_use, reasoning, text and +// step_finish events, the step-finish part carrying reason, snapshot, cost and +// tokens{total,input,output,reasoning,cache{read,write}}), and like the real +// one it names no model. + +// reasons indexes a record's list by field. +func reasons(us []oracle.Unpinned) map[oracle.Field]oracle.UnpinnedReason { + m := map[oracle.Field]oracle.UnpinnedReason{} + for _, u := range us { + m[u.Field] = u.Reason + } + return m +} + +// assertFixedLists holds every listed field to the fixed lists, once each. +func assertFixedLists(t *testing.T, us []oracle.Unpinned) { + t.Helper() + known := map[oracle.Field]bool{} + for _, f := range append(append([]oracle.Field{}, oracle.AskedFields...), oracle.ServedFields...) { + known[f] = true + } + seen := map[oracle.Field]bool{} + for _, u := range us { + if !known[u.Field] || seen[u.Field] { + t.Errorf("listed field %q is off the fixed lists or listed twice: %v", u.Field, us) + } + seen[u.Field] = true + } +} + +func streamFixture(t *testing.T, name string) string { + t.Helper() + p, err := filepath.Abs(filepath.Join("testdata", name)) + if err != nil { + t.Fatal(err) + } + t.Setenv(fakeStreamEnv, p) + return p +} + +func promptHash(req Request) string { + sum := sha256.Sum256([]byte(prompt(req))) + return hex.EncodeToString(sum[:]) +} + +// TestClaudeRecordsBothSidesFromARealStream is criterion 2: the run's answer +// carries the asked model the runner passed, the served models the result +// event's per-model usage names, its token counts and its cost (computed by +// the harness, and listed so), and the harness version from the init event. +func TestClaudeRecordsBothSidesFromARealStream(t *testing.T) { + f := newFake(t, "stream", Claude) + streamFixture(t, "claude-stream-2.1.302.jsonl") + req := f.request("ruthless-reviewer") + ans, _, err := newClaude("anthropic/haiku").Run(context.Background(), req) + if err != nil { + t.Fatalf("run: %v", err) + } + if ans.Model != "claude-haiku-5-5" || ans.ModelAsked != "haiku" || + !reflect.DeepEqual(ans.ServedModels, []string{"claude-haiku-5-5"}) { + t.Fatalf("models: served %q (%v), asked %q", ans.Model, ans.ServedModels, ans.ModelAsked) + } + if ans.HarnessVersion != "2.1.302" || ans.FinishReason != "end_turn" || ans.ServedProvider != "firstParty" { + t.Fatalf("version %q, finish %q, provider %q", ans.HarnessVersion, ans.FinishReason, ans.ServedProvider) + } + wantUsage := oracle.Usage{PromptTokens: 2, CompletionTokens: 76, ReasoningTokens: 72, CacheReadTokens: 11228, CacheWriteTokens: 18650} + if ans.Usage == nil || *ans.Usage != wantUsage { + t.Fatalf("usage = %+v, want %+v", ans.Usage, wantUsage) + } + if ans.Cost == nil || ans.Cost.Unit != oracle.CostUSD || ans.Cost.Amount < 0.00388 || ans.Cost.Amount > 0.00389 { + t.Fatalf("cost = %+v", ans.Cost) + } + if ans.PromptSHA256 != promptHash(req) { + t.Fatalf("prompt hash = %q, want the prompt's", ans.PromptSHA256) + } + got := reasons(ans.Unpinned) + want := map[oracle.Field]oracle.UnpinnedReason{ + oracle.AskedProviderPin: oracle.NotConfigured, oracle.AskedTemperature: oracle.NotConfigured, + oracle.AskedTopP: oracle.NotConfigured, oracle.AskedSeed: oracle.NotConfigured, + oracle.AskedEffort: oracle.NotConfigured, oracle.AskedContext: oracle.NotConfigured, + oracle.ServedGenerationID: oracle.NotReported, oracle.ServedCost: oracle.HarnessComputed, + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("unpinned = %v, want %v", got, want) + } + assertFixedLists(t, ans.Unpinned) +} + +// TestClaudeWithoutUsageListsEachMissing is criterion 4 on the claude +// runner: a stream that carries no usage, cost, version or stop reason lists +// each of them as not reported, and a run passed no model lists the asked +// model as not configured. +func TestClaudeWithoutUsageListsEachMissing(t *testing.T) { + f := newFake(t, "ok", Claude) + ans, _, err := newClaude("").Run(context.Background(), f.request("ruthless-reviewer")) + if err != nil { + t.Fatal(err) + } + if ans.Usage != nil || ans.Cost != nil || ans.ModelAsked != "" { + t.Fatalf("answer = %+v", ans) + } + got := reasons(ans.Unpinned) + for f, r := range map[oracle.Field]oracle.UnpinnedReason{ + oracle.AskedModel: oracle.NotConfigured, oracle.AskedHarnessVersion: oracle.NotReported, + oracle.ServedPromptTokens: oracle.NotReported, oracle.ServedCompletionTokens: oracle.NotReported, + oracle.ServedReasoningTokens: oracle.NotReported, oracle.ServedCacheReadTokens: oracle.NotReported, + oracle.ServedCacheWriteTokens: oracle.NotReported, oracle.ServedCost: oracle.NotReported, + oracle.ServedFinishReason: oracle.NotReported, oracle.ServedProvider: oracle.NotReported, + } { + if got[f] != r { + t.Errorf("%s = %q, want %q", f, got[f], r) + } + } + assertFixedLists(t, ans.Unpinned) +} + +// TestOpenCodeNeverStampsTheAskedModelAsServed is criterion 3 and +// iss-2610161709506207: opencode's run mode names no model, so the served +// model is empty and listed as not reported, never a copy of the asked one, +// and the token counts and cost are summed across its step-finish events. +func TestOpenCodeNeverStampsTheAskedModelAsServed(t *testing.T) { + f := newFake(t, "stream", OpenCode) + streamFixture(t, "opencode-run-1.18.31.jsonl") + req := f.request("ruthless-reviewer") + ans, _, err := newOpenCode("local/qwen3-coder").Run(context.Background(), req) + if err != nil { + t.Fatalf("run: %v", err) + } + if ans.Model != "" || ans.ModelAsked != "local/qwen3-coder" || len(ans.ServedModels) != 0 { + t.Fatalf("served %q (%v), asked %q: the asked model must never stand in for the served one", ans.Model, ans.ServedModels, ans.ModelAsked) + } + wantUsage := oracle.Usage{PromptTokens: 1600, CompletionTokens: 300, ReasoningTokens: 50, CacheReadTokens: 350, CacheWriteTokens: 100} + if ans.Usage == nil || *ans.Usage != wantUsage { + t.Fatalf("usage = %+v, want %+v", ans.Usage, wantUsage) + } + if ans.Cost == nil || ans.Cost.Unit != oracle.CostUSD || ans.Cost.Amount < 0.001999 || ans.Cost.Amount > 0.002001 { + t.Fatalf("cost = %+v, want 0.002 usd", ans.Cost) + } + if ans.FinishReason != "stop" || ans.Text != "done" || ans.PromptSHA256 != promptHash(req) { + t.Fatalf("finish %q, text %q, hash %q", ans.FinishReason, ans.Text, ans.PromptSHA256) + } + got := reasons(ans.Unpinned) + for f, r := range map[oracle.Field]oracle.UnpinnedReason{ + oracle.ServedModel: oracle.NotReported, oracle.ServedProvider: oracle.NotReported, + oracle.ServedGenerationID: oracle.NotReported, oracle.ServedCost: oracle.HarnessComputed, + oracle.AskedHarnessVersion: oracle.NotReported, + } { + if got[f] != r { + t.Errorf("%s = %q, want %q", f, got[f], r) + } + } + if _, listed := got[oracle.AskedModel]; listed { + t.Errorf("the asked model the runner passed is listed: %v", ans.Unpinned) + } + assertFixedLists(t, ans.Unpinned) +} + +// TestOpenCodeWithoutStepUsageListsItNotReported: a stream whose steps carry +// no tokens and no cost lists every count and the cost as not reported. +func TestOpenCodeWithoutStepUsageListsItNotReported(t *testing.T) { + f := newFake(t, "ok", OpenCode) + ans, _, err := newOpenCode("").Run(context.Background(), f.request("ruthless-reviewer")) + if err != nil { + t.Fatal(err) + } + if ans.Usage != nil || ans.Cost != nil || ans.Model != "" { + t.Fatalf("answer = %+v", ans) + } + got := reasons(ans.Unpinned) + for _, f := range []oracle.Field{oracle.ServedPromptTokens, oracle.ServedCost, oracle.ServedModel, oracle.ServedFinishReason} { + if got[f] != oracle.NotReported { + t.Errorf("%s = %q, want not_reported", f, got[f]) + } + } + if got[oracle.AskedModel] != oracle.NotConfigured { + t.Errorf("asked model with none passed = %q", got[oracle.AskedModel]) + } +} + +// TestTheRouteRecordKeepsTheAskedModelBesideTheServedOne: a dispatch through +// a runner puts both sides of its answer on the role's route record. +func TestTheRouteRecordKeepsTheAskedModelBesideTheServedOne(t *testing.T) { + f := newFake(t, "stream", Claude, OpenCode) + streamFixture(t, "opencode-run-1.18.31.jsonl") + h := newHarness(t, mustLoad(t, routed(routedMachine), ""), true) + out, err := h.d.Dispatch(context.Background(), f.request("ruthless-reviewer")) + if err != nil { + t.Fatal(err) + } + r := out.Receipt.Route + if r.Ran != OpenCode || r.Model != "" || r.ModelAsked != "local/qwen3-coder" || r.Usage == nil || r.Cost == nil || len(r.Unpinned) == 0 { + t.Fatalf("route = %+v", r) + } + if !reflect.DeepEqual(RecordOf(OpenCode, OpenCode, *out.Answer), r) { + t.Fatalf("RecordOf = %+v, want the route the dispatch recorded %+v", RecordOf(OpenCode, OpenCode, *out.Answer), r) + } +} + +// TestHarnessStringsAreCleanedAndBounded: a served model, provider, finish +// reason or version the harness supplied is percent-encoded of hidden and +// control runes and bounded before it reaches a record. +func TestHarnessStringsAreCleanedAndBounded(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "stream.jsonl") + long := make([]byte, 400) + for i := range long { + long[i] = 'v' + } + stream := `{"type":"system","subtype":"init","session_id":"s1","model":"claude-x","claude_code_version":"2.1\u001b[2J` + string(long) + `"}` + "\n" + + `{"type":"result","subtype":"success","is_error":false,"result":"ok","session_id":"s1","stop_reason":"end‮turn",` + + `"modelUsage":{"claude-x\u0007":{"provider":"first​Party"}}}` + "\n" + if err := os.WriteFile(p, []byte(stream), 0o600); err != nil { + t.Fatal(err) + } + ans, err := parseClaude([]byte(stream)) + if err != nil { + t.Fatal(err) + } + vals := append([]string{ans.HarnessVersion, ans.FinishReason, ans.ServedProvider}, ans.ServedModels...) + for _, v := range vals { + for _, r := range v { + if r < 0x20 || r == 0x7f || r == '‮' || r == '​' { + t.Errorf("%q carries rune %U", v, r) + } + } + if len(v) > maxHarnessText+len("...") { + t.Errorf("%q is %d bytes, over the bound", v, len(v)) + } + } +} + +// servedReasons is the served side's list for ans, as a run builds it. +func servedReasons(ans Answer) map[oracle.Field]oracle.UnpinnedReason { + var un oracle.Unpin + servedUnpinned(&un, ans.Model, ans.Sides, ans.counted) + return reasons(un.List()) +} + +// TestOpenCodeSumsPastTheirBoundAreNotReported: a cost summed past the +// largest float is not finite, so it is dropped and listed as not reported +// (and the state that records it still writes); a token sum that would +// overflow is unreadable, listed as not reported, and a later step does not +// bring it back. +func TestOpenCodeSumsPastTheirBoundAreNotReported(t *testing.T) { + maxInt := strconv.Itoa(math.MaxInt) + stream := strings.Join([]string{ + `{"type":"step_finish","part":{"reason":"tool-calls","cost":1e308,"tokens":{"input":` + maxInt + `,"output":5}}}`, + `{"type":"step_finish","part":{"reason":"tool-calls","cost":1e308,"tokens":{"input":1,"output":6}}}`, + `{"type":"step_finish","part":{"reason":"stop","cost":0.5,"tokens":{"input":3,"output":1}}}`, + `{"type":"text","part":{"type":"text","text":"done"}}`, + }, "\n") + ans, err := parseOpenCode([]byte(stream)) + if err != nil { + t.Fatal(err) + } + if ans.Cost != nil { + t.Fatalf("cost = %+v, want it dropped once its sum stopped being finite", ans.Cost) + } + if ans.Usage == nil || ans.Usage.PromptTokens != 0 || ans.Usage.CompletionTokens != 12 { + t.Fatalf("usage = %+v, want the prompt sum dropped and the completion sum kept", ans.Usage) + } + got := servedReasons(ans) + if got[oracle.ServedCost] != oracle.NotReported || got[oracle.ServedPromptTokens] != oracle.NotReported { + t.Fatalf("cost %q, prompt %q: want both not_reported", got[oracle.ServedCost], got[oracle.ServedPromptTokens]) + } + if _, listed := got[oracle.ServedCompletionTokens]; listed { + t.Fatalf("the completion sum is listed: %v", got) + } + if _, err := json.Marshal(RecordOf("opencode", "opencode", ans)); err != nil { + t.Fatalf("the record does not write: %v", err) + } +} + +// TestClaudeNegativeCountIsNotReported: a count the harness sends below zero +// is unreadable, listed as not reported, never recorded as a zero it +// reported; the counts beside it stand. +func TestClaudeNegativeCountIsNotReported(t *testing.T) { + stream := strings.Join([]string{ + `{"type":"system","subtype":"init","model":"claude-haiku-4-5","session_id":"s"}`, + `{"type":"result","subtype":"success","result":"ok","total_cost_usd":-1,"usage":{"input_tokens":-5,"output_tokens":3}}`, + }, "\n") + ans, err := parseClaude([]byte(stream)) + if err != nil { + t.Fatal(err) + } + got := servedReasons(ans) + if got[oracle.ServedPromptTokens] != oracle.NotReported || got[oracle.ServedCost] != oracle.NotReported { + t.Fatalf("prompt %q, cost %q: want both not_reported", got[oracle.ServedPromptTokens], got[oracle.ServedCost]) + } + if _, listed := got[oracle.ServedCompletionTokens]; listed || ans.Usage == nil || ans.Usage.CompletionTokens != 3 { + t.Fatalf("usage %+v, list %v", ans.Usage, got) + } +} diff --git a/internal/core/runner/testdata/claude-stream-2.1.302.jsonl b/internal/core/runner/testdata/claude-stream-2.1.302.jsonl new file mode 100644 index 0000000000..038251f7b9 --- /dev/null +++ b/internal/core/runner/testdata/claude-stream-2.1.302.jsonl @@ -0,0 +1,7 @@ +{"type":"system","subtype":"init","session_id":"00000000-0000-4000-8000-000000000166","model":"claude-haiku-5-5","permissionMode":"dontAsk","apiKeySource":"none","claude_code_version":"2.1.302","output_style":"default","fast_mode_state":"off","per_turn_effort_active":true,"uuid":"29c11f57-e6c5-4a05-9129-94e4ce87ff48"} +{"type":"system","subtype":"thinking_tokens","estimated_tokens":50,"estimated_tokens_delta":50,"session_id":"00000000-0000-4000-8000-000000000166","uuid":"d97f6465-40ca-4eba-b11a-3ce89c5f3f6e"} +{"type":"system","subtype":"thinking_tokens","estimated_tokens":210,"estimated_tokens_delta":160,"session_id":"00000000-0000-4000-8000-000000000166","uuid":"9b765bd6-19e1-4286-b188-ea19c69fb043"} +{"type":"assistant","message":{"model":"claude-haiku-5-5","id":"msg_011Cg7NkEg3KfQ3FqqpgSKCN","type":"message","role":"assistant","content":[{"type":"thinking","thinking":"","signature":"[stripped]"}],"container":null,"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":18650,"cache_read_input_tokens":11228,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":18650},"output_tokens":4,"service_tier":"standard","inference_geo":"global","speed":"standard"},"input_transformations":[],"diagnostics":null,"context_management":null},"parent_tool_use_id":null,"session_id":"00000000-0000-4000-8000-000000000166","uuid":"a6be9c91-0e7f-4298-b2a7-53217cc324db","timestamp":"2026-10-17T06:58:38.213Z","request_id":"req_011Cg7NkETdj1BULWKK4CoxQ","thinking_duration_ms":432,"thinking_display":"updates"} +{"type":"assistant","message":{"model":"claude-haiku-5-5","id":"msg_011Cg7NkEg3KfQ3FqqpgSKCN","type":"message","role":"assistant","content":[{"type":"text","text":"ok"}],"container":null,"stop_reason":null,"stop_sequence":null,"stop_details":null,"usage":{"input_tokens":2,"cache_creation_input_tokens":18650,"cache_read_input_tokens":11228,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":18650},"output_tokens":4,"service_tier":"standard","inference_geo":"global","speed":"standard"},"input_transformations":[],"diagnostics":null,"context_management":null},"parent_tool_use_id":null,"session_id":"00000000-0000-4000-8000-000000000166","uuid":"8a7a23c3-92eb-4f11-a3dd-dc41005b663d","timestamp":"2026-10-17T06:58:38.213Z","request_id":"req_011Cg7NkETdj1BULWKK4CoxQ"} +{"type":"rate_limit_event","rate_limit_info":{"status":"allowed","rateLimitType":"five_hour"},"session_id":"00000000-0000-4000-8000-000000000166","uuid":"7fc4372b-5186-45f2-9215-68ebbee9e7a0"} +{"duration_api_ms":1688,"stop_reason":"end_turn","session_id":"00000000-0000-4000-8000-000000000166","total_cost_usd":0.0038804800000000004,"usage":{"input_tokens":2,"cache_creation_input_tokens":18650,"cache_read_input_tokens":11228,"output_tokens":76,"output_tokens_details":{"thinking_tokens":72},"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":18650,"ephemeral_5m_input_tokens":0},"inference_geo":"global","iterations":[{"input_tokens":2,"output_tokens":76,"cache_read_input_tokens":11228,"cache_creation_input_tokens":18650,"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":18650},"type":"message"}],"speed":"standard","fallback_credit":null},"modelUsage":{"claude-haiku-5-5":{"inputTokens":2,"outputTokens":76,"cacheReadInputTokens":11228,"cacheCreationInputTokens":18650,"webSearchRequests":0,"costUSD":0.0038804800000000004,"contextWindow":1000000,"maxOutputTokens":128000,"thinkingTokens":72,"canonicalModel":"claude-haiku-5-5","provider":"firstParty","costBasis":"list"}},"permission_denials":[],"terminal_reason":"completed","fast_mode_state":"off","fast_mode_disabled_reason":"sdk_opt_in_required","subagent_stats":{"spawned":0,"requested":{"background":0,"foreground":0,"unset":0},"started_in_background":0,"max_depth":0,"spawned_by_subagents":0,"completed":0,"failed":0,"killed":{"parent":0,"user":0,"system":0},"refused":{"depth_limit":0,"concurrency_limit":0,"budget":0},"by_type":{}},"safety_stops":0,"is_error":false,"num_turns":1,"subtype":"success","api_error_status":null,"result":"ok","ttft_ms":1850,"type":"result","duration_ms":1869,"uuid":"46b523a4-4cb4-483e-a90a-01648eddb993","ttft_stream_ms":1418,"time_to_request_ms":185,"first_content_frame_ms":1418,"queued_turn_count":0,"result_index":0} diff --git a/internal/core/runner/testdata/opencode-run-1.18.31.jsonl b/internal/core/runner/testdata/opencode-run-1.18.31.jsonl new file mode 100644 index 0000000000..d5e86526e3 --- /dev/null +++ b/internal/core/runner/testdata/opencode-run-1.18.31.jsonl @@ -0,0 +1,7 @@ +{"type":"step_start","timestamp":1792200000000,"sessionID":"ses_fake166","part":{"id":"prt_1","sessionID":"ses_fake166","messageID":"msg_1","type":"step-start","snapshot":"4b825dc642cb6eb9a060e54bf8d69288fbee4904"}} +{"type":"tool_use","timestamp":1792200001000,"sessionID":"ses_fake166","part":{"id":"prt_2","sessionID":"ses_fake166","messageID":"msg_1","type":"tool","tool":"read","state":{"status":"completed"}}} +{"type":"step_finish","timestamp":1792200002000,"sessionID":"ses_fake166","part":{"id":"prt_3","sessionID":"ses_fake166","messageID":"msg_1","type":"step-finish","reason":"tool-calls","snapshot":"4b825dc642cb6eb9a060e54bf8d69288fbee4904","cost":0.0012,"tokens":{"total":1500,"input":1000,"output":200,"reasoning":50,"cache":{"read":200,"write":50}}}} +{"type":"step_start","timestamp":1792200003000,"sessionID":"ses_fake166","part":{"id":"prt_4","sessionID":"ses_fake166","messageID":"msg_2","type":"step-start","snapshot":"4b825dc642cb6eb9a060e54bf8d69288fbee4904"}} +{"type":"reasoning","timestamp":1792200004000,"sessionID":"ses_fake166","part":{"id":"prt_5","sessionID":"ses_fake166","messageID":"msg_2","type":"reasoning","text":"the receipt is written"}} +{"type":"text","timestamp":1792200005000,"sessionID":"ses_fake166","part":{"id":"prt_6","sessionID":"ses_fake166","messageID":"msg_2","type":"text","text":"done"}} +{"type":"step_finish","timestamp":1792200006000,"sessionID":"ses_fake166","part":{"id":"prt_7","sessionID":"ses_fake166","messageID":"msg_2","type":"step-finish","reason":"stop","snapshot":"4b825dc642cb6eb9a060e54bf8d69288fbee4904","cost":0.0008,"tokens":{"total":900,"input":600,"output":100,"reasoning":0,"cache":{"read":150,"write":50}}}} diff --git a/internal/core/surface/sentences.go b/internal/core/surface/sentences.go index a241fe6c3c..542f48be84 100644 --- a/internal/core/surface/sentences.go +++ b/internal/core/surface/sentences.go @@ -31,7 +31,7 @@ var sentences = map[string]string{ "abcd ahoy": "Detect abcd's install state and list its gaps, or report one mode a flag names: " + "Writes nothing; refuses any argument or two modes at once.", - "abcd ahoy connect": "Verify a provider with one call and configure it, or guide setup: " + + "abcd ahoy connect": "Verify a provider and configure it, or guide setup: " + "Writes " + abcdhome.Display() + "/ and the key's home, nothing guided; refuses no --model off a terminal.", "abcd ahoy credential": "List the credentials abcd reads, explain one, or verify and store it: " + "Writes the chosen home only with --home; refuses a value the adapter's call fails.", diff --git a/internal/surface/cli/ahoy_connect.go b/internal/surface/cli/ahoy_connect.go index bf05491d3b..6f2e2c1237 100644 --- a/internal/surface/cli/ahoy_connect.go +++ b/internal/surface/cli/ahoy_connect.go @@ -178,7 +178,7 @@ func keyState(home, name string) (state, from string) { // newAhoyConnectCommand builds `ahoy connect `. func newAhoyConnectCommand(asJSON *bool) *cobra.Command { var baseURL, home, keyName, resume, answer string - var guide bool + var guide, streamUsage bool var models []string var ptr credential.Pointer cmd := &cobra.Command{ @@ -211,6 +211,11 @@ func newAhoyConnectCommand(asJSON *bool) *cobra.Command { "or run the command in a terminal to pick one from the models the service lists"} } req := oracle.ConnectRequest{Roots: roots, Provider: args[0], BaseURL: baseURL, Models: models, Home: home, KeyName: keyName, Pointer: ptr} + // Unset, the setup asks for usage in the stream and settles the + // block's stream_usage itself when the server refuses it. + if cmd.Flags().Changed("stream-usage") { + req.StreamUsage = &streamUsage + } if len(models) == 0 { req.Pick = connectPick(cmd, roots, baseURL) } @@ -247,6 +252,12 @@ func newAhoyConnectCommand(asJSON *bool) *cobra.Command { line := func(s string) { fmt.Fprintf(w, " %s\n", termsafe.Sanitize(s)) } fmt.Fprintf(w, "abcd ahoy connect — %s verified and configured\n", termsafe.Sanitize(res.Provider)) line(fmt.Sprintf("verified: asked %s, %s reported %s", res.Verified.ModelAsked, res.Verified.Provider, res.Verified.ModelReported)) + switch { + case res.StreamUsageNote != "": + line("stream_usage: false, because " + res.StreamUsageNote) + case res.StreamUsage != nil: + line(fmt.Sprintf("stream_usage: %t, as given", *res.StreamUsage)) + } for _, p := range res.Wrote { line("wrote: " + p) } @@ -268,6 +279,8 @@ func newAhoyConnectCommand(asJSON *bool) *cobra.Command { "omitted at a terminal, the service's models are listed with the key and you pick one") cmd.Flags().StringVar(&home, "home", "", "where the key lives: external (--env, or --file and --field) | abcd (read from stdin, hidden at a terminal, into the owner-only "+credential.StorePath+") | keychain (read from stdin, hidden at a terminal, into the platform keychain) | none (a server that takes no key)") cmd.Flags().StringVar(&keyName, "key", "", "the credential's name (default: the provider's name)") + cmd.Flags().BoolVar(&streamUsage, "stream-usage", true, "whether calls ask the server for usage in the stream ("+openaiapi.StreamUsageField+"), written into the block as stream_usage; "+ + "omitted, the verification asks, and a server that refuses it is asked once more without it and the block is written with false") pointerFlags(cmd, &ptr) cmd.Flags().BoolVar(&guide, "guide", false, "work the values out one question a turn and print the command to paste into a terminal; writes nothing, and the provider name is optional") cmd.Flags().StringVar(&resume, "resume", "", "with --guide: the last turn's resume object, or - to read it from stdin") diff --git a/internal/surface/cli/ahoy_connect_test.go b/internal/surface/cli/ahoy_connect_test.go index 013aa95f8f..d86ba50fc6 100644 --- a/internal/surface/cli/ahoy_connect_test.go +++ b/internal/surface/cli/ahoy_connect_test.go @@ -313,3 +313,76 @@ func TestBareAhoyNamesASkippedRoute(t *testing.T) { } } } + +// usageRefusingProvider refuses with HTTP 400 every chat completion that asks +// for usage in the stream, and answers reply to one that does not, counting +// the chat requests. +func usageRefusingProvider(t *testing.T, reply string) (base string, calls *atomic.Int32) { + t.Helper() + calls = &atomic.Int32{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet { + http.NotFound(w, r) + return + } + calls.Add(1) + raw, _ := io.ReadAll(r.Body) + w.Header().Set("Content-Type", "application/json") + if bytes.Contains(raw, []byte("stream_options")) { + w.WriteHeader(http.StatusBadRequest) + _, _ = io.WriteString(w, `{"error":{"message":"Unrecognized request argument supplied: stream_options"}}`) + return + } + _, _ = io.WriteString(w, reply) + })) + t.Cleanup(srv.Close) + return srv.URL + "/v1", calls +} + +// TestAhoyConnectSaysItTurnedStreamUsageOff is iss-2610170934491794 through +// the front door: a server that refuses the usage request is verified asked +// again without it, and the setup says, in the text and the JSON form, that +// the block it wrote sets stream_usage to false. +func TestAhoyConnectSaysItTurnedStreamUsageOff(t *testing.T) { + hermeticEnv(t) + t.Chdir(t.TempDir()) + base, calls := usageRefusingProvider(t, completionReply("qwen/qwen3-8b")) + out, err := runCLIErr(t, "ahoy", "connect", "local", "--base-url", base, "--model", "qwen/qwen3-8b", "--home", "none") + if err != nil { + t.Fatalf("ahoy connect: %v\n%s", err, out) + } + if calls.Load() != 2 || !strings.Contains(string(out), "stream_usage to false") { + t.Fatalf("ahoy connect (%d call(s)):\n%s", calls.Load(), out) + } + base, _ = usageRefusingProvider(t, completionReply("qwen/qwen3-8b")) + jout, err := runCLIErr(t, "ahoy", "connect", "desk", "--base-url", base, "--model", "qwen/qwen3-8b", "--home", "none", "--json") + if err != nil { + t.Fatalf("ahoy connect --json: %v\n%s", err, jout) + } + var v struct { + StreamUsage *bool `json:"stream_usage"` + Note string `json:"stream_usage_note"` + } + if err := json.Unmarshal(jout, &v); err != nil || v.StreamUsage == nil || *v.StreamUsage || v.Note == "" { + t.Fatalf("--json = %s (%v)", jout, err) + } +} + +// TestAhoyConnectTakesStreamUsage: --stream-usage=false sends the +// verification without the usage request and writes the block's setting. +func TestAhoyConnectTakesStreamUsage(t *testing.T) { + hermeticEnv(t) + t.Chdir(t.TempDir()) + base, calls := usageRefusingProvider(t, completionReply("qwen/qwen3-8b")) + out, err := runCLIErr(t, "ahoy", "connect", "local", "--base-url", base, "--model", "qwen/qwen3-8b", "--home", "none", "--stream-usage=false") + if err != nil { + t.Fatalf("ahoy connect: %v\n%s", err, out) + } + if calls.Load() != 1 || !strings.Contains(string(out), "stream_usage: false") { + t.Fatalf("ahoy connect (%d call(s)):\n%s", calls.Load(), out) + } + raw, _ := os.ReadFile(abcdhome.Path(os.Getenv("HOME"), "config.json")) + if !strings.Contains(string(raw), `"stream_usage": false`) { + t.Fatalf("config.json:\n%s", raw) + } +} diff --git a/internal/surface/cli/build.go b/internal/surface/cli/build.go index 64460c7a19..304d640c0d 100644 --- a/internal/surface/cli/build.go +++ b/internal/surface/cli/build.go @@ -10,6 +10,7 @@ import ( "os/signal" "path/filepath" "sort" + "strconv" "strings" "syscall" "time" @@ -19,6 +20,7 @@ import ( "github.com/intentdriven/abcd/internal/core/implement" "github.com/intentdriven/abcd/internal/core/implement/loop" "github.com/intentdriven/abcd/internal/core/layered" + "github.com/intentdriven/abcd/internal/core/oracle" "github.com/intentdriven/abcd/internal/core/runner" "github.com/intentdriven/abcd/internal/fsutil" "github.com/intentdriven/abcd/internal/gitutil" @@ -793,7 +795,7 @@ func renderStepResult(w io.Writer, verb string, res loop.StepResult) { fmt.Fprintf(w, "%s: %s is complete\n", verb, res.RunID) } if r := res.Route; r != nil { - fmt.Fprintf(w, " ran on: the %s runner (asked %s; model %s)\n", termsafe.Sanitize(r.Ran), termsafe.Sanitize(r.Asked), termsafe.Sanitize(modelOrNone(r.Model))) + fmt.Fprintf(w, " ran on: the %s runner (%s)\n", termsafe.Sanitize(r.Ran), routeDetail(r)) } if fb := res.Fallback; fb != nil { fmt.Fprintf(w, " fallback: %s was routed to %s, which was %s (%s); %s runs it\n", termsafe.Sanitize(fb.Role), termsafe.Sanitize(fb.Asked), @@ -1315,7 +1317,28 @@ func routeSuffix(r *runner.RouteRecord) string { if r == nil { return "" } - return fmt.Sprintf(" — ran on the %s runner (asked %s; model %s)", termsafe.Sanitize(r.Ran), termsafe.Sanitize(r.Asked), termsafe.Sanitize(modelOrNone(r.Model))) + return fmt.Sprintf(" — ran on the %s runner (%s)", termsafe.Sanitize(r.Ran), routeDetail(r)) +} + +// routeDetail is a runner route's parenthesis: the route asked for and the +// model that served, then the model the runner asked for and the cost when +// the record carries them (itd-166). The token counts and the fields the run +// could not pin stay in the record, for a reader tracing a run back. +func routeDetail(r *runner.RouteRecord) string { + d := fmt.Sprintf("asked %s; model %s", termsafe.Sanitize(r.Asked), termsafe.Sanitize(modelOrNone(r.Model))) + if r.ModelAsked != "" { + d += "; model asked " + termsafe.Sanitize(r.ModelAsked) + } + if r.Cost != nil { + d += "; cost " + costText(*r.Cost) + } + return d +} + +// costText is a recorded cost as a route line names it: the amount as +// recorded, and its unit. +func costText(c oracle.Cost) string { + return strconv.FormatFloat(c.Amount, 'f', -1, 64) + " " + termsafe.Sanitize(c.Unit) } func modelOrNone(m string) string { diff --git a/internal/surface/cli/dispatch.go b/internal/surface/cli/dispatch.go index 878f46c7bd..bb703757fa 100644 --- a/internal/surface/cli/dispatch.go +++ b/internal/surface/cli/dispatch.go @@ -111,8 +111,14 @@ func withDispatchReceipt(v any, d *dispatched) any { func renderDispatchLine(w interface{ Write([]byte) (int, error) }, d *dispatched) { rc := d.receipt line := fmt.Sprintf("route: %s ran on %s", rc.Agent, rc.ConnectionUsed) - if rc.ProviderCall != nil { - line += fmt.Sprintf(", model asked %s, model reported %s", rc.ProviderCall.ModelAsked, rc.ProviderCall.ModelReported) + if pc := rc.ProviderCall; pc != nil { + line += fmt.Sprintf(", model asked %s, model reported %s", pc.ModelAsked, pc.ModelReported) + if pc.ServedProvider != "" { + line += ", served by " + pc.ServedProvider + } + if pc.Cost != nil { + line += ", cost " + costText(*pc.Cost) + } } fmt.Fprintf(w, " %s\n", termsafe.Sanitize(line)) } diff --git a/internal/surface/cli/route_audit_test.go b/internal/surface/cli/route_audit_test.go index ec3346ee48..e2aef2ae02 100644 --- a/internal/surface/cli/route_audit_test.go +++ b/internal/surface/cli/route_audit_test.go @@ -94,7 +94,8 @@ func TestIntentAuditIngestCarriesTheReceipt(t *testing.T) { var rc oracle.ReceiptRoute member(t, []byte(stdout), "route", &rc) want := oracle.ReceiptRoute{Agent: "intent-auditor", TierAsked: oracle.Economy, ConnectionUsed: oracle.Harness, - FallbackReason: rc.FallbackReason, Override: "intent-auditor=economy", SettingsSent: oracle.Settings{}} + FallbackReason: rc.FallbackReason, Override: "intent-auditor=economy", SettingsSent: oracle.Settings{}, + Unpinned: oracle.HandOffUnpinnedForTest("")} if rc.FallbackReason == "" || !receiptEqual(rc, want) { t.Fatalf("route = %+v", rc) } diff --git a/internal/surface/cli/route_reading_test.go b/internal/surface/cli/route_reading_test.go index 183da98fa8..b7c4fa7aab 100644 --- a/internal/surface/cli/route_reading_test.go +++ b/internal/surface/cli/route_reading_test.go @@ -47,7 +47,7 @@ func TestReadingIngestCarriesTheReceipt(t *testing.T) { member(t, []byte(stdout), "route", &rc) want := oracle.ReceiptRoute{Agent: "cold-reading-detection", TierAsked: oracle.Economy, ConnectionUsed: oracle.Harness, FallbackReason: rc.FallbackReason, Override: "cold-reading-detection=economy", SettingsSent: oracle.Settings{}, - ModelReported: "a-model"} + ModelReported: "a-model", Unpinned: oracle.HandOffUnpinnedForTest("a-model")} if rc.FallbackReason == "" || !receiptEqual(rc, want) { t.Fatalf("route = %+v", rc) } diff --git a/internal/surface/cli/route_test.go b/internal/surface/cli/route_test.go index 2a6e5d7cd3..7773a088a9 100644 --- a/internal/surface/cli/route_test.go +++ b/internal/surface/cli/route_test.go @@ -161,7 +161,7 @@ func TestNothingAcceptedReceiptIsHostDecidesOnTheHarness(t *testing.T) { var rc oracle.ReceiptRoute member(t, []byte(stdout), "route", &rc) want := oracle.ReceiptRoute{Agent: "principle-distiller", TierAsked: oracle.HostDecides, - ConnectionUsed: oracle.Harness, SettingsSent: oracle.Settings{}} + ConnectionUsed: oracle.Harness, SettingsSent: oracle.Settings{}, Unpinned: oracle.HandOffUnpinnedForTest("")} if !receiptEqual(rc, want) { t.Fatalf("route = %+v, want %+v", rc, want) } @@ -224,7 +224,7 @@ func TestDisembarkIngestsCarryTheReceipt(t *testing.T) { var rc oracle.ReceiptRoute member(t, []byte(stdout), "route", &rc) want := oracle.ReceiptRoute{Agent: v.agent, TierAsked: oracle.Local, ConnectionUsed: oracle.Harness, // abcd-lint:allow - FallbackReason: rc.FallbackReason, SettingsSent: oracle.Settings{}} + FallbackReason: rc.FallbackReason, SettingsSent: oracle.Settings{}, Unpinned: oracle.HandOffUnpinnedForTest("")} if rc.FallbackReason == "" || !receiptEqual(rc, want) { t.Fatalf("route = %+v", rc) } diff --git a/internal/surface/cli/sides_test.go b/internal/surface/cli/sides_test.go new file mode 100644 index 0000000000..b10971e570 --- /dev/null +++ b/internal/surface/cli/sides_test.go @@ -0,0 +1,122 @@ +package cli + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "regexp" + "strings" + "testing" + + "github.com/intentdriven/abcd/internal/core/intent" + "github.com/intentdriven/abcd/internal/core/oracle" + "github.com/intentdriven/abcd/internal/core/runner" +) + +// The tests in this file are itd-166 at the front door: the route lines name +// the asked model and the cost when a record carries them, and the +// consistency answer's route file carries both sides of the call that +// produced it, kept and reused whole, beside the files it always had. + +// TestTheRouteLineNamesTheAskedModelAndCost: a runner's route line names the +// model the runner asked for and the cost when present, and a route without +// either reads as it always has; a dispatched step's line names the served +// provider and the cost when the call reported them. +func TestTheRouteLineNamesTheAskedModelAndCost(t *testing.T) { + plain := runner.RouteRecord{Asked: runner.Claude, Ran: runner.Claude, Model: "fake-model"} + if got := routeSuffix(&plain); got != " — ran on the claude runner (asked claude; model fake-model)" { + t.Fatalf("a route without sides renders %q", got) + } + sided := runner.RouteRecord{Asked: runner.OpenCode, Ran: runner.OpenCode, + Sides: runner.Sides{ModelAsked: "local/qwen3-coder", Cost: &oracle.Cost{Amount: 0.002, Unit: oracle.CostUSD}}} + got := routeSuffix(&sided) + for _, want := range []string{"model none reported", "model asked local/qwen3-coder", "cost 0.002 usd"} { + if !strings.Contains(got, want) { + t.Errorf("route line %q lacks %q", got, want) + } + } + var b bytes.Buffer + renderDispatchLine(&b, &dispatched{receipt: oracle.ReceiptRoute{Agent: "scribe", ConnectionUsed: "openrouter", + ProviderCall: &oracle.CallRecord{ModelAsked: "typesafe/jev-1.13", ModelReported: "typesafe/jev-1.13-20260915", + ServedProvider: "Fireworks", Cost: &oracle.Cost{Amount: 0.00042, Unit: oracle.CostCredits}}}}) + for _, want := range []string{"model reported typesafe/jev-1.13-20260915", "served by Fireworks", "cost 0.00042 credits"} { + if !strings.Contains(b.String(), want) { + t.Errorf("dispatch line %q lacks %q", b.String(), want) + } + } + b.Reset() + renderDispatchLine(&b, &dispatched{receipt: oracle.ReceiptRoute{Agent: "scribe", ConnectionUsed: "openrouter", + ProviderCall: &oracle.CallRecord{ModelAsked: "m", ModelReported: "m"}}}) + if strings.Contains(b.String(), "cost") || strings.Contains(b.String(), "served by") { + t.Errorf("a call that reported neither names them: %q", b.String()) + } +} + +// TestTheConsistencyRouteFileCarriesBothSides is criterion 7's route-file +// half: a chunk's answer kept with a route carrying both sides is reused with +// that route whole, the chunks sent now carry their own, and the reviews +// directory holds no kind of file it did not before. +func TestTheConsistencyRouteFileCarriesBothSides(t *testing.T) { + srcRoot := repoRootFromTest(t) + repo := consistencyChunkCLIRepo(t) + t.Setenv("ABCD_PLUGIN_ROOT", srcRoot) + var em consistencyChunkEmitted + if err := json.Unmarshal(runCLI(t, "intent", "consistency", "--json"), &em); err != nil { + t.Fatal(err) + } + kept := `{"agent":"intent-auditor","connection_used":"openrouter","provider_call":{"provider":"openrouter",` + + `"model_asked":"typesafe/jev-1.13","model_reported":"local-model","prompt_sha256":"` + strings.Repeat("c", 64) + `",` + + `"adapter_added":["stream_options.include_usage"],"served_provider":"Fireworks","generation_id":"gen-1","finish_reason":"stop",` + + `"usage":{"prompt_tokens":120,"completion_tokens":30,"reasoning_tokens":12,"cache_read_tokens":100,"cache_write_tokens":20},` + + `"cost":{"amount":0.00042,"unit":"credits"},"unpinned":[{"field":"asked.provider_pin","reason":"not_configured"}]},"unpinned":null}` + if _, err := intent.KeepConsistencyAnswer(repo, em.Chunks[0].ReceiptID, + chunkFindingsJSON(t, repo, em.Chunks[0].ReceiptID, em.Chunks[0].RequestPath, em.Chunks[0].CorpusPath), []byte(kept)); err != nil { + t.Fatal(err) + } + receiptRe := regexp.MustCompile(`- receipt_id: (rcp-[0-9a-f]{12})`) + p := newChatFake(t, "local-model", func(body string) string { + m := receiptRe.FindStringSubmatch(body) + rel := ".abcd/.work.local/reviews/" + m[1] + return string(chunkFindingsJSON(t, repo, m[1], rel+".request.md", rel+".corpus.md")) + }) + pointMachine(t, os.Getenv("HOME"), p.srv.URL, false, "", "intent-auditor") + stdout, stderr, err := runCLISplit(t, "intent", "consistency", "--json") + if err != nil { + t.Fatalf("%v\n%s", err, stderr) + } + var routes []json.RawMessage + member(t, []byte(stdout), "chunk_routes", &routes) + if len(routes) != len(em.Chunks) { + t.Fatalf("chunk_routes lists %d receipts for %d chunks", len(routes), len(em.Chunks)) + } + var want, got map[string]any + _ = json.Unmarshal([]byte(kept), &want) + _ = json.Unmarshal(routes[0], &got) + if !jsonEqual(want["provider_call"], got["provider_call"]) { + t.Fatalf("the reused chunk's call record is not the kept one whole:\nkept %s\ngot %s", kept, routes[0]) + } + var sent oracle.ReceiptRoute + if err := json.Unmarshal(routes[1], &sent); err != nil || sent.ProviderCall == nil || + len(sent.ProviderCall.PromptSHA256) != 64 || len(sent.ProviderCall.Unpinned) == 0 { + t.Fatalf("a chunk sent now carries its own call's sides: %s", routes[1]) + } + entries, err := os.ReadDir(filepath.Join(repo, ".abcd/.work.local/reviews")) + if err != nil { + t.Fatal(err) + } + for _, e := range entries { + n := e.Name() + if !strings.HasSuffix(n, ".request.md") && !strings.HasSuffix(n, ".corpus.md") && !strings.HasSuffix(n, ".route.json") && + !strings.HasSuffix(n, ".findings.json") { + t.Errorf("the reviews directory holds %s, a kind of file the pass did not write before", n) + } + } +} + +// jsonEqual compares two decoded JSON values. +func jsonEqual(a, b any) bool { + x, _ := json.Marshal(a) + y, _ := json.Marshal(b) + return bytes.Equal(x, y) +}