What
eval-refresh.yml makes one real inference call per provider before the matrix starts. Its own comment explains why it exists:
On 13 August the OpenAI credit balance hit zero and every one of the thirty-seven Codex jobs ran, failed, and was paid for in wall-clock time before anyone looked. One request per provider, before the matrix starts, costs nothing and turns six hours of failure into a thirty-second one.
pnpm eval has no equivalent. A local run walks straight into the same wall.
It happened again
On 21 August, a local 24-cell run reached cell 6 before every Codex cell began erroring:
agent errored before emitting any transcript events (no tool calls, empty report).
Not scored. Check the agent CLI and the provider key: a quota or spend cap
surfaces exactly like this.
Sixteen cells, all dead, discovered by reading the error rather than by a check that runs in seconds. The per-cell error message is good — it names the likely cause — but it only appears once the run is already underway.
Suggested
Move the check out of the workflow and into the run path, so both callers share it:
- one real completion per provider that the selected experiments actually need — Anthropic only if a Claude experiment is in the matrix, OpenAI if a Codex experiment or any judged scenario is
- run it after the plan is resolved and before the first cell
- fail with the same actionable message the workflow uses
--dry skips it
The judge is the part most likely to be forgotten. OPENAI_API_KEY backs the default judge, so seven scenarios need OpenAI credit whichever agent produced the run. A matrix of only Claude experiments still fails on those seven if the balance is zero.
One trap to preserve
The workflow comment records it, and it is worth not relearning: /v1/models answers 200 with a zero balance, listing every model. Only a real completion returns credit_balance_exhausted. A liveness check built on the models endpoint is worse than none, because it reads as evidence.
Related: #41 — the same run also died from an agent wiping shared deployment config, and the two faults had different signatures.
What
eval-refresh.ymlmakes one real inference call per provider before the matrix starts. Its own comment explains why it exists:pnpm evalhas no equivalent. A local run walks straight into the same wall.It happened again
On 21 August, a local 24-cell run reached cell 6 before every Codex cell began erroring:
Sixteen cells, all dead, discovered by reading the error rather than by a check that runs in seconds. The per-cell error message is good — it names the likely cause — but it only appears once the run is already underway.
Suggested
Move the check out of the workflow and into the run path, so both callers share it:
--dryskips itThe judge is the part most likely to be forgotten.
OPENAI_API_KEYbacks the default judge, so seven scenarios need OpenAI credit whichever agent produced the run. A matrix of only Claude experiments still fails on those seven if the balance is zero.One trap to preserve
The workflow comment records it, and it is worth not relearning:
/v1/modelsanswers 200 with a zero balance, listing every model. Only a real completion returnscredit_balance_exhausted. A liveness check built on the models endpoint is worse than none, because it reads as evidence.Related: #41 — the same run also died from an agent wiping shared deployment config, and the two faults had different signatures.