diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index b292fd67..5eef1adc 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -489,3 +489,65 @@ with the two `action.yml` items above — one considered change to the action's run. Guard would need cross-method dataflow over each `Agent`-subclass turn-state class (write sites vs read sites), which the AST-only CExxx runner can't express in ~30 min — deferred. Caught in: Pi harness Phase 2 quality review. + +- [ ] A duration/count aggregate over run-task rows that fails to exclude + `mature_skipped`. Nothing guards it: the CExxx runner is Python-AST only and + this defect class lives in the evalboard's TypeScript. A codex nightly + rendered "1300 tasks · 15h 29m" for the 397 tasks that actually ran. Fixed + for the two whole-run sites by extracting `deriveRunDuration`; a general + guard needs a TS lint surface the harness does not have. + Caught in: timing-capture Phase 1 review. + +- [ ] Subtracting a SUM of intervals from a wall span where the intervals can + overlap. Semantic, not syntactic — an AST rule cannot tell a sum of + durations from a union. Antigravity shipped it: four concurrent 400ms tool + calls inside a 1000ms window summed to 1600ms and clamped generation to the + 0.0 the change existed to remove. The real guard is the replay-based + `assert_timing_captured` golden sensor, which now exists; a static rule + would not have caught it. Caught in: timing-capture Phase 3 review. + +- [ ] A test fixture whose field name does not exist on the type it models. + `parseMessages`'s `CommandEntry` keys on `tool_id`; a fixture using + `tool_use_id` never resolves, params fall back to `{}`, and every tool + weighs exactly 1 — which silently turned a "split by content size" test + into a 99%-thinking assertion that passed. TypeScript accepts it because + the fixtures are untyped object literals. Typing the fixture factories + against the real interfaces would guard the whole class; that is a + sweep across the evalboard test suite, not ~30 min. + Caught in: timing-capture Phase 5 review. + +### Deferred timing divergences (not guardrails — accounting gaps) + +Recorded here as well as in `docs/agents/HARNESS_PARITY.md` § Known +divergences, so the deferred-work record is one place. Measurements in +`c/time-bugs-audit.md`. + +- [ ] **Antigravity books orphan-poll waiting as agent duration** (audit P2-1). + A task can spend `0.8 × turn_timeout` waiting on a tool call that never + reaches DONE — 14 tasks, 9.6h of one 83h run. Only CLOSED tool intervals are + subtracted from a generation window, so that wait stays inside whichever + window contains it; the force-close records `execution_completed_at` while + leaving `duration_ms` as `None`. Deliberately out of scope of the timing + work: closing it means deciding whether a backgrounded tool's elapsed time + is model time (the model IS generating while it runs), which is a semantic + question, not a bug fix. + +- [ ] **Delegate (`delegate-sdk`) records no execution bounds** (audit P3-1). + It reports `duration_ms` but neither `execution_started_at` nor + `execution_completed_at`, so its tool calls cannot be placed on a timeline. + Coverage is ~88%, so it is not urgent. The agent lives in the separate + `coder_eval_uipath` repo; mirror the Codex change there + (`_item_timing` + threading the SDK stamps through the telemetry builders). + +- [ ] **Pre-existing, surfaced by this work's final review: `TokenUsage._adopt_legacy_input_tokens` + double-counts the cache buckets.** The validator copies a legacy record's + full-prompt `input_tokens` straight into `uncached_input_tokens`, and the + computed `input_tokens` then adds `cache_creation` + `cache_read` again. A + legacy record with `input_tokens=1000`, `cache_creation=200`, `cache_read=150` + reloads as 1350 prompt tokens and bills 1000 at the uncached rate instead of + 650. Affects every report, budget check and detached regrade over a + pre-split run that used prompt caching. NOT touched by the timing work + (token accounting was explicitly out of its scope) and not a guardrail + candidate — a real bug needing its own change, with a decision about + whether legacy records can be distinguished from current ones at all. + Caught in: timing-capture final review (gpt-5.6-sol). diff --git a/CLAUDE.md b/CLAUDE.md index 2a0d60c5..616945b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -236,7 +236,7 @@ make plugin-reference # the plugin's bundled criteria reference from the models Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction. -Recent additions, each traceable to a shipped defect: **CE056** (no bare `CODER_EVAL_IN_CONTAINER` literal outside `models/container_paths.py` — the CE053 shape again: a rename-safety constant that shipped beside the literal it replaced, and the straggler was the single WRITER, so a rename would have disarmed four security/correctness gates at once with nothing failing; CE052 cannot catch it because that rule inspects `if` guards and the writer is not one), **CE055** (a criterion `path:` in `tasks/` must be sandbox-relative — an absolute path is joined onto the sandbox root, which DISCARDS the root, so containment refuses it and the criterion can never match whatever the agent does; two in-tree tasks were broken this way and the pair is the argument for a static rule on top of the runtime `CheckerMisuseError`: `byod_smoke_test` IS in a CI bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not exist" for a file that existed, while `dockerfile_build_example` is in NO bucket, so nothing ran it and no runtime guard was ever reached — the fix is never to relax containment but to say what the criterion means, `run_command: test -f /opt/marker`, a claim about the container IMAGE rather than about the agent's workspace), **CE054** (an `environment_info` key that is READ must be WRITTEN somewhere in `src/` — the bag is `dict[str, Any]`, so nothing connects reader to writer, and the `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: `.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user guide both described it as protection it never provided), **CE048** (never call a Typer command function in process — its parameter defaults are `OptionInfo` sentinels, not values, and the sentinel is TRUTHY, so `in_place=None` silently selected the wrong branch; the fix is the `run_pipeline` / `run_evaluation` / `run_plan` split, and this rule is the one that also scans `tests/`, since that is the only place the defect occurs), **CE049** (never coalesce a possibly-unmeasured score to a numeric literal — `score or 0.0` publishes "measured and scored zero" while meaning "never measured", which is how an ungraded night reached four unfiltered `avg(Score)` dashboards as a real zero), **CE050** (no untyped `getattr` probe for a discriminated-union field — pyright cannot see the string, so a rename degrades the guard to a permanent no-op; scoped to criterion-shaped receivers because `command`/`tool`/`prompt` are far too common to flag on their own), **CE051** (a sandbox driver may not be rewritten silently — the driver IS the isolation boundary, so a downgrade must be an explicit, stamped, operator-visible decision), **CE053** (no bare run-record or run-LOG filename literal outside `path_utils` — widened to `docker.log` / `grade.docker.log` / `task.log` / `grade.log` after the same shape recurred: `docker.log` was produced in `isolation/` and consumed in `orchestration/` as three unrelated literals, and because the consumer guards its copy with `is_file()`, a rename would have silently discarded the only record of why a grading container failed — `TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals stayed unmigrated, including all three `rglob("task.json")` sites the constant's own comment cites as its reason to exist, so it created the second source of truth it argues against), **CE052** (an `os._exit` must sit inside a branch testing `CODER_EVAL_IN_CONTAINER` — it is the right primitive only for reaping the container's own disposable main process, and `run_task_internal_command` armed its heartbeat watchdog, a daemon thread whose whole authority is `os._exit(137)`, unconditionally: a test that invoked the command in-process left the pytest worker holding that thread, which exited the worker 40s later inside an unrelated test file, naming a different test on each run and on each platform with no traceback — and the dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, naming neither the test nor the cause), **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed), **CE057** (a module copied into the recorder directory beside a generated sandbox shim — `models.sandbox.SIDECAR_MODULES`, currently `argv_match.py` — may import stdlib only. The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not installed, so one package import makes every shadowed CLI die with an ImportError the agent reads as "the tool is broken", costing a whole run to diagnose. The rule derives its target set from that exported tuple and a test asserts it matches a file that exists — a lint rule guarding zero files must fail, not pass). +Recent additions, each traceable to a shipped defect: **CE058** (in `src/coder_eval/`, an unknown timing value may not become a numeric literal — `duration_ms is None` means *never timed* and `0.0` means *timed and instant*, so writing the literal publishes the second while meaning the first. One invariant, one id, five syntactic forms — a zero constructor keyword, `x or 0`, `x if x is not None else 0.0`, `if x.duration_ms is None: x.duration_ms = 0.0`, and a `model_copy(update={...})` dict (the shape the Antigravity DONE path writes through, which a keyword-only rule cannot see). Antigravity constructed EVERY message with `generation_duration_ms=0.0`, so the task page's Generation cell read `0ms` and its breakdown rendered `0%` for months with nothing failing; Codex published the SDK's `0.0` as a measured command duration, so `avg_command_time_ms` divided real milliseconds by a command count of which 70 of 211 in one nightly had never been timed. The fourth form is the one no existing rule shape covered and is where a live instance was hiding — `claude_code_agent._finalize_commands` set `0.0` on every command force-closed without a tool result, in the one harness a timing audit had called healthy. BLIND SPOT, stated in the rule's docstring: form 1 keys on the callee's spelling, so renaming the `AssistantMessageTelemetry` import alias silently disarms it there), **CE059** (in `src/coder_eval/agents/`, an `AssistantMessage` may not receive the same `ast.Name` for both `started_at` and `completed_at` — the Antigravity reducer read `datetime.now()` once and passed it as both bounds, so `started_at == completed_at` on 368 of 368 sampled messages. A separate id from CE058 because it is a separate invariant, a zero-length window whatever the duration field says, and one invariant per id is what makes a `# noqa` mean one thing. It does NOT fire when the same call passes `generation_duration_ms=None`: a call that says, in the field built to say it, that no window was measurable is not claiming one — that exemption is what keeps the rule pointed at the misleading case instead of accumulating four permanent suppressions on the rollout-rebuild and sub-agent-synthesis sites), **CE056** (no bare `CODER_EVAL_IN_CONTAINER` literal outside `models/container_paths.py` — the CE053 shape again: a rename-safety constant that shipped beside the literal it replaced, and the straggler was the single WRITER, so a rename would have disarmed four security/correctness gates at once with nothing failing; CE052 cannot catch it because that rule inspects `if` guards and the writer is not one), **CE055** (a criterion `path:` in `tasks/` must be sandbox-relative — an absolute path is joined onto the sandbox root, which DISCARDS the root, so containment refuses it and the criterion can never match whatever the agent does; two in-tree tasks were broken this way and the pair is the argument for a static rule on top of the runtime `CheckerMisuseError`: `byod_smoke_test` IS in a CI bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not exist" for a file that existed, while `dockerfile_build_example` is in NO bucket, so nothing ran it and no runtime guard was ever reached — the fix is never to relax containment but to say what the criterion means, `run_command: test -f /opt/marker`, a claim about the container IMAGE rather than about the agent's workspace), **CE054** (an `environment_info` key that is READ must be WRITTEN somewhere in `src/` — the bag is `dict[str, Any]`, so nothing connects reader to writer, and the `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: `.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user guide both described it as protection it never provided), **CE048** (never call a Typer command function in process — its parameter defaults are `OptionInfo` sentinels, not values, and the sentinel is TRUTHY, so `in_place=None` silently selected the wrong branch; the fix is the `run_pipeline` / `run_evaluation` / `run_plan` split, and this rule is the one that also scans `tests/`, since that is the only place the defect occurs), **CE049** (never coalesce a possibly-unmeasured score to a numeric literal — `score or 0.0` publishes "measured and scored zero" while meaning "never measured", which is how an ungraded night reached four unfiltered `avg(Score)` dashboards as a real zero), **CE050** (no untyped `getattr` probe for a discriminated-union field — pyright cannot see the string, so a rename degrades the guard to a permanent no-op; scoped to criterion-shaped receivers because `command`/`tool`/`prompt` are far too common to flag on their own), **CE051** (a sandbox driver may not be rewritten silently — the driver IS the isolation boundary, so a downgrade must be an explicit, stamped, operator-visible decision), **CE053** (no bare run-record or run-LOG filename literal outside `path_utils` — widened to `docker.log` / `grade.docker.log` / `task.log` / `grade.log` after the same shape recurred: `docker.log` was produced in `isolation/` and consumed in `orchestration/` as three unrelated literals, and because the consumer guards its copy with `is_file()`, a rename would have silently discarded the only record of why a grading container failed — `TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals stayed unmigrated, including all three `rglob("task.json")` sites the constant's own comment cites as its reason to exist, so it created the second source of truth it argues against), **CE052** (an `os._exit` must sit inside a branch testing `CODER_EVAL_IN_CONTAINER` — it is the right primitive only for reaping the container's own disposable main process, and `run_task_internal_command` armed its heartbeat watchdog, a daemon thread whose whole authority is `os._exit(137)`, unconditionally: a test that invoked the command in-process left the pytest worker holding that thread, which exited the worker 40s later inside an unrelated test file, naming a different test on each run and on each platform with no traceback — and the dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, naming neither the test nor the cause), **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed), **CE057** (a module copied into the recorder directory beside a generated sandbox shim — `models.sandbox.SIDECAR_MODULES`, currently `argv_match.py` — may import stdlib only. The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not installed, so one package import makes every shadowed CLI die with an ImportError the agent reads as "the tool is broken", costing a whole run to diagnose. The rule derives its target set from that exported tuple and a test asserts it matches a file that exists — a lint rule guarding zero files must fail, not pass). When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033–CE036 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE036 enforces the `live_verdict` determinism + monotonicity contract (`criteria/base.py`) that `EarlyStopWatcher`'s latching, deferred fail-stop, and flip-attribution silently depend on: monotonicity over arbitrary Python is undecidable, so instead of a static check it REPLAYS each live criterion against every prefix of recorded trajectories (`tests/lint/live_verdict_contract.py::CASES`) — on the authored ordering AND under seeded shuffles (`permuted_violations`, which catch order-sensitive bugs the authored walk misses) — and asserts the property directly, plus registry-derived coverage — every `LiveSuccessCriterion` in the union must have cases, and every polarity its instances claim via `live_decidable_polarities()` must actually be reached by one (otherwise a single always-`undecided` fixture would "cover" a type while proving nothing). Adding a live criterion therefore means adding `ContractCase`s in the same change. CE035 resolves every `steps..outputs.` / `needs..outputs.` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.) diff --git a/docs/agents/HARNESS_PARITY.md b/docs/agents/HARNESS_PARITY.md index 7da24b4e..4b429032 100644 --- a/docs/agents/HARNESS_PARITY.md +++ b/docs/agents/HARNESS_PARITY.md @@ -17,6 +17,73 @@ This page is the contract for what each run limit means per harness, plus the sh | `run_limits.task_timeout` | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | | `run_limits.stop_early` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` (event granularity) | cooperative `should_stop` (event granularity — Pi streams incrementally) | +## Timing capture + +What each harness records about *when* things happened, and how much of a task's +wall clock its numbers account for. + +| Field | claude-code | codex | antigravity | opencode | pi | +|---|---|---|---|---|---| +| `generation_duration_ms` source | harness clock: previous SDK event → this message | SDK item stamps, minus tool execution inside the window | harness clock: previous flush → this flush, minus tool execution inside the window | harness clock per CLI step, minus tool execution inside the step | harness clock per CLI turn, minus tool execution inside the turn | +| tool `duration_ms` source | measured around the tool result | SDK `completed_at_ms − started_at_ms`; the item's own `duration_ms` only as a fallback | measured ACTIVE → DONE | measured around the tool event | measured around the tool event | +| `execution_started_at` / `execution_completed_at` | derived from the measured duration | SDK stamps (both, or neither) | measured at ACTIVE / DONE | measured | measured | +| `generation_completed_at` | set | `None` — see below | `None` | `None` | `None` | +| `Σ generation + Σ tool ≈ turn duration` | yes | yes | yes | yes | yes | + +**`generation_duration_ms` is model-generation time, not `completed_at − started_at`.** +Four of the five harnesses interleave tool execution into a single generation +window. Antigravity reports a `Step` for the tool and only a later +`usage_metadata` `Step` cuts the message; Codex's message window is seeded from +the first item's start and extended to the last item's completion; OpenCode +opens its window at `step_start` and closes it at `step_finish`, and Pi at +`turn_start` / `turn_end`, with every tool call running inside. In all four the +span between the recorded bounds legitimately CONTAINS tool time that the model +did not spend generating, so all four subtract it — the **union** of the closed tool intervals +clipped to the window (`agents/_timing.py::busy_ms`), never the sum, because +tool calls overlap: Antigravity resolves several from one `Step` and backgrounds +anything over ten seconds, and Codex spawns collab agents concurrently. Summing +them over-subtracts by exactly the overlap and, with enough concurrency, drives +the result to a clamped zero. + +The consequence worth knowing: on an emission that carries *only* a tool call, +the whole measured window was that tool running, so the recorded generation +time is legitimately `0.0`. That is a measurement, not a placeholder — `None` +is what "never measured" looks like. Only `claude-code` does not need the +subtraction: it marks the end of the previous SDK event and reads again when +the next message arrives, so a tool's execution falls between two windows +rather than inside one. + +**Why Codex leaves `generation_completed_at` as `None`.** It means "when the +model finished emitting the `tool_use` block". Codex's stream does not carry +that per tool; deriving it from the flush time would be a guess. Note also that +`CommandTelemetry.timestamp` is the tool's own start on codex, antigravity, +opencode and pi, and the generation-completed moment on claude-code. Nothing +orders on it — `TurnRecord.commands` is sorted by `sequence_number` — but the +field's own docstring still describes only the claude-code reading. + +**Codex `duration_ms` covers more than the command run.** Derived from the item +stamps, it is the item's lifecycle (queueing and approval included) rather than +the SDK's own narrower command-execution figure, which it deliberately overrides +— the SDK reported `0` for 70 of 211 commands in one nightly. `fileChange` and +generic tool items now carry a duration where they previously carried none, so +`avg_command_time_ms` and `total_command_time_ms` for a Codex run describe every +tool call rather than shell commands alone. + +### Known divergences + +- **Delegate (`delegate-sdk`, out of tree)** records `duration_ms` but no + execution bounds, so its tool calls cannot be placed on a timeline. Its + coverage is ~88%. Mirror the Codex change in `coder_eval_uipath` + (audit P3-1). +- **Antigravity books orphan-poll waiting as agent duration.** A task can spend + `0.8 × turn_timeout` waiting on a tool call that never reaches DONE — 14 tasks + and 9.6h of one 83h run. Only CLOSED tool intervals are subtracted, so that + wait stays inside whichever generation window contains it, and the force-close + records `execution_completed_at` while leaving `duration_ms` as `None` + (audit P2-1). + +Both are deliberately deferred; see `c/time-bugs-audit.md` for the measurements. + ## `max_turns` counts visible turns on Codex and Antigravity A "visible turn" is one entry in the run's timeline: one resolved tool call. It is diff --git a/evalboard/app/page.tsx b/evalboard/app/page.tsx index 7a66f245..ee20f570 100644 --- a/evalboard/app/page.tsx +++ b/evalboard/app/page.tsx @@ -73,6 +73,40 @@ function fmtCost(c: number | null): string { return `$${c.toFixed(2)}`; } +// The Duration cell for a run row. `taskDurationSeconds` is compute time over +// the rows that actually ran, so when the nightly carried some forward as +// mature passes the cell says how many that was — otherwise "1300 tasks · +// 15h 29m" reads as a per-task rate over 1300 tasks when it describes 397. +function RunDurationCell({ + seconds, + tasksRun, + tasksExecuted, +}: { + seconds: number | null; + tasksRun: number; + tasksExecuted: number; +}) { + const skipped = tasksRun - tasksExecuted; + if (skipped <= 0) { + return ( + + {fmtDuration(seconds)} + + ); + } + return ( + + {fmtDuration(seconds)} +
+ {tasksExecuted} run +
+ + ); +} + // Rail-level q filter: substring match on tag name only. This is narrower // than getRunListing's q (which also matches taskId / humanized id) by // design — rails are a tag namespace, the table is a task namespace. @@ -428,9 +462,11 @@ export default async function Page({ {fmtCost(r.totalCostUsd)} - - {fmtDuration(r.taskDurationSeconds)} - + ); })} @@ -572,11 +608,11 @@ export default async function Page({ {fmtCost(r.totalCostUsd)} - - {fmtDuration( - r.taskDurationSeconds, - )} - + ); })} diff --git a/evalboard/app/runs/[id]/[...task]/__tests__/message-timeline.test.tsx b/evalboard/app/runs/[id]/[...task]/__tests__/message-timeline.test.tsx index cf5ade72..bbeb5638 100644 --- a/evalboard/app/runs/[id]/[...task]/__tests__/message-timeline.test.tsx +++ b/evalboard/app/runs/[id]/[...task]/__tests__/message-timeline.test.tsx @@ -13,6 +13,7 @@ function makeMessage(overrides: Partial = {}): MessageEvent { thinkingMs: null, textMs: 1000, toolGenMs: null, + mixedGenMs: null, blockTypes: ["text"], thinkingText: null, text: "hello", @@ -476,3 +477,264 @@ describe("MessageTimelineSection — sub-agent grouping", () => { expect(screen.getByText("Message timeline (1)")).toBeInTheDocument(); }); }); + +// The strip must reconcile: generation + tool exec are shown against the wall +// clock they should add up to, so a harness that stops reporting one of them +// is visible on the page instead of silently reading as fast. +describe("MessageTimelineSection — Unaccounted cell", () => { + // Each summary cell is