Skip to content

STATE.md

Latest commit

 

History

History
450 lines (351 loc) · 25.5 KB

STATE.md

File metadata and controls

450 lines (351 loc) · 25.5 KB
Raw

Where this project stands

Handoff note. Read this first when picking the work back up.

RESUME HERE — 2026-09-02

This repository is not the state document any more. The experiment record lives in the lab: agent-learning-lab/HANDOFF.md, starting at "What the eighth session changed, 2026-09-01". Read that first. This file now covers only what changed in the instrument.

The spine is at position 4 of 28 (B2 — plain baseline). B1 closed 2026-08-30. B2 has run and been scored; its gate is not closed. Nothing is green-and-unmerged in either repo: agent-observatory main is 13419ef, agent-learning-lab main is 66dd630.

What changed in the instrument since 2026-08-10

The 08-10 block below predates all of it — it was written before B2, before issue #65, and before PR #66.

change where
Per-run HOME and --disable plugins on the codex arm — the two arms were not the same experiment, and the record could not say so #66 → b39b85e
cmux shims stripped from PATH — a terminal host was putting $TMPDIR/cmux-cli-shims/<id>/{claude,codex} ahead of the real CLI, injecting session ids, settings and --dangerously-bypass-hook-trust runner/run-agent.sh §PATH
V4 — behaviour counters are nullable. model_calls, tool_calls, tool_failures drop NOT NULL, so a telemetry gap stops being indistinguishable from a genuine measured zero V4__behavior_not_measured.sql
V5 — reported_total_tokens for runtimes that give no input/output split; null when the runtime reports a breakdown V5__reported_total_tokens.sql
V6 — the surface record. user_settings_isolated, shims_stripped, agent_surface — a run now states the surface it ran on rather than leaving it to be inferred V6__agent_surface.sql
§13.1 in the analyzer — compare efficiency only among runs that passed, so a run that failed a quality gate cannot lower its arm's cost median #50 → 13419ef, runner/analyze-experiment.py:334

V4 is why an arm with no telemetry now reads as unmeasured rather than as the most efficient arm in the comparison. That is the correction that made codex n=5 legible in B2's baseline report.

Closed since

  • obs#51 — render unreported behaviour counters as unknown, not zero — closed 2026-09-02 as superseded by #66, not merged. It inferred absence from modelCalls == 0 && toolCalls == 0, and V4 made the columns nullable, so that heuristic would now blank a genuine measured zero.
  • obs#52 — BehaviorDto fabricates zeros — closed 2026-09-02. It is what V4 did.

What this repo still owes

obs#53 — record load alongside each run. The duration tail on both arms is contaminated by something the run record does not capture, and the record cannot separate a contaminated run from a clean one. See HANDOFF, "The sleep explains one run, and not the arm it was used to void."

History

Everything below is kept as the record and is no longer current. It describes the instrument as it stood on 2026-08-10.

RESUME HERE — 2026-08-10 (end of day) — historical

EXP-BE002-CLAUDEMD-V2 produced this project's first non-void result. Full write-up in preregistration-exp-be002-agentsmd.md, "Result — EXP-BE002-CLAUDEMD-V2".

VERDICT: INCONCLUSIVE — cost change +39% (p=0.00) does not clear the registered bar

10 + 10 measuring runs, arms interleaved, zero skill contamination, instructionsHash set on all 10 treatment runs and null on all 10 baseline runs.

metric baseline instructions change p
cost (primary) $0.1301 $0.1809 +39.0% <0.01
cache tokens 571 k 805 k +41.0% 0.01
tool calls 15.5 19.5 +25.8% 0.01
pass rate 100% 100% — —
F02 / F07 0 / 0 0 / 0 — —

INCONCLUSIVE is correct by the letter of the rule and a misleading word for it. KEEP needs cost to improve beyond MDE; REJECT needs F02 to increase. Cost moved beyond its MDE with p < .01 in the wrong direction and F02 stayed at zero, so neither branch fires. The rule has no branch for "significantly worse on the primary metric, without harming correctness". Noted, deliberately not patched — patching a decision rule with its result on screen is what the registration exists to prevent. Fix it before the next experiment.

The finding, which is not statistical

Zero crossover in 20 runs:

  • baseline, 10/10 — changed OrderController.kt only, validation hand-rolled inline.
  • instructions, 10/10 — changed Order.kt + OrderController.kt + GlobalExceptionHandler.kt: @Positive on the DTO, @Valid on the parameter, a MethodArgumentNotValidException handler.

That is the CLAUDE.md Conventions section followed literally. The instruction file works exactly as written — it buys a more idiomatic implementation, and on this task that costs 39% more for an outcome the evaluator scores identically.

Prediction 3 called the direction and the wrong cause. Carrying the file is ~5 300 cache-creation tokens (+21%); output tokens rose 50% and added lines 54%. Roughly two thirds of the premium is the extra work the file prescribes, one third is heavier context. So "use a shorter file" would not recover it.

Three of four predictions held, against one of four for the void V3.

The hook caveat is discharged — EXP-BE002-NOHOOKS, same day

Ran both arms again with --isolate-user-settings (--setting-sources project), which drops ~/.claude/settings.json and the 21 hooks in it while leaving CLAUDE.md readable. Manipulation check: 0 hook executions in all 20 runs, against a V2 baseline median of 24.5.

with hooks (V2) without hooks level change
baseline cost $0.1301 $0.1139 −12.5%
instructions cost $0.1809 $0.1572 −13.1%
premium +39.0% +38.1% −0.9 pp

The premium is not a hook artefact. Both arms got ~13% cheaper by the same proportion and the gap barely moved. Four of four predictions held. The behavioural split is now 50 of 50 with zero crossover.

Incidentally measured and worth keeping: hooks cost ~13% of a run on this machine, at ~1.6 executions per tool call.

Two corrections happened along the way, both worth remembering:

  • Hook counts were first published as 16.0/18.5 from grep -c over the event log. The collector batches many records per line, so that counted batches and undercounted by about a third. Correct figures 24.5/31.5. Never grep -c this log for per-run counts.
  • The caveat was first drafted as "+39% is an upper bound". Checking the per-arm rates showed that was a guess with a direction attached, so it was not published that way. The measurement then came out at +38.1% — roughly what "upper bound" implied, for reasons that claim did not have.

Next actions

  1. PR the branch exp/be002-claudemd (4 commits, pushed). It carries the void notice, the bug #13 fix, the class-based abort rule, and this result.
  2. Add a REJECT-on-significantly-worse branch to the decision rule, registered before the next experiment exists — not retrofitted to this one.
  3. The experiment this result argues for: a task where the idiomatic structure compounds — several endpoints, or a second change layered on the first. BE-002 is close to the worst case for a conventions file, since conventions buy consistency the task never exercises.
  4. Still open: ~/.claude/settings.json reaches the agent, so operator permission rules are inherited. Constant across arms so it does not bias this result, but "baseline" is not a pristine Claude Code. Narrower than #13, tracked, not fixed.

Two bugs found today, and the second one bit twice

#13 — the runs were not isolated. --strict-mcp-config was passed, with a comment arguing that user-scope config makes "the plain baseline vary by machine". The identical hole for plugins and their skills was open. A plugin skill executed in 5 of 23 V1 runs; one answered a bugfix task by writing docs/superpowers/plans/…md and changed no production file. skillsHash was null on every run because it hashes .github/skills.md inside the worktree, a path that cannot exist there — a field structurally incapable of reporting the problem it appeared to rule out. This voided EXP-BE002-CLAUDEMD and withdrew an F12 adjudication that was one commit from being published.

#2, fourth costume. The infrastructure detector was extended for API Error and Connection closed. The very next batch died on You've hit your session limit, a phrase it did not contain, and 16 runs recorded F03 "incorrect code" for a billing state. The fix is no longer a phrase: an agent that changed no file and called no tool did not attempt the task and cannot have failed it. That needs no vocabulary and does not go stale. It stays narrow — a run that explored and then stalled has tool calls, so it still counts against its arm.

Infrastructure gotchas

  • Session limit resets on a wall clock (resets 1pm (Europe/Prague) observed). A 20-run batch is roughly the budget; expect to split larger ones.
  • Colima's port forwarding wedges: containers healthy inside the VM, every host port dead. colima restart, then start containers by hand — none have a restart policy, including grafana-* and ai-portfolio-rabbitmq-1, which are not this project's.
  • make build-api before docker compose up --build. Spring silently ignores unknown JSON fields, so a stale image drops new evaluation fields while the guard looks like it works.

Status: the instrument works, and has now produced one valid result

Chapter 00 M0–M10 is built, tested and merged. make up && make demo works. The instrument was hardened substantially on 2026-08-09.

Six experiments attempted. Five void; the sixth is the result above.

experiment why it is void
EXP-BE002-AGENTSMD answer key in the agent's working tree
EXP-BE002-AGENTSMD-V2 #28 removed it from the tree, left it in git history
EXP-BE002-AGENTSMD-V3 the treatment was never loaded — see below
EXP-BE002-MODEL-TIER permission confound, and its registration was committed after nine runs had started
EXP-BE002-CLAUDEMD runs not isolated — a plugin skill from ~/.claude/ executed in 5 of 23 runs

V3 is void because Claude Code does not read AGENTS.md

This one was written up as a concluded result — INCONCLUSIVE, cost −12.8% at p = .04, pass rate 80% → 100%. The comparison was baseline against baseline.

agents-md-v1 is a Copilot-native customization. It was run against Claude, which loads CLAUDE.md. Verified by controlled test: an identical file named CLAUDE.md is read, and AGENTS.md is not. The runner installed the file, hashed it into instructionsHash, and all ten B1 runs carried the same hash — so it looked perfectly applied. It was applied to disk and never read.

That also resolves the replication anomaly logged below as unexplained: two "different" haiku arms four hours apart differed by 8/10 vs 10/10 and 19 vs 14 tool calls, the same magnitude as the "effect" — because both were in fact identical configurations.

Found in review of PR #33, not by two audit passes. The lesson is in "The thing most worth remembering" at the bottom.

The upside: the AGENTS.md question has never actually been tested. Rename the file to CLAUDE.md and its four predictions are live and untouched — a ~$4 experiment that would be this project's first real result.

Resuming mid-experiment

Run results are persisted in PostgreSQL by the runner, not held in any session, so a batch that is still running loses nothing when a context is cleared. To pick up:

curl -fsS http://localhost:8081/api/runs \
  | jq -r '[.[]|select(.experimentKey=="EXP-BE002-AGENTSMD-V3")]
           | sort_by(.startedAt) | .[]
           | "\(.variant) pass=\(.evaluation.passed) class=\(.evaluation.failureClass // "-")"'

ps -ef | grep run-agent shows whether a batch is still going. Re-running a batch that already completed would produce more than the registered 10 runs per arm, which the analysis tool refuses to score — so check the count before launching anything.

The finding that invalidates the previous results

The benchmark's answer key reached the agent — first in the working tree, then, after that was fixed, in git history.

run-agent.sh handed the agent a git worktree of the benchmarks repo, which contains:

tasks/BE-00X-.../acceptance/*.kt        the exact tests the run is graded on
tasks/BE-00X-.../fixtures/known-good/   a model solution
tasks/BE-00X-.../evaluator.sh           the checks and their exit codes
README.md                               since 2026-08-09, the trap described in prose

Proof it is not theoretical: pilot run a69933a0 ended its own summary with "This will run the existing OrderControllerTest suite plus the new BE002FunctionalTest and BE002ContractTest acceptance tests." Those filenames exist nowhere but the answer directory.

#28 did not close it. It deleted those paths from the working tree and committed the deletion, but a worktree shares its parent repository's object store, so inside the "stripped" tree both of these still worked — and were run against the live worktree of a run that was executing at the time:

git show --stat HEAD    # the setup commit's diff, naming BE002FunctionalTest.kt,
                        # BE002ContractTest.kt, evaluator.sh, fixtures/known-good/
git show HEAD^:tasks/BE-002-…/fixtures/known-good/…/OrderController.kt   # the solution

The setup commit's own message, "strip benchmark authoring material from the worktree", advertised that the material existed and had a previous state.

These are harness bugs five and six, and the only two that flatter the agent. The other four all made runs look worse than they were, which is easy to notice. A bias toward good results is not — and the second one survived a fix aimed directly at it.

The symptom that was already explained, and therefore never re-checked

Median cost and tool calls across every BE-002 stage, now that a clean arm exists:

stage answer key reachable via cost tool calls pass
pilot (n=5) working tree $0.213 20 4/5
post-README (n=3) working tree + README prose $0.115–0.143 9–13 —
V2 baseline (n=4) git history only $0.108–0.174 11–17 4/4
V3 baseline (n=10) nothing $0.190 19.5 8/10

The drop from the pilot's 20 tool calls to ~13 was written up as corroboration of the working-tree leak. It survived the fix for that leak completely, and it did not survive removing the git-history route either — the clean arm is back at 19.5 tool calls and $0.190, statistically indistinguishable from the pilot.

So the attribution was wrong twice over. The most likely cause of the dip is mundane: #28 shrank the tree from the whole benchmarks repo to sample-service, and a smaller tree takes fewer calls to explore. Nothing to do with reading the answer.

A symptom with an explanation attached stops being treated as evidence. The drop had a name, so nobody asked whether it disappeared when the named cause was removed — and it didn't, through two fixes. That check costs one query and is the only thing that would have caught either leak earlier.

What the leaks actually cost is still unmeasured. The clean arm's pass rate is 8/10 against the leaking arm's 4/4, and its two failures are both F07, but those are different n and different runs; treat it as suggestive, not as a measured effect.

What it invalidates and what survived

  • The BE-002 pilot (5 runs, 4 pass / 1 F07) and the power estimates derived from it.
  • EXP-BE002-AGENTSMD is void (3 of 20 runs) and EXP-BE002-AGENTSMD-V2 is void (4 of 20 runs, all baseline, all passing).
  • EXP-BE001-CLEAN did not test what it claimed. Its 5/5 pass, median 15 tool calls, $0.155, 103 s were measured on the stripped worktree — i.e. with the answer key still reachable through git. "Removing the answer key did not make BE-001 harder" was not tested. Whether BE-001 is genuinely easy is an open question again. Building BE-002 is still justified on its own merits, but that $0.75 disproof needs re-spending if the BE-001 claim is ever load-bearing.

Recommended next sequence

  1. Fix the leak. Done twice. #28 stripped the working tree and missed git history; the runner now builds the agent's tree with git archive of an allowlist into a fresh git init, and asserts on every run that there is exactly one commit and no object under tasks/.

  2. 5 clean BE-001 runs. Superseded — they were not clean. See above.

  3. B0 arm of EXP-BE002-AGENTSMD-V3. Done — 10 runs, 8 pass / 2 F07 / 0 F02.

  4. Re-derive the MDE and register it before any B1 run exists. Done — Amendment 2. Cost 19% → 24%, tool calls 30% → 36%, cache 36% → 32%, duration unusable at 175%. The clean task is more variable than the contaminated pilot implied, so the experiment is less sensitive than it was sold as.

  5. Run the B1 arm and score it. Void — the B1 arm installed AGENTS.md, which Claude Code does not read. Both arms were baseline. The instructionsHash matched across all ten runs and certified a treatment that never reached the model.

  6. Commit and PR the working tree. Done — #30, #32, #33, #44 and benchmarks#8.

  7. Run the experiment for real — this is the next action. agents-md-v1 renamed to CLAUDE.md, as EXP-BE002-CLAUDEMD. The four predictions in the registration have never been tested and stand unmodified; §32 still forbids editing the instruction file in response to results. Budget ~$4 and ~40 minutes. Do not launch before the runtime guard (below) is in place, or the same failure is available again.

  8. Then stop fixing the instrument. Seven PRs' worth of hardening on 2026-08-09 was defensible and none of it was a result. Instrument work is unbounded and always more satisfying than an experiment that may conclude "no effect". Publish with the caveats that remain.

    The counter-argument, and the reason step 1 happened twice: two of those bugs were validity bugs, not polish, and the second was found only because someone re-checked a fix that had already been declared done. "Stop fixing the instrument" means stop polishing it. It does not mean stop verifying the claims it rests on — and the way to tell them apart is whether a bug changes what a number means or only how it looks.

Merged on 2026-08-09

PR
#21 F13/F15 excluded from aggregates, the §20 minimum, and Prometheus summaries
#22 comparison reports cache tokens (reads + creations) and vendor cost
#23 BE-002 pilot written up, incl. the criterion it missed
#24 stop tracking the Memtrace daemon's state and PID
#25 pre-registration of EXP-BE002-AGENTSMD
#26 analyze-experiment.py — fails closed, emits REJECT/KEEP/INCONCLUSIVE, 28 tests
#27 handoff rewritten after the answer-key finding
#28 strip the answer key from the agent's worktree
bench#7 BE-002

No PRs open.

Uncommitted, in the working tree — historical, this tree no longer exists

Historical. This section described a working tree as of 2026-08-10; that tree no longer exists and everything listed has since been committed and merged. Of the two items under "Not done", the nullable BehaviorDto counters landed as V4; the recorded UAT of the five questions was not checked when this notice was written. Kept for the record; do not act on the table as a to-do list.

Not yet committed or PR'd — do this before anything else:

file
runner/run-agent.sh build the agent's repo from an allowlist via git archive + git init; assert one commit and no tasks/ objects
runner/derive-mde.py re-derive the MDE from a measured arm; refuses a partial arm
runner/test_derive_mde.py 19 tests — the formula, the exclusions, the incomplete-arm gate
runner/analyze-experiment.py MDE table updated to Amendment 2 (cost bar 19% → 24%)
docs/preregistration-…md Void (2), the replacement key -V3, and Amendment 2
docs/STATE.md this file

Full suite green: make test-runner → 47 tests.

Not done: a recorded UAT of the five questions; making BehaviorDto's counters nullable so a telemetry gap stops being indistinguishable from a genuine zero.

M11 is decided — deferred to Chapter 01, written up in ADR-001. Chapter 00 closes at two runtimes. The short version: a third adapter adds integration surface rather than answers, the bottleneck is benchmark tasks and not runtimes, and an adapter no experiment exercises is where the seventh harness bug would hide.

Practical notes

  • make build-api before docker compose build — the image copies a host-built jar, so rebuilding without it silently ships a stale API. Cost a confused debugging round.
  • Claude/haiku BE-002 run on a clean tree: mean $0.204, median $0.190, SD $0.038, ~82–167 s (n=10). Budget ~$4.10 for a 10+10. The spread is wide enough that single runs mean nothing — the first clean run was $0.279 / 27 calls and looked like a discovery until nine more arrived.
  • make run-benchmark RUNTIME=claude MODEL=haiku BENCHMARK=BE-002 — Copilot's quota is exhausted; Claude is not rate-limited here.
  • Cheapest Copilot model is gpt-5.4-mini (0 premium requests, verified by probing).
  • Host ports are overridden in infra/.env (gitignored): Grafana 3001, API 8081, web 5174.
  • Claude runs have traceId: null by design (#20 refused to fake span parity), so the five-questions walkthrough needs a Copilot run.

The thing most worth remembering

Thirteen bugs have now made the harness measure something other than the agent. Six were found on 2026-08-09, and the two that mattered most were not found by reading code at all: #11 came from an outside reviewer plus a two-minute controlled test, and #12 from reading what four runs actually changed, and #13 from reading what five runs actually loaded. The list below covers 1–6; 7–13 are in the audit and in the void notices.

The pattern across most of them: an environmental block keeps being recorded as a capability failure. Quota → F03. Permission block → F05. Dropped connection → F03. Each time it was fixed for the specific cause rather than the class, and each time it came back wearing a different hat. That is why the guard added on 2026-08-09 keys on "produced no implementation" rather than on any particular reason for it — and why it caught #12 on the first batch it ran against.

#13 is a different pattern, and a worse one: the harness certified a fact it had never checked. skillsHash: null was read as "this run had no skills". It was actually the hash of a path that cannot exist in the worktree, so it could never have returned anything else — a field structurally incapable of reporting the problem it appeared to rule out. The runner had already closed the identical hole for MCP servers one line above, with a comment explaining exactly why user-scope config must not reach the agent.

Ask of any reassuring field: what value would it show if the thing it denies were happening? If the answer is "the same value", it is not evidence.

The six original bugs:

  1. AGENTS.md, installed by the runner, counted as an agent scope violation.
  2. An exhausted Copilot quota recorded as F03, incorrect code.
  3. A background daemon's .memdb/ files counted as unrelated production changes.
  4. An acceptance suite's compiled classes, surviving in gitignored target/, running as "the existing tests" on the next evaluation of the same worktree.
  5. The answer key shipped in the agent's worktree.
  6. The answer key stayed in the agent's git history after 5 was "fixed".

The first four all made runs look worse than they were. Five and six make them look better, and between them they survived every review the project has had — which is the lesson. Errors that flatter a result do not announce themselves; nobody investigates a pass.

Six adds a second lesson, and it is the sharper one. The evidence for five — a 35% drop in tool calls — was still there after five was fixed, and no one looked, because that drop already had a name. An explained symptom stops being evidence. A fix is not confirmed by the story that motivated it; it is confirmed by the symptom disappearing, or by an assertion that fails loudly when the leak returns. There is now such an assertion, and it runs on every run.

Disbelieve a dramatic number until it is explained, and disbelieve a flattering one twice. Then check that the explanation still predicts the number after you have acted on it.