bench(worldmodel): re-run the CEM-budget sweep on the repaired benchmark - #57
Merged
Conversation
Ledger node 16. Nodes 10 and 11 swept CEM knobs under the shared-generator defect, so their paired reading did not hold (node 15). Re-run with per-episode generators, a fixed episode set, n=200 and the continuous endpoint. NODE 11's CONCLUSION SURVIVES. More CEM budget does not lift control: 0/5 seeds reach paired |t| = 2 on either endpoint in either mode. ITS POINT ESTIMATE DID NOT. Node 11 reported default 0.342+/-0.151 vs tuned 0.325+/-0.144, delta -0.017 -- tuned slightly WORSE. Repaired: signal default 0.335+/-0.011 tuned 0.360+/-0.016 delta +0.025 nuisance default 0.337+/-0.023 tuned 0.354+/-0.025 delta +0.017 The +/-0.15 node 11 called seed noise was mostly the benchmark's own defects -- the spread is now 14x tighter -- and the direction it reported was noise. A conclusion can be right while every number supporting it is not. AND THERE IS A WORSE FINDING UNDERNEATH. log(d_final/d_start) is POSITIVE in all 20 runs: +0.108/+0.082 (signal default/tuned), +0.128/+0.097 (nuisance). The planner ends 9-14% FARTHER from the goal than it started. Against trivial policies on the same episodes -- zero-action +0.034/+0.029, random-action +0.046/+0.060 -- the planner is WORSE THAN DOING NOTHING. Paired per-episode, planner minus zero-action is positive in 6/6 mode x seed comparisons (+0.055 to +0.155, sign test p ~ 0.03), though only 1/6 individually clears |t| = 2. Same direction against random-action, 6/6. This reframes nodes 11-15. They asked why the planner fails to beat chance; that is too generous. An UNINFORMATIVE planner scores like the zero policy. This one is consistently worse, which means something is systematically pointing it away from the goal -- a directional defect, not a power problem. Node 16 lists the candidates in order and names the cheapest discriminating test (sign/axis convention between the latent displacement and the action applied in _step) rather than proposing another sweep. Node 14's one-frame remedy did not fix control, so the frame offset alone is not sufficient to explain it. Ledger-only change. Verified: smoke test 2 consecutive runs; ledger gate and ADR gate green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ledger node 16. Nodes 10 and 11 swept CEM knobs under the shared-generator defect, so their paired reading did not hold. Re-run with per-episode generators, a fixed episode set, n=200 and the continuous endpoint.
Node 11's conclusion survives. Its point estimate does not.
More CEM budget does not lift control — 0/5 seeds reach paired |t| = 2 on either endpoint in either mode. That part holds.
The ±0.15 node 11 called seed noise was mostly the benchmark's own defects — the spread is now 14× tighter — and the direction it reported was noise. A conclusion can be right while every number supporting it is not.
And there is a worse finding underneath
log(d_final/d_start)is positive in all 20 runs: +0.108/+0.082 (signal default/tuned), +0.128/+0.097 (nuisance). The planner ends 9–14% farther from the goal than it started.Against trivial policies on the same episodes:
The planner is worse than doing nothing. Paired per-episode, planner minus zero-action is positive in 6/6 mode × seed comparisons (+0.055 to +0.155, sign test p ≈ 0.03) — though only 1/6 individually clears |t| = 2, so this is a consistent direction rather than an individually significant effect. Same direction against random-action, 6/6.
Why this reframes nodes 11–15
They asked why the planner fails to beat chance. That framing is too generous. An uninformative planner scores like the zero policy. This one is consistently worse, which means something is systematically pointing it away from the goal.
That is a directional defect, not a power problem — and it is cheap to localise, which the last five nodes of sweeping were not.
Node 16 lists candidates in order and names the cheapest discriminating test — sign/axis convention between the latent displacement and the action applied in
_step— rather than proposing another sweep. Note that node 14's one-frame remedy did not fix control, so the frame offset alone does not explain it.Scope
Ledger-only change; no code. Verified: smoke test 2 consecutive runs, ledger gate and ADR gate green.