Skip to content

bench(worldmodel): re-run the CEM-budget sweep on the repaired benchmark - #57

Merged
aurascoper merged 1 commit into
mainfrom
bench/rerun-cem-budget
Aug 2, 2026
Merged

bench(worldmodel): re-run the CEM-budget sweep on the repaired benchmark#57
aurascoper merged 1 commit into
mainfrom
bench/rerun-cem-budget

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Ledger node 16. Nodes 10 and 11 swept CEM knobs under the shared-generator defect, so their paired reading did not hold. Re-run with per-episode generators, a fixed episode set, n=200 and the continuous endpoint.

Node 11's conclusion survives. Its point estimate does not.

More CEM budget does not lift control — 0/5 seeds reach paired |t| = 2 on either endpoint in either mode. That part holds.

default(3,64) tuned(8,192) delta
node 11 (as recorded) 0.342 ±0.151 0.325 ±0.144 −0.017
repaired, signal 0.335 ±0.011 0.360 ±0.016 +0.025
repaired, nuisance 0.337 ±0.023 0.354 ±0.025 +0.017

The ±0.15 node 11 called seed noise was mostly the benchmark's own defects — the spread is now 14× tighter — and the direction it reported was noise. A conclusion can be right while every number supporting it is not.

And there is a worse finding underneath

log(d_final/d_start) is positive in all 20 runs: +0.108/+0.082 (signal default/tuned), +0.128/+0.097 (nuisance). The planner ends 9–14% farther from the goal than it started.

Against trivial policies on the same episodes:

policy signal nuisance
zero-action +0.034 +0.029
random-action +0.046 +0.060
planner +0.108 +0.128

The planner is worse than doing nothing. Paired per-episode, planner minus zero-action is positive in 6/6 mode × seed comparisons (+0.055 to +0.155, sign test p ≈ 0.03) — though only 1/6 individually clears |t| = 2, so this is a consistent direction rather than an individually significant effect. Same direction against random-action, 6/6.

Why this reframes nodes 11–15

They asked why the planner fails to beat chance. That framing is too generous. An uninformative planner scores like the zero policy. This one is consistently worse, which means something is systematically pointing it away from the goal.

That is a directional defect, not a power problem — and it is cheap to localise, which the last five nodes of sweeping were not.

Node 16 lists candidates in order and names the cheapest discriminating test — sign/axis convention between the latent displacement and the action applied in _step — rather than proposing another sweep. Note that node 14's one-frame remedy did not fix control, so the frame offset alone does not explain it.

Scope

Ledger-only change; no code. Verified: smoke test 2 consecutive runs, ledger gate and ADR gate green.

Ledger node 16. Nodes 10 and 11 swept CEM knobs under the shared-generator
defect, so their paired reading did not hold (node 15). Re-run with per-episode
generators, a fixed episode set, n=200 and the continuous endpoint.

NODE 11's CONCLUSION SURVIVES. More CEM budget does not lift control: 0/5 seeds
reach paired |t| = 2 on either endpoint in either mode.

ITS POINT ESTIMATE DID NOT. Node 11 reported default 0.342+/-0.151 vs tuned
0.325+/-0.144, delta -0.017 -- tuned slightly WORSE. Repaired:

  signal    default 0.335+/-0.011   tuned 0.360+/-0.016   delta +0.025
  nuisance  default 0.337+/-0.023   tuned 0.354+/-0.025   delta +0.017

The +/-0.15 node 11 called seed noise was mostly the benchmark's own defects --
the spread is now 14x tighter -- and the direction it reported was noise. A
conclusion can be right while every number supporting it is not.

AND THERE IS A WORSE FINDING UNDERNEATH.

log(d_final/d_start) is POSITIVE in all 20 runs: +0.108/+0.082 (signal
default/tuned), +0.128/+0.097 (nuisance). The planner ends 9-14% FARTHER from
the goal than it started.

Against trivial policies on the same episodes -- zero-action +0.034/+0.029,
random-action +0.046/+0.060 -- the planner is WORSE THAN DOING NOTHING. Paired
per-episode, planner minus zero-action is positive in 6/6 mode x seed
comparisons (+0.055 to +0.155, sign test p ~ 0.03), though only 1/6 individually
clears |t| = 2. Same direction against random-action, 6/6.

This reframes nodes 11-15. They asked why the planner fails to beat chance; that
is too generous. An UNINFORMATIVE planner scores like the zero policy. This one
is consistently worse, which means something is systematically pointing it away
from the goal -- a directional defect, not a power problem.

Node 16 lists the candidates in order and names the cheapest discriminating test
(sign/axis convention between the latent displacement and the action applied in
_step) rather than proposing another sweep. Node 14's one-frame remedy did not
fix control, so the frame offset alone is not sufficient to explain it.

Ledger-only change. Verified: smoke test 2 consecutive runs; ledger gate and ADR
gate green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
@aurascoper
aurascoper merged commit c39f595 into main Aug 2, 2026
2 checks passed
@aurascoper
aurascoper deleted the bench/rerun-cem-budget branch August 2, 2026 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant