bench(worldmodel): multi-step training works; control still does not move - #64
Merged
Conversation
…move
Ledger node 23. Node 22's named next test.
BLOCKER FOUND FIRST, and it is a finding: the corpus contains NO TRAJECTORIES.
generate() samples a fresh state per record (synthetic_1f.py:104), so record i's
post window is unrelated to record i+1's pre window. There was no multi-step
ground truth anywhere in the data. Node 22's test was not runnable as specified;
it needed generate(..., trajectory_length=K) and train_jepa(..., rollout_steps=K)
first.
M1 PASS defaults byte-identical, hash 8f74f84ac5aaebe2 -- a flag flip
M2 PASS post==pre within a trajectory, not across a boundary
M3 PASS oracle ladder bit-identical 100.0/91.5/73.5/37.0/34.5
M5 PASS h=8 rollout 0.02311 -> 0.00546, a 4.2x reduction (5.7x vs K=1 on the
same corpus). THE MECHANISM WORKS.
M4 FAIL pos 0.9725 against a registered floor of 0.9824.
M4 attributed rather than excused, by separating the two changes:
corpus K pos vel h=8
single-transition 1 0.9824 0.9740 0.02311
trajectory 1 0.9818 0.9709 0.03116
trajectory 6 0.9725 0.9896 0.00546
The corpus costs nothing; the multi-step loss trades -0.009 pos for +0.019 vel
and the rollout gain. A trade, not a regression -- but M4 was a hard floor.
AND NODE 22's OUTCOME GATE FAILS. Deployable predictor rollout, 3 seeds:
TARGET goal 36.3%, ONLINE goal 36.3%, against zero-action 37.0% and a gate of
"approach 76%". Cutting rollout error 5.7x moved control by nothing.
REVERTED per the registered rule, without tuning K. Also right on the merits: a
corpus mode and two parameters for no demonstrated benefit. Node 20 set the
precedent.
NODE 22's THREE-CONDITION MODEL IS NOW IN DOUBT. It predicted that treating the
last untreated condition would approach 76%. It was treated, measurably, and
control did not move. Two readings remain, and node 22's 2x2 cannot separate
them because its true-env arm used EXACT dynamics rather than merely better ones:
(a) the residual gap between 0.00546 and exact zero is still decisive, making
control a near-discontinuous function of forward-model error; or (b) a fourth
condition is unidentified.
Next test named and it adjudicates rather than adds a candidate: sweep control
against INJECTED rollout error, interpolating between the true env and the
learned predictor across 0, 0.00546 and 0.0231. One sweep, no new model.
Ledger-only. Verified: generator and forward_eval smoke tests pass; ledger and
ADR gates green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Node 22's named next test.
Blocker found first, and it is a finding
The corpus contains no trajectories.
generate()samples a fresh state per record (synthetic_1f.py:104), so record i's post window is unrelated to record i+1's pre window. There was no multi-step ground truth anywhere in the data — nothing to train a multi-step predictor against and nothing to validate one on.Node 22's test was not runnable as specified. It needed
generate(..., trajectory_length=K)andtrain_jepa(..., rollout_steps=K)first.Gates
8f74f84ac5aaebe2)pos≥ 0.9824M5 — the mechanism works
h=8 rollout error fell 5.7× against the same corpus at K=1. Multi-step training does exactly what it is supposed to do.
M4 — attributed, not excused
The corpus costs nothing. The multi-step loss trades −0.009
posfor +0.019veland the rollout gain — a trade, not a regression. But M4 was a registered hard floor and it was not met.And node 22's outcome gate fails
Deployable predictor rollout, 3 seeds:
Against zero-action's 37.0% and a gate of approach 76%. Cutting rollout error 5.7× moved control by nothing.
Reverted
Per the registered rule, without tuning K. Also right on the merits — a corpus mode and two parameters for no demonstrated benefit. Node 20 set the precedent.
synthetic_1f.pyandeeg_jepa.pyare unchanged on main; this node is the record.Node 22's three-condition model is now in doubt
It predicted that treating the last untreated condition would approach 76%. It was treated, measurably, and control did not move.
Two readings remain, and node 22's 2×2 cannot separate them because its true-env arm used exact dynamics rather than merely better ones:
Next test adjudicates rather than adds a candidate
Sweep control against injected rollout error, interpolating between the true env and the learned predictor across 0, 0.00546 and 0.0231. One sweep, no new model, and it settles a question five nodes have circled.
Ledger-only. Verified: generator and forward_eval smoke tests pass; ledger and ADR gates green.