Skip to content

bench(worldmodel): multi-step training works; control still does not move - #64

Merged
aurascoper merged 1 commit into
mainfrom
feat/multi-step-predictor
Aug 3, 2026
Merged

bench(worldmodel): multi-step training works; control still does not move#64
aurascoper merged 1 commit into
mainfrom
feat/multi-step-predictor

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Node 22's named next test.

Blocker found first, and it is a finding

The corpus contains no trajectories. generate() samples a fresh state per record (synthetic_1f.py:104), so record i's post window is unrelated to record i+1's pre window. There was no multi-step ground truth anywhere in the data — nothing to train a multi-step predictor against and nothing to validate one on.

Node 22's test was not runnable as specified. It needed generate(..., trajectory_length=K) and train_jepa(..., rollout_steps=K) first.

Gates

gate result
M1 defaults byte-identical (8f74f84ac5aaebe2) ✅ a flag flip, incumbent reproducible
M2 post==pre within a trajectory, not across ✅ both directions
M3 oracle ladder bit-identical ✅ 100.0 / 91.5 / 73.5 / 37.0 / 34.5
M5 multi-step error must improve 0.02311 → 0.00546, 4.2×
M4 pos ≥ 0.9824 0.9725

M5 — the mechanism works

h=8 rollout error fell 5.7× against the same corpus at K=1. Multi-step training does exactly what it is supposed to do.

M4 — attributed, not excused

corpus K pos vel h=8
single-transition (baseline) 1 0.9824 0.9740 0.02311
trajectory 1 0.9818 0.9709 0.03116
trajectory 6 0.9725 0.9896 0.00546

The corpus costs nothing. The multi-step loss trades −0.009 pos for +0.019 vel and the rollout gain — a trade, not a regression. But M4 was a registered hard floor and it was not met.

And node 22's outcome gate fails

Deployable predictor rollout, 3 seeds:

goal latent mean node 22
TARGET 36.3% 35.5%
ONLINE (one-frame) 36.3% 35.0%

Against zero-action's 37.0% and a gate of approach 76%. Cutting rollout error 5.7× moved control by nothing.

Reverted

Per the registered rule, without tuning K. Also right on the merits — a corpus mode and two parameters for no demonstrated benefit. Node 20 set the precedent. synthetic_1f.py and eeg_jepa.py are unchanged on main; this node is the record.

Node 22's three-condition model is now in doubt

It predicted that treating the last untreated condition would approach 76%. It was treated, measurably, and control did not move.

Two readings remain, and node 22's 2×2 cannot separate them because its true-env arm used exact dynamics rather than merely better ones:

  • (a) the residual gap between 0.00546 and exact zero is still decisive — control would be a near-discontinuous function of forward-model error, and the 76% arm only ever reachable with perfect dynamics
  • (b) a fourth condition is unidentified

Next test adjudicates rather than adds a candidate

Sweep control against injected rollout error, interpolating between the true env and the learned predictor across 0, 0.00546 and 0.0231. One sweep, no new model, and it settles a question five nodes have circled.

Ledger-only. Verified: generator and forward_eval smoke tests pass; ledger and ADR gates green.

…move

Ledger node 23. Node 22's named next test.

BLOCKER FOUND FIRST, and it is a finding: the corpus contains NO TRAJECTORIES.
generate() samples a fresh state per record (synthetic_1f.py:104), so record i's
post window is unrelated to record i+1's pre window. There was no multi-step
ground truth anywhere in the data. Node 22's test was not runnable as specified;
it needed generate(..., trajectory_length=K) and train_jepa(..., rollout_steps=K)
first.

  M1 PASS  defaults byte-identical, hash 8f74f84ac5aaebe2 -- a flag flip
  M2 PASS  post==pre within a trajectory, not across a boundary
  M3 PASS  oracle ladder bit-identical 100.0/91.5/73.5/37.0/34.5
  M5 PASS  h=8 rollout 0.02311 -> 0.00546, a 4.2x reduction (5.7x vs K=1 on the
           same corpus). THE MECHANISM WORKS.
  M4 FAIL  pos 0.9725 against a registered floor of 0.9824.

M4 attributed rather than excused, by separating the two changes:

  corpus              K   pos      vel      h=8
  single-transition   1   0.9824   0.9740   0.02311
  trajectory          1   0.9818   0.9709   0.03116
  trajectory          6   0.9725   0.9896   0.00546

The corpus costs nothing; the multi-step loss trades -0.009 pos for +0.019 vel
and the rollout gain. A trade, not a regression -- but M4 was a hard floor.

AND NODE 22's OUTCOME GATE FAILS. Deployable predictor rollout, 3 seeds:
TARGET goal 36.3%, ONLINE goal 36.3%, against zero-action 37.0% and a gate of
"approach 76%". Cutting rollout error 5.7x moved control by nothing.

REVERTED per the registered rule, without tuning K. Also right on the merits: a
corpus mode and two parameters for no demonstrated benefit. Node 20 set the
precedent.

NODE 22's THREE-CONDITION MODEL IS NOW IN DOUBT. It predicted that treating the
last untreated condition would approach 76%. It was treated, measurably, and
control did not move. Two readings remain, and node 22's 2x2 cannot separate
them because its true-env arm used EXACT dynamics rather than merely better ones:
(a) the residual gap between 0.00546 and exact zero is still decisive, making
control a near-discontinuous function of forward-model error; or (b) a fourth
condition is unidentified.

Next test named and it adjudicates rather than adds a candidate: sweep control
against INJECTED rollout error, interpolating between the true env and the
learned predictor across 0, 0.00546 and 0.0231. One sweep, no new model.

Ledger-only. Verified: generator and forward_eval smoke tests pass; ledger and
ADR gates green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
@aurascoper
aurascoper merged commit 59276d3 into main Aug 3, 2026
2 checks passed
@aurascoper
aurascoper deleted the feat/multi-step-predictor branch August 3, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant