fix(worldmodel): pin the episode set, per-episode CEM noise, paired endpoint - #55
Merged
Merged
Conversation
…ndpoint Items 3-5 of the benchmark repair. 1. EPISODE SET IS A PROPERTY OF THE BENCHMARK, NOT THE RUN. evaluate() passed seed = model_seed + 2 into _mpc_success, so every model seed drew a DIFFERENT episode set: "which episodes" was confounded with "which model". Measured previously, between-set difficulty carried sd 0.0853 -- about half the observed baseline swing -- and was being read as model variance. _mpc_success now takes `episode_seed` (default 2) and drops `seed` entirely. Episodes, baseline draws and planner noise all derive from it and nothing else. Result across 10 runs (2 latent dims x 5 seeds): random_baseline_success is 0.303 for EVERY run, against 0.254-0.483 before, and the episode sets are byte-identical. 2. A GENERATOR PER EPISODE. One torch.Generator consumed sequentially made the planning noise on episode i depend on how many draws episodes 0..i-1 had taken -- a function of n_samples * cem_iters * horizon. Any arm touching a CEM knob got different noise on the SAME episode, silently defeating pairing even with the episode set fixed. Nothing in the panel would have shown it. Nodes 10 and 11 swept exactly those knobs, so that sweep was not the paired comparison it appeared to be. 3. PAIRED CONTINUOUS ENDPOINT: per-episode log(d_final / d_start). The ratio rather than the difference, because episodes differ enormously in how far the goal starts and a raw difference is dominated by that. The log makes "halved the distance" one effect size wherever it happens. MEASURED, not simulated -- paired ld32 vs ld64 on identical episodes and identical planner noise: mean |t| binary 0.62 log-ratio 0.88 -> 1.43x median n for |t|=2: binary ~248 log-ratio ~154 A power simulation had suggested nearer 3-4x. The mechanism it identified is right -- only 3-8 of 20 pairs are discordant, which is why the binary endpoint is weak -- but the size was overstated, and 5 seeds is a noisy estimate of it. Reported here rather than repeated from the simulation. log_distance_ratio_mean is REPORTED, NOT ADOPTED as the adjudicator. Switching the primary endpoint in the same change that repins the episode set would confound the two; adoption is a separate pre-registered decision that still has to clear the blind criterion. final_distance_per_episode is also returned so a downstream paired analysis can compute either endpoint exactly instead of approximating the binary one from two marginal rates -- an approximation that overstated the advantage as 1.82x before it was replaced with the exact figure. NODE 14's success_rate VALUES NO LONGER REPRODUCE, by design: per-episode generators change planner noise. The episode set for model-seed 0 is unchanged (the old effective seed was 2 and the new fixed episode_seed is 2). Node 14's conclusions are untouched -- the frame ratio, Procrustes residual and one-frame remedy were all measured within a run, never across seeds. Recorded in a dated "Corrections to node 14, part 2" block rather than by editing node 14. Verified: smoke test passes on 3 consecutive runs and at 1 and 12 threads; ledger gate, ADR gate and both suites green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Items 3–5 of the benchmark repair.
1. The episode set is a property of the benchmark, not the run
evaluate()passedseed = model_seed + 2into_mpc_success, so every model seed drew a different episode set — "which episodes" was confounded with "which model". Between-set difficulty carried sd 0.0853, about half the observed baseline swing, and was being read as model variance._mpc_successnow takesepisode_seed(default 2) and dropsseedentirely. Episodes, baseline draws and planner noise all derive from it and nothing else.random_baseline_success2. A generator per episode
One
torch.Generatorconsumed sequentially made the planning noise on episode i depend on how many draws episodes 0…i−1 had taken — a function ofn_samples × cem_iters × horizon. Any arm touching a CEM knob got different noise on the same episode, silently defeating pairing even with the episode set fixed. Nothing in the panel would have shown it.Nodes 10 and 11 swept exactly those knobs. That sweep was not the paired comparison it appeared to be — not necessarily wrong, but its "within seed noise" reading rested on an assumption that did not hold.
3. Paired continuous endpoint —
log(d_final / d_start)The ratio, not the difference: episodes differ enormously in how far the goal starts, so a raw difference is dominated by episode difficulty. The log makes "halved the distance" one effect size wherever it happens.
Measured, not simulated — paired ld32 vs ld64 on identical episodes and identical planner noise:
1.43×, where a power simulation suggested nearer 3–4×. The mechanism the simulation identified is right — only 3–8 of 20 pairs are discordant, which is why binary is weak — but the size was overstated, and five seeds is a noisy estimate of it.
An earlier version of this measurement said 1.82×; that used an approximation of the paired binary sd from the two marginal rates.
final_distance_per_episodeis now returned so either endpoint can be computed exactly downstream, and the exact figure is 1.43×.log_distance_ratio_meanis reported, NOT adopted as the adjudicator. Switching the primary endpoint in the same change that repins the episode set would confound the two. Adoption is a separate pre-registered decision that still has to clear the blind criterion.Node 14's
success_ratevalues no longer reproduce — by designPer-episode generators change planner noise. The episode set for model-seed 0 is unchanged (the old effective seed was 2; the new fixed
episode_seedis 2), but every seed now scores that same set.Node 14's conclusions are untouched. The frame ratio, the Procrustes residual and the one-frame remedy were all measured within a run, never across seeds. Recorded in a dated
Corrections to node 14, part 2block rather than by editing node 14.Verification
Remaining
Raise n to ~200 — the last item, and now a config change rather than a design one.