Repository navigation
spec(ml): train_hslm -- main() trains the whole 3-block HSLM with STE and LAMB, writes MSLH checkpoints (Refs #10372) - #10387
Merged
Merged
Conversation
… and LAMB, writes MSLH checkpoints (Refs #10372) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the flat vector, xorshift guards zero (Refs #10372) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gHashTag
added a commit
that referenced
this pull request
Oct 18, 2026
…d specs (Refs #10387) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
PR DashboardGenerated at: 2026-10-18 10:54:52 UTC
Summary
Seal Status
|
Contributor
PR DashboardGenerated at: 2026-10-18 11:01:23 UTC
Summary
Seal Status
|
This was referenced Oct 18, 2026
Merged
Refs #10372 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
PR DashboardGenerated at: 2026-10-19 02:00:14 UTC
Summary
Seal Status
|
This was referenced Oct 19, 2026
Merged
Merged
gHashTag
marked this pull request as ready for review
October 19, 2026 03:33
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #10372
What
specs/ml/train_hslm.t27. Itsmain()trains the whole 3-block HSLM fromnn/hslm(ctx 27, vocab 729, embed 243, ff 729, 3 heads), on the real HTOK corpus from #10380.It reuses these through
use, without copying them:nn/hslm: forward, backward and cross-entropynn/ternary_ste:quantize_twnnn/hslm_data: the MSLH header and layoutWhat the spec adds:
lambStepSlice.total. Global-norm clip 1.0. Batch is 66 windows of 27 tokens, with starts drawn from a pinned-seed xorshift64./tmp/hslm/train.cfgis optional and holdskey=valuelines. Keys:steps total warmup eval_every ckpt_every seed speed resume synth eval_windows loss_positions gradcheck. Defaults: 32000 / 32000 / 2000 / 500 / 2000 / 1729 / 2 / 0 / 0 / 243 / 27 / 0.eval_everysteps, one line goes to stdout. It carries the step, the minibatch train loss and its ppl, the mean train loss since the last line, the held-out loss over 243 fixed evenly spacedvalid.tokwindows and its ppl, tok/s, s/step, lr and the pre-clip gnorm.ckpt_everysteps and at the end:/tmp/hslm/ckpt_<step>.binis MSLH: thenn/hslm_dataheader plus its layout, 4971796 bytes./tmp/hslm/ckpt_<step>.stateholds the full resumable state, because MSLH has no attention matrices, embedding or gains. It is a 64-byteT27Sheader (step, rng, beta powers, loss), then p, m and v as f64 LE: 51123376 bytes.main()assert, which gives a nonzero exit.resume=<step>restarts from the.statefile and continues bit-identically.Speed: same model, honest kernels
nn/hslmreference kernels as they areternary_matmulas a branch-free row axpy (bit-identical);trit_intandouter_addbranch-free;ternary_matmul_t_addwith 4 partial sums (backward only)50-step measurement (real tokens, speed 2, seed 1729)
Data: train.tok has 8557893 tokens and valid.tok has 7689354.
I rebuilt at 47aff93 (the duplicate-body fix) and reran the 50 steps. Every printed loss matched to the last digit. This run measured 1.534 s/step and 1161 tok/s while test-report shared the box.
After 50 steps the learning rate is still only 2.5e-5 of a 2000-step warmup, so the held-out loss moves just 6.5952 -> 6.5916; ln 729 = 6.5917.
The checkpoints at steps 25 and 50 were written and read back byte-equal.
Resume check: I ran again with
resume=25to step 50. The resultingckpt_50.binandckpt_50.statematched the uninterrupted run undercmp.Tests
t27c test-report specs/ml/train_hslm.t27gives 20/20 pass, 0 vacuous: the 14 below plus the 6 for the 27-position objective (next section). The original 14:Differences from R5 / the Zig trainer (trinity-training @ 3e9648c)
loss_positionspositions (default 27;loss_positions=1is R5's last-position objective), each position's with earlier K/V treated as constants (nn/hslmsection 5).nn/hslm's hash.encode_f32.Lab commands (Railway fs-lab, t27c at /root/t27, zig 0.16.0)
/root/t27-hslm-train/train_hslm(sha256 549bb065...334c, built from 47aff93).train.cfg, the defaults run all 32000 steps.The 27-position objective (3b5c87a, owner's choice 2026-10-18)
Why. A held-out eval (
specs/ml/eval_hslm.t27, #10467) showed that the last-position objective made the full-sequence loss worse. Between steps 2000 and 4000, last_loss went 5.75 -> 4.50 while full_loss rose 6.20 -> 6.54.What changed.
loss_positionstakes 1..27, default 27.loss_positions=1is the old step, to the bit.objective last27 train_loss_last27 .. train_ppl_last27 .. avg_loss_last27 ...valid_loss/valid_pplare still the last-position number, so they compare with run 1.valid_full_loss/valid_full_pplare new: the mean over all 27 positions of the same 243 held-out windows.tr_window_forward,tr_window_loss,tr_valid_lossand the checkpoint reader are unchanged.eval_hslm.t27at a6d8af0 still gives 10/10 tests against the new train_hslm and builds with--backend ml.New tests (6).
loss_positions_one_is_the_last_position_step_to_the_bit: a k=1 step equals the old step, with 0 of 2130138 parameters differing. A k=27 step moves more than 10^6 of them.the_27_position_loss_is_the_mean_of_27_forward_only_cross_entropies: checked for k = 27, 7 and 1, with gap < 1e-12. The measured gap is 0.uniform_logits_give_ln_729_at_every_position: with the head zeroed, the loss equals ln 729 = 6.591673732009 within 1e-12. The output-bias gradient equals 1/729 - count_j/27 within 1e-15 (measured 3.5e-17).Gradient check. It uses central differences with h = 1e-4 on window 700 of a seed-5 synthetic corpus, at speed 2. The bound is relative error < 1e-4, and the gradient magnitude must be > 1e-7 so the check is not vacuous.
fd_stopmoves the parameter only inside the scored position's forward, which matches the documented gradient (earlier K/V held).fd_truemoves it for the whole window.nn/hslmsection 5.Negative controls. Each mutates one line of
tr_window_grad_k. For each, test-report ran and the file was restored (sha256 checked).tokens[start+p+1]->tokens[start+p]p + k >= 27->p + k > 27Seal.
t27c seal --savewrote.trinity/seals/ml_train_hslm.json(spec sha256 fa8490d8...).seal --verifyreports all hashes MATCH: spec, zig, verilog, c, rust.Speed. This was measured over 50 real-token steps on the lab, seed 1729, speed 2, while run 1 shared the box.
At 50 steps of the 27-position run,
valid_losswas 6.5930 andvalid_full_loss6.6055. Step 0 gave the same 6.5952 as run 1.Run 2 (started 2026-10-18 14:03:44 UTC):
loss_positions=27 steps=8000 total=8000 seed=1729 eval_every=500 ckpt_every=1000; warmup, lr and batch are unchanged.ckpt_every=1000gives 8 checkpoints. 2000 would give only 4.run.log,ckpt_{2000,4000,6000}andeval_{2000,4000}.txtare kept in/tmp/hslm/run1_lastpos/.Nothing needed a compiler change. Host I/O uses the existing
host_open/read/write/close, with a NUL-terminated path buffer for the checkpoint names, and the clock is read from/proc/uptime.🤖 Generated with Claude Code