Skip to content

feat(ml): thread primitive + train_hslm windows on N threads, bit-identical at any count (#10876) - #10896

Merged
gHashTag merged 8 commits into
masterfrom
claude/hslm-threads
Oct 19, 2026
Merged

feat(ml): thread primitive + train_hslm windows on N threads, bit-identical at any count (#10876)#10896
gHashTag merged 8 commits into
masterfrom
claude/hslm-threads

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 19, 2026

Copy link
Copy Markdown
Owner

Closes #10876

What

Lever (a) of #10850: the 66 windows of one tr_train_step_k step run on N worker threads instead of one.

  • Compiler (thin Rust): threadlocal var form and thread_spawn / thread_join / thread_cores builtins for --backend ml (bootstrap/src/compiler.rs, +40/-47, dead code deleted; FROZEN_HASH moved).
  • Spec proving the primitive: specs/ml/thread_model.t27 (6 tests; parts on N threads sum bit-identically to the fixed-order sum, per-thread state starts at its initializer and does not leak).
  • specs/ml/train_hslm.t27: cfg key threads (default = cores, threads=1 is the serial path). Each window writes its own gradient slot in win_g; slots are summed in window order, so the result does not depend on the thread count. nn/hslm state and the b3_* scratch arrays are threadlocal. New test one_two_and_three_threads_give_one_step_the_same_bits_at_speed_2_and_3.

Association change (stated, not hidden)

The old step kept one running sum over windows. The new step sums per-window slots in fixed window order. Floating-point addition is not associative, so:

  • at k = 1 window per step the result is bit-equal to the old one;
  • at k > 1 it differs from the old binary: state hash 240eb9893ad8eba1 (old) vs a9f594b18f9fc17e (new). Loss agrees to 4 digits (train 6.5916, valid 6.5637) and the .bin hash is the same.

The new value is the same for every thread count, which is the point: determinism no longer depends on the machine.

Lab proof (Railway lab, 48 cores, 8 steps, seed 1729, loaded: load average 55-72 during the runs, so these are floors)

State hash after 8 steps, sha256(ckpt_8.state)[:16]: a9f594b18f9fc17e at threads = 1, 2, 4, 8, 16, 24, 48, at speed 2 and at speed 3. Loss identical at every count.

threads speed=2 tok/s speed=3 tok/s
old binary 244.6 484.1
1 238.1 / 245.6 407.3
2 470.8 773.1
4 750.7 1312.7
8 1095.8 1820.7
16 1322.4 1960.9
24 1488.1 1730.1
48 1166.6 1033.8

Speed-up at 16 threads: 5.4x at speed 2, 4.1x vs the old speed-3 binary. Past 24 threads the lab is oversubscribed (load > 48) and throughput falls.

Known costs

  • threads=1 at speed=3 is about 16% slower than the old binary (407 vs 484 tok/s): thread-local access on the b3_* scratch. At speed 2 it is level (238-246 vs 245). Not fixed here; a follow-up.
  • A Debug build wrote undefined as 0xaa into .data, 1.1 GB for win_g. win_g now starts from a zero initializer (.bss), .data is 178 MB.
  • specs/ml/eval_hslm.t27 is BLOCKED (use of undeclared identifier 'b3_q'). Inherited from spec(ml): train_hslm speed=3 -- one matrix over the 27 positions, bit-identical to speed=2 (Closes #10849) #10866 (it fails at this PR's base too), not caused here; follow-up.

Stack

This branch contains the commits of #10857 (issue #10850) until that merges; the diff of this PR alone is the thread work.

Tests on the lab: thread_model 6/6, train_hslm 30/30, t27t 17/17, ngram_hslm 8/8, all with 0 vacuous passes. Seals saved for ml_train_hslm and ml_thread_model.

🤖 Generated with Claude Code

gHashTag and others added 5 commits October 19, 2026 12:38
@gHashTag @claude
wip(train_hslm): profile=1 times the parts of a training step
f751d6a 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag @claude
wip(train_hslm): profile=2 reads the thread CPU clock
83fcc96 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag @claude
wip(t27t): the t27c CPU runner, its measured split and four lever bou…
9432608 
…nds (#10850)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag @claude
seal(ml): re-seal train_hslm with the profile; ledger row blocker is …
06968d7 
…now std.os.linux.timespec (#10850)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag @claude
wip(ml): threadlocal var + thread_spawn/join, train_hslm windows on N…
763f6b2 
… threads (#10876)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
Merged
Merged
Merged
This was referenced Oct 19, 2026
Closed
Merged
This was referenced Oct 19, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
@github-actions

github-actions Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-19 06:38:10 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 46
PRs with All Checks Green 4
READY 3
FAILING 46
PENDING 0
NO CHECKS YET 0

These columns do not partition: 3 + 46 + 0 + 0 = 49, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=509db03c8f4d != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions github-actions Bot mentioned this pull request Oct 19, 2026
Merged
@gHashTag
gHashTag enabled auto-merge (squash) October 19, 2026 06:45
This was referenced Oct 19, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
This was referenced Oct 19, 2026
Open
Draft
@gHashTag gHashTag mentioned this pull request Oct 19, 2026
Closed
7 of 8 tasks
This was referenced Oct 19, 2026
Merged
Merged
gHashTag and others added 2 commits October 19, 2026 14:41
@gHashTag @claude
Merge master into claude/hslm-threads (#10857 landed as a squash; tra…
ea22d60 
…in_hslm.t27 and its seal kept at the thread version)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@gHashTag
Merge remote-tracking branch 'origin/master' into claude/hslm-threads
2137ded
@github-actions github-actions Bot mentioned this pull request Oct 19, 2026
Merged
@github-actions

github-actions Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-19 07:57:04 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 44
PRs with All Checks Green 6
READY 1
FAILING 44
PENDING 0
NO CHECKS YET 0

These columns do not partition: 1 + 44 + 0 + 0 = 45, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=509db03c8f4d != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

This was referenced Oct 19, 2026
Draft
Merged
@github-actions

github-actions Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-19 08:08:19 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 45
PRs with All Checks Green 5
READY 2
FAILING 45
PENDING 0
NO CHECKS YET 0

These columns do not partition: 2 + 45 + 0 + 0 = 47, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=509db03c8f4d != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions github-actions Bot mentioned this pull request Oct 19, 2026
Merged
@gHashTag
gHashTag merged commit 356e7ce into master Oct 19, 2026
30 of 38 checks passed
This was referenced Oct 19, 2026
Closed
Merged
Closed
gHashTag added a commit that referenced this pull request Oct 19, 2026
@gHashTag @claude
compiler.rs: delete the tests that call the seven items #10896 remove…
fe414dd 
…d, move the seal (Closes #10957)

#10896 dropped Node::with_child, Parser::check_peek, icarus_lowerability_reason,
align_address, reset_end_cycle, path_delay and worst_path_delay to keep the
foreign-line budget net-negative, but eight test lines still called five of them,
so the unit tests no longer compiled. Delete those tests; nothing else changes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@gHashTag gHashTag mentioned this pull request Oct 19, 2026
Merged
gHashTag added a commit that referenced this pull request Oct 19, 2026
@gHashTag @claude
spec(ml): speed=4 scratch is threadlocal and the speed-4 step is bit-…
a9e20bf 
…identical at 1 and 3 threads (#10875)

Master's thread primitive (#10896) makes window scratch threadlocal; the
speed-4 buffers b4_q, b4_s and b4_wt follow. New test checks the step digest
at 1 and 3 threads at speed 4, and that it differs from speed 2. Re-sealed on
the lab (37 tests, 0 vacuous, all hashes MATCH).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
gHashTag added a commit that referenced this pull request Oct 19, 2026
@gHashTag @claude
zig emit: a spec that defines its own thread_spawn/thread_join/thread…
382265a 
…_cores does not get the thread runtime appended (Closes #10957)

#10896 appended the thread runtime to any Zig output that calls thread_spawn,
thread_join or thread_cores, so compiler/runtime/runtime.t27 (which defines and
calls its own thread_spawn) changed its gen_hash_zig and the triruntime and
runtime_triruntime seals drifted (coverage red). Guard the detection with
declared_fns, as the call rewrite already is. One line changed, none added.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
gHashTag added a commit that referenced this pull request Oct 19, 2026
@gHashTag @claude
spec(ml): train_hslm speed=4 -- int8 trit forward, error bounded agai…
c736177 
…nst speed=2 (Closes #10875) (#10908)

* spec(ml): train_hslm speed=4 -- int8 trit forward, one scale per row, error bounded against speed=2 (Closes #10875)

speed=4 is speed 3's window with every forward ternary matmul in integers:
activations rounded to int8 with one scale per row, trits transposed once per
call, sixteen independent i32 accumulators, output = acc * scale. The backward
stays speed 3's f64 code. Tests bound the kernel error (proved per-output
bound), logits, k-position loss and gradients against speed 2, and re-derive
the lab loss-curve gap and tok/s from recorded constants.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* seal(ml): restore master's train_hslm seal so the master merge is conflict-free (#10875)

The seal is regenerated on the lab in the next commit, after the merge.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* spec(ml): speed=4 scratch is threadlocal and the speed-4 step is bit-identical at 1 and 3 threads (#10875)

Master's thread primitive (#10896) makes window scratch threadlocal; the
speed-4 buffers b4_q, b4_s and b4_wt follow. New test checks the step digest
at 1 and 3 threads at speed 4, and that it differs from speed 2. Re-sealed on
the lab (37 tests, 0 vacuous, all hashes MATCH).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
gHashTag added a commit that referenced this pull request Oct 19, 2026
@gHashTag @claude
compiler.rs: delete the tests calling the seven items #10896 removed (C…
027a0e2 
…loses #10957) (#10958)

* compiler.rs: delete the tests that call the seven items #10896 removed, move the seal (Closes #10957)

#10896 dropped Node::with_child, Parser::check_peek, icarus_lowerability_reason,
align_address, reset_end_cycle, path_delay and worst_path_delay to keep the
foreign-line budget net-negative, but eight test lines still called five of them,
so the unit tests no longer compiled. Delete those tests; nothing else changes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* zig emit: a spec that defines its own thread_spawn/thread_join/thread_cores does not get the thread runtime appended (Closes #10957)

#10896 appended the thread runtime to any Zig output that calls thread_spawn,
thread_join or thread_cores, so compiler/runtime/runtime.t27 (which defines and
calls its own thread_spawn) changed its gen_hash_zig and the triruntime and
runtime_triruntime seals drifted (coverage red). Guard the detection with
declared_fns, as the call rewrite already is. One line changed, none added.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ml backend: threadlocal var + thread_spawn/join, and train_hslm windows on N threads with a fixed-order gradient sum

1 participant

@gHashTag