Skip to content

docs: layerdrop rung passes the subset gate (3.8088) #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Aug 30, 2026
Merged

docs: layerdrop rung passes the subset gate (3.8088) #89

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,3 +368,22 @@ int8; int8 is the client default. E2 (Modal 4-vCPU / 8 GiB — the production se
Server vs node-laptop tier: cold load 4s vs 13s session create, decode
~2.5-3x faster — the serving tier trades network for speed. E3 (browser
WASM/WebGPU) pending.

## ara-diac-small-layerdrop — the depth-cut rung PASSES on the subset (2026-08-30)

Encoder 12->6 from pretrained ByT5-small, layers copied VERBATIM
(no projection - the width-cut rungs failed at 74.68/82.96), Muon,
same clean r6 labels, 10,995 steps, final CE 0.016 (the scratch rung
converged near 0.9 - 50x lower train loss at the same step count).
300-paragraph subset gate (teacher reproduces 1.3205):

| rung | params | subset DER-CE |
|---|---|---|
| ByT5-small 1.0 (AdamW, full) | 300M | 3.658 |
| **layerdrop (enc 6, Muon)** | ~190M | **3.8088** |

Halving encoder depth costs 0.15pp on the subset - the pretrained
representation survives a depth cut that width surgery destroyed.
Full-set gate (1,200 paragraphs) in flight; int8 ~190MB, int4 ~95MB
(the browser-budget artifact). Survived two infra failures en route
(eviction without watchdog; a regressed d_kv derivation) - both fixed.
Loading