feat(distill): layer-drop bridge + ara-diac-small-layerdrop rung - #77
Merged
Conversation
added 2 commits
August 29, 2026 12:59
Both width-cut rungs failed (scratch 74.68, naive SVD stitch 82.96) - every matrix surgically damaged. Depth instead: encoder 12->6, width untouched, surviving layers copied verbatim; Muon per E3; int4 target ~75MB browser artifact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The third attack on the browser tier, motivated by the two failures:
width cuts damage every projection; a depth cut damages none of the
survivors. layer_drop_state copies every 2nd encoder layer verbatim
(12->6, decoder untouched, full d1472 width) from pretrained
ByT5-small; spec runs with Muon (E3) on the same clean labels; int4
export target ~75MB. Paired with the gap-closing plan: G1 = ship the
already-trained Muon student (run-005-muon best, 4.83 DER) as
ara-diac-small-2.0; G2 = label-scale rung for the 2.25pp residual.