Skip to content

paper: teacher at greedy — same-protocol shrink +1.60pp #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Aug 25, 2026
Merged

paper: teacher at greedy — same-protocol shrink +1.60pp #49

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/paper.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ All fp32 artifacts measured 0.0pp parity (torch reference vs ONNX, ≥500 sample

=== Capacity costs are real and are disclosed

The Hebrew student met its pre-registered budget (+5.58pp against a ≈5.6pp allowance, the widened budget reflecting the teacher's own headroom). The Arabic client student missed its strict +0.5pp budget by +2.34pp and ships with that number in its metadata, its model card, and the results log. The measured pattern across languages: ByT5-small → ByT5-base shrink costs are single-digit percentage points on diacritization but larger on G2P (Thai +7.63pp beam-4 / teacher-relative 2.85% greedy). Users choosing a tier choose a measured trade, not a mystery box.
The Hebrew student met its pre-registered budget (+5.58pp against a ≈5.6pp allowance, the widened budget reflecting the teacher's own headroom). The Arabic client student missed its strict +0.5pp budget by +2.34pp and ships with that number in its metadata, its model card, and the results log. The measured pattern across languages: ByT5-small → ByT5-base shrink costs are single-digit percentage points on diacritization, and — measured same-protocol at greedy — +1.60pp on Thai G2P. Users choosing a tier choose a measured trade, not a mystery box.

=== The Persian teacher is competitive with published state of the art

Expand Down Expand Up @@ -233,6 +233,14 @@ Porting beam search into the runtimes forced an apples-to-apples measurement on

Exact match barely moves; edit distance collapses by 4.2x. The signature is diagnostic: beam decoding is mangling precisely the sentences the model did not get exactly right.

The *teacher* is also affected, though less severely: measured at
greedy on the same harness and split, the umt5 teacher scores 1.25% PER
(EM 95.16%) against its published beam-4 4.43%. Same-protocol, the
client tier's true shrink cost is +1.60pp — not the +7.63pp the
beam-vs-beam comparison suggested. Both numbers in that comparison were
decode artifacts; the distillation itself was substantially better than
either published figure implied.

[[fig-decode]]
.(a) Top-1 log-probabilities at successive positions of a representative greedy decode against the uniform baseline; the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset: no normalization exponent recovers greedy quality.
[image::paper-assets/decode.png[Decode distributions and alpha sweep]]
Expand Down
Loading