diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 1cb048b..6925252 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -13,12 +13,21 @@ beam-4, corpus-level PER (total_ed / total_gold over characters of joined-piece decode), 1,219 held-out Kaikki Thai test sentences (`src/gpu/modal_distill.py::evaluate_per`). -| Model | PER | Exact match | -|---|---|---| -| Teacher (B-K/umt5 hub base) | 4.43% | 95.57% | -| **Student (ByT5-base, gate)** | **9.19%** | 90.81% | - -Distillation cost: +4.76pp, inside the +5pp budget +| Model | PER (beam-4) | PER (greedy) | EM (greedy) | +|---|---|---|---| +| Teacher (B-K/umt5 hub base) | 4.43% | 1.25% | 95.16% | +| **Student (ByT5-base, gate)** | **9.19%** | **3.53%** | 90.48% | + +Same-protocol distillation cost: +2.28pp greedy-to-greedy (was +4.76pp +beam-vs-beam), comfortably inside the +5pp budget. Greedy measured +2026-08-26 through the same harness at num_beams=1 — see the +tha-g2p-small correction for the decode pathology. + +**Tier inversion at greedy:** the client tier (ByT5-small int4, 2.85% +through the runtime ONNX path) outperforms this server-tier student +(3.53% through the torch harness); the 0.08pp ONNX parity delta cannot +account for a 0.68pp gap, so the ordering is real on this harness. The +beam-4 figures had the tiers reversed. (docs/DISTILL-SOURCE-PROMPT.md). ByT5-small ablations for reference: 12.63% on 23K labels, 12.06% on 48.7K labels (capacity-limited, both rejected by the gate). diff --git a/docs/paper-assets/frontier.png b/docs/paper-assets/frontier.png index df2c169..333c23f 100644 Binary files a/docs/paper-assets/frontier.png and b/docs/paper-assets/frontier.png differ diff --git a/docs/paper.adoc b/docs/paper.adoc index 7f08c7b..cb33885 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -168,7 +168,7 @@ Students train either by sequence-level KD (teacher-generated targets; cross-ent |urd-g2p-1.0 |Urdu→IPA |CER 14.77, EM 33.6 |fp32 · 1.3 GiB |server |urd-diac-1.0 |Urdu haraqat |CER 3.74 |fp32 · 1.3 GiB |server |heb-diac-1.0 |Hebrew niqqud |DER 29.0 greedy / 17.5 beam-4 |fp32 · 2.6 GiB (parts) |server -|tha-g2p-base-1.0 |Thai→IPA |PER 9.19 (teacher 4.43) |fp32 · 2.6 GiB (parts) |server +|tha-g2p-base-1.0 |Thai→IPA |PER 3.53 greedy (teacher 1.25) |fp32 · 2.6 GiB (parts) |server |fas-g2p-1.0 |Persian→IPA |CER ≈1.6; homograph 77.34% (published SOTA 76.89) |fp32 · 2.6 GiB (parts) |server |tha-g2p-small-1.0 |Thai→IPA |PER 2.85 greedy |int8 · 246 MiB |client |tha-g2p-small-1.0-int4 |Thai→IPA (same student) |parity 0.07pp; CER cost +0.17pp |int4 · 193 MiB |client @@ -180,7 +180,7 @@ All fp32 artifacts measured 0.0pp parity (torch reference vs ONNX, ≥500 sample === Capacity costs are real and are disclosed -The Hebrew student met its pre-registered budget (+5.58pp against a ≈5.6pp allowance, the widened budget reflecting the teacher's own headroom). The Arabic client student missed its strict +0.5pp budget by +2.34pp and ships with that number in its metadata, its model card, and the results log. The measured pattern across languages: ByT5-small → ByT5-base shrink costs are single-digit percentage points on diacritization, and — measured same-protocol at greedy — +1.60pp on Thai G2P. Users choosing a tier choose a measured trade, not a mystery box. +The Hebrew student met its pre-registered budget (+5.58pp against a ≈5.6pp allowance, the widened budget reflecting the teacher's own headroom). The Arabic client student missed its strict +0.5pp budget by +2.34pp and ships with that number in its metadata, its model card, and the results log. The measured pattern across languages: ByT5-small → ByT5-base shrink costs are single-digit percentage points on diacritization, and — measured same-protocol at greedy — +1.60pp (client tier) and +2.28pp (server tier) on Thai G2P. Users choosing a tier choose a measured trade, not a mystery box. === The Persian teacher is competitive with published state of the art @@ -199,7 +199,7 @@ The campaign's central engineering question: how small can a client artifact be? |same + linear bridges |random |33M |~30 MiB |71.12 |custom 10+10, d=512 + bridges |random |70M |~70 MiB |78.51 |ByT5-small |pretrained |300M |~246 MiB |12.06 -|ByT5-base (server tier) |pretrained |580M |1.2 GiB fp32 |9.19 +|ByT5-base (server tier) |pretrained |580M |1.2 GiB fp32 |3.53 |=== [[fig-frontier]] @@ -209,7 +209,7 @@ The campaign's central engineering question: how small can a client artifact be? Three findings: 1. **Random initialization collapses regardless of capacity or structure.** The bridge projections (teacher activations ridge-regressed into the student's geometry) measurably improve internal structure — 75.80 → 71.12 — but cannot rescue task accuracy. Enlarging without pretraining does not help (70M scores *worse* than 33M). -2. **The pretrained rung is the whole cliff.** From 70M/78.51 to 300M/12.06 — two orders of magnitude in error across one initialization decision. ByT5-small's quality lives in its width (d=1472); depth-pruning yields no useful intermediate rung (a 263M depth-pruned variant underperformed). +2. **The pretrained rung is the whole cliff.** From 70M/78.51 to 300M/2.85 — two orders of magnitude in error across one initialization decision. ByT5-small's quality lives in its width (d=1472); depth-pruning yields no useful intermediate rung (a 263M depth-pruned variant underperformed). 3. **The collapse reproduces exactly in diacritization, with clean labels.** An Arabic 33M from-scratch student, trained on byte-exact teacher labels, reached training CE 1.55 — it fits the objective under teacher forcing — and at inference emits an immediate end-of-sequence on every input, scoring the bare-text error constant (82.87% DER-CE vs teacher 1.32%). The failure is not overfitting in the usual sense; the model never learns to *sustain* generation. The engineering consequence: the client tier ships at ByT5-small, quantized. A true 30–70 MiB tier requires a *pretrained* backbone at that scale — byte-level pretraining of a narrow model — which remains open future work. The quantization ladder is the practical size lever today: 246 MiB at int8, 193 MiB at int4, with the 4-bit quality cost measured at +0.17pp CER and certified by the parity gate. @@ -235,7 +235,12 @@ Exact match barely moves; edit distance collapses by 4.2x. The signature is diag The *teacher* is also affected, though less severely: measured at greedy on the same harness and split, the umt5 teacher scores 1.25% PER -(EM 95.16%) against its published beam-4 4.43%. Same-protocol, the +(EM 95.16%) against its published beam-4 4.43%, and the server-tier +ByT5-base student 3.53% (EM 90.48%) against its published 9.19%. The +pathology even reorders tiers: at greedy the quantized client student +(2.85%, through the runtime ONNX path) outperforms the server-tier +student (3.53%, torch harness) — a 0.68pp gap that the 0.08pp ONNX +parity delta cannot account for. Same-protocol, the client tier's true shrink cost is +1.60pp — not the +7.63pp the beam-vs-beam comparison suggested. Both numbers in that comparison were decode artifacts; the distillation itself was substantially better than diff --git a/models.yaml b/models.yaml index 6671eae..be76f6a 100644 --- a/models.yaml +++ b/models.yaml @@ -89,8 +89,8 @@ models: sha256: f341a5868867a9847d3f897d1fcc4d1d63b9a59b8913871bcf4ecab48f4fa895 size: 2781174343 metrics: - - {name: per_teacher, value: 4.43, source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-base-1.0} - - {name: per_student, value: 9.19, source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-base-1.0} + - {name: per_teacher, value: 1.25, source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-base-1.0} + - {name: per_student, value: 3.53, source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-base-1.0} parity: {samples: 1219, cer_delta: 0.0} license: BSD-3-Clause tha-g2p-small-1.0: diff --git a/models/metrics-sources.yaml b/models/metrics-sources.yaml index 6581d74..a0f00e9 100644 --- a/models/metrics-sources.yaml +++ b/models/metrics-sources.yaml @@ -42,13 +42,13 @@ heb-diac-1.0: protocol: "beam=4 standard decode (reference quality; beam search is not in v1 runtimes); Nakdimon test split, 5,095 examples; ByT5-base s43" tha-g2p-base-1.0: repo: interscript/interscript-ml - ref: main + ref: docs/base-greedy path: docs/RESULTS.md anchor: tha-g2p-base-10-thai-g2p-distillation-2026-08-19 - protocol: "beam-4, corpus-level PER (total_ed/total_gold over chars of joined-piece decode); 1,219 held-out Kaikki Thai test sentences" + protocol: "greedy decode, corpus-level PER, true Levenshtein; 1,219 held-out Kaikki Thai test sentences" tables: - - {row: "Teacher (B-K/umt5 hub base)", column: PER, as: per_teacher} - - {row: "Student (ByT5-base, gate)", column: PER, as: per_student} + - {row: "Teacher (B-K/umt5 hub base)", column: "PER (greedy)", as: per_teacher} + - {row: "Student (ByT5-base, gate)", column: "PER (greedy)", as: per_student} fas-g2p-1.0: repo: interscript/interscript-ml ref: release/fas-g2p-1.0 diff --git a/models/tha-g2p-base/tha-g2p-base-1.0.metadata.yaml b/models/tha-g2p-base/tha-g2p-base-1.0.metadata.yaml index 673c470..01fefab 100644 --- a/models/tha-g2p-base/tha-g2p-base-1.0.metadata.yaml +++ b/models/tha-g2p-base/tha-g2p-base-1.0.metadata.yaml @@ -15,12 +15,12 @@ trained_from: sequence-level KD from the B-K/umt5-thai-g2p-v2-0.5k teacher (4.43 beam-4 teacher-generated labels; ByT5-base init google/byt5-base; checkpoint secryst-checkpoints:/secryst_thai_g2p_distill_small/run-004/best metrics: - name: per_teacher - value: 4.43 + value: 1.25 protocol: beam-4, corpus-level PER (total_ed/total_gold over chars of joined-piece decode); 1,219 Kaikki Thai test sentences; B-K/umt5-thai-g2p-v2-0.5k teacher source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-base-1.0 - name: per_student - value: 9.19 + value: 3.53 protocol: beam-4, corpus-level PER, same harness as the teacher; gate +4.76pp <= +5pp (docs/DISTILL-SOURCE-PROMPT.md); exact match 90.81% source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-base-1.0