diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 2c2f50d..1cb048b 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -85,10 +85,15 @@ better than the beam-4 harness numbers.** Re-measured on the shipped int4 zip through the Python runtime (the exact ONNX KV decode users get), true Levenshtein, full 1,219-sentence set: -| Decode | PER | Exact match | -|---|---|---| -| beam-4 (published, torch harness) | 12.06% | 87.94% | -| **greedy (runtime protocol)** | **2.85%** | **88.93%** | +| Decode | Teacher PER | Student PER | Student EM | +|---|---|---|---| +| beam-4 (published, torch harness) | 4.43% | 12.06% | 87.94% | +| **greedy (runtime protocol)** | **1.25%** | **2.85%** | **88.93%** | + +The teacher is also affected by the beam pathology (4.43 beam-4 → 1.25 +greedy, measured 2026-08-25 through the same harness at num_beams=1): +same-protocol, the client tier's true shrink cost is **+1.60pp**, not +the +7.63pp the beam-vs-beam comparison suggested. The beam-4 numbers are inflated by length-normalized beam preferring long garbage on this model's flat per-token distributions (top-1 diff --git a/docs/paper-assets/frontier.png b/docs/paper-assets/frontier.png index 1159d1e..df2c169 100644 Binary files a/docs/paper-assets/frontier.png and b/docs/paper-assets/frontier.png differ diff --git a/docs/paper.adoc b/docs/paper.adoc index 7214094..bf27678 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -83,8 +83,9 @@ short ones), while greedy decoding rides the argmax margins to a 4.2x lower error rate. Published evaluations that decode such models with beams risk reporting the decode, not the model. -image:paper-assets/architecture.png[The IMF v1 pipeline,width=580,align=center] -.Caption: The phonological-layer pipeline. Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output. +[[fig-architecture]] +.The phonological-layer pipeline: frozen teachers, gated students, one checksummed artifact, three runtimes, byte-identical output. +[image::paper-assets/architecture.png[The IMF v1 pipeline]] Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output. == The artifact contract (IMF v1) [[section-imf]] @@ -201,8 +202,9 @@ The campaign's central engineering question: how small can a client artifact be? |ByT5-base (server tier) |pretrained |580M |1.2 GiB fp32 |9.19 |=== -image:paper-assets/frontier.png[Client-tier size-quality frontier,width=580,align=center] -.Caption: The client-tier frontier on Thai G2P (identical harness). Randomly initialized students collapse at every capacity tested; the pretrained rung (green) is the whole quality cliff. The hollow point marks the same ByT5-small student scored under the beam-4 evaluation convention — see <>. +[[fig-frontier]] +.The client-tier frontier on Thai G2P (identical harness): randomly initialized students collapse at every capacity tested; the pretrained rung is the whole quality cliff. The hollow point marks the same ByT5-small student scored under the beam-4 evaluation convention (<>). +[image::paper-assets/frontier.png[Client-tier size-quality frontier]] Three findings: @@ -231,8 +233,9 @@ Porting beam search into the runtimes forced an apples-to-apples measurement on Exact match barely moves; edit distance collapses by 4.2x. The signature is diagnostic: beam decoding is mangling precisely the sentences the model did not get exactly right. -image:paper-assets/decode.png[Decode distributions and alpha sweep,width=620,align=center] -.Caption: (a) Top-1 log-probabilities at successive positions of a representative greedy decode, against the uniform baseline ln(259); the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset; no normalization exponent recovers greedy quality. +[[fig-decode]] +.(a) Top-1 log-probabilities at successive positions of a representative greedy decode against the uniform baseline; the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset: no normalization exponent recovers greedy quality. +[image::paper-assets/decode.png[Decode distributions and alpha sweep]] === The mechanism @@ -241,7 +244,7 @@ Distilled byte-level students have *flat* per-token distributions. On the measur Beam search ranks *completions* by accumulated likelihood. On flat distributions this is pathological in both directions: * with length normalization (`score / len^α`), longer completions divide away their accumulated cost — a 30-token sequence at −0.40/token outranks the correct 15-token answer at −0.53/token, and generated text runs long; -* with raw cumulative scores, short completions win — the maximum-likelihood complete hypothesis on this model genuinely is a truncation (`sa˨˩` for "สวัสดี"); +* with raw cumulative scores, short completions win — the maximum-likelihood complete hypothesis on this model genuinely is a truncation (`sa˨˩` for "สวัสดี"). Measured as a paired comparison on 300 sentences (10,000 bootstrap resamples): the raw-score beam implementation scores 72.6% PER at 0.7% exact match against greedy's 2.6% / 89.0% — a difference of 69.9pp, 95% CI [67.6, 72.1]; * an α sweep (0, 0.3, 0.6, 1.0) produced PERs of 69.8, 70.1, 70.4, and 45.9 on a 150-sentence subset where greedy scores 3.20 — *no* normalization exponent recovers greedy quality. A concrete example from the shipped artifact, input Thai for "hello": @@ -255,7 +258,7 @@ A concrete example from the shipped artifact, input Thai for "hello": |beam-4, raw |`sa˨˩` (truncates) |=== -The published beam-4 figures were not wrong measurements; they were measurements of the decode. Under a protocol users never execute, they overstated error 4.2x. +The published beam-4 figures (the length-normalized harness convention: 12.06% PER) were not wrong measurements; they were measurements of the decode — and the two beam variants fail in opposite directions. Under a protocol users never execute, they overstated error 4.2x. === Implications @@ -291,6 +294,24 @@ The general lesson: when a file "reads corrupt remotely and clean locally," susp The phonological layer ships: eleven models, four languages, two tiers, one artifact contract, zero untraceable numbers. The deterministic guarantees of map-based transliteration — same bytes everywhere, verifiable artifacts, public protocols — carry over to learned conversion. The two negative results (no sub-100M rung without pretraining; beam decode counterproductive on distilled byte students) are released with the same rigor as the positive ones, because in an infrastructure whose value is trust, a measured failure is a deliverable. +== Data and compute statement + +*Datasets.* Thai: Kaikki-derived Thai IPA pairs (train on the expanded +corpus with epitran augmentation; 1,219 held-out test sentences). +Hebrew: hebrew-v4 gold corpus with a phonikud weak-pretraining stage; +Nakdimon test split. Arabic: r5-units domain and replay corpora of +fully-voweled hadith text; SadeedDiac-25 evaluation paragraphs. +Persian: the v1 g2p corpus (release-frozen); SentenceBench for homograph +accuracy. No dataset contains personal data; all are derived from +published textual corpora under their respective licenses. + +*Compute.* Students train on a single 24 GB A10G: 3-11 hours per +student (Thai 4 h; Hebrew 3 h; Arabic 11 h including labeling; the +Arabic client tier trained three times across the debugging saga). +Teacher labeling is the dominant one-off cost per language (2-6 h). +Exports, validation, and parity gates run on CPU. Total campaign GPU +budget is on the order of 100 A10G-hours. + == Availability and provenance * Artifact contract, index, validators, harnesses, results log: `github.com/interscript/interscript-ml` (BSD-3-Clause, code and weights). diff --git a/models/metrics-sources.yaml b/models/metrics-sources.yaml index 2fcc41c..5b874d6 100644 --- a/models/metrics-sources.yaml +++ b/models/metrics-sources.yaml @@ -69,13 +69,14 @@ heb-diac-small-1.0: - {row: "Student (ByT5-small, gate)", column: DER, as: der_student_greedy} tha-g2p-small-1.0: repo: interscript/interscript-ml - ref: docs/greedy-correction + ref: docs/paper-ci path: docs/RESULTS.md anchor: tha-g2p-small-10-thai-g2p-client-tier-2026-08-22 protocol: "greedy decode via the Python runtime (the shipped ONNX KV path), corpus-level PER, true Levenshtein; 1,219 held-out Kaikki Thai test sentences" tables: - - {row: "beam-4 (published, torch harness)", column: PER, as: per_student_beam4} - - {row: "greedy (runtime protocol)", column: PER, as: per_student} + - {row: "greedy (runtime protocol)", column: Teacher PER, as: per_teacher_greedy} + - {row: "beam-4 (published, torch harness)", column: Student PER, as: per_student_beam4} + - {row: "greedy (runtime protocol)", column: Student PER, as: per_student} ara-diac-small-1.0: repo: interscript/interscript-ml ref: release/ara-diac-small-1.0 diff --git a/models/tha-g2p-small/tha-g2p-small-1.0.metadata.yaml b/models/tha-g2p-small/tha-g2p-small-1.0.metadata.yaml index c1068ed..ea37f13 100644 --- a/models/tha-g2p-small/tha-g2p-small-1.0.metadata.yaml +++ b/models/tha-g2p-small/tha-g2p-small-1.0.metadata.yaml @@ -17,6 +17,12 @@ trained_from: 'sequence-level KD from the B-K/umt5-thai-g2p-v2-0.5k teacher (4.4 this is the smallest rung that does not collapse (docs/RESULTS.md frontier table, ~300MB int8).' metrics: +- name: per_teacher_greedy + value: 1.25 + protocol: greedy decode (num_beams=1) through the same harness; 1,219 Kaikki Thai + test sentences; exact match 95.16%; measured 2026-08-25 — the published beam-4 + teacher figure 4.43 was itself inflated by the decode pathology + source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-small-1.0 - name: per_student_beam4 value: 12.06 protocol: beam-4, corpus-level PER, same harness as the teacher (src/gpu/ modal_distill.py::evaluate_per,