diff --git a/docs/paper-assets/architecture.png b/docs/paper-assets/architecture.png new file mode 100644 index 0000000..47bb22e Binary files /dev/null and b/docs/paper-assets/architecture.png differ diff --git a/docs/paper-assets/decode.png b/docs/paper-assets/decode.png new file mode 100644 index 0000000..2d94656 Binary files /dev/null and b/docs/paper-assets/decode.png differ diff --git a/docs/paper-assets/frontier.png b/docs/paper-assets/frontier.png new file mode 100644 index 0000000..1159d1e Binary files /dev/null and b/docs/paper-assets/frontier.png differ diff --git a/docs/paper.adoc b/docs/paper.adoc index 917b3d5..7214094 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -67,9 +67,24 @@ We apply two standard distillation regimes: *sequence-level* distillation, where Post-training weight quantization of transformer MatMuls (8-bit dynamic; 4-bit blockwise) is standard deployment practice for large language models. Two aspects are, to our knowledge, under-treated in the literature and addressed by our contract: (a) *parity gates must be precision-aware* — a flat torch-vs-ONNX error bound that passes at fp32 will spuriously fail quantized artifacts (we measure fp16 ≈ 0.43pp and int8 ≈ 0.84pp character-error deltas on an unquantized-verified model); and (b) quantized graphs may use operators unavailable to old consumer runtimes (our 4-bit artifacts require MatMulNBits, absent from the oldest Ruby consumer), so precision is a first-class, advertised property of each artifact rather than a silent implementation detail. -=== Beam search and length normalization - -Length-normalized beam search is the conventional decode for seq2seq evaluation. Murray & Chiang and others have analyzed its length biases; our contribution is a concrete, deployment-scale demonstration that on *distilled byte-level students* — whose per-token distributions are far flatter than their teachers' — likelihood-ranked completion is pathological in both directions (length normalization prefers long garbage; raw cumulative scores prefer short ones), while greedy decoding rides the argmax margins to a 4.2x lower error rate. Published evaluations that decode such models with beams risk reporting the decode, not the model. +=== Beam search and its pathologies + +Length-normalized beam search is the conventional decode for seq2seq +evaluation, and its failure modes are well studied: Murray & Chiang +(2018) analyze length biases and their corrections; Meister, Cotterell & +Vieira (2020) show that beam search's errors are not merely scoring +artifacts but reflect which completions the underlying model actually +prefers. Our result sits at their intersection, on a model class they +did not consider — *distilled byte-level students*, whose per-token +distributions are far flatter than their teachers' — and at deployment +scale: likelihood-ranked completion is pathological in both directions +(length normalization prefers long garbage; raw cumulative scores prefer +short ones), while greedy decoding rides the argmax margins to a 4.2x +lower error rate. Published evaluations that decode such models with +beams risk reporting the decode, not the model. + +image:paper-assets/architecture.png[The IMF v1 pipeline,width=580,align=center] +.Caption: The phonological-layer pipeline. Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output. == The artifact contract (IMF v1) [[section-imf]] @@ -120,6 +135,21 @@ Every language defines one harness, used for teacher and student identically: * *Arabic diacritization:* windowed DER-CE — inputs longer than 1,400 bytes split at word boundaries, decoded greedily per window, stitched, and the teacher's haraqat projected onto the input's letters through letter-level alignment, scored by the Misraj evaluator over SadeedDiac-25 paragraphs. The windowed protocol removes truncation losses on long hadith paragraphs. * *Persian G2P:* test-split CER plus a SentenceBench homograph accuracy (ezafe-normalized). +=== Training configuration + +[%autowidth,cols="1,1"] +|=== +|Setting |Value + +|Student init (client tier) |google/byt5-small (300M, d=1472, 12+4 layers) +|Optimizer |AdamW, lr 1e-4, cosine schedule, grad-norm clip 1.0 +|Batching |token-budget batches (32 x max(200, seq_max)); 3 epochs +|Truncation |per-language window (384 bytes Thai; 1,450 bytes Arabic) +|Labeling |teacher greedy/beam-4 generation, resumable, ASCII-escaped transport +|Checkpoints |every 500 steps, with labels-file digest for lineage-safe resume +|Hardware |single A10G (24 GB) per training; exports and parity on CPU +|=== + === Training regimes Students train either by sequence-level KD (teacher-generated targets; cross-entropy) or logit KD (KL + CE on gold). Client-tier students initialize from the ByT5-small pretrained backbone; we additionally constructed from-scratch custom byte-level architectures (33M and 70M parameters, with and without linear "bridge" projections that map teacher activations into the student's geometry via closed-form ridge regression) to test whether small, task-specific backbones could replace pretrained ones. They could not (<>). @@ -171,6 +201,9 @@ The campaign's central engineering question: how small can a client artifact be? |ByT5-base (server tier) |pretrained |580M |1.2 GiB fp32 |9.19 |=== +image:paper-assets/frontier.png[Client-tier size-quality frontier,width=580,align=center] +.Caption: The client-tier frontier on Thai G2P (identical harness). Randomly initialized students collapse at every capacity tested; the pretrained rung (green) is the whole quality cliff. The hollow point marks the same ByT5-small student scored under the beam-4 evaluation convention — see <>. + Three findings: 1. **Random initialization collapses regardless of capacity or structure.** The bridge projections (teacher activations ridge-regressed into the student's geometry) measurably improve internal structure — 75.80 → 71.12 — but cannot rescue task accuracy. Enlarging without pretraining does not help (70M scores *worse* than 33M). @@ -198,6 +231,9 @@ Porting beam search into the runtimes forced an apples-to-apples measurement on Exact match barely moves; edit distance collapses by 4.2x. The signature is diagnostic: beam decoding is mangling precisely the sentences the model did not get exactly right. +image:paper-assets/decode.png[Decode distributions and alpha sweep,width=620,align=center] +.Caption: (a) Top-1 log-probabilities at successive positions of a representative greedy decode, against the uniform baseline ln(259); the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset; no normalization exponent recovers greedy quality. + === The mechanism Distilled byte-level students have *flat* per-token distributions. On the measured artifact, the top-1 log-probability at a typical decoding position is ≈ −4.6 against a uniform baseline of ln(259) ≈ −5.6 — the argmax is barely distinguished from the field. Greedy decoding rides these tiny margins consistently and produces coherent output. @@ -208,6 +244,17 @@ Beam search ranks *completions* by accumulated likelihood. On flat distributions * with raw cumulative scores, short completions win — the maximum-likelihood complete hypothesis on this model genuinely is a truncation (`sa˨˩` for "สวัสดี"); * an α sweep (0, 0.3, 0.6, 1.0) produced PERs of 69.8, 70.1, 70.4, and 45.9 on a 150-sentence subset where greedy scores 3.20 — *no* normalization exponent recovers greedy quality. +A concrete example from the shipped artifact, input Thai for "hello": + +[%autowidth,cols="1,1"] +|=== +|Decode |Output + +|greedy |`sa˨˩.wat̚˨˩.diː˧` (correct) +|beam-4, α=1 |`sa˨˩.wat̚˨˩.sat̚˨˩.sa˨˩` (runs long) +|beam-4, raw |`sa˨˩` (truncates) +|=== + The published beam-4 figures were not wrong measurements; they were measurements of the decode. Under a protocol users never execute, they overstated error 4.2x. === Implications @@ -252,6 +299,19 @@ The phonological layer ships: eleven models, four languages, two tiers, one arti == References +* [%hardbreaks] + Yao, K., Zweig, G. _Sequence-to-sequence neural net models for grapheme-to-phoneme conversion._ INTERSPEECH 2015. + + +* Rao, K., Peng, F., Sak, H., Beaufays, F. _Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks._ ICASSP 2015. + + +* Yolchuyeva, B., Németh, G., Gyires-Tóth, B. _Grapheme-to-phoneme conversion with deep learning._ Speech Communication 114 (2019). + + +* Meister, I., Cotterell, R., Vieira, T. _If beam search is the answer, what was the question?_ EMNLP 2020. + + +* Dettmers, T., Lewis, M., Shleifer, S., Zettlemoyer, L. _LLM.int8(): 8-bit matrix multiplication for transformers at scale._ NeurIPS 2022. + + +* Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. _GPTQ: Accurate post-training quantization for generative pre-trained transformers._ ICLR 2023. + * [%hardbreaks] Open Geospatial Consortium. _Interoperable Script Conversion Systems_ (OGC Abstract Specification Topic). +