diff --git a/docs/paper-arabic/main.pdf b/docs/paper-arabic/main.pdf index faeafc6..40e3f6d 100644 Binary files a/docs/paper-arabic/main.pdf and b/docs/paper-arabic/main.pdf differ diff --git a/docs/paper-arabic/main.tex b/docs/paper-arabic/main.tex index 2235c31..aacfde6 100644 --- a/docs/paper-arabic/main.tex +++ b/docs/paper-arabic/main.tex @@ -3,9 +3,9 @@ \usepackage{booktabs} \usepackage{amsmath} \usepackage{hyperref} -\title{Data Quality over Scale: A 30M-Parameter Encoder for\\ Arabic Diacritization} +\title{Data Quality over Scale: Compact Models for\\ Arabic Diacritization and the Frontier Regression} \author{Interscript ML Team} -\date{August 2026} +\date{September 2026} \begin{document} \maketitle @@ -17,19 +17,23 @@ using a 1.5B-parameter decoder-only language model. We show that compact models---a character-level encoder of roughly 10M parameters and a 580M ByT5-base---\textbf{beat it directly on that benchmark with its own -evaluation code}, reaching \textbf{2.68\% DER (CE) and 1.60\% without -case endings} under a zero-skip windowed protocol: 2.7$\times$ and -3.3$\times$ better than Sadeed, ahead of GPT-4 (3.86\%) and -Gemini-Flash-2.0 (3.19\%/2.38\%). We additionally verify the LLM -frontier on this benchmark: GLM-5.2, a 2026 flagship, scores -2.51\%/1.55\% under a neutral protocol (2.69\%/1.72\% zero-skip) --- -the published Claude-3.7-Sonnet figure of 1.39\% does not reproduce -with a strictly stronger model, and our 580M model matches or beats the -verified frontier. A controlled RL study (rejection-sampling FT, -sequence-level GRPO, and entropy-weighted GRPO) finds \emph{no} further -gain at SFT convergence: the residual is knowledge-limited, not -policy-limited. Our -combined corpus merges the full 75M-word Tashkeela dump (cleaned with a +evaluation code}, reaching \textbf{2.29\% DER (CE) and 1.33\% without +case endings} under a zero-skip windowed protocol: 3.2$\times$ and +4.0$\times$ better than Sadeed, ahead of GPT-4 (3.86\%) and +Gemini-Flash-2.0 (3.19\%/2.38\%). We additionally measure the LLM +frontier on this benchmark and find a family-wide regression: GLM-5.2 +scores 2.51\%/2.69\% (raw/zero-skip) under a neutral protocol---vowel +knowledge matching our dedicated 580M teacher to within 0.02pp---while +every GLM successor we measured is 3--5$\times$ worse (5.3-Flash 8.80, +5.3 9.90, 4.7-flash 13.23 zero-skip), with per-position attribution +locating the loss in wrong haraqat rather than writing convention. +A distilled 300M client-tier student reaches 4.82\% (42\% error +reduction) through a controlled lever ladder whose two registered +negatives---multi-token-prediction auxiliary and constant-budget +register mixing---mirror the RL negative: at SFT convergence every +verified gain came from supervision quality, not from policy +optimization, frontier training techniques, or register diversification. +Our combined corpus merges the full 75M-word Tashkeela dump (cleaned with a Sadeed-style pipeline), Arabic Wikipedia, and QCRI EMNLP-2025 data, totalling 2.1M sentences; scaling 28$\times$ (75K $\rightarrow$ 2.1M) reduced in-domain DER from 2.42\% to 0.99\%. We further show that @@ -54,7 +58,7 @@ \section{Introduction} $\approx$10M parameters) that \textbf{beats Sadeed on SadeedDiac-25 with Misraj's own evaluator}: 3.25\% DER (CE) / 1.81\% (w/o CE) vs 7.29\% / 5.26\%, at 1/150th the parameters; - a 580M ByT5-base extends this to \textbf{2.81\% / 1.69\%}, + a 580M ByT5-base extends this to 2.81\% / 1.69\%, beating Gemini-Flash-2.0 on all four metrics under a zero-skip windowed protocol. \item A data ablation showing in-domain DER falls 2.42\% $\rightarrow$ 0.99\% as @@ -66,10 +70,29 @@ \section{Introduction} \item Paragraph-context training (joining corpus lines into ${\sim}$1400-byte units) closing the context gap to LLMs: 2.81$\rightarrow$2.68\% DER (CE), 1.69$\rightarrow$1.60\% (w/o CE). - \item A frontier verification: GLM-5.2 under a neutral protocol - lands at 2.51\%/2.69\% (raw/zero-skip), far from the published - Claude-3.7 1.39\%; our compact model beats the verified - frontier on the w/o-CE metrics. + \item A teacher lineage driven by supervision quality, not + architecture: a morphological-analysis auxiliary stream + (r6, 2.5793\% DER) and a teacher-labeled news-domain mix + (r7, \textbf{2.2864\% / 1.3343\%})---the best dedicated model + measured on this benchmark---with a controlled ablation + attributing the r6 gain to lexical-morphological knowledge + injection (IPA-aux 2.6588 vs none 2.6775 vs morph 2.5793). + \item A frontier regression axis: under our neutral protocol, + GLM-5.2 scores 2.69\% zero-skip DER---and every GLM successor + we measured is 3--5$\times$ worse (5.3-Flash 8.80, 5.3 9.90, + 4.7-flash 13.23), with per-position attribution locating the + loss in wrong haraqat (10.05\% of positions vs 5.2's 2.64\%, + which matches our 580M teacher's 2.62\%), not writing + convention (the dagger-alif effect is 0.125pp). + \item A distilled client tier at 300M parameters: 8.26\% + $\rightarrow$ \textbf{4.82\%} (42\% error reduction at + identical architecture) via a controlled lever ladder + (optimizer $-2.96$pp, fresher teacher labels $-0.47$pp, + longer training $-0.25$pp), plus two registered negatives + (multi-token-prediction auxiliary $+0.26$pp; + constant-budget register swap $+0.98$pp) showing neither + frontier-training techniques nor register mixing substitute + for supervision quality. \item A three-way RL negative result (RAFT, sequence-GRPO on Persian, entropy-weighted GTPO-GRPO on Arabic): all flat or negative at SFT convergence --- diacritization residual error is @@ -129,12 +152,17 @@ \subsection{Comparison on SadeedDiac-25 (their benchmark, their evaluator)} System & Params & DER (CE) & DER (w/o CE) & WER (CE) & WER (w/o CE) \\ \midrule Claude-3-7-Sonnet & --- & 1.3941 & 0.7693 & 4.6718 & 2.3098 \\ +\textbf{Ours (ByT5-base r7, morph-aux + news mix)} & 580M & \textbf{2.2864} & \textbf{1.3343} & --- & --- \\ +Ours (ByT5-base r6, morph-aux) & 580M & 2.5793 & 1.5317 & --- & --- \\ GLM-5.2 (our reproduction, raw) & --- & 2.5060 & 1.5537 & 7.9929 & 4.7509 \\ GLM-5.2 (our reproduction, zero-skip) & --- & 2.6911 & 1.7179 & 8.3037 & 5.0619 \\ +GLM-5.3-Flash (zero-skip, reasoning\_effort=low) & --- & 8.7978 & 6.6368 & 30.84 & 24.16 \\ +GLM-5.3 (zero-skip, reasoning\_effort=low) & --- & 9.8971 & 7.8219 & 31.07 & 24.63 \\ +glm-4.7-flash (zero-skip, thinking-disabled) & --- & 13.2256 & 10.3206 & 36.10 & 30.10 \\ Gemini-Flash-2.0 & --- & 3.1926 & 2.3783 & 7.9942 & 5.5044 \\ GPT-4 & --- & 3.8645 & 3.8645 & 5.2719 & 10.9274 \\ Sadeed & 1.5B & 7.2915 & 5.2625 & 13.7425 & 9.9245 \\ -\textbf{Ours (ByT5 r5 paragraph-context, windowed)} & 580M & \textbf{2.6775} & \textbf{1.5965} & \textbf{8.0919} & \textbf{4.3863} \\ +Ours (ByT5 r5 paragraph-context, windowed) & 580M & 2.6775 & 1.5965 & 8.0919 & 4.3863 \\ Ours (r5 + GTPO-GRPO, 700B windows) & 580M & 2.6597 & 1.5818 & 8.1165 & 4.3874 \\ Ours (ByT5 r3, windowed) & 580M & 2.8126 & 1.6877 & 8.4110 & 4.5739 \\ Ours (ByT5 r3, single-shot) & 580M & 2.8429 & 1.7589 & 8.4981 & 4.8859 \\ @@ -144,7 +172,11 @@ \subsection{Comparison on SadeedDiac-25 (their benchmark, their evaluator)} \end{tabular} \caption{Full 1,200-paragraph SadeedDiac-25, Misraj's ArabicDiacritizationEvaluator, default protocol. Published -LLM/Sadeed rows from the benchmark card. The windowed protocol +LLM/Sadeed rows from the benchmark card; the GLM-5.3-generation rows +are our reproductions (thinking cannot be disabled on 5.x---HTTP 400 +code 1210---so they run at \texttt{reasoning\_effort=low}, the nearest +expressible analog of plain completion; glm-4.7-flash still accepts +disabled thinking). The windowed protocol splits long inputs at word boundaries into $\leq$600-byte in-distribution windows with a $2\times$ generation cap (diacritized output is 1.4--1.6$\times$ input bytes; a naive input-length cap @@ -164,6 +196,51 @@ \subsection{Paragraph-context training} on both metrics. The same specialization costs +0.53 WER on the out-of-domain WikiNews-2024 multi-reference probe. +\subsection{The frontier regression axis} +The paragraph-context result above overtook GLM-5.2---but the frontier +moved underneath us. Measuring the GLM family under the identical +protocol (same 1,200 paragraphs, same evaluator, temperature 0, plain +completion where the API still allows it): +GLM-5.2 reproduces at 2.5060\% raw / 2.6911\% zero-skip; +GLM-5.3-Flash lands at 8.5721 / 8.7978; the full GLM-5.3 at +9.9760 / 9.8971; and glm-4.7-flash, the last plain-completion GLM +model, at 13.0035 / 13.2256 (12 resumable passes past sustained +provider rate limits; all 1,200 responses real). Per-position +attribution over the same paragraphs (convention-normalized, rates +over the 179{,}401 ground-truth-marked positions) localizes the loss: +GLM-5.2's wrong-haraqat rate is 2.64\%---matching our dedicated r7 +teacher's 2.62\% to within 0.02pp---while 5.3-Flash's is 10.05\% and +4.7-flash regresses on both axes at once (wrong 9.01\%, missing +6.67\%, the family's worst under-diacritization). The entire +dagger-alif (U+0670) convention effect is 0.125pp. Paired-bootstrap +CIs put every successor decisively behind 5.2 ($+7.9$ to $+8.0$pp, +CIs excluding zero by a wide margin). As frontier optimization has +shifted agentic, the newest generalists lost classical-Arabic mark +knowledge their predecessor had---while our dedicated 580M teacher +improved. + +\subsection{The distilled client tier} +The same benchmark disciplines a 300M distilled student shipped for +browser/edge inference. From the 1.0 rung (8.26\%), a pre-registered +lever ladder reached \textbf{4.82\%} (42\% error reduction at +identical architecture and artifact size): the Muon optimizer alone +contributes $-2.96$pp (a controlled 2$\times$2 factorial closes +additively: optimizer $-2.96$, memory-layer capacity $-0.70$, +combined $-3.43$), fresher r7 teacher labels $-0.47$pp, and longer +training a further $-0.25$pp (4.57\%). Two registered negatives +bracket the ladder: multi-token-prediction as a training auxiliary +scored 5.09\% ($+0.26$pp vs control, with a disclosed preemption +confound), and swapping news-domain training units for classical +Tashkeela at constant budget scored 5.81\% ($+0.98$pp)---the causal +test of the domain-coverage hypothesis failing in the swap direction. +Together with the RL negatives, the pattern is one line: at SFT +convergence, supervision quality dominates policy optimization, +frontier training techniques, and register diversification alike. The +tier ships as checksummed, runtime-agnostic artifacts (IMF v1) whose +quantization fragility was itself diagnosed and fixed---the quantized +output head, not the body, caused confident flip errors (9.34\% +$\rightarrow$ 0.26\% after the head-fp32 repair). + \subsection{RL at SFT convergence: three negative results} We tested whether the residual (98.7\% phonotactically legal alternates) can be sharpened by policy optimization: @@ -188,9 +265,11 @@ \subsection{RL at SFT convergence: three negative results} \section{Discussion} \textbf{Data curation dominates capacity.} A 150$\times$ smaller encoder beats a 1.5B model on its own benchmark once the corpus is cleaned and -scaled, and a 580M ByT5-base reaches 2.68\% DER (CE) / 1.60\% (w/o -CE)---beating Gemini-Flash-2.0 on all four metrics and the verified -GLM-5.2 frontier on the w/o-CE metrics. On our in-domain +scaled, and the 580M ByT5-base lineage reaches 2.29\% DER (CE) / 1.33\% +(w/o CE) through supervision-side moves alone (paragraph context, +morphological auxiliary, teacher-labeled domain mix)---beating +Gemini-Flash-2.0 on all four metrics and every GLM generation after +5.2 on every metric. On our in-domain split the models reach $\leq$1.4\% DER; the gap to 2.81\% on SadeedDiac-25 measures the domain shift (the benchmark is 50\% Classical Arabic, expert-reviewed, deliberately contamination-free). @@ -217,9 +296,10 @@ \section{Limitations} predicts haraqat for space and punctuation positions (its input vocab includes them), which the benchmark's word splitter treats as ghost tokens; we suppress non-letter haraqat at render time. The published -Claude-3.7-Sonnet row (1.39\%) does not reproduce under our neutral -protocol: a strictly stronger 2026 flagship (GLM-5.2) scores 2.51\% -raw, and we publish our reproduction protocol in full. WER (CE) trails +Claude-3.7-Sonnet row (1.39\%) is vendor-published under an +undisclosed protocol; we do not treat it as reproducible and note only +that a newer flagship measured under our fully published protocol +scores 2.51\% raw. WER (CE) trails GPT-4 and Gemini---LLMs copy words more faithfully at the cost of 2$\times$ worse DER. diff --git a/docs/paper-hebrew/main.pdf b/docs/paper-hebrew/main.pdf index e7bd2aa..bc2f91b 100644 Binary files a/docs/paper-hebrew/main.pdf and b/docs/paper-hebrew/main.pdf differ diff --git a/docs/paper-hebrew/main.tex b/docs/paper-hebrew/main.tex index c2e8b51..423b85c 100644 --- a/docs/paper-hebrew/main.tex +++ b/docs/paper-hebrew/main.tex @@ -5,7 +5,7 @@ \usepackage{hyperref} \title{Hebrew Diacritization Is Domain-Bound: Beating the SOTA Model\\ on Biblical Text and the Teamim Input-Format Effect} \author{Interscript ML Team} -\date{August 2026} +\date{September 2026} \begin{document} \maketitle @@ -16,14 +16,25 @@ near-SOTA error rates. We show that on the Biblical/Rablinic portions of the Nakdimon test split, the same SOTA model degrades to 35.6\% diacritization error rate (DER), while a ByT5-base seq2seq model trained on a mixed-domain -50K-pair corpus reaches 17.5\%---an 18-point margin. Error analysis shows -the vowel-pointing accuracy of our model at aligned positions is 97.1\%, +corpus reaches \textbf{16.43\%}---an 19-point margin---and greedy +decoding matches beam-4 on the production checkpoint, so the shipped +runtime path delivers reference quality. On the public-domain Dicta +ACL-2020 test corpora the same model is strong across modern, poetry, +and rabbinic line-level registers (0.52/0.24/0.45 DER), localizing the +weak surface specifically to paragraph-level Biblical text---register +coverage, not a general modern-text gap; we note D-Nikud, the recent +open TavBERT+Bi-LSTM system, publishes no numbers on these corpora. +Error analysis shows the vowel-pointing accuracy of our model at +aligned positions is 97.1\%, with \emph{zero} cantillation (teamim) errors: we identify a previously unreported input-format effect in which standard preprocessing leaves teamim in the model input, making cantillation a copy-through task. We further report negative results: 2.3$\times$ data scaling yields no DER -change, and character-level output-vote ensembling \emph{increases} DER -despite reducing per-character vowel errors. +change; character-level output-vote ensembling \emph{increases} DER +despite reducing per-character vowel errors; and transplanting the +Arabic campaign's morphological-auxiliary win to Hebrew is flat +(16.53 vs 16.43)---the auxiliary trick is not template-portable, the +residuals decompose differently. \end{abstract} \section{Introduction} @@ -84,8 +95,12 @@ \section{Data and the Input-Format Question} \section{Model and Training} ByT5-base (580M), seq2seq, undiacritized$\to$diacritized. 3 epochs, batch 8, LR $3{\times}10^{-4}$, label smoothing 0.1, beam 4 at -inference. Four checkpoints: v2 (22K corpus), v4 (50K corpus), s43, s44 -(seed replicas of v4). +inference. Checkpoint lineage: v2 (22K corpus), v4 (50K corpus), s43, +s44 (seed replicas); s45 adds a phonikud weak-pretrain stage (1.5M +machine-labeled lines, deduped, decontaminated); s46 (production) +garnishes stage 1 with 73.8K Dicta-labeled wiki lines; s47 transplants +the Arabic campaign's morphological-auxiliary stream and is flat +(16.53 vs init 16.43)---the Hebrew teacher line is closed at s46. \section{Experiments} \subsection{Main results} @@ -96,7 +111,10 @@ \subsection{Main results} System & DER & Notes \\ \midrule DictaBERT-large (SOTA) & 35.63\% & our run, native bare-text input \\ -\textbf{s43 (ours)} & \textbf{17.46\%} & teamim-preserving input \\ +\textbf{s46 (ours, production)} & \textbf{16.43\%} & beam-4; greedy 16.44\% \\ +s47 (ours, morph-aux transplant) & 16.53\% & negative vs init s46 \\ +s45 (ours, phonikud curriculum) & 16.58\% & \\ +s43 (ours) & 17.46\% & teamim-preserving input \\ s44 (ours) & 17.65\% & \\ v4 (ours, 50K) & 17.78\% & \\ v2 (ours, 22K) & 17.3\% & original recipe \\ @@ -104,7 +122,8 @@ \subsection{Main results} 3-way vote ensemble & 21.52\% & \emph{worse} than singles \\ \bottomrule \end{tabular} -\caption{Nakdimon test (Biblical/Rablinic), beam 4 unless noted.} +\caption{Nakdimon test (Biblical/Rablinic), 5{,}095 examples, beam 4 +unless noted.} \end{table} \subsection{Error decomposition} @@ -123,6 +142,32 @@ \subsection{Error decomposition} \caption{v4 predictions, per-consonant analysis.} \end{table} +\subsection{Modern-text surfaces: the Dicta ACL-2020 corpora} +The public-domain Dicta ACL-2020 test corpora (Modern/HebrewWiki, +Poetry, Rabbinic; densely vocalized at ${\sim}$0.8 marks/letter) +measure whether the model generalizes beyond the Biblical test: + +\begin{tabular}{lcc} +\toprule +Corpus & s46 greedy DER & Examples \\ +\midrule +Modern (HebrewWiki) & 0.5209\% & 253 \\ +Poetry & 0.2377\% & 959 \\ +Rabbinic (Bet Yosef) & 0.4523\% & 164 \\ +\bottomrule +\end{tabular} + +Protocol: line-level units ($\leq$512 chars), greedy, the same +seq2seq DER harness as the Nakdimon row. The weak surface is thus +specifically Nakdimon-style \emph{paragraph-level} Biblical text---not +a general modern-text gap. D-Nikud (TavBERT+Bi-LSTM, 2024), the +recent open system, publishes no numbers on these three corpora (its +tables are internal splits and the Nakdimon test set), so we record +our rows only; our harness additionally keeps the corpora's +angle-bracket matres-lectionis marks in input and ground truth rather +than the README's strip-and-check convention, disclosed as a protocol +deviation. + \subsection{Ablations} \begin{itemize} \item \textbf{Data scaling}: 22K$\to$50K changes DER 17.3\%$\to$17.8\% diff --git a/docs/paper-umbrella/main.pdf b/docs/paper-umbrella/main.pdf index 5c6379d..5d3c5a9 100644 Binary files a/docs/paper-umbrella/main.pdf and b/docs/paper-umbrella/main.pdf differ diff --git a/docs/paper-umbrella/main.tex b/docs/paper-umbrella/main.tex index 672da84..4cef7b7 100644 --- a/docs/paper-umbrella/main.tex +++ b/docs/paper-umbrella/main.tex @@ -5,7 +5,7 @@ \usepackage{hyperref} \title{Five Languages, Five Bottlenecks:\\ Diacritization and G2P for Transliteration at Interscript} \author{Interscript ML Team} -\date{August 2026} +\date{September 2026} \begin{document} \maketitle @@ -20,27 +20,34 @@ (Hebrew), labeled-data \emph{coverage} (Thai), output \emph{representation} (Persian, Urdu)---and a fixed ByT5-based framework with one surgical fix per language reached or beat the published -state of the art in each. Results: Arabic \textbf{2.68\% DER} (CE) / -1.60\% (w/o CE) on the SadeedDiac-25 benchmark with Misraj's own -evaluator under a zero-skip windowed protocol---2.6$\times$/3.1$\times$ +state of the art in each. Results: Arabic \textbf{2.29\% DER} (CE) / +1.33\% (w/o CE) on the SadeedDiac-25 benchmark with Misraj's own +evaluator under a zero-skip windowed protocol---3.2$\times$/4.0$\times$ better than the 1.5B-parameter Sadeed model, ahead of GPT-4 and -Gemini-Flash-2.0---and matching or beating our own verified GLM-5.2 -frontier reproduction (2.69\%/1.72\% zero-skip; the published -Claude-3.7 figure of 1.39\% does not reproduce under a neutral -protocol)---plus a contamination audit showing the benchmark's public -source corpus leaks 122 of its own paragraphs, and a three-way RL -negative result (RAFT, sequence-GRPO, entropy-weighted GRPO) showing -the residual is knowledge-limited, not policy-limited; Hebrew 17.5\% DER -on Biblical text where the SOTA model scores 35.6\%; Thai 2.32\% PER +Gemini-Flash-2.0---plus a frontier regression axis: GLM-5.2 matches our +dedicated 580M teacher's vowel accuracy to within 0.02pp, while every +GLM successor we measured is 3--5$\times$ worse (5.3-Flash 8.80, +5.3 9.90, 4.7-flash 13.23 zero-skip), the loss located by per-position +attribution in wrong haraqat, not writing convention; a 300M distilled +client tier at 4.82\% (42\% error reduction) whose lever ladder brackets +two registered negatives (MTP-auxiliary, register swap) mirroring the +three-way RL negative (RAFT, sequence-GRPO, entropy-weighted GRPO); a +contamination audit showing the benchmark's public source corpus leaks +122 of its own paragraphs; Hebrew 16.4\% DER +on Biblical text where the SOTA model scores 35.6\%, and strong +modern/poetry/rabbinic line-level surfaces (0.52/0.24/0.45 DER); +Thai 2.32\% PER via deterministic (non-LLM) augmentation; Persian homograph accuracy 77.3--89.5\% and diacritization 0.52\% CER from a G2P-only dataset; Urdu G2P 14.77\% CER (first learned baseline, 4.1$\times$ over rule-based) and diacritization 3.74\% CER from weak supervision. We distill cross-cutting lessons for phonological NLP: LLMs are unreliable -phonological labelers; evaluation domains and input formats silently -dominate comparisons; ensembling and curricula fail at data-limited -optima; policy optimization adds nothing at SFT convergence---every -verified gain came from data curation; and byte-level models are the +phonological labelers and are \emph{regressing} on classical knowledge +as their optimization shifts agentic; evaluation domains and input +formats silently dominate comparisons; ensembling and curricula fail +at data-limited optima; policy optimization adds nothing at SFT +convergence---every verified gain came from data or supervision +curation; and byte-level models are the right deployment target for multilingual TS runtimes. \end{abstract} @@ -62,8 +69,8 @@ \section{System and results} \toprule Language & Task & Result & Published reference \\ \midrule -Arabic & diacritization & \textbf{2.81/1.69\% DER} & Sadeed 7.29/5.26\%, Gemini 3.19/2.38\% (same benchmark+evaluator) \\ -Hebrew & diacritization & \textbf{17.46\% DER} (Biblical) & DictaBERT 35.6\% (same test) \\ +Arabic & diacritization & \textbf{2.29/1.33\% DER} & Sadeed 7.29/5.26\%, Gemini 3.19/2.38\%, GLM-5.3-Flash 8.80 (same benchmark+evaluator) \\ +Hebrew & diacritization & \textbf{16.43\% DER} (Biblical) & DictaBERT 35.6\% (same test) \\ Thai & G2P & \textbf{2.32\% PER} & baseline 6.37\% (same test) \\ Persian & G2P/HA & \textbf{89.5\%} / 77.3\% (SB) & Homo-GE2PE 76.9\% \\ Persian & diacritization & \textbf{0.52\% CER} & (none existed) \\ @@ -76,10 +83,21 @@ \section{System and results} \section{The bottleneck taxonomy} \begin{itemize} \item \textbf{Arabic --- data quality}: cleaning + 28$\times$ scaling - halved DER; a 30M encoder matches a 1.5B LM. + halved DER; a 30M encoder matches a 1.5B LM; supervision-side + moves (paragraph context, morphological auxiliary, teacher + labeled domain mix) then carried the 580M teacher from 2.81 to + 2.29\%; a distilled 300M client tier reached 4.82\% (42\% + error reduction) through a controlled lever ladder whose two + registered negatives (MTP-auxiliary $+0.26$pp, register swap + $+0.98$pp) bracket it---supervision quality, not technique, + moves this task. \item \textbf{Hebrew --- data domain}: the SOTA model collapses cross-domain (9$\times$); mixed-domain training wins; the teamim - input-format effect makes cantillation copy-through. + input-format effect makes cantillation copy-through; modern, + poetry, and rabbinic line-level surfaces are strong + (0.52/0.24/0.45 DER on the public-domain Dicta ACL-2020 test + corpora), so the residual is specifically paragraph-level + Biblical text. \item \textbf{Thai --- labeled coverage}: 10K dictionary $\to$ 60K via deterministic phonemizer labels; all architectural fixes failed, data fixed it. @@ -92,9 +110,16 @@ \section{The bottleneck taxonomy} \section{Cross-cutting lessons} \begin{enumerate} - \item \textbf{LLMs cannot label phonology.} Tone and haraqat - hallucination mirrors the known diacritization failure; epitran - labels are free of it and transfer through fine-tuning. + \item \textbf{LLMs cannot label phonology---and are regressing on + classical knowledge.} Tone and haraqat hallucination mirrors + the known diacritization failure; epitran labels are free of + it and transfer through fine-tuning. On SadeedDiac-25 the GLM + family regressed 3--5$\times$ across one generation boundary + (5.2 at 2.69\% zero-skip vs 5.3-Flash 8.80, 5.3 9.90, + 4.7-flash 13.23), with the loss attributed to wrong haraqat + rather than convention---dedicated distilled models are not + nostalgia; they are the only reproducible, protocol-pinned + artifacts on the right side of that axis. \item \textbf{Input formats and domains are hidden variables.} Teamim-preserving inputs and Biblical-vs-modern test domain each move DER by more than most modeling choices. @@ -109,7 +134,10 @@ \section{Cross-cutting lessons} \item \textbf{Byte-level models are deployment-friendly}: ByT5's tokenizer is \texttt{TextEncoder} in TypeScript---no vocab files, no WASM sentencepiece---ideal for interscript.org's - browser/Node targets. + browser/Node targets. Quantization fragility is diagnosable + and fixable at the artifact level: the quantized output head, + not the body, caused confident flip errors (9.34\% $\to$ + 0.26\% after the head-fp32 repair). \end{enumerate} \section{Deployment}