Skip to content

Commit a37ba7a

Browse files
author
Ronald Tse
committed
docs(paper): GLM-5.3-Flash leaderboard row (8.7978 zero-skip) + protocol note
1 parent f66a939 commit a37ba7a

1 file changed

Lines changed: 3 additions & 0 deletions

File tree

docs/paper.adoc

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,13 +206,16 @@ External claims are made only where a public benchmark exists and the full proto
206206
|**our teacher (r7, ByT5-base)** |580M |**2.2864**
207207
|our previous teacher (r6) |580M |2.5793
208208
|GLM-5.2 (reproduced, zero-skip) |— |2.6911
209+
|GLM-5.3-Flash (reproduced, zero-skip, reasoning_effort=low) |— |8.7978
209210
|Gemini Flash 2.0 (published) |— |3.1926
210211
|GPT-4 (published) |— |3.8645
211212
|Sadeed (published) |1.5B |7.2915
212213
|**our client student 2.0 (ByT5-small)** |300M |**4.822**
213214
|our client student 1.0 (ByT5-small) |300M |8.259
214215
|===
215216

217+
A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B — driven by Quranic-convention orthography (dagger-alif forms absent from the ground truth; 9.8% not-fully-diacritized words) and the reasoning floor. The frontier's newest generalist regressed on this classical-knowledge task while our dedicated 580M teacher improved; we record it rather than average it away.
218+
216219
The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B. The current rung (r7) adds a teacher-labeled news-domain mix to the morphological-auxiliary teacher (r6) and improves both surfaces at once: in-domain 2.5793 → 2.2864, and out-of-domain WikiNews-2024 multi-reference 19.82/12.46 → 17.38/11.83 (WER/DER) — the paragraph-specialization trade-off of the earlier lineage is gone.
217220

218221
The r6 teacher's auxiliary-task design is validated by a controlled ablation: an identical run differing only in the auxiliary stream's output representation — broad-phonemic IPA of the same training units (deterministic converter) in place of morphological analysis — scores 2.6588 (vs 2.6775 for no auxiliary task and 2.5793 for the morphological one), while a held-out probe confirms the phonemic projection itself was learned (2.3% CER on the IPA stream). The improvement is therefore attributable to lexical-morphological knowledge injection, not to phonemic supervision per se: the strong form of the "phonemes help diacritization" hypothesis fails where its weak form (any structured auxiliary projection beats none) barely holds.

0 commit comments

Comments
 (0)