From afdbb2a49002130e239b7611a138b533515b1b7b Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 7 Sep 2026 00:22:22 +0200 Subject: [PATCH] ml: the frontier leaderboard + client ladder; catalog to 23 models P1 currency: client-tier card moves to ara-diac-small-2.1 (DER 4.57; 2.0 was 4.82, 1.0 was 8.26) with fp16 listed; ara-diac-layerdrop-1.0 added (DER 5.78 at 63% of parameters, int4 95 MiB browser tier); ara-diac-2.0 status released; section deck states the 23-model catalog. P2 the leaderboard section: protocol-matched SadeedDiac-25 table (every cell from the ledger; the Claude-3.7 row labeled vendor-published, protocol undisclosed, not reproduced by us), the 300M client ladder (8.26 - 5.29 - 4.82 - 4.57) with the five registered negatives named, and the Hebrew same-test line. Astro build clean; the built page emits dist/client/ml/index.html with the section. --- src/pages/ml.astro | 228 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 218 insertions(+), 10 deletions(-) diff --git a/src/pages/ml.astro b/src/pages/ml.astro index 422a1fca..4561f58d 100644 --- a/src/pages/ml.astro +++ b/src/pages/ml.astro @@ -8,7 +8,7 @@ const serverModels = [ pair: "Arabic → haraqat", metric: "DER 2.29 · OOD WER 17.38", artifact: "fp32 · 2.7 GiB · parts", - status: "releasing", + status: "released", }, { id: "khm-latn-1.0", @@ -54,12 +54,20 @@ const serverModels = [ const clientModels = [ { - id: "ara-diac-small-2.0", + id: "ara-diac-small-2.1", task: "diacritization", pair: "Arabic → haraqat", - metric: "DER 4.82 (1.0 was 8.26)", - artifact: "int8 · ~0.5 GiB", - status: "releasing", + metric: "DER 4.57 (2.0 was 4.82 · 1.0 was 8.26)", + artifact: "int8 · 264 MiB · fp16 · 674 MiB", + status: "released", + }, + { + id: "ara-diac-layerdrop-1.0", + task: "diacritization", + pair: "Arabic → haraqat", + metric: "DER 5.78 · 63% of the parameters", + artifact: "int4 · 95 MiB · browser-native", + status: "released", }, { id: "tha-g2p-small-1.0", @@ -128,11 +136,11 @@ const clientModels = [

The catalogue

Every model, with its number.

- No model is published without a measured metric next to its teacher's, on the same - harness, in the open. Students are distilled from frozen teachers and gated at a - pre-agreed error budget. Each entry resolves from the models.yaml index; artifacts above - GitHub's 2 GiB cap ship as sha256-verified parts that the runtimes reassemble - transparently. + The index resolves 23 models across five languages. No model is published without a + measured metric next to its teacher's, on the same harness, in the open. Students are + distilled from frozen teachers and gated at a pre-agreed error budget. Each entry + resolves from the models.yaml index; artifacts above GitHub's 2 GiB cap ship as + sha256-verified parts that the runtimes reassemble transparently.

@@ -197,6 +205,153 @@ const clientModels = [ + +
+
+
+

The frontier, measured

+

580M parameters against the frontier.

+

+ SadeedDiac-25 — all 1,200 paragraphs, Misraj's own evaluator, windowed zero-skip + protocol, one instrument for every row. Our dedicated 580M teacher is the best + non-vendor system measured on it, 3.2× better than the 1.5B model the benchmark was + built around. And the frontier is moving the wrong way: every GLM successor to 5.2 is + 3–5× worse, losing classical haraqat (wrong vowels on 10.05% of marked positions for + 5.3-Flash, vs 5.2's 2.64% — which matches our 580M teacher's 2.62% to 0.02pp). +

+
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SystemParamsDER (CE)DER (w/o CE)Protocol note
Claude-3.7-Sonnet—1.39410.7693vendor-published · protocol undisclosed · not reproduced by us
Ours · ara-diac-2.0 (r7)580M2.28641.3343best dedicated model measured
GLM-5.2—2.50601.5537our reproduction · raw
Gemini-Flash-2.0—3.19262.3783same protocol
GPT-4—3.86453.8645same protocol
Sadeed1.5B7.29155.2625prior dedicated state of the art
GLM-5.3-Flash—8.79786.6368zero-skip · reasoning_effort=low
GLM-5.3—9.89717.8219zero-skip · reasoning_effort=low
glm-4.7-flash—13.225610.3206zero-skip · thinking-disabled
+
+

+ WER columns, decode protocols, and paired bootstrap CIs for every row: + rababa/docs/RESULTS.md. The GLM-5.3 family cannot disable thinking; low effort is its plainest protocol. +

+ +

+ The client ladder 300M student, identical architecture and artifact size +

+
    +
  • + rung 1.0 + AdamW · r6 labels · 3 ep + 8.26 + the shipped baseline +
    • +
  • + + Muon + optimizer swap + 5.29 + −2.96pp +
    • +
  • + + r7 labels + fresher teacher labels + 4.82 + −0.47pp +
    • +
  • + + 6 epochs + ara-diac-small-2.1 + 4.57 + −0.25pp · released +
    • +
+

+ What didn't move it — every rung pre-registered, measured, and kept in the log: + multi-token-prediction auxiliary 5.09 (+0.26pp); classical-register swap at constant + budget 5.81 (+0.98pp); register add at matched epochs 4.82 (flat — it cancelled the epoch + gain); on-policy GKD 6.00 (+1.18pp, the worst rung); depth-halved 5.78 (63% of the + parameters — shipped anyway as the browser tier). At SFT convergence, supervision quality + dominates. The ladder is closed. +

+

+ Hebrew, same discipline: 16.43% DER on Biblical Hebrew, where the modern-Hebrew state of + the art degrades to 35.6% on the same test. +

+
+
+ @@ -441,6 +596,59 @@ github.com/secryst # runtimes + training max-width: 42rem; line-height: 1.55; } + + /* ── The leaderboard ──────────────────────────────────────────── */ + .lb-wrap { + overflow-x: auto; + margin: 0 -1.5rem; + padding: 0 1.5rem; + } + .lb-table { + width: 100%; + border-collapse: collapse; + font-size: 0.9375rem; + } + .lb-table th { + font-family: var(--font-mono); + font-size: var(--text-micro); + letter-spacing: 0.12em; + text-transform: uppercase; + text-align: left; + color: var(--color-stone); + border-bottom: 2px solid var(--color-ink); + padding: 0.6rem 1rem 0.6rem 0; + } + .lb-table td { + padding: 0.7rem 1rem 0.7rem 0; + border-bottom: 1px dotted var(--color-rule); + vertical-align: baseline; + } + .lb-table th.tnum, + .lb-table td.tnum { + text-align: right; + font-variant-numeric: tabular-nums; + white-space: nowrap; + } + .lb-dim { + color: var(--color-stone); + font-size: 0.8125rem; + } + .lb-ours td { + background: color-mix(in srgb, var(--color-highlight) 12%, transparent); + } + .lb-note { + font-size: 0.875rem; + color: var(--color-stone); + margin-top: 1.5rem; + max-width: 48rem; + line-height: 1.55; + } + .lb-note a { + color: var(--color-highlight); + } + .ladder .ledger-artifact { + font-size: 0.8125rem; + } .ledger-note a { color: var(--color-highlight); }