Skip to content

Commit 0c61a9e

Browse files
ronaldtseronaldtse
committed
ml: the frontier leaderboard + client ladder; catalog to 23 models
P1 currency: client-tier card moves to ara-diac-small-2.1 (DER 4.57; 2.0 was 4.82, 1.0 was 8.26) with fp16 listed; ara-diac-layerdrop-1.0 added (DER 5.78 at 63% of parameters, int4 95 MiB browser tier); ara-diac-2.0 status released; section deck states the 23-model catalog. P2 the leaderboard section: protocol-matched SadeedDiac-25 table (every cell from the ledger; the Claude-3.7 row labeled vendor-published, protocol undisclosed, not reproduced by us), the 300M client ladder (8.26 - 5.29 - 4.82 - 4.57) with the five registered negatives named, and the Hebrew same-test line. Astro build clean; the built page emits dist/client/ml/index.html with the section.
1 parent c2c7898 commit 0c61a9e

1 file changed

Lines changed: 218 additions & 10 deletions

File tree

‎src/pages/ml.astro‎

Lines changed: 218 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ const serverModels = [
88
pair: "Arabic → haraqat",
99
metric: "DER 2.29 · OOD WER 17.38",
1010
artifact: "fp32 · 2.7 GiB · parts",
11-
status: "releasing",
11+
status: "released",
1212
},
1313
{
1414
id: "khm-latn-1.0",
@@ -54,12 +54,20 @@ const serverModels = [
5454
5555
const clientModels = [
5656
{
57-
id: "ara-diac-small-2.0",
57+
id: "ara-diac-small-2.1",
5858
task: "diacritization",
5959
pair: "Arabic → haraqat",
60-
metric: "DER 4.82 (1.0 was 8.26)",
61-
artifact: "int8 · ~0.5 GiB",
62-
status: "releasing",
60+
metric: "DER 4.57 (2.0 was 4.82 · 1.0 was 8.26)",
61+
artifact: "int8 · 264 MiB · fp16 · 674 MiB",
62+
status: "released",
63+
},
64+
{
65+
id: "ara-diac-layerdrop-1.0",
66+
task: "diacritization",
67+
pair: "Arabic → haraqat",
68+
metric: "DER 5.78 · 63% of the parameters",
69+
artifact: "int4 · 95 MiB · browser-native",
70+
status: "released",
6371
},
6472
{
6573
id: "tha-g2p-small-1.0",
@@ -128,11 +136,11 @@ const clientModels = [
128136
<p class="eyebrow">The catalogue</p>
129137
<h2>Every model, with its number.</h2>
130138
<p class="section-deck">
131-
No model is published without a measured metric next to its teacher's, on the same
132-
harness, in the open. Students are distilled from frozen teachers and gated at a
133-
pre-agreed error budget. Each entry resolves from the models.yaml index; artifacts above
134-
GitHub's 2&nbsp;GiB cap ship as sha256-verified parts that the runtimes reassemble
135-
transparently.
139+
The index resolves 23 models across five languages. No model is published without a
140+
measured metric next to its teacher's, on the same harness, in the open. Students are
141+
distilled from frozen teachers and gated at a pre-agreed error budget. Each entry
142+
resolves from the models.yaml index; artifacts above GitHub's 2&nbsp;GiB cap ship as
143+
sha256-verified parts that the runtimes reassemble transparently.
136144
</p>
137145
</header>
138146

@@ -197,6 +205,153 @@ const clientModels = [
197205
</div>
198206
</section>
199207

208+
<!-- ════════════════════════════════════════════════════════════════
209+
THE LEADERBOARD — one instrument, every system
210+
════════════════════════════════════════════════════════════════ -->
211+
<section class="section surface-deep" id="leaderboard">
212+
<div class="container">
213+
<header class="section-head">
214+
<p class="eyebrow">The frontier, measured</p>
215+
<h2>580M parameters against the frontier.</h2>
216+
<p class="section-deck">
217+
SadeedDiac-25 — all 1,200 paragraphs, Misraj's own evaluator, windowed zero-skip
218+
protocol, one instrument for every row. Our dedicated 580M teacher is the best
219+
non-vendor system measured on it, 3.2× better than the 1.5B model the benchmark was
220+
built around. And the frontier is moving the wrong way: every GLM successor to 5.2 is
221+
3–5× worse, losing classical haraqat (wrong vowels on 10.05% of marked positions for
222+
5.3-Flash, vs 5.2's 2.64% — which matches our 580M teacher's 2.62% to 0.02pp).
223+
</p>
224+
</header>
225+
226+
<div class="lb-wrap">
227+
<table class="lb-table">
228+
<thead>
229+
<tr>
230+
<th>System</th>
231+
<th class="tnum">Params</th>
232+
<th class="tnum">DER (CE)</th>
233+
<th class="tnum">DER (w/o CE)</th>
234+
<th>Protocol note</th>
235+
</tr>
236+
</thead>
237+
<tbody>
238+
<tr>
239+
<td>Claude-3.7-Sonnet</td>
240+
<td class="tnum">—</td>
241+
<td class="tnum">1.3941</td>
242+
<td class="tnum">0.7693</td>
243+
<td class="lb-dim">vendor-published · protocol undisclosed · not reproduced by us</td>
244+
</tr>
245+
<tr class="lb-ours">
246+
<td>Ours · ara-diac-2.0 (r7)</td>
247+
<td class="tnum">580M</td>
248+
<td class="tnum"><strong>2.2864</strong></td>
249+
<td class="tnum"><strong>1.3343</strong></td>
250+
<td>best dedicated model measured</td>
251+
</tr>
252+
<tr>
253+
<td>GLM-5.2</td>
254+
<td class="tnum">—</td>
255+
<td class="tnum">2.5060</td>
256+
<td class="tnum">1.5537</td>
257+
<td class="lb-dim">our reproduction · raw</td>
258+
</tr>
259+
<tr>
260+
<td>Gemini-Flash-2.0</td>
261+
<td class="tnum">—</td>
262+
<td class="tnum">3.1926</td>
263+
<td class="tnum">2.3783</td>
264+
<td class="lb-dim">same protocol</td>
265+
</tr>
266+
<tr>
267+
<td>GPT-4</td>
268+
<td class="tnum">—</td>
269+
<td class="tnum">3.8645</td>
270+
<td class="tnum">3.8645</td>
271+
<td class="lb-dim">same protocol</td>
272+
</tr>
273+
<tr>
274+
<td>Sadeed</td>
275+
<td class="tnum">1.5B</td>
276+
<td class="tnum">7.2915</td>
277+
<td class="tnum">5.2625</td>
278+
<td class="lb-dim">prior dedicated state of the art</td>
279+
</tr>
280+
<tr>
281+
<td>GLM-5.3-Flash</td>
282+
<td class="tnum">—</td>
283+
<td class="tnum">8.7978</td>
284+
<td class="tnum">6.6368</td>
285+
<td class="lb-dim">zero-skip · reasoning_effort=low</td>
286+
</tr>
287+
<tr>
288+
<td>GLM-5.3</td>
289+
<td class="tnum">—</td>
290+
<td class="tnum">9.8971</td>
291+
<td class="tnum">7.8219</td>
292+
<td class="lb-dim">zero-skip · reasoning_effort=low</td>
293+
</tr>
294+
<tr>
295+
<td>glm-4.7-flash</td>
296+
<td class="tnum">—</td>
297+
<td class="tnum">13.2256</td>
298+
<td class="tnum">10.3206</td>
299+
<td class="lb-dim">zero-skip · thinking-disabled</td>
300+
</tr>
301+
</tbody>
302+
</table>
303+
</div>
304+
<p class="lb-note">
305+
WER columns, decode protocols, and paired bootstrap CIs for every row:
306+
<a href="https://github.com/interscript/rababa/blob/main/docs/RESULTS.md"
307+
>rababa/docs/RESULTS.md</a
308+
>. The GLM-5.3 family cannot disable thinking; low effort is its plainest protocol.
309+
</p>
310+
311+
<h3 class="tier-label">
312+
The client ladder <span>300M student, identical architecture and artifact size</span>
313+
</h3>
314+
<ul class="ledger ladder">
315+
<li class="ledger-row">
316+
<span class="ledger-task">rung 1.0</span>
317+
<span class="ledger-body"><code class="ledger-id">AdamW · r6 labels · 3 ep</code></span>
318+
<span class="ledger-metric tnum">8.26</span>
319+
<span class="ledger-artifact">the shipped baseline</span>
320+
</li>
321+
<li class="ledger-row">
322+
<span class="ledger-task">+ Muon</span>
323+
<span class="ledger-body"><code class="ledger-id">optimizer swap</code></span>
324+
<span class="ledger-metric tnum">5.29</span>
325+
<span class="ledger-artifact">−2.96pp</span>
326+
</li>
327+
<li class="ledger-row">
328+
<span class="ledger-task">+ r7 labels</span>
329+
<span class="ledger-body"><code class="ledger-id">fresher teacher labels</code></span>
330+
<span class="ledger-metric tnum">4.82</span>
331+
<span class="ledger-artifact">−0.47pp</span>
332+
</li>
333+
<li class="ledger-row">
334+
<span class="ledger-task">+ 6 epochs</span>
335+
<span class="ledger-body"><code class="ledger-id">ara-diac-small-2.1</code></span>
336+
<span class="ledger-metric tnum">4.57</span>
337+
<span class="ledger-artifact">−0.25pp · released</span>
338+
</li>
339+
</ul>
340+
<p class="lb-note">
341+
What didn't move it — every rung pre-registered, measured, and kept in the log:
342+
multi-token-prediction auxiliary 5.09 (+0.26pp); classical-register swap at constant
343+
budget 5.81 (+0.98pp); register add at matched epochs 4.82 (flat — it cancelled the epoch
344+
gain); on-policy GKD 6.00 (+1.18pp, the worst rung); depth-halved 5.78 (63% of the
345+
parameters — shipped anyway as the browser tier). At SFT convergence, supervision quality
346+
dominates. The ladder is closed.
347+
</p>
348+
<p class="lb-note">
349+
Hebrew, same discipline: 16.43% DER on Biblical Hebrew, where the modern-Hebrew state of
350+
the art degrades to 35.6% on the same test.
351+
</p>
352+
</div>
353+
</section>
354+
200355
<!-- ════════════════════════════════════════════════════════════════
201356
THE CONTRACT — IMF v1
202357
════════════════════════════════════════════════════════════════ -->
@@ -441,6 +596,59 @@ github.com/secryst # runtimes + training</code></pre>
441596
max-width: 42rem;
442597
line-height: 1.55;
443598
}
599+
600+
/* ── The leaderboard ──────────────────────────────────────────── */
601+
.lb-wrap {
602+
overflow-x: auto;
603+
margin: 0 -1.5rem;
604+
padding: 0 1.5rem;
605+
}
606+
.lb-table {
607+
width: 100%;
608+
border-collapse: collapse;
609+
font-size: 0.9375rem;
610+
}
611+
.lb-table th {
612+
font-family: var(--font-mono);
613+
font-size: var(--text-micro);
614+
letter-spacing: 0.12em;
615+
text-transform: uppercase;
616+
text-align: left;
617+
color: var(--color-stone);
618+
border-bottom: 2px solid var(--color-ink);
619+
padding: 0.6rem 1rem 0.6rem 0;
620+
}
621+
.lb-table td {
622+
padding: 0.7rem 1rem 0.7rem 0;
623+
border-bottom: 1px dotted var(--color-rule);
624+
vertical-align: baseline;
625+
}
626+
.lb-table th.tnum,
627+
.lb-table td.tnum {
628+
text-align: right;
629+
font-variant-numeric: tabular-nums;
630+
white-space: nowrap;
631+
}
632+
.lb-dim {
633+
color: var(--color-stone);
634+
font-size: 0.8125rem;
635+
}
636+
.lb-ours td {
637+
background: color-mix(in srgb, var(--color-highlight) 12%, transparent);
638+
}
639+
.lb-note {
640+
font-size: 0.875rem;
641+
color: var(--color-stone);
642+
margin-top: 1.5rem;
643+
max-width: 48rem;
644+
line-height: 1.55;
645+
}
646+
.lb-note a {
647+
color: var(--color-highlight);
648+
}
649+
.ladder .ledger-artifact {
650+
font-size: 0.8125rem;
651+
}
444652
.ledger-note a {
445653
color: var(--color-highlight);
446654
}

0 commit comments

Comments
 (0)