Skip to content

ml: the frontier leaderboard + client ladder; catalog to 23 models #158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Sep 6, 2026
Merged

ml: the frontier leaderboard + client ladder; catalog to 23 models #158

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
228 changes: 218 additions & 10 deletions src/pages/ml.astro
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ const serverModels = [
pair: "Arabic → haraqat",
metric: "DER 2.29 · OOD WER 17.38",
artifact: "fp32 · 2.7 GiB · parts",
status: "releasing",
status: "released",
},
{
id: "khm-latn-1.0",
Expand Down Expand Up @@ -54,12 +54,20 @@ const serverModels = [

const clientModels = [
{
id: "ara-diac-small-2.0",
id: "ara-diac-small-2.1",
task: "diacritization",
pair: "Arabic → haraqat",
metric: "DER 4.82 (1.0 was 8.26)",
artifact: "int8 · ~0.5 GiB",
status: "releasing",
metric: "DER 4.57 (2.0 was 4.82 · 1.0 was 8.26)",
artifact: "int8 · 264 MiB · fp16 · 674 MiB",
status: "released",
},
{
id: "ara-diac-layerdrop-1.0",
task: "diacritization",
pair: "Arabic → haraqat",
metric: "DER 5.78 · 63% of the parameters",
artifact: "int4 · 95 MiB · browser-native",
status: "released",
},
{
id: "tha-g2p-small-1.0",
Expand Down Expand Up @@ -128,11 +136,11 @@ const clientModels = [
<p class="eyebrow">The catalogue</p>
<h2>Every model, with its number.</h2>
<p class="section-deck">
No model is published without a measured metric next to its teacher's, on the same
harness, in the open. Students are distilled from frozen teachers and gated at a
pre-agreed error budget. Each entry resolves from the models.yaml index; artifacts above
GitHub's 2&nbsp;GiB cap ship as sha256-verified parts that the runtimes reassemble
transparently.
The index resolves 23 models across five languages. No model is published without a
measured metric next to its teacher's, on the same harness, in the open. Students are
distilled from frozen teachers and gated at a pre-agreed error budget. Each entry
resolves from the models.yaml index; artifacts above GitHub's 2&nbsp;GiB cap ship as
sha256-verified parts that the runtimes reassemble transparently.
</p>
</header>

Expand Down Expand Up @@ -197,6 +205,153 @@ const clientModels = [
</div>
</section>

<!-- ════════════════════════════════════════════════════════════════
THE LEADERBOARD — one instrument, every system
════════════════════════════════════════════════════════════════ -->
<section class="section surface-deep" id="leaderboard">
<div class="container">
<header class="section-head">
<p class="eyebrow">The frontier, measured</p>
<h2>580M parameters against the frontier.</h2>
<p class="section-deck">
SadeedDiac-25 — all 1,200 paragraphs, Misraj's own evaluator, windowed zero-skip
protocol, one instrument for every row. Our dedicated 580M teacher is the best
non-vendor system measured on it, 3.2× better than the 1.5B model the benchmark was
built around. And the frontier is moving the wrong way: every GLM successor to 5.2 is
3–5× worse, losing classical haraqat (wrong vowels on 10.05% of marked positions for
5.3-Flash, vs 5.2's 2.64% — which matches our 580M teacher's 2.62% to 0.02pp).
</p>
</header>

<div class="lb-wrap">
<table class="lb-table">
<thead>
<tr>
<th>System</th>
<th class="tnum">Params</th>
<th class="tnum">DER (CE)</th>
<th class="tnum">DER (w/o CE)</th>
<th>Protocol note</th>
</tr>
</thead>
<tbody>
<tr>
<td>Claude-3.7-Sonnet</td>
<td class="tnum">—</td>
<td class="tnum">1.3941</td>
<td class="tnum">0.7693</td>
<td class="lb-dim">vendor-published · protocol undisclosed · not reproduced by us</td>
</tr>
<tr class="lb-ours">
<td>Ours · ara-diac-2.0 (r7)</td>
<td class="tnum">580M</td>
<td class="tnum"><strong>2.2864</strong></td>
<td class="tnum"><strong>1.3343</strong></td>
<td>best dedicated model measured</td>
</tr>
<tr>
<td>GLM-5.2</td>
<td class="tnum">—</td>
<td class="tnum">2.5060</td>
<td class="tnum">1.5537</td>
<td class="lb-dim">our reproduction · raw</td>
</tr>
<tr>
<td>Gemini-Flash-2.0</td>
<td class="tnum">—</td>
<td class="tnum">3.1926</td>
<td class="tnum">2.3783</td>
<td class="lb-dim">same protocol</td>
</tr>
<tr>
<td>GPT-4</td>
<td class="tnum">—</td>
<td class="tnum">3.8645</td>
<td class="tnum">3.8645</td>
<td class="lb-dim">same protocol</td>
</tr>
<tr>
<td>Sadeed</td>
<td class="tnum">1.5B</td>
<td class="tnum">7.2915</td>
<td class="tnum">5.2625</td>
<td class="lb-dim">prior dedicated state of the art</td>
</tr>
<tr>
<td>GLM-5.3-Flash</td>
<td class="tnum">—</td>
<td class="tnum">8.7978</td>
<td class="tnum">6.6368</td>
<td class="lb-dim">zero-skip · reasoning_effort=low</td>
</tr>
<tr>
<td>GLM-5.3</td>
<td class="tnum">—</td>
<td class="tnum">9.8971</td>
<td class="tnum">7.8219</td>
<td class="lb-dim">zero-skip · reasoning_effort=low</td>
</tr>
<tr>
<td>glm-4.7-flash</td>
<td class="tnum">—</td>
<td class="tnum">13.2256</td>
<td class="tnum">10.3206</td>
<td class="lb-dim">zero-skip · thinking-disabled</td>
</tr>
</tbody>
</table>
</div>
<p class="lb-note">
WER columns, decode protocols, and paired bootstrap CIs for every row:
<a href="https://github.com/interscript/rababa/blob/main/docs/RESULTS.md"
>rababa/docs/RESULTS.md</a
>. The GLM-5.3 family cannot disable thinking; low effort is its plainest protocol.
</p>

<h3 class="tier-label">
The client ladder <span>300M student, identical architecture and artifact size</span>
</h3>
<ul class="ledger ladder">
<li class="ledger-row">
<span class="ledger-task">rung 1.0</span>
<span class="ledger-body"><code class="ledger-id">AdamW · r6 labels · 3 ep</code></span>
<span class="ledger-metric tnum">8.26</span>
<span class="ledger-artifact">the shipped baseline</span>
</li>
<li class="ledger-row">
<span class="ledger-task">+ Muon</span>
<span class="ledger-body"><code class="ledger-id">optimizer swap</code></span>
<span class="ledger-metric tnum">5.29</span>
<span class="ledger-artifact">−2.96pp</span>
</li>
<li class="ledger-row">
<span class="ledger-task">+ r7 labels</span>
<span class="ledger-body"><code class="ledger-id">fresher teacher labels</code></span>
<span class="ledger-metric tnum">4.82</span>
<span class="ledger-artifact">−0.47pp</span>
</li>
<li class="ledger-row">
<span class="ledger-task">+ 6 epochs</span>
<span class="ledger-body"><code class="ledger-id">ara-diac-small-2.1</code></span>
<span class="ledger-metric tnum">4.57</span>
<span class="ledger-artifact">−0.25pp · released</span>
</li>
</ul>
<p class="lb-note">
What didn't move it — every rung pre-registered, measured, and kept in the log:
multi-token-prediction auxiliary 5.09 (+0.26pp); classical-register swap at constant
budget 5.81 (+0.98pp); register add at matched epochs 4.82 (flat — it cancelled the epoch
gain); on-policy GKD 6.00 (+1.18pp, the worst rung); depth-halved 5.78 (63% of the
parameters — shipped anyway as the browser tier). At SFT convergence, supervision quality
dominates. The ladder is closed.
</p>
<p class="lb-note">
Hebrew, same discipline: 16.43% DER on Biblical Hebrew, where the modern-Hebrew state of
the art degrades to 35.6% on the same test.
</p>
</div>
</section>

<!-- ════════════════════════════════════════════════════════════════
THE CONTRACT — IMF v1
════════════════════════════════════════════════════════════════ -->
Expand Down Expand Up @@ -441,6 +596,59 @@ github.com/secryst # runtimes + training</code></pre>
max-width: 42rem;
line-height: 1.55;
}

/* ── The leaderboard ──────────────────────────────────────────── */
.lb-wrap {
overflow-x: auto;
margin: 0 -1.5rem;
padding: 0 1.5rem;
}
.lb-table {
width: 100%;
border-collapse: collapse;
font-size: 0.9375rem;
}
.lb-table th {
font-family: var(--font-mono);
font-size: var(--text-micro);
letter-spacing: 0.12em;
text-transform: uppercase;
text-align: left;
color: var(--color-stone);
border-bottom: 2px solid var(--color-ink);
padding: 0.6rem 1rem 0.6rem 0;
}
.lb-table td {
padding: 0.7rem 1rem 0.7rem 0;
border-bottom: 1px dotted var(--color-rule);
vertical-align: baseline;
}
.lb-table th.tnum,
.lb-table td.tnum {
text-align: right;
font-variant-numeric: tabular-nums;
white-space: nowrap;
}
.lb-dim {
color: var(--color-stone);
font-size: 0.8125rem;
}
.lb-ours td {
background: color-mix(in srgb, var(--color-highlight) 12%, transparent);
}
.lb-note {
font-size: 0.875rem;
color: var(--color-stone);
margin-top: 1.5rem;
max-width: 48rem;
line-height: 1.55;
}
.lb-note a {
color: var(--color-highlight);
}
.ladder .ledger-artifact {
font-size: 0.8125rem;
}
.ledger-note a {
color: var(--color-highlight);
}
Expand Down