Skip to content

Feat/tts benchmark template - #284

Merged
robdmac merged 34 commits into
mainfrom
feat/tts-benchmark-template
Nov 17, 2026
Merged

robdmac merged 34 commits into
mainfrom
feat/tts-benchmark-template

Conversation

@robdmac

@robdmac robdmac commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

No description provided.

robdmac and others added 30 commits November 17, 2026 15:06
The open-weight TTS benchmark is the first one here that needs no agent, no
provider key and no orchestrator: it measures speech engines rather than
coding agents, so the whole run fits in a terminal's boot command and the
report it writes is served to a browser block beside it.

Two things the sandbox forces, both verified rather than assumed:

  talkito's .gitmodules points at git@github.com: for both submodules, and
  the sandbox has no SSH key, so --recurse-submodules cannot work. The three
  public repos are cloned over HTTPS into the layout tools/paths.py resolves.
  The image installs the talkito package but keeps no checkout, so all three
  have to be fetched at run time.

  tools/benchmark.py with no arguments opens an interactive picker that reads
  stdin and exits on EOF, so it cannot be driven from a boot command. --only
  names the models and skips it.

The web server starts before the clone rather than after the run, so the
browser block has a page from the first second instead of a connection error
for the length of the benchmark. Samples are synthesized before the sweep
because benchmark.py builds the page at the end, and a clip made afterwards
would not be in it.

Pinned to piper: it needs no torch at all and faster-whisper scores on CPU,
so the arm installs in seconds. A full sweep of all 73 providers is around
eleven hours and is not something a template should make easy to start by
accident.

Not yet validated end to end. The clone layout, dependency install, report
serving and browser wiring all work in the sandbox; the run itself stops at
benchmark/tools/paths.py, which requires tests-repo/test_tts_asr_roundtrip.py
— a file that is not in the public talkito-tests repo. That blocks anyone
trying to run the benchmark, not just this template.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Same shape as the agent-skills card - 1200x630, title, finding, chart, footer -
and the palette is sampled from it rather than guessed, so the two sit beside
each other.

Generated rather than drawn. The y-axis is reader preference, which arrives by
ballot and keeps moving, so a hand-made PNG is stale the week after it ships and
cannot be re-cut without redrawing it. The subtitle's r is computed in the same
pass as the points for the same reason: a re-cut must not leave a stale number
next to fresh data.

The x-axis is logarithmic, and that is the one deliberate difference from the
on-page chart. The page fixes x at 6.8s, which covers the engines that keep up
with real time; ten of the thirty rated engines are slower than that, out to 51s
a phrase. A card has no filter controls and no hover, so it has to show every
point it counts, and log x fits 0.12s and 51s in one frame. The flatness of the
cloud across two and a half decades is the finding: r = 0.21, and the sub-second
engines alone span 1 to 94.

Two layout bugs found by looking at the output rather than trusting the code.
Markers and labels are drawn in separate passes, because in one pass a later
marker paints over an earlier label - "NeuTTS-2E Q8" rendered as "euTTS-2E Q8".
And the y-axis runs to 105 while its ticks stop at 100, so the best-rated engine
sits below the frame rather than against it and its label has somewhere to go
that is not the densest part of the cloud.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Thirty-seven configurations now, up from thirty-six. Q8 is the only new row the
page shows: the export also carries the three NeuTTS-2E CPU builds, which
DROP_ROWS already excludes because they are the same weights on a slower path
and say nothing about the model.

Nothing else moved. Every cell of every pre-existing row is byte-identical to
the last import, so this is an addition rather than a re-measurement.

Q8 needed the three registries Q4 already had -- display name, model URL, and
the licence proof. Both quantizations are the same weights off the same model
card, so they cite it rather than the repository.

Two counts in the prose follow from the new row and are updated with it: the
medium.en gap is twenty-two of thirty-seven, and thirteen configurations rather
than twelve now sit above two times real time. The cutoff sentence still holds
unchanged -- OmniVoice at 1.51x is still the slowest that keeps up and
Chatterbox Q4 at 2.03x still the fastest that does not.

The ballot pool is synced to thirty-seven so Q8 can be ranked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Four notes change and nothing else does: the run is identical, only the prose
in the Notes column moved. The counts in the article still hold at thirty-seven
configurations and thirteen above two times real time, so no prose follows.

Three of the four were stale superlatives or stale figures, which is the failure
the benchmark's own README warns about -- a note claiming "slowest here" is a
claim about the whole table at the moment it was written, and adding a row
falsifies it silently:

  dots-tts    said 11x real time, measures 14.22x, and is now correctly named
              as the slowest rather than left implying Orpheus was
  zonos       said 8x, measures 11.07x
  orpheus-q4  claimed to be the slowest at 8.1x while ranking fourth; now
              states its size and speed and claims nothing about the table
  orpheus-q8  replaces a wrong count of published quantizations with a
              comparison against q4 that the two rows actually support

Each was checked against the columns rather than taken on trust: zonos really
is second on PESQ behind piper (4.02 to 4.10), dots-tts really is the slowest,
and q8's "half a point more accurate" is 7.70% against q4's 8.23%.

A fifth note was refused. The export reverted omnivoice to "no upstream model
card found, so licence is unverified", which is the claim CELL_OVERRIDES exists
to hold down -- the licence was since established as Apache-2.0 from k2-fsa's
own Space, and the export still prints "?" in that column. The override kept
both the licence and the note. Worth fixing upstream in build_benchmark_page.py
so the next export stops trying.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The benchmark submodule now ships its own harness, so tests-repo is gone from
the boot command. Running it from an empty workspace then turned up three more
things, none of which reading the code would have found -- and two of which fail
green.

  TALKITO_AUTO_APPROVE_DOWNLOADS=1
      talkito declines to fetch model weights when it cannot ask a human, and a
      boot command has no TTY. Without it every phrase fails with "Piper model
      unavailable", and the sweep still exits 0 having published a table reading
      0/84 phrases, WER 0.0%, RTF 0.00x. A green run that measured nothing is the
      worst outcome available here.

  ffmpeg
      make_samples.py shells out to it for the wav-to-mp3 step and the sandbox
      image has no ffmpeg. Missing, the numbers are all correct and the page has
      nothing to play.

  TTS_SAMPLE_DIR
      build_benchmark_page.py and build_export.py default it to a hardcoded
      /Users/robertmacrae/... path, so anywhere else they find no clips at all.
      make_samples.py resolves the same variable with expanduser and writes into
      an mp3 subdirectory, so the two want different values and it is set on the
      benchmark.py line rather than once at the top.

Validated from an empty workspace: clone, install, weight download, 84-phrase
sweep, Whisper scoring, page build, serving. About three and a half minutes.

  piper                  82/84  phrases   WER  5.0%   RTF  0.07x

The published Apple M2 figures are 4% and 0.04x. Real-time factor is wall-clock,
so a throttled container reading higher is expected rather than a discrepancy.
The report reaches the browser block at 51 KB with the clip embedded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
build_benchmark_page.py and build_export.py no longer default TTS_SAMPLE_DIR to
an absolute home directory, so the template no longer has to force one. It now
sets nothing, which is what the three scripts want: they need different
directories -- make_samples.py writes mp3s into an "mp3" subdirectory the page
builder reads and the export builder does not -- and each defaults correctly on
its own.

Forcing a single value was what made the export report "0 audio samples". The
workaround was right for the code it was written against and wrong the moment
that code was fixed, which is the usual life of a workaround.

Re-verified from an empty workspace:

  piper                  82/84  phrases   WER  5.3%   RTF  0.08x
  1 configurations, 1 audio samples

The page reaches the browser block at 50 KB with the clip embedded, and the
export zip now carries samples/piper.mp3 rather than nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Selection was uniform over whatever the voter had not heard, and nothing looked
at how often a configuration had been ranked overall. Across 43 real ballots that
left appearances spread 2 to 9 around a mean of 4.7, standard deviation 2.0 --
almost exactly the sqrt(4.7) a Poisson spread predicts, which is the signature of
no balancing at all. Six configurations sat under the display cutoff and showed
no human score, while others were measured twice over.

Selection is now weighted by 1/(count+1) via Efraimidis-Spirakis, which samples
without replacement in one pass -- necessary here because a ballot must never
offer the same clip twice. The anchor draw follows the same rule.

Deliberately a bias and not an ordering. Always serving the k rarest as a set
would leave them only ever ranked against each other, and Bradley-Terry cannot
place a disconnected clique on the same scale as the rest of the table. The
simulation says that worry is not just avoided but inverted -- at today's volume
the weighted selector reaches a connected comparison graph every time where
uniform manages it three runs in four, and it produces slightly more distinct
pairs, because spreading coverage beats concentrating it:

  ballots  selector    min   max  stdev  under-3  connected
       43  uniform     0.9  10.5   2.11      5.1        75%
       43  weighted    2.2   9.8   1.56      1.4       100%
      100  uniform     5.0  23.9   3.80      0.1       100%
      100  weighted    6.9  21.3   2.82      0.0       100%

The uniform arm reproduces the live numbers closely, which is the reason to
believe the weighted column.

1/(count+1) self-attenuates: it biases by over 3x while 2 against 9 decides
whether a row is visible, and fades to near-uniform once everything sits in the
twenties and the deficits stop mattering.

Counts come from an aggregate over counted ballots rather than by reading every
row, since this runs on each issue rather than only when the table renders.
Abandoned ballots are excluded -- they told us nothing about the clips they
offered, and counting them as coverage would starve exactly those clips.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Three rows read "8%" and only one of them was green. The export tones word error
on the unrounded rate while the column prints whole percents, so its cutoff at
0.08 fell inside a displayed band: Orpheus Q8 at 7.70% was under it, XTTS at 8.03%
and Orpheus Q4 at 8.23% were over. Identical text in different colours reads as a
rendering fault rather than a distinction, and no reader can recover the third
decimal that would justify it.

The threshold now applies to the number actually on the page, so two cells that
read the same are guaranteed to be coloured the same whatever the export's own
rounding does next. Green is six percent and below, which moves the boundary off
the crowded 8% band and onto one where the displayed value changes anyway.

Kokoro at 7% and Orpheus Q8 at 8% lose their green; nothing else moves. warn and
bad are left where the export put them, marking the 24%-and-worse rows: those are
not in question, and re-deriving them here would mean a second threshold to keep
in step with the first.

Verified from the rendered page rather than the data alone -- reading computed
styles back out of the table, every distinct displayed value now maps to exactly
one colour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
…e blocks

Two faults, both mine.

The benchmark was pinned to --only piper because I concluded the picker "reads
stdin and exits on EOF, so it cannot be driven from a boot command". That is true
of the docker exec I tested through and false of the thing it stands in for: a
terminal block is a PTY. Driving benchmark.py under a real pty shows the picker
renders and takes input exactly as it does on a laptop -- and it is now an
arrow-key TUI, which is not the shape of something meant to be bypassed. The
reader chooses; the boot command no longer chooses for them.

The duplicate terminal and browser were the setupGuide. A setupGuide is a prompt
addressed to Orcabot chat -- slopcodebench's opens "you are Orcabot chat" and
tells it in as many words not to call create_browser -- and mine was written as
documentation for a human, describing a dashboard that clones a repo and shows a
report. Chat read that as a brief and built both blocks, on top of the two the
template had already placed, leaving a second benchmark writing the same
measurement files as the first. The guide now opens by saying the dashboard is
already complete and forbids create_terminal and create_browser by name.

Interactive selection moves two things that were previously fixed. Samples can no
longer be synthesized before the sweep, because which engines to sample is only
known once the reader has picked, so the selection is read back out of
tools/sweep_rerun.json afterwards and the page and export are rebuilt. And the
page has to be seeded from tools/page_template.html before that rebuild:
build_benchmark_page.py substitutes into an existing page rather than writing one,
and benchmark.py only seeds it on a run it completes -- so quitting the picker
with prior measurements on disk died on FileNotFoundError with every clip already
synthesized. That path is now tested, because it is the one that found the bug.

Also corrects the seeding docs. There is no code seed for cloud:
seedStarterTemplates returns early unless SURFACE_TOKEN is set and only the
desktop app sets it, so every production row is a direct insert -- which is why
the five templates in the prod grid are all hand-inserted under one author.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Nothing plays. make_samples.py exists to embed one clip per engine, audio does
not reach the reader through the browser block, and that clip was the only reason
this template needed ffmpeg. ffmpeg was the only reason it ran apt-get. And
apt-get was the only reason it reached deb.nodesource.com, which is not in the
default egress allowlist and so raised an approval prompt on every fresh
workspace -- the sandbox image installs Node from NodeSource at build time and
leaves /etc/apt/sources.list.d/nodesource.sources behind, so any apt-get update
inside the sandbox reaches for it whatever it was asked to install.

One thing nobody wanted was pulling three the reader had to care about. All of it
goes: no clips, no ffmpeg, no apt-get, no egress prompt. The table is numbers,
which is what it was being read for.

That also retires two gotchas this template had accumulated. Samples no longer
have to be deferred until after the picker, and the page no longer has to be
seeded from page_template.html before a rebuild, because there is no rebuild.

Separately, the report renders as mojibake -- "TTS Engine Benchmark - 24
configurations" arriving as garbage where the em dash should be. The file is
valid UTF-8; it just says so nowhere. The page carries no <meta charset> and
python3 -m http.server sends "text/html" with no charset either, so the browser
falls back to its locale default and reads UTF-8 as cp1252. The tag is prepended
to the published copy rather than patched into the source, so the export zip
stays byte-identical to what the tools wrote.

The real fix for that belongs upstream in tools/page_template.html, which would
also cover the zip and any copy opened from disk rather than served.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
page_template.html declares utf-8 now, so the boot command copies the report
straight through and the injection never fires. Verified against a fresh clone:
the plain-copy branch is taken and the published page carries exactly one meta
tag rather than two.

The fallback stays. The failure it covers is silent and total -- a page that
declares no encoding, served by a server that sends none either, is read as the
browser's locale default and every em dash becomes mojibake -- and this repo
already keeps CELL_OVERRIDES for the same reason: an upstream fix that cannot
quietly revert is worth a few lines. Its comment now says it is a fallback rather
than describing a bug that is fixed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Nothing ever framed them. ReactFlow's fitView is off and cannot simply be turned
on: it fits at mount, when the items are still in flight with the query, so it
would frame an empty canvas and never run again. Blocks the reader creates are
placed relative to the current viewport by findAvailableSpace and are visible by
construction, which is why the gap went unnoticed.

It shows on a dashboard whose coordinates came from somewhere else. Every
hand-written template ships viewport {x:0,y:0,zoom:1} -- a typed placeholder
rather than a computed frame -- so a block at the origin renders under the header
and behind the workspace sidebar. All five benchmark templates carry it. The
three templates exported from a real dashboard carry genuine viewports (zoom
0.69, 0.79, 0.69) and always looked fine, which is why this survived.

The maths is extracted to placement.ts rather than written inline, following
findAvailableSpace: framing is pure geometry, and geometry is worth testing
without React in the way. Which paid immediately -- the first implementation had
the sidebar correction backwards. setCenter puts the given point at the container
midpoint, so screen x is (p - centerX) * zoom + width / 2 and pushing content
clear of the sidebar needs a SMALLER centre, not a larger one. Adding the inset
buries the content further underneath it. Two tests caught it, one of which I had
written to assert the wrong direction, so the projection assertions matter more
than the direction ones: they check where a block actually lands rather than
which way a number moved.

Zoom is capped at 1. Two blocks magnified to fill a wide screen reads as broken,
and zooming in is the one direction a reader can trivially do themselves.

Note for follow-up: ensureVisible carries the same "+ offsetX" correction and
looks wrong the same way, centring roughly half a sidebar too far left when it
rescues an off-screen block. Left alone here rather than changing two behaviours
in one go.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
setupGuide is not documentation. It is a walkthrough brief for Orcabot chat, and
carrying one has two effects that together explain the duplicate blocks.

The dashboards page sets orcabot_initial_prompt to "Walk me through setting up
this dashboard." whenever a template has a guide, and ChatPanel auto-sends it on
mount with the panel expanded -- which is why that sentence appeared as a user
message nobody typed. And the control plane injects the guide into every chat
message on that dashboard, wrapped in a system prompt reading "Use your tools to
do the work (... create_terminal/create_browser as needed)".

So the platform instructs chat to create blocks whenever a guide exists. The
previous fix put "Do NOT call create_terminal" inside the guide, which argued
with the instruction wrapped around it and treated the symptom: it stopped chat
building a second pair, but left it opening, speaking, and being told to build.

There is nothing here to walk through. No keys, no secrets, no provider, no
config file, and a terminal that starts itself. Dropping the field removes the
auto-prompt, the walkthrough system prompt, and the reason chat was invoked at
all. Verified end to end: setup_guide inserts NULL and dashboard creation returns
hasSetupGuide false, so no initial prompt is stashed.

What the guide explained now lives only in this directory's README, which is read
by people rather than injected into a model.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Drops the "no stored results in it" aside, which restated the sentence before it.
Says the report appears in the built-in browser rather than "next to it", which is
what the template actually does and is the more useful thing to know.

Removes both em dashes from the prose. The one left in the methodology section
became a comma, which the clause took without complaint.

The forty em dashes still on the rendered page are not prose: they are the
placeholder a Human cell shows when a configuration has fewer than three ballots,
from TtsResultsTable, and a dash for "no value" in a table is doing a different
job from a dash in a sentence. Left alone.

Dated 22 August, since the table has moved twice since the 17th -- Orpheus Q8
landed and four notes were corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
POST /analytics/events calls requireAuth and stamps every row with a user_id, so
an anonymous reader on a blog, docs or benchmark page produces no record at all.
/admin/metrics is built from users, sessions and those rows -- DAU/WAU/MAU,
signups, retention -- so it counts accounts, not visits, and a link shared
publicly is invisible to it. The only signal a shared page currently leaves is
second-order: whoever signed up afterwards.

Cloudflare Web Analytics fills exactly that gap and nothing else. Cookieless, no
cross-site identifier, no fingerprinting, so it needs no consent banner, which is
what makes it the right choice for pages read by people who have not agreed to
anything.

Two gates. Without NEXT_PUBLIC_CF_BEACON_TOKEN it renders nothing, so it stays
inert until a token is set and cannot leak from a local build. And it never
renders under NEXT_PUBLIC_DESKTOP_MODE: that build is served from localhost
inside Tauri, where beaconing a desktop user's navigation to a third party is not
something a local app should do, and would fail offline anyway.

Both paths verified against the served HTML rather than reasoned about: with a
token the script tag is present with the JSON attribute intact, and with none the
string cloudflareinsights does not appear on the page at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Compute per phrase spans two and a half decades, 0.12s to 51s, and the axis was
linear and pinned at 6.8s -- roughly where the engines that keep up with real
time end. Everything slower was drawn outside the plot: ten of the thirty rated
engines rendered off-canvas at up to cx=5278 in a 780-wide viewBox, while the
accessible label went on claiming thirty.

The word-error chart already had both halves of this right, including the reason
its bound derives from the data: "they were pinned to the range of one export,
and the next one moved a configuration past the right-hand end -- Dots-TTS
stopped being a partial run, finished the whole corpus, and went from 41.7s to
51.0s, which drew its marker outside the plot with nothing to say so." The same
lesson, unlearned one chart lower. This inherits both, so a future export cannot
push a marker off the end.

The scale moves into the component because the plotted set depends on which
engines have ratings, which is fetched. Label side and tooltip side now test
screen position rather than a fraction of the maximum: on a log axis 0.75 of the
range sits far right of three quarters across, so the value-based test they used
flipped almost nothing.

Verified against the rendered SVG with the live scores mocked in: 30 markers, 0
beyond the plot edge, max cx 736 of 780, ticks 0.1s/0.3s/1s/3s/10s/30s matching
the sibling chart's decade and half-decade convention.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Forty configurations now, up from thirty-seven. All three new rows clone from
the piper sample, which is worth knowing when reading their PESQ: it is a
no-reference estimate of signal quality, so an engine reproducing the
highest-PESQ engine in the table is not being scored on the same thing as one
synthesizing its own voice. IndexTTS lands second on PESQ behind piper.

VoxCPM answers a question asked publicly. A commenter listed it among "most
recent advances" the table misses; it was in fact already integrated and had
never been measured, because without a reference voice it ignores the input text.
Given one, it runs: 78/84 at 8% WER, Apache-2.0, 0.5B. Its PESQ is provisional
over 42 of 84 clips and the note says so.

Pocket TTS is Kyutai's, and its licence is the split the proof map exists for:
MIT as code on GitHub, CC-BY-4.0 as weights on the card. The table states the
weights licence, so the card is what the link evidences. Same for VoxCPM, where
the bare openbmb/VoxCPM path 401s and openbmb/VoxCPM-0.5B is the card that
matches the configuration measured. Every licence here was read off the card
rather than taken from a search result.

VoxCPM has no clip. The importer says so ("! no audio for voxcpm2-tts"), the
table disables its button and labels it, and the ballot pool syncs to
thirty-nine rather than forty, because a configuration nobody can play cannot be
ranked. The second line of the article promised every row could be played, which
was true until this import, so it now says almost every row and names the
exception. Synthesizing that clip would let the stronger sentence come back.

Counts follow from the new rows: twenty-five of forty lack a medium.en score,
and fifteen rather than thirteen sit above two times real time. The cutoff
sentence still holds unchanged.

Nothing drifted. The sixteen cells that differ from the last import are all the
normalisation layer working: REWRITE folding stoch-ff to st-ff and "under $5M"
back to "<$5M", and CELL_OVERRIDES holding supertonic's release date and
dots-tts's licence against an export that reverted both again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Three corrections, all to voxcpm2-tts, and nothing else in the table moves.

It has a sample now, so every one of the forty rows is playable again and the
second line of the article says so without the hedge it needed last import. The
ballot pool goes back to forty: a configuration that can be heard can be ranked.

Released fills in as 2024-12, matching the arXiv id its own card carries, and
PESQ settles at 3.78 from 3.83. The note drops the "provisional over 42 of 84
clips" caveat, which reads as the estimate now covering the corpus rather than
half of it -- worth knowing that the row is still partial in the other sense the
README describes, 78 of 84 phrases inside its time budget.

The note is better than the one it replaces in the way that matters here: it
says what the model is rather than what happened while measuring it. Tokenizer-
free, continuous speech space rather than discrete acoustic tokens, on MiniCPM-4
-- which is the "latent VAE, light AR" family a reader asked after by name. And
it states the gap between the upstream 0.17x on a GPU and 2.60x for q4_k on an
M2, so the row cannot be read as a claim about the model at its best.

Nothing drifted again. Eleven cells differ and nine are the normalisation layer:
REWRITE on three NeuTTS licences, and CELL_OVERRIDES holding omnivoice's licence,
size and date, bananamind-tts's size, and both Tada licences against an export
that reverted every one of them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Kyutai, OpenBMB and IndexTeam arrived with Pocket TTS, VoxCPM and IndexTTS last
import and were never added here, because the chip list is maintained by hand
and nothing checks it against the table.

Checking it turned up a fourth: yl4579, whose StyleTTS2 has been in the table
all along without a chip. That one is not from this import; it has simply never
been there. Cross-referencing every row's MODEL_URL against the listed
organisations is what found it, and is worth doing whenever rows are added.

Two apparent gaps are not gaps. cosyvoice3 points at QwenAudio and qwen3-tts-vd
at Qwen while the chip links QwenLM: three handles, one organisation, already
represented.

Logos are each organisation's own GitHub avatar, which is what the rest of the
list uses and what the comment at the top of this file argues for: one
consistent source, published by the organisation to identify itself, implying no
endorsement.

This pushes the list to twenty-five chips and therefore to three rows, which is
not what it was trimmed to twice before. Left at a readable size rather than
shrunk again -- the measurements are in the message that follows, and the fix if
two rows still matter is a smaller one than it looks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Twenty-five chips do not fit two rows at any readable size. The only sizing that
managed it was 0.70rem text with 5px padding, which shrinks all twenty-five to
tidy up one row's wrapping - the wall this was always going to hit, and the
previous pass said so: "the next vendor after that needs another pass rather
than another trim."

So the constraint goes and the space gets used. The chips are bigger than they
were before either trim - 0.85rem on 9px by 14px padding, 38px tall with 18px
logos, against 0.78rem and 32px - and the twenty-five wrap 8 + 9 + 8 with 167px
spare on the last row, where the trimmed size left 11 + 11 + 3 and 840px of that
row empty. A third row with three chips in it reads as a mistake; a third row
with eight reads as a list.

Measured in-page across candidate sizings rather than guessed, which is also how
the ceiling was found: 0.88rem tips to four rows. Worth recording that shrinking
the logos buys nothing, because the width is in the labels, not the images -
tested, not assumed.

Narrower viewports wrap further as they should: four rows at 1024, five at 768.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Forty-six configurations now. Magpie is NVIDIA's 357M multilingual checkpoint at
six precisions, and one card evidences all six, so the licence proof and model
link point at nvidia/magpie_tts_multilingual_357m for each. NVIDIA already had a
vendor chip.

Two sentences in the article stopped being true, and one of them stopped being
true in substance rather than in detail.

The cutoff paragraph said "nothing sits awkwardly against it", and named
OmniVoice at 1.51x and Chatterbox Q4 at 2.03x either side of the line. The
boundary is now Magpie q5_k at 1.98x and Magpie q4_k at 2.01x: two builds of one
checkpoint, three hundredths apart, which the table's own notes describe as
indistinguishable. That is the definition of sitting awkwardly against it, so
the paragraph now says what the boundary is, says the Real time only control is
a rough divide rather than a verdict near it, and says outright that it used to
be clean and is not any more.

The passes rule said feed-forward engines get one. These six get three, on
purpose: six builds differing only in precision need more samples than one run
to say whether they differ at all, and the answer at three runs is that they do
not -- 6.5-7.4% mean word error across the ladder while individual variants
swing by up to 2.4% between runs. Worth stating rather than leaving as an
unexplained 252 in a column the page does not show.

Counts follow: thirty-one of forty-six lack a medium.en score, and nineteen
rather than fifteen sit above two times real time.

One upstream inconsistency, neither ours nor blocking: the card's prose says
364M where its own repo id and this table say 357M.

Verified rendered -- 49 rows for 46 configurations plus the cutoff divider, 46
play buttons, none disabled, six Magpie rows, and all four prose changes live.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The band marking the two-times cutoff rendered three times in one table, each
copy claiming everything below it was slower than real time. Two of the three
were lies, and the rows between them were on both sides of the line.

boundaryAfter tested each adjacent pair and drew a rule wherever the cutoff was
crossed, which is correct only if the ordering separates the two groups. The
comment above BOUNDARY_SORTS says exactly that -- "the only two where a line
drawn at the cutoff means anything ... under any other sort the fast and slow
engines interleave and the boundary is not a place" -- but the code took the
sort column as proof of it rather than checking.

Sorting by RTF separates them by construction. Sorting by compute per phrase did
too, for as long as the two agreed on order, and Magpie ended that: RTF is
compute per second of audio, so builds differing in how long they speak can
order one way per phrase and the other way per second. Under the default sort the
cutoff is now crossed three times, between q8_0 at 1.83x and q4_k at 2.01x, back
under at q5_k at 1.98x, and over again at f16 at 2.13x.

So contiguity is counted rather than assumed: exactly one crossing means the
groups are contiguous and the rule is a place, more than one means they
interleave and there is nothing to draw. Verified by sort -- none under compute
per phrase, one under RTF, none under word error, where it never belonged.

The article promised the line under both sorts, so it now promises it under RTF,
says compute per phrase used to draw it, and says why it stopped. That is the
same fact the cutoff paragraph already had to admit two commits ago: near the
line, per-phrase and per-second compute no longer agree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
…ault sort

Two hundredths, and the whole reason is what lives inside them. Magpie q4_k at
2.01x, q6_k at 2.02x and Chatterbox Q4 at 2.03x sat just above a round 2, and
the table's own notes say the Magpie ladder is not separable - 6.5-7.4% mean word
error across six builds while individual variants swing by up to 2.4% between
runs. Splitting three engines from measurements the table reports as
indistinguishable is a distinction the data cannot carry.

It also split them inconsistently. RTF is compute per second of audio, so builds
that differ in how long they speak order one way per phrase and another per
second: at 2 the cutoff was crossed three times under the default sort, and the
rule marking it had nowhere honest to go, so the contiguity check suppressed it
on the view most readers land on.

At 2.05 the boundary is a place again. Chatterbox Q4 at 2.03x is the slowest
that keeps up, Tada 3B at 2.13x the fastest that does not, and the gap is 0.093x
where it had narrowed to 0.029x. One crossing under both boundary sorts, so the
rule returns to compute per phrase without the contiguity check being loosened -
it is still counted, it now simply passes.

The cutoff lives in the run data rather than the component, so the filter's own
label follows it: "Real time only (under 2.05x)". Thirty configurations keep up
now rather than twenty-seven.

The article says why the line is not round, rather than leaving 2.05 looking
arbitrary. An unexplained odd number reads as a threshold picked to flatter
something; the reason here is the opposite, and worth the two sentences.

Verified by sort: one rule under compute per phrase, one under RTF, none under
word error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
All six were det-ff and are ar-lm. The class came from the export rather than
from anything here - the importer only ever renames stoch-ff to st-ff - so this
is upstream's correction, and it is the right one. NVIDIA's own card describes a
causal transformer decoder that "autoregressively predicts discrete audio codec
tokens", refined over stacked frames and decoded by NanoCodec.

The table was already arguing against itself, which is the part worth keeping.
This page defines det-ff as "a single forward pass with a deterministic duration
predictor, so timing is identical every run and it cannot hallucinate" - and the
Magpie rows are averaged over three free-seed runs precisely because they do not
behave that way, spanning 6.5-7.4% mean word error while a single variant swings
by up to 2.4%. A deterministic model has nothing to average.

The corrected note now says so where a reader can check it: autoregressive
despite the size, the GGUF declares a causal decoder with a codebook refinement
head, and syntheses vary run to run. That is evidence rather than a label.

Class counts move from det-ff 14 / st-ff 5 / ar-lm 27 to 8 / 5 / 33, and the six
change colour in the preference chart, where they were the only large cluster
painted as feed-forward.

Nothing else moved: forty-six configurations, forty-six clips, the 2.05x rule
still drawn once under the default sort.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Peak RSS is the column most likely to decide whether an engine is usable at all,
and until now it could only be sorted, not filtered. "What fits on the machine I
have" was a question the table could answer only by reading down a column.

The stops are spaced by ratio rather than evenly. RSS runs 246 MB to 8.6 GB, but
twenty of the forty-six engines fit inside a gigabyte and only three sit above
four, so an even track would spend most of its length separating the handful of
largest models and almost none of it where the choosing happens. Twelve stops
from 256 MB to no limit, and it starts at no limit so nothing is hidden until
someone asks for less.

A slider rather than a number field because the question is a budget being felt
out against the count beside it, not a figure known in advance - the useful
motion is dragging until the table stops containing what you wanted.

The value is rendered as a size and repeated on aria-valuetext, since a range
input otherwise announces "8", which says nothing; and the label has a minimum
width so the filter row does not shuffle sideways while the handle moves.

Verified against the distribution: 1 configuration under 256 MB, 5 under 512,
20 under 1 GB, 43 under 4 GB, 46 unfiltered. The cutoff rule behaves correctly
underneath it too - it disappears once the filter leaves every remaining engine
on one side of the real-time line, which is the contiguity check doing its job
for a filter it knows nothing about.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Accent orange is what this page uses for a live state - the sounding play
button, the open note, the engaged real-time filter - so a permanently filled
orange track read as something being flagged rather than a budget being set. The
slider now takes the table's own border colour, which is where a control that is
part of the furniture belongs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Forty-seven configurations. FastPitch goes because NVIDIA's own Magpie
supersedes it - same vendor, an autoregressive model with a published
quantisation ladder against a 2023 checkpoint that only reached the table by
speaking an AI disclosure the harness had to strip. It leaves via DROP_ROWS, so
the measurement stays in the raw export; nothing in the prose referred to it, and
its clip is removed automatically.

Dots-TTS MeanFlow closes a gap a reader asked about publicly. It was measured but
absent last cut, which made "you miss meanflow" true in the only sense that
counts. It is the same 2B model the Dots-TTS row measures as SOAR, and the note
says the like-for-like comparison is still being taken rather than implying one.

The Fish Audio row needed its licence corrected before it could be published, and
this is the one column where being wrong costs somebody something. The export
prints Apache-2.0. The model card's frontmatter says `license: other` with
`license_name: fish-audio-research-license`, and its body says research and
non-commercial use is free while commercial use needs a separate agreement.
Apache-2.0 says the opposite, so the row now reads "Fish Audio Research
(non-commercial)", pinned so the next export cannot revert it, and the proof link
points at the card that says so.

Its note also claimed "the largest weights in this table at 6.0 GB and second
slowest at 10.2x". Tada 3B carries 6.4 GB against this row's 5.9, and Dots-TTS at
14.22x and Zonos at 11.07x are both slower, so it is third. Both rankings are
dropped for the figures themselves, which a reader can check against the columns
beside them.

Fish Audio joins the vendor chips, found by re-running the cross-check of every
row's MODEL_URL against the listed organisations. Counts follow: thirty-two of
forty-seven lack a medium.en score, eighteen sit above 2.05x. The real-time rule
still resolves to one crossing, and the boundary is unchanged at Chatterbox Q4
and Tada 3B.

The preview card is re-cut because FastPitch was rated, at 17.2 over three
ballots, so it was a plotted point and not only a row.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Forty-eight configurations. Both corrections made here last import came back
fixed upstream, so the local override goes: keeping it would have replaced a
correct value with a paraphrase of itself, which is not what CELL_OVERRIDES is
for. The licence now reads "Fish research (non-commercial)" in upstream's own
words rather than mine, and the note is better than the one I wrote, because it
carries the restriction into the prose where a reader meets it: "Research
licence: commercial use needs a separate agreement with Fish Audio."

The overrides that remain are the ones upstream keeps reverting - omnivoice for
the fifth cut running, bananamind-tts, supertonic, dots-tts, both Tadas. Those
are guards. A guard against something already fixed is just a fork.

The new row is a control, and it is the most interesting thing in this export.
dots-soar-acpp is the same SOAR checkpoint as the dots-tts row, at q8_0 through
audio.cpp instead of f16 through crispasr, which isolates runtime and precision
from the MeanFlow distillation. Measured against it the distillation is worth
1.16x rather than the 3.3x it appeared to be worth against the f16 crispasr row.
A reader asked publicly whether this table missed MeanFlow; it now carries the
distillation, its control, and the number that separates them.

chatterbox also drops a superlative of its own accord - "the family's best PESQ
at 3.61" becomes "PESQ 3.64 - but turbo beats it on every axis", which is both
a corrected value and a claim that cannot go stale.

Counts follow: thirty-three of forty-eight lack a medium.en score, nineteen sit
above 2.05x. The boundary is unchanged at Chatterbox Q4 and Tada 3B, still one
crossing, and the preview card is re-cut.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Every autoregressive configuration is now averaged over at least three passes,
and the timings moved a long way: thirty-five real-time factors and thirty-four
per-phrase figures changed, some by more than half. Magpie f16 went from 6.98s a
phrase to 2.71s, dots-soar-acpp from 18.59s to 9.32s.

That retires the reason the cutoff was not round. It sat at 2.05x because Magpie
q4_k and q6_k measured 2.01x and 2.02x, and splitting a ladder the table calls
indistinguishable is not a distinction the data can carry. Those builds now
measure between 1.53x and 1.81x, so nothing is inside the gap and the boundary is
Chatterbox Q4 at 1.99x against Tada 3B at 2.21x - a wider margin than the one
2.05 was invented to buy.

The boundary rule no longer appears under the default sort, at any cutoff. I
scanned 1.8 to 2.5 and every value leaves three crossings, because real-time
factor is compute per second of audio while the sort is compute per phrase, and
audio durations now vary enough that magpie-f32 and chatterbox each sit on the
wrong side of their neighbours. The contiguity check does the right thing and
draws nothing; the article says so rather than leaving a reader wondering where
the line went.

The passes paragraph now explains itself. Sampling is the reason the two classes
differ - a deterministic forward pass gives the same audio twice, so a second run
measures nothing - and Chatterbox Turbo is the illustration: one pass put it
joint-best on word error at 4%, three put it at 5% and behind three others. A
handful of configurations that still moved got four passes or five.

The corrected chatterbox note is in too, checked figure by figure. Turbo no
longer "beats it on every axis": it is ahead on word error and 4.3x quicker, and
behind on PESQ at 3.30 against 3.61, which the note now calls quality against
speed rather than a clean win.

Counts: thirty-three of forty-eight lack a medium.en score, seventeen sit above
2x. Also caught the filter description still claiming 2.05x while the control
beside it said 2x.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Two limits now: 2x real time, and 6.5 seconds of compute a phrase. The rule uses
whichever matches the active sort.

This fixes the boundary properly rather than by choosing a luckier number.
Sorting by one measure and cutting on another can interleave, and after the
three-pass rerun it always did - every cutoff from 1.8 to 2.5 left three
crossings under the default sort, because real-time factor is compute per second
of audio and engines differ in how long they speak. Sorting by a measure and
cutting on that same measure can only cross once, so the rule is now drawn under
both speed columns instead of neither.

6.5s is not an invented threshold. It sits in a 0.76s gap, between Chatterbox Q4
at 6.27s and Q8 at 7.03s, and classifies the table identically to 2x but for one
row: Magpie's full-precision build, quick per phrase at 3.79s and slow per second
at 2.31x. A reader who clicks a different header is shown the same population,
not a different one.

The label follows the measure - "over 6.5s of compute a phrase" against "slower
than real time, over 2x" - because a line cut on seconds should not be explained
in multiples of real time. Both limits live in the run data beside each other,
since where a line sits is an editorial choice rather than a rendering detail.

The article said sorting by compute per phrase does not draw the line, which was
true for exactly one commit. It now says the line is cut on whichever column you
sorted by, and gives both boundaries.

Verified by sort: one rule under compute per phrase between Chatterbox Q4 and Q8,
one under RTF between Chatterbox Q4 and Tada 3B, none under word error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
robdmac and others added 4 commits November 17, 2026 15:07
New row from the n=3 export: LFM2 backbone over NVIDIA's NeMo nano codec,
370M params, 10% WER at 3.86 PESQ, 1.50x RTF / 6.86s a phrase.

Purely additive - zero drift across the 48 pre-existing rows.

Licence rewrite generalised from a per-model rule to a pattern, so
"LFM1.0, under $10M" and "NeuTTS, under $5M" both shorten the same way.
Verified NeuTTS renders identically to before. The proof link is the model
card, which declares `license: lfm1.0`; the $10M ceiling is a term of that
licence rather than of the card, same as NeuTTS's $5M.

Kani makes the two speed measures disagree in a second, opposite direction:
fast per second of audio at 1.50x but slow per phrase at 6.86s, where
Magpie F32 is 3.79s and 2.31x. Methodology updated - that pair is now the
argument for cutting on the sorted column, since no single ordering puts
both on the right side of one cut.

Prose counts refreshed against the new row set: medium.en now misses
thirty-four of forty-nine, and the synth boundary reads 6.27s against
6.86s. The seventeen above real time is unchanged.

Verified by render: divider falls directly above Kani under the compute
sort, vendor chip and licence cell correct.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The real-time line is drawn on whichever speed column you sorted by, so it
needs a limit in each measure. Moving the per-phrase limit from 6.5s to 7s
buys agreement between the two: at 6.5s both Magpie F32 and Kani sat on the
wrong side of one of them, and at 7s only Magpie does, which is what the
number is chosen for.

The cost is a tight boundary, and it is stated rather than glossed. 7s falls
in a 0.17s gap, Kani at 6.86s against Chatterbox Q8 at 7.03s, where 6.5s had
0.59s of clear air. That is well inside what three passes resolve, so the
methodology now says those two share a position instead of claiming nothing
sits awkwardly against the line - the same standard applied to Magpie's
2.01/2.02/2.05 ladder. 7.5s is not a way out: it pulls Q8 inside at 2.21x
and buys a fresh disagreement for the one it removes.

Counts move with it: 33 within the per-phrase line and 16 above, while the
seventeen above 2x real time are unchanged, that line not having moved.

Vendor chips: adding nineninesix took the set to 27, which spilled 8+9+9+1
and stranded Zyphra alone on a fourth row. Horizontal padding 14px -> 13px
gives an even 9+9+9 with 82px spare. The label carries the width, so the
font stays at 0.85rem and the logos are untouched. Measured in-page across
a grid of font/padding/gap values, not guessed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
"Avg synth" named the operation rather than the quantity, and spent ten
characters doing it in a column four wide. It is a mean of a time, so it is
now written as one: x̄ t, with the t italic as a variable. Renamed at the
importer so the data, both scatter charts, the OG card and the prose all read
from one name.

The logo column sits between the engine name and Human, with no title and no
sort, because it carries no value to order by. It answers "whose model is
this" per row - which the chip list above the table could not, since nothing
there says which chip belongs to which row.

Each logo is resolved from the model link the row already carries rather than
from a second config-to-vendor table, so it cannot drift from the name beside
it. Three orgs need an alias (Qwen ships from three accounts, and the Magpie
rows point at model cards under `nvidia` rather than the NeMo repo); that is
done by explicit alias rather than by loosening the URL match, since a prefix
rule would eventually put the wrong logo on a row, which is worse than none.

Verified both directions: all 49 rows resolve, all 27 vendors are used by at
least one row, every icon file exists, and in-page all 49 images decode with
27 distinct vendors and no page errors.

The vendor list moved to ttsVendors, shared by the chips and the table, so a
vendor cannot be added to one and forgotten in the other - which is how the
chips fell out of step with the rows before.

The header is blank to the eye but announces "Vendor" to a screen reader; a
bare th would leave the column's cells unlabelled in table mode.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The card in the repo was built on 27 August and had not moved since, because
the scores API was unreachable from the machine doing the importing and the
local database has no ballots at all - building from that would have emptied
the chart rather than refreshed it.

Prod is reachable again, so it is rebuilt from real votes: 37 engines with at
least three ballots, r = 0.10. The headline claim is unchanged, and still
holds on the larger sample - compute barely predicts preference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
@robdmac
robdmac merged commit 1b6aee5 into main Nov 17, 2026
1 check passed
@robdmac
robdmac deleted the feat/tts-benchmark-template branch November 17, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant