Repository navigation
Feat/tts benchmark template - #284
Merged
Merged
Conversation
The open-weight TTS benchmark is the first one here that needs no agent, no provider key and no orchestrator: it measures speech engines rather than coding agents, so the whole run fits in a terminal's boot command and the report it writes is served to a browser block beside it. Two things the sandbox forces, both verified rather than assumed: talkito's .gitmodules points at git@github.com: for both submodules, and the sandbox has no SSH key, so --recurse-submodules cannot work. The three public repos are cloned over HTTPS into the layout tools/paths.py resolves. The image installs the talkito package but keeps no checkout, so all three have to be fetched at run time. tools/benchmark.py with no arguments opens an interactive picker that reads stdin and exits on EOF, so it cannot be driven from a boot command. --only names the models and skips it. The web server starts before the clone rather than after the run, so the browser block has a page from the first second instead of a connection error for the length of the benchmark. Samples are synthesized before the sweep because benchmark.py builds the page at the end, and a clip made afterwards would not be in it. Pinned to piper: it needs no torch at all and faster-whisper scores on CPU, so the arm installs in seconds. A full sweep of all 73 providers is around eleven hours and is not something a template should make easy to start by accident. Not yet validated end to end. The clone layout, dependency install, report serving and browser wiring all work in the sandbox; the run itself stops at benchmark/tools/paths.py, which requires tests-repo/test_tts_asr_roundtrip.py — a file that is not in the public talkito-tests repo. That blocks anyone trying to run the benchmark, not just this template. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Same shape as the agent-skills card - 1200x630, title, finding, chart, footer - and the palette is sampled from it rather than guessed, so the two sit beside each other. Generated rather than drawn. The y-axis is reader preference, which arrives by ballot and keeps moving, so a hand-made PNG is stale the week after it ships and cannot be re-cut without redrawing it. The subtitle's r is computed in the same pass as the points for the same reason: a re-cut must not leave a stale number next to fresh data. The x-axis is logarithmic, and that is the one deliberate difference from the on-page chart. The page fixes x at 6.8s, which covers the engines that keep up with real time; ten of the thirty rated engines are slower than that, out to 51s a phrase. A card has no filter controls and no hover, so it has to show every point it counts, and log x fits 0.12s and 51s in one frame. The flatness of the cloud across two and a half decades is the finding: r = 0.21, and the sub-second engines alone span 1 to 94. Two layout bugs found by looking at the output rather than trusting the code. Markers and labels are drawn in separate passes, because in one pass a later marker paints over an earlier label - "NeuTTS-2E Q8" rendered as "euTTS-2E Q8". And the y-axis runs to 105 while its ticks stop at 100, so the best-rated engine sits below the frame rather than against it and its label has somewhere to go that is not the densest part of the cloud. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Thirty-seven configurations now, up from thirty-six. Q8 is the only new row the page shows: the export also carries the three NeuTTS-2E CPU builds, which DROP_ROWS already excludes because they are the same weights on a slower path and say nothing about the model. Nothing else moved. Every cell of every pre-existing row is byte-identical to the last import, so this is an addition rather than a re-measurement. Q8 needed the three registries Q4 already had -- display name, model URL, and the licence proof. Both quantizations are the same weights off the same model card, so they cite it rather than the repository. Two counts in the prose follow from the new row and are updated with it: the medium.en gap is twenty-two of thirty-seven, and thirteen configurations rather than twelve now sit above two times real time. The cutoff sentence still holds unchanged -- OmniVoice at 1.51x is still the slowest that keeps up and Chatterbox Q4 at 2.03x still the fastest that does not. The ballot pool is synced to thirty-seven so Q8 can be ranked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Four notes change and nothing else does: the run is identical, only the prose
in the Notes column moved. The counts in the article still hold at thirty-seven
configurations and thirteen above two times real time, so no prose follows.
Three of the four were stale superlatives or stale figures, which is the failure
the benchmark's own README warns about -- a note claiming "slowest here" is a
claim about the whole table at the moment it was written, and adding a row
falsifies it silently:
dots-tts said 11x real time, measures 14.22x, and is now correctly named
as the slowest rather than left implying Orpheus was
zonos said 8x, measures 11.07x
orpheus-q4 claimed to be the slowest at 8.1x while ranking fourth; now
states its size and speed and claims nothing about the table
orpheus-q8 replaces a wrong count of published quantizations with a
comparison against q4 that the two rows actually support
Each was checked against the columns rather than taken on trust: zonos really
is second on PESQ behind piper (4.02 to 4.10), dots-tts really is the slowest,
and q8's "half a point more accurate" is 7.70% against q4's 8.23%.
A fifth note was refused. The export reverted omnivoice to "no upstream model
card found, so licence is unverified", which is the claim CELL_OVERRIDES exists
to hold down -- the licence was since established as Apache-2.0 from k2-fsa's
own Space, and the export still prints "?" in that column. The override kept
both the licence and the note. Worth fixing upstream in build_benchmark_page.py
so the next export stops trying.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The benchmark submodule now ships its own harness, so tests-repo is gone from
the boot command. Running it from an empty workspace then turned up three more
things, none of which reading the code would have found -- and two of which fail
green.
TALKITO_AUTO_APPROVE_DOWNLOADS=1
talkito declines to fetch model weights when it cannot ask a human, and a
boot command has no TTY. Without it every phrase fails with "Piper model
unavailable", and the sweep still exits 0 having published a table reading
0/84 phrases, WER 0.0%, RTF 0.00x. A green run that measured nothing is the
worst outcome available here.
ffmpeg
make_samples.py shells out to it for the wav-to-mp3 step and the sandbox
image has no ffmpeg. Missing, the numbers are all correct and the page has
nothing to play.
TTS_SAMPLE_DIR
build_benchmark_page.py and build_export.py default it to a hardcoded
/Users/robertmacrae/... path, so anywhere else they find no clips at all.
make_samples.py resolves the same variable with expanduser and writes into
an mp3 subdirectory, so the two want different values and it is set on the
benchmark.py line rather than once at the top.
Validated from an empty workspace: clone, install, weight download, 84-phrase
sweep, Whisper scoring, page build, serving. About three and a half minutes.
piper 82/84 phrases WER 5.0% RTF 0.07x
The published Apple M2 figures are 4% and 0.04x. Real-time factor is wall-clock,
so a throttled container reading higher is expected rather than a discrepancy.
The report reaches the browser block at 51 KB with the clip embedded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
build_benchmark_page.py and build_export.py no longer default TTS_SAMPLE_DIR to an absolute home directory, so the template no longer has to force one. It now sets nothing, which is what the three scripts want: they need different directories -- make_samples.py writes mp3s into an "mp3" subdirectory the page builder reads and the export builder does not -- and each defaults correctly on its own. Forcing a single value was what made the export report "0 audio samples". The workaround was right for the code it was written against and wrong the moment that code was fixed, which is the usual life of a workaround. Re-verified from an empty workspace: piper 82/84 phrases WER 5.3% RTF 0.08x 1 configurations, 1 audio samples The page reaches the browser block at 50 KB with the clip embedded, and the export zip now carries samples/piper.mp3 rather than nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Selection was uniform over whatever the voter had not heard, and nothing looked
at how often a configuration had been ranked overall. Across 43 real ballots that
left appearances spread 2 to 9 around a mean of 4.7, standard deviation 2.0 --
almost exactly the sqrt(4.7) a Poisson spread predicts, which is the signature of
no balancing at all. Six configurations sat under the display cutoff and showed
no human score, while others were measured twice over.
Selection is now weighted by 1/(count+1) via Efraimidis-Spirakis, which samples
without replacement in one pass -- necessary here because a ballot must never
offer the same clip twice. The anchor draw follows the same rule.
Deliberately a bias and not an ordering. Always serving the k rarest as a set
would leave them only ever ranked against each other, and Bradley-Terry cannot
place a disconnected clique on the same scale as the rest of the table. The
simulation says that worry is not just avoided but inverted -- at today's volume
the weighted selector reaches a connected comparison graph every time where
uniform manages it three runs in four, and it produces slightly more distinct
pairs, because spreading coverage beats concentrating it:
ballots selector min max stdev under-3 connected
43 uniform 0.9 10.5 2.11 5.1 75%
43 weighted 2.2 9.8 1.56 1.4 100%
100 uniform 5.0 23.9 3.80 0.1 100%
100 weighted 6.9 21.3 2.82 0.0 100%
The uniform arm reproduces the live numbers closely, which is the reason to
believe the weighted column.
1/(count+1) self-attenuates: it biases by over 3x while 2 against 9 decides
whether a row is visible, and fades to near-uniform once everything sits in the
twenties and the deficits stop mattering.
Counts come from an aggregate over counted ballots rather than by reading every
row, since this runs on each issue rather than only when the table renders.
Abandoned ballots are excluded -- they told us nothing about the clips they
offered, and counting them as coverage would starve exactly those clips.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Three rows read "8%" and only one of them was green. The export tones word error on the unrounded rate while the column prints whole percents, so its cutoff at 0.08 fell inside a displayed band: Orpheus Q8 at 7.70% was under it, XTTS at 8.03% and Orpheus Q4 at 8.23% were over. Identical text in different colours reads as a rendering fault rather than a distinction, and no reader can recover the third decimal that would justify it. The threshold now applies to the number actually on the page, so two cells that read the same are guaranteed to be coloured the same whatever the export's own rounding does next. Green is six percent and below, which moves the boundary off the crowded 8% band and onto one where the displayed value changes anyway. Kokoro at 7% and Orpheus Q8 at 8% lose their green; nothing else moves. warn and bad are left where the export put them, marking the 24%-and-worse rows: those are not in question, and re-deriving them here would mean a second threshold to keep in step with the first. Verified from the rendered page rather than the data alone -- reading computed styles back out of the table, every distinct displayed value now maps to exactly one colour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
…e blocks Two faults, both mine. The benchmark was pinned to --only piper because I concluded the picker "reads stdin and exits on EOF, so it cannot be driven from a boot command". That is true of the docker exec I tested through and false of the thing it stands in for: a terminal block is a PTY. Driving benchmark.py under a real pty shows the picker renders and takes input exactly as it does on a laptop -- and it is now an arrow-key TUI, which is not the shape of something meant to be bypassed. The reader chooses; the boot command no longer chooses for them. The duplicate terminal and browser were the setupGuide. A setupGuide is a prompt addressed to Orcabot chat -- slopcodebench's opens "you are Orcabot chat" and tells it in as many words not to call create_browser -- and mine was written as documentation for a human, describing a dashboard that clones a repo and shows a report. Chat read that as a brief and built both blocks, on top of the two the template had already placed, leaving a second benchmark writing the same measurement files as the first. The guide now opens by saying the dashboard is already complete and forbids create_terminal and create_browser by name. Interactive selection moves two things that were previously fixed. Samples can no longer be synthesized before the sweep, because which engines to sample is only known once the reader has picked, so the selection is read back out of tools/sweep_rerun.json afterwards and the page and export are rebuilt. And the page has to be seeded from tools/page_template.html before that rebuild: build_benchmark_page.py substitutes into an existing page rather than writing one, and benchmark.py only seeds it on a run it completes -- so quitting the picker with prior measurements on disk died on FileNotFoundError with every clip already synthesized. That path is now tested, because it is the one that found the bug. Also corrects the seeding docs. There is no code seed for cloud: seedStarterTemplates returns early unless SURFACE_TOKEN is set and only the desktop app sets it, so every production row is a direct insert -- which is why the five templates in the prod grid are all hand-inserted under one author. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Nothing plays. make_samples.py exists to embed one clip per engine, audio does not reach the reader through the browser block, and that clip was the only reason this template needed ffmpeg. ffmpeg was the only reason it ran apt-get. And apt-get was the only reason it reached deb.nodesource.com, which is not in the default egress allowlist and so raised an approval prompt on every fresh workspace -- the sandbox image installs Node from NodeSource at build time and leaves /etc/apt/sources.list.d/nodesource.sources behind, so any apt-get update inside the sandbox reaches for it whatever it was asked to install. One thing nobody wanted was pulling three the reader had to care about. All of it goes: no clips, no ffmpeg, no apt-get, no egress prompt. The table is numbers, which is what it was being read for. That also retires two gotchas this template had accumulated. Samples no longer have to be deferred until after the picker, and the page no longer has to be seeded from page_template.html before a rebuild, because there is no rebuild. Separately, the report renders as mojibake -- "TTS Engine Benchmark - 24 configurations" arriving as garbage where the em dash should be. The file is valid UTF-8; it just says so nowhere. The page carries no <meta charset> and python3 -m http.server sends "text/html" with no charset either, so the browser falls back to its locale default and reads UTF-8 as cp1252. The tag is prepended to the published copy rather than patched into the source, so the export zip stays byte-identical to what the tools wrote. The real fix for that belongs upstream in tools/page_template.html, which would also cover the zip and any copy opened from disk rather than served. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
page_template.html declares utf-8 now, so the boot command copies the report straight through and the injection never fires. Verified against a fresh clone: the plain-copy branch is taken and the published page carries exactly one meta tag rather than two. The fallback stays. The failure it covers is silent and total -- a page that declares no encoding, served by a server that sends none either, is read as the browser's locale default and every em dash becomes mojibake -- and this repo already keeps CELL_OVERRIDES for the same reason: an upstream fix that cannot quietly revert is worth a few lines. Its comment now says it is a fallback rather than describing a bug that is fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Nothing ever framed them. ReactFlow's fitView is off and cannot simply be turned
on: it fits at mount, when the items are still in flight with the query, so it
would frame an empty canvas and never run again. Blocks the reader creates are
placed relative to the current viewport by findAvailableSpace and are visible by
construction, which is why the gap went unnoticed.
It shows on a dashboard whose coordinates came from somewhere else. Every
hand-written template ships viewport {x:0,y:0,zoom:1} -- a typed placeholder
rather than a computed frame -- so a block at the origin renders under the header
and behind the workspace sidebar. All five benchmark templates carry it. The
three templates exported from a real dashboard carry genuine viewports (zoom
0.69, 0.79, 0.69) and always looked fine, which is why this survived.
The maths is extracted to placement.ts rather than written inline, following
findAvailableSpace: framing is pure geometry, and geometry is worth testing
without React in the way. Which paid immediately -- the first implementation had
the sidebar correction backwards. setCenter puts the given point at the container
midpoint, so screen x is (p - centerX) * zoom + width / 2 and pushing content
clear of the sidebar needs a SMALLER centre, not a larger one. Adding the inset
buries the content further underneath it. Two tests caught it, one of which I had
written to assert the wrong direction, so the projection assertions matter more
than the direction ones: they check where a block actually lands rather than
which way a number moved.
Zoom is capped at 1. Two blocks magnified to fill a wide screen reads as broken,
and zooming in is the one direction a reader can trivially do themselves.
Note for follow-up: ensureVisible carries the same "+ offsetX" correction and
looks wrong the same way, centring roughly half a sidebar too far left when it
rescues an off-screen block. Left alone here rather than changing two behaviours
in one go.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
setupGuide is not documentation. It is a walkthrough brief for Orcabot chat, and carrying one has two effects that together explain the duplicate blocks. The dashboards page sets orcabot_initial_prompt to "Walk me through setting up this dashboard." whenever a template has a guide, and ChatPanel auto-sends it on mount with the panel expanded -- which is why that sentence appeared as a user message nobody typed. And the control plane injects the guide into every chat message on that dashboard, wrapped in a system prompt reading "Use your tools to do the work (... create_terminal/create_browser as needed)". So the platform instructs chat to create blocks whenever a guide exists. The previous fix put "Do NOT call create_terminal" inside the guide, which argued with the instruction wrapped around it and treated the symptom: it stopped chat building a second pair, but left it opening, speaking, and being told to build. There is nothing here to walk through. No keys, no secrets, no provider, no config file, and a terminal that starts itself. Dropping the field removes the auto-prompt, the walkthrough system prompt, and the reason chat was invoked at all. Verified end to end: setup_guide inserts NULL and dashboard creation returns hasSetupGuide false, so no initial prompt is stashed. What the guide explained now lives only in this directory's README, which is read by people rather than injected into a model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Drops the "no stored results in it" aside, which restated the sentence before it. Says the report appears in the built-in browser rather than "next to it", which is what the template actually does and is the more useful thing to know. Removes both em dashes from the prose. The one left in the methodology section became a comma, which the clause took without complaint. The forty em dashes still on the rendered page are not prose: they are the placeholder a Human cell shows when a configuration has fewer than three ballots, from TtsResultsTable, and a dash for "no value" in a table is doing a different job from a dash in a sentence. Left alone. Dated 22 August, since the table has moved twice since the 17th -- Orpheus Q8 landed and four notes were corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
POST /analytics/events calls requireAuth and stamps every row with a user_id, so an anonymous reader on a blog, docs or benchmark page produces no record at all. /admin/metrics is built from users, sessions and those rows -- DAU/WAU/MAU, signups, retention -- so it counts accounts, not visits, and a link shared publicly is invisible to it. The only signal a shared page currently leaves is second-order: whoever signed up afterwards. Cloudflare Web Analytics fills exactly that gap and nothing else. Cookieless, no cross-site identifier, no fingerprinting, so it needs no consent banner, which is what makes it the right choice for pages read by people who have not agreed to anything. Two gates. Without NEXT_PUBLIC_CF_BEACON_TOKEN it renders nothing, so it stays inert until a token is set and cannot leak from a local build. And it never renders under NEXT_PUBLIC_DESKTOP_MODE: that build is served from localhost inside Tauri, where beaconing a desktop user's navigation to a third party is not something a local app should do, and would fail offline anyway. Both paths verified against the served HTML rather than reasoned about: with a token the script tag is present with the JSON attribute intact, and with none the string cloudflareinsights does not appear on the page at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Compute per phrase spans two and a half decades, 0.12s to 51s, and the axis was linear and pinned at 6.8s -- roughly where the engines that keep up with real time end. Everything slower was drawn outside the plot: ten of the thirty rated engines rendered off-canvas at up to cx=5278 in a 780-wide viewBox, while the accessible label went on claiming thirty. The word-error chart already had both halves of this right, including the reason its bound derives from the data: "they were pinned to the range of one export, and the next one moved a configuration past the right-hand end -- Dots-TTS stopped being a partial run, finished the whole corpus, and went from 41.7s to 51.0s, which drew its marker outside the plot with nothing to say so." The same lesson, unlearned one chart lower. This inherits both, so a future export cannot push a marker off the end. The scale moves into the component because the plotted set depends on which engines have ratings, which is fetched. Label side and tooltip side now test screen position rather than a fraction of the maximum: on a log axis 0.75 of the range sits far right of three quarters across, so the value-based test they used flipped almost nothing. Verified against the rendered SVG with the live scores mocked in: 30 markers, 0 beyond the plot edge, max cx 736 of 780, ticks 0.1s/0.3s/1s/3s/10s/30s matching the sibling chart's decade and half-decade convention. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Forty configurations now, up from thirty-seven. All three new rows clone from
the piper sample, which is worth knowing when reading their PESQ: it is a
no-reference estimate of signal quality, so an engine reproducing the
highest-PESQ engine in the table is not being scored on the same thing as one
synthesizing its own voice. IndexTTS lands second on PESQ behind piper.
VoxCPM answers a question asked publicly. A commenter listed it among "most
recent advances" the table misses; it was in fact already integrated and had
never been measured, because without a reference voice it ignores the input text.
Given one, it runs: 78/84 at 8% WER, Apache-2.0, 0.5B. Its PESQ is provisional
over 42 of 84 clips and the note says so.
Pocket TTS is Kyutai's, and its licence is the split the proof map exists for:
MIT as code on GitHub, CC-BY-4.0 as weights on the card. The table states the
weights licence, so the card is what the link evidences. Same for VoxCPM, where
the bare openbmb/VoxCPM path 401s and openbmb/VoxCPM-0.5B is the card that
matches the configuration measured. Every licence here was read off the card
rather than taken from a search result.
VoxCPM has no clip. The importer says so ("! no audio for voxcpm2-tts"), the
table disables its button and labels it, and the ballot pool syncs to
thirty-nine rather than forty, because a configuration nobody can play cannot be
ranked. The second line of the article promised every row could be played, which
was true until this import, so it now says almost every row and names the
exception. Synthesizing that clip would let the stronger sentence come back.
Counts follow from the new rows: twenty-five of forty lack a medium.en score,
and fifteen rather than thirteen sit above two times real time. The cutoff
sentence still holds unchanged.
Nothing drifted. The sixteen cells that differ from the last import are all the
normalisation layer working: REWRITE folding stoch-ff to st-ff and "under $5M"
back to "<$5M", and CELL_OVERRIDES holding supertonic's release date and
dots-tts's licence against an export that reverted both again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Three corrections, all to voxcpm2-tts, and nothing else in the table moves. It has a sample now, so every one of the forty rows is playable again and the second line of the article says so without the hedge it needed last import. The ballot pool goes back to forty: a configuration that can be heard can be ranked. Released fills in as 2024-12, matching the arXiv id its own card carries, and PESQ settles at 3.78 from 3.83. The note drops the "provisional over 42 of 84 clips" caveat, which reads as the estimate now covering the corpus rather than half of it -- worth knowing that the row is still partial in the other sense the README describes, 78 of 84 phrases inside its time budget. The note is better than the one it replaces in the way that matters here: it says what the model is rather than what happened while measuring it. Tokenizer- free, continuous speech space rather than discrete acoustic tokens, on MiniCPM-4 -- which is the "latent VAE, light AR" family a reader asked after by name. And it states the gap between the upstream 0.17x on a GPU and 2.60x for q4_k on an M2, so the row cannot be read as a claim about the model at its best. Nothing drifted again. Eleven cells differ and nine are the normalisation layer: REWRITE on three NeuTTS licences, and CELL_OVERRIDES holding omnivoice's licence, size and date, bananamind-tts's size, and both Tada licences against an export that reverted every one of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Kyutai, OpenBMB and IndexTeam arrived with Pocket TTS, VoxCPM and IndexTTS last import and were never added here, because the chip list is maintained by hand and nothing checks it against the table. Checking it turned up a fourth: yl4579, whose StyleTTS2 has been in the table all along without a chip. That one is not from this import; it has simply never been there. Cross-referencing every row's MODEL_URL against the listed organisations is what found it, and is worth doing whenever rows are added. Two apparent gaps are not gaps. cosyvoice3 points at QwenAudio and qwen3-tts-vd at Qwen while the chip links QwenLM: three handles, one organisation, already represented. Logos are each organisation's own GitHub avatar, which is what the rest of the list uses and what the comment at the top of this file argues for: one consistent source, published by the organisation to identify itself, implying no endorsement. This pushes the list to twenty-five chips and therefore to three rows, which is not what it was trimmed to twice before. Left at a readable size rather than shrunk again -- the measurements are in the message that follows, and the fix if two rows still matter is a smaller one than it looks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Twenty-five chips do not fit two rows at any readable size. The only sizing that managed it was 0.70rem text with 5px padding, which shrinks all twenty-five to tidy up one row's wrapping - the wall this was always going to hit, and the previous pass said so: "the next vendor after that needs another pass rather than another trim." So the constraint goes and the space gets used. The chips are bigger than they were before either trim - 0.85rem on 9px by 14px padding, 38px tall with 18px logos, against 0.78rem and 32px - and the twenty-five wrap 8 + 9 + 8 with 167px spare on the last row, where the trimmed size left 11 + 11 + 3 and 840px of that row empty. A third row with three chips in it reads as a mistake; a third row with eight reads as a list. Measured in-page across candidate sizings rather than guessed, which is also how the ceiling was found: 0.88rem tips to four rows. Worth recording that shrinking the logos buys nothing, because the width is in the labels, not the images - tested, not assumed. Narrower viewports wrap further as they should: four rows at 1024, five at 768. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Forty-six configurations now. Magpie is NVIDIA's 357M multilingual checkpoint at six precisions, and one card evidences all six, so the licence proof and model link point at nvidia/magpie_tts_multilingual_357m for each. NVIDIA already had a vendor chip. Two sentences in the article stopped being true, and one of them stopped being true in substance rather than in detail. The cutoff paragraph said "nothing sits awkwardly against it", and named OmniVoice at 1.51x and Chatterbox Q4 at 2.03x either side of the line. The boundary is now Magpie q5_k at 1.98x and Magpie q4_k at 2.01x: two builds of one checkpoint, three hundredths apart, which the table's own notes describe as indistinguishable. That is the definition of sitting awkwardly against it, so the paragraph now says what the boundary is, says the Real time only control is a rough divide rather than a verdict near it, and says outright that it used to be clean and is not any more. The passes rule said feed-forward engines get one. These six get three, on purpose: six builds differing only in precision need more samples than one run to say whether they differ at all, and the answer at three runs is that they do not -- 6.5-7.4% mean word error across the ladder while individual variants swing by up to 2.4% between runs. Worth stating rather than leaving as an unexplained 252 in a column the page does not show. Counts follow: thirty-one of forty-six lack a medium.en score, and nineteen rather than fifteen sit above two times real time. One upstream inconsistency, neither ours nor blocking: the card's prose says 364M where its own repo id and this table say 357M. Verified rendered -- 49 rows for 46 configurations plus the cutoff divider, 46 play buttons, none disabled, six Magpie rows, and all four prose changes live. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The band marking the two-times cutoff rendered three times in one table, each copy claiming everything below it was slower than real time. Two of the three were lies, and the rows between them were on both sides of the line. boundaryAfter tested each adjacent pair and drew a rule wherever the cutoff was crossed, which is correct only if the ordering separates the two groups. The comment above BOUNDARY_SORTS says exactly that -- "the only two where a line drawn at the cutoff means anything ... under any other sort the fast and slow engines interleave and the boundary is not a place" -- but the code took the sort column as proof of it rather than checking. Sorting by RTF separates them by construction. Sorting by compute per phrase did too, for as long as the two agreed on order, and Magpie ended that: RTF is compute per second of audio, so builds differing in how long they speak can order one way per phrase and the other way per second. Under the default sort the cutoff is now crossed three times, between q8_0 at 1.83x and q4_k at 2.01x, back under at q5_k at 1.98x, and over again at f16 at 2.13x. So contiguity is counted rather than assumed: exactly one crossing means the groups are contiguous and the rule is a place, more than one means they interleave and there is nothing to draw. Verified by sort -- none under compute per phrase, one under RTF, none under word error, where it never belonged. The article promised the line under both sorts, so it now promises it under RTF, says compute per phrase used to draw it, and says why it stopped. That is the same fact the cutoff paragraph already had to admit two commits ago: near the line, per-phrase and per-second compute no longer agree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
…ault sort Two hundredths, and the whole reason is what lives inside them. Magpie q4_k at 2.01x, q6_k at 2.02x and Chatterbox Q4 at 2.03x sat just above a round 2, and the table's own notes say the Magpie ladder is not separable - 6.5-7.4% mean word error across six builds while individual variants swing by up to 2.4% between runs. Splitting three engines from measurements the table reports as indistinguishable is a distinction the data cannot carry. It also split them inconsistently. RTF is compute per second of audio, so builds that differ in how long they speak order one way per phrase and another per second: at 2 the cutoff was crossed three times under the default sort, and the rule marking it had nowhere honest to go, so the contiguity check suppressed it on the view most readers land on. At 2.05 the boundary is a place again. Chatterbox Q4 at 2.03x is the slowest that keeps up, Tada 3B at 2.13x the fastest that does not, and the gap is 0.093x where it had narrowed to 0.029x. One crossing under both boundary sorts, so the rule returns to compute per phrase without the contiguity check being loosened - it is still counted, it now simply passes. The cutoff lives in the run data rather than the component, so the filter's own label follows it: "Real time only (under 2.05x)". Thirty configurations keep up now rather than twenty-seven. The article says why the line is not round, rather than leaving 2.05 looking arbitrary. An unexplained odd number reads as a threshold picked to flatter something; the reason here is the opposite, and worth the two sentences. Verified by sort: one rule under compute per phrase, one under RTF, none under word error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
All six were det-ff and are ar-lm. The class came from the export rather than from anything here - the importer only ever renames stoch-ff to st-ff - so this is upstream's correction, and it is the right one. NVIDIA's own card describes a causal transformer decoder that "autoregressively predicts discrete audio codec tokens", refined over stacked frames and decoded by NanoCodec. The table was already arguing against itself, which is the part worth keeping. This page defines det-ff as "a single forward pass with a deterministic duration predictor, so timing is identical every run and it cannot hallucinate" - and the Magpie rows are averaged over three free-seed runs precisely because they do not behave that way, spanning 6.5-7.4% mean word error while a single variant swings by up to 2.4%. A deterministic model has nothing to average. The corrected note now says so where a reader can check it: autoregressive despite the size, the GGUF declares a causal decoder with a codebook refinement head, and syntheses vary run to run. That is evidence rather than a label. Class counts move from det-ff 14 / st-ff 5 / ar-lm 27 to 8 / 5 / 33, and the six change colour in the preference chart, where they were the only large cluster painted as feed-forward. Nothing else moved: forty-six configurations, forty-six clips, the 2.05x rule still drawn once under the default sort. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Peak RSS is the column most likely to decide whether an engine is usable at all, and until now it could only be sorted, not filtered. "What fits on the machine I have" was a question the table could answer only by reading down a column. The stops are spaced by ratio rather than evenly. RSS runs 246 MB to 8.6 GB, but twenty of the forty-six engines fit inside a gigabyte and only three sit above four, so an even track would spend most of its length separating the handful of largest models and almost none of it where the choosing happens. Twelve stops from 256 MB to no limit, and it starts at no limit so nothing is hidden until someone asks for less. A slider rather than a number field because the question is a budget being felt out against the count beside it, not a figure known in advance - the useful motion is dragging until the table stops containing what you wanted. The value is rendered as a size and repeated on aria-valuetext, since a range input otherwise announces "8", which says nothing; and the label has a minimum width so the filter row does not shuffle sideways while the handle moves. Verified against the distribution: 1 configuration under 256 MB, 5 under 512, 20 under 1 GB, 43 under 4 GB, 46 unfiltered. The cutoff rule behaves correctly underneath it too - it disappears once the filter leaves every remaining engine on one side of the real-time line, which is the contiguity check doing its job for a filter it knows nothing about. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Accent orange is what this page uses for a live state - the sounding play button, the open note, the engaged real-time filter - so a permanently filled orange track read as something being flagged rather than a budget being set. The slider now takes the table's own border colour, which is where a control that is part of the furniture belongs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Forty-seven configurations. FastPitch goes because NVIDIA's own Magpie supersedes it - same vendor, an autoregressive model with a published quantisation ladder against a 2023 checkpoint that only reached the table by speaking an AI disclosure the harness had to strip. It leaves via DROP_ROWS, so the measurement stays in the raw export; nothing in the prose referred to it, and its clip is removed automatically. Dots-TTS MeanFlow closes a gap a reader asked about publicly. It was measured but absent last cut, which made "you miss meanflow" true in the only sense that counts. It is the same 2B model the Dots-TTS row measures as SOAR, and the note says the like-for-like comparison is still being taken rather than implying one. The Fish Audio row needed its licence corrected before it could be published, and this is the one column where being wrong costs somebody something. The export prints Apache-2.0. The model card's frontmatter says `license: other` with `license_name: fish-audio-research-license`, and its body says research and non-commercial use is free while commercial use needs a separate agreement. Apache-2.0 says the opposite, so the row now reads "Fish Audio Research (non-commercial)", pinned so the next export cannot revert it, and the proof link points at the card that says so. Its note also claimed "the largest weights in this table at 6.0 GB and second slowest at 10.2x". Tada 3B carries 6.4 GB against this row's 5.9, and Dots-TTS at 14.22x and Zonos at 11.07x are both slower, so it is third. Both rankings are dropped for the figures themselves, which a reader can check against the columns beside them. Fish Audio joins the vendor chips, found by re-running the cross-check of every row's MODEL_URL against the listed organisations. Counts follow: thirty-two of forty-seven lack a medium.en score, eighteen sit above 2.05x. The real-time rule still resolves to one crossing, and the boundary is unchanged at Chatterbox Q4 and Tada 3B. The preview card is re-cut because FastPitch was rated, at 17.2 over three ballots, so it was a plotted point and not only a row. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Forty-eight configurations. Both corrections made here last import came back fixed upstream, so the local override goes: keeping it would have replaced a correct value with a paraphrase of itself, which is not what CELL_OVERRIDES is for. The licence now reads "Fish research (non-commercial)" in upstream's own words rather than mine, and the note is better than the one I wrote, because it carries the restriction into the prose where a reader meets it: "Research licence: commercial use needs a separate agreement with Fish Audio." The overrides that remain are the ones upstream keeps reverting - omnivoice for the fifth cut running, bananamind-tts, supertonic, dots-tts, both Tadas. Those are guards. A guard against something already fixed is just a fork. The new row is a control, and it is the most interesting thing in this export. dots-soar-acpp is the same SOAR checkpoint as the dots-tts row, at q8_0 through audio.cpp instead of f16 through crispasr, which isolates runtime and precision from the MeanFlow distillation. Measured against it the distillation is worth 1.16x rather than the 3.3x it appeared to be worth against the f16 crispasr row. A reader asked publicly whether this table missed MeanFlow; it now carries the distillation, its control, and the number that separates them. chatterbox also drops a superlative of its own accord - "the family's best PESQ at 3.61" becomes "PESQ 3.64 - but turbo beats it on every axis", which is both a corrected value and a claim that cannot go stale. Counts follow: thirty-three of forty-eight lack a medium.en score, nineteen sit above 2.05x. The boundary is unchanged at Chatterbox Q4 and Tada 3B, still one crossing, and the preview card is re-cut. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Every autoregressive configuration is now averaged over at least three passes, and the timings moved a long way: thirty-five real-time factors and thirty-four per-phrase figures changed, some by more than half. Magpie f16 went from 6.98s a phrase to 2.71s, dots-soar-acpp from 18.59s to 9.32s. That retires the reason the cutoff was not round. It sat at 2.05x because Magpie q4_k and q6_k measured 2.01x and 2.02x, and splitting a ladder the table calls indistinguishable is not a distinction the data can carry. Those builds now measure between 1.53x and 1.81x, so nothing is inside the gap and the boundary is Chatterbox Q4 at 1.99x against Tada 3B at 2.21x - a wider margin than the one 2.05 was invented to buy. The boundary rule no longer appears under the default sort, at any cutoff. I scanned 1.8 to 2.5 and every value leaves three crossings, because real-time factor is compute per second of audio while the sort is compute per phrase, and audio durations now vary enough that magpie-f32 and chatterbox each sit on the wrong side of their neighbours. The contiguity check does the right thing and draws nothing; the article says so rather than leaving a reader wondering where the line went. The passes paragraph now explains itself. Sampling is the reason the two classes differ - a deterministic forward pass gives the same audio twice, so a second run measures nothing - and Chatterbox Turbo is the illustration: one pass put it joint-best on word error at 4%, three put it at 5% and behind three others. A handful of configurations that still moved got four passes or five. The corrected chatterbox note is in too, checked figure by figure. Turbo no longer "beats it on every axis": it is ahead on word error and 4.3x quicker, and behind on PESQ at 3.30 against 3.61, which the note now calls quality against speed rather than a clean win. Counts: thirty-three of forty-eight lack a medium.en score, seventeen sit above 2x. Also caught the filter description still claiming 2.05x while the control beside it said 2x. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Two limits now: 2x real time, and 6.5 seconds of compute a phrase. The rule uses whichever matches the active sort. This fixes the boundary properly rather than by choosing a luckier number. Sorting by one measure and cutting on another can interleave, and after the three-pass rerun it always did - every cutoff from 1.8 to 2.5 left three crossings under the default sort, because real-time factor is compute per second of audio and engines differ in how long they speak. Sorting by a measure and cutting on that same measure can only cross once, so the rule is now drawn under both speed columns instead of neither. 6.5s is not an invented threshold. It sits in a 0.76s gap, between Chatterbox Q4 at 6.27s and Q8 at 7.03s, and classifies the table identically to 2x but for one row: Magpie's full-precision build, quick per phrase at 3.79s and slow per second at 2.31x. A reader who clicks a different header is shown the same population, not a different one. The label follows the measure - "over 6.5s of compute a phrase" against "slower than real time, over 2x" - because a line cut on seconds should not be explained in multiples of real time. Both limits live in the run data beside each other, since where a line sits is an editorial choice rather than a rendering detail. The article said sorting by compute per phrase does not draw the line, which was true for exactly one commit. It now says the line is cut on whichever column you sorted by, and gives both boundaries. Verified by sort: one rule under compute per phrase between Chatterbox Q4 and Q8, one under RTF between Chatterbox Q4 and Tada 3B, none under word error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
New row from the n=3 export: LFM2 backbone over NVIDIA's NeMo nano codec, 370M params, 10% WER at 3.86 PESQ, 1.50x RTF / 6.86s a phrase. Purely additive - zero drift across the 48 pre-existing rows. Licence rewrite generalised from a per-model rule to a pattern, so "LFM1.0, under $10M" and "NeuTTS, under $5M" both shorten the same way. Verified NeuTTS renders identically to before. The proof link is the model card, which declares `license: lfm1.0`; the $10M ceiling is a term of that licence rather than of the card, same as NeuTTS's $5M. Kani makes the two speed measures disagree in a second, opposite direction: fast per second of audio at 1.50x but slow per phrase at 6.86s, where Magpie F32 is 3.79s and 2.31x. Methodology updated - that pair is now the argument for cutting on the sorted column, since no single ordering puts both on the right side of one cut. Prose counts refreshed against the new row set: medium.en now misses thirty-four of forty-nine, and the synth boundary reads 6.27s against 6.86s. The seventeen above real time is unchanged. Verified by render: divider falls directly above Kani under the compute sort, vendor chip and licence cell correct. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The real-time line is drawn on whichever speed column you sorted by, so it needs a limit in each measure. Moving the per-phrase limit from 6.5s to 7s buys agreement between the two: at 6.5s both Magpie F32 and Kani sat on the wrong side of one of them, and at 7s only Magpie does, which is what the number is chosen for. The cost is a tight boundary, and it is stated rather than glossed. 7s falls in a 0.17s gap, Kani at 6.86s against Chatterbox Q8 at 7.03s, where 6.5s had 0.59s of clear air. That is well inside what three passes resolve, so the methodology now says those two share a position instead of claiming nothing sits awkwardly against the line - the same standard applied to Magpie's 2.01/2.02/2.05 ladder. 7.5s is not a way out: it pulls Q8 inside at 2.21x and buys a fresh disagreement for the one it removes. Counts move with it: 33 within the per-phrase line and 16 above, while the seventeen above 2x real time are unchanged, that line not having moved. Vendor chips: adding nineninesix took the set to 27, which spilled 8+9+9+1 and stranded Zyphra alone on a fourth row. Horizontal padding 14px -> 13px gives an even 9+9+9 with 82px spare. The label carries the width, so the font stays at 0.85rem and the logos are untouched. Measured in-page across a grid of font/padding/gap values, not guessed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
"Avg synth" named the operation rather than the quantity, and spent ten characters doing it in a column four wide. It is a mean of a time, so it is now written as one: x̄ t, with the t italic as a variable. Renamed at the importer so the data, both scatter charts, the OG card and the prose all read from one name. The logo column sits between the engine name and Human, with no title and no sort, because it carries no value to order by. It answers "whose model is this" per row - which the chip list above the table could not, since nothing there says which chip belongs to which row. Each logo is resolved from the model link the row already carries rather than from a second config-to-vendor table, so it cannot drift from the name beside it. Three orgs need an alias (Qwen ships from three accounts, and the Magpie rows point at model cards under `nvidia` rather than the NeMo repo); that is done by explicit alias rather than by loosening the URL match, since a prefix rule would eventually put the wrong logo on a row, which is worse than none. Verified both directions: all 49 rows resolve, all 27 vendors are used by at least one row, every icon file exists, and in-page all 49 images decode with 27 distinct vendors and no page errors. The vendor list moved to ttsVendors, shared by the chips and the table, so a vendor cannot be added to one and forgotten in the other - which is how the chips fell out of step with the rows before. The header is blank to the eye but announces "Vendor" to a screen reader; a bare th would leave the column's cells unlabelled in table mode. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The card in the repo was built on 27 August and had not moved since, because the scores API was unreachable from the machine doing the importing and the local database has no ballots at all - building from that would have emptied the chart rather than refreshed it. Prod is reachable again, so it is rebuilt from real votes: 37 engines with at least three ballots, r = 0.10. The headline claim is unchanged, and still holds on the larger sample - compute barely predicts preference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.