Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
8b1950a
Add a TTS benchmark template, and say on the page that it can be run
robdmac Aug 21, 2026
5f3808c
Add a social preview card for the TTS benchmark
robdmac Aug 21, 2026
7e694c7
Import the export that adds Orpheus 3B Q8
robdmac Aug 21, 2026
5d173a0
Take the corrected notes from the re-cut export
robdmac Aug 21, 2026
3d796d4
Make the TTS benchmark template actually run, and prove it
robdmac Aug 21, 2026
d6dbf80
Drop the sample-directory workaround, now that upstream resolves it
robdmac Aug 21, 2026
e1b86c7
Bias ballot selection toward the clips ranked least
robdmac Aug 21, 2026
00b337a
Colour word error on what the cell shows, and stop green at 6%
robdmac Aug 21, 2026
087dce7
Run the picker instead of forcing piper, and stop chat duplicating th…
robdmac Aug 21, 2026
3fd774f
Drop the listening clips, and declare the report's encoding
robdmac Aug 21, 2026
429f1e0
Take the upstream charset fix; keep the guard dormant
robdmac Aug 21, 2026
5850500
Frame every block once when a dashboard's items first render
robdmac Aug 22, 2026
2ba073c
Ship no setupGuide, so chat stays minimised and silent
robdmac Aug 22, 2026
a2cfcfa
Tighten the run-it-yourself note, and redate the article
robdmac Aug 22, 2026
982f579
Add a Cloudflare Web Analytics beacon for readers who never sign in
robdmac Aug 22, 2026
291ad05
Put the preference chart on a log x-axis, like the chart above it
robdmac Aug 23, 2026
6eca9b0
Import the 43-row export: VoxCPM, IndexTTS and Pocket TTS
robdmac Aug 23, 2026
e5e208c
Take the corrected VoxCPM row, and give it back its clip
robdmac Aug 23, 2026
0e8f243
Add the four vendors whose models are in the table but had no chip
robdmac Aug 23, 2026
4584259
Size the vendor chips for three rows instead of squeezing them into two
robdmac Aug 23, 2026
af838f9
Import the six Magpie quantisations, and fix what they falsify
robdmac Aug 24, 2026
8f80847
Draw the real-time rule once, or not at all
robdmac Aug 24, 2026
99a6d11
Move the real-time line to 2.05x, which puts the rule back on the def…
robdmac Aug 24, 2026
76078f3
Take the corrected Magpie class: autoregressive, not feed-forward
robdmac Aug 24, 2026
74a1079
Add a memory budget slider beside the class filters
robdmac Aug 24, 2026
6b93720
Colour the memory slider like the table, not like the accent
robdmac Aug 24, 2026
66d676c
Add Dots-TTS MeanFlow and Fish Audio S2 Pro, drop FastPitch
robdmac Aug 24, 2026
eb6b60c
Take the audited export, and drop the override it made redundant
robdmac Aug 24, 2026
4079566
Take the three-pass rerun, and put the real-time line back on a round 2
robdmac Aug 27, 2026
638d900
Cut the real-time line on whichever column the table is sorted by
robdmac Aug 27, 2026
a6f2304
Add Kani TTS to the open-weight TTS benchmark
robdmac Sep 6, 2026
08bc4e3
Move the compute-per-phrase cutoff to 7s, fit vendor chips in three rows
robdmac Sep 6, 2026
542cad9
Rename the compute column to x̄ t, add a vendor logo column
robdmac Sep 6, 2026
b2a8f92
Rebuild the TTS preview card against live scores
robdmac Sep 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 30 additions & 5 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,21 @@ results surface in a browser/note block.
| Benchmark | Dir | Runnable? | Execution model |
|-----------|-----|-----------|-----------------|
| slop-code-bench | [`slopcodebench/`](slopcodebench/) | ✅ yes | **Host** (no Docker) via the fork's tmux executor |
| Open-weight TTS | [`talkito-tts/`](talkito-tts/) | ✅ yes | **Host**, one PTY; no agent, no keys; reader picks engines interactively |
| SWE-bench (Lite/Verified/Full) | [`swebench/`](swebench/) | ✅ yes | Inference on host + **Docker-per-task** eval (Modal or DinD) |
| SWE-bench Pro (public 731) | [`swebench-pro/`](swebench-pro/) | ✅ yes | Inference on host + **Docker-per-task** eval (`--use_local_docker` or Modal) |
| Terminal-Bench 2.1 | [`terminal-bench/`](terminal-bench/) | ✅ yes | **Harbor** harness; Docker-per-task (or `--env daytona`/Modal remote) |
| FrontierCode (Diamond) | [`frontiercode/`](frontiercode/) | ❌ no | **Closed / submission-only** — no public harness, see the note |

## Two architecture families

These describe the *coding* benchmarks, where something has to run an agent under
test. The TTS benchmark belongs to neither: it measures speech engines rather than
agents, so it has no agent-under-test, needs no provider key, and its entire run is
one terminal's boot command with a browser block reading the report. It is the only
template here that skips the config wizard and the orchestrator entirely — see
[`talkito-tts/`](talkito-tts/).

**Family A — host-executable.** The agent-under-test runs directly in the
Orcabot sandbox VM; no inner container. The sandbox VM *is* the isolation
boundary. This is slop-code-bench (via the fork's host-tmux executor), where
Expand Down Expand Up @@ -66,14 +74,31 @@ benchmarks/bin/seed-benchmark-templates swebench # just one, by dir name
It upserts each `benchmarks/*/template.json` (delete-by-name then insert,
`status='approved'`) against the desktop d1-shim (`D1_URL`, default
`http://127.0.0.1:9001`). `slopcodebench/bin/scb-insert-template` is a thin
wrapper that seeds only the SlopCodeBench template. For a **remote/cloud** control
plane, export from an existing dashboard via the templates API (`POST /templates`)
instead — direct INSERT is the local/curated path.
wrapper that seeds only the SlopCodeBench template.

For a **remote/cloud** control plane there is no code seed at all:
`templates.seedStarterTemplates` returns early unless `SURFACE_TOKEN` is set, and only
the desktop app sets it, so `starter-templates.json` never reaches production. Every
production row is a direct insert. Use the script's `--sql` mode:

```
AUTHOR_ID=<a real production users.id> AUTHOR_NAME=Orcabot \
benchmarks/bin/seed-benchmark-templates --sql talkito-tts > /tmp/seed.sql
cd controlplane && npx wrangler d1 execute orcabot-db-production --remote --file /tmp/seed.sql
```

`AUTHOR_ID` is not optional. It defaults to `dev-desktop`, which exists on the desktop
shim but not in a cloud or wrangler-dev database, and `dashboard_templates.author_id`
is a foreign key into `users` — the insert then fails with `FOREIGN KEY constraint
failed`, quietly enough to look like success. Grep the output for `ERROR` rather than
trusting a clean exit.

## Status

All templates here are **prototypes**. slop-code-bench is validated end-to-end
(see its README). The Family-B templates are grounded in each harness's real
All templates here are **prototypes**. slop-code-bench and the open-weight TTS
benchmark are validated end-to-end (see their READMEs); the TTS one is the
cheapest to re-verify, since a piper run is about three and a half minutes and
needs no key. The Family-B templates are grounded in each harness's real
commands but are **not** validated end-to-end from here, because they need a
Docker-capable backend (Modal token or DinD sandbox) + provider keys that aren't
available in this environment. Each dir's README states exactly what's verified
Expand Down
132 changes: 132 additions & 0 deletions benchmarks/talkito-tts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# talkito TTS benchmark

Runs the open-weight TTS benchmark from
[talkito-benchmark](https://github.com/robdmac/talkito-benchmark) inside one Orcabot
terminal and shows the report it produces in a browser block beside it. This is the
runnable version of the published
[Testing open-weight Text To Speech models](https://orcabot.com/benchmarks/open-weight-tts)
table: same harness, same 84-phrase corpus, same page.

The reader chooses which engines to measure. The terminal opens the benchmark's own
checklist — arrows move, space toggles, `n` clears, Enter runs — and everything after
that is automatic.

## Why this one has no orchestrator

Every other template here drives a benchmark through an agent, because those
benchmarks need an agent-under-test, provider keys and per-task containers. This one
needs none of that. The benchmark is a Python program that synthesizes a fixed corpus
and scores it, so the whole run fits in a single boot command and the only moving
parts are a terminal and a browser block.

That also makes it the cheapest template here to run: no LLM calls, no API key, no
tokens. It costs CPU time and nothing else.

**It ships no `setupGuide`, deliberately.** That field is not documentation — it is a
walkthrough brief for Orcabot chat, and its presence has two effects. The dashboards
page sets `orcabot_initial_prompt` to "Walk me through setting up this dashboard.",
which `ChatPanel` auto-sends on mount with the panel expanded. And the control plane
injects the guide into *every* chat message on that dashboard, under a system prompt
reading "Use your tools to do the work (… `create_terminal`/`create_browser` as
needed)".

An earlier version carried a guide that described the dashboard, and chat did as the
wrapper told it: two terminals and two browsers, with a second benchmark writing the
same measurement files as the first. Replacing the text with "do not create anything"
only argued with the instruction wrapped around it. Removing the field removes the
auto-prompt, the walkthrough system prompt, and the reason chat was invoked at all —
so it stays minimised and silent, which is right for a dashboard with no keys, no
provider choice, no config, and a terminal that starts itself.

## Architecture

terminal block ──runs──> tools/benchmark.py (interactive picker)
│ │
│ └──> tts-benchmark.html
└──serves──> python3 -m http.server 8099 <──reads──┘
▲
browser block ──opens──> http://localhost:8099/

The terminal starts the web server *before* it starts cloning, serving a placeholder
that refreshes every ten seconds. Without that, the browser block would show a
connection error for the whole run and only become useful at the very end. When the
report is built it is copied over the placeholder and the next refresh picks it up.

## What the sandbox forces

All of these were found by running it, not by reading it, and two of them fail
*quietly* — the run still exits 0 and still publishes a page.

**HTTPS clones, not submodules.** talkito's `.gitmodules` points at
`git@github.com:robdmac/...`, and the sandbox has no SSH key, so
`git clone --recurse-submodules` cannot work. The boot command clones the two public
repos over HTTPS instead and arranges them as

/workspace/talkito/ talkito
/workspace/talkito/benchmark/ talkito-benchmark

The harness ships inside talkito-benchmark, so `tests-repo` is not needed.
`tools/paths.py` still wants a talkito **checkout** rather than the installed package
— it looks for `talkito/tts.py` — and `benchmark/` inside `talkito/` is the layout it
resolves via `../..`.

**`TALKITO_AUTO_APPROVE_DOWNLOADS=1`.** talkito refuses to fetch model weights when it
cannot ask a human. The picker is interactive, so it *could* ask — but the prompt would
land in the middle of a sweep the reader has walked away from, so it is answered in
advance. Without it every phrase fails with `Piper model unavailable` and the sweep
still reports `rc=0`, having published a table reading `0/84 phrases, WER 0.0%,
RTF 0.00x`. A green run measuring nothing is the worst failure available here.

**No listening clips, and therefore no ffmpeg.** `make_samples.py` exists to embed one
playable clip per engine, and it shells out to ffmpeg for the wav-to-mp3 step. Audio
does not reach the reader through the browser block, so the clips are not synthesized
at all — which removes ffmpeg, which removes the only `apt-get` this template ran,
which removes an egress approval prompt for `deb.nodesource.com`. The sandbox image
installs Node from NodeSource at build time and leaves the apt source configured, so
*any* `apt-get update` inside the sandbox reaches for it. The table is numbers.

**The published page's encoding is declared upstream now.** It was not: the report
carried no `<meta charset>` and `python3 -m http.server` sends `text/html` with no
charset either, so a browser fell back to its locale default and read valid UTF-8 as
cp1252 — `TTS Engine Benchmark — 24 configurations` arriving with garbage where the em
dash should be. `tools/page_template.html` declares it now, so the boot command simply
copies. It keeps a fallback that prepends the tag when a page lacks one, because the
failure is silent and total, and it writes to the published copy rather than the source
so the export zip stays byte-identical to what the tools wrote.

## Choosing engines

Everything is ticked when the picker opens, and a **full sweep is around eleven
hours** — longer on a first run, because every engine downloads its own weights. Press
`n` to clear, then space to pick a few.

piper is the cheapest complete arm in the table: no torch at all, and scoring runs
through faster-whisper on CPU via CTranslate2, so the install is seconds rather than
gigabytes and the run is about three and a half minutes. Most other engines pull torch.

The picker shows what each engine cost on previous runs, and `?` where it has never
been measured.

## Status

**Validated end to end in the Docker sandbox**, from an empty workspace: clone,
dependency install, weight download, 84-phrase sweep, Whisper scoring, page build and
serving. A piper run finishes

piper 82/84 phrases WER 5.3% RTF 0.08x

against 4% and 0.04x for the published Apple M2 figures — close, and real-time factor
is wall-clock so a throttled container is expected to read higher.

The picker was driven under a real PTY to confirm it renders and takes input the way a
terminal block presents it, and the quit-without-running path was tested separately.
The report reaches the browser block as a table of numbers with its encoding declared;
nothing is playable, by design.

## Seeding

benchmarks/bin/seed-benchmark-templates talkito-tts

A cloud control plane needs a direct insert instead: `seedStarterTemplates` returns
early unless `SURFACE_TOKEN` is set, and only the desktop app sets it, so the code seed
never runs in production. See [`../README.md`](../README.md).
40 changes: 40 additions & 0 deletions benchmarks/talkito-tts/template.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"_comment": "Importable Orcabot dashboard template. Unlike the other benchmark templates this one needs no orchestrator, no agent and no API key: the terminal's boot command runs the whole benchmark and serves its report on localhost, and the browser block is pointed at that server. Seed with bin/seed-benchmark-templates.",
"name": "TTS Benchmark Runner",
"description": "Run the open-weight TTS benchmark and read its report",
"category": "coding",
"items": [
{
"placeholderId": "runner",
"type": "terminal",
"content": "{\"name\": \"TTS benchmark\", \"agentic\": false, \"bootCommand\": \"PORT=8099\\n# talkito refuses to fetch model weights when it cannot ask a human. The picker\\n# below is interactive, but the download prompt would land in the middle of a\\n# sweep the reader has walked away from, so it is answered in advance.\\nexport TALKITO_AUTO_APPROVE_DOWNLOADS=1\\nROOT=/workspace/talkito\\nBENCH=$ROOT/benchmark\\nSERVE=/workspace/tts-report\\necho \\\"== Orcabot: talkito TTS benchmark ==\\\"\\n\\n# Serve a placeholder first, so the browser block has something at a stable URL from\\n# the first second rather than an error until the run finishes.\\nmkdir -p \\\"$SERVE\\\"\\n[ -f \\\"$SERVE/index.html\\\" ] || printf '%s' '<!doctype html><meta http-equiv=\\\"refresh\\\" content=\\\"10\\\"><title>TTS benchmark</title><body style=\\\"font:15px/1.6 system-ui;padding:2rem;max-width:34rem\\\"><h2>Waiting for a run...</h2><p>Pick engines in the terminal. This page reloads every 10 seconds and the report replaces it when the run finishes.</p></body>' > \\\"$SERVE/index.html\\\"\\ncurl -sf -o /dev/null --max-time 2 \\\"http://127.0.0.1:$PORT/\\\" || ( cd \\\"$SERVE\\\" && nohup python3 -m http.server \\\"$PORT\\\" >/workspace/.tts-serve.log 2>&1 & )\\n\\n# talkito's .gitmodules uses git@github.com: URLs and the sandbox has no SSH key,\\n# so --recurse-submodules fails; both repos are cloned over HTTPS instead.\\n# The harness ships inside talkito-benchmark, so tests-repo is not needed -- but\\n# tools/paths.py still wants a talkito CHECKOUT (it looks for talkito/tts.py, not\\n# the installed package), and benchmark/ inside talkito/ is the layout it resolves.\\n[ -d \\\"$ROOT/.git\\\" ] || git clone --depth 1 https://github.com/robdmac/talkito.git \\\"$ROOT\\\"\\n[ -d \\\"$BENCH/.git\\\" ] || git clone --depth 1 https://github.com/robdmac/talkito-benchmark.git \\\"$BENCH\\\"\\n\\n# piper needs no torch at all, and faster-whisper scores on CPU through CTranslate2,\\n# so the cheapest arm installs in seconds. Heavier engines pull their own weights on\\n# first use and the picker warns what each has cost before.\\npython3 -c 'import piper' 2>/dev/null || pip3 install --break-system-packages -q piper-tts\\npython3 -c 'import faster_whisper' 2>/dev/null || pip3 install --break-system-packages -q faster-whisper\\n\\ncd \\\"$BENCH\\\" || exec bash\\ncat <<'BANNER'\\n\\n Everything is selected by default and a full sweep is around eleven hours.\\n Press n to clear, then space to pick a few. piper alone takes about three minutes.\\n\\nBANNER\\n# No --only: this is the interactive picker, and a terminal block is a real PTY, so\\n# the arrow-key menu works exactly as it does on a laptop.\\npython3 tools/benchmark.py\\nrc=$?\\n\\n# No listening clips: make_samples.py exists to embed one per engine, but audio does\\n# not reach the reader through the browser block, and it is the only reason this\\n# needed ffmpeg -- which in turn was the only reason the sandbox ran apt-get, which\\n# reaches deb.nodesource.com through a leftover apt source and trips egress approval.\\n# Dropping playback drops all three.\\n\\nif [ -f tts-benchmark.html ]; then\\n # page_template.html declares utf-8 upstream, so this normally just copies. The\\n # fallback stays because the failure it guards is silent and total: python3 -m\\n # http.server sends \\\"text/html\\\" with no charset, and a page that also declares\\n # none is read as the browser's locale default, turning every em dash into\\n # mojibake. It writes to the published copy rather than the source, so the export\\n # zip stays byte-identical to what the tools wrote, and never doubles the tag.\\n if grep -qi '<meta[^>]*charset' tts-benchmark.html; then\\n cp -f tts-benchmark.html \\\"$SERVE/index.html\\\"\\n else\\n { printf '%s\\\\n' '<meta charset=\\\"utf-8\\\">'; cat tts-benchmark.html; } > \\\"$SERVE/index.html\\\"\\n fi\\n echo \\\"report published -> http://localhost:$PORT/\\\"\\nelse\\n echo \\\"no tts-benchmark.html was produced; measurements (if any) are under $BENCH/tools/\\\"\\nfi\\necho \\\"[tts benchmark done rc=$rc]\\\"\\nexec bash\", \"subagentIds\": [], \"skillIds\": [], \"mcpToolIds\": []}",
"position": {
"x": 0,
"y": 0
},
"size": {
"width": 560,
"height": 560
}
},
{
"placeholderId": "report",
"type": "browser",
"content": "http://localhost:8099/",
"position": {
"x": 600,
"y": 0
},
"size": {
"width": 900,
"height": 620
}
}
],
"edges": [],
"viewport": {
"x": 0,
"y": 0,
"zoom": 1
}
}
Loading
Loading