Test framework, stage 4: the benchmarks in Catch2 by RyeMutt · Pull Request #478 · AlchemyViewer/Alchemy

RyeMutt · 2026-10-14T10:25:02Z

Description

Stage 4 of the test framework's move from TUT to Catch2 (TM25): the six hand-rolled benchmark executables become Catch2 benchmark files, each built into the test binary whose link set it matches. Six executables go.

Benchmark	Binary	Benchmarks
`alscript_bench`	`alscript_tests`	85
`alregex_bench`	`llcommon_tests`	25, for the old 34 cells (repeated Boost cells measured once, the worked-out speedup column dropped)
`llsimd_bench`	`llmath_tests`	81 on NEON; x86 builds keep their extra rows
`altextdiff_bench`, `altextengine_bench`	`llui_tests`	48 and 108
`altextdraw_bench`	`llui_gl_tests`	12

It stacks on #475 (rye/tm24-test-fixes).

The harness (f400470)

BENCH_SOURCES: al_add_test_binary builds benchmark files into the binary beside its tests. Their cases are tagged [!benchmark], which Catch2 hides, so the test files' entries, the combined run and per-case entries all leave them out.
One entry per benchmark file: bench/<binary>/<stem>, labelled benchmark and manual as the old executables were. It runs alone and reports to the console and to Testing/benchmarks/<binary>_<stem>.xml; Catch2's JSON reporter drops benchmarks. BUILD_AND_RUN_BENCHMARKS still runs ctest -L benchmark -V.
Optimised builds only: every benchmark case starts with ll_test::skipUnlessOptimised() (test/altestbenchmark.h). The code now compiles in every configuration, and the entry reads as skipped outside an optimised build, as the old exit 125 did.
Self-tests: two entries check the filters from outside. A benchmark file's cases are hidden from its tests' filter, and run by its benchmark entry's filter.

The conversions (509176f..fc5bdf2), by hand

Names: each timed cell of an old table is one BENCHMARK, named for its row and column. A case is a section of the old table.
Per operation: where the old number divided a pass by the operations in it, such as a keystroke typed and taken back, each call is now one operation. Where it was per element over a pass, a call is the pass, named for what it covers.
Checks: the lines that said what is right ("errors found (0 is right)") are now CHECKs. Counts that are neither times nor right or wrong are printed, as before.
No more sinks: the volatile sinks go; each body returns what it made.
llsimd keeps its codegen: each call runs over local views of arrays made once a run. A kernel run over the arrays in a static reloaded their bounds for every element, since any store might have moved them, and was 40% slower over the cached arrays for it.
The editor benches start fresh: each case makes its own editors and lets them go, closing the find's worker and putting focus back, so the state guard finds the process as it was.

al_add_bench goes (853378b).

Related Issues

Please link to a relevant GitHub issue for additional context.

Issue Link: none. This follows #471, #472, #474 and #475.

Checklist

I have provided a clear title and detailed description for this pull request.
If useful, I have included media such as screenshots and video to show off my changes.
I have tested the changes locally and verified they work as intended.
All new and existing tests pass.
Code follows the project's style guidelines.
Documentation has been updated if needed.
Any dependent changes have been merged and published in downstream modules.
I have reviewed the contributing guidelines.

Additional Notes

Tested on macOS arm64, RelWithDebInfo:
- the full build is clean;
- all 477 tests of ctest -LE manual pass, and so do all 19 pending combined runs;
- all seven benchmark entries pass, with no state guard report.
Before and after, on the same machine: each of the 359 benchmarks was compared with three runs of the old executable, and with one run back to back with the new. The old median was scaled to what a new call covers, and the old helpers' clock read taken off (about 15 ns a pass, read inside their timed loop). The new mean and its 95% interval agree, except where the old number is explained:
- Small memcpys (1 KB and under): the old figures were mostly that clock read.
- altextengine's "an edit with every match lit" and "a squiggle on every other line":
  - The old figures: 125 µs an edit, against 2 µs here.
  - Why: the old rows followed Replace All undone, which leaves about 20,000 lines dirty in the highlighter. Every edit walks that list until idle time lexes them, and a bench has no idle time. That is reproduced at 89 µs an edit.
  - In the viewer: idle time lexes them within a few frames.
- The history rows: the old typing run grew on every pass. Each typed character is now a step of its own.
- altextdraw's first frame: the old run measured it while its caches filled. It came out slower than the same frame with squiggles drawn.
- Left: large memcpys and some of llsimd's tight loops move by 5–15% either way. Each kernel's output still feeds the checksum it returns.
- Single rows: a few moved between two runs of the new, such as the new solver over a fragment, and agreed with the old in the other.
Runtime: at Catch2's 100 samples a benchmark, the six take 11.5 minutes, where the old took 100 seconds; alscript takes 6 of them. They stay out of the default run.
Not done:
- CI uploads no benchmark XML, since it never runs the manual label.
- llkdu's proprietary llimagej2ckdu_test is still TUT, and must be converted before TUT goes (TM26).
Merge order: this should follow Test framework, stage 3: llui's tests in Catch2 #471, Test framework: fix the old test bugs in llui's tests #472, Test framework, stage 3: newview's tests in Catch2 #474 and Test framework: fix the old test bugs in newview's tests #475. Once they merge, its base becomes develop.

🤖 Generated with Claude Code

coderabbitai · 2026-10-14T10:25:14Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 48 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI
Review profile: CHILL
Plan: Advanced
Run ID: 1f945ca9-ccd5-4e66-8172-b34b0f261560

📥 Commits

Reviewing files that changed from the base of the PR and between 67e4e77 and 64ed06d.

📒 Files selected for processing: 15

indra/alscript/CMakeLists.txt
indra/alscript/tests/alscript_bench.cpp
indra/cmake/ALAddTestBinary.cmake
indra/cmake/LLAddBuildTest.cmake
indra/llcommon/CMakeLists.txt
indra/llcommon/tests/alregex_bench.cpp
indra/llmath/CMakeLists.txt
indra/llmath/tests/llsimd_bench.cpp
indra/llui/CMakeLists.txt
indra/llui/tests/altextdiff_bench.cpp
indra/llui/tests/altextdraw_bench.cpp
indra/llui/tests/altextengine_bench.cpp
indra/test/CMakeLists.txt
indra/test/altestbenchmark.h
indra/test/tests/altestsupport_bench.cpp

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI
Review profile: CHILL
Plan: Advanced
Run ID: efe50f7b-a912-4491-9cfb-072765803dd6

📥 Commits

Reviewing files that changed from the base of the PR and between 16b2b31 and 67e4e77.

📒 Files selected for processing: 15

indra/alscript/CMakeLists.txt
indra/alscript/tests/alscript_bench.cpp
indra/cmake/ALAddTestBinary.cmake
indra/cmake/LLAddBuildTest.cmake
indra/llcommon/CMakeLists.txt
indra/llcommon/tests/alregex_bench.cpp
indra/llmath/CMakeLists.txt
indra/llmath/tests/llsimd_bench.cpp
indra/llui/CMakeLists.txt
indra/llui/tests/altextdiff_bench.cpp
indra/llui/tests/altextdraw_bench.cpp
indra/llui/tests/altextengine_bench.cpp
indra/test/CMakeLists.txt
indra/test/altestbenchmark.h
indra/test/tests/altestsupport_bench.cpp

💤 Files with no reviewable changes: 1

indra/cmake/LLAddBuildTest.cmake

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Summary

Summary by CodeRabbit

Testing
- Integrated benchmarks into the existing test suite, with separate benchmark runs and reports.
- Converted benchmarks across scripting, math, UI, text processing, and test support to structured benchmark cases.
- Benchmark cases requiring optimized builds or graphics support are skipped when those requirements aren’t met.

The change integrates benchmark sources into Catch2 test binaries and replaces standalone timing runners with Catch2 benchmark cases across scripting, regex, SIMD, and UI benchmark suites.

Changes: Catch2 Benchmark Integration

Layer / File(s)	Summary
Benchmark test harness `indra/cmake/ALAddTestBinary.cmake`, `indra/cmake/LLAddBuildTest.cmake`, `indra/test/*`	The test-binary helper now compiles benchmark sources and registers per-file benchmark CTest entries. The standalone benchmark helper is removed. Test support adds `skipUnlessOptimised()` and a small benchmark case with CTest coverage.
Scripting benchmark conversion `indra/alscript/CMakeLists.txt`, `indra/alscript/tests/alscript_bench.cpp`	The alscript benchmark joins the test binary and uses Catch2 test cases and benchmark blocks for analysis, typing, include, preprocessing, formatting, and message workloads.
Regex benchmark conversion `indra/llcommon/CMakeLists.txt`, `indra/llcommon/tests/alregex_bench.cpp`	The regex benchmark joins the llcommon test binary. URL, RLVa, lexer, chat, and date workloads use Catch2 benchmark bodies that return accumulated results.
SIMD benchmark conversion `indra/llmath/CMakeLists.txt`, `indra/llmath/tests/llsimd_bench.cpp`	The SIMD benchmark joins the llmath test binary. Catch2 cases cover memory, vector, transform, batch, and morph workloads; morph samples alternate weights.
UI diff and draw benchmarks `indra/llui/CMakeLists.txt`, `indra/llui/tests/altextdiff_bench.cpp`, `indra/llui/tests/altextdraw_bench.cpp`	The UI test binaries include text-diff and text-draw benchmarks. Their Catch2 cases use shared fixtures and scoped UI setup; draw measurements include editor drawing and flushing.
Text-engine benchmark conversion `indra/llui/CMakeLists.txt`, `indra/llui/tests/altextengine_bench.cpp`	The text-engine benchmark uses Catch2 cases for editor operations, with reusable fixtures and scoped cleanup for views and workers.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 67e4e

This change moves the standalone benchmark programs into the existing Catch2 test binaries as manual, opt-in benchmarks. Default test runs are unchanged, and no concrete defect remains, so the change is ready to merge.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 35.82% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 67 functions across 8 files. (6 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the stage of the test framework migration and the conversion of benchmarks to Catch2.
Description check		The description is detailed and covers the changes, testing results, related work, limitations, merge order, and checklist. The issue link field is not populated, but the description references the re…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 35.82% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 67 functions across 8 files. (6 skipped: 6 unsupported.)

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{A rabbit times a script at dawn,
Then hops through samples, one by one.
Regex, vectors, editors join,
Catch2 records each measured point.
The bunny files its chart and runs.}

_{Comment @coderabbitai help to get the list of available commands.}

TM25 moves the six benchmark executables into Catch2 (TMD16). This is the harness for it. - al_add_test_binary takes BENCH_SOURCES: benchmark files built into the binary beside its tests. Their cases are tagged [!benchmark], which Catch2 hides, so no test file's entry, the combined run or a per-case entry runs them. - Each benchmark file is an entry of its own, bench/<binary>/<stem>, running "[#<stem>][!benchmark]" with the console reporter and an XML one, Testing/benchmarks/<binary>_<stem>.xml (Catch2's JSON reporter drops benchmarks). It is labelled benchmark and manual, as the old executables were, so BUILD_AND_RUN_BENCHMARKS runs it and the default run does not. It runs alone, since anything beside it would move its numbers. - altestbenchmark.h gives a benchmark file Catch2's macros and ll_test::skipUnlessOptimised(), every benchmark case's first line: the code compiles in every configuration and runs only where the numbers mean something, as the old executables exited 125 elsewhere. - test_support_tests checks the filters from outside: a benchmark file's cases are hidden from the filter its tests' entries run, and its benchmark entry's filter runs them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BENCH_alscript becomes alscript_bench.cpp in alscript_tests: seven cases, one a section of the old table, and 85 benchmarks, one an old timed cell, each named for its row and column. The two typing cases choose their solver as the old loop did, not by the binary's rerun. - The lines that said what is right, such as "errors found (0 is right)" and "the questions below answer (3 is right)", are checks. - The script's bytes and the problems found are printed, as before. - The volatile sink and its checksum go: each body returns what it made. Checked against four runs of the old executable on this Mac: in a run of the new, every benchmark's mean agrees with the old median, inside its 95% interval or the old runs' own spread. Two of the new solver's rows over a fragment moved between two runs of the new, as the new solver's rows do in the old table. The run takes six minutes, at Catch2's 100 samples, where the old took 37 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BENCH_alregex becomes alregex_bench.cpp in llcommon_tests: seven cases, one a section of the old table, and 25 benchmarks for its 34 cells. A Boost number the old table printed beside several RE2 variants is measured once, and the speedup column, which was worked out, goes. Each benchmark is named for what it runs and what a call covers: ten lines of chat, 770 places in a script, 10,000 log lines. Checked against four runs of the old executable on this Mac: every benchmark's mean agrees with the old median, once the old helper's clock read is taken off, about 15 ns a pass, which a pass of one line or row carried. The run takes 23 seconds, where the old took 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BENCH_llsimd becomes llsimd_bench.cpp in llmath_tests: seven cases, one a section of the old table, and 81 benchmarks on NEON, one an old cell; x86 builds add the rows they always had. A call is a whole pass, named for how many elements it covers, where the old number was per element. The processor and ISA line is printed once a run. - The arrays are made once a run, where each row made its own, and each call runs over views of them it holds as locals. Run over the arrays in a static, a kernel reloaded their bounds for every element, since any store it made might have moved them, and the cached rows were 40% slower for it. The kernel is copied into the call for the same reason: the transforms' matrix is a local too. - The morph rows add their weight and take it away in turn, where the old ones added it on every pass without end. Checked against four runs of the old executable on this Mac. Most rows agree with the old median. The memcpy rows up to 1 KB came out at a fifth to a half of the old: the old helper read the clock after every pass, about 15 ns, and a pass was one copy. The large copies and a few kernels move by 5 to 15% either way between runs, the batch kernels' faster in every run, each with its output still feeding the checksum it returns. The run takes 76 seconds, where the old took 12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BENCH_altextdiff, BENCH_altextengine and BENCH_altextdraw become benchmark files: the first two in llui_tests, over the headless UI the binary makes, and the drawing in llui_gl_tests, on its shared context. Each case makes the editors it measures and lets them go, closing the find's worker and putting focus back, so the state guard finds the process as it was. - altextdiff: four cases and 48 benchmarks, one an old cell. The keystroke rows type a character and take it away in turn, a call a keystroke, as the old number was per keystroke. What each comparison says of some edits is printed, as before. - altextengine: eighteen cases and 108 benchmarks. A case starts from editors of its own where the old rows ran one after another over the same two. Rows that put the text back over several steps take a step a call. The history rows break the typing run before each character they type, so a sample's undos take back exactly what it typed. - altextdraw: three cases and 12 benchmarks, a frame each, the draw and the flush on the clock and the GPU waited for off it. The draw calls a frame makes are printed. Checked against four runs of the old executables on this Mac. All of altextdiff's and altextdraw's agree with the old medians but altextdraw's first frame, which the old run measured while its caches filled: it came out slower than the same frame with squiggles on. altextengine's agree but where the old rows ran over what earlier rows left: - An edit with every match lit, or a squiggle on every other line, cost 125 us an edit in the old run and 2 us here. The old rows came after Replace All undone, which left 20,000 lines dirty in the highlighter, and every edit walks that list until idle time lexes them, which a bench never gives. Fresh editors measure the layers. - The history's notation and LLSD rows typed their character into one run that grew on every pass; here each is a step of its own. - Two vim rows, which in the old run followed all of the above, are 7% and 36% faster. The three take 3.8 minutes, where the old took 50 seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Every benchmark is a file of a test binary now, so the function that built one into an executable of its own has nothing left to build. BUILD_BENCHMARKS builds the binaries with benchmark files, and BUILD_AND_RUN_BENCHMARKS runs ctest -L benchmark as it did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added c/cpp llui llcommon cmake llmath alscript labels Oct 14, 2026

RyeMutt added this pull request to stack #473 October 14, 2026 10:41

RyeMutt and others added 6 commits October 14, 2026 06:53

RyeMutt force-pushed the rye/tests-tm25 branch from 67e4e77 to 64ed06d Compare October 14, 2026 10:53

RyeMutt merged commit 860b500 into develop Oct 14, 2026
3 of 4 checks passed

RyeMutt deleted the rye/tests-tm25 branch October 14, 2026 10:53

RyeMutt mentioned this pull request Oct 14, 2026

Test framework, stage 5: TUT removed #479

Merged

5 of 9 tasks

Navigation Menu

Repository navigation

Test framework, stage 4: the benchmarks in Catch2 - #478

Test framework, stage 4: the benchmarks in Catch2#478
RyeMutt merged 6 commits into
rye/tm24-test-fixesfrom
rye/tests-tm25

RyeMutt commented Oct 14, 2026

Uh oh!

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading

Review limit reached

Summary by CodeRabbit

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

RyeMutt commented Oct 14, 2026

Description

Related Issues

Checklist

Additional Notes

Uh oh!

coderabbitai Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Summary by CodeRabbit

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading