bench: cross-version benchmarks for v0.4.0, v1.0.0 and v1.0.2

[davydog187]

Records the first cross-version benchmark campaign and establishes the convention for benchmarking every release going forward. Related to #267 (left open — the per-workload perf-parity work it tracks continues).

What's here

• bench_results/ — one directory per released version (v0.4.0/, v1.0.0/, v1.0.2/), each with raw Benchee stdout for all 11 workloads, a parsed summary.json, and an environment.md. The cross-version report is versions-2026-07-28.md; it supersedes the 1.0.0-era numbers in benchmarks/BASELINE.md (left untouched as the historical gate record). README.md documents how to add the next release's column.
• Four new workloads closing gaps a published suite couldn't ship without: patterns (the pattern engine had zero benchmark coverage), metamethods (: self-dispatch, multi-hop __index, arithmetic metamethods), pcall_varargs, and vm_new. Existing workloads now thread Lua.load_chunk! state so the chunk path measures correctly on pre-1.0 refs.
• A hosted results page at /benchmarks on the website app — a dead view that renders the committed summary.json files, so every figure on it derives from the recorded data rather than being transcribed. Version directories are discovered by glob, so recording a new release adds a column without a code change; see What updates itself in bench_results/README.md.
• Changelog correction for the 1.0.2 Lua.new/1 entry, now quoting measured figures.

Headlines (medians, chunk path, Luerl 1.5.1 as same-run control)

Workload	v0.4.0	v1.0.0	v1.0.2	v1.0.2 vs Luerl
Lua.new() steady-state	21.33 µs	36.67 µs	0.58 µs	26× faster
Lua.new() allocation	73.78 KB	91.88 KB	0.88 KB	—
fibonacci fib(30)	702.84 ms	792.42 ms	434.12 ms	1.7× faster
closures	368.92 µs	473.54 µs	373.71 µs	1.04× faster
string.format (literal-heavy)	3.91 ms	1.03 ms	716.04 µs	5.5× faster
pcall success path	290.50 µs	282.13 µs	222.88 µs	1.3× faster
pcall raise + catch	469.29 µs	877.33 µs	727.13 µs	1.55× slower ⚠️
varargs + multi-return	2.40 ms	4.01 ms	3.28 ms	1.36× slower ⚠️

1.0.2 is the first release faster than Luerl on most workloads. The two ⚠️ rows plus large-list host-boundary decode are called out in the report as the 1.1.x targets (the raise path partly reflects PUC-conformant position-prefixed error messages that Luerl doesn't build).

Measurement discipline

Apple M4 · Elixir 1.20.0 / OTP 29 · LUA_BENCH_MODE=full · strictly serial, one mix run at a time, quiet machine · each ref in a throwaway detached worktree · medians quoted. Luerl runs inside every benchmark as a control; its spread across the three runs was under 1%, so cross-version deltas are attributable to the library.

Changelog note

The 1.0.2 entry now reads "~60x faster for the default configuration (36.7µs → 0.6µs median …) and ~5.5x faster when a custom sandbox is passed" — the measured full-mode figures from benchmarks/vm_new.exs under mix run.

[davydog187]

why is this in here?

[davydog187]

Two different kinds of junk, both now stripped in 971d18c.

Lines 1–7 are a zoxide shell-configuration notice — local shell state that leaked into the capture because the benchmark was driven through an interactive shell.

Lines 9–14 are a real compiler warning, but it belongs to v0.4.0 itself, not to main. At that tag Lua.Util has a format_error/2 whose last clause is unreachable:

$ git show v0.4.0:lib/lua/util.ex | sed -n "177p"
  defp format_error(_, {:undefined, args}) do

main has only format_error/1, so there is nothing to fix here — it is compile output from a released tag we measure but do not change. That is also why the warning appears in the v0.4.0/ captures and not in v1.0.0/ or v1.0.2/.

The strip removes leading lines only, stopping at each file's first real output line: 344 deletions, 0 insertions, nothing inside a capture touched.

[davydog187]

this is specific to my machine, this shouldn't be in here

[davydog187]

Agreed — replaced with <repo> and <worktree> placeholders in c9dce4f.

The file already used <worktree> a few lines further down (cp -R <repo>/benchmarks <worktree>/benchmarks), and the v1.0.0/environment.md notes were written generically from the start, so this just brings v0.4.0 in line. It was the only file in bench_results/ with machine paths.

[davydog187]

why are we doing this this way (html file instead of writing a live view / dead view and rendering data)???? we also never link to t his anywhere

[davydog187]

Fair — rewritten as a dead view in d50cb56. priv/static/benchmarks.html is gone, along with its static_paths entry.

Now: get "/benchmarks" → PageController.benchmarks/2 → page_html/benchmarks.html.heex, inside Layouts.app, so it gets the nav, footer and theme toggle for free. Linked from the desktop nav, the mobile menu, and the footer's Product column.

Rendering data, not transcribing it. Website.Benchmarks reads the committed bench_results/<version>/summary.json files at compile time (@external_resource, so editing results recompiles). The table, the ratio plot and the headline tiles all derive from it — the plot is server-rendered divs with computed percentages, so the client JS is gone too. As a check that the parsing is right, every figure was diffed against what the static page carried by hand: all 36 table cells and all 27 plot ratios match.

On keeping it easy to update — this was the part worth getting right, so the page tracks the directories rather than a hardcoded list:

• a new bench_results/<version>/summary.json becomes a new column (glob + Version.compare/2, no list to extend)
• the newest versions-<date>.md becomes the linked report and the page's dated eyebrow
• the tiles and the "still behind Luerl" figures re-derive, including the "N× faster than " deltas

Two things still need a human, and bench_results/README.md now says so under What updates itself: a new workload needs a row spec (which cases are worth showing is editorial), and the prose is written, not generated.

One consequence worth flagging: the compile-time read means the results directory has to be in the Docker build context, which it was not — the Dockerfile only copied mix.exs, lib/ and website/. Added a COPY bench_results line next to the existing README.md-is-read-at-compile-time precedent, and the module raises a pointed error rather than silently rendering an empty page if it is ever missing. I built the release container and fetched the page from it to confirm: 200 with all figures present, old /benchmarks.html path 404s.

Also fixed the "1.0.2 is the release cut today" line you would have hated in six months — it now reads from the data.

[davydog187]

Code review (AI)

Overview

Records the first cross-version benchmark campaign (v0.4.0 / v1.0.0 / v1.0.2 with Luerl 1.5.1 as a same-run control), adds four workloads (patterns, metamethods, pcall_varargs, vm_new), fixes chunk-state threading so the chunk path is valid on pre-1.0 refs, publishes a self-contained results page at /benchmarks.html, and corrects the 1.0.2 changelog's Lua.new/1 figures. Establishes the per-release bench_results/<version>/ convention.

Verified

• Numbers reproduce. Spot-checked report/page/PR-body figures against the committed summary.json files: fibonacci medians (434.12 ms vs 741.95 µs→ms control), all four vm_new shapes across all three refs, and the derived multipliers (63× vs 1.0.0, 26× vs Luerl, 5.5× for the custom-exclude shape) all match the raw data. 3a0d392 is on origin/main.
• Chunk-state threading fix is correct. At v0.4.0 a %Lua.Chunk{} holds a ref into the state it was loaded against, so {chunk, _} = Lua.load_chunk!(...) invalidated the chunk. Every script now threads the returned state; the table_ops refactor to a single load_chunks helper with Enum.map_reduce is both correct and tighter than the five copy-pasted blocks it replaces.
• Website page is genuinely self-contained: no external scripts/styles/fonts; the only outbound references are two GitHub hrefs. Dark/light handled via both prefers-color-scheme and :root[data-theme=…] overrides. The static_paths addition is the minimal change.
• Formatting: all touched .exs files pass mix format --check-formatted; website/lib/website_web.ex passes under the website app's own .formatter.exs (the root formatter disagrees with pre-existing lines, not the changed one).
• Measurement honesty is the standout quality of this PR: the oop control-drift warning, the sub-µs deviation caveat, the raise-path conformance asymmetry, the "v0.4.0 is Luerl → its ratio column is the noise floor" framing, and excluding the noisy n=10/n=100 table cells are all exactly the caveats a reader would otherwise trip over.

Findings

1. The v1.0.2 tag does not exist (locally or on origin — latest is v1.0.1), yet mix.exs is at 1.0.2, the changelog entry is dated 2026-07-28, the results directory is bench_results/v1.0.2/, and the README convention this PR establishes says "After tagging, run the full suite against the tag." The convention's own exemplar doesn't satisfy it. Separately, docs.source_ref: "v#{@version}" means HexDocs source links for 1.0.2 point at a nonexistent ref. Recommend pushing the v1.0.2 tag at 3a0d392 (the recorded release commit) before or alongside merging; environment.md already discloses the measurement was from main, so no re-run is needed.
2. Merge-order coupling on absolute links. The changelog and the hosted page footer link to blob/main/bench_results/versions-2026-07-28.md, which 404s until this PR merges. Intentional (the changelog ships in the Hex package, so a relative link can't work) — just don't cherry-pick the changelog hunk anywhere ahead of the report.
3. Nit: the identical 4-line "state returned by load_chunk!/2 is threaded…" comment appears in 7 scripts. The constraint is worth stating (the code alone can't show why discarding the state breaks v0.4.0), but one sentence pointing at the report's Reproduction §3 would carry the same weight without the 7-way duplication. Fine to leave.
4. Nit: the hosted page's "1.0.2 is the release cut today" will read oddly in six months; the dated eyebrow line mostly covers it.

No correctness, security, or performance concerns in the code itself — the only library-code change is one entry appended to static_paths. No test coverage is expected for benchmark scripts and a static page.

Verdict: ship it (after pushing the tag). 🌿