Skip to content

docs: regenerate the benchmark figures and add their scripts - #24

Merged
AlexanderSoftCode merged 5 commits into
mainfrom
docs/benchmark-figures
Nov 1, 2026
Merged

AlexanderSoftCode merged 5 commits into
mainfrom
docs/benchmark-figures

Conversation

@AlexanderSoftCode

@AlexanderSoftCode AlexanderSoftCode commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Context

The README's benchmark figures were last generated in August, on a CNN that no longer matches examples/cifar10/v1_cifar10.py by scripts that couldn't be regenerated from the repo. This PR moves that tooling into benchmarks/, re-measures on the model from cifar10_breakdown.build_model, and commits the results the figures are drawn from.

Commits are split into the build_model refactor, the new scripts, the regenerated AMD figures, the RTX 3050 results with the green recolor, and the move of the figures into notebooks/assets/benchmarks/.

Benchmark scripts

  • cifar10_paths.py trains on NumPy, fallback CuPy and fused kernels, for the throughput and convergence figures. --fused-only adds another GPU to the convergence figure without the slower paths.
  • cifar10_vram.py records live pool bytes inside the real model.train(), for the VRAM figure.
  • plot.py draws every figure from benchmarks/history/, so a figure can be redrawn without a GPU.
  • build_model now takes a device, and results record the commit of the aether tree being measured, so a run against an older tag is labelled correctly.

The old fallback override no longer worked on this code, and the new one fixes both problems:

  • It moved Adam's buffers to the host while the weights stayed on the GPU, which crashed on both v1.1.0 and main.
  • It left BatchNorm unfused while the fused ReLUs stayed skipped, so on main the fallback model silently ran without its ReLUs.

Figures

The legacy series is gone, because examples/legacy has no GlobalAvgPool and can't run this model.

Measure (RX 9070 XT, 265K for NumPy) Before After
Fused kernels 3.17 ms/step 4.85 ms/step
CuPy, no custom kernels 52.6 ms/step 202 ms/step
NumPy CPU 326 ms/step 1,865 ms/step
Peak transient VRAM, fused vs no kernels 32 vs 180 MiB 80 vs 420 MiB

The two columns are not comparable: the model is larger and the batch size went from 128 to 256. Conv2d also only honors fp16 on the fused path, so the fallback paths compute it in fp32, and the README now says so.

benchmark_release is the fair comparison, measuring both versions on the same model and GPU:

GPU v1.1.0 v1.2 Gain
AMD Radeon RX 9070 XT 10.24 ms/step 4.86 ms/step 2.1×
NVIDIA RTX 3050 Laptop 49.9 ms/step 21.7 ms/step 2.3×

Each GPU has its own panel and scale, so each one is compared only with itself. This figure is for the v1.2 release notes, not the README.

The convergence figure adds the 3050's fused path as a dashed curve, to show both vendors' kernels training the same model. Its title and equal-loss line still use only the AMD paths.

The NVIDIA runs were run connected to the wall.

Behavior changes

None. Nothing in aether/ changed. matplotlib is only needed by plot.py and isn't added to pyproject.toml.

The only visible change is where the benchmark PNGs live: they moved from notebooks/assets/ to notebooks/assets/benchmarks/. plot.py now writes there and the README image URLs point at the new path, so any external link to the old paths will break.

Testing

  • python3 -m unittest discover tests: 1190 tests pass on NumPy and CuPy (78 skipped).
  • No new regression tests, since nothing in aether/ changed.
  • ruff check benchmarks passes.
  • All results come from clean worktrees of cbaa753 (v1.1.0) and 089667c, on the RX 9070 XT (ROCm) and the RTX 3050 Laptop (CUDA).
  • Viewed the charts for label collisions and layout.

@AlexanderSoftCode
AlexanderSoftCode merged commit fc06f99 into main Nov 1, 2026
5 checks passed
…figures

Adds the RTX 3050 release and fused-path results, a per-GPU release figure with the
v1.2.0 gain on its bar, and a green fused-kernel palette across the figures. The
3050's convergence curve gets its own green shade and a dash pattern that fits the
legend handle; the release stages are drawn in shades of the fused green.
@AlexanderSoftCode
AlexanderSoftCode deleted the docs/benchmark-figures branch November 1, 2026 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant