Repository navigation
docs: regenerate the benchmark figures and add their scripts - #24
Merged
Merged
Conversation
…figures Adds the RTX 3050 release and fused-path results, a per-GPU release figure with the v1.2.0 gain on its bar, and a green fused-kernel palette across the figures. The 3050's convergence curve gets its own green shade and a dash pattern that fits the legend handle; the release stages are drawn in shades of the fused green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The README's benchmark figures were last generated in August, on a CNN that no longer matches
examples/cifar10/v1_cifar10.pyby scripts that couldn't be regenerated from the repo. This PR moves that tooling intobenchmarks/, re-measures on the model fromcifar10_breakdown.build_model, and commits the results the figures are drawn from.Commits are split into the build_model refactor, the new scripts, the regenerated AMD figures, the RTX 3050 results with the green recolor, and the move of the figures into notebooks/assets/benchmarks/.
Benchmark scripts
cifar10_paths.pytrains on NumPy, fallback CuPy and fused kernels, for the throughput and convergence figures.--fused-onlyadds another GPU to the convergence figure without the slower paths.cifar10_vram.pyrecords live pool bytes inside the realmodel.train(), for the VRAM figure.plot.pydraws every figure frombenchmarks/history/, so a figure can be redrawn without a GPU.build_modelnow takes a device, and results record the commit of theaethertree being measured, so a run against an older tag is labelled correctly.The old fallback override no longer worked on this code, and the new one fixes both problems:
Figures
The legacy series is gone, because
examples/legacyhas noGlobalAvgPooland can't run this model.The two columns are not comparable: the model is larger and the batch size went from 128 to 256. Conv2d also only honors fp16 on the fused path, so the fallback paths compute it in fp32, and the README now says so.
benchmark_releaseis the fair comparison, measuring both versions on the same model and GPU:Each GPU has its own panel and scale, so each one is compared only with itself. This figure is for the v1.2 release notes, not the README.
The convergence figure adds the 3050's fused path as a dashed curve, to show both vendors' kernels training the same model. Its title and equal-loss line still use only the AMD paths.
The NVIDIA runs were run connected to the wall.
Behavior changes
None. Nothing in
aether/changed. matplotlib is only needed byplot.pyand isn't added topyproject.toml.The only visible change is where the benchmark PNGs live: they moved from
notebooks/assets/tonotebooks/assets/benchmarks/.plot.pynow writes there and the README image URLs point at the new path, so any external link to the old paths will break.Testing
python3 -m unittest discover tests: 1190 tests pass on NumPy and CuPy (78 skipped).aether/changed.ruff check benchmarkspasses.cbaa753(v1.1.0) and089667c, on the RX 9070 XT (ROCm) and the RTX 3050 Laptop (CUDA).