Repository navigation
Benchmark sharding basic functionality - #1813
Conversation
…CONNECTIVITY BECOMES MULTITHREADED
asv preimports the benchmark suite before it runs any setup_cache (asv/runner.py, spawner.preimport() ahead of the run loop), so on a cold cache bench_connectivity's import-time preload_topologies is what fills it -- serially, in the forkserver parent, before a single benchmark starts. CachedFixtures.setup_cache then finds everything already built, and prime(workers=...) never runs on the path it was written for. Filling it from the CLI first puts those reads back in the parallel prime. Worth a second or two on the GitHub runners, which only see the oQU grids; worth rather more on a machine that can reach the four dyamond grids on campaign storage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Text only: module docstrings cut to purpose, key constraints and usage; comments to one or two lines of why. Fixes two that had drifted from the code (a stale stderr claim in _threads.resolve, a contradictory matrix note in asv.conf.hpc.json) and a comment above the wheels cache that described the durations cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NeighborhoodReduce ran at r=15, where a 120km face has ~600 neighbors and time_dataset_reduce alone took over a minute per round -- the class that bounded the slowest shard. It now runs at 5, shared with NeighborhoodDask as NEIGHBORHOOD_RADIUS. NeighborhoodBuild swept r over 1, 5 and 15 for every benchmark. The timings now skip 15 (1.3s per call at 120km) and the track_* benchmarks skip 1, where peakmem and nbytes barely move (362k vs 384k at 480km). asv reads params from the method first, so this stays one class and the kernels still compile once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Same shards, caches and pre-builds; about 630 fewer lines. - _merge: union rows and max durations. Every shard's benchmarks.json already lists the whole suite (asv saves all it discovers), and the conflict, ordering and column-realignment paths were unreachable. - _partition: group by class. No setup_cache outside _fixtures spans classes, so the union-find never merged more. --shard now implies --asv-args; the uncalled --bench-args and --config-out are gone, and the report lists the heaviest benchmarks in place of the workflow's inline script. - _machine: write asv's detected specs under the pinned name directly, replacing `asv machine --yes` plus a rename. - Drop the unused _threads.py and asv.conf.hpc.json. The latter had drifted from asv.conf.json (no conda_environment_file), so HPC runs built a different env than CI. stage.pbs now always sets NUMBA_NUM_THREADS (default 8; numba's default counts SMT siblings) and passes -a timeout=1800. - local.sh pins each shard to whole physical cores in socket order, so concurrent shards no longer share SMT siblings on nodes that number them N and N + cores. - PR workflow: YAML anchors for the steps repeated across jobs, one SHARDS setting, and a shorter baseline resolution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ASV BenchmarkingBenchmark Comparison ResultsBenchmarks that have stayed the same:
Benchmarks that have got worse:
|
…ns from main asv_runner re-runs setup before every sample, about six times per benchmark process. NeighborhoodReduce.time_reduce timed a 5ms kernel at 120km but took 27s, and NeighborhoodDask.time_mean 49s for 1-40ms calls, almost all setup. That made NeighborhoodReduce alone larger than a quarter of the suite, which capped a 4-shard split at 3.2x. - NeighborhoodReduce and NeighborhoodDask build their fixture, warmup and neighborhood once per process. The timed calls only read them. The grid caches a single ball tree and time_dataset_reduce leaves it on edges, so setup restores the face tree each sample, as a fresh setup did. - main's benchmark workflow saves the commit it just ran under the asv-results cache key. PR caches are scoped to their PR, so every PR's first run had no durations and split by count (this branch's first run: shards of 3:11, 4:48, 2:17 and 9:49). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sevans711
left a comment
There was a problem hiding this comment.
Great overall, I'm excited for very fast ASV benchmark runs! I read through all the changes and they look pretty reasonable to me. Because this is all for benchmarks, not anything user-facing, I didn't take the time to spin up a local or HPC environment to test it directly for myself. It seems to work, and I think that should be good enough? And, if there's some subtle bug that gets missed here, it could always be fixed in a follow-up PR, without users ever being affected directly.
I think there's only one important thing to fix before merging, and that would be the roughly 11% to 30% increases to track_peakmem for many benchmarks here. I would hope to see no changes to peakmem usage, to avoid re-introducing something like #1605. Any guesses on what is causing this?
Beyond that, I also have a few clarifying questions, just so I can understand this a bit better:
- Where do the sharding timing weights come from when you make a new github action ASV run? I can understand from the code that if you run things locally you are just using the previous job's weights to decide how to group jobs into shards. Are these being cached somewhere on github? What happens if the cache doesn't exist?
- Can you clarify, are the changes in
mpas_ocean.pyrelated to the sharding at all, or are they a completely separate improvement (i.e. is this part of the original issue or an expansion of PR scope)? I don't necessarily have any concerns about those changes, just trying to understand the motivation for them a bit better.
|
@Sevans711 The memory issues are ongoing, I think this issue is something about
|
There was a problem hiding this comment.
I would recommend fusing/melding commits together so that the history of changes are more tractable.
My understanding of the changes in this PR,
The benchmark test suite run using asv (Airspeed Velocity) in github CI workflow takes up to 20-30 mins to run. This PR shards (breaks up) the benchmark test suite into multiple independent jobs, containing batches of tests, that run in parallel. Running these test shards/batches in parallel reduces the overall time taken to run the benchmark. However since the project does not use custom testing frameworks that support sharding the mechanism of sharding/batching tests is implemented manually.
The PR contains the following changes,
- A helper script, _machine.py, that captures the "machine/node name" from the environment (or platform)
- Helper scripts for sharding/partitioning the tests into batches and gathering the test results
** Script _partition.py : Reads benchmark.json (asv output, one for each run) from result dirs to calculate mean "duration" (wallclock time for benchmark) for each benchmark. The benchmarks are binned greedily (sort benchmarks using duration, longest first, and add to bin with the lowest load) into separate shards/bins (number of bins is a parameter to script). The asv config file (parameter to script) is updated with the shard information.
** Script _merge.py : Since the shard workflow jobs are run in separate directories (and the results are in these directories) this script consolidates/merges the results (benchmarks.json) and machine files (machine.json) in each result directory into a separate result directory (benchmarks.json in this directory contains results from all shard jobs). - Since all shard jobs need to use the same input files (oQU*.nc), asv files (wheels) these files are cached
- Misc fixes
** Adding env for OMP, MKL, BLAS
** Updates to MPAS neighborhood tests
|
As the number of benchmark runs (previous/older) increase you might also want to investigate alternatives to scanning results from all previous runs to calculate mean benchmark runtimes (when binning the tests to shard jobs). |
Thank you, that clarifies things, and sounds good for now! It makes sense that a more robust mechanism might be a good avenue for future improvement (but that can be scoped to a separate PR).
Sounds good, I did a closer read-through of those changes with this in mind, and they look reasonable to me.
I don't think these memory issues are on |
- _merge: seed each file from the first shard that has it and fold later shards' results in, telling result files by their "results" key rather than a list of whole-file names. rel is now relpath, since keys keep the machine directory. - The results artifact is named with runner.os again, as on main. The comment workflow still downloads asv-benchmark-results-Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
load_weights keeps every sample, so the median costs nothing over the mean and ignores a single cold-numba outlier once there are three or more, as in a local results dir with history. CI restores two samples per benchmark (base and PR), where the two agree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Skip machine.json by its missing "results", as _merge does, and index result_columns without a fallback: every v2 results file (asv >= 0.5) has a duration column, and asv itself won't read a v1 file without asv update. The median doesn't depend on order, so the glob needn't be sorted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
benchmark_classes maps each class (or module, for a bare function) to its (benchmark name, cost) pairs, and class_costs to their total, in place of classes, class_cost and owner. Plans are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@Sevans711 The memory issues are still on main, you can see them crop up in other PRs occasionally. They've proven fairly difficult to nail down, so I'd defer that to another PR. |
Sevans711
left a comment
There was a problem hiding this comment.
The memory issues are still on main, you can see them crop up in other PRs occasionally. They've proven fairly difficult to nail down, so I'd defer that to another PR.
True, I can see some of them on other PRs occasionally. Though, not nearly as many or as consistently as they seem to occur here.
Happy to approve this due to the huge ASV suite speedup, with the understanding that this would be investigated further later, but that it comes with the cost of seeing these memory tracking issues more often for now. I don't think this problem will get lost or forgotten, but if you have a few minutes to open up another issue report to track it, before this PR merges, that would be great!
This issue is now being tracked here: #1891 |
Closes #1814
Overview
This PR reworks the ASV CI workflow to split the benchmarking suite into separate jobs, allowing for parallel benchmarking runs.
Basically, all the benchmarks are scored and are roughly load-balanced based on the number of threads available. Each job runs independently, so there is some overhead with redundant benchmark cache builds. This work can also support larger HPC-scale jobs, but I'm deferring the actual HPC scripts to another PR.
Current timings show benchmark suite wall time usually around 6-8 minutes, versus 20-30 minutes for the current baseline.
PR Checklist
General
Testing & Benchmarking
Documentation and Examples
docs/api.rst; internal (private) function names start with an underscore (_)AI Disclosure
AI Usage: Claude Opus 5, Opus 5.5