Skip to content

Benchmark sharding basic functionality - #1813

Merged
cmdupuis3 merged 63 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/bench_shards
Oct 14, 2026
Merged

cmdupuis3 merged 63 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/bench_shards

Conversation

@cmdupuis3

@cmdupuis3 cmdupuis3 commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1814

Overview

This PR reworks the ASV CI workflow to split the benchmarking suite into separate jobs, allowing for parallel benchmarking runs.

Basically, all the benchmarks are scored and are roughly load-balanced based on the number of threads available. Each job runs independently, so there is some overhead with redundant benchmark cache builds. This work can also support larger HPC-scale jobs, but I'm deferring the actual HPC scripts to another PR.

Current timings show benchmark suite wall time usually around 6-8 minutes, versus 20-30 minutes for the current baseline.

PR Checklist

General

  • An issue is created and linked
  • Added appropriate labels (if your uxarray repo permissions allow it)
  • Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

  • There is adequate test coverage of changes from this PR (add new tests if needed)
  • If this PR could affect performance, ran ASV benchmarks and confirmed they show expected behavior (add a new benchmark if necessary)

Documentation and Examples

  • Docstrings updated with any function changes, and included in all new functions
  • User (public) functions added to docs/api.rst; internal (private) function names start with an underscore (_)

AI Disclosure

AI Usage: Claude Opus 5, Opus 5.5

  • I have tested and take responsibility for all AI-generated content in my PR.

cmdupuis3 and others added 30 commits August 21, 2026 12:07
asv preimports the benchmark suite before it runs any setup_cache
(asv/runner.py, spawner.preimport() ahead of the run loop), so on a cold
cache bench_connectivity's import-time preload_topologies is what fills
it -- serially, in the forkserver parent, before a single benchmark
starts. CachedFixtures.setup_cache then finds everything already built,
and prime(workers=...) never runs on the path it was written for.

Filling it from the CLI first puts those reads back in the parallel
prime. Worth a second or two on the GitHub runners, which only see the
oQU grids; worth rather more on a machine that can reach the four
dyamond grids on campaign storage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cmdupuis3 and others added 4 commits October 6, 2026 16:44
Text only: module docstrings cut to purpose, key constraints and usage;
comments to one or two lines of why. Fixes two that had drifted from the
code (a stale stderr claim in _threads.resolve, a contradictory matrix note
in asv.conf.hpc.json) and a comment above the wheels cache that described
the durations cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NeighborhoodReduce ran at r=15, where a 120km face has ~600 neighbors and
time_dataset_reduce alone took over a minute per round -- the class that
bounded the slowest shard. It now runs at 5, shared with NeighborhoodDask
as NEIGHBORHOOD_RADIUS.

NeighborhoodBuild swept r over 1, 5 and 15 for every benchmark. The
timings now skip 15 (1.3s per call at 120km) and the track_* benchmarks
skip 1, where peakmem and nbytes barely move (362k vs 384k at 480km).
asv reads params from the method first, so this stays one class and the
kernels still compile once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Same shards, caches and pre-builds; about 630 fewer lines.

- _merge: union rows and max durations. Every shard's benchmarks.json
  already lists the whole suite (asv saves all it discovers), and the
  conflict, ordering and column-realignment paths were unreachable.
- _partition: group by class. No setup_cache outside _fixtures spans
  classes, so the union-find never merged more. --shard now implies
  --asv-args; the uncalled --bench-args and --config-out are gone, and
  the report lists the heaviest benchmarks in place of the workflow's
  inline script.
- _machine: write asv's detected specs under the pinned name directly,
  replacing `asv machine --yes` plus a rename.
- Drop the unused _threads.py and asv.conf.hpc.json. The latter had
  drifted from asv.conf.json (no conda_environment_file), so HPC runs
  built a different env than CI. stage.pbs now always sets
  NUMBA_NUM_THREADS (default 8; numba's default counts SMT siblings) and
  passes -a timeout=1800.
- local.sh pins each shard to whole physical cores in socket order, so
  concurrent shards no longer share SMT siblings on nodes that number
  them N and N + cores.
- PR workflow: YAML anchors for the steps repeated across jobs, one
  SHARDS setting, and a shorter baseline resolution.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cmdupuis3 cmdupuis3 self-assigned this Oct 7, 2026
@cmdupuis3 cmdupuis3 added benchmarking Related to benchmarks, memory usage, and/or time profiling run-benchmark Run ASV benchmark workflow labels Oct 7, 2026
@cmdupuis3
cmdupuis3 marked this pull request as ready for review October 7, 2026 23:35
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have stayed the same:

Change Before [0287eb0] After [785345e] Ratio Benchmark (Parameter)
997±30μs 978±10μs 0.98 connectivity.Connectivity.time_edge_face('120km')
487±10μs 483±20μs 0.99 connectivity.Connectivity.time_edge_face('480km')
4.00±0.4ms 4.01±0.04ms 1.00 connectivity.Connectivity.time_edge_node('120km')
1.29±0.04ms 1.32±0.02ms 1.02 connectivity.Connectivity.time_edge_node('480km')
70.7±9μs 85.0±8μs ~1.20 connectivity.Connectivity.time_face_edge('120km')
71.2±4μs 81.1±10μs ~1.14 connectivity.Connectivity.time_face_edge('480km')
987±40μs 998±20μs 1.01 connectivity.Connectivity.time_face_face('120km')
413±20μs 433±10μs 1.05 connectivity.Connectivity.time_face_face('480km')
65.6±20μs 67.5±8μs 1.03 connectivity.Connectivity.time_face_node('120km')
62.4±5μs 65.3±10μs 1.05 connectivity.Connectivity.time_face_node('480km')
495±30μs 511±40μs 1.03 connectivity.Connectivity.time_n_nodes_per_face('120km')
426±20μs 426±20μs 1.00 connectivity.Connectivity.time_n_nodes_per_face('480km')
1.40±0.02ms 1.39±0.03ms 1.00 connectivity.Connectivity.time_node_edge('120km')
498±20μs 492±10μs 0.99 connectivity.Connectivity.time_node_edge('480km')
79.0±2ms 87.4±6ms ~1.11 connectivity.Connectivity.time_node_face('120km')
5.06±0.02ms 5.20±0.05ms 1.03 connectivity.Connectivity.time_node_face('480km')
1.42M 1.42M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('120km')
106k 106k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('480km')
6.48M 6.48M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('120km')
420k 420k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('480km')
2.42k 2.42k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('120km')
2.42k 2.42k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('480km')
1.6M 1.6M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('120km')
101k 101k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('480km')
2.54k 2.54k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('120km')
2.54k 2.54k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('480km')
240k 240k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('120km')
26.3k 26.3k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('480km')
1.9M 1.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('120km')
127k 127k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('480km')
11.9M 11.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('120km')
747k 747k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('480km')
8.62±0.1ms 8.59±0.1ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
2.77±0.2ms 2.78±0.02ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
10.2±0.2ms 10.2±9ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
1.56±0.02ms 1.62±0.03ms 1.04 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
57.3k 57.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
12.3k 12.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
123k 123k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
128 128 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.27M 1.27M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
50.1k 50.1k 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.48M 1.48M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
712 712 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.85M 1.85M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
1.85M 1.85M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
2.01M 2.01M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
35.6k 35.6k 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.13±0.01μs 1.14±0.01μs 1.01 geometry_kernels.AccucrossKernels.time_accucross
2.58±0.03μs 2.57±0.04μs 1.00 geometry_kernels.AccucrossKernels.time_accucross_pair
430±9ns 426±20ns 0.99 geometry_kernels.EFTPrimitives.time_acc_sqrt_re
441±20ns 436±20ns 0.99 geometry_kernels.EFTPrimitives.time_diff_of_products
366±10ns 406±20ns ~1.11 geometry_kernels.EFTPrimitives.time_two_prod
366±9ns 375±9ns 1.03 geometry_kernels.EFTPrimitives.time_two_sum
662±9ns 667±20ns 1.01 geometry_kernels.GCAConstLatIntersection.time_accux_constlat_kernel
741±9ns 717±30ns 0.97 geometry_kernels.GCAConstLatIntersection.time_gca_const_lat_intersection
821±40ns 781±40ns 0.95 geometry_kernels.GCAConstLatIntersection.time_try_gca_const_lat_intersection
791±5ns 791±8ns 1.00 geometry_kernels.GCAGCAIntersection.time_accux_gca_kernel
882±30ns 881±10ns 1.00 geometry_kernels.GCAGCAIntersection.time_gca_gca_intersection
1.04±0.04μs 1.00±0.01μs 0.96 geometry_kernels.GCAGCAIntersection.time_try_gca_gca_intersection
51.0±0.4μs 52.8±0.4μs 1.03 geometry_kernels.OrientPredicates.time_on_minor_arc
50.7±0.7μs 52.4±0.6μs 1.03 geometry_kernels.OrientPredicates.time_orient3d_on_sphere
3.05±0ms 3.09±0.01ms 1.01 geometry_samebody.SameBodyConstLat.time_accux_dispatch
1.16±0ms 1.16±0ms 1.00 geometry_samebody.SameBodyConstLat.time_accux_kernel
2.29±0ms 2.28±0ms 0.99 geometry_samebody.SameBodyConstLat.time_fp64_dispatch
148±0.1μs 147±0.6μs 1.00 geometry_samebody.SameBodyConstLat.time_fp64_kernel
29.5±0.03ms 29.6±0.02ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_dispatch
6.28±0.01ms 6.33±0ms 1.01 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_kernel
23.6±0.07ms 23.7±0.4ms 1.01 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_dispatch
861±2μs 870±7μs 1.01 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_kernel
902±10ms 903±8ms 1.00 import.Imports.timeraw_import_uxarray
2.46±0.04ms 2.57±0.6ms 1.04 mpas_ocean.CheckNorm.time_check_norm('120km')
2.09±0.06ms 2.11±0.07ms 1.01 mpas_ocean.CheckNorm.time_check_norm('480km')
1.13±0.01ms 1.17±0.01ms 1.04 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
568±20μs 563±20μs 0.99 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
678±20μs 674±20μs 0.99 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
613±10μs 615±5μs 1.00 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
5.28±0.02ms 5.39±0.04ms 1.02 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
3.83±0.03ms 3.94±0.05ms 1.03 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
98.3±1ms 95.8±0.4ms 0.97 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
9.32±0.02ms 9.82±0.3ms 1.05 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
18.7±0.05ms 18.8±0.1ms 1.01 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
977±20μs 965±20μs 0.99 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
9.95±0.05ms 10.1±0.06ms 1.02 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
621±9μs 641±10μs 1.03 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
604±8ms 605±3ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 1)
302±5ms 311±2ms 1.03 mpas_ocean.CrossSections.time_const_lat('120km', 2)
156±2ms 160±0.9ms 1.02 mpas_ocean.CrossSections.time_const_lat('120km', 4)
545±9ms 553±2ms 1.02 mpas_ocean.CrossSections.time_const_lat('480km', 1)
273±3ms 276±0.6ms 1.01 mpas_ocean.CrossSections.time_const_lat('480km', 2)
145±0.9ms 143±0.1ms 0.99 mpas_ocean.CrossSections.time_const_lat('480km', 4)
338M 363M 1.07 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 1)
338M 363M 1.08 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 2)
338M 363M 1.08 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 4)
26.1±0.2ms 25.9±0.08ms 0.99 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
2.90±0.02ms 2.94±0.01ms 1.01 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
14.3±0.8ms 14.6±0.3ms 1.03 mpas_ocean.FaceAreas.time_face_areas('120km')
4.43±0.1ms 4.35±0.2ms 0.98 mpas_ocean.FaceAreas.time_face_areas('480km')
229k 229k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('120km')
14.3k 14.3k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('480km')
2.12M 2.12M 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('120km')
718k 718k 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('480km')
327±3ms 345±3ms 1.05 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', False)
42.0±0.1ms 44.6±0.7ms 1.06 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', False)
5.91±0.5ms 5.28±0.3ms ~0.89 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', True)
13.4±0.3ms 13.5±0.08ms 1.01 mpas_ocean.Gradient.time_gradient('120km')
1.82±0.01ms 1.84±0.01ms 1.01 mpas_ocean.Gradient.time_gradient('480km')
457k 457k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('120km')
28.7k 28.7k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('480km')
3.2M 3.2M 1.00 mpas_ocean.Gradient.track_peakmem_gradient('120km')
204k 204k 1.00 mpas_ocean.Gradient.track_peakmem_gradient('480km')
334M 363M 1.09 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('120km')
292±10μs 294±10μs 1.01 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
148±2μs 152±10μs 1.03 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
209±10μs 206±3μs 0.99 mpas_ocean.Integrate.time_integrate('120km')
192±4μs 189±3μs 0.98 mpas_ocean.Integrate.time_integrate('480km')
18.4M 18.4M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('120km')
1.2M 1.2M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('480km')
183±1ms 193±0.8ms 1.06 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
181±0.7ms 196±1ms 1.08 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
182±1ms 196±4ms 1.08 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
13.3±0.1ms 13.3±0.06ms 1.00 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
12.9±0.09ms 13.6±0.2ms 1.05 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
12.9±0.05ms 13.6±0.1ms 1.05 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
245±1ms 246±2ms 1.01 mpas_ocean.NeighborhoodBuild.time_build('120km', 1.0)
509±2ms 506±4ms 1.00 mpas_ocean.NeighborhoodBuild.time_build('120km', 5.0)
13.3±0.04ms 13.4±0.1ms 1.01 mpas_ocean.NeighborhoodBuild.time_build('480km', 1.0)
16.4±0.02ms 16.4±0.03ms 1.00 mpas_ocean.NeighborhoodBuild.time_build('480km', 5.0)
242±0.7ms 241±1ms 1.00 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 1.0)
500±2ms 504±1ms 1.01 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 5.0)
12.8±0.02ms 13.1±0.2ms 1.02 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 1.0)
16.1±0.1ms 16.2±0.05ms 1.01 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 5.0)
612.76 612.76 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 15.0)
74.17 74.17 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 5.0)
37.29 37.29 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 15.0)
6.57 6.57 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 5.0)
141M 141M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 15.0)
17.4M 17.4M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 5.0)
563k 563k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 15.0)
123k 123k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 5.0)
145M 145M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 15.0)
21.5M 21.5M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 5.0)
825k 825k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 15.0)
384k 384k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 5.0)
40.0±0.2ms 40.1±0.5ms 1.00 mpas_ocean.NeighborhoodDask.time_mean('120km', 'grid_chunks')
18.2±0.09ms 18.2±0.04ms 1.00 mpas_ocean.NeighborhoodDask.time_mean('120km', 'numpy')
37.2±0.6ms 35.6±0.5ms 0.96 mpas_ocean.NeighborhoodDask.time_mean('120km', 'time_chunks')
10.6±0.2ms 10.8±0.2ms 1.02 mpas_ocean.NeighborhoodDask.time_mean('480km', 'grid_chunks')
717±8μs 721±5μs 1.01 mpas_ocean.NeighborhoodDask.time_mean('480km', 'numpy')
7.37±0.08ms 7.49±0.09ms 1.02 mpas_ocean.NeighborhoodDask.time_mean('480km', 'time_chunks')
5.83M 5.83M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'grid_chunks')
2.75M 2.75M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'numpy')
5.68M 5.68M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'time_chunks')
757k 758k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'grid_chunks')
177k 177k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'numpy')
538k 539k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'time_chunks')
3.18±0.03s 3.31±0.04s 1.04 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'mean')
3.40±0.02s 3.41±0.04s 1.00 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'median')
85.9±0.8ms 87.3±0.3ms 1.02 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'mean')
87.2±3ms 88.6±0.3ms 1.02 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'median')
373±3ms 370±7ms 0.99 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'mean')
396±4ms 387±3ms 0.98 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'median')
12.2±0.2ms 11.6±0.2ms 0.95 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'mean')
12.1±0.2ms 12.5±0.1ms 1.03 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'median')
2.65±0.03ms 2.69±0.04ms 1.01 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'mean')
27.8±0.1ms 27.6±0.3ms 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'median')
151±4μs 152±20μs 1.01 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'mean')
383±5μs 379±3μs 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'median')
234k 234k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'mean')
235k 235k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'median')
19.4k 19.5k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'mean')
19.5k 19.5k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'median')
404±20μs 346±20μs ~0.86 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
329±20μs 338±30μs 1.03 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
356±40μs 383±40μs 1.07 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
309±40μs 345±40μs ~1.12 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
109±1ms 109±2ms 1.00 mpas_ocean.RemapDownsample.time_bilinear_remapping
17.3±0.07ms 17.7±0.2ms 1.02 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
15.5±0.09ms 15.8±0.05ms 1.02 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
1.11±0.01s 1.10±0.01s 0.99 mpas_ocean.RemapUpsample.time_bilinear_remapping
25.9±0.1ms 25.9±0.2ms 1.00 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
11.4±0.04ms 11.7±0.04ms 1.02 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
8.42±0.1ms 8.39±0.1ms 1.00 mpas_ocean.ZonalAverage.time_zonal_average('120km')
5.06±0.04ms 5.18±0.09ms 1.02 mpas_ocean.ZonalAverage.time_zonal_average('480km')
339M 363M 1.07 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('120km')
1.0332130440355918 1.0200674615218868 0.99 nogil_scaling.GILScaling.track_gil_scaling
7.51±0.07ms 7.68±0.1ms 1.02 quad_hexagon.QuadHexagon.time_open_dataset
6.31±0.05ms 6.55±0.1ms 1.04 quad_hexagon.QuadHexagon.time_open_grid
408 408 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_dataset
392 392 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_grid
73.6k 73.5k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_dataset
72.7k 71.7k 0.99 quad_hexagon.QuadHexagon.track_peakmem_open_grid
920±2ms 917±2ms 1.00 to_raster.ToRaster.time_to_raster((10000.0, 10.0))
2.23±0.03s 2.22±0s 0.99 to_raster.ToRaster.time_to_raster((10000.0, 100.0))
2.57±0.01s 2.56±0.02s 1.00 to_raster.ToRaster.time_to_raster((100000.0, 100.0))
271M 271M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 10.0))
684M 684M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 100.0))
768M 768M 1.00 to_raster.ToRaster.track_peakmem((100000.0, 100.0))

Benchmarks that have got worse:

Change Before [0287eb0] After [785345e] Ratio Benchmark (Parameter)
+ 318M 363M 1.14 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
+ 319M 363M 1.14 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
+ 320M 363M 1.14 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
+ 319M 363M 1.14 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
+ 276M 363M 1.31 import.Imports.track_peakmem_import_uxarray
+ 324M 363M 1.12 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 1)
+ 324M 363M 1.12 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 2)
+ 321M 363M 1.13 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 4)
+ 50.2±0.7ms 57.4±1ms 1.15 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', True)
+ 314M 363M 1.16 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('480km')
+ 322M 363M 1.13 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('480km')

…ns from main

asv_runner re-runs setup before every sample, about six times per benchmark
process. NeighborhoodReduce.time_reduce timed a 5ms kernel at 120km but took
27s, and NeighborhoodDask.time_mean 49s for 1-40ms calls, almost all setup.
That made NeighborhoodReduce alone larger than a quarter of the suite, which
capped a 4-shard split at 3.2x.

- NeighborhoodReduce and NeighborhoodDask build their fixture, warmup and
  neighborhood once per process. The timed calls only read them. The grid
  caches a single ball tree and time_dataset_reduce leaves it on edges, so
  setup restores the face tree each sample, as a fresh setup did.
- main's benchmark workflow saves the commit it just ran under the
  asv-results cache key. PR caches are scoped to their PR, so every PR's
  first run had no durations and split by count (this branch's first run:
  shards of 3:11, 4:48, 2:17 and 9:49).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Sevans711 Sevans711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great overall, I'm excited for very fast ASV benchmark runs! I read through all the changes and they look pretty reasonable to me. Because this is all for benchmarks, not anything user-facing, I didn't take the time to spin up a local or HPC environment to test it directly for myself. It seems to work, and I think that should be good enough? And, if there's some subtle bug that gets missed here, it could always be fixed in a follow-up PR, without users ever being affected directly.

I think there's only one important thing to fix before merging, and that would be the roughly 11% to 30% increases to track_peakmem for many benchmarks here. I would hope to see no changes to peakmem usage, to avoid re-introducing something like #1605. Any guesses on what is causing this?

Beyond that, I also have a few clarifying questions, just so I can understand this a bit better:

  1. Where do the sharding timing weights come from when you make a new github action ASV run? I can understand from the code that if you run things locally you are just using the previous job's weights to decide how to group jobs into shards. Are these being cached somewhere on github? What happens if the cache doesn't exist?
  2. Can you clarify, are the changes in mpas_ocean.py related to the sharding at all, or are they a completely separate improvement (i.e. is this part of the original issue or an expansion of PR scope)? I don't necessarily have any concerns about those changes, just trying to understand the motivation for them a bit better.

@cmdupuis3

cmdupuis3 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

@Sevans711 The memory issues are ongoing, I think this issue is something about subprocess_peak_rss on main.

  1. At the moment, they are basically cached and updated as a running average upon every benchmark run. To me, this makes sense on HPC, but I'm looking into possibly using a more robust mechanism, like having one of the bots here scan the repo's benchmark runs and re-weight them if they get too unbalanced.
  2. These are mostly unrelated, but are important because the neighborhood filters are otherwise the heaviest jobs atm, and they are harder to load-balance against. The two change sets are: (a) pruning expensive benchmarks by removing the large-radius jobs where we don't need them, and (b) running NeighborhoodReduce.setup per shard rather than per sample.

@jayeshkrishna jayeshkrishna left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend fusing/melding commits together so that the history of changes are more tractable.

My understanding of the changes in this PR,

The benchmark test suite run using asv (Airspeed Velocity) in github CI workflow takes up to 20-30 mins to run. This PR shards (breaks up) the benchmark test suite into multiple independent jobs, containing batches of tests, that run in parallel. Running these test shards/batches in parallel reduces the overall time taken to run the benchmark. However since the project does not use custom testing frameworks that support sharding the mechanism of sharding/batching tests is implemented manually.

The PR contains the following changes,

  • A helper script, _machine.py, that captures the "machine/node name" from the environment (or platform)
  • Helper scripts for sharding/partitioning the tests into batches and gathering the test results
    ** Script _partition.py : Reads benchmark.json (asv output, one for each run) from result dirs to calculate mean "duration" (wallclock time for benchmark) for each benchmark. The benchmarks are binned greedily (sort benchmarks using duration, longest first, and add to bin with the lowest load) into separate shards/bins (number of bins is a parameter to script). The asv config file (parameter to script) is updated with the shard information.
    ** Script _merge.py : Since the shard workflow jobs are run in separate directories (and the results are in these directories) this script consolidates/merges the results (benchmarks.json) and machine files (machine.json) in each result directory into a separate result directory (benchmarks.json in this directory contains results from all shard jobs).
  • Since all shard jobs need to use the same input files (oQU*.nc), asv files (wheels) these files are cached
  • Misc fixes
    ** Adding env for OMP, MKL, BLAS
    ** Updates to MPAS neighborhood tests

Comment thread benchmarks/helpers/_merge.py Resolved
Comment thread benchmarks/helpers/_merge.py Resolved
Comment thread benchmarks/helpers/_merge.py Resolved
Comment thread benchmarks/helpers/_partition.py Resolved
Comment thread benchmarks/helpers/_partition.py Resolved
Comment thread benchmarks/helpers/_partition.py Resolved
Comment thread benchmarks/mpas_ocean.py Resolved
@jayeshkrishna

Copy link
Copy Markdown

As the number of benchmark runs (previous/older) increase you might also want to investigate alternatives to scanning results from all previous runs to calculate mean benchmark runtimes (when binning the tests to shard jobs).

@Sevans711

Copy link
Copy Markdown
Collaborator
  1. At the moment, they are basically cached and updated as a running average upon every benchmark run. To me, this makes sense on HPC, but I'm looking into possibly using a more robust mechanism, like having one of the bots here scan the repo's benchmark runs and re-weight them if they get too unbalanced.

Thank you, that clarifies things, and sounds good for now! It makes sense that a more robust mechanism might be a good avenue for future improvement (but that can be scoped to a separate PR).

  1. These are mostly unrelated, but are important because the neighborhood filters are otherwise the heaviest jobs atm, and they are harder to load-balance against. The two change sets are: (a) pruning expensive benchmarks by removing the large-radius jobs where we don't need them, and (b) running NeighborhoodReduce.setup per shard rather than per sample.

Sounds good, I did a closer read-through of those changes with this in mind, and they look reasonable to me.

The memory issues are ongoing, I think this issue is something about subprocess_peak_rss on main.

I don't think these memory issues are on main currently? I haven't seen them in other PRs, and #1710 (comment) doesn't show them either (I merged main into that branch a few days ago to have it re-run the benchmarks there, just to double-check). I think this is the last outstanding point from me, and I would be happy to approve if it can be resolved!

cmdupuis3 and others added 4 commits October 12, 2026 13:45
- _merge: seed each file from the first shard that has it and fold later
  shards' results in, telling result files by their "results" key rather
  than a list of whole-file names. rel is now relpath, since keys keep the
  machine directory.
- The results artifact is named with runner.os again, as on main. The
  comment workflow still downloads asv-benchmark-results-Linux.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
load_weights keeps every sample, so the median costs nothing over the mean
and ignores a single cold-numba outlier once there are three or more, as in
a local results dir with history. CI restores two samples per benchmark
(base and PR), where the two agree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Skip machine.json by its missing "results", as _merge does, and index
result_columns without a fallback: every v2 results file (asv >= 0.5) has
a duration column, and asv itself won't read a v1 file without asv update.
The median doesn't depend on order, so the glob needn't be sorted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
benchmark_classes maps each class (or module, for a bare function) to its
(benchmark name, cost) pairs, and class_costs to their total, in place of
classes, class_cost and owner. Plans are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

@Sevans711 The memory issues are still on main, you can see them crop up in other PRs occasionally. They've proven fairly difficult to nail down, so I'd defer that to another PR.

@Sevans711 Sevans711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The memory issues are still on main, you can see them crop up in other PRs occasionally. They've proven fairly difficult to nail down, so I'd defer that to another PR.

True, I can see some of them on other PRs occasionally. Though, not nearly as many or as consistently as they seem to occur here.

Happy to approve this due to the huge ASV suite speedup, with the understanding that this would be investigated further later, but that it comes with the cost of seeing these memory tracking issues more often for now. I don't think this problem will get lost or forgotten, but if you have a few minutes to open up another issue report to track it, before this PR merges, that would be great!

@cmdupuis3
cmdupuis3 merged commit 6d6c6da into UXARRAY:main Oct 14, 2026
22 checks passed
@Sevans711

Copy link
Copy Markdown
Collaborator

it comes with the cost of seeing these memory tracking issues more often for now. I don't think this problem will get lost or forgotten, but if you have a few minutes to open up another issue report to track it, before this PR merges, that would be great!

This issue is now being tracked here: #1891

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmarking Related to benchmarks, memory usage, and/or time profiling run-benchmark Run ASV benchmark workflow

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parallelized benchmark jobs

3 participants