Skip to content

(Checking how ASV benchmarks look without any code changes) - #1710

Draft
Sevans711 wants to merge 11 commits into
mainfrom
sevans/asv-benchmarks-baseline
Draft

Sevans711 wants to merge 11 commits into
mainfrom
sevans/asv-benchmarks-baseline

Conversation

@Sevans711

Copy link
Copy Markdown
Collaborator

Not intended to be merged to main, just trying to check how ASV benchmarks look currently when the code is unchanged.

There's probably a better solution than having a draft PR for this, but this is the simplest/fastest way I could think of right now. I already attempted but struggled to generate ASV benchmark runs on my local machine with the same output format as what the run-benchmark label produces via github CI.

I originally created this with the intention of determining whether the reported benchmark performance degradation in #1705 is actually caused by the code changes there or not. Maybe this should be closed soon after that check, or maybe it should be open for a longer-term basis to repeat checks like this in the future? For example, it might have been useful to have something like this available for initially discovering #1605.

@Sevans711 Sevans711 added the run-benchmark Run ASV benchmark workflow label Aug 26, 2026
@cmdupuis3

Copy link
Copy Markdown
Collaborator

I did this too, actually. But an easier way to get the benchmarks running is to push your branch, then run ASV benchmarks directly in the Actions tab. That way you don't have to mess with a PR to get them running.

@github-actions

github-actions Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved:

Change Before [db33c50] After [a60d20f] Ratio Benchmark (Parameter)
- 677k 602k 0.89 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'grid_chunks')

Benchmarks that have stayed the same:

Change Before [db33c50] After [a60d20f] Ratio Benchmark (Parameter)
977±9μs 966±30μs 0.99 connectivity.Connectivity.time_edge_face('120km')
468±10μs 487±10μs 1.04 connectivity.Connectivity.time_edge_face('480km')
6.41±0.3ms 6.12±0.07ms 0.95 connectivity.Connectivity.time_edge_node('120km')
1.32±0.01ms 1.26±0.04ms 0.96 connectivity.Connectivity.time_edge_node('480km')
82.1±5μs 74.1±20μs ~0.90 connectivity.Connectivity.time_face_edge('120km')
72.9±10μs 71.9±8μs 0.99 connectivity.Connectivity.time_face_edge('480km')
1.01±0.02ms 986±30μs 0.98 connectivity.Connectivity.time_face_face('120km')
412±20μs 404±20μs 0.98 connectivity.Connectivity.time_face_face('480km')
64.8±20μs 70.6±20μs 1.09 connectivity.Connectivity.time_face_node('120km')
65.8±6μs 68.8±20μs 1.05 connectivity.Connectivity.time_face_node('480km')
484±20μs 481±30μs 0.99 connectivity.Connectivity.time_n_nodes_per_face('120km')
427±10μs 421±10μs 0.99 connectivity.Connectivity.time_n_nodes_per_face('480km')
1.38±0.02ms 1.37±0.04ms 0.99 connectivity.Connectivity.time_node_edge('120km')
497±20μs 479±20μs 0.96 connectivity.Connectivity.time_node_edge('480km')
86.0±3ms 78.3±3ms 0.91 connectivity.Connectivity.time_node_face('120km')
5.12±0.2ms 5.18±0.05ms 1.01 connectivity.Connectivity.time_node_face('480km')
1.42M 1.42M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('120km')
106k 106k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('480km')
6.48M 6.48M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('120km')
419k 419k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('480km')
2.42k 2.42k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('120km')
2.42k 2.42k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('480km')
1.6M 1.6M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('120km')
101k 101k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('480km')
2.54k 2.54k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('120km')
2.54k 2.54k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('480km')
241k 241k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('120km')
26.8k 26.8k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('480km')
1.9M 1.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('120km')
127k 127k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('480km')
11.9M 11.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('120km')
747k 747k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('480km')
8.53±1ms 8.54±0.08ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
2.76±0.06ms 2.77±0.04ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
10.3±0.2ms 10.3±9ms 1.01 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
1.55±0.04ms 1.55±0.01ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
57.3k 57.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
12.3k 12.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
123k 123k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
128 128 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.27M 1.27M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
50.1k 50.1k 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.48M 1.48M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
712 712 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.85M 1.85M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
1.85M 1.85M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
2.01M 2.01M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
35.6k 35.6k 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
794±40ns 834±50ns 1.05 geometry_kernels.AccucrossKernels.time_accucross
1.74±0.05μs 1.74±0.06μs 1.00 geometry_kernels.AccucrossKernels.time_accucross_pair
246±5ns 251±9ns 1.02 geometry_kernels.EFTPrimitives.time_acc_sqrt_re
235±5ns 231±4ns 0.98 geometry_kernels.EFTPrimitives.time_diff_of_products
220±9ns 210±5ns 0.95 geometry_kernels.EFTPrimitives.time_two_prod
216±9ns 266±40ns ~1.23 geometry_kernels.EFTPrimitives.time_two_sum
351±4ns 345±10ns 0.98 geometry_kernels.GCAConstLatIntersection.time_accux_constlat_kernel
451±9ns 451±4ns 1.00 geometry_kernels.GCAConstLatIntersection.time_gca_const_lat_intersection
451±10ns 546±100ns ~1.21 geometry_kernels.GCAConstLatIntersection.time_try_gca_const_lat_intersection
821±10ns 796±40ns 0.97 geometry_kernels.GCAGCAIntersection.time_accux_gca_kernel
982±60ns 892±20ns ~0.91 geometry_kernels.GCAGCAIntersection.time_gca_gca_intersection
1.08±0.02μs 1.05±0.02μs 0.97 geometry_kernels.GCAGCAIntersection.time_try_gca_gca_intersection
22.6±0.6μs 22.8±0.3μs 1.01 geometry_kernels.OrientPredicates.time_on_minor_arc
22.2±0.5μs 22.8±0.2μs 1.03 geometry_kernels.OrientPredicates.time_orient3d_on_sphere
3.05±0ms 3.06±0ms 1.00 geometry_samebody.SameBodyConstLat.time_accux_dispatch
1.16±0ms 1.17±0ms 1.00 geometry_samebody.SameBodyConstLat.time_accux_kernel
2.29±0ms 2.30±0.02ms 1.00 geometry_samebody.SameBodyConstLat.time_fp64_dispatch
148±0.7μs 147±0.6μs 1.00 geometry_samebody.SameBodyConstLat.time_fp64_kernel
40.8±0.02ms 40.9±0.02ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_dispatch
3.81±0.01ms 3.80±0.01ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_kernel
37.8±0.01ms 37.8±0.03ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_dispatch
484±20μs 468±5μs 0.97 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_kernel
920±7ms 945±20ms 1.03 import.Imports.timeraw_import_uxarray
1 1 1.00 lazy_grid_construction.LazyGridConstruction.track_computes_isel
2 2 1.00 lazy_grid_construction.LazyGridConstruction.track_computes_open_grid_chunked
101±0.5ms 98.4±2ms 0.97 lazy_grid_construction.OpenGridChunked.time_open_grid('120km')
140±2ms 133±3ms 0.95 lazy_grid_construction.OpenGridChunked.time_open_grid('480km')
2.01M 2.11M 1.05 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('120km')
2.29M 2.29M 1.00 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('480km')
2.51±0.04ms 2.51±0.03ms 1.00 mpas_ocean.CheckNorm.time_check_norm('120km')
2.00±0.3ms 2.04±0.05ms 1.02 mpas_ocean.CheckNorm.time_check_norm('480km')
544±10μs 550±8μs 1.01 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
296±2μs 299±4μs 1.01 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
377±4μs 376±5μs 1.00 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
326±3μs 332±5μs 1.02 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
5.33±0.06ms 5.31±0.02ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
3.86±0.03ms 3.92±0.07ms 1.01 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
98.5±0.4ms 96.9±0.6ms 0.98 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
9.55±0.06ms 9.86±0.4ms 1.03 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
9.53±0.03ms 9.49±0.04ms 1.00 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
626±30μs 613±20μs 0.98 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
7.03±0.03ms 7.00±0.02ms 1.00 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
474±10μs 469±10μs 0.99 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
469±30ms 455±20ms 0.97 mpas_ocean.CrossSections.time_const_lat('120km', 1)
222±1ms 231±9ms 1.04 mpas_ocean.CrossSections.time_const_lat('120km', 2)
114±0.5ms 114±2ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 4)
384±2ms 382±0.5ms 0.99 mpas_ocean.CrossSections.time_const_lat('480km', 1)
193±1ms 194±2ms 1.01 mpas_ocean.CrossSections.time_const_lat('480km', 2)
106±7ms 104±7ms 0.98 mpas_ocean.CrossSections.time_const_lat('480km', 4)
26.5±0.2ms 26.6±0.06ms 1.00 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
3.08±0.03ms 3.20±0.09ms 1.04 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
9.76±0.5ms 13.2±0.7ms ~1.36 mpas_ocean.FaceAreas.time_face_areas('120km')
2.80±0.09ms 3.00±0.2ms 1.07 mpas_ocean.FaceAreas.time_face_areas('480km')
229k 229k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('120km')
14.3k 14.3k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('480km')
2.12M 2.12M 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('120km')
698k 699k 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('480km')
167±3ms 161±2ms 0.96 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', False)
54.2±0.8ms 53.5±0.8ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', True)
32.3±0.2ms 31.7±0.2ms 0.98 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', False)
5.20±0.1ms 5.03±0.2ms 0.97 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', True)
8.41±0.3ms 8.60±0.3ms 1.02 mpas_ocean.Gradient.time_gradient('120km')
986±10μs 1.00±0.02ms 1.02 mpas_ocean.Gradient.time_gradient('480km')
457k 457k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('120km')
28.7k 28.7k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('480km')
3.2M 3.2M 1.00 mpas_ocean.Gradient.track_peakmem_gradient('120km')
204k 204k 1.00 mpas_ocean.Gradient.track_peakmem_gradient('480km')
170±5μs 153±3μs ~0.90 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
74.7±0.3μs 70.5±0.3μs 0.94 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
203±1μs 205±1μs 1.01 mpas_ocean.Integrate.time_integrate('120km')
186±2μs 192±1μs 1.03 mpas_ocean.Integrate.time_integrate('480km')
18.4M 18.4M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('120km')
1.2M 1.2M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('480km')
186±1ms 184±1ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
186±3ms 183±0.5ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
185±2ms 183±0.9ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
13.0±0.05ms 13.4±0.2ms 1.03 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
13.0±0.08ms 13.4±0.2ms 1.04 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
13.1±0.07ms 13.3±0.3ms 1.02 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
246±1ms 245±0.4ms 0.99 mpas_ocean.NeighborhoodBuild.time_build('120km', 1.0)
505±0.8ms 506±2ms 1.00 mpas_ocean.NeighborhoodBuild.time_build('120km', 5.0)
13.5±0.1ms 13.4±0.04ms 0.99 mpas_ocean.NeighborhoodBuild.time_build('480km', 1.0)
16.5±0.2ms 16.5±0.1ms 1.00 mpas_ocean.NeighborhoodBuild.time_build('480km', 5.0)
241±0.1ms 241±1ms 1.00 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 1.0)
503±2ms 499±2ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 5.0)
13.1±0.07ms 12.9±0.02ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 1.0)
16.0±0.06ms 16.3±0.02ms 1.02 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 5.0)
612.76 612.76 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 15.0)
74.17 74.17 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 5.0)
37.29 37.29 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 15.0)
6.57 6.57 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 5.0)
141M 141M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 15.0)
17.4M 17.4M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 5.0)
563k 563k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 15.0)
123k 123k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 5.0)
145M 145M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 15.0)
21.5M 21.5M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 5.0)
825k 825k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 15.0)
384k 384k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 5.0)
27.1±0.5ms 27.3±0.7ms 1.01 mpas_ocean.NeighborhoodDask.time_mean('120km', 'grid_chunks')
11.1±0.3ms 11.5±0.5ms 1.04 mpas_ocean.NeighborhoodDask.time_mean('120km', 'numpy')
23.5±0.6ms 25.0±1ms 1.06 mpas_ocean.NeighborhoodDask.time_mean('120km', 'time_chunks')
5.90±0.1ms 8.14±3ms ~1.38 mpas_ocean.NeighborhoodDask.time_mean('480km', 'grid_chunks')
403±5μs 409±9μs 1.01 mpas_ocean.NeighborhoodDask.time_mean('480km', 'numpy')
4.01±0.06ms 4.23±0.2ms 1.05 mpas_ocean.NeighborhoodDask.time_mean('480km', 'time_chunks')
5.9M 5.9M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'grid_chunks')
2.75M 2.75M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'numpy')
5.68M 5.68M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'time_chunks')
177k 177k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'numpy')
535k 534k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'time_chunks')
4.58±0.01s 4.53±0.01s 0.99 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'mean')
4.65±0.01s 4.64±0.01s 1.00 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'median')
122±0.8ms 123±0.2ms 1.01 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'mean')
124±1ms 123±0.7ms 0.99 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'median')
511±3ms 510±4ms 1.00 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'mean')
545±3ms 545±3ms 1.00 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'median')
16.8±0.2ms 16.7±0.05ms 1.00 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'mean')
17.2±0.06ms 17.2±0.07ms 1.00 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'median')
4.58±0.01ms 4.51±0.01ms 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'mean')
37.5±0.2ms 37.7±0.08ms 1.01 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'median')
241±10μs 245±10μs 1.02 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'mean')
541±9μs 542±10μs 1.00 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'median')
234k 234k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'mean')
235k 235k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'median')
19.5k 19.5k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'mean')
19.5k 19.5k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'median')
380±40μs 401±30μs 1.05 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
419±40μs 390±10μs 0.93 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
325±40μs 354±20μs 1.09 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
332±30μs 325±30μs 0.98 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
110±0.8ms 109±1ms 0.99 mpas_ocean.RemapDownsample.time_bilinear_remapping
17.2±0.08ms 17.7±0.2ms 1.03 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
15.8±0.2ms 15.6±0.2ms 0.99 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
1.12±0.02s 1.12±0.01s 1.01 mpas_ocean.RemapUpsample.time_bilinear_remapping
27.1±0.2ms 26.5±0.4ms 0.98 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
12.1±0.08ms 11.6±0.05ms 0.95 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
9.12±0.2ms 8.58±0.3ms 0.94 mpas_ocean.ZonalAverage.time_zonal_average('120km')
5.35±0.2ms 5.44±0.1ms 1.02 mpas_ocean.ZonalAverage.time_zonal_average('480km')
1.0378378847782574 1.0237262427297702 0.99 nogil_scaling.GILScaling.track_gil_scaling
226±2ms 230±4ms 1.01 plots.PlotHealpix.time_default_plot(6)
2.31±0.01s 2.33±0.07s 1.01 plots.PlotHealpix.time_default_plot(8)
28M 28M 1.00 plots.PlotHealpix.track_peakmem_default_plot(6)
443M 443M 1.00 plots.PlotHealpix.track_peakmem_default_plot(8)
4.40±0.1ms 4.44±0.1ms 1.01 quad_hexagon.QuadHexagon.time_open_dataset
3.55±0.03ms 3.62±0.1ms 1.02 quad_hexagon.QuadHexagon.time_open_grid
408 408 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_dataset
392 392 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_grid
73.5k 73.1k 0.99 quad_hexagon.QuadHexagon.track_peakmem_open_dataset
72.7k 72.4k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_grid
1.11±0s 1.12±0s 1.00 to_raster.ToRaster.time_to_raster((10000.0, 10.0))
2.74±0.03s 2.71±0.02s 0.99 to_raster.ToRaster.time_to_raster((10000.0, 100.0))
3.13±0.02s 3.09±0.03s 0.99 to_raster.ToRaster.time_to_raster((100000.0, 100.0))
271M 271M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 10.0))
684M 684M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 100.0))
768M 768M 1.00 to_raster.ToRaster.track_peakmem((100000.0, 100.0))

Benchmarks that have got worse:

Change Before [db33c50] After [a60d20f] Ratio Benchmark (Parameter)
+ 322M 413M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
+ 323M 413M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
+ 324M 413M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
+ 323M 413M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
+ 298M 410M 1.37 import.Imports.track_peakmem_import_uxarray
+ 342M 413M 1.21 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 1)
+ 342M 413M 1.21 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 2)
+ 342M 413M 1.21 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 4)
+ 326M 413M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 1)
+ 326M 413M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 2)
+ 325M 413M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 4)
+ 338M 409M 1.21 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('120km')
+ 317M 409M 1.29 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('480km')
+ 342M 409M 1.19 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('120km')
+ 325M 409M 1.26 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('480km')

@Sevans711

Copy link
Copy Markdown
Collaborator Author

I did this too, actually. But an easier way to get the benchmarks running is to push your branch, then run ASV benchmarks directly in the Actions tab. That way you don't have to mess with a PR to get them running.

Ah, that makes sense! Does that lead to a nice comment with the benchmark results too? I suppose even if there isn't a comment it's definitely less noisy than making a draft PR

@Sevans711 Sevans711 mentioned this pull request Aug 26, 2026
7 tasks done
@cmdupuis3

Copy link
Copy Markdown
Collaborator

Nah, you'd have to dig. I've been mostly looking at total time though, so for me it doesn't matter.

@dylannelson

Copy link
Copy Markdown
Member

Leaving a note here in case it gets solved and disappears in the above benchmark run, we have been seeing a *1000 or so speedup on a test even when nothing changes, it appears as "no change". Very odd:

image

This appears in this PR and others as well (at the time of writing).

@Sevans711

Sevans711 commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator Author

@cmdupuis3 I've been noticing that the "import uxarray" benchmark is always showing an improvement, and the mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'grid_chunks') is always showing things getting worse. Do you have any guesses about what is causing these quirks?

EDIT: Ah, I agree now I was wrong about the latter… I felt like I've been seeing NeighborhoodDask benchmarks in that category in a few different PRs, but now I can see that it isn't actually always showing up as "got worse", only sometimes!

@cmdupuis3

Copy link
Copy Markdown
Collaborator

@Sevans711 If it's around 300M on a memory benchmark, that usually indicates caching behavior. I think the pure import is pretty indicative. It can be hard to measure consistently under caching conditions, so for the other benchmarks, maybe they aren't rigorous enough.

The other one seems like a one-off, you can just compare it with previous benchmark runs and if there's no pattern, I wouldn't worry about it much.

@Sevans711 Sevans711 mentioned this pull request Oct 12, 2026
8 tasks done
@dylannelson

dylannelson commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Because of the large number of test that consistently show up in the "worse" tab, it's a bit hard to read through the benchmarks. Seems like the peakmem benchmarks and a few others still consistently appear worse across this PR and a few others, which can be a bit confusing when looking at benchmarking PR's like the one below:

Another PR on benchmarking
image

The work I'm doing with grid cleaning
image

Here:
image

Since ~15 tests are giving us likely wrong results, should we think about removing them so we can actually see what the real slowdowns are? A bit hard to read in it's current state

@Sevans711

Copy link
Copy Markdown
Collaborator Author

Because of the large number of test that consistently show up in the "worse" tab, it's a bit hard to read through the benchmarks. Seems like the peakmem benchmarks and a few others still consistently appear worse across this PR and a few others, which can be a bit confusing when looking at benchmarking PR's like the one below

I agree, it would be nice to get this fixed. Would you be willing to open an issue report for it, so that it ends up going onto the iteration boards at some point? (Sidenote: I originally commented about this during the PR which introduced it, see #1813 (review).)

My preferred solution would be to fix the benchmarking suite instead of to remove the tests altogether; anecdotally I recall that in at least one of my PRs since #1813 merged, I would have accidentally introduced a memory usage degradation if not for these tests. Fixing the underlying problem would be great, though if that is too difficult, maybe a temporary workaround could be to increase the "noise" threshold ASV uses to decide whether to report something as a degradation, for these benchmarks in particular.

@Sevans711 Sevans711 added the benchmarking Related to benchmarks, memory usage, and/or time profiling label Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmarking Related to benchmarks, memory usage, and/or time profiling run-benchmark Run ASV benchmark workflow

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants