Skip to content

Optimize azimuthal_mean with one sparse ring operator - #1847

Open
cmdupuis3 wants to merge 17 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/lazy-azimuthal
Open

cmdupuis3 wants to merge 17 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/lazy-azimuthal

Conversation

@cmdupuis3

@cmdupuis3 cmdupuis3 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1848

Overview

Speedup for azimuthal means (~100x on the oQU benchmark), lazy dask support, and an incidental float32 accuracy fix.


azimuthal_mean looped over the radii: each ring ran its own kd-tree query, built a subgrid with isel (without ignore_grid), recomputed face_areas on it, and computed its weighted_mean eagerly. Now:

  • Ring membership: one kd-tree query, padded by 1e-9 so the tree's rounding can't drop faces on the outer circle, then each face is binned by its exact squared chord distance (r_(i-1) < d <= r_i). Hit counts match main on the test and tutorial meshes; they can only differ for a face center within rounding of a ring boundary.
  • Means: the rings are disjoint, so every ring mean is one row of a single sparse (n_radius, n_ring_faces) operator weighted by the parent grid's face areas. No subgrids are built.
  • Dask: dask-backed input now stays lazy (main returned NumPy). Each input n_face chunk that holds ring faces is reduced to partial ring sums, and the partial sums are added up in a tree. No task gathers every face inside outer_radius, so peak memory per task is about one input chunk, whatever the radius. Chunks without ring faces aren't read, and the other dims keep their chunks.
  • Lazily loaded data without dask: read as contiguous slices covering at most twice the ring faces, rather than by fancy indexing, which netCDF4 serves face by face (6,681 ring faces x 24 times on oQU120: 459 ms fancy-indexed vs 3.3 ms in 38 slices).
  • In-memory data: the ring faces are taken with one fancy index, which is cheaper than slicing when nothing has to be read from a file.

Output dims, coords, attrs, name and NaN handling (NaN faces add nothing to the sum but still count toward the ring area) are unchanged, and UxDataset.azimuthal_mean gets the same speedup since it calls this per variable. Accumulation is now float64: on the float32 dyamond test grid, main's ring means are off by up to ~1.6e-7 relative against a float64 reference, while the new ones are within ~3e-16. On float64 data the two agree to ~1e-15 relative. Chunked and in-memory inputs add the same terms in a different order, so they can differ by rounding.

Adds AzimuthalAverage to the oQU benchmarks (bottomDepth, 61 rings out to 60 degrees around (-30, 0), kd-tree and face areas pre-cached). Locally: oQU480 175 ms -> 1.1 ms, oQU120 292 ms -> 3.3 ms.

Docs: the docstring describes how dask-backed and file-backed data are read, and the azimuthal-average user guide gains a chunked example plus notes on lazy execution, estimating the memory of a task, and numerical/boundary differences.

New tests:

  • test_ring_membership checks the rings against brute force, including faces exactly on ring edges and radii past 180 degrees.
  • test_file_backed_reads_are_bounded counts reads through an xarray BackendArray and checks that at most twice the selected faces are read.
  • test_dask_holds_one_input_chunk_at_a_time records the size of every intermediate dask result and checks that none is larger than one input chunk.

The old tests can be seen with these parameter combos:

Old test New test and parameters
test_gaussian test_monotone_profile[gaussian--1]: name="gaussian", sign=-1
test_inverse_gaussian test_monotone_profile[inverse_gaussian-1]: name="inverse_gaussian", sign=1
test_non_zero_hit_counts test_matches_reference[center0-None] and [center0-chunks1]: center=(45, 0), chunks is None or {"n_face": 50, "time": 1}
test_zero_hit_counts test_matches_reference[center1-None] and [center1-chunks1]: center=(-45, 0) (outside the subset), same two chunks values

PR Checklist

General

  • An issue is created and linked
  • Added appropriate labels (if your uxarray repo permissions allow it)
  • Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

  • There is adequate test coverage of changes from this PR (add new tests if needed)
  • If this PR could affect performance, ran ASV benchmarks and confirmed they show expected behavior (add a new benchmark if necessary)

Documentation and Examples

  • Docstrings updated with any function changes, and included in all new functions
  • User (public) functions added to docs/api.rst; internal (private) function names start with an underscore (_)

AI Disclosure

AI Usage: Claude Opus 5.5

  • I have tested and take responsibility for all AI-generated content in my PR.

azimuthal_mean built a full subgrid for every ring (isel without
ignore_grid), recomputed face_areas on it, and forced a compute per ring.
The rings are disjoint, so the per-ring area-weighted means now form one
sparse (n_radius, n_ring_faces) operator over the ring faces, gathered
once with ignore_grid=True and weighted by the parent grid's areas. It is
applied through a single apply_ufunc, so dask-backed input stays lazy.

Ring membership, hit counts, output dims/coords/attrs and NaN handling
(NaN faces add nothing to the sum but still count toward the ring area)
are unchanged. Accumulation is now float64: on float32 grids the old
float32 sums were off by up to ~3e-7 relative against a float64
reference; the new result matches it to ~3e-16.

Adds AzimuthalAverage to the oQU benchmarks (21 rings, out to 20 degrees):
480km 43.9 ms -> 0.7 ms, 120km 50.0 ms -> 1.2 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cmdupuis3 cmdupuis3 added scalability Related to scalability & performance efforts run-benchmark Run ASV benchmark workflow accuracy Related to mathematical accuracy of operations, such as numerical precision or rounding errors labels Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved:

Change Before [ce8690d] After [b20bba8] Ratio Benchmark (Parameter)
- 457±8ms 2.46±0.2ms 0.01 mpas_ocean.AzimuthalAverage.time_azimuthal_average('120km')
- 408±3ms 1.01±0.01ms 0 mpas_ocean.AzimuthalAverage.time_azimuthal_average('480km')
- 676±60μs 591±4μs 0.87 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
- 515±10μs 389±7μs 0.76 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
- 10.0±0.1ms 8.46±0.05ms 0.85 mpas_ocean.Gradient.time_gradient('120km')

Benchmarks that have stayed the same:

Change Before [ce8690d] After [b20bba8] Ratio Benchmark (Parameter)
952±20μs 944±7μs 0.99 connectivity.Connectivity.time_edge_face('120km')
439±10μs 455±10μs 1.04 connectivity.Connectivity.time_edge_face('480km')
6.11±0.07ms 6.22±0.1ms 1.02 connectivity.Connectivity.time_edge_node('120km')
1.20±0.05ms 1.22±0.02ms 1.02 connectivity.Connectivity.time_edge_node('480km')
66.8±10μs 67.3±10μs 1.01 connectivity.Connectivity.time_face_edge('120km')
60.0±6μs 61.7±4μs 1.03 connectivity.Connectivity.time_face_edge('480km')
952±20μs 945±40μs 0.99 connectivity.Connectivity.time_face_face('120km')
389±20μs 387±20μs 1 connectivity.Connectivity.time_face_face('480km')
64.0±3μs 62.0±4μs 0.97 connectivity.Connectivity.time_face_node('120km')
58.2±3μs 56.5±5μs 0.97 connectivity.Connectivity.time_face_node('480km')
531±20μs 505±20μs 0.95 connectivity.Connectivity.time_n_nodes_per_face('120km')
381±10μs 387±30μs 1.02 connectivity.Connectivity.time_n_nodes_per_face('480km')
1.35±0.02ms 1.30±0.06ms 0.96 connectivity.Connectivity.time_node_edge('120km')
469±10μs 466±9μs 0.99 connectivity.Connectivity.time_node_edge('480km')
89.5±0.8ms 87.5±0.7ms 0.98 connectivity.Connectivity.time_node_face('120km')
5.54±0.1ms 5.67±0.2ms 1.02 connectivity.Connectivity.time_node_face('480km')
1.42M 1.42M 1 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('120km')
105k 105k 1 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('480km')
6.48M 6.48M 1 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('120km')
419k 419k 1 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('480km')
2.05k 2.05k 1 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('120km')
2.05k 2.05k 1 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('480km')
1.6M 1.6M 1 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('120km')
101k 101k 1 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('480km')
2.18k 2.18k 1 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('120km')
2.18k 2.18k 1 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('480km')
240k 240k 1 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('120km')
26k 26k 1 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('480km')
1.9M 1.9M 1 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('120km')
127k 127k 1 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('480km')
11.9M 11.9M 1 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('120km')
747k 747k 1 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('480km')
9.18±0.2ms 8.89±0.1ms 0.97 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
2.56±0.04ms 2.55±0.04ms 1 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
11.2±0.3ms 10.8±7ms 0.97 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
1.35±0.04ms 1.32±0.05ms 0.98 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
57.3k 57.3k 1 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
12.3k 12.3k 1 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
123k 123k 1 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
128 128 1 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.27M 1.27M 1 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
50.1k 50.1k 1 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.48M 1.48M 1 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
712 712 1 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.73M 1.73M 1 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
1.71M 1.71M 1 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.87M 1.87M 1 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
35.3k 35.3k 1 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
781±30ns 720±30ns 0.92 geometry_kernels.AccucrossKernels.time_accucross
1.65±0.02μs 1.66±0.04μs 1 geometry_kernels.AccucrossKernels.time_accucross_pair
271±10ns 261±10ns 0.96 geometry_kernels.EFTPrimitives.time_acc_sqrt_re
260±9ns 251±6ns 0.96 geometry_kernels.EFTPrimitives.time_diff_of_products
221±4ns 220±7ns 1 geometry_kernels.EFTPrimitives.time_two_prod
220±0.5ns 225±10ns 1.02 geometry_kernels.EFTPrimitives.time_two_sum
371±4ns 361±0.4ns 0.97 geometry_kernels.GCAConstLatIntersection.time_accux_constlat_kernel
465±9ns 461±8ns 0.99 geometry_kernels.GCAConstLatIntersection.time_gca_const_lat_intersection
470±8ns 461±9ns 0.98 geometry_kernels.GCAConstLatIntersection.time_try_gca_const_lat_intersection
791±30ns 812±10ns 1.03 geometry_kernels.GCAGCAIntersection.time_accux_gca_kernel
932±50ns 927±40ns 0.99 geometry_kernels.GCAGCAIntersection.time_gca_gca_intersection
1.08±0.03μs 1.07±0.03μs 0.99 geometry_kernels.GCAGCAIntersection.time_try_gca_gca_intersection
21.4±0.6μs 21.4±0.2μs 1 geometry_kernels.OrientPredicates.time_on_minor_arc
21.3±0.5μs 20.8±0.1μs 0.98 geometry_kernels.OrientPredicates.time_orient3d_on_sphere
3.04±0ms 3.07±0ms 1.01 geometry_samebody.SameBodyConstLat.time_accux_dispatch
1.16±0ms 1.16±0ms 1 geometry_samebody.SameBodyConstLat.time_accux_kernel
2.28±0ms 2.29±0.01ms 1 geometry_samebody.SameBodyConstLat.time_fp64_dispatch
148±0.3μs 148±0.1μs 1 geometry_samebody.SameBodyConstLat.time_fp64_kernel
41.8±0.2ms 42.0±0.04ms 1 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_dispatch
3.87±0.02ms 3.91±0.01ms 1.01 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_kernel
38.6±0.4ms 38.9±0.03ms 1.01 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_dispatch
480±30μs 470±10μs 0.98 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_kernel
826±4ms 807±7ms 0.98 import.Imports.timeraw_import_uxarray
1 1 1 lazy_grid_construction.LazyGridConstruction.track_computes_isel
2 2 1 lazy_grid_construction.LazyGridConstruction.track_computes_open_grid_chunked
98.3±0.6ms 99.4±3ms 1.01 lazy_grid_construction.OpenGridChunked.time_open_grid('120km')
132±2ms 136±1ms 1.03 lazy_grid_construction.OpenGridChunked.time_open_grid('480km')
1.99M 2.01M 1.01 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('120km')
2.23M 2.11M 0.94 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('480km')
2.15±0.1ms 2.06±0.03ms 0.96 mpas_ocean.CheckNorm.time_check_norm('120km')
1.27±0.09ms 1.18±0.01ms 0.93 mpas_ocean.CheckNorm.time_check_norm('480km')
287±2μs 292±6μs 1.02 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
327±8μs 310±5μs 0.95 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
4.48±0.03ms 4.40±0.05ms 0.98 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
3.12±0.1ms 3.09±0.04ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
112±2ms 110±1ms 0.98 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
9.86±0.2ms 9.96±0.2ms 1.01 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
9.31±0.03ms 9.33±0.04ms 1 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
617±30μs 581±30μs 0.94 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
6.80±0.04ms 6.86±0.06ms 1.01 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
417±9μs 412±20μs 0.99 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
366±8ms 352±8ms 0.96 mpas_ocean.CrossSections.time_const_lat('120km', 1)
189±3ms 179±2ms 0.95 mpas_ocean.CrossSections.time_const_lat('120km', 2)
98.8±0.7ms 92.2±0.9ms 0.93 mpas_ocean.CrossSections.time_const_lat('120km', 4)
302±3ms 301±2ms 1 mpas_ocean.CrossSections.time_const_lat('480km', 1)
155±1ms 149±2ms 0.96 mpas_ocean.CrossSections.time_const_lat('480km', 2)
78.2±0.4ms 76.7±2ms 0.98 mpas_ocean.CrossSections.time_const_lat('480km', 4)
28.6±0.07ms 28.8±0.1ms 1.01 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
3.44±0.4ms 3.49±0.01ms 1.01 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
7.54±0.2ms 7.59±0.5ms 1.01 mpas_ocean.FaceAreas.time_face_areas('120km')
2.43±0.07ms 2.64±0.2ms 1.08 mpas_ocean.FaceAreas.time_face_areas('480km')
229k 229k 1 mpas_ocean.FaceAreas.track_nbytes_face_areas('120km')
14.3k 14.3k 1 mpas_ocean.FaceAreas.track_nbytes_face_areas('480km')
2.12M 2.12M 1 mpas_ocean.FaceAreas.track_peakmem_face_areas('120km')
679k 679k 1 mpas_ocean.FaceAreas.track_peakmem_face_areas('480km')
32.8±0.5ms 33.7±0.5ms 1.03 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', 'exclude')
96.9±2ms 98.0±0.9ms 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', 'split')
3.23±0.2ms 3.37±0.2ms 1.04 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', 'exclude')
17.6±0.1ms 18.8±0.6ms 1.07 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', 'split')
1.44±0.01ms 1.40±0.02ms 0.97 mpas_ocean.Gradient.time_gradient('480km')
457k 457k 1 mpas_ocean.Gradient.track_nbytes_gradient('120km')
28.7k 28.7k 1 mpas_ocean.Gradient.track_nbytes_gradient('480km')
3.44M 3.44M 1 mpas_ocean.Gradient.track_peakmem_gradient('120km')
226k 226k 1 mpas_ocean.Gradient.track_peakmem_gradient('480km')
389±10μs 390±6μs 1 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
156±7μs 150±5μs 0.96 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
191±4μs 203±9μs 1.06 mpas_ocean.Integrate.time_integrate('120km')
162±0.7μs 165±8μs 1.02 mpas_ocean.Integrate.time_integrate('480km')
18.4M 18.4M 1 mpas_ocean.Integrate.track_nbytes_integrate('120km')
1.2M 1.2M 1 mpas_ocean.Integrate.track_nbytes_integrate('480km')
139±1ms 139±1ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
137±0.8ms 138±0.5ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
138±0.8ms 138±1ms 1 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
9.65±0.04ms 9.68±0.04ms 1 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
9.89±0.09ms 10.0±0.08ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
9.66±0.02ms 10.1±0.2ms 1.05 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
149±3ms 150±0.7ms 1 mpas_ocean.NeighborhoodBuild.time_build('120km', 1.0)
335±5ms 332±2ms 0.99 mpas_ocean.NeighborhoodBuild.time_build('120km', 5.0)
9.86±0.3ms 9.90±0.09ms 1 mpas_ocean.NeighborhoodBuild.time_build('480km', 1.0)
12.5±0.2ms 12.1±0.2ms 0.97 mpas_ocean.NeighborhoodBuild.time_build('480km', 5.0)
148±1ms 144±2ms 0.97 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 1.0)
333±3ms 325±2ms 0.98 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 5.0)
9.70±0.1ms 9.66±0.1ms 1 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 1.0)
12.2±0.2ms 11.8±0.3ms 0.96 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 5.0)
612.76 612.76 1 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 15.0)
74.17 74.17 1 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 5.0)
37.29 37.29 1 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 15.0)
6.57 6.57 1 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 5.0)
141M 141M 1 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 15.0)
17.4M 17.4M 1 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 5.0)
563k 563k 1 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 15.0)
123k 123k 1 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 5.0)
145M 145M 1 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 15.0)
21.5M 21.5M 1 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 5.0)
824k 824k 1 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 15.0)
384k 384k 1 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 5.0)
27.5±0.5ms 26.3±0.1ms 0.96 mpas_ocean.NeighborhoodDask.time_mean('120km', 'grid_chunks')
11.6±0.1ms 11.1±0.04ms 0.96 mpas_ocean.NeighborhoodDask.time_mean('120km', 'numpy')
24.5±0.2ms 24.0±0.5ms 0.98 mpas_ocean.NeighborhoodDask.time_mean('120km', 'time_chunks')
6.13±0.1ms 5.73±0.05ms 0.94 mpas_ocean.NeighborhoodDask.time_mean('480km', 'grid_chunks')
335±5μs 329±5μs 0.98 mpas_ocean.NeighborhoodDask.time_mean('480km', 'numpy')
4.26±0.06ms 3.93±0.05ms 0.92 mpas_ocean.NeighborhoodDask.time_mean('480km', 'time_chunks')
5.82M 5.82M 1 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'grid_chunks')
2.75M 2.75M 1 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'numpy')
5.66M 5.67M 1 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'time_chunks')
676k 669k 0.99 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'grid_chunks')
177k 177k 1 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'numpy')
521k 528k 1.01 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'time_chunks')
3.15±0.01s 3.07±0.02s 0.97 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'mean')
3.30±0.01s 3.25±0.03s 0.99 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'median')
90.5±0.2ms 92.2±3ms 1.02 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'mean')
92.9±0.6ms 90.7±0.3ms 0.98 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'median')
343±1ms 336±1ms 0.98 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'mean')
382±2ms 371±2ms 0.97 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'median')
12.8±0.03ms 12.5±0.3ms 0.97 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'mean')
13.5±0.04ms 12.6±0.2ms 0.93 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'median')
3.79±0ms 3.75±0.02ms 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'mean')
38.3±0.09ms 37.4±0.6ms 0.98 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'median')
198±3μs 185±3μs 0.93 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'mean')
542±5μs 534±20μs 0.98 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'median')
234k 234k 1 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'mean')
235k 235k 1 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'median')
19.5k 19.5k 1 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'mean')
19.6k 19.6k 1 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'median')
384±9μs 362±8μs 0.94 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
352±4μs 348±10μs 0.99 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
351±6μs 349±6μs 0.99 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
327±6μs 312±10μs 0.95 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
97.5±0.4ms 94.3±1ms 0.97 mpas_ocean.RemapDownsample.time_bilinear_remapping
15.8±0.2ms 15.2±0.4ms 0.96 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
14.2±0.1ms 13.7±0.2ms 0.97 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
1.07±0s 1.09±0.01s 1.02 mpas_ocean.RemapUpsample.time_bilinear_remapping
23.4±0.1ms 23.9±0.3ms 1.02 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
11.0±0.1ms 11.2±0.2ms 1.02 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
6.29±0.3ms 6.31±0.1ms 1 mpas_ocean.ZonalAverage.time_zonal_average('120km')
3.20±0.2ms 3.27±0.2ms 1.02 mpas_ocean.ZonalAverage.time_zonal_average('480km')
0.9988628196804635 0.9919406236780645 0.99 nogil_scaling.GILScaling.track_gil_scaling
1.27±0.02ms 1.33±0.03ms 1.05 open_dataset.OpenDataset.time_open_dataset('dataset', 'dataset')
2.22±0.01ms 2.33±0.02ms 1.05 open_dataset.OpenDataset.time_open_dataset('dataset', 'file')
5.02±0.03ms 5.08±0.03ms 1.01 open_dataset.OpenDataset.time_open_dataset('file', 'dataset')
5.97±0.04ms 6.09±0.1ms 1.02 open_dataset.OpenDataset.time_open_dataset('file', 'file')
172±1μs 176±3μs 1.02 open_dataset.OpenDataset.time_open_dataset('grid', 'dataset')
1.11±0ms 1.19±0.02ms 1.07 open_dataset.OpenDataset.time_open_dataset('grid', 'file')
1.27±0.02s 1.30±0.03s 1.02 plots.PlotContour.time_contour_projected
225±3ms 228±4ms 1.01 plots.PlotHealpix.time_default_plot(6)
2.23±0.06s 2.23±0.02s 1 plots.PlotHealpix.time_default_plot(8)
28M 28M 1 plots.PlotHealpix.track_peakmem_default_plot(6)
443M 443M 1 plots.PlotHealpix.track_peakmem_default_plot(8)
4.77±0.1ms 4.57±0.09ms 0.96 quad_hexagon.QuadHexagon.time_open_dataset
3.93±0.03ms 3.88±0.1ms 0.99 quad_hexagon.QuadHexagon.time_open_grid
408 408 1 quad_hexagon.QuadHexagon.track_nbytes_open_dataset
392 392 1 quad_hexagon.QuadHexagon.track_nbytes_open_grid
71.2k 71.3k 1 quad_hexagon.QuadHexagon.track_peakmem_open_dataset
71.1k 71.1k 1 quad_hexagon.QuadHexagon.track_peakmem_open_grid
10.4±0.1ms 11.0±8ms 1.05 subset.BoundingBox.time_bounding_box('120km')
5.82±0.2ms 6.00±0.1ms 1.03 subset.BoundingBox.time_bounding_box('480km')
2.38M 2.38M 1 subset.BoundingBox.track_peakmem_bounding_box('120km')
203k 203k 1 subset.BoundingBox.track_peakmem_bounding_box('480km')
2.97±1ms 2.88±6ms 0.97 subset.BoundingBoxHealpix.time_bounding_box(6)
12.4±0.3ms 12.0±0.2ms 0.97 subset.BoundingBoxHealpix.time_bounding_box(8)
2.24M 2.24M 1 subset.BoundingBoxHealpix.track_peakmem_bounding_box(6)
13.2M 13.2M 1 subset.BoundingBoxHealpix.track_peakmem_bounding_box(8)
656±3ms 629±4ms 0.96 to_raster.ToRaster.time_to_raster((10000.0, 10.0))
1.38±0.01s 1.34±0.01s 0.97 to_raster.ToRaster.time_to_raster((10000.0, 100.0))
1.63±0.02s 1.56±0.01s 0.96 to_raster.ToRaster.time_to_raster((100000.0, 100.0))
270M 270M 1 to_raster.ToRaster.track_peakmem((10000.0, 10.0))
680M 680M 1 to_raster.ToRaster.track_peakmem((10000.0, 100.0))
765M 765M 1 to_raster.ToRaster.track_peakmem((100000.0, 100.0))

Benchmarks that have got worse:

Change Before [ce8690d] After [b20bba8] Ratio Benchmark (Parameter)
+ 304M 388M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
+ 305M 388M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
+ 307M 388M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
+ 305M 388M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
+ 278M 388M 1.4 import.Imports.track_peakmem_import_uxarray
+ 327M 388M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 1)
+ 327M 388M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 2)
+ 327M 388M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 4)
+ 308M 388M 1.26 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 1)
+ 307M 388M 1.26 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 2)
+ 307M 388M 1.26 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 4)
+ 328M 390M 1.19 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('120km')
+ 302M 390M 1.29 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('480km')
+ 328M 390M 1.19 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('120km')
+ 310M 390M 1.26 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('480km')

The ring loop queried the kd-tree once per radius for the whole disc,
then sorted it and ran setdiff1d against the previous disc, so its cost
grew with rings x faces inside. At 60 degrees on oQU120 it was 93% of
the runtime. One query at the outermost circle now finds every candidate
face, and each is binned by its squared chord distance, summed in the
kd-tree's own order.

The tree accepts or rejects whole nodes on distance bounds that carry
their own rounding, so within an ulp or so of a circle it can disagree
with d**2 <= r**2, in either direction. Faces within 1e-12 of a ring's
edge are settled by querying the tree at that radius, as the per-radius
loop did, which keeps ring membership identical. Checked against the
old loop on oQU480, oQU120 and the dyamond subset over 4050 cases,
15,000 of them exact ties on a ring's edge; results are bit-identical
to the previous commit.

The AzimuthalAverage benchmark now runs out to 60 degrees (61 rings),
where the ring faces rather than per-call overhead set the cost:

  480km  141 ms (before 833840d) -> 1.8 ms (833840d) -> 0.5 ms
  120km  169 ms                   -> 10.1 ms           -> 1.6 ms

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cmdupuis3 cmdupuis3 self-assigned this Oct 23, 2026
@cmdupuis3 cmdupuis3 removed the run-benchmark Run ASV benchmark workflow label Oct 23, 2026
cmdupuis3 and others added 3 commits October 23, 2026 17:45
Fancy-indexing a lazily backed (netCDF) variable reads face by face, so
gathering the ring faces took 87 ms on oQU120 against 1.6 ms for the
whole call on loaded data; the benchmark fixture loads data, hiding it.
The span of ring faces is now read contiguously and indexed in memory.

Dask input with no face inside outer_radius raised ZeroDivisionError in
dask's gufunc rechunking; n_face is already one chunk, so allow_rechunk
is now off.

Ring membership is now d <= r on the computed squared distances, as
documented, rather than reproducing the kd-tree's pruning rounding at
ring edges. The single query is padded so the tree can't drop faces on
the outer circle. Only faces exactly on a ring edge can change ring.

  oQU, 61 rings      loaded            lazy
  480km              0.40 -> 0.33 ms   5.9 -> 0.40 ms
  120km              1.57 -> 0.92 ms   86.7 -> 1.0 ms

Tests share one brute-force ring oracle; the eager, dask and hit-count
checks run as one parametrized reference test, inside and outside the
grid subset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cmdupuis3 cmdupuis3 added run-benchmark Run ASV benchmark workflow and removed run-benchmark Run ASV benchmark workflow labels Oct 26, 2026
The consolidated monotone test dropped non-finite differences, so a NaN
in a non-empty ring passed silently. Only the empty ring at radius 0 is
NaN, and res[1:] already excludes it, so neither profile needs the
filter; the old test_gaussian had none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cmdupuis3
cmdupuis3 marked this pull request as ready for review October 26, 2026 23:21

@Sevans711 Sevans711 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes here look good overall, and a 100x speedup is awesome, I'm excited to get this merged into main!

I have two small concerns: (1) with the tests, and (2) in the possibility for tiny differences between these results and the results from main.

(1) I can tell that some of the new tests in test_azimuthal.py look like they should match the old tests (e.g. test_monotone_profile should cover the old test_gaussian and test_inverse_gaussian I think) but I'm not really sure why the old tests needed to be deleted. It would be easier to approve this if the old tests were maintained as-is, and new tests were added, instead of completely deleting/refactoring the old tests. Refactoring the tests at the same time as the code means there's nothing left to clearly demonstrate that the old test suite's guarantees are still 100% guaranteed to hold true after the changes too. Even though it looks like the new tests cover the same things, maybe there's some subtlety I'm missing.

(2) The results after this change are not bit-for-bit identical to the results before it. For a pure performance change I would have probably expected bit-for-bit identical results, and maybe a test that demonstrates as much, as well. Any guesses as to why the values are different? This PR would be very easy to approve if the results were bit-for-bit identical, but it's more difficult to approve when there are tiny differences.

Minimal reproducer for (2):

psi = ux.tutorial.open_dataset("outCSne30-vortex")['psi']
result = psi.azimuthal_mean((10, 20), 40, 2)

Running this on main and on this branch, and then subtracting the two, I see some values are exactly the same (diff=0) while some have diff=1.11e-16, 2.22e-16, or 4.44e-16.

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

@Sevans711

(1) They were refactored mostly to condense the code, but they are also more flexible. The old tests can be seen with these parameter combos:

Old test New test and parameters
test_gaussian test_monotone_profile[gaussian--1]: name="gaussian", sign=-1
test_inverse_gaussian test_monotone_profile[inverse_gaussian-1]: name="inverse_gaussian", sign=1
test_non_zero_hit_counts test_matches_reference[center0-None] and [center0-chunks1]: center=(45, 0), chunks is None or {"n_face": 50, "time": 1}
test_zero_hit_counts test_matches_reference[center1-None] and [center1-chunks1]: center=(-45, 0) (outside the subset), same two chunks values

I think the git diff is being stupid though, you'll have to compare them side-by-side, or maybe find a good diff tool (git's default tool is terrible even by 2010 standards). Claude could also be helpful.

(2) That is true, this code doesn't reproduce exactly. The new code should be more accurate since it's doing arithmetic n float64 rather than float32. The old error should be around 10^-8, while the error now should be around 10^-16.

@cmdupuis3 cmdupuis3 added run-benchmark Run ASV benchmark workflow and removed run-benchmark Run ASV benchmark workflow labels Oct 27, 2026
@Sevans711

Copy link
Copy Markdown
Collaborator

For (1), thank you for clarifying the parameter combos to get from old tests to new tests. I was also sort of hoping to just get the old tests restored since that makes review easier, but I do agree the new tests look like they should reproduce the old ones and look like they should be reasonable. If you don't want to restore the old tests, I'll just make sure to run them on this branch locally to ensure they still pass, before I approve.

For (2), can you provide some sort of evidence to demonstrate that the new code is indeed always (or at least almost always) accurate (or, more accurate)? It looks like there's 1 correctness test being introduced here, as test_matches_reference, with a couple different numerical parameter inputs, but I'm not sure yet if this covers a wide enough variety of cases to demonstrate that the new code is always correct / more accurate.

If precision is really the only reason there is any difference between here and main, one option could be to locally introduce float64 precision on main's version of the code and show that leads to a bit-for-bit match here. If that is indeed the case I would be happy to approve this PR as-is (assuming the old tests from (1) still pass).

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

(1) Yeah, I'd rather not restore them. The more horizontal alternative would be to do a massive test consolidation sweep, so I think it's reasonable to clean up bits as we go.

(2) Casting main to float64 inherently won't match float32, you'd basically be doing the same thing I'm doing here.

Try this:

from fractions import Fraction

import numpy as np

import uxarray as ux
from uxarray.core.azimuthal import _ring_membership
from uxarray.grid.coordinates import _lonlat_rad_to_xyz

base = "test/meshfiles/mpas/dyamond-30km/"
uxds = ux.open_dataset(base + "gradient_grid_subset.nc", base + "gradient_data_subset.nc")
center, outer_radius, radius_step = (45, 0), 2, 0.25

radii = np.arange(0.0, outer_radius + radius_step, radius_step)
chords = 2.0 * np.sin(np.deg2rad(radii) / 2.0)
center_xyz = np.array(_lonlat_rad_to_xyz(*np.deg2rad(center)))
faces, ring_ids = _ring_membership(uxds.uxgrid._get_scipy_kd_tree(), center_xyz, chords)
areas = uxds.uxgrid.face_areas.values


def exact_mean(x, a):
    """sum(x * a) / sum(a) in exact rational arithmetic."""
    return sum(Fraction(float(xi)) * Fraction(float(ai)) for xi, ai in zip(x, a)) / sum(
        Fraction(float(ai)) for ai in a
    )


def rel_err(value, exact):
    return float(abs(Fraction(float(value)) - exact) / abs(exact))


def report(name, data):
    print(f"\n{name}  (data {data.dtype}, areas {areas.dtype})")
    new = data.azimuthal_mean(
        center_coord=center, outer_radius=outer_radius, radius_step=radius_step
    ).values
    print(f"{'ring':>4} {'faces':>6} {'old (float32) err':>18} {'new (float64) err':>18}")
    for i in range(1, radii.size):
        f = faces[ring_ids == i]
        x, a = data.values[f], areas[f]
        exact = exact_mean(x, a)
        old = (x * a).sum() / a.sum()  # what weighted_mean() computed: all float32
        print(f"{i:>4} {f.size:>6} {rel_err(old, exact):>18.1e} {rel_err(new[i], exact):>18.1e}")


# 1. The real test field
report("gaussian", uxds["gaussian"])

# 2. A constant field: the exact mean is the constant itself
const = uxds["gaussian"].copy(data=np.full(uxds["gaussian"].shape, 0.1, dtype=np.float32))
report("constant 0.1f", const)

@Sevans711

Copy link
Copy Markdown
Collaborator

(1) Yeah, I'd rather not restore them. The more horizontal alternative would be to do a massive test consolidation sweep, so I think it's reasonable to clean up bits as we go.

Fair enough!

(2) Casting main to float64 inherently won't match float32, you'd basically be doing the same thing I'm doing here.

I think there was a misunderstanding: I wasn't expecting "main casted to float64" to match "main now (which is float32)". But, your previous claim:

the new code should be more accurate since it's doing arithmetic in float64 rather than float32.

made me expect "main casted to float64" should match "this PR". Is that the correct understanding / can you show whether that is actually true? If it's true then I would happily agree that the only reason there is a difference is the float64 versus float32, and approve the PR on the basis of "float64 is more accurate than float32"!

@cmdupuis3

cmdupuis3 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

Basically yes, there is no other reason why this code is more accurate than that we're using float64 instead of float32. It's not exactly the same as doing that on main, but it's close, the difference being that you'd need to have the data in float64 to have a fair comparison, but if it is, it would be nearly the same. The change in math on this branch necessarily results in some differences of a few ulps.

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

@Sevans711 So, there are some differences in azimuthal_mean depending on which faces end up on the ring. Generally the difference is small, but in some cases they can be significantly different if the faces are exactly on a ring.

I believe the actual problem is that the existing algorithm (here and on main) uses whole faces and doesn't split faces at the ring edges and weight the partial faces accordingly. That's probably an accuracy issue for another PR though.

@Sevans711

Copy link
Copy Markdown
Collaborator

So, there are some differences in azimuthal_mean depending on which faces end up on the ring. Generally the difference is small, but in some cases they can be significantly different if the faces are exactly on a ring.

I believe the actual problem is that the existing algorithm (here and on main) uses whole faces and doesn't split faces at the ring edges and weight the partial faces accordingly. That's probably an accuracy issue for another PR though.

Thank you for taking a closer look into this! When reading through the code I was also a bit confused about why each face gets assigned to exactly 1 ring, without any splitting and weighting. I agree that accuracy issue should be scoped to a different PR.

I'm not sure how "stable" the azimuthal_mean feature is currently expected to be, or how much downstream friction it might cause for users to see their results suddenly change (either slightly, or significantly in rare cases). This is especially true for this PR, which doesn't really seem like it's intending to change the accuracy, and isn't exposing any sort of user-facing kwarg that could make it easier to discover why the results are suddenly a bit different.

I'd be happy to defer to other reviewers' thoughts on this one. If other uxarray maintainers are happy to proceed with tiny (and rarely somewhat significant) differences here, especially given that the method might need pretty major accuracy updating in the future to properly split and weight faces, I'd be happy to proceed with it as well. But, it feels like a quick team discussion/decision might be the easiest way to proceed here.

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

@Sevans711 The original PR #1255 is pretty recent.

@rajeeja

rajeeja commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Let’s hold this out of this month’s incremental release while we address the file-backed loading concern and clarify the ring-boundary behavior change.

@rajeeja rajeeja left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmdupuis3 Please update the branch with main before making these revisions. In uxarray/core/azimuthal.py, bound the file-backed reads instead of loading the entire span between selected faces, and add a backend-read regression test; locally, selecting 12 faces loaded 133 across all time steps. Please also update the existing docs/user-guide/azimuthal-average.ipynb with a chunked example and notes on lazy execution, single-face-chunk memory requirements, and numerical/boundary differences.

@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@Sevans711 Sevans711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before I give a detailed review I'm waiting for a different uxarray maintainer to explicitly confirm they feel that the numerical changes to results here are acceptable. This could be during a review, a comment on the PR, or during an internal discussion. We discussed briefly internally and my understanding was that @rajeeja was leaning in that direction but wanted to take a closer look before confirming.

(Reminder to myself for when I review: as per my previous comments, I will double-check some tests locally to confirm I am happy with the changes to tests here. I will also make sure to review the new changes, especially the updates to azimuthal-average.ipynb.)

@erogluorhan erogluorhan left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EDIT: Was later duplicated by the next review accidentally; hence, text of this has been removed.

@erogluorhan erogluorhan left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does what #1848 asks: no per-ring subgrid, one sparse operator, and chunked input stays lazy. I checked the open points from earlier reviews against 8191975e.

  • @Sevans711 (1), old tests: main's test/core/test_azimuthal.py run against this branch passes 4/4.
  • @Sevans711 (2), numerical differences: I compared main and this branch on outCSne30, MPAS QU and the dyamond subset (10 centers, float32/float64 data, (time, n_face) and (n_face, time) order, chunked).
    • Hit counts were identical everywhere.
    • Means differ by ≤2e-13 relative on grids with float64 areas.
    • On dyamond, whose face_areas are float32, they differ by up to 9e-8 even for float64 data. That is main summing the ring areas in float32; this branch matches a float64 oracle to rtol 1e-12.
    • Ring membership changes only when a face center lies exactly on a ring radius. Over the PR's 200 tie cases, main's per-ring kd-tree queries disagree with the documented r_(i-1) < d <= r_i in 85 (dyamond) and 70 (outCSne30), and this branch in none.
  • rajeeja, bounded file-backed reads: addressed. test_file_backed_reads_are_bounded fails if _read_faces reads one span instead. The notebook section covers chunking, memory and boundary behavior.
  • Other checks:
    • Mutation of each part of azimuthal.py: the running max, the 1e-9 pad, NaN→0, empty rings→NaN and the single chunk each make a test fail.
    • Chunked input: 0 dask computes during the call (20 on main).
    • 21 rings on outCSne30: 37 ms → 0.46 ms.

Before merge: please update the description, which is stale after the later commits. It says ring membership is unchanged, which the notebook now contradicts, and it quotes the old benchmark (21 rings to 20°) while the code runs 61 rings to 60°. One notebook nit inline.

Comment thread docs/user-guide/azimuthal-average.ipynb Resolved
@erogluorhan

Copy link
Copy Markdown
Member

@Sevans711 , so explicitly stating after my review, the numerical changes look okay to me. @rajeeja any thoughts?

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

@erogluorhan The PR text has been refreshed also.

@Sevans711
Sevans711 self-requested a review November 17, 2026 19:16
@dylannelson

Copy link
Copy Markdown
Member

1 - Memory Concerns

There seemed to be some memory concerns that came up:

The dask path gathers every face inside outer_radius into one n_face chunk before the sparse multiply. With data chunked only along n_face, that block holds every ring face for every time step, and the peak is 4–5x the block. (time=100, n_face) random data on a 0.25° lat-lon grid (1,036,800 faces) in 10,000-face chunks, peak from tracemalloc:

(time=100, n_face) on 1.04M faces, 10k-face chunks main PR Fix
outer 30° 25 MiB, 3.8–3.9 s 201 MiB, 0.9 s 25 MiB, 0.8–0.9 s
outer 90° 45 MiB, 27–29 s 1,998 MiB, 2.6–2.7 s 48 MiB, 1.7–1.8 s

Claude suggested a fix could look like this:

Diff against 1661420
--- a/uxarray/core/azimuthal.py
+++ b/uxarray/core/azimuthal.py
@@ -30,12 +30,11 @@ def _compute_azimuthal_mean(uxda, center_xyz, chord_radii):
     # Only the data variable is gathered: the caller rebuilds the coords, and
     # loading them here would read any lazy ones once per contiguous read
     variable = uxda.variable
-    if variable.chunks is None:
-        ring_data = _read_faces(variable, faces)
-    else:
-        # dask reads only the chunks that hold ring faces; the sparse multiply
-        # spans every ring face, so they're gathered into a single chunk
-        ring_data = variable.isel(n_face=faces).chunk({"n_face": -1})
+    dims = ["radius" if dim == "n_face" else dim for dim in uxda.dims]
+    if variable.chunks is not None:
+        means = _dask_ring_means(variable, faces, weights, hit_count)
+        return means.transpose(*dims), hit_count
+    ring_data = _read_faces(variable, faces)
 
     means = xr.apply_ufunc(
         _apply_ring_weights,
@@ -52,10 +51,52 @@ def _compute_azimuthal_mean(uxda, center_xyz, chord_radii):
         },
     )
 
-    dims = ["radius" if dim == "n_face" else dim for dim in uxda.dims]
     return means.transpose(*dims), hit_count
 
 
+def _dask_ring_means(variable, faces, weights, hit_count):
+    """Applies the ring operator to each input n_face chunk and sums the partial
+    results, so ring faces are never gathered into one chunk: each task holds one
+    input chunk and an (other dims x n_radius) partial sum."""
+    import dask.array as da
+
+    v = variable.transpose(..., "n_face")
+    arr = v.data
+    n_radius = weights.shape[0]
+    dtype = np.result_type(weights.dtype, arr.dtype)
+    out_chunks = arr.chunks[:-1] + ((n_radius,),)
+    bounds = np.cumsum((0,) + arr.chunks[-1])
+    parts = []
+    for lo_face, hi_face in zip(bounds[:-1], bounds[1:]):
+        lo, hi = np.searchsorted(faces, [lo_face, hi_face])  # faces are ascending
+        if lo == hi:
+            continue
+        part = arr[..., lo_face:hi_face].map_blocks(
+            _partial_ring_sums,
+            local=faces[lo:hi] - lo_face,
+            weights=weights[:, lo:hi],
+            chunks=out_chunks,
+            dtype=dtype,
+            meta=np.array((), dtype=dtype),
+        )
+        parts.append(part)
+    if parts:
+        # a tree reduction: a chain of + is fused into one task that needs every input chunk
+        total = da.stack(parts).sum(axis=0)
+    else:
+        total = da.zeros(arr.shape[:-1] + (n_radius,), chunks=out_chunks, dtype=dtype)
+    total = da.where(hit_count == 0, np.nan, total)  # empty rings
+    return xr.Variable(v.dims[:-1] + ("radius",), total)
+
+
+def _partial_ring_sums(block, local, weights):
+    values = block[..., local]
+    values = np.where(np.isnan(values), 0, values)
+    flat = values.reshape(math.prod(values.shape[:-1]), values.shape[-1])
+    sums = (weights @ flat.T).T
+    return sums.reshape(values.shape[:-1] + (weights.shape[0],))
+
+
 def _ring_membership(kdtree, center_xyz, chord_radii):

2 - Slight potential speedup

In-memory NumPy data goes through the file-read path _read_faces, when it may not need to, slight speed up by changing this:

Fix, tested in uxarray/core/azimuthal.py#L33-L34:

-    if variable.chunks is None:
+    if variable.chunks is None and variable._in_memory:
+        # already in memory: one fancy index is cheaper than contiguous reads
+        ring_data = variable.isel(n_face=faces)
+    elif variable.chunks is None:
         ring_data = _read_faces(variable, faces)

Dask-backed data is now reduced to partial ring sums per input n_face
chunk and added up in a tree, so no task gathers every face inside
outer_radius and memory no longer grows with it. In-memory data skips
the contiguous-read path and uses a single fancy index. The apply_ufunc
wrapper is replaced by one _ring_sums helper shared by all paths.

Adds a regression test that no dask task holds more than one input
chunk, and updates the docstring and the user guide's memory and
precision notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

@dylannelson Both changes have been incorporated

@cmdupuis3 cmdupuis3 added the run-benchmark Run ASV benchmark workflow label Nov 18, 2026

@dylannelson dylannelson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the concerns I had have been addressed, there were no blockers, and the speed up was seen locally and in the benchmarks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Related to mathematical accuracy of operations, such as numerical precision or rounding errors run-benchmark Run ASV benchmark workflow scalability Related to scalability & performance efforts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

azimuthal_mean builds a full subgrid for every ring

5 participants