Repository navigation
Conversation
azimuthal_mean built a full subgrid for every ring (isel without ignore_grid), recomputed face_areas on it, and forced a compute per ring. The rings are disjoint, so the per-ring area-weighted means now form one sparse (n_radius, n_ring_faces) operator over the ring faces, gathered once with ignore_grid=True and weighted by the parent grid's areas. It is applied through a single apply_ufunc, so dask-backed input stays lazy. Ring membership, hit counts, output dims/coords/attrs and NaN handling (NaN faces add nothing to the sum but still count toward the ring area) are unchanged. Accumulation is now float64: on float32 grids the old float32 sums were off by up to ~3e-7 relative against a float64 reference; the new result matches it to ~3e-16. Adds AzimuthalAverage to the oQU benchmarks (21 rings, out to 20 degrees): 480km 43.9 ms -> 0.7 ms, 120km 50.0 ms -> 1.2 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ASV BenchmarkingBenchmark Comparison ResultsBenchmarks that have improved:
Benchmarks that have stayed the same:
Benchmarks that have got worse:
|
The ring loop queried the kd-tree once per radius for the whole disc, then sorted it and ran setdiff1d against the previous disc, so its cost grew with rings x faces inside. At 60 degrees on oQU120 it was 93% of the runtime. One query at the outermost circle now finds every candidate face, and each is binned by its squared chord distance, summed in the kd-tree's own order. The tree accepts or rejects whole nodes on distance bounds that carry their own rounding, so within an ulp or so of a circle it can disagree with d**2 <= r**2, in either direction. Faces within 1e-12 of a ring's edge are settled by querying the tree at that radius, as the per-radius loop did, which keeps ring membership identical. Checked against the old loop on oQU480, oQU120 and the dyamond subset over 4050 cases, 15,000 of them exact ties on a ring's edge; results are bit-identical to the previous commit. The AzimuthalAverage benchmark now runs out to 60 degrees (61 rings), where the ring faces rather than per-call overhead set the cost: 480km 141 ms (before 833840d) -> 1.8 ms (833840d) -> 0.5 ms 120km 169 ms -> 10.1 ms -> 1.6 ms Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fancy-indexing a lazily backed (netCDF) variable reads face by face, so gathering the ring faces took 87 ms on oQU120 against 1.6 ms for the whole call on loaded data; the benchmark fixture loads data, hiding it. The span of ring faces is now read contiguously and indexed in memory. Dask input with no face inside outer_radius raised ZeroDivisionError in dask's gufunc rechunking; n_face is already one chunk, so allow_rechunk is now off. Ring membership is now d <= r on the computed squared distances, as documented, rather than reproducing the kd-tree's pruning rounding at ring edges. The single query is padded so the tree can't drop faces on the outer circle. Only faces exactly on a ring edge can change ring. oQU, 61 rings loaded lazy 480km 0.40 -> 0.33 ms 5.9 -> 0.40 ms 120km 1.57 -> 0.92 ms 86.7 -> 1.0 ms Tests share one brute-force ring oracle; the eager, dask and hit-count checks run as one parametrized reference test, inside and outside the grid subset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The consolidated monotone test dropped non-finite differences, so a NaN in a non-empty ring passed silently. Only the empty ring at radius 0 is NaN, and res[1:] already excludes it, so neither profile needs the filter; the old test_gaussian had none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
The changes here look good overall, and a 100x speedup is awesome, I'm excited to get this merged into main!
I have two small concerns: (1) with the tests, and (2) in the possibility for tiny differences between these results and the results from main.
(1) I can tell that some of the new tests in test_azimuthal.py look like they should match the old tests (e.g. test_monotone_profile should cover the old test_gaussian and test_inverse_gaussian I think) but I'm not really sure why the old tests needed to be deleted. It would be easier to approve this if the old tests were maintained as-is, and new tests were added, instead of completely deleting/refactoring the old tests. Refactoring the tests at the same time as the code means there's nothing left to clearly demonstrate that the old test suite's guarantees are still 100% guaranteed to hold true after the changes too. Even though it looks like the new tests cover the same things, maybe there's some subtlety I'm missing.
(2) The results after this change are not bit-for-bit identical to the results before it. For a pure performance change I would have probably expected bit-for-bit identical results, and maybe a test that demonstrates as much, as well. Any guesses as to why the values are different? This PR would be very easy to approve if the results were bit-for-bit identical, but it's more difficult to approve when there are tiny differences.
Minimal reproducer for (2):
psi = ux.tutorial.open_dataset("outCSne30-vortex")['psi']
result = psi.azimuthal_mean((10, 20), 40, 2)Running this on main and on this branch, and then subtracting the two, I see some values are exactly the same (diff=0) while some have diff=1.11e-16, 2.22e-16, or 4.44e-16.
|
(1) They were refactored mostly to condense the code, but they are also more flexible. The old tests can be seen with these parameter combos:
I think the git diff is being stupid though, you'll have to compare them side-by-side, or maybe find a good diff tool (git's default tool is terrible even by 2010 standards). Claude could also be helpful. (2) That is true, this code doesn't reproduce exactly. The new code should be more accurate since it's doing arithmetic n float64 rather than float32. The old error should be around 10^-8, while the error now should be around 10^-16. |
|
For (1), thank you for clarifying the parameter combos to get from old tests to new tests. I was also sort of hoping to just get the old tests restored since that makes review easier, but I do agree the new tests look like they should reproduce the old ones and look like they should be reasonable. If you don't want to restore the old tests, I'll just make sure to run them on this branch locally to ensure they still pass, before I approve. For (2), can you provide some sort of evidence to demonstrate that the new code is indeed always (or at least almost always) accurate (or, more accurate)? It looks like there's 1 correctness test being introduced here, as If precision is really the only reason there is any difference between here and |
|
(1) Yeah, I'd rather not restore them. The more horizontal alternative would be to do a massive test consolidation sweep, so I think it's reasonable to clean up bits as we go. (2) Casting Try this: from fractions import Fraction
import numpy as np
import uxarray as ux
from uxarray.core.azimuthal import _ring_membership
from uxarray.grid.coordinates import _lonlat_rad_to_xyz
base = "test/meshfiles/mpas/dyamond-30km/"
uxds = ux.open_dataset(base + "gradient_grid_subset.nc", base + "gradient_data_subset.nc")
center, outer_radius, radius_step = (45, 0), 2, 0.25
radii = np.arange(0.0, outer_radius + radius_step, radius_step)
chords = 2.0 * np.sin(np.deg2rad(radii) / 2.0)
center_xyz = np.array(_lonlat_rad_to_xyz(*np.deg2rad(center)))
faces, ring_ids = _ring_membership(uxds.uxgrid._get_scipy_kd_tree(), center_xyz, chords)
areas = uxds.uxgrid.face_areas.values
def exact_mean(x, a):
"""sum(x * a) / sum(a) in exact rational arithmetic."""
return sum(Fraction(float(xi)) * Fraction(float(ai)) for xi, ai in zip(x, a)) / sum(
Fraction(float(ai)) for ai in a
)
def rel_err(value, exact):
return float(abs(Fraction(float(value)) - exact) / abs(exact))
def report(name, data):
print(f"\n{name} (data {data.dtype}, areas {areas.dtype})")
new = data.azimuthal_mean(
center_coord=center, outer_radius=outer_radius, radius_step=radius_step
).values
print(f"{'ring':>4} {'faces':>6} {'old (float32) err':>18} {'new (float64) err':>18}")
for i in range(1, radii.size):
f = faces[ring_ids == i]
x, a = data.values[f], areas[f]
exact = exact_mean(x, a)
old = (x * a).sum() / a.sum() # what weighted_mean() computed: all float32
print(f"{i:>4} {f.size:>6} {rel_err(old, exact):>18.1e} {rel_err(new[i], exact):>18.1e}")
# 1. The real test field
report("gaussian", uxds["gaussian"])
# 2. A constant field: the exact mean is the constant itself
const = uxds["gaussian"].copy(data=np.full(uxds["gaussian"].shape, 0.1, dtype=np.float32))
report("constant 0.1f", const) |
Fair enough!
I think there was a misunderstanding: I wasn't expecting "main casted to float64" to match "main now (which is float32)". But, your previous claim:
made me expect "main casted to float64" should match "this PR". Is that the correct understanding / can you show whether that is actually true? If it's true then I would happily agree that the only reason there is a difference is the float64 versus float32, and approve the PR on the basis of "float64 is more accurate than float32"! |
|
Basically yes, there is no other reason why this code is more accurate than that we're using float64 instead of float32. It's not exactly the same as doing that on main, but it's close, the difference being that you'd need to have the data in float64 to have a fair comparison, but if it is, it would be nearly the same. The change in math on this branch necessarily results in some differences of a few ulps. |
|
@Sevans711 So, there are some differences in I believe the actual problem is that the existing algorithm (here and on main) uses whole faces and doesn't split faces at the ring edges and weight the partial faces accordingly. That's probably an accuracy issue for another PR though. |
Thank you for taking a closer look into this! When reading through the code I was also a bit confused about why each face gets assigned to exactly 1 ring, without any splitting and weighting. I agree that accuracy issue should be scoped to a different PR. I'm not sure how "stable" the azimuthal_mean feature is currently expected to be, or how much downstream friction it might cause for users to see their results suddenly change (either slightly, or significantly in rare cases). This is especially true for this PR, which doesn't really seem like it's intending to change the accuracy, and isn't exposing any sort of user-facing kwarg that could make it easier to discover why the results are suddenly a bit different. I'd be happy to defer to other reviewers' thoughts on this one. If other uxarray maintainers are happy to proceed with tiny (and rarely somewhat significant) differences here, especially given that the method might need pretty major accuracy updating in the future to properly split and weight faces, I'd be happy to proceed with it as well. But, it feels like a quick team discussion/decision might be the easiest way to proceed here. |
|
@Sevans711 The original PR #1255 is pretty recent. |
|
Let’s hold this out of this month’s incremental release while we address the file-backed loading concern and clarify the ring-boundary behavior change. |
rajeeja
left a comment
There was a problem hiding this comment.
@cmdupuis3 Please update the branch with main before making these revisions. In uxarray/core/azimuthal.py, bound the file-backed reads instead of loading the entire span between selected faces, and add a backend-read regression test; locally, selecting 12 faces loaded 133 across all time steps. Please also update the existing docs/user-guide/azimuthal-average.ipynb with a chunked example and notes on lazy execution, single-face-chunk memory requirements, and numerical/boundary differences.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Sevans711
left a comment
There was a problem hiding this comment.
Before I give a detailed review I'm waiting for a different uxarray maintainer to explicitly confirm they feel that the numerical changes to results here are acceptable. This could be during a review, a comment on the PR, or during an internal discussion. We discussed briefly internally and my understanding was that @rajeeja was leaning in that direction but wanted to take a closer look before confirming.
(Reminder to myself for when I review: as per my previous comments, I will double-check some tests locally to confirm I am happy with the changes to tests here. I will also make sure to review the new changes, especially the updates to azimuthal-average.ipynb.)
There was a problem hiding this comment.
This does what #1848 asks: no per-ring subgrid, one sparse operator, and chunked input stays lazy. I checked the open points from earlier reviews against 8191975e.
- @Sevans711 (1), old tests:
main'stest/core/test_azimuthal.pyrun against this branch passes 4/4. - @Sevans711 (2), numerical differences: I compared
mainand this branch on outCSne30, MPAS QU and the dyamond subset (10 centers, float32/float64 data,(time, n_face)and(n_face, time)order, chunked).- Hit counts were identical everywhere.
- Means differ by ≤2e-13 relative on grids with float64 areas.
- On dyamond, whose
face_areasare float32, they differ by up to 9e-8 even for float64 data. That ismainsumming the ring areas in float32; this branch matches a float64 oracle to rtol 1e-12. - Ring membership changes only when a face center lies exactly on a ring radius. Over the PR's 200 tie cases,
main's per-ring kd-tree queries disagree with the documentedr_(i-1) < d <= r_iin 85 (dyamond) and 70 (outCSne30), and this branch in none.
- rajeeja, bounded file-backed reads: addressed.
test_file_backed_reads_are_boundedfails if_read_facesreads one span instead. The notebook section covers chunking, memory and boundary behavior. - Other checks:
- Mutation of each part of
azimuthal.py: the running max, the 1e-9 pad, NaN→0, empty rings→NaN and the single chunk each make a test fail. - Chunked input: 0 dask computes during the call (20 on
main). - 21 rings on outCSne30: 37 ms → 0.46 ms.
- Mutation of each part of
Before merge: please update the description, which is stale after the later commits. It says ring membership is unchanged, which the notebook now contradicts, and it quotes the old benchmark (21 rings to 20°) while the code runs 61 rings to 60°. One notebook nit inline.
|
@Sevans711 , so explicitly stating after my review, the numerical changes look okay to me. @rajeeja any thoughts? |
|
@erogluorhan The PR text has been refreshed also. |
1 - Memory ConcernsThere seemed to be some memory concerns that came up: The dask path gathers every face inside
Claude suggested a fix could look like this: Diff against 1661420--- a/uxarray/core/azimuthal.py
+++ b/uxarray/core/azimuthal.py
@@ -30,12 +30,11 @@ def _compute_azimuthal_mean(uxda, center_xyz, chord_radii):
# Only the data variable is gathered: the caller rebuilds the coords, and
# loading them here would read any lazy ones once per contiguous read
variable = uxda.variable
- if variable.chunks is None:
- ring_data = _read_faces(variable, faces)
- else:
- # dask reads only the chunks that hold ring faces; the sparse multiply
- # spans every ring face, so they're gathered into a single chunk
- ring_data = variable.isel(n_face=faces).chunk({"n_face": -1})
+ dims = ["radius" if dim == "n_face" else dim for dim in uxda.dims]
+ if variable.chunks is not None:
+ means = _dask_ring_means(variable, faces, weights, hit_count)
+ return means.transpose(*dims), hit_count
+ ring_data = _read_faces(variable, faces)
means = xr.apply_ufunc(
_apply_ring_weights,
@@ -52,10 +51,52 @@ def _compute_azimuthal_mean(uxda, center_xyz, chord_radii):
},
)
- dims = ["radius" if dim == "n_face" else dim for dim in uxda.dims]
return means.transpose(*dims), hit_count
+def _dask_ring_means(variable, faces, weights, hit_count):
+ """Applies the ring operator to each input n_face chunk and sums the partial
+ results, so ring faces are never gathered into one chunk: each task holds one
+ input chunk and an (other dims x n_radius) partial sum."""
+ import dask.array as da
+
+ v = variable.transpose(..., "n_face")
+ arr = v.data
+ n_radius = weights.shape[0]
+ dtype = np.result_type(weights.dtype, arr.dtype)
+ out_chunks = arr.chunks[:-1] + ((n_radius,),)
+ bounds = np.cumsum((0,) + arr.chunks[-1])
+ parts = []
+ for lo_face, hi_face in zip(bounds[:-1], bounds[1:]):
+ lo, hi = np.searchsorted(faces, [lo_face, hi_face]) # faces are ascending
+ if lo == hi:
+ continue
+ part = arr[..., lo_face:hi_face].map_blocks(
+ _partial_ring_sums,
+ local=faces[lo:hi] - lo_face,
+ weights=weights[:, lo:hi],
+ chunks=out_chunks,
+ dtype=dtype,
+ meta=np.array((), dtype=dtype),
+ )
+ parts.append(part)
+ if parts:
+ # a tree reduction: a chain of + is fused into one task that needs every input chunk
+ total = da.stack(parts).sum(axis=0)
+ else:
+ total = da.zeros(arr.shape[:-1] + (n_radius,), chunks=out_chunks, dtype=dtype)
+ total = da.where(hit_count == 0, np.nan, total) # empty rings
+ return xr.Variable(v.dims[:-1] + ("radius",), total)
+
+
+def _partial_ring_sums(block, local, weights):
+ values = block[..., local]
+ values = np.where(np.isnan(values), 0, values)
+ flat = values.reshape(math.prod(values.shape[:-1]), values.shape[-1])
+ sums = (weights @ flat.T).T
+ return sums.reshape(values.shape[:-1] + (weights.shape[0],))
+
+
def _ring_membership(kdtree, center_xyz, chord_radii):2 - Slight potential speedupIn-memory NumPy data goes through the file-read path Fix, tested in - if variable.chunks is None:
+ if variable.chunks is None and variable._in_memory:
+ # already in memory: one fancy index is cheaper than contiguous reads
+ ring_data = variable.isel(n_face=faces)
+ elif variable.chunks is None:
ring_data = _read_faces(variable, faces) |
Dask-backed data is now reduced to partial ring sums per input n_face chunk and added up in a tree, so no task gathers every face inside outer_radius and memory no longer grows with it. In-memory data skips the contiguous-read path and uses a single fancy index. The apply_ufunc wrapper is replaced by one _ring_sums helper shared by all paths. Adds a regression test that no dask task holds more than one input chunk, and updates the docstring and the user guide's memory and precision notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@dylannelson Both changes have been incorporated |
dylannelson
left a comment
There was a problem hiding this comment.
All the concerns I had have been addressed, there were no blockers, and the speed up was seen locally and in the benchmarks.
Closes #1848
Overview
Speedup for azimuthal means (~100x on the oQU benchmark), lazy dask support, and an incidental float32 accuracy fix.
azimuthal_meanlooped over the radii: each ring ran its own kd-tree query, built a subgrid withisel(withoutignore_grid), recomputedface_areason it, and computed itsweighted_meaneagerly. Now:r_(i-1) < d <= r_i). Hit counts matchmainon the test and tutorial meshes; they can only differ for a face center within rounding of a ring boundary.(n_radius, n_ring_faces)operator weighted by the parent grid's face areas. No subgrids are built.mainreturned NumPy). Each inputn_facechunk that holds ring faces is reduced to partial ring sums, and the partial sums are added up in a tree. No task gathers every face insideouter_radius, so peak memory per task is about one input chunk, whatever the radius. Chunks without ring faces aren't read, and the other dims keep their chunks.Output dims, coords, attrs, name and NaN handling (NaN faces add nothing to the sum but still count toward the ring area) are unchanged, and
UxDataset.azimuthal_meangets the same speedup since it calls this per variable. Accumulation is now float64: on the float32 dyamond test grid,main's ring means are off by up to ~1.6e-7 relative against a float64 reference, while the new ones are within ~3e-16. On float64 data the two agree to ~1e-15 relative. Chunked and in-memory inputs add the same terms in a different order, so they can differ by rounding.Adds
AzimuthalAverageto the oQU benchmarks (bottomDepth, 61 rings out to 60 degrees around (-30, 0), kd-tree and face areas pre-cached). Locally: oQU480 175 ms -> 1.1 ms, oQU120 292 ms -> 3.3 ms.Docs: the docstring describes how dask-backed and file-backed data are read, and the azimuthal-average user guide gains a chunked example plus notes on lazy execution, estimating the memory of a task, and numerical/boundary differences.
New tests:
test_ring_membershipchecks the rings against brute force, including faces exactly on ring edges and radii past 180 degrees.test_file_backed_reads_are_boundedcounts reads through an xarrayBackendArrayand checks that at most twice the selected faces are read.test_dask_holds_one_input_chunk_at_a_timerecords the size of every intermediate dask result and checks that none is larger than one input chunk.The old tests can be seen with these parameter combos:
test_gaussiantest_monotone_profile[gaussian--1]:name="gaussian",sign=-1test_inverse_gaussiantest_monotone_profile[inverse_gaussian-1]:name="inverse_gaussian",sign=1test_non_zero_hit_countstest_matches_reference[center0-None]and[center0-chunks1]:center=(45, 0),chunksisNoneor{"n_face": 50, "time": 1}test_zero_hit_countstest_matches_reference[center1-None]and[center1-chunks1]:center=(-45, 0)(outside the subset), same twochunksvaluesPR Checklist
General
Testing & Benchmarking
Documentation and Examples
docs/api.rst; internal (private) function names start with an underscore (_)AI Disclosure
AI Usage: Claude Opus 5.5