Repository navigation
Merge coincident nodes when constructing any Grid - #1692
Conversation
Match nodes in Cartesian space rather than the lon/lat plane so pole and antimeridian nodes are recognized as the same point, and store the resulting polar faces as triangles instead of quads with a repeated corner.
Canonicalize duplicate node indices in the face-node connectivity before building the dual, so grids with repeated nodes produce a correct dual instead of being rejected. Also vectorize the duplicate lookup and deprecate the now redundant check_duplicate_nodes argument.
Sevans711
left a comment
There was a problem hiding this comment.
Mostly looks like a good fix! I think it still needs a little bit of extra work, and I left some inline comments accordingly.
|
Actually, apologies for not including this during the original review, comment but one more thought: does this actually fully close the original issue? The issue writeup makes it sound to me like duplicate nodes should be handled immediately upon constructing the grid, not just during one functionality (get_dual). Is there a reason that duplicate nodes should be handled only during get_dual, instead of immediately? (Do all other current/planned functions work properly regardless of whether there are duplicate nodes?) |
float32 input (e.g. real climate datasets) silently ran the whole xyz/tolerance pipeline at float32 precision, causing pole/antimeridian merges to fail or merge only partially.
Match nodes in Cartesian space rather than the lon/lat plane so pole and antimeridian nodes are recognized as the same point, and store the resulting polar faces as triangles instead of quads with a repeated corner.
float32 input (e.g. real climate datasets) silently ran the whole xyz/tolerance pipeline at float32 precision, causing pole/antimeridian merges to fail or merge only partially.
Extend #865's fix beyond the dual mesh: canonicalize duplicate/coincident node indices in connectivity for every Grid construction path, not just construct_dual. Detection is now tolerance-based (unit-sphere chordal distance) instead of exact lon/lat match, so pole-degenerate duplicates are also caught. Node coordinate arrays are left untouched by design; only connectivity is remapped to canonical indices, with any resulting repeated face corners collapsed.
construct_dual no longer needs its own per-call duplicate detection and remap, and get_dual() no longer needs to hard-gate on duplicate node indices, since Grid construction now canonicalizes them structurally before any of this code runs.
Since duplicate node coordinates are intentionally left unreferenced by connectivity, a node KDTree/BallTree built over the raw coordinate array could select an index no face actually points to, silently returning empty or wrong nearest-neighbor results. Build the "nodes" tree only over live (referenced) indices and translate query results back to original index space.
polars' unique() with maintain_order unset does not guarantee row order across runs, so the node index assigned to a given corner coordinate could vary between reads of the same file. This is normally harmless, but it made canonical-node selection for coincident duplicates (e.g. pole points with differing longitude) flaky from run to run.
test_dual_duplicate: validate() now succeeds since connectivity is fully canonicalized (duplicate coordinates remain by design, but nothing references a dead index anymore). test_grid_nn_subset: max valid k for a node search is now bounded by the live node count, not raw node count. test_to_geodataframe_preserves_antimeridian_faces: pole-coincident corners with differing longitude are now also merged, shifting the antimeridian face count.
Merging pole-adjacent duplicate nodes was collapsing each face's own locally-meaningful longitude at the pole into one arbitrary canonical value, which corrupted lat/lon bounds and broke zonal weight computation for cube-sphere grids near the poles.
Sevans711
left a comment
There was a problem hiding this comment.
Suggestion: please change the title of this PR to reflect the full scope. Duplicate nodes are now handled directly whenever constructing a Grid, not just when constructing the dual mesh. I was confused about why _check_duplicate_nodes_indices checks were removed despite construct_dual() being unchanged. I think the reason is because all Grid objects are now guaranteed to not have duplicate nodes.
I tried leaving a full review but I kept getting the feeling that something weird was happening, a nagging feeling like "hey I think I've looked at this code before and left an inline comment, why am I reviewing it again?" Then I realized many of the changes here are also in #1690 which hasn't been merged yet. That led to duplicating review work and will probably lead to needing to apply fixes multiple times. I'm not sure the cleanest way forwards at this point, but I might suggest waiting to merge this until after #1690 gets merged. At least, I will want to do another close review of this PR after that PR merges, because I lost track of which things I already looked at closely and which I need to consider again.
…des' into rajeeja/coincident-nodes # Conflicts: # test/io/test_structured.py # uxarray/core/dataarray.py # uxarray/core/dataset.py # uxarray/grid/grid.py # uxarray/io/_structured.py
"Coincident nodes" is the domain term used by TempestRemap, MOAB, and our own validation module; "dedupe" reads like a dataframe operation. Also note in the docstring that we keep the redundant coordinates and only remap connectivity, where TempestRemap and MOAB delete and renumber.
_coincident_node_canonical_indices exempts pole nodes from merging so each face touching a pole keeps its own longitude. The mask used np.isclose(|z|, 1.0, atol=tolerance), which is wrong twice: tolerance is a chord radius everywhere else in the function, and np.isclose's default rtol=1e-5 swamped atol entirely. The carve-out spanned 1 - |z| <= 1.001e-5, a chord of 4.5e-3 or ~28 km on Earth, so any node within 28 km of a pole was silently exempt from merging. For a point at colatitude t, 1 - |z| = 1 - cos(t) = chord**2 / 2, so the correct cap is tolerance**2 / 2 with rtol pinned to zero.
Sevans711
left a comment
There was a problem hiding this comment.
Looks like some great progress! All of my previous inline comments have been addressed, the benchmarks aren't showing slowdowns, and my previous timing test of ux.open_grid() on a 15km MPAS grid now shows extremely similar speeds (within 1%) on this PR and on main.
I will plan to take one more look through these changes before approving, but I would prefer to delay that until at least a majority of @cmdupuis3's inline comments have been addressed. (Currently I see many unresolved inline comments, and no direct replies, so at a glance I can't tell whether they have been addressed yet.) Please feel free to re-request a review from me once more of those points have been resolved!
cmdupuis3
left a comment
There was a problem hiding this comment.
Some snippets worth investigating:
ux.open_grid("test/meshfiles/ugrid/geoflow-small/grid.nc").get_dual().n_face # 3842
ux.open_grid("test/meshfiles/ugrid/geoflow-small/grid.nc", chunks={}).get_dual() # GridInvalidErrorux.Grid.from_topology(np.array([0., 1, 1, 0, np.nan]), np.array([0., 0, 1, 1, np.nan]),
np.array([[0, 1, 2, 3]]))
# main: builds; branch: ValueError: Node XYZ coordinates must contain only finite values.F = 1e20 # face 1 is a triangle padded with the fill value
clon = np.array([[0., 1, 1, 0], [1, 2, 1, F]]); clat = np.array([[0., 0, 1, 1], [0, 0, 1, F]])
xr.Dataset({
"grid_center_lon": ("grid_size", [0.5, 1.33]), "grid_center_lat": ("grid_size", [0.5, 0.33]),
"grid_corner_lon": (("grid_size", "grid_corners"), clon, {"units": "degrees", "_FillValue": F}),
"grid_corner_lat": (("grid_size", "grid_corners"), clat, {"units": "degrees", "_FillValue": F}),
"grid_imask": ("grid_size", np.array([1, 1], np.int32)), "grid_dims": ("grid_rank", np.array([2], np.int32)),
}).to_netcdf("scrip_fill.nc")
ux.open_grid("scrip_fill.nc")
# main: opens (face_lon from grid_center_lon; the padded face's area is NaN, as before)
# branch: ValueError: Node XYZ coordinates must contain only finite values.p = ux.open_grid("test/meshfiles/exodus/outCSne8/outCSne8.g")
fnc = np.vstack([p.face_node_connectivity.values, p.face_node_connectivity.values[:1]]) # repeated face
g = ux.Grid.from_topology(p.node_lon.values, p.node_lat.values, fnc)
g.get_dual() # still uses node 384
ux.UxDataArray(np.zeros(g.n_node), dims=["n_node"], uxgrid=g).get_dual() # 384 merged into 0
g.get_dual().get_dual() # GridInvalidErrorThis one's more of a robustness issue, not necessarily blocking; one bad face renders the whole grid unopenable.
ux.open_grid("bowtie.nc") # GridInvalidError
ux.open_dataset("bowtie.nc", "bowtie.nc") # GridInvalidError
ux.Grid.from_dataset(ds, merge_coincident_nodes=False) # GridInvalidError (flag isn't forwarded)
ux.open_grid("bowtie.nc", merge_coincident_nodes=False) # TypeError (goes to the xarray backend)
Sevans711
left a comment
There was a problem hiding this comment.
After one final read through, this looks reasonable to me. All of my comments have been addressed, and most other comments have been addressed as well. I'm assuming that @cmdupuis3 would take a closer look at the remaining suggestions to investigate the snippets above (in #1692 (review)); I didn't take a closer look at them, personally. I'm happy to approve this one, thank you for all your efforts on it!
cmdupuis3
left a comment
There was a problem hiding this comment.
There's an issue with the merge_coincident_nodes setting's persistence:
import numpy as np, xarray as xr, uxarray as ux
# outCSne8 plus one extra node that duplicates node 0; one face uses the copy.
p = ux.open_grid("test/meshfiles/exodus/outCSne8/outCSne8.g")
lon = np.append(p.node_lon.values, p.node_lon.values[0])
lat = np.append(p.node_lat.values, p.node_lat.values[0])
fnc = p.face_node_connectivity.values.copy()
r, c = np.argwhere(fnc == 0)[0]
fnc[r, c] = lon.size - 1
ds = xr.Dataset({
"mesh": ((), 0, {"cf_role": "mesh_topology", "topology_dimension": 2,
"node_coordinates": "lon lat", "face_node_connectivity": "faces"}),
"lon": ("nodes", lon, {"standard_name": "longitude"}),
"lat": ("nodes", lat, {"standard_name": "latitude"}),
"faces": (("faces_dim", "corners"), fnc, {"start_index": 0}),
})
ux.open_grid(ds, merge_coincident_nodes=False).get_dual() # GridInvalidError: Duplicate nodes found
ux.open_grid(ds.chunk(), merge_coincident_nodes=False).get_dual() # 386 faces: the opt-out is ignoredProbably just need to save the setting:
@@ Grid.__init__
self._is_subset = is_subset
+ self._merge_coincident_nodes = merge_coincident_nodes
@@ Grid._prepare_for_dual
return type(self)(
grid_ds,
self.source_grid_spec,
source_dims_dict=self._source_dims_dict,
+ merge_coincident_nodes=self._merge_coincident_nodes,
)
Also, merge_coincident_nodes=False is now how users open a file that fails with "nonconsecutive repeated corners", but neither open_grid nor Grid.from_dataset lists it in their docstrings. The comment in Grid.__init__ (grid.py:194) also still calls it internal. Both functions take it through **kwargs, so a user can't find the name from the signature. A short parameter entry in both docstrings would fix this.
|
Thanks for the review and for catching this. Fixed in 673018d: deferred dual preparation now preserves the merge setting, and both entry points document the opt-out. |
cmdupuis3
left a comment
There was a problem hiding this comment.
The last bug is fixed, approving
Closes #865
geoflow-smallpreviously could not produce a dual at all and now yields 3840 faces; the test asserts this and fails on main._find_duplicate_nodesis vectorized withnp.uniqueinstead of a per-node dict.Grid.get_dual(check_duplicate_nodes=...)is now ignored and deprecated rather than removed, so existing callers keep working.UxDataArray.get_dualandUxDataset.get_dualstill raiseGridInvalidError, since node-centered data cannot be remapped onto a merged node set.(lon, lat), such as poles and the antimeridian, are handled by Merge coincident pole and antimeridian nodes in structured grids #1690, now merged into main.