Repository navigation
Filtering when reading a Grid file, avoid time leaking into the Grid - #1667
dylannelson wants to merge 17 commits into
Conversation
Time was getting into grid data, causing issues when subsetting down the line. This should clean the data in a way that shouldn't happen in the future
pre-commit run --files uxarray/io/_ugrid.py Seemed to turn up a few issues with spaces and line lengths
erogluorhan
left a comment
There was a problem hiding this comment.
Maybe too early for a review, but I'd like to share some thoughts:
-
timewas shown to exist in a grid from a user reading in a grid and data from a single fileA little more clarity could be helpful here. IIRC,
timeis not shown directly in theGridobject but in theGrid._ds. -
That said, I'd like to know what actual purpose we had with that object attribute, i.e.
_ds. If it was to keep complete track of the original file content throughxarray.Dataset, I'd say this dropping at the level of_dsmight break that. -
Maybe we'd want to fix it at the
Griditself rather than altering_ds, but would it be possible? -
Finally, ugrid (i.e.
_ugrid.py) is only one of the several formats we support. What about other formats in our I/O, e.g. _mpas.py etc. Their grid files can come in a similar way. Maybe we will need to do this fix at the Grid level rather than particular I/O modules?
#1444 — Solution historyThis is a bit about each version of the idea, why it's located where it is, and what the final code looks like. Version 1 —
|
Through _read_ugrid? |
Through Grid.__init__? |
_ds holds subgrid_*_indices? |
|
|---|---|---|---|
| Path A (file open) | Yes | Yes | No |
| Path B (subset) | No | Yes | Yes (needed by _slice_from_grid) |
- Filter in
_read_ugrid(V5): Path A only.timeis dropped as the grid is read; Path B never enters_read_ugrid, so itssubgrid_*_indicessurvive, and its sliced grid (copied from the already-cleangrid._ds) comes out clean for free. - Filter in
Grid.__init__: both paths. It would stripsubgrid_face_indicesfrom Path B's sliced grid →_slice_from_gridhits aKeyError. Fixes the leak but breaks subsetting.
Final code preview
Basically came down to two edits, both in uxarray/io/_ugrid.py.
Edit 1 — add one import at the top of the file (ugrid is already imported; only DESCRIPTOR_NAMES is new):
import numpy as np
import xarray as xr
import uxarray.conventions.ugrid as ugrid
from uxarray.constants import INT_DTYPE, INT_FILL_VALUE
from uxarray.grid.connectivity import _replace_fill_values
from uxarray.conventions.descriptors import DESCRIPTOR_NAMES # <---- this lineEdit 2 — filter at the end of _read_ugrid, then define the helper directly after it:
def _read_ugrid(ds):
"""Parses an unstructured grid dataset and encodes it in the UGRID
conventions."""
# ... (topology parse, coord/connectivity renames, dim swaps — unchanged) ...
dim_dict[ds["face_node_connectivity"].dims[1]] = ugrid.N_MAX_FACE_NODES_DIM
ds = ds.swap_dims(dim_dict)
# ===== NEW =====
# Strip non-grid extras (e.g. a stray scalar `time` coordinate, or unrelated data variables)
ds = _keep_only_grid_vars(ds)
# ===== end NEW =====
return ds, dim_dict
# ===== NEW helper — place directly after _read_ugrid =====
def _keep_only_grid_vars(ds):
"""Return ``ds`` with only recognized UGRID grid variables/coordinates.
Anything else on the dataset (a stray scalar ``time`` coordinate, unrelated
data variables, etc.) is dropped so it cannot leak onto ``grid._ds``.
Runs on the file-read path only.
"""
# uxarray's own canonical grid-variable names (the same lists Grid filters against)
keep = {"grid_topology"}
keep.update(ugrid.SPHERICAL_COORD_NAMES) # node/edge/face lon-lat
keep.update(ugrid.CARTESIAN_COORD_NAMES) # node/edge/face x-y-z
keep.update(ugrid.CONNECTIVITY_NAMES) # face_node_connectivity, edge_node_connectivity, ...
keep.update(DESCRIPTOR_NAMES) # n_nodes_per_face, face_areas, boundary_*_indices, ...
# drop_vars removes variables/coords by name (never bare dimensions), so grid
# dims survive with their variables; errors="ignore" tolerates absent names.
drop = [name for name in ds.variables if name not in keep]
return ds.drop_vars(drop, errors="ignore")
# ===== end NEW helper =====
# ===== existing code =====
def _encode_ugrid(ds):
"""Encodes an unstructured grid represented under a ``Grid`` object as a
``xr.Dataset`` with an updated grid topology variable."""
if "grid_topology" in ds:
ds = ds.drop_vars(["grid_topology"])
# ... (rest of _encode_ugrid, unchanged) ...
Sorry saw this after I was typing all the above content. Edit: Here's an update, more focused on # 4 to start This is a great question and I wish I had realized sooner. Trying to find a solution that was very high level, like in So I had to look at 2 questions:
Looking at # 1, I performed a similar check like before, read in a demo dataset, add time, save, reopen, check if there's time, and see if it crashes when there's a subset. Here's what I got
So basically from what I'm seeing right now, we have 3 different outcomes
I'll keep looking for a solution for # 2, but worried at it's impact, especially since every file type seems to have a unique way to handle the data when opening. |
|
The table in your latest comment showing no leaks for UGRID confused me a bit (From your other words, it sounds like UGRID has the time leaking). Could you clarify? |
|
@erogluorhan "the
|
It's no leak for UGRID in my current version, which fixed it. |
|
and for
that's effectively what I had originally, via The issue with this is that it fixes a symptom, without addressing the cause. We discussed this early on and it was turned down when we originally discussed it. It's basically a patch in the |
You're right; "without altering |
|
Other readers like MPAS, ESMF, Exodus, ICON, and FESOM2 still leave |
|
This new push should have everything we have wanted from the last few discussions extra_dims = set(grid_ds.dims) - set(DIM_NAMES)
grid_ds = grid_ds.drop_dims(extra_dims)The This solution is different in that it is dropping non-grid coordinates by name, which catches the scalar one that we've been having trouble with. Here's what this solution looks like instead: grid_coord_names = set(ugrid.SPHERICAL_COORD_NAMES) | set(ugrid.CARTESIAN_COORD_NAMES)
stray = [c for c in ds.coords if c not in grid_coord_names]
return ds.drop_vars(stray, errors="ignore")TestingNotebook Preview: Downstream impactPerformed a few downstream tests with claude code to test what affects it may have, here's the results of what it looked at:
|
ASV BenchmarkingBenchmark Comparison ResultsBenchmarks that have stayed the same:
Benchmarks that have got worse:
|
Sevans711
left a comment
There was a problem hiding this comment.
This looks like a relatively clean fix overall, with lots of tests run by hand to confirm things seem to be working!
I left a few minor comments inline, and I have a couple larger comments here:
- I had a suspicion that extra dimensions (not just extra coordinates) on input datasets would be an issue, but I tested a few things locally and found that it wasn't a problem (for some reason…). For example, if you had a time dimension on the input dataset, instead of just a time coordinate, it doesn't seem to lead to a crash later (although, it does leak into
Grid._ds). It isn't really obvious to me why it wouldn't crash…. Can you clarify why this might be the case, and/or add a test to ensure subsetting still works even after adding a time dimension to the grid? - Would you also be able to create a file with a time coordinate somewhere in the test suite (in meshfiles folder), and test that, too, instead of only having a test where the time coordinate gets assigned after loading the dataset? That would help me to be more convinced that this fully closes the original issue. It's a bit tricky to know for certain because the original issue report doesn't contain a complete verifiable example. But, the workflow there seemed to be "load from file, then subset (which crashes)" so ideally a test here could include that complete workflow as well.
| import uxarray.grid.slice as slice_module | ||
| from uxarray.grid.slice import _remap_dense, _remap_kernel, _remap_searchsorted | ||
|
|
||
| import numpy as np |
There was a problem hiding this comment.
remove repeated numpy import
There was a problem hiding this comment.
No clue how I missed that twice, thanks for the catch, should be gone
| from uxarray.core.dataarray import UxDataArray | ||
|
|
||
|
|
||
| def _drop_non_grid_coords(ds): |
There was a problem hiding this comment.
Relocate to uxarray/grid/utils.py or some other file, rather than defining this helper function directly in the main grid.py file
There was a problem hiding this comment.
Got it, should be moved there now
I agree with this, I think we shouldn't let extra dimensions either into the
I don't think we should expose a grid file with time coord/dim into our testing suite as it is uncommon to have such a presence and it would be misleading in the future. |
There was a problem hiding this comment.
Thanks very much for this fix! @Sevans711 has already made great points inline and overall ( see (1) here, but I believe (2) should be avoided), so I am approving it to make way once they're addressed.
Ah, for (2), I was going based on the file at That was leading me to think it would be reasonable enough to include a second version of this file which is extremely similar but just has a scalar time coordinate instead of a Time dimension, and then call Either way, I do still want to see evidence of the full original workflow being reproduced somewhere before I am ready to approve this as fully closing the original issue. Ideally this would involve an example of "load from file, then subset" crashing on |
|
@Sevans711
The summary of a path deep dive through many files and functions basically came down to:
Which happens about here it xarray: xarray/core/coordinates.py#L1214 Filtering dimensions seemed to be distinct from this problem (filtering coordinates) in my mind. Like I don't think we saw anything that indicates a time dimension could appear naturally like this. The downstream affects of filtering dimensions is different than coordinates too. If my understanding seems wrong, and a dimension filter here seems good, maybe we can scope this out over a call because there's lots of threads that have felt tangled while exploring this, and I don't want to overcompensate in this solution because it might introduce new problems. Maybe of note, that uxarray also won't report any Dims that it has that aren't in DIM_NAMES - which is a static list (in grid.py): @property
def dims(self) -> set:
"""Names of all unstructured grid dimensions."""
from uxarray.conventions.ugrid import DIM_NAMES
return set([dim for dim in DIM_NAMES if dim in self._ds.dims])so it shouldn't ever really report anything outside of:
|
There was a problem hiding this comment.
As far as I could understand from the whole conversation in this PR, especially the last parts of it, maybe we got a little too focused/concerned on fixing the reported case only, i.e. scalar-only coords colliding during isel() when .subset is called? I think the fix should be different, more comprehensive, than that, and I have a few thoughts on what that pathway should look like:
- The main problem is when there is a non-spatial coord leakage into the
Grid._ds, whether scalar or dimensional, we should clean it up before wrapping up the Grid object. - We should avoid trying to fix this at the low-level I/O modules (
_mpas,_icon, etc.) That is to say, the current approach (_drop_non_grid_coordsbeing called inGridconstructor instead) makes sense to me. - I don't agree with dropping only scalar coords. The actual overarching fix here should rather cover dimensional coords as well. It should actually avoid the unwanted removal of any spatial coords, e.g.
vlonetc. in ICON. That said,_drop_non_grid_coords's current version focusing only on scalar coords seems incorrect to me. Also, avoiding the unwanted removal of spatial coords should easily be achievable because such coords should have the size of the number of either geometric element (nodes, faces, or edges).
To wrap up the above thoughts, I've put together a notebook (pr_1667_grid_filter.html) where I test our main branch for:
(1) a MPAS case with a single-file that has both grid and data in it,
(2) Quad Hexagon grid and data files,
(3) A single-file ICON case
All of these cases use files from our testing suite. I used the previous version of _drop_non_grid_coords that removed dimensional coords (i.e. not only scalar ones). I put explanatory comments into each cell, but in summary:
(1) has no leakage for this particular MPAS case, but since I was able to see Time leakage in the other MPAS case @dylannelson was pointing out priorly, it seems safe to solve potential problem at the Grid level as this PR currently attempts, rather than making model output-based assumptions.
(2) With the expanded Quad Hexagon case, I showed that in a single file scenario where data variables have dimensional coords but not necessarily the grid variables, we might see dimensional Time coord leaking into Grid. So, fixing only scalar coords maynot be a complete fix.
3. Spatial coords such as vlon etc. are also dropped after _drop_non_grid_coords call. Solution should be check for each coord's size, and if it is equal to the number of either faces, edges, or nodes, don't remove it.
|
@Sevans711 |
Thank you for working through this! I agree it is quite difficult to reproduce without more details about the original issue. This notebook might be as close as possible to reproducing without having more information. I am still a little suspicious about these lines: ds = xr.open_dataset("demo_single_file.nc").isel(time=0)
uxds = ux.open_dataset(ds, "demo_single_file.nc")To me it feels unlikely that something like "call .isel() on the grid dataset then pass that into the uxarray open_dataset call" would have happened in a user workflow. But, at this point I also can't say that that isn't what happened, either. For the purposes of "does this PR close the original issue?" your efforts here have convinced me it is reasonable to move forward assuming the answer is yes. (If not, it is always possible for another issue to be posted later, after all.) That said, what would your thoughts be on opening a new issue which this PR would close as well, something like "Grid should include only grid-related information"? I think that could be a good way to clarify why this PR is going through so many iterations, to me it is sounding like @erogluorhan is asking for an issue like that to be solved here too. Is that accurate? I also have a minor note about this:
I suspect it would probably be cleaner to check for each coord's dims instead of sizes; only keep coords which have grid dims (e.g. n_face). This could avoid spawning another issue like #1616 in the future. Is there any case where this wouldn't work out? |
|
After further discussion, the team proposed a solution that covers more cases to be certain no other cases of data leaking could occur as well, and not be specific to scalar. The approach looks like:
Basically we check to make sure everything is related to: ['n_edge', 'n_face', 'n_max_edge_edges', 'n_max_edge_faces',
'n_max_edge_nodes', 'n_max_face_edges', 'n_max_face_faces',
'n_max_face_nodes', 'n_max_node_edges', 'n_max_node_faces',
'n_node', 'two']and if not, it gets removed. Testing this logic on a variety of files with grids, we get some outliers: mpas/QU/480/grid.nc [spec=MPAS]
Variable 'boundary_node_indices' would be REMOVED : non-grid dim(s) ['n_boundary_nodes'] (dims=('n_boundary_nodes',))
mpas/QU/mesh.QU.1920km.151026.nc [spec=MPAS]
Variable 'boundary_node_indices' would be REMOVED : non-grid dim(s) ['n_boundary_nodes'] (dims=('n_boundary_nodes',))
mpas/QU/oQU480.231010.nc [spec=MPAS]
Variable 'boundary_node_indices' would be REMOVED : non-grid dim(s) ['n_boundary_nodes'] (dims=('n_boundary_nodes',))
ugrid/fesom/fesom.mesh.diag.nc [spec=UGRID]
Variable 'nod_area' would be REMOVED : non-grid dim(s) ['nz'] (dims=('nz', 'n_node'))
Variable 'edge_node_connectivity' would be REMOVED : non-grid dim(s) ['n2'] (dims=('n_edge', 'n2'))
Variable 'edge_face_connectivity' would be REMOVED : non-grid dim(s) ['n2'] (dims=('n_edge', 'n2'))
Variable 'nod_in_elem2D' would be REMOVED : non-grid dim(s) ['N'] (dims=('N', 'n_node'))
Variable 'edge_cross_dxdy' would be REMOVED : non-grid dim(s) ['n4'] (dims=('n4', 'n_edge'))
Coordinate 'nz' would be REMOVED : non-grid dim(s) ['nz'] (dims=('nz',))
Coordinate 'nz1' would be REMOVED : non-grid dim(s) ['nz1'] (dims=('nz1',))Some of these look relevant, like |
|
I think we are on the same page but just to clarify, I think the steps are:
Would you be able to also open a corresponding issue to track this? It might be easier to refer back to this in the future if these details get discussed on an issue focused on this topic, to make sure they don't end up getting buried inside this PR discussion (if the PR discussion continues to grow).
Thank you for checking this ahead of time, these feel like important cases to consider! Clarifying, is this function being applied directly to the data from the file, or after the data has already been parsed by the relevant uxarray io routines? For example, I would have guessed that |
Sevans711
left a comment
There was a problem hiding this comment.
Thank you for the thorough investigation and updates here! The tests look reasonable by-eye, and I ran some of them locally against main to confirm they fail on main but are fixed by this PR.
I would prefer to just always drop non grid data during Grid.__init__, instead of introducing an opt-out option which gets added to the public API and woven throughout various parts of the codebase. My understanding of uxarray developer team preferences in general is that there is usually a hesitance to add new options to the public API (see e.g. the discussion in #1566).
Additionally, could you clarify a bit further what is the motivation for dropping scalar data variables? My understanding was that they weren't getting in the way of anything. Dropping them also led to needing to whitelist a few in particular: "grid_topology", "max_face_radius", plus a helper function for CRS variables (are these always scalars too? I might be misunderstanding the CRS variables case). The whitelist would be an ongoing maintenance burden whenever any new features are added which cache a scalar result on the grid. If the scalar data variables really aren't getting in the way of anything, my preference would be to just keep all of them, instead of building and maintaining a whitelist for them.
| set(ugrid.SPHERICAL_COORD_NAMES) | ||
| | set(ugrid.CARTESIAN_COORD_NAMES) | ||
| | set(ugrid.CONNECTIVITY_NAMES) | ||
| | set(DESCRIPTOR_NAMES) |
There was a problem hiding this comment.
What is the point of whitelisting these explicitly? Aren't they all being kept automatically, because they all correspond to variables with only grid dimensions?
If it is possible to claim the simpler invariant of "only keep variables with only grid dims, plus a few exceptions" by dropping these from keep, instead of needing to have code like "only keep [long list of whitelisted variables], plus variables with only grid dims, plus a few exceptions", I think that would be preferable.
| "bounds", | ||
| "face_bounds_lon", | ||
| "face_bounds_lat", |
There was a problem hiding this comment.
I suspect that these three ("bounds", "face_bounds_lon", "face_bounds_lat") would all be kept automatically if "min_max" and "lat_lon" were added to the list of dimensions being kept. (Locally, something like grid_dims = grid_dims + {"min_max", "lat_lon"}.) To me this would make more sense and represent a lower ongoing maintenance burden than explicitly whitelisting these three variables.
E.g., maybe a new feature like "face_bounds_x" might be added some day, with a "min_max" dimension. If the rule here is to consider "min_max" dimension as something to keep, there would be no need to touch this part of the code, but if it's just an explicit whitelist then "face_bounds_x" would need to be added here too.
There was a problem hiding this comment.
+1. I checked it: with lat_lon and min_max counted as grid dims, re-reading a grid that already has bounds and face_bounds_lon keeps them without listing names.
| "bounds", | ||
| "face_bounds_lon", | ||
| "face_bounds_lat", | ||
| "max_face_radius", |
There was a problem hiding this comment.
For scalar vars like "grid_topology" and "max_face_radius" which need to be whitelisted, I would recommend defining the list somewhere in the conventions folder instead of here, and import it, similarly to ugrid.DIM_NAMES.
There was a problem hiding this comment.
Actually… this, and the exceptions with CRS variables below, is only relevant if the decision is to drop scalar data variables. Wouldn't it be easier to reason through and maintain if we just keep all scalar data variables? (Still drop scalar coordinates, though.) Leaving a note in my overall review about this to clarify.
There was a problem hiding this comment.
+1. The test I suggested on test_non_grid_data.py:30 shows subsetting works with a scalar time data variable, even on main. See my review summary.
|
@Sevans711 I see a lot of the comments above are about whitelisting and how it seems like a burden or a pain to manage, and while I agree, I don't think we can make such broad and strict rules like "Remove all of __" without making exceptions, due to the wide variance of data and models coming in, and how they are used at various points in the code. Since the beginning of issue, I've expressed concerns with putting it in
This one I don't understand though, because I thought this has always been a part of the problem with time. Whether a stray scalar time reads back as a coordinate or a data variable depends only on how the file was written. It’s a coordinate if another variable’s coordinates attribute lists it, and a variable otherwise. If you want to remove this rule and it's own whitelist that's fine, it's just going to let in more data, which is part of what we're trying to avoid. The API concerns I get too, but I think it was a very rare but potentially critical safety net for if the whitelist method doesn't let something through and the user says "Hey this variable/dim didn't get through, but I need it right now" and whatever model they're using can quickly suggest turning off the cleaning so it gets through, without having to wait for us to add it to a model specific exception and make a release. Just like how this entire fix is only for weird unexpected scenarios, I also thought it could be also be good to have a backdoor for other weird and unexpected scenarios. We can cut it out, but I think it could result in a user being unable to proceed because this growing list of filters stopped something they needed to get through. I was told early on we shouldn't even have thing like If you want to take a shot at making a solution that could work with all edge cases and without a whitelist, send me what that could look like, but I haven't really found or thought of anything that works like that. I could respond to each of the questions above, but don't want to continue to do so if the whole idea is going to be scrapped and rescoped again. |
cmdupuis3
left a comment
There was a problem hiding this comment.
- A grid opted out with a stray time coordinate raises IndexError on isel. Also, a grid coordinate on n_face (lev) shows up in data subsets.
in dataarray.py:_slice_from_grid, add .data:
if self._face_centered():
da_sliced = self.isel(
- n_face=sliced_grid._ds["_subgrid_face_indices"], ignore_grid=True
+ n_face=sliced_grid._ds["_subgrid_face_indices"].data, ignore_grid=True
)
elif self._edge_centered():
da_sliced = self.isel(
- n_edge=sliced_grid._ds["_subgrid_edge_indices"], ignore_grid=True
+ n_edge=sliced_grid._ds["_subgrid_edge_indices"].data, ignore_grid=True
)
elif self._node_centered():
da_sliced = self.isel(
- n_node=sliced_grid._ds["_subgrid_node_indices"], ignore_grid=True
+ n_node=sliced_grid._ds["_subgrid_node_indices"].data, ignore_grid=True
)
- geoflow-small loses mesh_depth from to_xarray(), and no warning is raised.
import warnings, uxarray as ux
path = "test/meshfiles/ugrid/geoflow-small/grid.nc"
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter("always")
new = ux.open_grid(path) # branch default
old = ux.open_grid(path, drop_non_grid_data=False) # == main
sorted(set(old.to_xarray().variables) - set(new.to_xarray().variables)) # ['mesh_depth']
[str(x.message) for x in w if "drop" in str(x.message).lower()] # []Recommend defaulting drop_non_grid_data to false:
# uxarray/core/api.py open_grid(...)
- drop_non_grid_data: bool = True,
+ drop_non_grid_data: bool = False,
# uxarray/grid/grid.py Grid.__init__(...)
- drop_non_grid_data: bool = True,
+ drop_non_grid_data: bool = False,
# uxarray/grid/grid.py Grid.from_dataset(...)
- drop_non_grid_data = kwargs.get("drop_non_grid_data", True)
+ drop_non_grid_data = kwargs.get("drop_non_grid_data", False)
- Speed issue with
ds[name]lookups in_drop_non_grid_data, because every call materializes a new dataarray; recommend iterating throughds.variablesinstead.
|
@dylannelson, the back-and-forth on this PR is mostly on me. I approved on Aug 20 and reversed myself the next morning. I pushed back on the This wasn't a normal PR: we were learning the problem while growing the fix, and you and then @Sevans711's consistent review carried most of that. So instead of adding another opinion, I'll make the decision that unblocks the rest (review below). You've asked for a call a few times. Until reading below, if you still need that, let's do 30 minutes this week and close it out. |
erogluorhan
left a comment
There was a problem hiding this comment.
Here is my call on the open design questions. With these three changes I think this is ready:
- Filter once, on read. Call
_drop_non_grid_dataat the end of the auto-detect branch ofGrid.from_dataset, not inGrid.__init__. That branch runs after every_read_*, so it is still one format-agnostic place (my Aug 11 point 4). It's also what @rajeeja suggested on Aug 17 and @Sevans711 on Aug 26. Copies and subsets no longer re-filter, so thebounds/face_bounds_*/max_face_radiusentries and theslice.py/copy()plumbing go away. - Keep scalar data variables, drop only scalar coordinates (@Sevans711's rule). Only coordinates are attached to the subset indexer, so a scalar
timedata variable can't cause #1444; the test suggested inline checks this. Adding @Sevans711'slat_lon/min_maxdims meansgrid_topologyandmax_face_radiusdon't need listing either. - No
drop_non_grid_dataoption. We build the grid, and users shouldn't need to know or choose what we keep from the source file. The option is also only partly wired:from_topology,from_pointsand the other constructors ignore it, and 4 of its 9 hand-offs are untested. With 1 and 2 in place, there is little left for it to guard against.
I prototyped this (diff below). The filter becomes "grid dims plus lat_lon/min_max". It still has to keep descriptors (MPAS boundary_node_indices) and the grid_mapping CRS variable (needed for rioxarray's scalar spatial_ref coordinate). On every grid in test/meshfiles it keeps exactly what this branch keeps. The #1444 tests still pass, and in the full suite only the opt-out test and the scalar_data_var case fail, as expected.
Housekeeping:
- Merge
main; the branch has conflicts now. - In the description: remove the stray
(after "Closes #1444" and add "Closes #1720". Also note thatGrid._ds/to_xarray()no longer carry non-grid variables (e.g. geoflowmesh_depth, FESOM2nod_area), so it gets into the compatibility section of the release notes. - @rajeeja, you were right about ICON/FESOM2.
clon/clat/vlon/...are kept now, andtest_grid_fileschecks it. Could you withdraw or update your Aug 20 change request?
Prototype diff (filter placement and rule only; removing the option, its plumbing and the test updates are not included)
--- a/uxarray/grid/grid.py
+++ b/uxarray/grid/grid.py
@@ -203,7 +203,7 @@
# drop variables that don't belong on the grid (e.g. a `time` carried in from a
# file holding both grid and data), so they can't collide during subsetting.
self._drop_non_grid_data = drop_non_grid_data
- self._ds = _drop_non_grid_data(grid_ds) if drop_non_grid_data else grid_ds
+ self._ds = grid_ds
# source grid specification (i.e. UGRID, MPAS, SCRIP, etc.)
self.source_grid_spec = source_grid_spec
@@ -325,6 +325,7 @@
raise GridInvalidError(
f"Unsupported source_grid_spec={source_grid_spec!r}, in {cls.__name__}.from_dataset()"
)
+ grid_ds = _drop_non_grid_data(grid_ds)
else:
# custom source grid spec is provided
source_grid_spec = kwargs.get("source_grid_spec", None)
--- a/uxarray/grid/utils.py
+++ b/uxarray/grid/utils.py
@@ -17,46 +17,22 @@
def _drop_non_grid_data(ds):
- """Drop variables that don't belong on a ``Grid`` (e.g. a ``time`` carried in
- from a file that holds both the grid and its data).
-
- A variable or coordinate is dropped when it has no dimensions, or when any of
- its dimensions is not a UGRID grid dimension (``ugrid.DIM_NAMES``).
-
- Variables uxarray reads or caches on the grid are always kept, whatever their
- dimensions: grid coordinates, connectivity, descriptors (including MPAS
- ``boundary_node_indices``), the cached ``bounds``, ``face_bounds_lon``,
- ``face_bounds_lat`` and ``max_face_radius``, ``grid_topology``, and any CRS
- variable named by a ``grid_mapping`` attribute (read by projected-grid
- detection).
- """
- grid_dims = set(ugrid.DIM_NAMES)
-
- keep = (
- set(ugrid.SPHERICAL_COORD_NAMES)
- | set(ugrid.CARTESIAN_COORD_NAMES)
- | set(ugrid.CONNECTIVITY_NAMES)
- | set(DESCRIPTOR_NAMES)
- | {
- "grid_topology",
- "bounds",
- "face_bounds_lon",
- "face_bounds_lat",
- "max_face_radius",
- }
- )
- # CRS variables, referenced by name from a ``grid_mapping`` attribute
- # (CF also allows the extended form "crs: x y")
- for name in ds.variables:
- grid_mapping = ds[name].attrs.get("grid_mapping")
- if isinstance(grid_mapping, str):
- keep.update(token.rstrip(":") for token in grid_mapping.split())
-
+ """Drop variables and coordinates with a non-grid dimension, and scalar coordinates."""
+ grid_dims = set(ugrid.DIM_NAMES) | {"lat_lon", "min_max"}
+ # descriptors (MPAS boundary_node_indices) and CRS variables named by grid_mapping
+ keep = set(DESCRIPTOR_NAMES) | {
+ var.attrs["grid_mapping"].split(":")[0]
+ for var in ds.variables.values()
+ if isinstance(var.attrs.get("grid_mapping"), str)
+ }
drop = [
name
- for name in ds.variables
+ for name, var in ds.variables.items()
if name not in keep
- and (ds[name].ndim == 0 or any(dim not in grid_dims for dim in ds[name].dims))
+ and (
+ any(dim not in grid_dims for dim in var.dims)
+ or (var.ndim == 0 and name in ds.coords)
+ )
]
return ds.drop_vars(drop)
| # drop variables that don't belong on the grid (e.g. a `time` carried in from a | ||
| # file holding both grid and data), so they can't collide during subsetting. | ||
| self._drop_non_grid_data = drop_non_grid_data | ||
| self._ds = _drop_non_grid_data(grid_ds) if drop_non_grid_data else grid_ds |
There was a problem hiding this comment.
Move this call to the end of the auto-detect branch in from_dataset, right after the _read_* chain (see summary). In my prototype, test_bounding_box_with_stray_grid_time and test_single_file_with_grid_and_time_varying_data pass unchanged. test_stray_variables_are_dropped only needs ux.Grid.from_dataset(...) instead of ux.Grid(...).
| "coord_mixing_grid_and_non_grid_dims": lambda ds: ds.assign_coords( | ||
| z=(("n_node", "elevation"), np.zeros((ds.sizes["n_node"], 3))) | ||
| ), | ||
| "scalar_data_var": lambda ds: ds.assign( |
There was a problem hiding this comment.
Instead of asserting that a scalar time data variable is dropped, could this test assert that subsetting works with it present? It passes on main, on this branch and on the prototype:
grid_ds = xr.open_dataset(gridpath("ugrid", "quad-hexagon", "grid.nc"))
uxgrid = ux.open_grid(grid_ds.assign(time=xr.DataArray(np.datetime64("2016-01-01"))))
uxda = ux.UxDataArray(np.ones((2, uxgrid.n_face)), dims=("time", "n_face"),
coords={"time": TIMES}, uxgrid=uxgrid)
res = uxda.subset.bounding_box((-10, 10), (-10, 10))
np.testing.assert_array_equal(res["time"].values, TIMES)If it ever fails, rule 4 was right and we keep it.
| assert not set(dropped) & set(uxgrid._ds.variables) | ||
|
|
||
|
|
||
| def test_known_variable_on_misnamed_dim_kept(quad_hexagon_ds): |
There was a problem hiding this comment.


Closes #1444 (
Overview
timewas shown to exist in a grid from a user reading in a grid and data from a single file.subsetthere was a crash related to time being different on the grid vs the datatimeshouldn't realistically be on the grid to start with, so this was a concern, and was causing downstream issues not addressed when reading in the dataGrid.__init__or_read_ugrid, both were tested extensivelyGrid.__init__is run in multiple scenarios, including when a file is read from disk, but_slice_from_gridit can cause issues.timebecause it wasn't a dimension_read_ugridoccurs primarily when a file is read from disk_read_ugrid(ds)only solved the the issue for that file typeGrid.__init__as it solved the issue in all data types, but required a different approachtimegone from grid coords and grid variablesnode_lon,node_lat,face_node_connectivitypreservedn_node,n_face> 0)bounding_boxruns, no errortimepreservedsubgrid_face_indicessel(time=...)still workstimeaxis + coord strippedspatial_ref, which wasn't seen in any of the test data, but if it exists, needs to be read by_is_projected_gridincoordinates.pyChanges Made:
_drop_non_grid_data_drop_non_grid_datafor cleaning stray coordinates_drop_non_grid_datainGrid.__init__Grid.__init__Grid.from_datasetopen_gridslice.pyWe are effectively following these rules now:
timein some circumstances, which was a primary goal of this PRSPHERICAL_COORD_NAMES,CARTESIAN_COORD_NAMES): node_lon, face_x, etcCONNECTIVITY_NAMES): face_node_connectivity, face_edge_connectivity, etcDESCRIPTOR_NAMES): face_areas, n_nodes_per_face, etcbounds,face_bounds_lon,face_bounds_lat,max_face_radius, etcgrid_mappingattributeExpected Usage
This cleaning will occur during any and all grid creations
PR Checklist
General
Testing & Benchmarking
Documentation
docs/api.rst_)AI Disclosure
AI Usage:
Claude Code