Skip to content

Filtering when reading a Grid file, avoid time leaking into the Grid - #1667

Open
dylannelson wants to merge 17 commits into
mainfrom
dylannelson/time-filtering
Open

dylannelson wants to merge 17 commits into
mainfrom
dylannelson/time-filtering

Conversation

@dylannelson

@dylannelson dylannelson commented Aug 11, 2026 •

Copy link
Copy Markdown
Member

Closes #1444 (

Overview

  1. time was shown to exist in a grid from a user reading in a grid and data from a single file
  2. Later when the user tried to use .subset there was a crash related to time being different on the grid vs the data
  3. time shouldn't realistically be on the grid to start with, so this was a concern, and was causing downstream issues not addressed when reading in the data
  4. There was 2 main ideas of where to place filtering to avoid this, Grid.__init__ or _read_ugrid, both were tested extensively
    • Grid.__init__ is run in multiple scenarios, including when a file is read from disk, but
      • in other scenarios like _slice_from_grid it can cause issues.
      • This solution also wasn't properly removing the time because it wasn't a dimension
    • the path to _read_ugrid occurs primarily when a file is read from disk
      • _read_ugrid(ds) only solved the the issue for that file type
    • We went back to Grid.__init__ as it solved the issue in all data types, but required a different approach
  5. This has been tested in multiple ways and is shown to:
    • Scalar time gone from grid coords and grid variables
    • node_lon, node_lat, face_node_connectivity preserved
    • Grid non-empty (n_node, n_face > 0)
    • bounding_box runs, no error
    • Data time preserved
    • Subset returned faces
    • Sliced grid keeps subgrid_face_indices
    • sel(time=...) still works
    • Dimensional time axis + coord stripped
    • Leaked data variable stripped
  6. Following further requests made in Sam's comment below, we added further scope to account for nearly any kind of other data that could come in that we may not want, even scenarios we haven't seen
  7. Handled a few edge cases with the new rules above, which required a new list to be made to allow models with unique data that can't/wouldn't normally be created within UXarray
  8. Added an optional argument in the API to turn of this cleaning if there is ever data we didn't account for, but the user wants/needs for whatever reason. I expect this to be very unlikely to be used by a typical user, as most users aren't accessing data directly on the grid, but this could give a workaround while we handle a patch or to allow them to keep the data on the grid when they export (may be useful for if they want to compare grids before/after changes without worrying about the cleaning affecting results). This seemed potentially more revevant as I saw more and more instances of model specific data being needed for it's particular IO, and removing it could result in downstream issues. There was a similar case with the CRS coordinate spatial_ref, which wasn't seen in any of the test data, but if it exists, needs to be read by _is_projected_grid in coordinates.py

Changes Made:

  1. Added test for _drop_non_grid_data
  2. Added function _drop_non_grid_data for cleaning stray coordinates
  3. Called _drop_non_grid_data in Grid.__init__
  4. Connected the parts of the API where this function can occur, or where it needs info from when/if it was run. It occurs in 4 places:
    • Grid.__init__
    • Grid.from_dataset
    • open_grid
    • slice.py

We are effectively following these rules now:

  1. Drop any data variable that has a dim not in DIM_NAMES.
  2. Drop any coordinate that has a dim not in DIM_NAMES.
  3. Drop any scalar coordinate.
  4. Drop some scalar data variables (changed)
    • The three scalar data variables uxarray needs (grid_topology, max_face_radius, and a CRS variable) are kept
    • Not removing any scalar data variables results in letting in time in some circumstances, which was a primary goal of this PR
    • We can revert this change easily of needed, but scalar data variables seem to be a common way for unwanted data to get in from what we've seen
  5. Never drop a variable on our defined lists/conventions (new)
    • Grid coordinate names (SPHERICAL_COORD_NAMES, CARTESIAN_COORD_NAMES): node_lon, face_x, etc
    • Connectivity names (CONNECTIVITY_NAMES): face_node_connectivity, face_edge_connectivity, etc
    • Descriptor names (DESCRIPTOR_NAMES): face_areas, n_nodes_per_face, etc
    • Cached values: bounds, face_bounds_lon, face_bounds_lat, max_face_radius, etc
    • CRS variables: any variable named in another variable’s grid_mapping attribute

Expected Usage

This cleaning will occur during any and all grid creations

PR Checklist

General

  • An issue is created and linked
  • Added appropriate labels (if your uxarray repo permissions allow it)
  • Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

  • Adequate tests are created if there is new functionality
  • Tests are not too basic (such as simply calling a function and nothing else)
  • Tests cover all major paths in your new functions
  • If this PR could affect performance, ran ASV benchmarks and confirmed they show expected behavior (add a new benchmark if necessary)

Documentation

  • [N/A] Docstrings have been added to all new functions
  • [N/A] Docstrings have been updated with any function changes
  • [N/A] User (public) functions have been added to docs/api.rst
  • Internal (private) function names start with an underscore (_)

AI Disclosure

AI Usage:
Claude Code

  • I take responsibility for all AI-generated content in my PR.
  • I have tested all AI-generated content in my PR.

Time was getting into grid data, causing issues when subsetting down the line. This should clean the data in a way that shouldn't happen in the future
pre-commit run --files uxarray/io/_ugrid.py
Seemed to turn up a few issues with spaces and line lengths

@erogluorhan erogluorhan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe too early for a review, but I'd like to share some thoughts:

  1. time was shown to exist in a grid from a user reading in a grid and data from a single file

    A little more clarity could be helpful here. IIRC, time is not shown directly in the Grid object but in the Grid._ds.

  2. That said, I'd like to know what actual purpose we had with that object attribute, i.e. _ds. If it was to keep complete track of the original file content through xarray.Dataset, I'd say this dropping at the level of _ds might break that.

  3. Maybe we'd want to fix it at the Grid itself rather than altering _ds, but would it be possible?

  4. Finally, ugrid (i.e. _ugrid.py) is only one of the several formats we support. What about other formats in our I/O, e.g. _mpas.py etc. Their grid files can come in a similar way. Maybe we will need to do this fix at the Grid level rather than particular I/O modules?

@dylannelson

Copy link
Copy Markdown
Member Author

#1444 — Solution history

This is a bit about each version of the idea, why it's located where it is, and what the final code looks like.


Version 1 — .values guard in _slice_from_grid

# uxarray/core/dataarray.py — _slice_from_grid()
n_face=sliced_grid._ds["subgrid_face_indices"].values   # was: [...["subgrid_face_indices"]]
  • Location: within _slice_from_grid in core/dataarray.py.
  • Idea: Pass the face indexer as .values (a plain array) so xarray does positional indexing with no coordinate alignment. It fixes the crash for every subset method, but only guards the symptom — the stray time stays on the grid, so it never addresses why the grid carries a time.

Version 2 — Name-based clean build in _read_ugrid

# uxarray/io/_ugrid.py — end of _read_ugrid()
ds = _keep_only_grid_vars(ds)   # keep only recognized grid vars/coords, drop the rest
  • Location: at the end of _read_ugrid in io/_ugrid.py.
  • Idea: Filter the parsed dataset down to recognized grid variables, dropping extras where the grid is built. This removes the stray time at its source (the leaky reader), so nothing downstream can collide. It's essentially the final answer — a simpler-looking alternative was proposed next, which prompted the comparison below.

Version 3 — drop_dims in Grid.__init__ (team's idea)

# uxarray/grid/grid.py — Grid.__init__
extra_dims = set(grid_ds.dims) - set(DIM_NAMES)
grid_ds = grid_ds.drop_dims(extra_dims)
  • Location: at the top of Grid.__init__ in grid/grid.py.
  • Idea: Drop any dimension not in DIM_NAMES, centrally for every reader, to remove stray non-grid data by axis. It captured a dimensional time, but the leaked time here is a scalar (0-d) coord with no dimension — so drop_dims finds nothing to remove and the bug survives.

Version 4 — "Best of both" (drop_dims + 0-d/name drop)

grid_ds = grid_ds.drop_dims(set(grid_ds.dims) - set(DIM_NAMES))              # dimensional extras
drop = [n for n, v in grid_ds.variables.items() if n not in KEEP and v.ndim == 0]
grid_ds = grid_ds.drop_vars(drop, errors="ignore")                          # scalar extras
  • Location: at the top of Grid.__init__ in grid/grid.py.
  • Idea: Keep V3's drop_dims and add a second pass dropping 0-d non-grid vars, catching the scalar time it missed. It works, but uses two mechanisms for what one can do — V5's name-based keep-list covers scalar, dimensional, and data-var leaks in a single pass.

Version 5 — Single name-based keep-list in _read_ugrid (final)

# uxarray/io/_ugrid.py — end of _read_ugrid()
def _keep_only_grid_vars(ds):
    keep = {"grid_topology", *SPHERICAL_COORD_NAMES, *CARTESIAN_COORD_NAMES,
            *CONNECTIVITY_NAMES, *DESCRIPTOR_NAMES}
    return ds.drop_vars([n for n in ds.variables if n not in keep], errors="ignore")
  • Location: at the end of _read_ugrid in io/_ugrid.py — different location (see next section).
  • Idea: One name-based keep-list that drops every kind of non-grid extra (scalar, dimensional, data vars) in a single pass, collapsing V2–V4 into the simplest complete fix. Could be paired with the .values option, but didn't seem necessary.

Why V5 lives in _read_ugrid, not Grid.__init__

Two code paths build a Grid. They share Grid.__init__ but not _read_ugrid, so why is it in _read_ugrid?

Path A — opening a grid from a file (where the time leaks in):

ux.open_grid(...)                   core/api.py
  └─ _open_dataset_with_fallback()  # full xr.Dataset — stray `time` included
  └─ Grid.from_dataset(...)         grid.py
       └─ no source_grid_spec
       └─ _parse_grid_type()
       └─ _read_ugrid(...)          io/_ugrid.py   ← renames in place, returns EVERYTHING (leak enters here)
  └─ Grid.__init__(...)             grid.py   # self._ds = grid_ds

Path B — building a sliced grid during a subset (trying not to disturb this path):

uxda.subset.bounding_box(...)
  └─ Grid.isel(...)                    grid.py
       └─ _slice_face_indices(...)     grid/slice.py
            ds = grid._ds.isel(...)    # copy of cleaned grid
            ds["subgrid_face_indices"] = ...
       └─ Grid.from_dataset(...)       slice.py
            └─ grid_ds = ds            (_read_ugrid is skipped)
  └─ Grid.__init__(...)                grid.py   # self._ds holds subgrid_*_indices
Through _read_ugrid? Through Grid.__init__? _ds holds subgrid_*_indices?
Path A (file open) Yes Yes No
Path B (subset) No Yes Yes (needed by _slice_from_grid)
  • Filter in _read_ugrid (V5): Path A only. time is dropped as the grid is read; Path B never enters _read_ugrid, so its subgrid_*_indices survive, and its sliced grid (copied from the already-clean grid._ds) comes out clean for free.
  • Filter in Grid.__init__: both paths. It would strip subgrid_face_indices from Path B's sliced grid → _slice_from_grid hits a KeyError. Fixes the leak but breaks subsetting.

Final code preview

Basically came down to two edits, both in uxarray/io/_ugrid.py.

Edit 1 — add one import at the top of the file (ugrid is already imported; only DESCRIPTOR_NAMES is new):

import numpy as np
import xarray as xr

import uxarray.conventions.ugrid as ugrid
from uxarray.constants import INT_DTYPE, INT_FILL_VALUE
from uxarray.grid.connectivity import _replace_fill_values
from uxarray.conventions.descriptors import DESCRIPTOR_NAMES  # <---- this line

Edit 2 — filter at the end of _read_ugrid, then define the helper directly after it:

def _read_ugrid(ds):
    """Parses an unstructured grid dataset and encodes it in the UGRID
    conventions."""

    # ... (topology parse, coord/connectivity renames, dim swaps — unchanged) ...

    dim_dict[ds["face_node_connectivity"].dims[1]] = ugrid.N_MAX_FACE_NODES_DIM

    ds = ds.swap_dims(dim_dict)

    # ===== NEW =====
    # Strip non-grid extras (e.g. a stray scalar `time` coordinate, or unrelated data variables)
    ds = _keep_only_grid_vars(ds)
    # ===== end NEW =====

    return ds, dim_dict


# ===== NEW helper — place directly after _read_ugrid =====
def _keep_only_grid_vars(ds):
    """Return ``ds`` with only recognized UGRID grid variables/coordinates.

    Anything else on the dataset (a stray scalar ``time`` coordinate, unrelated
    data variables, etc.) is dropped so it cannot leak onto ``grid._ds``.

    Runs on the file-read path only.
    """
    # uxarray's own canonical grid-variable names (the same lists Grid filters against)
    keep = {"grid_topology"}
    keep.update(ugrid.SPHERICAL_COORD_NAMES)   # node/edge/face lon-lat
    keep.update(ugrid.CARTESIAN_COORD_NAMES)   # node/edge/face x-y-z
    keep.update(ugrid.CONNECTIVITY_NAMES)      # face_node_connectivity, edge_node_connectivity, ...
    keep.update(DESCRIPTOR_NAMES)              # n_nodes_per_face, face_areas, boundary_*_indices, ...

    # drop_vars removes variables/coords by name (never bare dimensions), so grid
    # dims survive with their variables; errors="ignore" tolerates absent names.
    drop = [name for name in ds.variables if name not in keep]
    return ds.drop_vars(drop, errors="ignore")
# ===== end NEW helper =====


# ===== existing code =====
def _encode_ugrid(ds):
    """Encodes an unstructured grid represented under a ``Grid`` object as a
    ``xr.Dataset`` with an updated grid topology variable."""

    if "grid_topology" in ds:
        ds = ds.drop_vars(["grid_topology"])
    # ... (rest of _encode_ugrid, unchanged) ...

@dylannelson

dylannelson commented Aug 11, 2026 •

Copy link
Copy Markdown
Member Author

@erogluorhan

4. Finally, ugrid (i.e. `_ugrid.py`) is only one of the several formats we support. What about other formats in our I/O, e.g. _mpas.py etc. Their grid files can come in a similar way. Maybe we will need to do this fix at the Grid level rather than particular I/O modules?

Sorry saw this after I was typing all the above content.

Edit: Here's an update, more focused on # 4 to start

This is a great question and I wish I had realized sooner. Trying to find a solution that was very high level, like in Grid.__innit__ seemed like a it made the most sense but kept causing problems downstream. Looking to something less high up made it easier, but I totally missed that it would only work for certain file types

So I had to look at 2 questions:

  1. Does this leak happen for other file types?
    • If no, we are good, and it was a ugrid specific issue
    • If yes, and it's every other file, then we may have to find some function at a higher level that can handle all the leaks the same
  2. Is there somewhere up, that can solve this problem by interacting with all the different file types in the same way, without disturbing other functionality
    • I've looked at this before and was struggling to find a place

Looking at # 1, I performed a similar check like before, read in a demo dataset, add time, save, reopen, check if there's time, and see if it crashes when there's a subset. Here's what I got

Reader Demo grid tested (test/meshfiles/…) How grid vars are populated Leaks?
MPAS mpas/QU/mesh.QU.1920km.151026.nc DataArrays derived from in_ds yes
ESMF esmf/ne30/ne30pg3.grid.nc xr.DataArray(<DataArray>, …) yes
ICON icon/R02B04/icon_grid_0010_R02B04_G.nc in_ds["…"].T - 1 (DataArray) yes
Exodus exodus/outCSne8/outCSne8.g DataArrays from the source yes
SCRIP scrip/ne30pg2/grid.nc xr.DataArray(<numpy>, …) no
GEOS-CS geos-cs/c12/test-c12.native.nc4 in_ds["…"].values.ravel() no
UGRID ugrid/quad-hexagon/grid.nc renames the input dataset in place no
Structured / FESOM2 structured/outCSne30_vortex_structured.nc (FESOM2 untested) crashes when re-opening yes but worse

So basically from what I'm seeing right now, we have 3 different outcomes

  • it's fine as is (good, 3 file types)
  • it's the same problem as ugrid (bad)
  • it introduces a problem that crashes when reading (worse than what happened with ugrid)

I'll keep looking for a solution for # 2, but worried at it's impact, especially since every file type seems to have a unique way to handle the data when opening.

@erogluorhan

Copy link
Copy Markdown
Member

The table in your latest comment showing no leaks for UGRID confused me a bit (From your other words, it sounds like UGRID has the time leaking). Could you clarify?

@dylannelson

Copy link
Copy Markdown
Member Author

@erogluorhan
For # 1 and # 2 and # 3, I think from what I'm seeing, _ds is used for a lot of functionality, and the added time being in there causes it to leak out into anything that can call _ds (which is a lot, shown below). The "Grid" and "_ds" can't be thought of as separate, because they are so heavily intertwined.

"the Grid" is essentially _ds

Almost every Grid property is a thin read of self._ds. Straight from uxarray/grid/grid.py (trimmed to the returns):

@property
def n_face(self):                 return self._ds.sizes["n_face"]          # :878
@property
def face_node_connectivity(self): return self._ds["face_node_connectivity"]  # :1261
@property
def attrs(self):                  return self._ds.attrs                    # :836
@property
def coordinates(self):            # only the NAMES the conventions recognize:
    return {c for c in SPHERICAL_COORD_NAMES if c in self._ds} | \
           {c for c in CARTESIAN_COORD_NAMES if c in self._ds}            # :799
@property
def dims(self):
    return {d for d in DIM_NAMES if d in self._ds.dims}                   # :787

Line references (pinned to 17c20f91):

The same pattern appears to be true for others as well, like: n_node, n_edge, node_lon, every *_connectivity,
face_areas, bounds, … — all return self._ds[...].

This can be a problem because time can leak onto other calls, like what we saw with subset. When the subset pulls _ds["subgrid_face_indices"] out as an index array, time rides along and collides with the data's time.

@dylannelson

dylannelson commented Aug 12, 2026 •

Copy link
Copy Markdown
Member Author

@erogluorhan

The table in your latest comment showing no leaks for UGRID confused me a bit (From your other words, it sounds like UGRID has the time leaking). Could you clarify?

It's no leak for UGRID in my current version, which fixed it. main would instead show that yes, there is a leak in UGRID

@dylannelson

dylannelson commented Aug 12, 2026 •

Copy link
Copy Markdown
Member Author

and for

fixing at the Grid without altering _ds,

that's effectively what I had originally, via .values. (Version 1 in the big comment above)

The issue with this is that it fixes a symptom, without addressing the cause. We discussed this early on and it was turned down when we originally discussed it. It's basically a patch in the .isel indexer (_slice_from_grid) so that when the data is on the way out, we strip any extra data that may cause a clash. This works for our subset incident, but it won't permanently remove the leaked time and it won't guarantee it won't cause issues in other functions,

@erogluorhan

Copy link
Copy Markdown
Member

and for

fixing at the Grid without altering _ds,

that's effectively what I had originally, via .values. (Version 1 in the big comment above)

The issue with this is that it fixes a symptom, without addressing the cause. We discussed this early on and it was turned down when we originally discussed it. It's basically a patch in the .isel indexer (_slice_from_grid) so that when the data is on the way out, we strip any extra data that may cause a clash. This works for our subset incident, but it won't permanently remove the leaked time and it won't guarantee it won't cause issues in other functions,

You're right; "without altering _ds" doesn't seem feasible or ideal, but I think we should discuss if the fix should be in the individual I/O modules (like _mpas etc.) or right before the Grid object creation, i.e. after the specific grid format reading. Thougts?

@rajeeja

rajeeja commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Other readers like MPAS, ESMF, Exodus, ICON, and FESOM2 still leave time on _ds — MPAS reproduces the same IndexError from #1444. Could the filter move to the end of the auto-detect branch in Grid.from_dataset, before the return cls(...) at grid.py:323, so it covers all formats at once? The keep-list also drops bounds and max_face_radius when re-reading a uxarray-written grid.

@dylannelson

Copy link
Copy Markdown
Member Author

This new push should have everything we have wanted from the last few discussions
Early on we had the idea of moving the solution to grid.py in Grid.__init__ and doing the filtering there. The filtering would look at the dimensions and remove any that didn't belong. Unfortunately this didn't work, while also introducing downstream issues. It looked like this:

extra_dims = set(grid_ds.dims) - set(DIM_NAMES)
grid_ds = grid_ds.drop_dims(extra_dims)

The time in #1444 is a scalar (0-d) coordinate — it has no dimension, so it never appears in grid_ds.dims, extra_dims comes back empty, and drop_dims doesn't change it. I moved on from looking in grid.py at this point but now I'm back to it, and it seems to be working.

This solution is different in that it is dropping non-grid coordinates by name, which catches the scalar one that we've been having trouble with. Here's what this solution looks like instead:

grid_coord_names = set(ugrid.SPHERICAL_COORD_NAMES) | set(ugrid.CARTESIAN_COORD_NAMES)
stray = [c for c in ds.coords if c not in grid_coord_names]
return ds.drop_vars(stray, errors="ignore")

Testing

Before:
image
After:
image

Notebook Preview:
18-new-fix-demo.html

Downstream impact

Performed a few downstream tests with claude code to test what affects it may have, here's the results of what it looked at:

__init__ runs on every construction path — from_dataset (all formats), from_topology/from_healpix/etc., the subset's sliced grids, and Grid.copy(). Static sweep:

Area Finding Verdict
Grid properties (n_face, node_lon, …) read by name; recognized coords kept safe
Reads of stray coords off grid._ds grep '_ds.coords' / _ds["time"] → none safe
Grid↔data linking (open_dataset) by dimensions (_map_dims_to_ugrid), not coords safe
Subset bookkeeping subgrid_face/edge/node_indices are data vars safe
Grid.copy() re-runs __init__ on a clean _ds safe (idempotent)
Encoders / to_xarray no .coords use safe
Every .coords read in the package on the data objects / shapely — never grid._ds safe
Tests none assert coords on grid._ds safe

@dylannelson dylannelson added run-benchmark Run ASV benchmark workflow extra dims Handling of non-grid dimensions, such as time or altitude labels Aug 19, 2026
@github-actions

github-actions Bot commented Aug 19, 2026 •

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have stayed the same:

Change Before [ce8690d] After [95e2397] Ratio Benchmark (Parameter)
952±30μs 947±30μs 0.99 connectivity.Connectivity.time_edge_face('120km')
446±10μs 456±10μs 1.02 connectivity.Connectivity.time_edge_face('480km')
6.12±0.1ms 6.11±0.1ms 1.00 connectivity.Connectivity.time_edge_node('120km')
1.20±0.01ms 1.20±0.03ms 1.00 connectivity.Connectivity.time_edge_node('480km')
66.4±2μs 63.3±3μs 0.95 connectivity.Connectivity.time_face_edge('120km')
63.1±9μs 59.4±8μs 0.94 connectivity.Connectivity.time_face_edge('480km')
942±20μs 955±20μs 1.01 connectivity.Connectivity.time_face_face('120km')
392±30μs 375±20μs 0.96 connectivity.Connectivity.time_face_face('480km')
67.2±10μs 63.8±3μs 0.95 connectivity.Connectivity.time_face_node('120km')
57.4±5μs 58.2±10μs 1.01 connectivity.Connectivity.time_face_node('480km')
545±40μs 537±80μs 0.98 connectivity.Connectivity.time_n_nodes_per_face('120km')
394±20μs 386±20μs 0.98 connectivity.Connectivity.time_n_nodes_per_face('480km')
1.33±0.05ms 1.35±0.04ms 1.01 connectivity.Connectivity.time_node_edge('120km')
475±8μs 458±10μs 0.96 connectivity.Connectivity.time_node_edge('480km')
85.4±1ms 86.2±1ms 1.01 connectivity.Connectivity.time_node_face('120km')
5.48±0.2ms 5.38±0.03ms 0.98 connectivity.Connectivity.time_node_face('480km')
1.42M 1.42M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('120km')
105k 105k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('480km')
6.48M 6.48M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('120km')
419k 419k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('480km')
2.05k 2.05k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('120km')
2.05k 2.05k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('480km')
1.6M 1.6M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('120km')
101k 101k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('480km')
2.18k 2.18k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('120km')
2.18k 2.18k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('480km')
240k 240k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('120km')
26k 26k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('480km')
1.9M 1.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('120km')
127k 127k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('480km')
11.9M 11.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('120km')
747k 747k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('480km')
7.20±1ms 7.25±0.1ms 1.01 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
2.33±0.04ms 2.49±0.09ms 1.07 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
8.04±0.03ms 8.05±8ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
1.45±0.02ms 1.47±0.03ms 1.01 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
57.3k 57.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
12.3k 12.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
123k 123k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
128 128 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.27M 1.27M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
50.1k 50.1k 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
516k 516k 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
712 712 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.77M 1.77M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
1.75M 1.75M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.91M 1.91M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
35.3k 35.3k 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.14±0.03μs 1.18±0.04μs 1.04 geometry_kernels.AccucrossKernels.time_accucross
2.58±0.02μs 2.63±0.05μs 1.02 geometry_kernels.AccucrossKernels.time_accucross_pair
546±10ns 491±30ns ~0.90 geometry_kernels.EFTPrimitives.time_acc_sqrt_re
511±20ns 476±20ns 0.93 geometry_kernels.EFTPrimitives.time_diff_of_products
406±30ns 426±20ns 1.05 geometry_kernels.EFTPrimitives.time_two_prod
426±20ns 441±20ns 1.04 geometry_kernels.EFTPrimitives.time_two_sum
552±20ns 570±20ns 1.03 geometry_kernels.GCAConstLatIntersection.time_accux_constlat_kernel
778±20ns 793±60ns 1.02 geometry_kernels.GCAConstLatIntersection.time_gca_const_lat_intersection
742±50ns 764±60ns 1.03 geometry_kernels.GCAConstLatIntersection.time_try_gca_const_lat_intersection
796±30ns 832±40ns 1.04 geometry_kernels.GCAGCAIntersection.time_accux_gca_kernel
887±20ns 907±30ns 1.02 geometry_kernels.GCAGCAIntersection.time_gca_gca_intersection
1.05±0.01μs 1.08±0.03μs 1.03 geometry_kernels.GCAGCAIntersection.time_try_gca_gca_intersection
40.0±0.5μs 40.3±0.7μs 1.01 geometry_kernels.OrientPredicates.time_on_minor_arc
39.1±3μs 38.2±2μs 0.98 geometry_kernels.OrientPredicates.time_orient3d_on_sphere
4.10±0.03ms 4.04±0.1ms 0.99 geometry_samebody.SameBodyConstLat.time_accux_dispatch
719±30μs 754±5μs 1.05 geometry_samebody.SameBodyConstLat.time_accux_kernel
4.00±0ms 4.00±0.02ms 1.00 geometry_samebody.SameBodyConstLat.time_fp64_dispatch
108±5μs 113±0.06μs 1.04 geometry_samebody.SameBodyConstLat.time_fp64_kernel
29.6±0.1ms 29.6±0.04ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_dispatch
6.26±0.01ms 6.28±0.03ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_kernel
23.7±0.2ms 24.1±0.04ms 1.01 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_dispatch
859±2μs 858±4μs 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_kernel
762±10ms 752±10ms 0.99 import.Imports.timeraw_import_uxarray
1 1 1.00 lazy_grid_construction.LazyGridConstruction.track_computes_isel
2 2 1.00 lazy_grid_construction.LazyGridConstruction.track_computes_open_grid_chunked
97.8±1ms 95.5±0.5ms 0.98 lazy_grid_construction.OpenGridChunked.time_open_grid('120km')
133±2ms 135±3ms 1.01 lazy_grid_construction.OpenGridChunked.time_open_grid('480km')
2.01M 2.1M 1.04 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('120km')
2.11M 2.3M 1.09 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('480km')
1.22±0.06ms 1.30±0.06ms 1.06 mpas_ocean.CheckNorm.time_check_norm('120km')
829±20μs 860±10μs 1.04 mpas_ocean.CheckNorm.time_check_norm('480km')
638±50μs 617±30μs 0.97 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
271±8μs 284±7μs 1.05 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
413±30μs 441±10μs 1.07 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
304±2μs 310±5μs 1.02 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
4.15±0.07ms 4.11±0.03ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
3.12±0.04ms 3.00±0.1ms 0.96 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
107±0.6ms 105±1ms 0.98 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
9.31±0.04ms 9.16±0.3ms 0.98 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
13.3±0.04ms 13.2±0.05ms 0.99 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
639±20μs 638±30μs 1.00 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
6.70±0.01ms 6.70±0.07ms 1.00 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
369±50μs 384±30μs 1.04 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
549±3ms 545±7ms 0.99 mpas_ocean.CrossSections.time_const_lat('120km', 1)
284±4ms 274±1ms 0.96 mpas_ocean.CrossSections.time_const_lat('120km', 2)
143±1ms 143±0.6ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 4)
489±3ms 490±2ms 1.00 mpas_ocean.CrossSections.time_const_lat('480km', 1)
240±2ms 237±3ms 0.99 mpas_ocean.CrossSections.time_const_lat('480km', 2)
124±2ms 118±1ms 0.96 mpas_ocean.CrossSections.time_const_lat('480km', 4)
28.6±0.2ms 28.9±0.1ms 1.01 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
3.39±0.06ms 3.47±0.09ms 1.02 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
6.24±0.2ms 6.33±0.3ms 1.01 mpas_ocean.FaceAreas.time_face_areas('120km')
1.98±0.1ms 2.11±0.2ms 1.07 mpas_ocean.FaceAreas.time_face_areas('480km')
229k 229k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('120km')
14.3k 14.3k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('480km')
2.12M 2.12M 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('120km')
687k 687k 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('480km')
53.7±1ms 54.3±0.7ms 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', 'exclude')
153±2ms 158±2ms 1.03 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', 'split')
4.83±0.2ms 4.87±0.1ms 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', 'exclude')
30.3±0.9ms 31.0±0.8ms 1.02 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', 'split')
9.05±0.6ms 9.89±1ms 1.09 mpas_ocean.Gradient.time_gradient('120km')
1.32±0.04ms 1.37±0.03ms 1.04 mpas_ocean.Gradient.time_gradient('480km')
457k 457k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('120km')
28.7k 28.7k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('480km')
3.44M 3.44M 1.00 mpas_ocean.Gradient.track_peakmem_gradient('120km')
226k 226k 1.00 mpas_ocean.Gradient.track_peakmem_gradient('480km')
349±10μs 341±20μs 0.98 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
149±4μs 142±6μs 0.96 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
110±3μs 140±8μs ~1.28 mpas_ocean.Integrate.time_integrate('120km')
82.1±2μs 85.7±2μs 1.04 mpas_ocean.Integrate.time_integrate('480km')
18.4M 18.4M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('120km')
1.2M 1.2M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('480km')
103±0.8ms 98.3±2ms 0.96 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
102±1ms 97.6±1ms 0.96 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
103±1ms 99.3±2ms 0.96 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
7.34±0.1ms 7.21±0.2ms 0.98 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
7.48±0.01ms 7.29±0.1ms 0.97 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
7.36±0.2ms 7.28±0.1ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
142±0.5ms 143±1ms 1.01 mpas_ocean.NeighborhoodBuild.time_build('120km', 1.0)
318±2ms 320±4ms 1.01 mpas_ocean.NeighborhoodBuild.time_build('120km', 5.0)
9.41±0.3ms 9.07±0.06ms 0.96 mpas_ocean.NeighborhoodBuild.time_build('480km', 1.0)
11.8±0.08ms 11.4±0.1ms 0.96 mpas_ocean.NeighborhoodBuild.time_build('480km', 5.0)
139±1ms 135±0.7ms 0.97 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 1.0)
310±1ms 314±2ms 1.01 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 5.0)
9.15±0.05ms 9.04±0.08ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 1.0)
11.5±0.2ms 11.4±0.1ms 0.98 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 5.0)
612.76 612.76 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 15.0)
74.17 74.17 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 5.0)
37.29 37.29 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 15.0)
6.57 6.57 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 5.0)
141M 141M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 15.0)
17.4M 17.4M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 5.0)
563k 563k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 15.0)
123k 123k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 5.0)
145M 145M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 15.0)
21.5M 21.5M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 5.0)
824k 824k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 15.0)
384k 384k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 5.0)
24.9±0.7ms 27.6±0.2ms ~1.11 mpas_ocean.NeighborhoodDask.time_mean('120km', 'grid_chunks')
11.1±0.1ms 12.0±0.4ms 1.08 mpas_ocean.NeighborhoodDask.time_mean('120km', 'numpy')
24.0±0.4ms 24.8±0.8ms 1.04 mpas_ocean.NeighborhoodDask.time_mean('120km', 'time_chunks')
5.62±0.09ms 5.93±0.2ms 1.06 mpas_ocean.NeighborhoodDask.time_mean('480km', 'grid_chunks')
330±6μs 324±7μs 0.98 mpas_ocean.NeighborhoodDask.time_mean('480km', 'numpy')
3.70±0.06ms 3.84±0.1ms 1.04 mpas_ocean.NeighborhoodDask.time_mean('480km', 'time_chunks')
5.82M 5.82M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'grid_chunks')
2.75M 2.75M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'numpy')
5.68M 5.68M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'time_chunks')
744k 744k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'grid_chunks')
177k 177k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'numpy')
529k 528k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'time_chunks')
4.36±0s 4.34±0.01s 1.00 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'mean')
4.48±0.01s 4.46±0s 1.00 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'median')
117±1ms 115±0.1ms 0.98 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'mean')
117±0.7ms 116±0.4ms 0.99 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'median')
491±3ms 492±2ms 1.00 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'mean')
520±3ms 522±3ms 1.01 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'median')
16.2±0.1ms 16.0±0.2ms 0.98 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'mean')
16.3±0.05ms 16.2±0.05ms 0.99 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'median')
4.52±0.04ms 4.53±0.06ms 1.00 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'mean')
37.1±0.2ms 36.8±0.02ms 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'median')
190±8μs 198±8μs 1.04 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'mean')
474±10μs 469±20μs 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'median')
234k 234k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'mean')
235k 235k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'median')
19.5k 19.5k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'mean')
19.6k 19.6k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'median')
350±3μs 337±20μs 0.96 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
332±3μs 324±10μs 0.98 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
328±10μs 317±10μs 0.97 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
309±7μs 303±7μs 0.98 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
103±0.6ms 104±0.2ms 1.01 mpas_ocean.RemapDownsample.time_bilinear_remapping
16.5±0.1ms 17.1±0.1ms 1.04 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
15.1±0.08ms 15.5±0.2ms 1.03 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
788±4ms 774±4ms 0.98 mpas_ocean.RemapUpsample.time_bilinear_remapping
21.3±0.2ms 21.0±0.3ms 0.99 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
9.80±0.06ms 9.71±0.2ms 0.99 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
7.59±0.09ms 7.54±0.5ms 0.99 mpas_ocean.ZonalAverage.time_zonal_average('120km')
4.55±0.04ms 4.62±0.2ms 1.01 mpas_ocean.ZonalAverage.time_zonal_average('480km')
1.067844816237592 1.0563824632451835 0.99 nogil_scaling.GILScaling.track_gil_scaling
1.27±0.02ms 1.30±0.01ms 1.03 open_dataset.OpenDataset.time_open_dataset('dataset', 'dataset')
2.22±0.02ms 2.32±0.03ms 1.05 open_dataset.OpenDataset.time_open_dataset('dataset', 'file')
5.06±0.03ms 5.11±0.02ms 1.01 open_dataset.OpenDataset.time_open_dataset('file', 'dataset')
5.99±0.04ms 6.05±0.04ms 1.01 open_dataset.OpenDataset.time_open_dataset('file', 'file')
173±0.7μs 174±1μs 1.01 open_dataset.OpenDataset.time_open_dataset('grid', 'dataset')
1.13±0.01ms 1.12±0ms 0.99 open_dataset.OpenDataset.time_open_dataset('grid', 'file')
1.64±0.01s 1.58±0.02s 0.97 plots.PlotContour.time_contour_projected
184±2ms 191±2ms 1.04 plots.PlotHealpix.time_default_plot(6)
1.81±0.04s 1.80±0.02s 1.00 plots.PlotHealpix.time_default_plot(8)
28M 28M 1.00 plots.PlotHealpix.track_peakmem_default_plot(6)
443M 443M 1.00 plots.PlotHealpix.track_peakmem_default_plot(8)
8.04±0.02ms 8.33±0.2ms 1.04 quad_hexagon.QuadHexagon.time_open_dataset
7.01±0.01ms 7.31±0.09ms 1.04 quad_hexagon.QuadHexagon.time_open_grid
408 408 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_dataset
392 392 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_grid
71.3k 71.3k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_dataset
71.1k 71k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_grid
5.82±0.8ms 6.22±5ms 1.07 subset.BoundingBox.time_bounding_box('120km')
3.01±0.1ms 3.10±0.1ms 1.03 subset.BoundingBox.time_bounding_box('480km')
2.38M 2.38M 1.00 subset.BoundingBox.track_peakmem_bounding_box('120km')
203k 203k 1.00 subset.BoundingBox.track_peakmem_bounding_box('480km')
2.67±0.09ms 2.81±0.1ms 1.05 subset.BoundingBoxHealpix.time_bounding_box(6)
11.5±1ms 12.0±2ms 1.05 subset.BoundingBoxHealpix.time_bounding_box(8)
2.24M 2.24M 1.00 subset.BoundingBoxHealpix.track_peakmem_bounding_box(6)
13.2M 13.2M 1.00 subset.BoundingBoxHealpix.track_peakmem_bounding_box(8)
735±4ms 731±3ms 0.99 to_raster.ToRaster.time_to_raster((10000.0, 10.0))
1.53±0.01s 1.54±0.01s 1.01 to_raster.ToRaster.time_to_raster((10000.0, 100.0))
1.77±0s 1.80±0.01s 1.01 to_raster.ToRaster.time_to_raster((100000.0, 100.0))
270M 270M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 10.0))
680M 680M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 100.0))
765M 765M 1.00 to_raster.ToRaster.track_peakmem((100000.0, 100.0))

Benchmarks that have got worse:

Change Before [ce8690d] After [95e2397] Ratio Benchmark (Parameter)
+ 304M 389M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
+ 305M 389M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
+ 307M 389M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
+ 305M 389M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
+ 278M 388M 1.4 import.Imports.track_peakmem_import_uxarray
+ 327M 390M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 1)
+ 328M 390M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 2)
+ 328M 390M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 4)
+ 308M 390M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 1)
+ 308M 390M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 2)
+ 308M 390M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 4)
+ 328M 390M 1.19 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('120km')
+ 301M 390M 1.29 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('480km')
+ 328M 388M 1.18 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('120km')
+ 308M 388M 1.26 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('480km')

@dylannelson
dylannelson marked this pull request as ready for review August 19, 2026 19:34

@Sevans711 Sevans711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a relatively clean fix overall, with lots of tests run by hand to confirm things seem to be working!

I left a few minor comments inline, and I have a couple larger comments here:

  1. I had a suspicion that extra dimensions (not just extra coordinates) on input datasets would be an issue, but I tested a few things locally and found that it wasn't a problem (for some reason…). For example, if you had a time dimension on the input dataset, instead of just a time coordinate, it doesn't seem to lead to a crash later (although, it does leak into Grid._ds). It isn't really obvious to me why it wouldn't crash…. Can you clarify why this might be the case, and/or add a test to ensure subsetting still works even after adding a time dimension to the grid?
  2. Would you also be able to create a file with a time coordinate somewhere in the test suite (in meshfiles folder), and test that, too, instead of only having a test where the time coordinate gets assigned after loading the dataset? That would help me to be more convinced that this fully closes the original issue. It's a bit tricky to know for certain because the original issue report doesn't contain a complete verifiable example. But, the workflow there seemed to be "load from file, then subset (which crashes)" so ideally a test here could include that complete workflow as well.

Comment thread test/test_subset.py Outdated
import uxarray.grid.slice as slice_module
from uxarray.grid.slice import _remap_dense, _remap_kernel, _remap_searchsorted

import numpy as np

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove repeated numpy import

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No clue how I missed that twice, thanks for the catch, should be gone

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sevans711 please resolve or comment.

Comment thread uxarray/grid/grid.py Outdated
from uxarray.core.dataarray import UxDataArray


def _drop_non_grid_coords(ds):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relocate to uxarray/grid/utils.py or some other file, rather than defining this helper function directly in the main grid.py file

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, should be moved there now

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sevans711 please resolve or comment.

@erogluorhan

Copy link
Copy Markdown
Member
  1. I had a suspicion that extra dimensions (not just extra coordinates) on input datasets would be an issue, but I tested a few things locally and found that it wasn't a problem (for some reason…). For example, if you had a time dimension on the input dataset, instead of just a time coordinate, it doesn't seem to lead to a crash later (although, it does leak into Grid._ds). It isn't really obvious to me why it wouldn't crash…. Can you clarify why this might be the case, and/or add a test to ensure subsetting still works even after adding a time dimension to the grid?

I agree with this, I think we shouldn't let extra dimensions either into the Grid regardless of whether they lead to crash or not.

  1. Would you also be able to create a file with a time coordinate somewhere in the test suite (in meshfiles folder), and test that, too, instead of only having a test where the time coordinate gets assigned after loading the dataset? That would help me to be more convinced that this fully closes the original issue. It's a bit tricky to know for certain because the original issue report doesn't contain a complete verifiable example. But, the workflow there seemed to be "load from file, then subset (which crashes)" so ideally a test here could include that complete workflow as well.

I don't think we should expose a grid file with time coord/dim into our testing suite as it is uncommon to have such a presence and it would be misleading in the future.

@erogluorhan erogluorhan left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much for this fix! @Sevans711 has already made great points inline and overall ( see (1) here, but I believe (2) should be avoided), so I am approving it to make way once they're addressed.

@Sevans711

Copy link
Copy Markdown
Collaborator

Thanks very much for this fix! @Sevans711 has already made great points inline and overall ( see (1) here, but I believe (2) should be avoided), so I am approving it to make way once they're addressed.

Ah, for (2), I was going based on the file at gridpath("mpas", "QU", "oQU480.231010.nc") which is treated as dataset data and grid data during test_open_dataset_single_combined_xarray_dataset in test_api.py (related: #1479). That file already has a Time dimension in it. So, the testing suite does already include a "grid file" with time dimension, though it isn't really a simple grid file, rather it is a single file grid + data situation.

That was leading me to think it would be reasonable enough to include a second version of this file which is extremely similar but just has a scalar time coordinate instead of a Time dimension, and then call ux.open_dataset on it. What would your thoughts be on something like that, @erogluorhan?

Either way, I do still want to see evidence of the full original workflow being reproduced somewhere before I am ready to approve this as fully closing the original issue. Ideally this would involve an example of "load from file, then subset" crashing on main but not crashing on this branch. I spent a little time trying to reproduce something like that on my end, but I'm not succeeding quickly and I want to avoid duplicating work if you've already run such a test. @dylannelson my apologies if you have already shared test results like this - would you be able to point me to them if this test was already performed somewhere?

@dylannelson

Copy link
Copy Markdown
Member Author

@Sevans711
For 1, yeah the weirdness of dimensions vs coordinates here is what has taken me a while to understand and work around. It seems to have to do with how dimensions are used during .isel indexing. When it's a scalar coordinate it clashes with other existing times, whereas when its a dimension, it only attaches to variables that share a time with it, so time doesn't end up on the index at all.

  • this may be why other issues related to this one claim that subsetting didn't work, maybe because it had a time but didn't share the time so it did nothing or something wrong? (really hard to tell without seeing the data) But I'm not sure, I haven't explore this case much as it was never a dimension, and we don't know if it can be naturally.
  • note the time dimension will still be in the Grid._ds, it just won't raise conflicts. We can forcefully make it a dim, but it doesn't get carried into subsetting functionality at all.

The summary of a path deep dive through many files and functions basically came down to:

xarray attaches a coordinate to a variable only when the coordinate's dimensions are a subset of the variable's dimensions.

When it's scalar time → dims = (). And () is a subset of everything, so it glues onto the n_face index array.
Dimensional time → dims = (time,). (time,) is not a subset of (n_face,), so it stays off the index array.

Which happens about here it xarray: xarray/core/coordinates.py#L1214

Filtering dimensions seemed to be distinct from this problem (filtering coordinates) in my mind. Like I don't think we saw anything that indicates a time dimension could appear naturally like this. The downstream affects of filtering dimensions is different than coordinates too. If my understanding seems wrong, and a dimension filter here seems good, maybe we can scope this out over a call because there's lots of threads that have felt tangled while exploring this, and I don't want to overcompensate in this solution because it might introduce new problems.

Maybe of note, that uxarray also won't report any Dims that it has that aren't in DIM_NAMES - which is a static list (in grid.py):

    @property
    def dims(self) -> set:
        """Names of all unstructured grid dimensions."""
        from uxarray.conventions.ugrid import DIM_NAMES

        return set([dim for dim in DIM_NAMES if dim in self._ds.dims])

so it shouldn't ever really report anything outside of:

DIM_NAMES = [
    "n_node",
    "n_edge",
    "n_face",
    "n_max_face_nodes",
    "n_max_face_edges",
    "n_max_face_faces",
    "n_max_edge_nodes",
    "n_max_edge_edges",
    "n_max_edge_faces",
    "n_max_node_faces",
    "n_max_node_edges",
    "two",
]
  1. Was about to respond why we didn't add local files, but @erogluorhan got there first
    Basically all my local tests read a file, made a alternate version with time, saved, reopened, check if it has time, and run subset to see if it crashes. This effectively does the same because the same Grid.__init__ should be called either way. If you're interested in seeing that we could hop on a call and I can demo. I have a few notebooks dedicated to this.

@erogluorhan
erogluorhan self-requested a review August 20, 2026 21:01

@erogluorhan erogluorhan left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I could understand from the whole conversation in this PR, especially the last parts of it, maybe we got a little too focused/concerned on fixing the reported case only, i.e. scalar-only coords colliding during isel() when .subset is called? I think the fix should be different, more comprehensive, than that, and I have a few thoughts on what that pathway should look like:

  1. The main problem is when there is a non-spatial coord leakage into the Grid._ds, whether scalar or dimensional, we should clean it up before wrapping up the Grid object.
  2. We should avoid trying to fix this at the low-level I/O modules (_mpas, _icon, etc.) That is to say, the current approach (_drop_non_grid_coords being called in Grid constructor instead) makes sense to me.
  3. I don't agree with dropping only scalar coords. The actual overarching fix here should rather cover dimensional coords as well. It should actually avoid the unwanted removal of any spatial coords, e.g. vlon etc. in ICON. That said, _drop_non_grid_coords's current version focusing only on scalar coords seems incorrect to me. Also, avoiding the unwanted removal of spatial coords should easily be achievable because such coords should have the size of the number of either geometric element (nodes, faces, or edges).

To wrap up the above thoughts, I've put together a notebook (pr_1667_grid_filter.html) where I test our main branch for:
(1) a MPAS case with a single-file that has both grid and data in it,
(2) Quad Hexagon grid and data files,
(3) A single-file ICON case

All of these cases use files from our testing suite. I used the previous version of _drop_non_grid_coords that removed dimensional coords (i.e. not only scalar ones). I put explanatory comments into each cell, but in summary:

(1) has no leakage for this particular MPAS case, but since I was able to see Time leakage in the other MPAS case @dylannelson was pointing out priorly, it seems safe to solve potential problem at the Grid level as this PR currently attempts, rather than making model output-based assumptions.
(2) With the expanded Quad Hexagon case, I showed that in a single file scenario where data variables have dimensional coords but not necessarily the grid variables, we might see dimensional Time coord leaking into Grid. So, fixing only scalar coords maynot be a complete fix.
3. Spatial coords such as vlon etc. are also dropped after _drop_non_grid_coords call. Solution should be check for each coord's size, and if it is equal to the number of either faces, edges, or nodes, don't remove it.

@dylannelson

Copy link
Copy Markdown
Member Author

@Sevans711
here's a custom notebook for you, with a full process and my best attempt at rebuilding everything the way the original user may have had things set up on his end. Please keep in mind, I don't have the original data, or have any clue what it contained or how it was read in, and what happened to after. I had to find a way to construct the data that could produce similar outputs, and more strangely, finding a way to read in the data from a single file that causes the times to be split in the weird way that it was. This was like trying to reproduce someone's cookie recipe, but only knowing they used eggs.

21-original-workflow-demo.ipynb

21-original-workflow-demo.html

@Sevans711

Copy link
Copy Markdown
Collaborator

@Sevans711 here's a custom notebook for you, with a full process and my best attempt at rebuilding everything the way the original user may have had things set up on his end. Please keep in mind, I don't have the original data, or have any clue what it contained or how it was read in, and what happened to after. I had to find a way to construct the data that could produce similar outputs, and more strangely, finding a way to read in the data from a single file that causes the times to be split in the weird way that it was. This was like trying to reproduce someone's cookie recipe, but only knowing they used eggs.

21-original-workflow-demo.ipynb

21-original-workflow-demo.html

Thank you for working through this! I agree it is quite difficult to reproduce without more details about the original issue. This notebook might be as close as possible to reproducing without having more information. I am still a little suspicious about these lines:

ds = xr.open_dataset("demo_single_file.nc").isel(time=0)
uxds = ux.open_dataset(ds, "demo_single_file.nc")

To me it feels unlikely that something like "call .isel() on the grid dataset then pass that into the uxarray open_dataset call" would have happened in a user workflow. But, at this point I also can't say that that isn't what happened, either. For the purposes of "does this PR close the original issue?" your efforts here have convinced me it is reasonable to move forward assuming the answer is yes. (If not, it is always possible for another issue to be posted later, after all.)

That said, what would your thoughts be on opening a new issue which this PR would close as well, something like "Grid should include only grid-related information"? I think that could be a good way to clarify why this PR is going through so many iterations, to me it is sounding like @erogluorhan is asking for an issue like that to be solved here too. Is that accurate?

I also have a minor note about this:

Solution should be check for each coord's size, and if it is equal to the number of either faces, edges, or nodes, don't remove it.

I suspect it would probably be cleaner to check for each coord's dims instead of sizes; only keep coords which have grid dims (e.g. n_face). This could avoid spawning another issue like #1616 in the future. Is there any case where this wouldn't work out?

@dylannelson

dylannelson commented Aug 25, 2026 •

Copy link
Copy Markdown
Member Author

After further discussion, the team proposed a solution that covers more cases to be certain no other cases of data leaking could occur as well, and not be specific to scalar. The approach looks like:

  • Remove any data variables and dimensions with non-grid dims
  • Read through all the Data Variables and Dimensions, check their dimensions, if any are not in DIM_NAMES, drop that data variable/dimension

Basically we check to make sure everything is related to:

['n_edge', 'n_face', 'n_max_edge_edges', 'n_max_edge_faces', 
'n_max_edge_nodes', 'n_max_face_edges', 'n_max_face_faces', 
'n_max_face_nodes', 'n_max_node_edges', 'n_max_node_faces', 
'n_node', 'two']

and if not, it gets removed. Testing this logic on a variety of files with grids, we get some outliers:

mpas/QU/480/grid.nc   [spec=MPAS]
   Variable   'boundary_node_indices' would be REMOVED  : non-grid dim(s) ['n_boundary_nodes']   (dims=('n_boundary_nodes',))

mpas/QU/mesh.QU.1920km.151026.nc   [spec=MPAS]
   Variable   'boundary_node_indices' would be REMOVED  : non-grid dim(s) ['n_boundary_nodes']   (dims=('n_boundary_nodes',))

mpas/QU/oQU480.231010.nc   [spec=MPAS]
   Variable   'boundary_node_indices' would be REMOVED  : non-grid dim(s) ['n_boundary_nodes']   (dims=('n_boundary_nodes',))

ugrid/fesom/fesom.mesh.diag.nc   [spec=UGRID]
   Variable   'nod_area' would be REMOVED  : non-grid dim(s) ['nz']   (dims=('nz', 'n_node'))
   Variable   'edge_node_connectivity' would be REMOVED  : non-grid dim(s) ['n2']   (dims=('n_edge', 'n2'))
   Variable   'edge_face_connectivity' would be REMOVED  : non-grid dim(s) ['n2']   (dims=('n_edge', 'n2'))
   Variable   'nod_in_elem2D' would be REMOVED  : non-grid dim(s) ['N']   (dims=('N', 'n_node'))
   Variable   'edge_cross_dxdy' would be REMOVED  : non-grid dim(s) ['n4']   (dims=('n4', 'n_edge'))
   Coordinate 'nz' would be REMOVED  : non-grid dim(s) ['nz']   (dims=('nz',))
   Coordinate 'nz1' would be REMOVED  : non-grid dim(s) ['nz1']   (dims=('nz1',))

Some of these look relevant, like edge_node_connectivity or edge_face_connectivity getting removed because it has dimension n2, but there's nothing in the ugrid conventions that we are using that mentions n2 or any of the others.
@erogluorhan , @Sevans711 , @rajeeja
So how do we feel about the variables/coordinates removed above? Is the function doing what it's supposed to, or should we extend the whitelist for these instances?

@Sevans711

Sevans711 commented Aug 26, 2026 •

Copy link
Copy Markdown
Collaborator

I think we are on the same page but just to clarify, I think the steps are:

  • Remove any data variables which have any non-grid dims (i.e. any dims not in DIM_NAMES).
  • Remove any coordinates which have any non-grid dims.
  • Remove any scalar coordinates.
  • (Do not remove any scalar data variables)

Would you be able to also open a corresponding issue to track this? It might be easier to refer back to this in the future if these details get discussed on an issue focused on this topic, to make sure they don't end up getting buried inside this PR discussion (if the PR discussion continues to grow).

Some of these look relevant, like edge_node_connectivity or edge_face_connectivity getting removed because it has dimension n2, but there's nothing in the ugrid conventions that we are using that mentions n2 or any of the others. … So how do we feel about the variables/coordinates removed above? Is the function doing what it's supposed to, or should we extend the whitelist for these instances?

Thank you for checking this ahead of time, these feel like important cases to consider! Clarifying, is this function being applied directly to the data from the file, or after the data has already been parsed by the relevant uxarray io routines? For example, I would have guessed that edge_node_connectivity variable should have dimension "two", not "n2", after being parsed by existing io routines. I think the non-grid-dims-stripping should only occur after io routines parse the data, not directly on data from files. (If this was already from after the io routines parse the data, then I will defer to others' thoughts about these data variables, I don't really have any particular direct intuitions about them yet.)

@Sevans711 Sevans711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the thorough investigation and updates here! The tests look reasonable by-eye, and I ran some of them locally against main to confirm they fail on main but are fixed by this PR.

I would prefer to just always drop non grid data during Grid.__init__, instead of introducing an opt-out option which gets added to the public API and woven throughout various parts of the codebase. My understanding of uxarray developer team preferences in general is that there is usually a hesitance to add new options to the public API (see e.g. the discussion in #1566).

Additionally, could you clarify a bit further what is the motivation for dropping scalar data variables? My understanding was that they weren't getting in the way of anything. Dropping them also led to needing to whitelist a few in particular: "grid_topology", "max_face_radius", plus a helper function for CRS variables (are these always scalars too? I might be misunderstanding the CRS variables case). The whitelist would be an ongoing maintenance burden whenever any new features are added which cache a scalar result on the grid. If the scalar data variables really aren't getting in the way of anything, my preference would be to just keep all of them, instead of building and maintaining a whitelist for them.

Comment thread uxarray/grid/utils.py
Comment on lines +36 to +39
set(ugrid.SPHERICAL_COORD_NAMES)
| set(ugrid.CARTESIAN_COORD_NAMES)
| set(ugrid.CONNECTIVITY_NAMES)
| set(DESCRIPTOR_NAMES)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point of whitelisting these explicitly? Aren't they all being kept automatically, because they all correspond to variables with only grid dimensions?

If it is possible to claim the simpler invariant of "only keep variables with only grid dims, plus a few exceptions" by dropping these from keep, instead of needing to have code like "only keep [long list of whitelisted variables], plus variables with only grid dims, plus a few exceptions", I think that would be preferable.

Comment thread uxarray/grid/utils.py Outdated
Comment on lines +42 to +44
"bounds",
"face_bounds_lon",
"face_bounds_lat",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect that these three ("bounds", "face_bounds_lon", "face_bounds_lat") would all be kept automatically if "min_max" and "lat_lon" were added to the list of dimensions being kept. (Locally, something like grid_dims = grid_dims + {"min_max", "lat_lon"}.) To me this would make more sense and represent a lower ongoing maintenance burden than explicitly whitelisting these three variables.

E.g., maybe a new feature like "face_bounds_x" might be added some day, with a "min_max" dimension. If the rule here is to consider "min_max" dimension as something to keep, there would be no need to touch this part of the code, but if it's just an explicit whitelist then "face_bounds_x" would need to be added here too.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. I checked it: with lat_lon and min_max counted as grid dims, re-reading a grid that already has bounds and face_bounds_lon keeps them without listing names.

Comment thread uxarray/grid/utils.py Outdated
"bounds",
"face_bounds_lon",
"face_bounds_lat",
"max_face_radius",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For scalar vars like "grid_topology" and "max_face_radius" which need to be whitelisted, I would recommend defining the list somewhere in the conventions folder instead of here, and import it, similarly to ugrid.DIM_NAMES.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually… this, and the exceptions with CRS variables below, is only relevant if the decision is to drop scalar data variables. Wouldn't it be easier to reason through and maintain if we just keep all scalar data variables? (Still drop scalar coordinates, though.) Leaving a note in my overall review about this to clarify.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. The test I suggested on test_non_grid_data.py:30 shows subsetting works with a scalar time data variable, even on main. See my review summary.

@dylannelson

dylannelson commented Nov 11, 2026 •

Copy link
Copy Markdown
Member Author

@Sevans711 I see a lot of the comments above are about whitelisting and how it seems like a burden or a pain to manage, and while I agree, I don't think we can make such broad and strict rules like "Remove all of __" without making exceptions, due to the wide variance of data and models coming in, and how they are used at various points in the code. Since the beginning of issue, I've expressed concerns with putting it in Grid.__init__ because Grid.__init__ isn't used just when a file is read from disk, but at other times like subsetting as well, and data has been added and changed in between. The scope of this started with a single case, and expanded a lot broader since then. Along the way, I haven't found any single solution that's just easily works without edge cases we must account for. At this point I don't know what else to add or explain without once again coming up with another solution that will also likely have weird edge cases we have to account for.

Additionally, could you clarify a bit further what is the motivation for dropping scalar data variables?

This one I don't understand though, because I thought this has always been a part of the problem with time. Whether a stray scalar time reads back as a coordinate or a data variable depends only on how the file was written. It’s a coordinate if another variable’s coordinates attribute lists it, and a variable otherwise. If you want to remove this rule and it's own whitelist that's fine, it's just going to let in more data, which is part of what we're trying to avoid.

The API concerns I get too, but I think it was a very rare but potentially critical safety net for if the whitelist method doesn't let something through and the user says "Hey this variable/dim didn't get through, but I need it right now" and whatever model they're using can quickly suggest turning off the cleaning so it gets through, without having to wait for us to add it to a model specific exception and make a release. Just like how this entire fix is only for weird unexpected scenarios, I also thought it could be also be good to have a backdoor for other weird and unexpected scenarios. We can cut it out, but I think it could result in a user being unable to proceed because this growing list of filters stopped something they needed to get through. I was told early on we shouldn't even have thing like time on the grid in the first place, but we did, along with a lot of other data, and getting rid of it needs to be done carefully as data on the grid can't be just wiped out. Many other parts of the code have been developed to add and interact with data on the grid. As we discussed earlier "without altering _ds" doesn't seem feasible or ideal

If you want to take a shot at making a solution that could work with all edge cases and without a whitelist, send me what that could look like, but I haven't really found or thought of anything that works like that. I could respond to each of the questions above, but don't want to continue to do so if the whole idea is going to be scrapped and rescoped again.

@cmdupuis3 cmdupuis3 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. A grid opted out with a stray time coordinate raises IndexError on isel. Also, a grid coordinate on n_face (lev) shows up in data subsets.

in dataarray.py:_slice_from_grid, add .data:

         if self._face_centered():
             da_sliced = self.isel(
-                n_face=sliced_grid._ds["_subgrid_face_indices"], ignore_grid=True
+                n_face=sliced_grid._ds["_subgrid_face_indices"].data, ignore_grid=True
             )

         elif self._edge_centered():
             da_sliced = self.isel(
-                n_edge=sliced_grid._ds["_subgrid_edge_indices"], ignore_grid=True
+                n_edge=sliced_grid._ds["_subgrid_edge_indices"].data, ignore_grid=True
             )

         elif self._node_centered():
             da_sliced = self.isel(
-                n_node=sliced_grid._ds["_subgrid_node_indices"], ignore_grid=True
+                n_node=sliced_grid._ds["_subgrid_node_indices"].data, ignore_grid=True
             )
  1. geoflow-small loses mesh_depth from to_xarray(), and no warning is raised.
import warnings, uxarray as ux

path = "test/meshfiles/ugrid/geoflow-small/grid.nc"
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    new = ux.open_grid(path)                         # branch default
old = ux.open_grid(path, drop_non_grid_data=False)   # == main

sorted(set(old.to_xarray().variables) - set(new.to_xarray().variables))  # ['mesh_depth']
[str(x.message) for x in w if "drop" in str(x.message).lower()]          # []

Recommend defaulting drop_non_grid_data to false:

# uxarray/core/api.py  open_grid(...)
-    drop_non_grid_data: bool = True,
+    drop_non_grid_data: bool = False,
# uxarray/grid/grid.py  Grid.__init__(...)
-        drop_non_grid_data: bool = True,
+        drop_non_grid_data: bool = False,
# uxarray/grid/grid.py  Grid.from_dataset(...)
-        drop_non_grid_data = kwargs.get("drop_non_grid_data", True)
+        drop_non_grid_data = kwargs.get("drop_non_grid_data", False)
  1. Speed issue with ds[name] lookups in _drop_non_grid_data, because every call materializes a new dataarray; recommend iterating through ds.variables instead.

@erogluorhan

Copy link
Copy Markdown
Member

@dylannelson, the back-and-forth on this PR is mostly on me. I approved on Aug 20 and reversed myself the next morning. I pushed back on the ndim == 0 narrowing that @rajeeja asked for and you had just implemented. Your Aug 25 question got a partial answer and then sat for ten weeks. That's four different scopes in three months.

This wasn't a normal PR: we were learning the problem while growing the fix, and you and then @Sevans711's consistent review carried most of that.

So instead of adding another opinion, I'll make the decision that unblocks the rest (review below). You've asked for a call a few times. Until reading below, if you still need that, let's do 30 minutes this week and close it out.

@erogluorhan erogluorhan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is my call on the open design questions. With these three changes I think this is ready:

  1. Filter once, on read. Call _drop_non_grid_data at the end of the auto-detect branch of Grid.from_dataset, not in Grid.__init__. That branch runs after every _read_*, so it is still one format-agnostic place (my Aug 11 point 4). It's also what @rajeeja suggested on Aug 17 and @Sevans711 on Aug 26. Copies and subsets no longer re-filter, so the bounds / face_bounds_* / max_face_radius entries and the slice.py / copy() plumbing go away.
  2. Keep scalar data variables, drop only scalar coordinates (@Sevans711's rule). Only coordinates are attached to the subset indexer, so a scalar time data variable can't cause #1444; the test suggested inline checks this. Adding @Sevans711's lat_lon / min_max dims means grid_topology and max_face_radius don't need listing either.
  3. No drop_non_grid_data option. We build the grid, and users shouldn't need to know or choose what we keep from the source file. The option is also only partly wired: from_topology, from_points and the other constructors ignore it, and 4 of its 9 hand-offs are untested. With 1 and 2 in place, there is little left for it to guard against.

I prototyped this (diff below). The filter becomes "grid dims plus lat_lon/min_max". It still has to keep descriptors (MPAS boundary_node_indices) and the grid_mapping CRS variable (needed for rioxarray's scalar spatial_ref coordinate). On every grid in test/meshfiles it keeps exactly what this branch keeps. The #1444 tests still pass, and in the full suite only the opt-out test and the scalar_data_var case fail, as expected.

Housekeeping:

  • Merge main; the branch has conflicts now.
  • In the description: remove the stray ( after "Closes #1444" and add "Closes #1720". Also note that Grid._ds / to_xarray() no longer carry non-grid variables (e.g. geoflow mesh_depth, FESOM2 nod_area), so it gets into the compatibility section of the release notes.
  • @rajeeja, you were right about ICON/FESOM2. clon/clat/vlon/... are kept now, and test_grid_files checks it. Could you withdraw or update your Aug 20 change request?
Prototype diff (filter placement and rule only; removing the option, its plumbing and the test updates are not included)
--- a/uxarray/grid/grid.py
+++ b/uxarray/grid/grid.py
@@ -203,7 +203,7 @@
         # drop variables that don't belong on the grid (e.g. a `time` carried in from a
         # file holding both grid and data), so they can't collide during subsetting.
         self._drop_non_grid_data = drop_non_grid_data
-        self._ds = _drop_non_grid_data(grid_ds) if drop_non_grid_data else grid_ds
+        self._ds = grid_ds
 
         # source grid specification (i.e. UGRID, MPAS, SCRIP, etc.)
         self.source_grid_spec = source_grid_spec
@@ -325,6 +325,7 @@
                     raise GridInvalidError(
                         f"Unsupported source_grid_spec={source_grid_spec!r}, in {cls.__name__}.from_dataset()"
                     )
+                grid_ds = _drop_non_grid_data(grid_ds)
             else:
                 # custom source grid spec is provided
                 source_grid_spec = kwargs.get("source_grid_spec", None)
--- a/uxarray/grid/utils.py
+++ b/uxarray/grid/utils.py
@@ -17,46 +17,22 @@
 
 
 def _drop_non_grid_data(ds):
-    """Drop variables that don't belong on a ``Grid`` (e.g. a ``time`` carried in
-    from a file that holds both the grid and its data).
-
-    A variable or coordinate is dropped when it has no dimensions, or when any of
-    its dimensions is not a UGRID grid dimension (``ugrid.DIM_NAMES``).
-
-    Variables uxarray reads or caches on the grid are always kept, whatever their
-    dimensions: grid coordinates, connectivity, descriptors (including MPAS
-    ``boundary_node_indices``), the cached ``bounds``, ``face_bounds_lon``,
-    ``face_bounds_lat`` and ``max_face_radius``, ``grid_topology``, and any CRS
-    variable named by a ``grid_mapping`` attribute (read by projected-grid
-    detection).
-    """
-    grid_dims = set(ugrid.DIM_NAMES)
-
-    keep = (
-        set(ugrid.SPHERICAL_COORD_NAMES)
-        | set(ugrid.CARTESIAN_COORD_NAMES)
-        | set(ugrid.CONNECTIVITY_NAMES)
-        | set(DESCRIPTOR_NAMES)
-        | {
-            "grid_topology",
-            "bounds",
-            "face_bounds_lon",
-            "face_bounds_lat",
-            "max_face_radius",
-        }
-    )
-    # CRS variables, referenced by name from a ``grid_mapping`` attribute
-    # (CF also allows the extended form "crs: x y")
-    for name in ds.variables:
-        grid_mapping = ds[name].attrs.get("grid_mapping")
-        if isinstance(grid_mapping, str):
-            keep.update(token.rstrip(":") for token in grid_mapping.split())
-
+    """Drop variables and coordinates with a non-grid dimension, and scalar coordinates."""
+    grid_dims = set(ugrid.DIM_NAMES) | {"lat_lon", "min_max"}
+    # descriptors (MPAS boundary_node_indices) and CRS variables named by grid_mapping
+    keep = set(DESCRIPTOR_NAMES) | {
+        var.attrs["grid_mapping"].split(":")[0]
+        for var in ds.variables.values()
+        if isinstance(var.attrs.get("grid_mapping"), str)
+    }
     drop = [
         name
-        for name in ds.variables
+        for name, var in ds.variables.items()
         if name not in keep
-        and (ds[name].ndim == 0 or any(dim not in grid_dims for dim in ds[name].dims))
+        and (
+            any(dim not in grid_dims for dim in var.dims)
+            or (var.ndim == 0 and name in ds.coords)
+        )
     ]
     return ds.drop_vars(drop)
 

Comment thread uxarray/grid/grid.py Outdated
# drop variables that don't belong on the grid (e.g. a `time` carried in from a
# file holding both grid and data), so they can't collide during subsetting.
self._drop_non_grid_data = drop_non_grid_data
self._ds = _drop_non_grid_data(grid_ds) if drop_non_grid_data else grid_ds

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this call to the end of the auto-detect branch in from_dataset, right after the _read_* chain (see summary). In my prototype, test_bounding_box_with_stray_grid_time and test_single_file_with_grid_and_time_varying_data pass unchanged. test_stray_variables_are_dropped only needs ux.Grid.from_dataset(...) instead of ux.Grid(...).

Comment thread test/grid/grid/test_non_grid_data.py Outdated
"coord_mixing_grid_and_non_grid_dims": lambda ds: ds.assign_coords(
z=(("n_node", "elevation"), np.zeros((ds.sizes["n_node"], 3)))
),
"scalar_data_var": lambda ds: ds.assign(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of asserting that a scalar time data variable is dropped, could this test assert that subsetting works with it present? It passes on main, on this branch and on the prototype:

grid_ds = xr.open_dataset(gridpath("ugrid", "quad-hexagon", "grid.nc"))
uxgrid = ux.open_grid(grid_ds.assign(time=xr.DataArray(np.datetime64("2016-01-01"))))
uxda = ux.UxDataArray(np.ones((2, uxgrid.n_face)), dims=("time", "n_face"),
                      coords={"time": TIMES}, uxgrid=uxgrid)
res = uxda.subset.bounding_box((-10, 10), (-10, 10))
np.testing.assert_array_equal(res["time"].values, TIMES)

If it ever fails, rule 4 was right and we keep it.

assert not set(dropped) & set(uxgrid._ds.variables)


def test_known_variable_on_misnamed_dim_kept(quad_hexagon_ds):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Fine to leave the keep-by-name for coordinates and connectivity as it is in this PR. The proper fix for the n2 shape is #1717, already fixed in #1736 (FESOM2 now gets two). In my prototype nothing in test/meshfiles needs the name list, so a follow-up can remove it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extra dims Handling of non-grid dimensions, such as time or altitude run-benchmark Run ASV benchmark workflow

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.subset with da that has time dimension

5 participants