Skip to content

Improved cleaning of grids objects to avoid leaked data #1720

Description

@dylannelson

Proposed new feature or change:

While diagnosing #1444 and developing #1667, we noticed an increasing scope. It started with being concerned about a specific instance of time leaking onto a grid object, causing downstream issues while using subset.
After finding a fix for this scenario, we decided we wanted to make a more robust grid object cleaning function, that can clean grid objects in a way that avoids other potentially damaging situations like this. Doing so we found other interactions with the IO, like #1717 that showed we need to further investigate making rules around cleaning, as there is a surprisingly unique range of incoming data variables, with characteristics that are uncommon, but comparable to what we imagine "leaked" data to look like.
A few main goals:

  1. Reduce the amount of potential incoming grid data variables
    • Based on what we don't need or don't expect to have after a grid file is read in
  2. Reduce the amount of potential incoming coordinates
    • Based on what we don't need or don't expect to have after a grid file is read in, and if they are 0 dimensional (dims=() or "scalar")
  3. Make a method that can run without needing model/format specific solutions
  4. Ensure that no models have their unique attributes removed if they are needed
  5. Ensure that the function is placed within the file read process so that it doesn't disrupt other existing parts

Added notes:

  • We are interested in keeping at a minimum our existing GRID_DIMS:
['n_edge', 'n_face', 'n_max_edge_edges', 'n_max_edge_faces', 
'n_max_edge_nodes', 'n_max_face_edges', 'n_max_face_faces', 
'n_max_face_nodes', 'n_max_node_edges', 'n_max_node_faces', 
'n_node', 'two']
  • We want cases like we've seen, and cases we've not seen, to be covered by this fix (in these examples, time should be assumed to be any variable we don't want/need):
    • time present as a dimension coordinate (1-D, time(time), ndim == 1)
    • time present as a scalar (0-dimensional) coordinate (dims == (), ndim == 0)
    • time present as a dimension without a coordinate (a bare time dimension carrying data variables, no time coordinate)
    • Meaningful dimensions (dimensions within DIM_NAMES) that should be there, but aren't, due to it's unique IO process

These are some of the instances we've seen and want to keep them all in mind when finding a solution. This may be blocked until we understand #1717 better.

Activity

  1. self-assigned this
    on Aug 28, 2026
  2. added
    extra dimsHandling of non-grid dimensions, such as time or altitude
    and removed
    new featureNew user-facing functionality
    on Aug 30, 2026
  3. Sevans711 commented on Aug 31, 2026

    @Sevans711
    Collaborator

    Thank you for opening an issue on this! This description mostly matches my understanding of the problem. I have a few tiny comments below. We might already be on the same page about these but I'm including them here to clarify:

    1. Ensure that the function is placed within the file read process so that it doesn't disrupt other existing parts

    Rather, I was thinking the idea was to put the function directly within the Grid object constructor (i.e. Grid.__init__). I wouldn't consider this to be part of the file read process; it acts on the dataset that gets provided to Grid, regardless of whether the grid is being created from file data or from an xarray.Dataset provided directly as Grid(grid_ds, ...).

    • We want cases like we've seen, and cases we've not seen, to be covered by this fix (in these examples, time should be assumed to be any variable we don't want/need):
      • time present as a dimension coordinate (1-D, time(time), ndim == 1)
      • time present as a scalar (0-dimensional) coordinate (dims == (), ndim == 0)
      • time present as a dimension without a coordinate (a bare time dimension carrying data variables, no time coordinate)
      • Meaningful dimensions (dimensions within DIM_NAMES) that should be there, but aren't, due to it's unique IO process

    These are good examples. If the goal is a comprehensive list I would add a few kinds of coordinates that could theoretically pop up and aren't covered above:

    • time_since_epoch present as a non-dimension coordinate (e.g. 1D coordinate with time dimension)
    • special_index present as a (non-dimension) coordinate with N>1 dimensions (e.g. special_index with dims "time" and "elevation", with value len(time) * time_index + elevation_index; the value itself isn't important, just hoping to clarify that 2D+ coords across non-grid dims are possible)
    • z present as a (non-dimension) coordinate with N>1 dimensions, at least one of which is not a grid dimension (e.g. x, y, z representing geocentric coordinates of data could each have dimensions "node_lon", "node_lat", "elevation"; elevation is not a grid dimension so these coords would need to be removed.)

    This may be blocked until we understand #1717 better.

    I agree this would be blocked by #1717, though to me it seems like 1717 is already a well-understood and clearly-reported issue with a simple fix (rename "n2" to "two" in fesom io functions), so I would maybe rephrase to "until #1717 is solved and merged" instead. Is 1717 more complicated than I think?

  4. dylannelson commented on Aug 31, 2026

    @dylannelson
    MemberAuthor

    @Sevans711

    Is 1717 more complicated than I think?

    I don't know, we'll see as I get into it, haven't started yet.

    so I would maybe rephrase to "until #1717 is solved and merged" instead

    What is the value of discussing semantics like this? Saying "X is blocked by Y" and "X is blocked until Y is solved and merged" sounds basically the same. I need to do X before Y. That's all I'm trying to say. Not making any statement on it's difficulty, and if I was, I'd do it on that issue's discussion, no?

  5. Sevans711 commented on Aug 31, 2026

    @Sevans711
    Collaborator

    What is the value of discussing semantics like this?

    I think the only value of it here is just to make sure we agree on the underlying ideas. For some reason the phrasing "blocked until we understand 1717" sounded to me like a claim about the difficulty of 1717. Your reply above helped me see that wasn't actually your intention, so I believe we are on the same page here - thank you for clarifying! I agree it makes the most sense to put claims about an issue's difficulty directly on that issue's discussion.

  6. moved this from 📚 Backlog to 📝 To-Do in UXarray Developmenton Aug 31, 2026
  7. moved this from 📝 To-Do to 🏗 In progress in UXarray Developmenton Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

extra dimsHandling of non-grid dimensions, such as time or altitude

Fields

Priority

Medium

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions