Skip to content

Support one-file open_dataset#1479

Open
rajeeja wants to merge 15 commits into
mainfrom
rajeeja/onefile_open_dataset
Open

Support one-file open_dataset#1479
rajeeja wants to merge 15 commits into
mainfrom
rajeeja/onefile_open_dataset

Conversation

@rajeeja

@rajeeja rajeeja commented Mar 30, 2026

Copy link
Copy Markdown
Contributor

Closes #345 by allowing ux.open_dataset(file) for combined grid-and-data files.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for opening combined grid-and-data files via a single-argument ux.open_dataset(file) call, aligning with Issue #345’s request to handle datasets stored in one file.

Changes:

  • Made filename_or_obj optional in uxarray.core.api.open_dataset, defaulting to the grid file path when omitted.
  • Updated open_dataset docstring to document and exemplify the one-file usage.
  • Added a unit test covering the new one-argument open_dataset behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
uxarray/core/api.py Makes filename_or_obj optional and adds logic/docs for single-file grid+data opening.
test/core/test_api.py Adds a regression test for calling ux.open_dataset() with a single argument.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread uxarray/core/api.py
Comment thread uxarray/core/api.py
Comment thread test/core/test_api.py Outdated
@rajeeja rajeeja marked this pull request as ready for review April 1, 2026 20:44

@erogluorhan erogluorhan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for taking this one. Please find my comments below

Comment thread uxarray/core/api.py Outdated
Comment thread uxarray/core/api.py Outdated
Comment thread uxarray/core/api.py Outdated
Comment thread uxarray/core/api.py
Comment thread uxarray/core/api.py
Comment thread uxarray/core/api.py
Comment thread test/core/test_api.py
@rajeeja rajeeja requested a review from erogluorhan April 7, 2026 19:13
@rajeeja rajeeja force-pushed the rajeeja/onefile_open_dataset branch 2 times, most recently from cc35e13 to d65e10c Compare June 8, 2026 20:47
@rajeeja rajeeja force-pushed the rajeeja/onefile_open_dataset branch from d65e10c to 2dd95de Compare June 15, 2026 21:27
rajeeja and others added 2 commits July 13, 2026 13:04
being able to ux.open_dataset(single_file) makes it more likely to accidentally call ux.open_dataset(data_file), which crashes here. Old message was "Could not recognize dataset format", which confused me, because the data_file certainly stores a valid dataset...

New message clarifies it is an issue with not being able to get a uxgrid from this xarray.Dataset. This should provide a much better hint in the ux.open_dataset(data_file) case that the issue is the file doesn't contain enough info to make a grid.

@Sevans711 Sevans711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes mostly make sense, code looks reasonable, tests cover new cases. Tested locally and behavior seems reasonable. I read through the older comments in the thread on this PR as well. A few lingering thoughts / comments:

I was surprised to discover that this enables ux.open_dataset(grid_file) regardless of whether the grid file also contains "non-grid" data. Added a comment about this, I think it could be cleared up with just a slight change to the docstring.

I was confused by the "Directory-based gridfiles" crash. Nothing in the docstring suggested that the inputs could be directories in the first place, so I didn't realize directory-based gridfiles even existed at first. Suggested rephrasing. Also, I separately may be trying to add support for ux.open_dataset(directory) in some simple cases soon (see #1555). Clarifying this now will help with that.

Finally, I committed a minor change to error message when failing due to typo like ux.open_dataset(data_file). Before the error was confusing; now it is clearer that the issue is being unable to parse uxgrid information from the inputs.

Comment thread uxarray/core/api.py
Comment on lines +367 to +373
grid_filename_or_obj : str | os.PathLike[Any] | dict | xr.Dataset
Strings and Path objects are interpreted as a path to a grid file. Xarray Datasets assume that
each member variable is in the UGRID conventions and will be used to create a ``ux.Grid``. Similarly, a dictionary
containing UGRID variables can be used to create a ``ux.Grid``
filename_or_obj : str | os.PathLike[Any]
filename_or_obj : str | os.PathLike[Any] | xr.Dataset, optional
String or Path object as a path to a netCDF file or an OpenDAP URL that
stores the actual data set. It is the same ``filename_or_obj`` in
``xarray.open_dataset``.
stores the actual data set, or an already-open ``xarray.Dataset``. It

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to clarify the docs a bit:

  • grid_filename_or_obj docstring could start with something like "Grid information for the UxDataset" to help quickly clarify this is always the grid info.
  • grid_filename_or_obj docstring should clarify it can be a path to a directory containing grid files (but only if filename_or_obj is provided).
  • filename_or_obj should clarify to add something like "Or, it can be omitted to treat the grid as data, for any grid file"

The third point I am saying because these changes enable you to do open_dataset(grid_file) for any grid file, regardless of whether it is "combined" with data or not.

Comment thread uxarray/core/api.py Outdated
if isinstance(grid_filename_or_obj, (str, os.PathLike)):
if os.path.isdir(grid_filename_or_obj):
raise ValueError(
"Directory-based grids require a separate data file when calling ux.open_dataset()."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a "Directory-based grid?" This was my first thought. I explored the code a bit more to see there is one special case of directory-based grids.

I would maybe prefer an error like: "open_dataset(directory) is currently not supported. Consider supplying a path to a grid file instead. In special cases (e.g. for FESOM2 ASCII Dataset) directory-based grids can be recognized, but only if also providing a data file, such as via open_dataset(directory, data_file)."

Although… actually I am not really sure why we can't extend "open a directory-based-grid as a UxDataset" the same way we would be supporting "open a single-file grid as a UxDataset". How confusing is the downstream xarray crash?

@rajeeja

rajeeja commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@Sevans711 Docstrings clarified (grid-vs-data roles, single-arg reads non-grid vars as data) and the directory error message rewritten. Also noted partial / non-UGRID / non-Earth grids can be opened, features degrading as needed.

@Sevans711

Copy link
Copy Markdown
Collaborator

@rajeeja Thank you for adding those clarifications! One more thought on the new wording, and then I think the rest looks good and I would be happy to approve. Current wording:

If omitted,grid_filename_or_obj is also used as the data source, allowing a combined grid-and-data file or xarray.Dataset to be opened with a single argument. In this single-argument form the file is used both to build the grid and to load the data; any non-grid variables it contains are read as data.

I would change this to something like (bold here to show changes):

If omitted,grid_filename_or_obj is also used as the data source, allowing a combined grid-and-data file or xarray.Dataset to be opened with a single argument. In this single-argument form the file is used to build the grid and also treated as data; all variables it contains, including grid variables, will be included as data_vars in the result.

This clarifies the unexpected behavior I was noticing. E.g., result['node_lon'] exists when building the dataset this way, in addition to things like result['bottomDepth'] or other physical data. This behavior seems reasonable enough, it just wasn't my initial intuition about what would happen, so clarifying the docs should be sufficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 👀 In review

Development

Successfully merging this pull request may close these issues.

Investigate allowing the construction of UxDataset from a single grid & data file

5 participants