Repository navigation
Can't call open_mfdataset without creating chunked dask arrays #9038
Description
What happened?
Passing chunks=None to xr.open_dataset/open_mfdataset is supposed to avoid using dask at all, returning lazily-indexed numpy arrays even if dask is installed. However chunks=None doesn't currently work for xr.open_mfdataset as it gets silently coerced internally to chunks={}, which creates dask chunks aligned with the on-disk files.
Offending line of code:
Line 1040 in 12123be
What did you expect to happen?
Passing chunks=None to open_mfdataset should return lazily-indexed numpy arrays, like open_dataset does.
Minimal Complete Verifiable Example
ds = xr.tutorial.open_dataset("air_temperature")
ds1 = ds.isel(time=slice(None, 1000))
ds2 = ds.isel(time=slice(1000, None))
ds1.to_netcdf('air1.nc')
ds2.to_netcdf('air2.nc')
combined = xr.open_mfdataset(['air1.nc', 'air2.nc'], chunks=None)
print(type(combined['air'].data))MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
dask.array.core.ArrayAnything else we need to know?
As the default is None, changing this without changing the default would be a breaking change. But the current behaviour is also not intended.
Environment
main
Activity
dcherian commentedon May 21, 2024
Can't do this without virtual concat machinery (#4628) which someone decided to implement elsewhere 🙄 ;)
We could change the default to
chunks={}in anticipation though.TomNicholas commentedon May 21, 2024
😅
It's still broken at the moment though - I had a (ridiculous) case where I don't care that concat will load everything in memory, I just want to completely avoid creating
dask.arrayobjects, and right now there is no possible input option toopen_mfdatasetto do that.That's probably more useful, as well as actually being consistent.
Illviljan commentedon May 22, 2024
See #5704 for changing
chunks={}and more discussion.weiji14 commentedon Jun 12, 2026
Agree with having the ability to totally avoid dask and/or any chunking manager, and have the possibility of letting the backend engine handle things on its own. Will also call out that the lazy-indexing shouldn't just return NumPy arrays, but allow for other Array API types (e.g. CuPy arrays).
To elaborate on my use-case, I'm hitting into issues at xarray-contrib/cupy-xarray#81 (comment) because
xr.open_mfdatasetcan't avoid using dask at all, and dask defaults to returning numpy.arrays, but I want CuPy arrays 🙂 (edit: went down the 🐇🕳️ and found #8733 (comment) that mentioned cupy!). Usingxr.open_dataset(..., chunks=None)works fine though, it is just thatchunks=chunks or {}part that is tripping things up inxr.open_mfdataset.Add requires_dask marker to two unit tests checking open_mfdataset