Repository navigation
Description
Since #1692 merged, eager (in-memory) grids and lazy (dask) grids now give different results, as eager grids get deduplicated but dask grids do not. Though this wasn't highlighted in the PR overview, the code comments provided some explanation for that, and claimed that de-duplication would occur later, whenever any duplicate-aware operation occurs:
uxarray/uxarray/grid/connectivity.py
Lines 651 to 653 in ce8690d
| # Preserve lazy readers such as chunked SCRIP. Canonicalization needs a | |
| # global spatial search and is deferred until a duplicate-aware operation | |
| # explicitly requests the map. |
However, that is not what actually happens. Simple operations like grid_eager == grid_lazy now give different results due to different face_node_connectivity values. The difference also isn't just "cosmetic"; even something as simple as grid.n_edge now provides different results, and grid.validate() now fails on lazy grids with any duplicates. Discovered this while working on #1870, as some of the regression tests proposed there started to fail recently; if those tests had merged first then they would have blocked #1692 before this issue hit main.
cc: @rajeeja, @cmdupuis3
Expected Behavior
I expected that loading a grid using chunks=None (i.e., load into memory immediately) will lead me to the same exact results as a grid using chunks="auto" or any other non-None value. I would expect chunks to only change how memory is managed and when the data is loaded, for efficiency purposes only, with no effect on any of the final results.
Minimal Examples
Producing two examples here for convenience, one using a very small grid with Grid.from_topology, and another using one of the tutorial files.
Example 1
import dask.array as da
import numpy as np
import uxarray as ux
# Two triangles sharing an edge along the prime meridian. Triangle A writes
# that edge at lon=0 and triangle B at lon=360, so they are stored as separate
# nodes (as in SCRIP files such as outCSne8); dedup should merge nodes 3 and 5 into 1 and 2.
node_lon = np.array([350.0, 0.0, 0.0, 360.0, 10.0, 360.0])
node_lat = np.array([0.0, 0.0, 10.0, 0.0, 0.0, 10.0])
face_node_connectivity = np.array([[0, 1, 2], [3, 4, 5]])
grid_eager = ux.Grid.from_topology(node_lon, node_lat, face_node_connectivity)
grid_lazy = ux.Grid.from_topology(
da.from_array(node_lon),
da.from_array(node_lat),
da.from_array(face_node_connectivity),
)
# expect (and, currently is) [[0 1 2] [1 4 2]]; was [[0 1 2] [3 4 5]] before #1692
print(grid_eager.face_node_connectivity.values)
# expect [[0 1 2] [1 4 2]]; actually get [[0 1 2] [3 4 5]], currently and before #1692.
print(grid_lazy.face_node_connectivity.values)
print(grid_eager == grid_lazy) # currently False, expected True
# Not purely cosmetic, e.g. n_edge disagrees (currently (5,6), expected same number):
grid_eager.n_edge, grid_lazy.n_edge
# Also, grid_lazy.validate() now fails with GridInvalidError, but it succeeded before #1692.
grid_lazy.validate()Example 2
import numpy as np
import uxarray as ux
grid_eager = ux.tutorial.open_grid("scrip-outCSne8")
grid_lazy = ux.tutorial.open_grid("scrip-outCSne8", chunks=-1)
print(grid_eager == grid_lazy) # currently False; True before #1692
# Same node coords, but different connectivities:
assert np.array_equal(grid_eager.node_lon.values, grid_lazy.node_lon.values)
assert np.array_equal(grid_eager.node_lat.values, grid_lazy.node_lat.values)
fnc_eager = grid_eager.face_node_connectivity.values
fnc_lazy = grid_lazy.face_node_connectivity.values
# currently 16 faces differ, expected 0 (was 0 before #1692)
print((fnc_eager != fnc_lazy).any(axis=1).sum(), "faces differ")
# expect (and currently is) 15 unreferenced nodes; was 0 before #1692.
print(grid_eager.n_node - len(np.unique(fnc_eager)), "unreferenced nodes (eager)")
# expect 15 unreferenced nodes; actually get 0 currently AND before #1692
print(grid_lazy.n_node - len(np.unique(fnc_lazy)), "unreferenced nodes (lazy)")Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Projects
- StatusShow more project fields📚 Backlog
Activity
update tests due to #1905