Skip to content

Eager and lazy grids with any duplicates now give different results, because #1692 deduplicated eager grids only #1905

Open

Description

@Sevans711

Since #1692 merged, eager (in-memory) grids and lazy (dask) grids now give different results, as eager grids get deduplicated but dask grids do not. Though this wasn't highlighted in the PR overview, the code comments provided some explanation for that, and claimed that de-duplication would occur later, whenever any duplicate-aware operation occurs:

# Preserve lazy readers such as chunked SCRIP. Canonicalization needs a
# global spatial search and is deferred until a duplicate-aware operation
# explicitly requests the map.

However, that is not what actually happens. Simple operations like grid_eager == grid_lazy now give different results due to different face_node_connectivity values. The difference also isn't just "cosmetic"; even something as simple as grid.n_edge now provides different results, and grid.validate() now fails on lazy grids with any duplicates. Discovered this while working on #1870, as some of the regression tests proposed there started to fail recently; if those tests had merged first then they would have blocked #1692 before this issue hit main.

cc: @rajeeja, @cmdupuis3

Expected Behavior

I expected that loading a grid using chunks=None (i.e., load into memory immediately) will lead me to the same exact results as a grid using chunks="auto" or any other non-None value. I would expect chunks to only change how memory is managed and when the data is loaded, for efficiency purposes only, with no effect on any of the final results.

Minimal Examples

Producing two examples here for convenience, one using a very small grid with Grid.from_topology, and another using one of the tutorial files.

Example 1

import dask.array as da
import numpy as np
import uxarray as ux

# Two triangles sharing an edge along the prime meridian. Triangle A writes
# that edge at lon=0 and triangle B at lon=360, so they are stored as separate
# nodes (as in SCRIP files such as outCSne8); dedup should merge nodes 3 and 5 into 1 and 2.
node_lon = np.array([350.0, 0.0, 0.0, 360.0, 10.0, 360.0])
node_lat = np.array([0.0, 0.0, 10.0, 0.0, 0.0, 10.0])
face_node_connectivity = np.array([[0, 1, 2], [3, 4, 5]])

grid_eager = ux.Grid.from_topology(node_lon, node_lat, face_node_connectivity)
grid_lazy = ux.Grid.from_topology(
    da.from_array(node_lon),
    da.from_array(node_lat),
    da.from_array(face_node_connectivity),
)

# expect (and, currently is) [[0 1 2] [1 4 2]]; was [[0 1 2] [3 4 5]] before #1692
print(grid_eager.face_node_connectivity.values)
# expect [[0 1 2] [1 4 2]]; actually get [[0 1 2] [3 4 5]], currently and before #1692.
print(grid_lazy.face_node_connectivity.values)   
print(grid_eager == grid_lazy)   # currently False, expected True
# Not purely cosmetic, e.g. n_edge disagrees (currently (5,6), expected same number):
grid_eager.n_edge, grid_lazy.n_edge
# Also, grid_lazy.validate() now fails with GridInvalidError, but it succeeded before #1692.
grid_lazy.validate()

Example 2

import numpy as np
import uxarray as ux

grid_eager = ux.tutorial.open_grid("scrip-outCSne8")
grid_lazy = ux.tutorial.open_grid("scrip-outCSne8", chunks=-1)

print(grid_eager == grid_lazy)  # currently False; True before #1692

# Same node coords, but different connectivities:
assert np.array_equal(grid_eager.node_lon.values, grid_lazy.node_lon.values)
assert np.array_equal(grid_eager.node_lat.values, grid_lazy.node_lat.values)
fnc_eager = grid_eager.face_node_connectivity.values
fnc_lazy = grid_lazy.face_node_connectivity.values
# currently 16 faces differ, expected 0 (was 0 before #1692)
print((fnc_eager != fnc_lazy).any(axis=1).sum(), "faces differ")
# expect (and currently is) 15 unreferenced nodes; was 0 before #1692.
print(grid_eager.n_node - len(np.unique(fnc_eager)), "unreferenced nodes (eager)")
# expect 15 unreferenced nodes; actually get 0 currently AND before #1692 
print(grid_lazy.n_node - len(np.unique(fnc_lazy)), "unreferenced nodes (lazy)")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingscalabilityRelated to scalability & performance efforts

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions