Skip to content

fix(array): detect overlapping source and destination across store roots - #459

Draft
d-v-b wants to merge 2 commits into
mainfrom
fix/from-array-overlapping-store-roots
Draft

d-v-b wants to merge 2 commits into
mainfrom
fix/from-array-overlapping-store-roots

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 17, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Fix from_array overwriting its source array when the destination store is rooted inside the source's directory: overlapping store paths are now detected across stores, not only within one store.

Summary

from_array(..., overwrite=True) checks that the destination doesn't overlap the source before init_array deletes the destination. The check only ran when the two stores compared equal (_stores_share_data). Two stores on the same data with different roots compare unequal, so the check passed. The agent reproduced the data loss on main (d4d4d38):

src = zarr.create_array(LocalStore(d), name="a", shape=(4,), chunks=(2,), dtype="i4")
src[:] = [1, 2, 3, 4]
zarr.from_array(LocalStore(d / "a"), data=src, overwrite=True)[:]  # [0 0 0 0]; the source is gone too

This PR adds a private hook, Store._data_location() -> tuple[Hashable, str] | None, returning (backend, root):

Store backend root
LocalStore filesystem resolved absolute path
MemoryStore, ManagedMemoryStore the dict object "" / path
ZipStore resolved file path, or the file object ""
FsspecStore the filesystem's cache token path
WrapperStore (incl. LoggingStore, CacheStore) delegates to the wrapped store
others (e.g. ObjectStore) None: keeps the previous equality-based check

_store_paths_overlap replaces _stores_share_data: same backend, then whether the roots joined with the paths nest (by /-separated directory, so a and ab don't overlap).

Behavior change

The check covers both directions, as before: a destination inside the source array's directory also raises, because its keys can collide with the source's chunk keys (a destination named c inside a v3 source writes into c/…). The store2 test fixture rooted the second store at tmp_path / "store2", inside the first store's root, so test_creation_from_other_zarr_format[*-local-local] wrote an array inside the source array's directory. That layout now raises; the fixture uses its own temporary directory instead.

The error message now names full locations (str(StorePath)), since the two stores can differ.

Tests
  • test_store_paths_overlap: 11 store pairs (nested and sibling LocalStore roots, read-only views, shared and equal-content MemoryStores, ManagedMemoryStore paths, two ZipStores on one file, nested FsspecStore roots, a LoggingStore wrapper, different backends), each checked in both argument orders.
  • test_from_array_overwrite_nested_store_root_raises: the reproducer above raises and leaves the source intact.
  • Full suite locally: 13380 passed, 1319 skipped, 9 xfailed. mypy and pre-commit clean.
Context

Found while auditing every use of Store.__eq__ (instrumented across the test suite): this overlap check is the only production caller, and it needs "do these stores share data", which equality can't answer. The planned storage redesign replaces store value equality with a backend-sharing check; this hook is the first piece of that.

🤖 Generated with Claude Code

`from_array(..., overwrite=True)` refused to overwrite a destination that
overlaps its source only when both stores compared equal. A destination
store rooted inside the source array's directory (for example
`LocalStore(d / "a")` for an array in `LocalStore(d)` at `a`) compared
unequal, so `init_array` deleted the source's chunks and the copy wrote
fill values.

Stores now report where their data lives through a private
`Store._data_location()` hook, `(backend, root)`, implemented for the
local, memory, managed-memory, zip, fsspec and wrapper stores. The
overlap check compares backends and then whether the roots joined with
the paths nest. Stores without a location keep the previous
equality-based check.

The `store2` test fixture put the second store inside the first store's
directory, which this check now reports as overlapping; it gets its own
temporary directory.

Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant