Repository navigation
Conversation
`from_array(..., overwrite=True)` refused to overwrite a destination that overlaps its source only when both stores compared equal. A destination store rooted inside the source array's directory (for example `LocalStore(d / "a")` for an array in `LocalStore(d)` at `a`) compared unequal, so `init_array` deleted the source's chunks and the copy wrote fill values. Stores now report where their data lives through a private `Store._data_location()` hook, `(backend, root)`, implemented for the local, memory, managed-memory, zip, fsspec and wrapper stores. The overlap check compares backends and then whether the roots joined with the paths nest. Stores without a location keep the previous equality-based check. The `store2` test fixture put the second store inside the first store's directory, which this check now reports as overlapping; it gets its own temporary directory. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Fix
from_arrayoverwriting its source array when the destination store is rooted inside the source's directory: overlapping store paths are now detected across stores, not only within one store.Summary
from_array(..., overwrite=True)checks that the destination doesn't overlap the source beforeinit_arraydeletes the destination. The check only ran when the two stores compared equal (_stores_share_data). Two stores on the same data with different roots compare unequal, so the check passed. The agent reproduced the data loss onmain(d4d4d38):This PR adds a private hook,
Store._data_location() -> tuple[Hashable, str] | None, returning(backend, root):LocalStoreMemoryStore,ManagedMemoryStore""/pathZipStore""FsspecStorepathWrapperStore(incl.LoggingStore,CacheStore)ObjectStore)None: keeps the previous equality-based check_store_paths_overlapreplaces_stores_share_data: same backend, then whether the roots joined with the paths nest (by/-separated directory, soaandabdon't overlap).Behavior change
The check covers both directions, as before: a destination inside the source array's directory also raises, because its keys can collide with the source's chunk keys (a destination named
cinside a v3 source writes intoc/…). Thestore2test fixture rooted the second store attmp_path / "store2", inside the first store's root, sotest_creation_from_other_zarr_format[*-local-local]wrote an array inside the source array's directory. That layout now raises; the fixture uses its own temporary directory instead.The error message now names full locations (
str(StorePath)), since the two stores can differ.Tests
test_store_paths_overlap: 11 store pairs (nested and siblingLocalStoreroots, read-only views, shared and equal-contentMemoryStores,ManagedMemoryStorepaths, twoZipStores on one file, nestedFsspecStoreroots, aLoggingStorewrapper, different backends), each checked in both argument orders.test_from_array_overwrite_nested_store_root_raises: the reproducer above raises and leaves the source intact.Context
Found while auditing every use of
Store.__eq__(instrumented across the test suite): this overlap check is the only production caller, and it needs "do these stores share data", which equality can't answer. The planned storage redesign replaces store value equality with a backend-sharing check; this hook is the first piece of that.🤖 Generated with Claude Code