Skip to content

test(store): check every built-in store against one contract - #461

Closed
d-v-b wants to merge 1 commit into
mainfrom
test/store-conformance-machine
Closed

d-v-b wants to merge 1 commit into
mainfrom
test/store-conformance-machine

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 17, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

This adds a Hypothesis state machine that runs every Store operation against ten built-in stores at once and compares each result with one reference contract. Known disagreements live in a ledger that the tests keep exact, so it doubles as the list of interim store fixes.

What the test does
  • Stores: MemoryStore, ManagedMemoryStore, LocalStore, ZipStore, FsspecStore over file and over memory, ObjectStore over obstore's local and memory stores, LoggingStore(MemoryStore) and LatencyStore(LocalStore).
  • Operations: set, set_if_not_exists, delete, delete_dir, list, list_prefix, list_dir, getsize_prefix, is_empty, exists, get (with range, offset and suffix byte requests) and getsize.
  • The contract (module docstring):
    • a prefix names a directory: "a" and "a/" are the same directory, "" is the root, and list_prefix("lat") never returns lat_bnds/...;
    • exists, get and getsize concern keys only;
    • byte ranges past the end of a value return the bytes that exist.
  • The ledger: each KnownDivergence records what a store does instead, as a predicate. It excuses only that behavior, not the whole operation.
    • UNMODELED_DIVERGENCES holds cases the machine avoids, because a mismatch would leave the stores in different states.
    • test_divergence_ledger_is_exact runs every entry's witness against every store. Fixing a store, or a store starting to diverge, fails until the ledger is updated.
  • Run time: the quick variants run in normal CI in about 3 s. The slow_hypothesis variants take about 60 s and are picked up by just hypothesis (tests/test_store/test_stateful*).
Divergences recorded (25 entries)
Root escapes

delete_dir reaches outside the store root on LocalStore and LatencyStore(LocalStore) for both ../x and absolute paths, and on fsspec over file for ../x. test_delete_stays_in_root records these as strict xfails against a sentinel directory inside the test's temporary directory, so an escape can only delete the sentinel.

zarr itself can't reach this, because StorePath runs normalize_path, which rejects ./.. segments and strips a leading /. Direct Store users can. During development, a generated delete_dir("/") made LocalStore call shutil.rmtree("/"). Paths that leave the root are no longer generated.

Found while building it, not covered here
  • WrapperStore doesn't forward getsize, so wrapped stores read the whole object to get its size.
  • ObjectStore.list_dir starts a coroutine when called, so it needs a running event loop at call time.
  • FsspecStore.from_url("file://…") can't write nested keys without auto_mkdir=True.
Verification
  • Quick and witness tests pass: 248 passed, 9 skipped (ZipStore can't delete), 5 xfailed (root escapes).
  • Two randomized --run-slow-hypothesis runs passed, along with ruff, mypy and pre-commit.
  • An independent review checked the ledger store by store and found two missing stores and three predicates that were too loose. All are fixed.
  • Reach measured over 60 runs of 50 steps: trees average 3.7 keys, with up to 20; each directory read sees directories, keys, names cut short (la for lat) and names below a key.

🤖 Generated with Claude Code

Add a Hypothesis state machine that applies each Store operation to ten
built-in stores at once (memory, managed memory, local, zip, fsspec over
file and memory, obstore over local and memory, and the logging and
latency wrappers) and compares every result with a reference model.

A disagreement fails the test unless it is in a ledger of known
divergences that records what the store does instead. A witness test
runs each ledger entry against every store, so an entry must name exactly
the stores that diverge: fixing a store, or a store starting to diverge,
fails the test until the ledger is updated.

The ledger starts with 25 entries, among them:
- list_prefix("lat") returns lat_bnds/... on memory, zip and the
  managed and logging memory stores (zarr-developers#3773, zarr-developers#4645);
- directories emptied by deletes are still listed by list_dir on the
  filesystem stores, and block writing a key of the same name (zarr-developers#4161);
- delete_dir("a") deletes the key "a" on fsspec;
- obstore raises for byte ranges at or past the end of a value, for
  zero-length ranges, and for most operations on a name below a key.

delete_dir on LocalStore and fsspec over file reaches outside the store
root for relative paths, and for absolute paths on LocalStore; a
sentinel-directory test records this as strict xfails.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b

d-v-b commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Superseded by zarr-developers#4646, which carries this change rebased on upstream main plus a wrapper-forwarding test and fix.

@d-v-b d-v-b closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant