Skip to content

test(store): check every built-in store and wrapper against one contract - #4646

Draft
d-v-b wants to merge 5 commits into
zarr-developers:mainfrom
d-v-b:test/store-contract
Draft

d-v-b wants to merge 5 commits into
zarr-developers:mainfrom
d-v-b:test/store-contract

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

WrapperStore, and so LoggingStore, now forwards getsize, getsize_prefix, get_ranges_sync and supports_consolidated_metadata to the store it wraps. Before this change, a wrapper read a whole value to measure its size, and reported support for consolidated metadata even when the wrapped store reported none. LoggingStore now also logs range reads (get_ranges and get_ranges_sync).

What the PR adds

Two tests that check every built-in store against one contract, so a store that drifts fails CI instead of failing a user.

tests/test_store/test_stateful_conformance.py: a Hypothesis RuleBasedStateMachine that runs one sequence of set, set_if_not_exists, delete, delete_dir, get (with byte ranges), exists, getsize, list, list_prefix, list_dir, getsize_prefix and is_empty against a reference model of the store contract. It runs on ten store configurations: memory, managed-memory, local, zip (append-only), fsspec over file and memory, obstore over local and memory, logging and latency wrappers. The contract it checks:

  • a prefix names a directory: "a" ≡ "a/", "" is the root, and list_prefix("lat") never returns lat_bnds/...;
  • exists, get and getsize concern keys only;
  • byte ranges past the end of a value follow Python slicing.

The stores disagree in many places today. Each disagreement is an entry in a ledger (KNOWN_DIVERGENCES, plus UNMODELED_DIVERGENCES for writes the model cannot follow). Each entry has a short witness program, and test_divergence_ledger_is_exact runs every witness on every store and requires "diverges ⇔ listed". A store that is fixed, or that starts diverging, fails that test until the ledger is updated. The ledger therefore documents the cross-store behavior as it is now. Entries that come from POSIX filesystem errors (NotADirectoryError for a name below a key) apply only off Windows. Two Windows-only entries record obstore-local raising GenericError for exists/getsize on a directory. Stores skip one at a time: fsspec stores need fsspec ≥ 2024.12.0 (AsyncFileSystemWrapper), obstore stores need obstore, and the rest always run. This PR made such an update: latency-local now forwards getsize to LocalStore, so it joins local's two getsize entries.

test_delete_stays_in_root records, as strict xfails, that LocalStore.delete/delete_dir (and fsspec over file, for relative paths) follow .. and absolute paths out of the store root. It uses a sentinel directory inside tmp_path. The state machine never generates such paths: during development, an unconfined delete_dir("/") reached shutil.rmtree("/").

tests/test_store/test_wrapper.py: RecordingStore overrides every Store member (plus get_sync/set_sync/delete_sync) to record its arguments and return a distinct value. For each wrapper (WrapperStore, LoggingStore, LatencyStore, CacheStore) and each member, the test checks that the wrapped store sees exactly one call of the same method with the same arguments, and that the wrapper returns its result. test_every_store_member_is_checked fails when Store gains a member without a recipe here, so a new method can't silently skip the wrappers. REROUTED lists the methods a wrapper implements itself on purpose (for example, LatencyStore reads ranges through its own get to add latency), and the test requires each entry to still hold. Every recipe gives each defaulted argument a non-default value, so a wrapper that drops an argument is caught. test_logging_store_logs_every_store_method checks that LoggingStore logs every method it forwards.

Source changes
  • WrapperStore gains getsize, getsize_prefix, get_ranges_sync and supports_consolidated_metadata. Without them, these fell back to the Store defaults on the wrapper:
    • getsize read the whole value through get;
    • getsize_prefix listed and then read every value;
    • get_ranges_sync bypassed any override on the wrapped store;
    • supports_consolidated_metadata always returned True. Wrapping a store that manages its own consolidation (and returns False) in LoggingStore would therefore let consolidate_metadata write to it.
  • get_ranges_sync takes None for the coalescing kwargs to mean "use the wrapped store's default", like WrapperStore.get_ranges.
  • LoggingStore logs get_ranges, get_ranges_sync, _get_many, _set_many and supports_consolidated_metadata. Before, range reads reached the wrapped store without being logged or counted.
  • CacheStore.getsize now asks the source store directly, instead of going through the cache-aware get.
  • LatencyStore gains get_ranges_sync, routed through its own get_sync like its get_ranges. Without it, the new forwarding would have removed latency from sync range reads. LatencyStore.getsize no longer adds latency, because it no longer goes through get.
Found but not fixed here
  • CacheStore forwards delete_dir, set_if_not_exists, _set_many and clear to the source store without touching the cache, so later reads can serve stale entries.
  • CacheStore.__enter__ raises NotImplementedError, because _with_store does, so with CacheStore(...) fails. It's in REROUTED with that reason.
Testing
  • tests/test_store, tests/test_experimental, tests/test_codecs/test_sharding.py: pass locally on macOS. The fsspec-too-old path was checked by raising the version threshold locally. The Windows entries are confirmed only by CI.
  • An independent agent review found the first push's Windows and min_deps failures, LoggingStore's unlogged range reads, a loose unmodeled-divergence match in the ledger test, and a recipe that passed a default value. This revision addresses all of them.
  • The quick conformance runs (max_examples=25) take about 3 s. slow_hypothesis variants run for about 60 s each under just hypothesis.
  • Before the source fix, the wrapper test fails for getsize, getsize_prefix, get_ranges_sync and supports_consolidated_metadata on every wrapper.

🤖 Generated with Claude Code

d-v-b and others added 3 commits November 17, 2026 18:42
Add a Hypothesis state machine that applies each Store operation to ten
built-in stores at once (memory, managed memory, local, zip, fsspec over
file and memory, obstore over local and memory, and the logging and
latency wrappers) and compares every result with a reference model.

A disagreement fails the test unless it is in a ledger of known
divergences that records what the store does instead. A witness test
runs each ledger entry against every store, so an entry must name exactly
the stores that diverge: fixing a store, or a store starting to diverge,
fails the test until the ledger is updated.

The ledger starts with 25 entries, among them:
- list_prefix("lat") returns lat_bnds/... on memory, zip and the
  managed and logging memory stores (zarr-developers#3773, zarr-developers#4645);
- directories emptied by deletes are still listed by list_dir on the
  filesystem stores, and block writing a key of the same name (zarr-developers#4161);
- delete_dir("a") deletes the key "a" on fsspec;
- obstore raises for byte ranges at or past the end of a value, for
  zero-length ranges, and for most operations on a name below a key.

delete_dir on LocalStore and fsspec over file reaches outside the store
root for relative paths, and for absolute paths on LocalStore; a
sentinel-directory test records this as strict xfails.

Assisted-by: ClaudeCode:claude-opus-5-5
WrapperStore didn't forward getsize, getsize_prefix, get_ranges_sync or
supports_consolidated_metadata, so wrappers fell back to the Store
defaults: getsize read the whole value, and supports_consolidated_metadata
was True whatever the wrapped store said. LatencyStore now routes
get_ranges_sync through its own get_sync, as it does get_ranges.

A new test wraps a store that records every call and checks that each
wrapper passes every Store method straight through, apart from the
methods a wrapper reroutes on purpose. The conformance ledger now lists
latency-local with local for getsize, which it now forwards.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.99%. Comparing base (b0020f2) to head (791c0cf).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4646      +/-   ##
==========================================
+ Coverage   94.90%   94.99%   +0.08%     
==========================================
  Files          96       96              
  Lines       14055    14096      +41     
==========================================
+ Hits        13339    13390      +51     
+ Misses        716      706      -10     
Files with missing lines Coverage Δ
src/zarr/storage/_logging.py 100.00% <100.00%> (ø)
src/zarr/storage/_wrapper.py 100.00% <100.00%> (+1.88%) ⬆️
src/zarr/testing/store.py 98.59% <100.00%> (+0.26%) ⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

d-v-b and others added 2 commits November 17, 2026 22:15
… every platform

LoggingStore now logs get_ranges, get_ranges_sync, _get_many, _set_many
and supports_consolidated_metadata. A new test checks that it logs every
Store method it forwards, and another that each forwarding recipe gives
every defaulted argument a non-default value.

The conformance test now skips stores per factory instead of per module,
so memory, local and zip still run without obstore or fsspec, and fsspec
stores skip on fsspec < 2024.12.0. The ledger's below-a-key entries are
POSIX-only, and two Windows-only entries record obstore-local raising for
a directory. An unmodeled divergence now explains a ledger step only when
the steps end with its whole witness.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s#4646 fragment

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant