Skip to content

feat(zarr-storage): add a Zarr-agnostic storage package with a new store API - #460

Draft
d-v-b wants to merge 6 commits into
mainfrom
feat/zarr-storage-api
Draft

d-v-b wants to merge 6 commits into
mainfrom
feat/zarr-storage-api

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 17, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Add zarr-storage, a new package with a Zarr-agnostic key-value storage API: a Store view over memory, local-filesystem and obstore backends, with sync and async methods, conditional writes, byte-range reads and reads into caller buffers. It ships a conformance suite in zarr_storage.testing that any backend can run.

What the API is

packages/zarr-storage/DESIGN.md is the contract; it records which source each decision came from (the roadmap, zng_storage, the zarr-python-planning store proposals, TensorStore, zarrs).

  • Store is a view of a backend at a root. It is the one place the key grammar, the root, read-only and closed states are enforced; backends only see backend paths. store / "a" returns a view on the same backend. The constructed store owns the backend's lifecycle. There is no value equality: shares_backend and overlaps answer what callers ask.
  • Keys: no leading/trailing /, no empty, . or .. segments, no NUL. Prefix arguments name directories ("a" ≡ "a/").
  • get (None when missing), get_ranges (one generation), get_into (caller's buffer), head, put/delete with if_match/if_none_match (applied=False instead of raising), delete_dir, list, list_dir (Key | Prefix | Entry). Every method has an _async twin; capability methods are self-typed, so calling one a backend lacks is a type error.
  • Backends: MemoryBackend; LocalBackend (staged writes renamed into place under per-directory locks taken ancestor before descendant; unique nanosecond mtime stamps so generations don't repeat on filesystems that reuse inodes; conditional writes refused where file times are rounded); ObjectStoreBackend over obstore.
Testing
  • zarr_storage.testing.StoreSuite plus capability mixins (ConditionalPutSuite, KeyAndDirectorySuite, UniqueGenerationsSuite, ConditionalRaceSuite, ...) and refusing counterparts. Every test runs through the sync methods and again through the async methods.
  • StoreSuite.test_state_machine: a Hypothesis state machine that drives random operations through random views against a dict model, including retired-generation replays.
  • Race tests (concurrent creates, conditional put vs delete, ranges under a rewriting writer), a tripwire that fails any blocking obstore call made on an event-loop thread, and forced-interleaving tests for the local backend's locking.
  • 1472 tests pass on macOS and in a Linux container (python:3.12-slim, overlay on ext4); pyright strict and ruff are clean. The agent ran five adversarial review rounds with mutation testing and stress scripts; their findings are fixed or listed under "Open" in DESIGN.md.
Not in this PR
  • fsspec and zip backends, transactions, range coalescing, caching.
  • Moving the current store API into zarr_storage.legacy, the adapter in zarr, and the hierarchy layer; see the storage plan.
  • No Windows run (the Windows name rules are unit-tested only); no CI workflow for the package yet.

🤖 Generated with Claude Code

…ore API

Add `packages/zarr-storage`, a key-value storage layer that knows nothing
about the Zarr spec. `DESIGN.md` is the contract.

- `Store` is a view of a backend at a root: the one place that checks the
  key grammar, applies the root, and enforces read-only and closed states.
  `store / "a"` returns a view on the same backend; the constructed store
  owns the backend's lifecycle; there is no value equality
  (`shares_backend`, `overlaps` instead).
- Prefix arguments name directories; `list_dir` yields `Key`, `Prefix` or
  `Entry`; `delete` removes one key and `delete_dir` a directory.
- Reads return `None` for a missing key and take byte ranges (range,
  offset, suffix); `get_into` reads into a caller's buffer. Writes and
  deletes take `if_match` / `if_none_match` and report `applied=False`
  instead of raising. Every method has an `_async` twin.
- Backends: `MemoryBackend`, `LocalBackend` (staged writes renamed into
  place under per-directory locks, unique nanosecond mtime stamps so
  generations don't repeat on filesystems that reuse inodes), and
  `ObjectStoreBackend` over obstore.
- `zarr_storage.testing`: a conformance suite with capability mixins, run
  through both the sync and async methods, plus a Hypothesis state machine
  that checks random operations through random views against a model.

Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…l operations

Reads return `Found | Missing` (and `FoundRanges`, `Copied`) instead of
`ReadResult | None`, so a missing key is a value an HTTP 404 maps onto.
`get_if_changed` adds conditional reads that return `Unchanged` without
reading bytes (HTTP 304, obstore `If-None-Match`).

Conditional writes become their own methods: `put` returns `Written`,
`put_if_absent` returns `Written | Rejected`, `put_if_match` and
`delete_if_match` live in the `WritableIfMatch` and `DeletableIfMatch`
capability protocols. Each method returns only the outcomes it can
produce, and `delete_if_match` on an object store is now a type error
rather than a runtime refusal. Backends share one implementation per
operation, parameterized by a precondition function.

Assisted-by: ClaudeCode:claude-opus-5-5
…s into

`ReadableInto` and `ReadableIntoAsync` are now generic in the type of
buffer the backend writes into, and `Store.get_into` accepts exactly that
type. Host backends are `ReadableInto[Buffer]` and convert the buffer
themselves; a GPU direct storage backend can be `ReadableInto` of its
device buffer type, so passing it host memory (or the reverse) is a type
error. The caller allocates, so the store needs no allocator.

`Store.get_into` no longer falls back to `get` plus a copy for backends
without `get_into`: the method is a type error on them. The obstore
backend gains `get_into` (a read, then a copy), using the shared
`copy_into` helper.

Assisted-by: ClaudeCode:claude-opus-5-5
Store gains get_many, get_into_many, put_many and delete_many, with async
twins. A batch is a loop of the single operation run concurrently where
the backend can, with no atomicity: reads are lazy and yield
(request, result) pairs as they finish; writes are eager and return
results in the order given. Keys are checked before any operation
starts, and a write batch naming a key twice raises ValueError. After a
failure a batch starts nothing new, lets running operations finish, and
raises BatchError (an ExceptionGroup) whose completed and remaining
fields survive except*.

Store runs any backend's batches serially; the optional ReadableMany,
ReadableIntoMany[O] and WritableMany capabilities (and async twins) let
a backend run them its own way. LocalBackend runs them on its thread
pool, and ObjectStoreBackend's async batches keep `concurrency`
requests in flight. The shared runners live in zarr_storage._batch.

The state machine's new batch rules found a race in ObjectStoreBackend:
a read or delete under a file, concurrent with deleting that file,
raised the store's raw error. A refusal carrying ENOTDIR is now read as
an absent key directly, instead of by re-checking the parents.

Assisted-by: ClaudeCode:claude-opus-5-5
Store keeps the sync operations; the new AsyncStore offers the same
operations under the same names as coroutines and async iterators
(`await astore.get(k)`, `async for k in astore.list()`), so neither
class carries the other's methods. Both derive from a private _View that
holds the backend, root, read-only flag and ownership, and does key
validation, root joining and read-only checks once. `store.as_async()`
and `astore.as_sync()` are non-owning views on the same backend.

Lifecycle differs between the two: Store has close() and `with`;
AsyncStore has `await aclose()` and `async with`, which awaits a
ClosableAsync backend's aclose (new capability, for cleanup that awaits a
client session) and calls close on any other. LocalBackend gains aclose,
running its pool shutdown on a worker thread.

BREAKING CHANGE: Store's `*_async` methods are gone; use
`store.as_async()` or construct an AsyncStore.

Assisted-by: ClaudeCode:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant