Repository navigation
Conversation
…ore API Add `packages/zarr-storage`, a key-value storage layer that knows nothing about the Zarr spec. `DESIGN.md` is the contract. - `Store` is a view of a backend at a root: the one place that checks the key grammar, applies the root, and enforces read-only and closed states. `store / "a"` returns a view on the same backend; the constructed store owns the backend's lifecycle; there is no value equality (`shares_backend`, `overlaps` instead). - Prefix arguments name directories; `list_dir` yields `Key`, `Prefix` or `Entry`; `delete` removes one key and `delete_dir` a directory. - Reads return `None` for a missing key and take byte ranges (range, offset, suffix); `get_into` reads into a caller's buffer. Writes and deletes take `if_match` / `if_none_match` and report `applied=False` instead of raising. Every method has an `_async` twin. - Backends: `MemoryBackend`, `LocalBackend` (staged writes renamed into place under per-directory locks, unique nanosecond mtime stamps so generations don't repeat on filesystems that reuse inodes), and `ObjectStoreBackend` over obstore. - `zarr_storage.testing`: a conformance suite with capability mixins, run through both the sync and async methods, plus a Hypothesis state machine that checks random operations through random views against a model. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
…l operations Reads return `Found | Missing` (and `FoundRanges`, `Copied`) instead of `ReadResult | None`, so a missing key is a value an HTTP 404 maps onto. `get_if_changed` adds conditional reads that return `Unchanged` without reading bytes (HTTP 304, obstore `If-None-Match`). Conditional writes become their own methods: `put` returns `Written`, `put_if_absent` returns `Written | Rejected`, `put_if_match` and `delete_if_match` live in the `WritableIfMatch` and `DeletableIfMatch` capability protocols. Each method returns only the outcomes it can produce, and `delete_if_match` on an object store is now a type error rather than a runtime refusal. Backends share one implementation per operation, parameterized by a precondition function. Assisted-by: ClaudeCode:claude-opus-5-5
…s into `ReadableInto` and `ReadableIntoAsync` are now generic in the type of buffer the backend writes into, and `Store.get_into` accepts exactly that type. Host backends are `ReadableInto[Buffer]` and convert the buffer themselves; a GPU direct storage backend can be `ReadableInto` of its device buffer type, so passing it host memory (or the reverse) is a type error. The caller allocates, so the store needs no allocator. `Store.get_into` no longer falls back to `get` plus a copy for backends without `get_into`: the method is a type error on them. The obstore backend gains `get_into` (a read, then a copy), using the shared `copy_into` helper. Assisted-by: ClaudeCode:claude-opus-5-5
Store gains get_many, get_into_many, put_many and delete_many, with async twins. A batch is a loop of the single operation run concurrently where the backend can, with no atomicity: reads are lazy and yield (request, result) pairs as they finish; writes are eager and return results in the order given. Keys are checked before any operation starts, and a write batch naming a key twice raises ValueError. After a failure a batch starts nothing new, lets running operations finish, and raises BatchError (an ExceptionGroup) whose completed and remaining fields survive except*. Store runs any backend's batches serially; the optional ReadableMany, ReadableIntoMany[O] and WritableMany capabilities (and async twins) let a backend run them its own way. LocalBackend runs them on its thread pool, and ObjectStoreBackend's async batches keep `concurrency` requests in flight. The shared runners live in zarr_storage._batch. The state machine's new batch rules found a race in ObjectStoreBackend: a read or delete under a file, concurrent with deleting that file, raised the store's raw error. A refusal carrying ENOTDIR is now read as an absent key directly, instead of by re-checking the parents. Assisted-by: ClaudeCode:claude-opus-5-5
Store keeps the sync operations; the new AsyncStore offers the same operations under the same names as coroutines and async iterators (`await astore.get(k)`, `async for k in astore.list()`), so neither class carries the other's methods. Both derive from a private _View that holds the backend, root, read-only flag and ownership, and does key validation, root joining and read-only checks once. `store.as_async()` and `astore.as_sync()` are non-owning views on the same backend. Lifecycle differs between the two: Store has close() and `with`; AsyncStore has `await aclose()` and `async with`, which awaits a ClosableAsync backend's aclose (new capability, for cleanup that awaits a client session) and calls close on any other. LocalBackend gains aclose, running its pool shutdown on a worker thread. BREAKING CHANGE: Store's `*_async` methods are gone; use `store.as_async()` or construct an AsyncStore. Assisted-by: ClaudeCode:claude-opus-5-5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Add
zarr-storage, a new package with a Zarr-agnostic key-value storage API: aStoreview over memory, local-filesystem and obstore backends, with sync and async methods, conditional writes, byte-range reads and reads into caller buffers. It ships a conformance suite inzarr_storage.testingthat any backend can run.What the API is
packages/zarr-storage/DESIGN.mdis the contract; it records which source each decision came from (the roadmap,zng_storage, thezarr-python-planningstore proposals, TensorStore, zarrs).Storeis a view of a backend at a root. It is the one place the key grammar, the root, read-only and closed states are enforced; backends only see backend paths.store / "a"returns a view on the same backend. The constructed store owns the backend's lifecycle. There is no value equality:shares_backendandoverlapsanswer what callers ask./, no empty,.or..segments, no NUL. Prefix arguments name directories ("a"≡"a/").get(Nonewhen missing),get_ranges(one generation),get_into(caller's buffer),head,put/deletewithif_match/if_none_match(applied=Falseinstead of raising),delete_dir,list,list_dir(Key | Prefix | Entry). Every method has an_asynctwin; capability methods are self-typed, so calling one a backend lacks is a type error.MemoryBackend;LocalBackend(staged writes renamed into place under per-directory locks taken ancestor before descendant; unique nanosecond mtime stamps so generations don't repeat on filesystems that reuse inodes; conditional writes refused where file times are rounded);ObjectStoreBackendover obstore.Testing
zarr_storage.testing.StoreSuiteplus capability mixins (ConditionalPutSuite,KeyAndDirectorySuite,UniqueGenerationsSuite,ConditionalRaceSuite, ...) and refusing counterparts. Every test runs through the sync methods and again through the async methods.StoreSuite.test_state_machine: a Hypothesis state machine that drives random operations through random views against a dict model, including retired-generation replays.python:3.12-slim, overlay on ext4); pyright strict and ruff are clean. The agent ran five adversarial review rounds with mutation testing and stress scripts; their findings are fixed or listed under "Open" inDESIGN.md.Not in this PR
zarr_storage.legacy, the adapter inzarr, and the hierarchy layer; see the storage plan.🤖 Generated with Claude Code