diff --git a/packages/zarr-storage/CHANGELOG.md b/packages/zarr-storage/CHANGELOG.md new file mode 100644 index 0000000000..7c4bc92cad --- /dev/null +++ b/packages/zarr-storage/CHANGELOG.md @@ -0,0 +1,3 @@ +# Release notes + + diff --git a/packages/zarr-storage/DESIGN.md b/packages/zarr-storage/DESIGN.md new file mode 100644 index 0000000000..46edd3d635 --- /dev/null +++ b/packages/zarr-storage/DESIGN.md @@ -0,0 +1,234 @@ +# zarr-storage design (v0) + +Status: draft, 2026-11-17. This is the new storage API from the storage plan +(`proposals/store-hierarchy-layering.md` in the planning worktree). It is built in iterations; each +iteration ends with every backend passing the shared conformance suite. + +Sources, and what each contributed: + +- **zng_storage** (Davis Bennett's earlier prototype): capability protocols instead of a class + hierarchy, a sync floor with an async family beside it, generations and result types, `store / "sub"`. +- **TensorStore**: a store is a driver plus a path (`KvStore{driver, path}`); generations with + "unconditional" and "absent" meanings; reserved file-name suffixes for the file driver's own files; + in-flight operations outlive the last reference. +- **zarrs**: `list_dir` returning keys and prefixes; the `StoreKey`/`StorePrefix` grammar; a byte-range + type with suffix ranges; a missing key is a value (`Option`), not an error. +- **zarr-python-planning store proposals**: result types and `ObjectMetadata`, conditional writes that + report a refusal instead of raising, the conformance-suite layout. +- **HTTP and Rust `object_store`**: a missing key and an unchanged value are outcomes (`404`, `304`), + not errors; conditional operations are their own methods (`put` vs `put_opts`). +- **The storage plan**: directory prefixes, `Key | Prefix | Entry` listing, `delete` vs `delete_dir`, + views with the root applied once, the constructed store owns the backend, no value equality. +- **The roadmap**: separate sync and async surfaces, reads into caller-provided buffers, typed explicit + configuration, transactions later. + +## Shape + +``` +Backend capability protocols over backend paths; owns shared state and lifecycle + ▲ +_View (backend, root, read_only, owner): validates keys, applies the root, relativizes + ▲ listings, enforces read-only; `store / "a"` is a view on the same backend +Store the operations as sync methods; `close()`, `with`; `as_async()` +AsyncStore the same names as coroutines; `aclose()`, `async with`; `as_sync()` +``` + +A **backend** is an object that implements some capability protocols (`Readable`, `Writable`, +`Listable`, `WritableIfMatch`, `DeletableIfMatch`, and their async twins) on *backend paths*: +`/`-separated strings, already joined with the root. Backends never see relative keys and never join or strip paths. They hold everything that must exist once: the dict, +the filesystem root, a client, locks, concurrency limits. + +A **`Store`** or **`AsyncStore`** is the only object users hold. Each is a view: a backend plus a +root. All key validation, root joining, listing relativization and read-only enforcement happen in +their shared base, once, so no backend can get them wrong on its own (the source of most bugs in the legacy API: #2190, #2467, #3639, #3922, #4204, +#4452). + +**Sync and async are two classes** (as zarrs has separate sync and async storage traits): `Store` +calls a backend's sync protocols and `AsyncStore` its async ones, under the same method names, so +`store.get(k)` and `await astore.get(k)` read the same way and neither class carries the other's +methods. A backend with both surfaces serves both; `store.as_async()` and `astore.as_sync()` are +views on the same backend that don't own it. Lifecycle differs too: `Store` has `close()` and +`with`, `AsyncStore` has `await aclose()` and `async with`, which awaits a `ClosableAsync` +backend's `aclose` (a backend whose cleanup awaits a client session) and calls `close` on any other. + +## Keys and directories + +- **Key:** a string with no leading `/`, no trailing `/`, no empty segment, no `.` or `..` segment, no + NUL character, and not empty. Validated by `Store` on every key argument; `InvalidKeyError` + otherwise. A backend may refuse more with `InvalidKeyError`: the local backend refuses its own + reserved suffix (as TensorStore's file driver does), names too long for the filesystem, and on + Windows the names Windows reads as something else (`:`, `\`, device names like `NUL`, a trailing `.` + or space). +- **Directory:** every prefix argument names a directory. `""` is the root; `"a"` and `"a/"` are the same + directory; otherwise it follows the key grammar. `list("lat")` never sees `lat_bnds/…`. +- **Root:** a store's root is a directory in the backend. `store / "a/b"` appends to it. + +## Operations (`Store`; `AsyncStore` has each under the same name, as a coroutine or async iterator) + +| Method | Returns | Notes | +|---|---|---| +| `get(key, *, byte_range=None)` | `Found \| Missing` | | +| `get_if_changed(key, generation, *, byte_range=None)` | `Found \| Missing \| Unchanged` | `Unchanged` without reading bytes when the value still has `generation` (HTTP `If-None-Match`), to revalidate a cached copy | +| `get_ranges(key, ranges)` | `FoundRanges \| Missing` | all parts from one generation (object stores pin later reads to the first read's generation, and fall back to one whole read if it changed) | +| `get_into(key, out, *, byte_range=None)` | `Copied \| Missing` | `ReadableInto[O]` capability; `Copied.size` bytes written into `out`, a buffer the caller allocated, of the type the backend reads into (see below); `ValueError` if it is too small or can't be written | +| `head(key)` | `ObjectMetadata \| Missing` | size, generation, last-modified | +| `exists(key)` | `bool` | | +| `put(key, value)` | `Written` | `value` is any C-contiguous buffer, copied; `KeyConflictError` where the backend can't hold a key and a directory with one name | +| `put_if_absent(key, value)` | `Written \| Rejected` | `Rejected` carries the generation of the value already there | +| `put_if_match(key, value, generation)` | `Written \| Rejected \| Missing` | `WritableIfMatch` capability; compare-and-swap | +| `delete(key)` | `Deleted \| Missing` | one key; deleting a directory name finds it `Missing` | +| `delete_if_match(key, generation)` | `Deleted \| Rejected \| Missing` | `DeletableIfMatch` capability | +| `delete_dir(directory)` | `None` | everything under the directory | +| `get_many(requests)` | `Iterator[(request, Found \| Missing)]` | a request is a key or `(key, byte_range)`, echoed back; lazy, in the order reads finish | +| `get_into_many(requests)` | `Iterator[(request, Copied \| Missing)]` | `(key, out)` or `(key, out, byte_range)`; `out` typed as for `get_into` | +| `put_many(items)` | `list[(key, Written)]` | `(key, value)` pairs; eager, results in the order given; a key named twice raises `ValueError` | +| `delete_many(keys)` | `list[(key, Deleted \| Missing)]` | as `put_many` | +| `list(directory="")` | `Iterator[str]` | every key under the directory, relative to the store's root | +| `list_dir(directory="")` | `Iterator[Key \| Prefix \| Entry]` | direct children, names relative to the directory; `Entry` when the backend can't tell; a `Prefix` may hold no keys (an empty directory on a filesystem) | + +**Results are closed unions of small frozen types**, one per outcome, so a `match` handles each and a +type checker sees what's left. A method returns only the outcomes it can produce: `get` can't be +`Unchanged`, `put` can't be `Rejected`. That is why conditional operations are separate methods +rather than keyword arguments, and it lets a capability a backend lacks be a type error +(`delete_if_match` on an object store) instead of a runtime refusal. A refusal some stores can only +discover at runtime (`put_if_match` on obstore's `LocalStore`, or on a filesystem that rounds file +times) still raises `UnsupportedOperationError`. `Missing` and the read results carry `read_at`, when +the backend read, so a cache can age an absence like a value. + +**Reads into caller buffers, on any device.** `ReadableInto` is generic in the buffer type the +backend writes into, and `Store.get_into` accepts exactly that type. The built-in backends are +`ReadableInto[Buffer]`: any writable C-contiguous host buffer. A GPU direct storage backend (cuFile, +kvikio) would be `ReadableInto[DeviceBuffer]`, reading from disk straight into device memory, and +passing it a host buffer is a type error (and the reverse). The caller allocates, because the caller +knows the size and the device; the store never needs an allocator, so it stays free of Zarr's buffer +classes (zarr-python's `prototype` argument puts allocation in the store instead). `get` keeps +returning host bytes, which is what metadata documents need. Not yet: `get_ranges_into` (several +ranges into several buffers, from one generation, for sharded chunks) and the write-side twin +`put_from`. + +**Batches are for throughput, not atomicity.** `get_many`, `get_into_many`, `put_many` and +`delete_many` each do what a loop of the single operation would, and promise nothing more: each +read finds whatever generation is current when it runs, and a batch of writes can land partly +(`get_ranges`, which reads several parts of one value from one generation, is a consistency +guarantee, not a batch). Each result is paired with its request, so `dict(store.get_many(keys))` +works and order doesn't matter. Reads are lazy and come in the order they finish, so a caller can +decode chunks as they arrive. Writes are eager, because a lazy write nobody iterates would silently +write nothing; they return results in the order given. Every key is checked, and every value viewed, +before any operation starts, and a write batch that names a key twice raises, since which write +wins would be a race. If operations raise, the batch starts no new ones, lets those running finish, +and raises `BatchError`, an `ExceptionGroup` of the failures whose `completed` holds the writes that +landed with their results and whose `remaining` holds the requests that failed or never started +(`except*` keeps both). Conditional operations have no batch form: compare-and-swap coordinates +single documents, and a batch of it belongs to transactions. + +`Store` provides every batch method for any backend that can read or write, running the single +operation one at a time; a backend that has a batch capability (`ReadableMany`, +`ReadableIntoMany[O]`, `WritableMany`, and async twins) is used instead. The rule for such +fallbacks: `Store` may fall back when only speed differs, never when behavior does. A serial batch +returns what a concurrent one would; a `get_into` that read and then copied would not put the bytes +where the backend's buffer type says, which is why `get_into` has no fallback. The batch runners in +`zarr_storage._batch` implement the windowing and failure rule once, for `Store` and backends alike. + +Listing arguments are checked when the method is called; the items are produced lazily, and taking +one after the store closes raises `StoreClosedError`. + +**Byte ranges** (zarrs `ByteRange`, TensorStore `OptionalByteRangeRequest`): `RangeRequest(start, end)`, +`OffsetRequest(start)`, `SuffixRequest(length)`. An end past the value is clamped; a start past the value +raises `InvalidByteRangeError`; a suffix longer than the value returns the whole value. + +**Generations** are opaque and compared only by equality. Backends mint them: a counter (memory), +`(device, inode, mtime_ns, size)` (local, as TensorStore's file driver), an ETag (object stores). + +## Views, lifecycle, identity + +- `store / "a"`, `store.with_read_only(True)` and `store.as_async()` (or `as_sync()`) return views + on the same backend. A view of a + read-only store is read-only; `with_read_only(False)` on one raises `ReadOnlyError`. +- The store the user constructed owns the backend. `close()` (`aclose()` on an `AsyncStore`) on it + closes the backend; on a derived view it does nothing. After the backend closes, every view raises `StoreClosedError`. +- No value equality: `==` is identity. `shares_backend(other)` and `overlaps(other)` answer the questions + callers actually ask. +- Pickling carries the backend's configuration and the view's root. A pickled memory store carries a + copy of its data. + +## Errors + +`StorageError` is the base. `InvalidKeyError` and `InvalidByteRangeError` also subclass `ValueError`; +`UnsupportedOperationError` also subclasses `NotImplementedError`. `KeyConflictError`, `ReadOnlyError` +and `StoreClosedError` complete the set. A missing key is not an error, and neither is reading under a +key (`a/b` when `a` is a key): that reads as absent. + +## Backends + +| Backend | Generations | `put_if_match` / `delete_if_match` | Key and directory with one name | Pickles | +|---|---|---|---|---| +| `MemoryBackend` | counter | yes / yes | yes | a copy of the data | +| `LocalBackend` | `(dev, inode, mtime_ns, size)`, with each write stamping a unique ns mtime | yes / yes, on filesystems that keep ns times | no (`KeyConflictError`) | the root | +| `ObjectStoreBackend` | ETag and version (obstore's `LocalStore` ETag can repeat on ext4) | if the store supports it / not implemented | as the store does | as the store does | + +Every backend has `put_if_absent` and `get_if_changed`. `LocalBackend` runs batches, sync and +async, `concurrency` operations at once on its threads; `ObjectStoreBackend` keeps `concurrency` +requests of an async batch in flight and runs sync batches one request at a time; `MemoryBackend` +uses `Store`'s serial batches, which are as fast for it. + +`LocalBackend` stages each write in a reserved directory under the root and renames it into place, +holding the lock of the key's directory; the lock lives in that directory, so overlapping roots and +other processes share it. Locks are always taken ancestor before descendant. `delete_dir` removes a +tree bottom-up under each directory's lock, so a write in progress finishes before its directory +goes. Deletes, and writes whose precondition fails, can leave directories behind (pruning raced +every concurrent write). A later write of a key with such a directory's name reclaims it with +`rmdir`, only if no key lies under it, so a reclaim can never remove a key; otherwise the write +raises `KeyConflictError`, whatever its preconditions. A symlink in a key's place is that key's +file. Each write stamps its file with a nanosecond mtime unique within the process, because +filesystems such as ext4 reuse inodes at once and advance mtimes only per clock tick; on a +filesystem that rounds file times, `put_if_match` and `delete_if_match` raise +`UnsupportedOperationError`. +`LocalBackend` doesn't sync to disk, and needs a root that doesn't span filesystems. + +`ObjectStoreBackend.get_ranges` reads the first range unpinned, takes the generation and size from +that response, and reads the rest pinned to it (`If-Match`, and checked against each response); +on a mismatch it reads the value once, whole, and slices it. It does not use obstore's own +`get_ranges`, which can panic when a value shrinks mid-read. Only obstore's `LocalStore` reads a +refusal under a file as an absent key (judged from the refusal's own `ENOTDIR`, since looking at +the parents afterwards races a concurrent delete of the parent), and only there does a write reclaim an empty directory in +its way; for every other store, an error is raised. obstore's `LocalStore` has no locking, so +writes that concurrently create and remove the same directories can surface its errors. + +## Conformance suite + +`zarr_storage.testing.StoreSuite` holds the contract every backend meets; optional capabilities come +as mixins a backend's test class adds (`ConditionalPutSuite`, `KeyAndDirectorySuite`, +`PickleSuite`, ...) or their refusing counterparts (`NoConditionalPutSuite`, +`KeyDirectoryConflictSuite`, `ConditionalRaceSuite`, ...). Every test runs twice: through `Store`, +and through an `AsyncStore` over the same backend on one event loop, so the two classes meet one +contract. + +`StoreSuite.test_state_machine` drives random sequences of `put`, `put_if_absent`, `put_if_match`, +`put_many`, `delete`, `delete_if_match`, `delete_many`, `delete_dir` and listings through random views, and checks every step against a +dict model (`zarr_storage.testing._stateful.StoreMachine`), again through both surfaces. After every +step, each stored key reads back as the model says, through `get` and `get_many`, and +`get_if_changed` with its generation is `Unchanged`. The obstore +tests also fail any async run that makes a blocking obstore call. + +## Configuration + +No global configuration. Every knob (local I/O concurrency, how many requests an obstore batch +keeps in flight) is a constructor parameter with its default in the signature. + +## Open + +- Durability: no backend syncs to disk; a `sync` option on `LocalBackend` would. +- Staged files left by a crash are never read but are never removed either. +- Windows rules are implemented and unit-tested, not yet run on Windows. +- Sync batches over obstore run one request at a time; a thread pool, or a native batch call, + would overlap them. +- `get_ranges_into`, `put_from`, and a GPU direct storage backend whose `get_into_many` submits one + cuFile batch. + +## Iterations + +1. **Core:** types, `Store`, the memory and local backends, the conformance suite in + `zarr_storage.testing`, pickling. +2. **Remote and archive backends:** obstore (S3, GCS, Azure, HTTP; async-native), fsspec, zip. +3. **Composition:** range coalescing, caching, retry, a sync bridge for async-only backends, URL specs. +4. **Transactions:** staged writes committed with generations, memory first. diff --git a/packages/zarr-storage/LICENSE.txt b/packages/zarr-storage/LICENSE.txt new file mode 100644 index 0000000000..1e8da4d242 --- /dev/null +++ b/packages/zarr-storage/LICENSE.txt @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2015-2025 Zarr Developers + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/packages/zarr-storage/README.md b/packages/zarr-storage/README.md new file mode 100644 index 0000000000..0843002d86 --- /dev/null +++ b/packages/zarr-storage/README.md @@ -0,0 +1,3 @@ +# zarr-storage + +A Zarr-agnostic key-value storage layer. Pre-alpha; see `DESIGN.md`. diff --git a/packages/zarr-storage/changes/460.feature.md b/packages/zarr-storage/changes/460.feature.md new file mode 100644 index 0000000000..618fe3e1d8 --- /dev/null +++ b/packages/zarr-storage/changes/460.feature.md @@ -0,0 +1 @@ +Add `zarr-storage`, a new package with a Zarr-agnostic key-value storage API: `Store` and `AsyncStore` views over memory, local-filesystem and obstore backends, with the same operations as sync methods and as coroutines, conditional reads and writes (`get_if_changed`, `put_if_absent`, `put_if_match`, `delete_if_match`), byte-range reads, reads into caller buffers, and batches (`get_many`, `get_into_many`, `put_many`, `delete_many`) that run concurrently where the backend can. It ships a conformance suite in `zarr_storage.testing` that any backend can run. diff --git a/packages/zarr-storage/changes/README.md b/packages/zarr-storage/changes/README.md new file mode 100644 index 0000000000..6ea6eb1939 --- /dev/null +++ b/packages/zarr-storage/changes/README.md @@ -0,0 +1,25 @@ +Writing a changelog entry for `zarr-storage` +--------------------------------------------- + +Fragments in **this** directory are released notes for the `zarr-storage` +package only — kept separate from the parent zarr-python `changes/` +directory so a PR touching only `packages/zarr-storage/` produces a +release note for this package only. + +Please put a new file in this directory named `xxxx..md`, where + +- `xxxx` is the pull request number associated with this entry +- `` is one of: + - feature + - bugfix + - doc + - removal + - misc + +Inside the file, please write a short description of what you have +changed, and how it impacts users of `zarr-storage`. + +A `zarr-storage` release runs `towncrier build` in `packages/zarr-storage/`, +which consumes the fragments here and updates `CHANGELOG.md`. Fragments +that describe parent zarr-python changes (not the storage package) +belong in the top-level `changes/` directory, not here. diff --git a/packages/zarr-storage/justfile b/packages/zarr-storage/justfile new file mode 100644 index 0000000000..cafbf32765 --- /dev/null +++ b/packages/zarr-storage/justfile @@ -0,0 +1,59 @@ +# Development verbs for the zarr-storage package. Recipes run with this +# directory as the working directory regardless of where `just` is invoked. + +# List available recipes +# Quoted arguments must survive delegation from the root Justfile. +set positional-arguments + +default: + @just --list + +# Run the test suite on the project interpreter; extra args are passed to pytest +test *args: + uv run --group test pytest tests "$@" + +# The oldest and newest interpreters the package is declared for: the floor is +# `requires-python`, the ceiling the newest classifier, both read from +# pyproject.toml. CI's matrix is written out in its workflow, so a change to +# the declared range is made there too. +# uv fetches an interpreter it does not have. A difference in how a version +# materialises annotations or resolves a stub shows up here, not only in CI. +# Run the test suite on the oldest and newest supported interpreters +test-versions *args: + #!/usr/bin/env bash + set -euo pipefail + read -r floor ceiling < <(uv run python - <<'EOF' + import re + import tomllib + + project = tomllib.load(open("pyproject.toml", "rb"))["project"] + floor = re.fullmatch(r">=\s*(\d+\.\d+)", project["requires-python"]).group(1) + declared = [ + classifier.rsplit(" ", 1)[1] + for classifier in project["classifiers"] + if classifier.startswith("Programming Language :: Python :: 3.") + ] + print(floor, max(declared, key=lambda version: tuple(map(int, version.split("."))))) + EOF + ) + for version in "$floor" "$ceiling"; do + echo "== python $version ==" + uv run --python "$version" --group test pytest tests "$@" + done + +# Lint with the same invocation CI uses +lint: + uvx ruff check . + +# Run under the interpreter CI runs pyright under, so a version-dependent +# stdlib or stub difference shows up here rather than only in CI. +# Type-check the package, sources and tests alike +typecheck: + uv run --python 3.11 --group test --with pyright pyright + +# Run everything CI runs for this package, on both ends of the version range +check: lint typecheck test-versions + +# Preview the changelog that the next release would generate +changelog-draft: + uvx towncrier build --draft --version Unreleased diff --git a/packages/zarr-storage/pyproject.toml b/packages/zarr-storage/pyproject.toml new file mode 100644 index 0000000000..2ef3287654 --- /dev/null +++ b/packages/zarr-storage/pyproject.toml @@ -0,0 +1,156 @@ +[build-system] +requires = ["hatchling>=1.29.0", "hatch-vcs", "setuptools-scm>=9"] +build-backend = "hatchling.build" + +[project] +name = "zarr-storage" +dynamic = ["version"] +description = "A Zarr-agnostic key-value storage layer for Zarr implementations." +readme = "README.md" +requires-python = ">=3.11" +license = "MIT" +license-files = ["LICENSE.txt"] +authors = [ + { name = "Davis Bennett", email = "davis.v.bennett@gmail.com" }, +] +classifiers = [ + "Development Status :: 2 - Pre-Alpha", + "Intended Audience :: Developers", + "Intended Audience :: Information Technology", + "Intended Audience :: Science/Research", + "License :: OSI Approved :: MIT License", + "Operating System :: OS Independent", + "Programming Language :: Python", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", + "Topic :: Scientific/Engineering", + "Topic :: Software Development :: Libraries :: Python Modules", + "Typing :: Typed", +] +keywords = ["zarr", "storage", "key-value"] +dependencies = [ + # `typing_extensions.Buffer`, the buffer-protocol type, before Python 3.12's + # `collections.abc.Buffer`. + "typing_extensions>=4.6", +] + +[project.optional-dependencies] +# `zarr_storage.backends.ObjectStoreBackend`. +obstore = ["obstore>=0.12"] +# The `zarr_storage.testing` conformance suite. +testing = ["pytest>=8", "hypothesis>=6.85"] + +[project.urls] +Homepage = "https://github.com/zarr-developers/zarr-python" +Source = "https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-storage" +Issues = "https://github.com/zarr-developers/zarr-python/issues" +Changelog = "https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-storage/CHANGELOG.md" + +[dependency-groups] +test = [ + "pytest>=8", + "hypothesis>=6.85", + "obstore>=0.12", +] + +[tool.hatch.version] +source = "vcs" +tag-pattern = '^zarr_storage-v(?P.+)$' +raw-options = { root = "../..", scm = { git = { describe_command = "git describe --dirty --tags --long --match zarr_storage-v*" } }, local_scheme = "no-local-version", fallback_version = "0.0.0" } + +[tool.hatch.build.targets.wheel] +packages = ["src/zarr_storage"] + +# An allowlist, so nothing that merely happens to sit in the package directory +# can ride along in a release. +[tool.hatch.build.targets.sdist] +include = [ + "/src", + "/tests", + "/justfile", + "/CHANGELOG.md", +] + +[tool.ruff] +extend = "../../pyproject.toml" +target-version = "py311" + +[tool.ruff.lint] +# Use `object` for "any value"; a genuinely dynamic annotation needs a noqa +# saying why. +extend-select = ["ANN401"] + +[tool.pytest.ini_options] +minversion = "8" +testpaths = ["tests"] +xfail_strict = true +addopts = ["-ra", "--strict-config", "--strict-markers"] +filterwarnings = ["error"] + +[tool.numpydoc_validation] +checks = [ + "GL10", + "SS04", + "PR02", + "PR03", + "PR05", + "PR06", +] + +# Pyright is the only checker that runs on this package. +[tool.pyright] +include = ["src", "tests"] +enableTypeIgnoreComments = false +reportUnnecessaryTypeIgnoreComment = "error" +extraPaths = ["."] +typeCheckingMode = "strict" +pythonVersion = "3.11" + +[[tool.pyright.executionEnvironments]] +root = "tests" +reportMissingTypeArgument = false +reportPrivateUsage = false +reportUnknownArgumentType = false +reportUnknownLambdaType = false +reportUnknownMemberType = false +reportUnknownParameterType = false +reportUnknownVariableType = false +reportUnusedFunction = false + +[tool.towncrier] +directory = "changes" +filename = "CHANGELOG.md" +package = "zarr_storage" +underlines = ["", "", ""] +title_format = "## {version} ({project_date})" +issue_format = "[#{issue}](https://github.com/zarr-developers/zarr-python/pull/{issue})" +start_string = "\n" +template = "towncrier-template.md" + +[[tool.towncrier.type]] +directory = "feature" +name = "Features" +showcontent = true + +[[tool.towncrier.type]] +directory = "bugfix" +name = "Bugfixes" +showcontent = true + +[[tool.towncrier.type]] +directory = "doc" +name = "Improved Documentation" +showcontent = true + +[[tool.towncrier.type]] +directory = "removal" +name = "Deprecations and Removals" +showcontent = true + +[[tool.towncrier.type]] +directory = "misc" +name = "Misc" +showcontent = true diff --git a/packages/zarr-storage/src/zarr_storage/__init__.py b/packages/zarr-storage/src/zarr_storage/__init__.py new file mode 100644 index 0000000000..510c1e6ef0 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/__init__.py @@ -0,0 +1,113 @@ +"""A Zarr-agnostic key-value storage layer. + +A `Store` is a view of a backend at a root directory, used through sync methods; an `AsyncStore` +is the same view used through coroutines, with the same method names. Backends implement capability +protocols (`Readable`, `Writable`, `Listable`, their batch variants, and async twins) over backend +paths; the stores validate keys, apply their root, enforce read-only access, and are the only +objects most callers need. +""" + +from zarr_storage._errors import ( + BatchError, + InvalidByteRangeError, + InvalidKeyError, + KeyConflictError, + ReadOnlyError, + StorageError, + StoreClosedError, + UnsupportedOperationError, +) +from zarr_storage._protocols import ( + Backend, + ClosableAsync, + DeletableIfMatch, + DeletableIfMatchAsync, + Listable, + ListableAsync, + Readable, + ReadableAsync, + ReadableInto, + ReadableIntoAsync, + ReadableIntoMany, + ReadableIntoManyAsync, + ReadableMany, + ReadableManyAsync, + Writable, + WritableAsync, + WritableIfMatch, + WritableIfMatchAsync, + WritableMany, + WritableManyAsync, +) +from zarr_storage._store import AsyncStore, Store +from zarr_storage._types import ( + ByteRange, + Copied, + Deleted, + Entry, + Found, + FoundRanges, + Generation, + Key, + ListItem, + Missing, + ObjectMetadata, + OffsetRequest, + Prefix, + RangeRequest, + Rejected, + SuffixRequest, + Unchanged, + Written, +) + +__all__ = [ + "AsyncStore", + "Backend", + "BatchError", + "ByteRange", + "ClosableAsync", + "Copied", + "DeletableIfMatch", + "DeletableIfMatchAsync", + "Deleted", + "Entry", + "Found", + "FoundRanges", + "Generation", + "InvalidByteRangeError", + "InvalidKeyError", + "Key", + "KeyConflictError", + "ListItem", + "Listable", + "ListableAsync", + "Missing", + "ObjectMetadata", + "OffsetRequest", + "Prefix", + "RangeRequest", + "ReadOnlyError", + "Readable", + "ReadableAsync", + "ReadableInto", + "ReadableIntoAsync", + "ReadableIntoMany", + "ReadableIntoManyAsync", + "ReadableMany", + "ReadableManyAsync", + "Rejected", + "StorageError", + "Store", + "StoreClosedError", + "SuffixRequest", + "Unchanged", + "UnsupportedOperationError", + "Writable", + "WritableAsync", + "WritableIfMatch", + "WritableIfMatchAsync", + "WritableMany", + "WritableManyAsync", + "Written", +] diff --git a/packages/zarr-storage/src/zarr_storage/_batch.py b/packages/zarr-storage/src/zarr_storage/_batch.py new file mode 100644 index 0000000000..e850519e57 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_batch.py @@ -0,0 +1,111 @@ +"""Running one operation over many requests, for `Store`'s fallbacks and backends' batch methods. + +Both runners yield `(index, result)` pairs as operations finish, so callers map results back to +their requests. A runner keeps at most `window` operations in flight. After the first failure it +starts no new operations, lets the running ones finish (yielding those that succeed), then raises +an `ExceptionGroup` of every failure. `Store` turns that group into a `BatchError`. +""" + +from __future__ import annotations + +import asyncio +from concurrent.futures import FIRST_COMPLETED, wait +from typing import TYPE_CHECKING, Any, TypeVar, TypeVarTuple, Unpack + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Callable, Coroutine, Iterator, Sequence + from concurrent.futures import Executor, Future + +__all__ = ["run_many", "run_many_async"] + +R = TypeVar("R") +Ts = TypeVarTuple("Ts") + +_MESSAGE = "Operations of a batch failed." + + +def run_many( + fn: Callable[[Unpack[Ts]], R], + requests: Sequence[tuple[*Ts]], + executor: Executor | None = None, + window: int = 1, +) -> Iterator[tuple[int, R]]: + """Call `fn(*request)` for each request: one at a time, or on `executor`, `window` at once.""" + if executor is None: + return _run_serially(fn, requests) + return _run_on(executor, fn, requests, window) + + +def _run_serially( + fn: Callable[[Unpack[Ts]], R], requests: Sequence[tuple[*Ts]] +) -> Iterator[tuple[int, R]]: + for index, request in enumerate(requests): + try: + result = fn(*request) + except Exception as error: # noqa: BLE001 -- every failure is reported, in the group + raise ExceptionGroup(_MESSAGE, [error]) from None + yield index, result + + +def _run_on( + executor: Executor, + fn: Callable[[Unpack[Ts]], R], + requests: Sequence[tuple[*Ts]], + window: int, +) -> Iterator[tuple[int, R]]: + pending: dict[Future[R], int] = {} + errors: list[Exception] = [] + started = 0 + try: + while pending or (not errors and started < len(requests)): + while not errors and started < len(requests) and len(pending) < window: + pending[executor.submit(fn, *requests[started])] = started + started += 1 + done, _ = wait(pending, return_when=FIRST_COMPLETED) + for future in done: + index = pending.pop(future) + error = future.exception() + if error is None: + yield index, future.result() + elif isinstance(error, Exception): + errors.append(error) + else: + raise error + finally: + for future in pending: + future.cancel() + if errors: + raise ExceptionGroup(_MESSAGE, errors) + + +async def run_many_async( + fn: Callable[[Unpack[Ts]], Coroutine[Any, Any, R]], + requests: Sequence[tuple[*Ts]], + window: int = 1, +) -> AsyncIterator[tuple[int, R]]: + """Await `fn(*request)` for each request, `window` at once.""" + pending: dict[asyncio.Task[R], int] = {} + errors: list[Exception] = [] + started = 0 + try: + while pending or (not errors and started < len(requests)): + while not errors and started < len(requests) and len(pending) < window: + pending[asyncio.ensure_future(fn(*requests[started]))] = started + started += 1 + done, _ = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED) + for task in done: + index = pending.pop(task) + error = task.exception() + if error is None: + yield index, task.result() + elif isinstance(error, Exception): + errors.append(error) + else: + raise error + finally: + for task in pending: + task.cancel() + if pending: + await asyncio.gather(*pending, return_exceptions=True) + if errors: + raise ExceptionGroup(_MESSAGE, errors) diff --git a/packages/zarr-storage/src/zarr_storage/_buffers.py b/packages/zarr-storage/src/zarr_storage/_buffers.py new file mode 100644 index 0000000000..3c6840ba81 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_buffers.py @@ -0,0 +1,41 @@ +"""Host-memory buffers: the byte views stores take values as, and copying a read into one.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr_storage._types import Copied, Found, Missing + +if TYPE_CHECKING: + from typing_extensions import Buffer + +__all__ = ["byte_view", "copy_into"] + + +def byte_view(value: Buffer, *, writable: bool) -> memoryview: + """A flat, C-contiguous byte view of a buffer. + + Raises `ValueError` if the buffer is not C-contiguous, or if `writable` and it is read-only. + """ + view = memoryview(value) + if not view.c_contiguous: + raise ValueError("The buffer must be C-contiguous.") + if writable and view.readonly: + raise ValueError("The output buffer must be writable.") + return view.cast("B") if view.format != "B" or view.ndim != 1 else view + + +def copy_into(result: Found | Missing, out: memoryview) -> Copied | Missing: + """The `get_into` outcome of a `get`, copying a found value into `out`. + + For a backend that can't read into a buffer directly. + """ + match result: + case Found(value=value, generation=generation, read_at=read_at): + n = len(value) + if n > len(out): + raise ValueError(f"The output buffer holds {len(out)} bytes; the value has {n}.") + out[:n] = value + return Copied(n, generation, read_at) + case Missing(): + return result diff --git a/packages/zarr-storage/src/zarr_storage/_byte_range.py b/packages/zarr-storage/src/zarr_storage/_byte_range.py new file mode 100644 index 0000000000..5bc70c1705 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_byte_range.py @@ -0,0 +1,38 @@ +"""Resolving a byte range against the size of a value.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr_storage._errors import InvalidByteRangeError +from zarr_storage._types import OffsetRequest, RangeRequest, SuffixRequest + +if TYPE_CHECKING: + from zarr_storage._types import ByteRange + +__all__ = ["resolve"] + + +def resolve(byte_range: ByteRange | None, size: int) -> tuple[int, int]: + """The `(start, end)` a byte range covers in a value of `size` bytes. + + An end past the value is clamped to `size`. A start past the value raises + `InvalidByteRangeError`; a start equal to `size` selects nothing. + """ + match byte_range: + case None: + return 0, size + case RangeRequest(start=start, end=end): + if start > size: + raise InvalidByteRangeError( + f"The range starts at byte {start}, past the end of a {size}-byte value." + ) + return start, min(end, size) + case OffsetRequest(start=start): + if start > size: + raise InvalidByteRangeError( + f"The range starts at byte {start}, past the end of a {size}-byte value." + ) + return start, size + case SuffixRequest(length=length): + return max(size - length, 0), size diff --git a/packages/zarr-storage/src/zarr_storage/_errors.py b/packages/zarr-storage/src/zarr_storage/_errors.py new file mode 100644 index 0000000000..fb47ea55d1 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_errors.py @@ -0,0 +1,90 @@ +"""The errors a store raises. A missing key is not one of them: reads return `Missing`.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Self + +if TYPE_CHECKING: + from collections.abc import Sequence + +__all__ = [ + "BatchError", + "InvalidByteRangeError", + "InvalidKeyError", + "KeyConflictError", + "ReadOnlyError", + "StorageError", + "StoreClosedError", + "UnsupportedOperationError", +] + + +class StorageError(Exception): + """Base class for every error this package raises.""" + + +class InvalidKeyError(StorageError, ValueError): + """A key or directory does not follow the key grammar, or names something the backend reserves.""" + + +class InvalidByteRangeError(StorageError, ValueError): + """A byte range starts past the end of the value it reads.""" + + +class KeyConflictError(StorageError): + """A key collides with a directory of the same name on a backend that can't hold both. + + Object stores and in-memory stores hold `a` and `a/b` side by side; a filesystem can't. + """ + + +class ReadOnlyError(StorageError): + """A write was attempted through a read-only store.""" + + +class StoreClosedError(StorageError): + """An operation was attempted after the store's backend was closed.""" + + +class UnsupportedOperationError(StorageError, NotImplementedError): + """The store's backend does not implement the requested operation.""" + + +class BatchError(ExceptionGroup[Exception], StorageError): + """Some operations of a batch (`get_many`, `put_many`, ...) raised. + + After the first failure a batch starts no new operations; the ones already running finish. The + group holds every exception those operations raised. `completed` pairs each request that + finished with its result, and `remaining` holds the requests that failed or never started: a + write among them may or may not have landed. + """ + + completed: tuple[tuple[object, object], ...] + remaining: tuple[object, ...] + + def __new__( + cls, + message: str, + exceptions: Sequence[Exception], + *, + completed: tuple[tuple[object, object], ...], + remaining: tuple[object, ...], + ) -> Self: + self = super().__new__(cls, message, exceptions) + self.completed = completed + self.remaining = remaining + return self + + def __init__( + self, + message: str, + exceptions: Sequence[Exception], + *, + completed: tuple[tuple[object, object], ...], + remaining: tuple[object, ...], + ) -> None: + super().__init__(message, exceptions) + + def derive(self, excs: Sequence[Exception]) -> BatchError: # pyright: ignore[reportIncompatibleMethodOverride] -- a batch only ever holds `Exception`s + """The same batch error, holding `excs`; `except*` builds its subgroups with this.""" + return BatchError(self.message, excs, completed=self.completed, remaining=self.remaining) diff --git a/packages/zarr-storage/src/zarr_storage/_keys.py b/packages/zarr-storage/src/zarr_storage/_keys.py new file mode 100644 index 0000000000..bbda939fda --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_keys.py @@ -0,0 +1,71 @@ +"""The key grammar, and the joins between a store's root and its keys. + +Every path operation in this package goes through these functions, so the rules live in one place. +""" + +from __future__ import annotations + +from zarr_storage._errors import InvalidKeyError + +__all__ = [ + "check_key", + "is_within", + "join", + "normalize_directory", + "relativize", +] + + +def check_key(key: str) -> str: + """Return `key` unchanged if it follows the key grammar; raise `InvalidKeyError` otherwise. + + A key is not empty, has no leading or trailing `/`, no empty segment, no `.` or `..` segment, + and no NUL character. Backends may refuse more (a filesystem refuses names it can't represent). + """ + if not isinstance(key, str): # pyright: ignore[reportUnnecessaryIsInstance] + raise InvalidKeyError(f"A key must be a str; got {type(key).__name__}.") + if key == "": + raise InvalidKeyError("A key must not be empty.") + if "\x00" in key: + raise InvalidKeyError(f"A key must not contain a NUL character; got {key!r}.") + if key.startswith("/") or key.endswith("/"): + raise InvalidKeyError(f"A key must not start or end with '/'; got {key!r}.") + for segment in key.split("/"): + if segment in ("", ".", ".."): + raise InvalidKeyError( + f"A key must not contain an empty, '.' or '..' segment; got {key!r}." + ) + return key + + +def normalize_directory(directory: str) -> str: + """Return the canonical form of a directory: `""` for the root, otherwise a key. + + A trailing `/` is accepted and removed, so `"a"` and `"a/"` name the same directory. + """ + if not isinstance(directory, str): # pyright: ignore[reportUnnecessaryIsInstance] + raise InvalidKeyError(f"A directory must be a str; got {type(directory).__name__}.") + if directory in ("", "/"): + return "" + return check_key(directory.removesuffix("/")) + + +def join(directory: str, key: str) -> str: + """Join a normalized directory and a key (or normalized directory).""" + if directory == "": + return key + if key == "": + return directory + return f"{directory}/{key}" + + +def relativize(path: str, directory: str) -> str: + """The part of `path` under the normalized `directory`. `path` must be inside it.""" + if directory == "": + return path + return path.removeprefix(f"{directory}/") + + +def is_within(path: str, directory: str) -> bool: + """Whether `path` is `directory` itself or lies under it (both normalized).""" + return directory == "" or path == directory or path.startswith(f"{directory}/") diff --git a/packages/zarr-storage/src/zarr_storage/_protocols.py b/packages/zarr-storage/src/zarr_storage/_protocols.py new file mode 100644 index 0000000000..7058b87ca7 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_protocols.py @@ -0,0 +1,280 @@ +"""Capability protocols that backends implement. + +A backend works on *backend paths*: keys and directories already joined with a store's root and +already checked against the key grammar. A backend never joins, strips or validates paths beyond +what it reserves for itself. `Store` does all of that, once, for every backend. + +Directories are passed normalized: `""` for the backend's root, otherwise a key without a trailing +`/`. + +Each capability has a sync protocol and an async twin whose methods end in `_async`. A backend +implements whichever it can. `Store` calls the sync protocols and `AsyncStore` the async ones, and +their methods are typed so that calling one the backend lacks is a type error. Compare-and-swap on a generation (`WritableIfMatch`, `DeletableIfMatch`) is a capability +of its own, since many stores can't do it; `put_if_absent` is part of `Writable`. + +The batch capabilities (`ReadableMany`, `ReadableIntoMany`, `WritableMany` and their async twins) +are optional and change only speed: `Store.get_many` and its siblings work on any backend that +can read or write, one operation at a time, and use a backend's batch methods when it has them. A +batch method yields `(index, result)` pairs as operations finish and follows the failure rule of +`zarr_storage._batch`, whose runners implement it. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Protocol, TypeVar, runtime_checkable + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Iterator, Sequence + + from zarr_storage._types import ( + ByteRange, + Copied, + Deleted, + Found, + FoundRanges, + Generation, + ListItem, + Missing, + ObjectMetadata, + Rejected, + Unchanged, + Written, + ) + +__all__ = [ + "Backend", + "ClosableAsync", + "DeletableIfMatch", + "DeletableIfMatchAsync", + "Listable", + "ListableAsync", + "Readable", + "ReadableAsync", + "ReadableInto", + "ReadableIntoAsync", + "ReadableIntoMany", + "ReadableIntoManyAsync", + "ReadableMany", + "ReadableManyAsync", + "Writable", + "WritableAsync", + "WritableIfMatch", + "WritableIfMatchAsync", + "WritableMany", + "WritableManyAsync", +] + + +O_contra = TypeVar("O_contra", contravariant=True) +"""The type of buffer a backend's `get_into` writes into.""" + + +class Backend(Protocol): + """What every backend provides: a lifecycle and whether it accepts writes at all.""" + + @property + def closed(self) -> bool: + """Whether `close` has been called.""" + ... + + @property + def writable(self) -> bool: + """Whether the backend can write. A store over a backend that can't is read-only.""" + ... + + def close(self) -> None: + """Release the backend's resources. Idempotent.""" + ... + + +@runtime_checkable +class ClosableAsync(Backend, Protocol): + """A backend whose cleanup awaits something, such as closing a client session. + + `AsyncStore.aclose` awaits `aclose` on such a backend, and calls `close` on any other. + """ + + async def aclose(self) -> None: + """`close`, as a coroutine. Idempotent.""" + ... + + +class Readable(Backend, Protocol): + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + """The value at `path`, or the requested part of it.""" + ... + + def get_if_changed( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: + """`get`, or `Unchanged` without reading any bytes if the value has `generation`.""" + ... + + def get_ranges(self, path: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: + """Several parts of the value at `path`, all from one generation.""" + ... + + def head(self, path: str) -> ObjectMetadata | Missing: + """The size and generation of the value at `path`.""" + ... + + +class ReadableInto(Backend, Protocol[O_contra]): + """Reads into a buffer the caller allocated, of the type the backend names. + + A backend over host memory takes any `Buffer` (`ReadableInto[Buffer]`). A backend that reads + into another device's memory, such as GPU direct storage, takes that device's buffer type + instead, and `Store.get_into` then accepts exactly that type. Where the bytes land is the + backend's business; `Copied` reports how many and from which generation. + """ + + def get_into(self, path: str, out: O_contra, byte_range: ByteRange | None) -> Copied | Missing: + """Copy the value (or part) into `out`. + + Raises `ValueError` if `out` is shorter than the bytes to copy, or can't be written. + """ + ... + + +class Writable(Backend, Protocol): + def put(self, path: str, value: memoryview) -> Written: + """Write `value` at `path`.""" + ... + + def put_if_absent(self, path: str, value: memoryview) -> Written | Rejected: + """Write `value` at `path` if the key holds no value.""" + ... + + def delete(self, path: str) -> Deleted | Missing: + """Delete the key at `path`. A directory is not a key.""" + ... + + def delete_dir(self, directory: str) -> None: + """Delete every key under `directory`.""" + ... + + +class WritableIfMatch(Backend, Protocol): + def put_if_match( + self, path: str, value: memoryview, generation: Generation + ) -> Written | Rejected | Missing: + """Write `value` at `path` if the key's value has `generation`.""" + ... + + +class DeletableIfMatch(Backend, Protocol): + def delete_if_match(self, path: str, generation: Generation) -> Deleted | Rejected | Missing: + """Delete the key at `path` if its value has `generation`.""" + ... + + +@runtime_checkable +class ReadableMany(Backend, Protocol): + def get_many( + self, requests: Sequence[tuple[str, ByteRange | None]] + ) -> Iterator[tuple[int, Found | Missing]]: + """`get` for each `(path, byte_range)`, yielding `(index, result)` as each finishes.""" + ... + + +@runtime_checkable +class ReadableIntoMany(Backend, Protocol[O_contra]): + def get_into_many( + self, requests: Sequence[tuple[str, O_contra, ByteRange | None]] + ) -> Iterator[tuple[int, Copied | Missing]]: + """`get_into` for each `(path, out, byte_range)`, yielding `(index, result)`.""" + ... + + +@runtime_checkable +class WritableMany(Backend, Protocol): + def put_many(self, requests: Sequence[tuple[str, memoryview]]) -> Iterator[tuple[int, Written]]: + """`put` for each `(path, value)`, yielding `(index, result)` as each finishes.""" + ... + + def delete_many(self, paths: Sequence[str]) -> Iterator[tuple[int, Deleted | Missing]]: + """`delete` for each path, yielding `(index, result)` as each finishes.""" + ... + + +class Listable(Backend, Protocol): + def list(self, directory: str) -> Iterator[str]: + """Every key under `directory`, as backend paths, in no particular order.""" + ... + + def list_dir(self, directory: str) -> Iterator[ListItem]: + """The direct children of `directory`, named relative to it.""" + ... + + +class ReadableAsync(Backend, Protocol): + async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: ... + + async def get_if_changed_async( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: ... + + async def get_ranges_async( + self, path: str, ranges: Sequence[ByteRange] + ) -> FoundRanges | Missing: ... + + async def head_async(self, path: str) -> ObjectMetadata | Missing: ... + + +class ReadableIntoAsync(Backend, Protocol[O_contra]): + async def get_into_async( + self, path: str, out: O_contra, byte_range: ByteRange | None + ) -> Copied | Missing: ... + + +class WritableAsync(Backend, Protocol): + async def put_async(self, path: str, value: memoryview) -> Written: ... + + async def put_if_absent_async(self, path: str, value: memoryview) -> Written | Rejected: ... + + async def delete_async(self, path: str) -> Deleted | Missing: ... + + async def delete_dir_async(self, directory: str) -> None: ... + + +class WritableIfMatchAsync(Backend, Protocol): + async def put_if_match_async( + self, path: str, value: memoryview, generation: Generation + ) -> Written | Rejected | Missing: ... + + +class DeletableIfMatchAsync(Backend, Protocol): + async def delete_if_match_async( + self, path: str, generation: Generation + ) -> Deleted | Rejected | Missing: ... + + +@runtime_checkable +class ReadableManyAsync(Backend, Protocol): + def get_many_async( + self, requests: Sequence[tuple[str, ByteRange | None]] + ) -> AsyncIterator[tuple[int, Found | Missing]]: ... + + +@runtime_checkable +class ReadableIntoManyAsync(Backend, Protocol[O_contra]): + def get_into_many_async( + self, requests: Sequence[tuple[str, O_contra, ByteRange | None]] + ) -> AsyncIterator[tuple[int, Copied | Missing]]: ... + + +@runtime_checkable +class WritableManyAsync(Backend, Protocol): + def put_many_async( + self, requests: Sequence[tuple[str, memoryview]] + ) -> AsyncIterator[tuple[int, Written]]: ... + + def delete_many_async( + self, paths: Sequence[str] + ) -> AsyncIterator[tuple[int, Deleted | Missing]]: ... + + +class ListableAsync(Backend, Protocol): + def list_async(self, directory: str) -> AsyncIterator[str]: ... + + def list_dir_async(self, directory: str) -> AsyncIterator[ListItem]: ... diff --git a/packages/zarr-storage/src/zarr_storage/_store.py b/packages/zarr-storage/src/zarr_storage/_store.py new file mode 100644 index 0000000000..de7dedc71b --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_store.py @@ -0,0 +1,742 @@ +"""`Store` and `AsyncStore`: views of a backend at a root, through sync methods or coroutines. + +They are where the key grammar, the root, read-only access and the closed check are enforced, +once, for every backend. A backend only sees backend paths. +""" + +from __future__ import annotations + +from collections.abc import AsyncGenerator, Generator +from typing import TYPE_CHECKING, Final, Generic, Self, TypeVar, cast, final + +from zarr_storage._batch import run_many, run_many_async +from zarr_storage._buffers import byte_view +from zarr_storage._errors import BatchError, ReadOnlyError, StoreClosedError +from zarr_storage._keys import check_key, is_within, join, normalize_directory, relativize +from zarr_storage._protocols import ( + Backend, + ClosableAsync, + ReadableIntoMany, + ReadableIntoManyAsync, + ReadableMany, + ReadableManyAsync, + WritableMany, + WritableManyAsync, +) +from zarr_storage._types import Missing + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Iterable, Iterator, Sequence + from types import TracebackType + + from typing_extensions import Buffer + + from zarr_storage._protocols import ( + DeletableIfMatch, + DeletableIfMatchAsync, + Listable, + ListableAsync, + Readable, + ReadableAsync, + ReadableInto, + ReadableIntoAsync, + Writable, + WritableAsync, + WritableIfMatch, + WritableIfMatchAsync, + ) + from zarr_storage._types import ( + ByteRange, + Copied, + Deleted, + Found, + FoundRanges, + Generation, + ListItem, + ObjectMetadata, + Rejected, + Unchanged, + Written, + ) + +__all__ = ["AsyncStore", "Store"] + +B_co = TypeVar("B_co", bound=Backend, covariant=True) +O = TypeVar("O") +T = TypeVar("T") +R = TypeVar("R") +Q = TypeVar("Q", bound="str | tuple[str, ByteRange | None]") +"""A `get_many` request: a key, or a key and the part of its value to read.""" + + +class _View(Generic[B_co]): + """What `Store` and `AsyncStore` share: a backend, a root, read-only access, ownership.""" + + __slots__ = ("_backend", "_owner", "_read_only", "_root") + + def __init__(self, backend: B_co, *, root: str = "", read_only: bool = False) -> None: + self._backend: Final[B_co] = backend + self._root: Final[str] = normalize_directory(root) + self._read_only: Final[bool] = read_only or not backend.writable + self._owner = True + + # ----------------------------------------------------------------- views and identity + + @property + def backend(self) -> B_co: + """The backend this store reads and writes through.""" + return self._backend + + @property + def root(self) -> str: + """This store's directory within the backend; `""` is the backend's root.""" + return self._root + + @property + def read_only(self) -> bool: + """Whether writes through this store are refused.""" + return self._read_only + + @property + def closed(self) -> bool: + """Whether the backend has been closed.""" + return self._backend.closed + + def _view(self, *, root: str, read_only: bool) -> Self: + view = type(self)(self._backend, root=root, read_only=read_only) + view._owner = False + return view + + def __truediv__(self, directory: str) -> Self: + """A view on the same backend, rooted at `directory` under this store's root.""" + return self._view( + root=join(self._root, normalize_directory(directory)), read_only=self._read_only + ) + + def with_read_only(self, read_only: bool = True) -> Self: + """A view on the same backend and root that refuses writes, or accepts them. + + A read-only store can't produce a writable view: `with_read_only(False)` on one raises + `ReadOnlyError`. + """ + if self._read_only and not read_only: + raise ReadOnlyError(f"{self!r} is read-only; its views are read-only too.") + return self._view(root=self._root, read_only=read_only) + + def shares_backend(self, other: _View[Backend]) -> bool: + """Whether `other` reads and writes through the same backend object.""" + return self._backend is other._backend + + def overlaps(self, other: _View[Backend]) -> bool: + """Whether a key could be visible through both stores: same backend, nested roots.""" + return self.shares_backend(other) and ( + is_within(self._root, other._root) or is_within(other._root, self._root) + ) + + def __repr__(self) -> str: + flags = ", read_only=True" if self._read_only else "" + return f"{type(self).__name__}({self._backend!r}, root={self._root!r}{flags})" + + def __reduce__(self) -> tuple[object, ...]: + return (_restore_store, (type(self), self._backend, self._root, self._read_only)) + + # ----------------------------------------------------------------- path handling + + def _check_open(self) -> None: + if self._backend.closed: + raise StoreClosedError(f"{self!r} is closed.") + + def _path(self, key: str) -> str: + self._check_open() + return join(self._root, check_key(key)) + + def _directory(self, directory: str) -> str: + self._check_open() + return join(self._root, normalize_directory(directory)) + + def _open_items(self, items: Iterator[T]) -> Iterator[T]: + """`items`, refusing to continue once the backend closes.""" + for item in items: + self._check_open() + yield item + + async def _open_items_async(self, items: AsyncIterator[T]) -> AsyncIterator[T]: + async for item in items: + self._check_open() + yield item + + def _writable_path(self, key: str) -> str: + path = self._path(key) + if self._read_only: + raise ReadOnlyError(f"{self!r} is read-only.") + return path + + def _writable_directory(self, directory: str) -> str: + path = self._directory(directory) + if self._read_only: + raise ReadOnlyError(f"{self!r} is read-only.") + return path + + def _read_request( + self, request: str | tuple[str, ByteRange | None] + ) -> tuple[str, ByteRange | None]: + if isinstance(request, str): + return self._path(request), None + key, byte_range = request + return self._path(key), byte_range + + def _into_request( + self, request: tuple[str, O] | tuple[str, O, ByteRange | None] + ) -> tuple[str, O, ByteRange | None]: + if len(request) == 2: + key, out = request + return self._path(key), out, None + key, out, byte_range = request + return self._path(key), out, byte_range + + def _writable_paths(self, keys: Sequence[str]) -> list[str]: + """The backend path of each key, refusing a key named twice: its outcome would be a race.""" + self._check_open() + if self._read_only: + raise ReadOnlyError(f"{self!r} is read-only.") + paths = [self._writable_path(key) for key in keys] + seen: set[str] = set() + for key, path in zip(keys, paths, strict=True): + if path in seen: + raise ValueError(f"The batch names the key {key!r} more than once.") + seen.add(path) + return paths + + +@final +class Store(_View[B_co]): + """A backend seen from a root directory, through sync methods. + + Construct a store from a backend; the store you construct owns the backend and closes it. + `store / "a"`, `with_read_only()` and `as_async()` return views on the same backend, which + share its state and never close it. A view of a read-only store is read-only too. + + Which methods a store offers depends on its backend's capabilities: a method whose capability + the backend lacks is a type error. `AsyncStore` offers the same operations as coroutines. + + `backend` gives direct access to the backend, which bypasses the key grammar, the root, and + read-only and closed checks. + + Stores have no value equality: `==` is identity. Use `shares_backend` and `overlaps` to compare + stores. + + Parameters + ---------- + backend + The backend to read and write through. + root + The directory within the backend that this store's keys are relative to. + read_only + Refuse writes through this store. A backend that can't write makes every store over it + read-only. + """ + + __slots__ = () + + def __init__(self, backend: B_co, *, root: str = "", read_only: bool = False) -> None: + super().__init__(backend, root=root, read_only=read_only) + + def as_async(self) -> AsyncStore[B_co]: + """An `AsyncStore` view on the same backend and root; it doesn't own the backend.""" + view = AsyncStore(self._backend, root=self._root, read_only=self._read_only) + view._owner = False + return view + + # ----------------------------------------------------------------- lifecycle + + def close(self) -> None: + """Close the backend, if this store owns it (it was constructed, not derived).""" + if self._owner: + self._backend.close() + + def __enter__(self) -> Self: + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self.close() + + # ----------------------------------------------------------------- batch results + + def _streamed( + self, batch: Sequence[T], results: Iterator[tuple[int, R]] + ) -> Iterator[tuple[T, R]]: + """`results` paired with their requests; a failure raises `BatchError`.""" + done: set[int] = set() + try: + while True: + # Checked before each result is taken, which may start more reads. + self._check_open() + pair = next(results, None) + if pair is None: + return + index, result = pair + done.add(index) + yield batch[index], result + except ExceptionGroup as group: + raise _batch_error(group, batch, {}, done) from None + finally: + # A caller that stops early must not leave the batch's reads to the garbage collector. + if isinstance(results, Generator): + results.close() + + @staticmethod + def _finished(batch: Sequence[T], results: Iterator[tuple[int, R]]) -> list[tuple[T, R]]: + """Every result, in request order; a failure raises `BatchError` with those that landed.""" + completed: dict[int, R] = {} + try: + completed.update(results) # one pair at a time, so a failure keeps those before it + except ExceptionGroup as group: + raise _batch_error(group, batch, completed, completed.keys()) from None + return [(request, completed[index]) for index, request in enumerate(batch)] + + # ----------------------------------------------------------------- reads + + def get( + self: Store[Readable], key: str, *, byte_range: ByteRange | None = None + ) -> Found | Missing: + """The value at `key`, or the part `byte_range` selects.""" + return self._backend.get(self._path(key), byte_range) + + def get_if_changed( + self: Store[Readable], + key: str, + generation: Generation, + *, + byte_range: ByteRange | None = None, + ) -> Found | Missing | Unchanged: + """`get`, unless the value still has `generation`: then `Unchanged`, and no bytes are read. + + This revalidates a cached copy, as an HTTP `If-None-Match` request does. + """ + return self._backend.get_if_changed(self._path(key), generation, byte_range) + + def get_ranges( + self: Store[Readable], key: str, ranges: Sequence[ByteRange] + ) -> FoundRanges | Missing: + """Several parts of the value at `key`, all from one generation.""" + return self._backend.get_ranges(self._path(key), ranges) + + def get_into( + self: Store[ReadableInto[O]], key: str, out: O, *, byte_range: ByteRange | None = None + ) -> Copied | Missing: + """Copy the value at `key` (or part of it) into `out`, a buffer the caller allocated. + + `out` is whatever buffer type the backend reads into: any writable C-contiguous `Buffer` + for the built-in backends, device memory for a GPU direct storage backend. Raises + `ValueError` if `out` is shorter than the bytes to copy or can't be written. + """ + return self._backend.get_into(self._path(key), out, byte_range) + + def head(self: Store[Readable], key: str) -> ObjectMetadata | Missing: + """The size and generation of the value at `key`.""" + return self._backend.head(self._path(key)) + + def exists(self: Store[Readable], key: str) -> bool: + """Whether `key` holds a value.""" + return not isinstance(self.head(key), Missing) + + # ----------------------------------------------------------------- writes + + def put(self: Store[Writable], key: str, value: Buffer) -> Written: + """Write `value` at `key`.""" + return self._backend.put(self._writable_path(key), byte_view(value, writable=False)) + + def put_if_absent(self: Store[Writable], key: str, value: Buffer) -> Written | Rejected: + """Write `value` at `key` if the key holds no value; `Rejected` if it holds one.""" + return self._backend.put_if_absent( + self._writable_path(key), byte_view(value, writable=False) + ) + + def put_if_match( + self: Store[WritableIfMatch], key: str, value: Buffer, generation: Generation + ) -> Written | Rejected | Missing: + """Write `value` at `key` if the key's value has `generation`. + + `Rejected` if the key holds another generation, `Missing` if it holds no value. + """ + return self._backend.put_if_match( + self._writable_path(key), byte_view(value, writable=False), generation + ) + + def delete(self: Store[Writable], key: str) -> Deleted | Missing: + """Delete the key `key`. A directory is not a key: deleting one finds it `Missing`.""" + return self._backend.delete(self._writable_path(key)) + + def delete_if_match( + self: Store[DeletableIfMatch], key: str, generation: Generation + ) -> Deleted | Rejected | Missing: + """Delete the key `key` if its value has `generation`.""" + return self._backend.delete_if_match(self._writable_path(key), generation) + + def delete_dir(self: Store[Writable], directory: str = "") -> None: + """Delete every key under `directory` (by default, every key in this store).""" + self._backend.delete_dir(self._writable_directory(directory)) + + # ----------------------------------------------------------------- batches + + def get_many( + self: Store[Readable], requests: Iterable[Q] + ) -> Iterator[tuple[Q, Found | Missing]]: + """`get` for many keys, yielding `(request, result)` pairs as reads finish. + + A request is a key, or a `(key, byte_range)` pair, and comes back as it was given, so + `dict(store.get_many(keys))` maps keys to results. Keys are checked when `get_many` is + called; the reads happen as the pairs are taken, in no particular order. A batch is not + atomic: each read is its own `get`, from whatever generation it finds (`get_ranges` reads + several parts of one value from one generation). + + If reads raise, the iterator yields the reads already running that succeed, then raises + `BatchError` with the requests left in `remaining`; its `completed` is empty, since those + results were yielded. + """ + batch = list(requests) + self._check_open() + paths = [self._read_request(request) for request in batch] + backend = self._backend + if isinstance(backend, ReadableMany): + return self._streamed(batch, backend.get_many(paths)) + return self._streamed(batch, run_many(backend.get, paths)) + + def get_into_many( + self: Store[ReadableInto[O]], + requests: Iterable[tuple[str, O] | tuple[str, O, ByteRange | None]], + ) -> Iterator[tuple[tuple[str, O] | tuple[str, O, ByteRange | None], Copied | Missing]]: + """`get_into` for many `(key, out)` or `(key, out, byte_range)` requests, as `get_many`. + + Each request names its own buffer; two requests sharing one write it in no set order. + """ + batch = list(requests) + self._check_open() + paths = [self._into_request(request) for request in batch] + backend = self._backend + if isinstance(backend, ReadableIntoMany): + many = cast("ReadableIntoMany[O]", backend) + return self._streamed(batch, many.get_into_many(paths)) + return self._streamed(batch, run_many(backend.get_into, paths)) + + def put_many( + self: Store[Writable], items: Iterable[tuple[str, Buffer]] + ) -> list[tuple[str, Written]]: + """`put` each `(key, value)`, returning `(key, result)` pairs in the order given. + + Every key is checked, and every value viewed, before anything is written; a key named twice + raises `ValueError`. A batch is not atomic. If writes raise, the writes already running + finish, no others start, and `BatchError` reports the keys written (`completed`, with their + results) and the rest (`remaining`). + """ + batch = list(items) + keys = [key for key, _ in batch] + paths = self._writable_paths(keys) + requests = [ + (path, byte_view(value, writable=False)) + for path, (_, value) in zip(paths, batch, strict=True) + ] + backend = self._backend + if isinstance(backend, WritableMany): + return self._finished(keys, backend.put_many(requests)) + return self._finished(keys, run_many(backend.put, requests)) + + def delete_many( + self: Store[Writable], keys: Iterable[str] + ) -> list[tuple[str, Deleted | Missing]]: + """`delete` each key, returning `(key, result)` pairs in the order given, as `put_many`.""" + batch = list(keys) + paths = self._writable_paths(batch) + backend = self._backend + if isinstance(backend, WritableMany): + return self._finished(batch, backend.delete_many(paths)) + return self._finished(batch, run_many(backend.delete, [(path,) for path in paths])) + + # ----------------------------------------------------------------- listing + + def list(self: Store[Listable], directory: str = "") -> Iterator[str]: + """Every key under `directory`, relative to this store's root, in no particular order. + + The directory is checked when `list` is called; iterating after the store closes raises + `StoreClosedError`. + """ + paths = self._backend.list(self._directory(directory)) + return self._open_items(relativize(path, self._root) for path in paths) + + def list_dir(self: Store[Listable], directory: str = "") -> Iterator[ListItem]: + """The direct children of `directory`, named relative to it. + + A backend may report a `Prefix` that holds no keys, such as an empty directory on a + filesystem. + """ + return self._open_items(self._backend.list_dir(self._directory(directory))) + + +@final +class AsyncStore(_View[B_co]): + """A backend seen from a root directory, through coroutines. + + The async twin of `Store`: the same operations under the same names, awaited, over the + backend's async capabilities (`ReadableAsync`, `WritableAsync`, ...). Listings and lazy batches + are async iterators. Construct one from a backend, or take `store.as_async()`; `as_sync()` + goes back. Close it with `await store.aclose()` or `async with`. + + Parameters + ---------- + backend + The backend to read and write through. + root + The directory within the backend that this store's keys are relative to. + read_only + Refuse writes through this store. A backend that can't write makes every store over it + read-only. + """ + + __slots__ = () + + def __init__(self, backend: B_co, *, root: str = "", read_only: bool = False) -> None: + super().__init__(backend, root=root, read_only=read_only) + + def as_sync(self) -> Store[B_co]: + """A `Store` view on the same backend and root; it doesn't own the backend.""" + view = Store(self._backend, root=self._root, read_only=self._read_only) + view._owner = False + return view + + # ----------------------------------------------------------------- lifecycle + + async def aclose(self) -> None: + """Close the backend, if this store owns it (it was constructed, not derived). + + A backend that is `ClosableAsync` is closed with `aclose`; any other with `close`. + """ + if not self._owner: + return + backend = self._backend + if isinstance(backend, ClosableAsync): + await backend.aclose() + else: + backend.close() + + async def __aenter__(self) -> Self: + return self + + async def __aexit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + await self.aclose() + + # ----------------------------------------------------------------- batch results + + async def _streamed( + self, batch: Sequence[T], results: AsyncIterator[tuple[int, R]] + ) -> AsyncIterator[tuple[T, R]]: + done: set[int] = set() + try: + while True: + self._check_open() + pair = await anext(results, None) + if pair is None: + return + index, result = pair + done.add(index) + yield batch[index], result + except ExceptionGroup as group: + raise _batch_error(group, batch, {}, done) from None + finally: + if isinstance(results, AsyncGenerator): + await results.aclose() + + @staticmethod + async def _finished( + batch: Sequence[T], results: AsyncIterator[tuple[int, R]] + ) -> list[tuple[T, R]]: + completed: dict[int, R] = {} + try: + async for index, result in results: + completed[index] = result # noqa: PERF403 -- a failure keeps those before it + except ExceptionGroup as group: + raise _batch_error(group, batch, completed, completed.keys()) from None + return [(request, completed[index]) for index, request in enumerate(batch)] + + # ----------------------------------------------------------------- reads + + async def get( + self: AsyncStore[ReadableAsync], key: str, *, byte_range: ByteRange | None = None + ) -> Found | Missing: + """As `Store.get`.""" + return await self._backend.get_async(self._path(key), byte_range) + + async def get_if_changed( + self: AsyncStore[ReadableAsync], + key: str, + generation: Generation, + *, + byte_range: ByteRange | None = None, + ) -> Found | Missing | Unchanged: + """As `Store.get_if_changed`.""" + return await self._backend.get_if_changed_async(self._path(key), generation, byte_range) + + async def get_ranges( + self: AsyncStore[ReadableAsync], key: str, ranges: Sequence[ByteRange] + ) -> FoundRanges | Missing: + """As `Store.get_ranges`.""" + return await self._backend.get_ranges_async(self._path(key), ranges) + + async def get_into( + self: AsyncStore[ReadableIntoAsync[O]], + key: str, + out: O, + *, + byte_range: ByteRange | None = None, + ) -> Copied | Missing: + """As `Store.get_into`.""" + return await self._backend.get_into_async(self._path(key), out, byte_range) + + async def head(self: AsyncStore[ReadableAsync], key: str) -> ObjectMetadata | Missing: + """As `Store.head`.""" + return await self._backend.head_async(self._path(key)) + + async def exists(self: AsyncStore[ReadableAsync], key: str) -> bool: + """As `Store.exists`.""" + return not isinstance(await self.head(key), Missing) + + # ----------------------------------------------------------------- writes + + async def put(self: AsyncStore[WritableAsync], key: str, value: Buffer) -> Written: + """As `Store.put`.""" + return await self._backend.put_async( + self._writable_path(key), byte_view(value, writable=False) + ) + + async def put_if_absent( + self: AsyncStore[WritableAsync], key: str, value: Buffer + ) -> Written | Rejected: + """As `Store.put_if_absent`.""" + return await self._backend.put_if_absent_async( + self._writable_path(key), byte_view(value, writable=False) + ) + + async def put_if_match( + self: AsyncStore[WritableIfMatchAsync], key: str, value: Buffer, generation: Generation + ) -> Written | Rejected | Missing: + """As `Store.put_if_match`.""" + return await self._backend.put_if_match_async( + self._writable_path(key), byte_view(value, writable=False), generation + ) + + async def delete(self: AsyncStore[WritableAsync], key: str) -> Deleted | Missing: + """As `Store.delete`.""" + return await self._backend.delete_async(self._writable_path(key)) + + async def delete_if_match( + self: AsyncStore[DeletableIfMatchAsync], key: str, generation: Generation + ) -> Deleted | Rejected | Missing: + """As `Store.delete_if_match`.""" + return await self._backend.delete_if_match_async(self._writable_path(key), generation) + + async def delete_dir(self: AsyncStore[WritableAsync], directory: str = "") -> None: + """As `Store.delete_dir`.""" + await self._backend.delete_dir_async(self._writable_directory(directory)) + + # ----------------------------------------------------------------- batches + + def get_many( + self: AsyncStore[ReadableAsync], requests: Iterable[Q] + ) -> AsyncIterator[tuple[Q, Found | Missing]]: + """As `Store.get_many`.""" + batch = list(requests) + self._check_open() + paths = [self._read_request(request) for request in batch] + backend = self._backend + if isinstance(backend, ReadableManyAsync): + return self._streamed(batch, backend.get_many_async(paths)) + return self._streamed(batch, run_many_async(backend.get_async, paths)) + + def get_into_many( + self: AsyncStore[ReadableIntoAsync[O]], + requests: Iterable[tuple[str, O] | tuple[str, O, ByteRange | None]], + ) -> AsyncIterator[tuple[tuple[str, O] | tuple[str, O, ByteRange | None], Copied | Missing]]: + """As `Store.get_into_many`.""" + batch = list(requests) + self._check_open() + paths = [self._into_request(request) for request in batch] + backend = self._backend + if isinstance(backend, ReadableIntoManyAsync): + many = cast("ReadableIntoManyAsync[O]", backend) + return self._streamed(batch, many.get_into_many_async(paths)) + return self._streamed(batch, run_many_async(backend.get_into_async, paths)) + + async def put_many( + self: AsyncStore[WritableAsync], items: Iterable[tuple[str, Buffer]] + ) -> list[tuple[str, Written]]: + """As `Store.put_many`.""" + batch = list(items) + keys = [key for key, _ in batch] + paths = self._writable_paths(keys) + requests = [ + (path, byte_view(value, writable=False)) + for path, (_, value) in zip(paths, batch, strict=True) + ] + backend = self._backend + if isinstance(backend, WritableManyAsync): + return await self._finished(keys, backend.put_many_async(requests)) + return await self._finished(keys, run_many_async(backend.put_async, requests)) + + async def delete_many( + self: AsyncStore[WritableAsync], keys: Iterable[str] + ) -> list[tuple[str, Deleted | Missing]]: + """As `Store.delete_many`.""" + batch = list(keys) + paths = self._writable_paths(batch) + backend = self._backend + if isinstance(backend, WritableManyAsync): + return await self._finished(batch, backend.delete_many_async(paths)) + return await self._finished( + batch, run_many_async(backend.delete_async, [(path,) for path in paths]) + ) + + # ----------------------------------------------------------------- listing + + def list(self: AsyncStore[ListableAsync], directory: str = "") -> AsyncIterator[str]: + """As `Store.list`.""" + paths = self._backend.list_async(self._directory(directory)) + return self._open_items_async(_relativized(paths, self._root)) + + def list_dir(self: AsyncStore[ListableAsync], directory: str = "") -> AsyncIterator[ListItem]: + """As `Store.list_dir`.""" + return self._open_items_async(self._backend.list_dir_async(self._directory(directory))) + + +def _batch_error( + group: ExceptionGroup[Exception], + batch: Sequence[T], + completed: dict[int, R], + done: Iterable[int], +) -> BatchError: + finished = set(done) + return BatchError( + f"{len(group.exceptions)} of {len(batch)} operations in a batch failed.", + group.exceptions, + completed=tuple((batch[index], result) for index, result in sorted(completed.items())), + remaining=tuple(request for index, request in enumerate(batch) if index not in finished), + ) + + +async def _relativized(paths: AsyncIterator[str], root: str) -> AsyncIterator[str]: + async for path in paths: + yield relativize(path, root) + + +def _restore_store( + cls: type[_View[Backend]], backend: Backend, root: str, read_only: bool +) -> _View[Backend]: + return cls(backend, root=root, read_only=read_only) diff --git a/packages/zarr-storage/src/zarr_storage/_types.py b/packages/zarr-storage/src/zarr_storage/_types.py new file mode 100644 index 0000000000..3c50860351 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/_types.py @@ -0,0 +1,190 @@ +"""Values that stores take and return. All are immutable.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import NewType + +__all__ = [ + "ByteRange", + "Copied", + "Deleted", + "Entry", + "Found", + "FoundRanges", + "Generation", + "Key", + "ListItem", + "Missing", + "ObjectMetadata", + "OffsetRequest", + "Prefix", + "RangeRequest", + "Rejected", + "SuffixRequest", + "Unchanged", + "Written", +] + +Generation = NewType("Generation", object) +"""An opaque token for the state of one key, compared only by equality. + +Backends mint it: a counter (memory), `(device, inode, mtime_ns, size)` (local files), an ETag +(object stores). Two reads that return equal generations read the same value. +""" + + +def _check_offsets(**offsets: int) -> None: + for name, value in offsets.items(): + if type(value) is not int: + raise TypeError(f"{name} must be an int; got {type(value).__name__}.") + + +@dataclass(frozen=True, slots=True) +class RangeRequest: + """The bytes from `start` (inclusive) to `end` (exclusive).""" + + start: int + end: int + + def __post_init__(self) -> None: + _check_offsets(start=self.start, end=self.end) + if self.start < 0 or self.end < self.start: + raise ValueError( + f"A range needs 0 <= start <= end; got start={self.start}, end={self.end}." + ) + + +@dataclass(frozen=True, slots=True) +class OffsetRequest: + """The bytes from `start` (inclusive) to the end of the value.""" + + start: int + + def __post_init__(self) -> None: + _check_offsets(start=self.start) + if self.start < 0: + raise ValueError(f"An offset must be >= 0; got {self.start}.") + + +@dataclass(frozen=True, slots=True) +class SuffixRequest: + """The last `length` bytes of the value, or the whole value if it is shorter.""" + + length: int + + def __post_init__(self) -> None: + _check_offsets(length=self.length) + if self.length < 0: + raise ValueError(f"A suffix length must be >= 0; got {self.length}.") + + +ByteRange = RangeRequest | OffsetRequest | SuffixRequest +"""A part of a value. An end past the value is clamped; a start past the value is an error.""" + + +@dataclass(frozen=True, slots=True) +class Found: + """A read found the key: its bytes (or the requested part) and their generation.""" + + value: memoryview + generation: Generation + read_at: float | None = None + """When the store read the value, in seconds since the UNIX epoch, if the backend knows.""" + + +@dataclass(frozen=True, slots=True) +class FoundRanges: + """A read found the key: the bytes of several ranges, all from one generation.""" + + values: tuple[memoryview, ...] + generation: Generation + read_at: float | None = None + + +@dataclass(frozen=True, slots=True) +class Copied: + """A read found the key and copied `size` bytes into the caller's buffer.""" + + size: int + generation: Generation + read_at: float | None = None + + +@dataclass(frozen=True, slots=True) +class Missing: + """The key holds no value (an HTTP store's `404`). + + `read_at` records when the store looked, if the backend knows, so an absence can be cached like + a value. + """ + + read_at: float | None = None + + +@dataclass(frozen=True, slots=True) +class Unchanged: + """`get_if_changed` named the key's current generation, so no bytes were read (an HTTP `304`).""" + + generation: Generation + read_at: float | None = None + + +@dataclass(frozen=True, slots=True) +class Written: + """A write stored its value, which has `generation`.""" + + generation: Generation + + +@dataclass(frozen=True, slots=True) +class Deleted: + """A delete removed the key's value, which had `generation`.""" + + generation: Generation + + +@dataclass(frozen=True, slots=True) +class Rejected: + """A conditional write or delete changed nothing: the key holds a value with `generation`. + + `put_if_absent` is rejected by any value; `put_if_match` and `delete_if_match` by a value whose + generation is not the one they named. + """ + + generation: Generation + + +@dataclass(frozen=True, slots=True) +class ObjectMetadata: + """What a store knows about one key without reading its value.""" + + size: int + generation: Generation + last_modified: float | None + """Seconds since the UNIX epoch, or `None` if the backend does not track it.""" + + +@dataclass(frozen=True, slots=True) +class Key: + """A key directly inside the listed directory. `name` has no `/`.""" + + name: str + + +@dataclass(frozen=True, slots=True) +class Prefix: + """A directory directly inside the listed directory. `name` has no `/`.""" + + name: str + + +@dataclass(frozen=True, slots=True) +class Entry: + """A child of the listed directory that the backend can't classify as a key or a directory.""" + + name: str + + +ListItem = Key | Prefix | Entry +"""One child of a listed directory. A name can appear both as a `Key` and as a `Prefix`.""" diff --git a/packages/zarr-storage/src/zarr_storage/backends/__init__.py b/packages/zarr-storage/src/zarr_storage/backends/__init__.py new file mode 100644 index 0000000000..d0dd4b42ec --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/backends/__init__.py @@ -0,0 +1,24 @@ +"""Backends: the objects a `Store` reads and writes through. + +`ObjectStoreBackend` needs the `obstore` extra and is imported on first use. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr_storage.backends._local import LocalBackend +from zarr_storage.backends._memory import MemoryBackend + +if TYPE_CHECKING: + from zarr_storage.backends._obstore import ObjectStoreBackend + +__all__ = ["LocalBackend", "MemoryBackend", "ObjectStoreBackend"] + + +def __getattr__(name: str) -> object: + if name == "ObjectStoreBackend": + from zarr_storage.backends._obstore import ObjectStoreBackend + + return ObjectStoreBackend + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/packages/zarr-storage/src/zarr_storage/backends/_common.py b/packages/zarr-storage/src/zarr_storage/backends/_common.py new file mode 100644 index 0000000000..2cd5984317 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/backends/_common.py @@ -0,0 +1,68 @@ +"""Helpers shared by backends: preconditions, and listing a flat set of keys. + +A precondition is a function of a key's current generation (`None` if the key is absent) that +returns the outcome that refuses the operation, or `None` to let it proceed. A backend writes each +operation once, parameterized by its precondition; the return type follows from the precondition's, +so `put` (with `always`) returns only `Written`. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from zarr_storage._keys import is_within, relativize +from zarr_storage._types import Key, Missing, Prefix, Rejected + +if TYPE_CHECKING: + from collections.abc import Iterable, Iterator + + from zarr_storage._types import Generation, ListItem + +__all__ = ["IfMatch", "always", "children", "if_absent", "strictly_within"] + + +def always(current: Generation | None) -> None: + """The precondition of an unconditional operation: it always proceeds.""" + + +def if_absent(current: Generation | None) -> Rejected | None: + """Proceed only if the key holds no value.""" + return None if current is None else Rejected(current) + + +@dataclass(frozen=True, slots=True) +class IfMatch: + """Proceed only if the key's value has `generation`.""" + + generation: Generation + + def __call__(self, current: Generation | None) -> Rejected | Missing | None: + if current is None: + return Missing() + if current != self.generation: + return Rejected(current) + return None + + +def strictly_within(path: str, directory: str) -> bool: + """Whether `path` lies under `directory` (normalized) and is not `directory` itself.""" + return path != directory and is_within(path, directory) + + +def children(paths: Iterable[str], directory: str) -> Iterator[ListItem]: + """The direct children of `directory` among a flat set of key paths. + + A path directly inside `directory` is a `Key`; the first segment of a deeper path is a + `Prefix`, reported once. A name can be both. + """ + seen_prefixes: set[str] = set() + for path in paths: + if not strictly_within(path, directory): + continue + name, sep, _ = relativize(path, directory).partition("/") + if not sep: + yield Key(name) + elif name not in seen_prefixes: + seen_prefixes.add(name) + yield Prefix(name) diff --git a/packages/zarr-storage/src/zarr_storage/backends/_local.py b/packages/zarr-storage/src/zarr_storage/backends/_local.py new file mode 100644 index 0000000000..e6a22f68eb --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/backends/_local.py @@ -0,0 +1,870 @@ +"""A backend over a directory on the local filesystem.""" + +from __future__ import annotations + +import asyncio +import contextlib +import errno +import os +import shutil +import stat +import sys +import threading +import time +import uuid +from concurrent.futures import ThreadPoolExecutor +from functools import partial +from pathlib import Path +from typing import TYPE_CHECKING, ParamSpec, TypedDict, TypeVar + +from zarr_storage._batch import run_many, run_many_async +from zarr_storage._buffers import byte_view +from zarr_storage._byte_range import resolve +from zarr_storage._errors import ( + InvalidKeyError, + KeyConflictError, + StoreClosedError, + UnsupportedOperationError, +) +from zarr_storage._types import ( + Copied, + Deleted, + Found, + FoundRanges, + Generation, + Key, + Missing, + ObjectMetadata, + Prefix, + Unchanged, + Written, +) +from zarr_storage.backends._common import IfMatch, always, if_absent + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Callable, Generator, Iterator, Sequence + from io import BufferedReader + + from typing_extensions import Buffer + + from zarr_storage._types import ByteRange, ListItem, Rejected + +__all__ = ["RESERVED_SUFFIX", "LocalBackend", "segment_problem"] + +RESERVED_SUFFIX = ".__zarr_local" +"""File names ending with this suffix belong to the backend: temporary files and lock files. + +Keys with a segment ending in it are refused, and listings skip such names, as TensorStore's file +driver does for its own lock files. +""" + +_LOCK_NAME = f"lock{RESERVED_SUFFIX}" +_STAGING_NAME = f"staging{RESERVED_SUFFIX}" +"""The directory under the root where writes stage values before renaming them into place.""" + +_WRITE_ATTEMPTS = 64 +"""How many times a write starts over after a concurrent `delete_dir` removed its directory.""" + +_WINDOWS = sys.platform == "win32" + +_WINDOWS_DEVICE_NAMES = frozenset( + ["CON", "PRN", "AUX", "NUL"] + + [f"{name}{i}" for name in ("COM", "LPT") for i in [*range(1, 10), "¹", "²", "³"]] +) +_WINDOWS_FORBIDDEN_CHARACTERS = frozenset('<>:"\\|?*') | frozenset(chr(i) for i in range(32)) + +P = ParamSpec("P") +R = TypeVar("R") +T = TypeVar("T") + + +class _State(TypedDict): + root: str + concurrency: int + closed: bool + + +class _Retry(Exception): + """A concurrent `delete_dir` removed a directory a write was using; start the write over.""" + + +_TRANSIENT_ERRNOS = (errno.ENOENT, errno.EINVAL) +"""What creating a file in, or renaming into, a directory being removed reports. + +macOS reports EINVAL rather than ENOENT for a file created in a directory being removed. +""" + + +def _generation(st: os.stat_result) -> Generation: + return Generation((st.st_dev, st.st_ino, st.st_mtime_ns, st.st_size)) + + +_stamp_lock = threading.Lock() +_last_stamp = 0 + + +def _unique_stamp() -> int: + """A nanosecond time, at least now, and later than any this process has handed out. + + Every write stamps its file's mtime with one. A filesystem can reuse an inode as soon as a file + is deleted (ext4 does) and only advances mtimes once per clock tick, so without the stamp a + generation `(dev, inode, mtime_ns, size)` could come back for a different value, and a stale + `if_match` would succeed. + """ + global _last_stamp + with _stamp_lock: + _last_stamp = max(time.time_ns(), _last_stamp + 1) + return _last_stamp + + +def _set_mtime(fd: int, path: Path, stamp: int) -> None: + if os.utime in os.supports_fd: + os.utime(fd, ns=(stamp, stamp)) + else: + os.utime(path, ns=(stamp, stamp)) + + +_PROBE_STAMP = 1_700_000_000_123_456_789 +"""An mtime with every digit significant, to see whether a filesystem keeps nanoseconds.""" + + +def segment_problem(segment: str, *, windows: bool) -> str | None: + """Why a filesystem can't hold `segment` as one file or directory name, or `None` if it can. + + On every platform the backend's reserved suffix is refused. With `windows=True`, so are the + names Windows reads as something else: separators, `:` (a drive or stream), other forbidden + characters, device names such as `NUL` or `com1.txt`, and names ending in `.` or a space, which + Windows strips. + """ + if segment.endswith(RESERVED_SUFFIX): + return f"names ending in {RESERVED_SUFFIX!r} are reserved by the local backend" + if not windows: + return None + if not _WINDOWS_FORBIDDEN_CHARACTERS.isdisjoint(segment): + return 'Windows forbids the characters <>:"|?*\\ and control characters in names' + if segment.split(".")[0].upper() in _WINDOWS_DEVICE_NAMES: + return "Windows reserves device names such as NUL, CON and COM1" + if segment.endswith((".", " ")): + return "Windows strips a trailing '.' or space from names" + return None + + +def _check_segments(path: str) -> None: + for segment in path.split("/"): + problem = segment_problem(segment, windows=_WINDOWS) + if problem is not None: + raise InvalidKeyError(f"{path!r} can't be stored on this filesystem: {problem}.") + + +@contextlib.contextmanager +def _key_errors(path: str) -> Generator[None, None, None]: + """Report a name the filesystem can't hold as an invalid key.""" + try: + yield + except OSError as e: + if e.errno == errno.ENAMETOOLONG: + raise InvalidKeyError(f"{path!r} is too long for this filesystem.") from e + raise + + +def _open_key(file: Path) -> BufferedReader | None: + """The file at `file` opened for reading, or `None` if it is absent or is a directory.""" + try: + return open(file, "rb") + except (FileNotFoundError, IsADirectoryError, NotADirectoryError): + return None + except PermissionError: + # Windows refuses to open a directory with PermissionError. + if file.is_dir(): + return None + raise + + +def _read(f: BufferedReader, st: os.stat_result, byte_range: ByteRange | None) -> Found: + """The part of the open file `f`, whose stat is `st`, that `byte_range` selects.""" + start, end = resolve(byte_range, st.st_size) + f.seek(start) + return Found(memoryview(f.read(end - start)), _generation(st), time.time()) + + +def _read_exact_into(f: BufferedReader, out: memoryview) -> None: + filled = 0 + while filled < len(out): + n = f.readinto(out[filled:]) + if n == 0: + raise OSError("The file shrank while it was being read.") + filled += n + + +def _stat_key(file: Path) -> os.stat_result | None: + """The file's stat, or `None` if it is absent or is not a regular file.""" + try: + st = file.stat() + except (FileNotFoundError, NotADirectoryError): + return None + return st if stat.S_ISREG(st.st_mode) else None + + +@contextlib.contextmanager +def _directory_lock(directory: Path) -> Generator[None, None, None]: + """Hold the exclusive lock of one directory. + + Every write and delete of a key holds the lock of the key's directory while it checks + preconditions and renames or unlinks. The lock lives in the directory itself, so two backends + whose roots overlap use the same lock for the same key. `delete_dir` removes a directory only + while holding its lock; a waiter that wakes holding the removed lock file tries again. + + Locks are taken ancestor before descendant, never the other way, so they can't deadlock. + + Raises `_Retry` if the directory is being removed. + """ + lock_path = directory / _LOCK_NAME + while True: + try: + fd = os.open(lock_path, os.O_RDWR | os.O_CREAT, 0o644) + except NotADirectoryError as e: + # The directory was removed and a key of the same name written in its place. + raise _Retry from e + except OSError as e: + if e.errno in _TRANSIENT_ERRNOS: + raise _Retry from e + raise + try: + _lock(fd) + held = os.fstat(fd) + try: + current = os.stat(lock_path) + except (FileNotFoundError, NotADirectoryError): + current = None + if current is not None and (current.st_dev, current.st_ino) == ( + held.st_dev, + held.st_ino, + ): + break + _unlock(fd) + except BaseException: + os.close(fd) + raise + os.close(fd) + try: + yield + finally: + _unlock(fd) + os.close(fd) + + +def _replace_if( + file: Path, + path: str, + temp: Path, + staged: Generation, + check: Callable[[Generation | None], R | None], +) -> Written | R: + """Rename `temp` to `file` if `check` lets it. The caller holds the directory's lock. + + `staged` is the generation of `temp`; a rename keeps the inode, mtime and size, so it is the + generation of the written key too. A directory in the way is a conflict, whatever the + preconditions, unless it holds no keys; then it gives way. + """ + if _is_real_directory(file) and not _reclaim(file): + raise KeyConflictError(f"{path!r} is a directory, so it can't also be a key.") + st = _stat_key(file) + refusal = check(None if st is None else _generation(st)) + if refusal is not None: + return refusal + try: + os.replace(temp, file) + except (IsADirectoryError, PermissionError) as e: + if _is_real_directory(file): + # Deletes leave directories behind; one that holds no keys gives way to the key. + if not _reclaim(file): + raise KeyConflictError( + f"{path!r} is a directory, so it can't also be a key." + ) from e + raise _Retry from e + if isinstance(e, IsADirectoryError): + raise _Retry from e # The directory in the way vanished meanwhile. + raise + except OSError as e: + if e.errno in _TRANSIENT_ERRNOS: + raise _Retry from e + raise + return Written(staged) + + +def _is_real_directory(path: Path) -> bool: + """Whether `path` is a directory itself, not a symlink to one (a symlink is a key's file).""" + return path.is_dir() and not path.is_symlink() + + +def _reclaim(directory: Path) -> bool: + """Remove `directory` if no key lies under it, and say whether it is gone. + + Only empty directories are removed, with `rmdir`, deepest first, each under its own lock + (ancestor before descendant). A file anywhere stops it, and so does a writer adding one meanwhile: + `rmdir` refuses a directory that isn't empty, so a reclaim can never remove a key. Another + operation can recreate a directory's lock file between its removal and the `rmdir`; then the + directory is scanned again. + """ + for _ in range(_WRITE_ATTEMPTS): + try: + with _directory_lock(directory): + with os.scandir(directory) as entries: + children = [e for e in entries if e.name != _LOCK_NAME] + for entry in children: + if not entry.is_dir(follow_symlinks=False) or entry.name.endswith( + RESERVED_SUFFIX + ): + return False + if not _reclaim(Path(entry.path)): + return False + os.unlink(directory / _LOCK_NAME) + os.rmdir(directory) + except (_Retry, FileNotFoundError): + return True + except OSError as e: + if e.errno == errno.ENOTEMPTY: + continue + raise + return True + return False + + +def _remove_tree(directory: Path, *, remove_self: bool) -> None: + """Delete every key under `directory`, and its subdirectories, each under its own lock. + + The directory's lock is held while its children are removed, and a child's lock is taken while + the parent's is held (ancestor before descendant), so a writer holding a directory's lock always + finishes before the directory goes. With `remove_self`, `directory` is removed too, unless a + concurrent writer has added to it meanwhile; directories aren't keys, so that is not an error. + """ + try: + with _directory_lock(directory): + with os.scandir(directory) as entries: + children = [e for e in entries if e.name != _LOCK_NAME] + for entry in children: + if entry.name.endswith(RESERVED_SUFFIX): + # Below the root, this is a nested backend's staging directory; at the root, + # it is this backend's own and stays. + if remove_self: + shutil.rmtree(entry.path, ignore_errors=True) + continue + if entry.is_dir(follow_symlinks=False): + _remove_tree(Path(entry.path), remove_self=True) + else: + with contextlib.suppress(FileNotFoundError): + os.unlink(entry.path) + if remove_self: + os.unlink(directory / _LOCK_NAME) + try: + os.rmdir(directory) + except OSError as e: + if e.errno != errno.ENOTEMPTY: + raise + except (_Retry, FileNotFoundError, NotADirectoryError): + return # The directory is already gone. + + +class LocalBackend: + """Keys stored as files under a root directory. + + A write goes to a temporary file in a staging directory under the root and is renamed into + place, so readers see the old value or the new one, never part of either. The root must not + span filesystems (no mount points under it), since a rename can't cross one. Writes are not + synced to disk. + + Writes and deletes hold the lock of the key's directory while they check preconditions and + rename, which makes conditional writes atomic across threads, processes and backends whose + roots overlap, on one machine. + + Deleting a key leaves its directory in place, so `list_dir` can report a `Prefix` that holds no + keys; a later write of a key with that directory's name removes it if no key lies under it. `delete_dir` removes the + tree bottom-up, taking each directory's lock first, so a write in progress finishes before its + directory goes; it is not atomic, and a key written while it runs may survive it. A write whose + directory a concurrent `delete_dir` removes starts over, with a short backoff, up to 64 times. + + Temporary files and lock files end in `RESERVED_SUFFIX` and are never listed. A crash can leave + staged files behind in the staging directory; they are never read. + + Key names follow the filesystem: on case-insensitive or Unicode-normalizing filesystems (the + defaults on macOS and Windows), keys that differ only in case or normalization are the same key. + + Parameters + ---------- + root + The directory that holds the keys. + create + Create `root` if it doesn't exist. With `create=False`, a missing root raises + `FileNotFoundError`. + concurrency + The number of threads that run the async methods' file operations, and the number of + operations a batch (`get_many`, `put_many`, ...) runs at once. + """ + + def __init__( + self, root: str | os.PathLike[str], *, create: bool = True, concurrency: int = 16 + ) -> None: + if concurrency < 1: + raise ValueError(f"concurrency must be >= 1; got {concurrency}.") + self._root = Path(root).resolve() + if create: + self._root.mkdir(parents=True, exist_ok=True) + elif not self._root.is_dir(): + raise FileNotFoundError(f"The root directory {str(self._root)!r} does not exist.") + self._concurrency = concurrency + self._executor: ThreadPoolExecutor | None = None + self._executor_lock = threading.Lock() + self._probe_lock = threading.Lock() + self._exact_times: bool | None = None + self._closed = False + + @property + def root(self) -> Path: + """The directory that holds the keys.""" + return self._root + + def __repr__(self) -> str: + return f"LocalBackend({str(self._root)!r})" + + def __getstate__(self) -> _State: + return {"root": str(self._root), "concurrency": self._concurrency, "closed": self._closed} + + def __setstate__(self, state: _State) -> None: + self._root = Path(state["root"]) + self._concurrency = state["concurrency"] + self._executor = None + self._executor_lock = threading.Lock() + self._probe_lock = threading.Lock() + self._exact_times = None + self._closed = state["closed"] + + # --- Backend + + @property + def closed(self) -> bool: + return self._closed + + @property + def writable(self) -> bool: + return True + + def close(self) -> None: + with self._executor_lock: + self._closed = True + executor, self._executor = self._executor, None + if executor is not None: + executor.shutdown(wait=True) + + async def aclose(self) -> None: + """`close` on a worker thread: it waits for running file operations to finish.""" + await asyncio.to_thread(self.close) + + # --- paths + + def _file(self, path: str) -> Path: + _check_segments(path) + return self._root / path + + def _directory(self, directory: str) -> Path: + if directory == "": + return self._root + _check_segments(directory) + return self._root / directory + + # --- Readable + + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + with _key_errors(path): + f = _open_key(self._file(path)) + if f is None: + return Missing(time.time()) + with f: + return _read(f, os.fstat(f.fileno()), byte_range) + + def get_if_changed( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: + with _key_errors(path): + f = _open_key(self._file(path)) + if f is None: + return Missing(time.time()) + with f: + st = os.fstat(f.fileno()) + if _generation(st) == generation: + return Unchanged(generation, time.time()) + return _read(f, st, byte_range) + + def get_ranges(self, path: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: + with _key_errors(path): + f = _open_key(self._file(path)) + if f is None: + return Missing(time.time()) + parts: list[memoryview] = [] + with f: + st = os.fstat(f.fileno()) + for byte_range in ranges: + start, end = resolve(byte_range, st.st_size) + f.seek(start) + parts.append(memoryview(f.read(end - start))) + return FoundRanges(tuple(parts), _generation(st), time.time()) + + def get_into(self, path: str, out: Buffer, byte_range: ByteRange | None) -> Copied | Missing: + view = byte_view(out, writable=True) + with _key_errors(path): + f = _open_key(self._file(path)) + if f is None: + return Missing(time.time()) + with f: + st = os.fstat(f.fileno()) + start, end = resolve(byte_range, st.st_size) + n = end - start + if n > len(view): + raise ValueError(f"The output buffer holds {len(view)} bytes; the value has {n}.") + f.seek(start) + _read_exact_into(f, view[:n]) + return Copied(n, _generation(st), time.time()) + + def head(self, path: str) -> ObjectMetadata | Missing: + with _key_errors(path): + st = _stat_key(self._file(path)) + if st is None: + return Missing(time.time()) + return ObjectMetadata(st.st_size, _generation(st), st.st_mtime) + + # --- Writable + + def put(self, path: str, value: memoryview) -> Written: + return self._put(path, value, always) + + def put_if_absent(self, path: str, value: memoryview) -> Written | Rejected: + return self._put(path, value, if_absent) + + def put_if_match( + self, path: str, value: memoryview, generation: Generation + ) -> Written | Rejected | Missing: + self._require_exact_times() + return self._put(path, value, IfMatch(generation)) + + def _put( + self, path: str, value: memoryview, check: Callable[[Generation | None], R | None] + ) -> Written | R: + file = self._file(path) + last = _Retry() + with _key_errors(path): + for attempt in range(_WRITE_ATTEMPTS): + temp: Path | None = None + try: + temp, staged = self._stage(value) + return self._commit(file, path, temp, staged, check) + except _Retry as retry: + last = retry + time.sleep(min(1e-4 * 2**attempt, 1e-2)) + finally: + if temp is not None: + with contextlib.suppress(OSError): + temp.unlink(missing_ok=True) + raise OSError( + f"Gave up writing {path!r}: a concurrent delete kept removing its directory." + ) from last.__cause__ + + def _require_exact_times(self) -> None: + """Refuse conditional writes on a filesystem that rounds file times. + + Generations need the nanosecond stamps writes set; a filesystem that rounds them (FAT, + HFS+, some network filesystems) could hand an old generation out again. + """ + with self._probe_lock: + if self._exact_times is None: + self._exact_times = self._probe_exact_times() + if not self._exact_times: + raise UnsupportedOperationError( + f"The filesystem under {str(self._root)!r} rounds file times, so generations " + "aren't unique enough for conditional writes." + ) + + def _probe_exact_times(self) -> bool: + staging = self._root / _STAGING_NAME + staging.mkdir(parents=True, exist_ok=True) + probe = staging / f"probe-{uuid.uuid4().hex}" + try: + with open(probe, "xb") as f: + _set_mtime(f.fileno(), probe, _PROBE_STAMP) + return os.fstat(f.fileno()).st_mtime_ns == _PROBE_STAMP + finally: + probe.unlink(missing_ok=True) + + def _stage(self, value: memoryview) -> tuple[Path, Generation]: + """Write `value` to a new file in the staging directory, outside every key's directory. + + Raises `_Retry` if a concurrent delete removed the root or the staging directory. + """ + staging = self._root / _STAGING_NAME + temp = staging / uuid.uuid4().hex + try: + staging.mkdir(parents=True, exist_ok=True) + with open(temp, "xb") as f: + f.write(value) + f.flush() + _set_mtime(f.fileno(), temp, _unique_stamp()) + staged = _generation(os.fstat(f.fileno())) + except (FileExistsError, NotADirectoryError) as e: + if self._root.exists() and not self._root.is_dir(): + raise KeyConflictError( + f"The root {str(self._root)!r} is now a file (a key of an enclosing store)." + ) from e + # `mkdir` saw the directory, then a concurrent delete removed it. + raise _Retry from e + except OSError as e: + if e.errno in _TRANSIENT_ERRNOS: + raise _Retry from e + raise + return temp, staged + + def _has_key_parent(self, file: Path) -> bool: + """Whether a directory between the root and `file` is a regular file (a key).""" + parent = file.parent + while parent != self._root: + if _stat_key(parent) is not None: + return True + parent = parent.parent + return False + + def _commit( + self, + file: Path, + path: str, + temp: Path, + staged: Generation, + check: Callable[[Generation | None], R | None], + ) -> Written | R: + """Rename `temp` to `file` if `check` lets it, under the lock of `file`'s directory. + + Raises `_Retry` if a concurrent `delete_dir` removed the directory on the way. + """ + try: + file.parent.mkdir(parents=True, exist_ok=True) + except (FileExistsError, NotADirectoryError) as e: + if self._has_key_parent(file): + raise KeyConflictError( + f"A parent of {path!r} is a key, so it can't also be a directory." + ) from e + # `mkdir` saw the directory, then a concurrent delete removed it before `mkdir` checked + # that it was one. + raise _Retry from e + except OSError as e: + if e.errno in _TRANSIENT_ERRNOS: + raise _Retry from e + raise + with _directory_lock(file.parent): + return _replace_if(file, path, temp, staged, check) + + def delete(self, path: str) -> Deleted | Missing: + return self._delete(path, always) + + def delete_if_match(self, path: str, generation: Generation) -> Deleted | Rejected | Missing: + self._require_exact_times() + return self._delete(path, IfMatch(generation)) + + def _delete( + self, path: str, check: Callable[[Generation | None], R | None] + ) -> Deleted | Missing | R: + file = self._file(path) + with _key_errors(path): + try: + with _directory_lock(file.parent): + st = _stat_key(file) + if st is None: + return Missing(time.time()) + current = _generation(st) + refusal = check(current) + if refusal is not None: + return refusal + file.unlink() + except (_Retry, NotADirectoryError): + # The key's directory doesn't exist, or a parent is a key: the key is absent. + return Missing(time.time()) + return Deleted(current) + + def delete_dir(self, directory: str) -> None: + target = self._directory(directory) + with _key_errors(directory): + if target == self._root: + _remove_tree(self._root, remove_self=False) + elif target.is_dir() and not target.is_symlink(): + _remove_tree(target, remove_self=True) + + # --- Listable + + def list(self, directory: str) -> Iterator[str]: + target = self._directory(directory) + prefix = "" if directory == "" else f"{directory}/" + with _key_errors(directory): + return iter(list(_walk(target, prefix))) + + def list_dir(self, directory: str) -> Iterator[ListItem]: + target = self._directory(directory) + with _key_errors(directory): + try: + entries = list(os.scandir(target)) + except (FileNotFoundError, NotADirectoryError): + return iter(()) + return _classify(entries) + + # --- batches: the operations above, `concurrency` at once on the backend's threads + + def get_many( + self, requests: Sequence[tuple[str, ByteRange | None]] + ) -> Iterator[tuple[int, Found | Missing]]: + return run_many(self.get, requests, self._pool(), self._concurrency) + + def get_into_many( + self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] + ) -> Iterator[tuple[int, Copied | Missing]]: + return run_many(self.get_into, requests, self._pool(), self._concurrency) + + def put_many(self, requests: Sequence[tuple[str, memoryview]]) -> Iterator[tuple[int, Written]]: + return run_many(self.put, requests, self._pool(), self._concurrency) + + def delete_many(self, paths: Sequence[str]) -> Iterator[tuple[int, Deleted | Missing]]: + return run_many(self.delete, [(path,) for path in paths], self._pool(), self._concurrency) + + # --- async twins: run the sync methods on the backend's threads + + def _pool(self) -> ThreadPoolExecutor: + with self._executor_lock: + if self._closed: + raise StoreClosedError(f"{self!r} is closed.") + if self._executor is None: + self._executor = ThreadPoolExecutor( + max_workers=self._concurrency, thread_name_prefix="zarr-storage-local" + ) + return self._executor + + async def _run(self, fn: Callable[P, T], *args: P.args, **kwargs: P.kwargs) -> T: + loop = asyncio.get_running_loop() + return await loop.run_in_executor(self._pool(), partial(fn, *args, **kwargs)) + + async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + return await self._run(self.get, path, byte_range) + + async def get_if_changed_async( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: + return await self._run(self.get_if_changed, path, generation, byte_range) + + async def get_ranges_async( + self, path: str, ranges: Sequence[ByteRange] + ) -> FoundRanges | Missing: + return await self._run(self.get_ranges, path, ranges) + + async def head_async(self, path: str) -> ObjectMetadata | Missing: + return await self._run(self.head, path) + + async def get_into_async( + self, path: str, out: Buffer, byte_range: ByteRange | None + ) -> Copied | Missing: + return await self._run(self.get_into, path, out, byte_range) + + async def put_async(self, path: str, value: memoryview) -> Written: + return await self._run(self.put, path, value) + + async def put_if_absent_async(self, path: str, value: memoryview) -> Written | Rejected: + return await self._run(self.put_if_absent, path, value) + + async def put_if_match_async( + self, path: str, value: memoryview, generation: Generation + ) -> Written | Rejected | Missing: + return await self._run(self.put_if_match, path, value, generation) + + async def delete_async(self, path: str) -> Deleted | Missing: + return await self._run(self.delete, path) + + async def delete_if_match_async( + self, path: str, generation: Generation + ) -> Deleted | Rejected | Missing: + return await self._run(self.delete_if_match, path, generation) + + async def delete_dir_async(self, directory: str) -> None: + await self._run(self.delete_dir, directory) + + def get_many_async( + self, requests: Sequence[tuple[str, ByteRange | None]] + ) -> AsyncIterator[tuple[int, Found | Missing]]: + return run_many_async(self.get_async, requests, self._concurrency) + + def get_into_many_async( + self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] + ) -> AsyncIterator[tuple[int, Copied | Missing]]: + return run_many_async(self.get_into_async, requests, self._concurrency) + + def put_many_async( + self, requests: Sequence[tuple[str, memoryview]] + ) -> AsyncIterator[tuple[int, Written]]: + return run_many_async(self.put_async, requests, self._concurrency) + + def delete_many_async( + self, paths: Sequence[str] + ) -> AsyncIterator[tuple[int, Deleted | Missing]]: + return run_many_async(self.delete_async, [(path,) for path in paths], self._concurrency) + + async def list_async(self, directory: str) -> AsyncIterator[str]: + for path in await self._run(_materialize, self.list(directory)): + yield path + + async def list_dir_async(self, directory: str) -> AsyncIterator[ListItem]: + for item in await self._run(_materialize, self.list_dir(directory)): + yield item + + +def _materialize(items: Iterator[T]) -> list[T]: + return list(items) + + +def _walk(directory: Path, prefix: str) -> Iterator[str]: + try: + entries = list(os.scandir(directory)) + except (FileNotFoundError, NotADirectoryError): + return + for entry in entries: + if entry.name.endswith(RESERVED_SUFFIX): + continue + if entry.is_dir(follow_symlinks=False): + yield from _walk(Path(entry.path), f"{prefix}{entry.name}/") + else: + yield f"{prefix}{entry.name}" + + +def _classify(entries: list[os.DirEntry[str]]) -> Iterator[ListItem]: + for entry in entries: + if entry.name.endswith(RESERVED_SUFFIX): + continue + if entry.is_dir(follow_symlinks=False): + yield Prefix(entry.name) + else: + yield Key(entry.name) + + +if sys.platform == "win32": + import msvcrt + + def _lock(fd: int) -> None: + os.lseek(fd, 0, os.SEEK_SET) + while True: + try: + msvcrt.locking(fd, msvcrt.LK_LOCK, 1) + except OSError: + continue + return + + def _unlock(fd: int) -> None: + os.lseek(fd, 0, os.SEEK_SET) + msvcrt.locking(fd, msvcrt.LK_UNLCK, 1) + +else: + import fcntl + + def _lock(fd: int) -> None: + fcntl.flock(fd, fcntl.LOCK_EX) + + def _unlock(fd: int) -> None: + fcntl.flock(fd, fcntl.LOCK_UN) diff --git a/packages/zarr-storage/src/zarr_storage/backends/_memory.py b/packages/zarr-storage/src/zarr_storage/backends/_memory.py new file mode 100644 index 0000000000..e7b9f7e335 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/backends/_memory.py @@ -0,0 +1,251 @@ +"""An in-process backend over a dict.""" + +from __future__ import annotations + +import threading +import time +from dataclasses import dataclass +from typing import TYPE_CHECKING, TypedDict, TypeVar + +from zarr_storage._buffers import byte_view +from zarr_storage._byte_range import resolve +from zarr_storage._types import ( + Copied, + Deleted, + Found, + FoundRanges, + Generation, + Missing, + ObjectMetadata, + Unchanged, + Written, +) +from zarr_storage.backends._common import IfMatch, always, children, if_absent, strictly_within + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Callable, Iterator, Sequence + + from typing_extensions import Buffer + + from zarr_storage._types import ByteRange, ListItem, Rejected + +__all__ = ["MemoryBackend"] + +R = TypeVar("R") + + +@dataclass(frozen=True, slots=True) +class _Value: + data: bytes + generation: int + last_modified: float + + +class _State(TypedDict): + values: dict[str, _Value] + next_generation: int + closed: bool + + +class MemoryBackend: + """Values held in a dict in this process. + + Thread-safe. Generations are a counter that increases with every write. A pickled + `MemoryBackend` carries a copy of its values; the copy and the original are independent. + """ + + def __init__(self) -> None: + self._values: dict[str, _Value] = {} + self._next_generation = 1 + self._lock = threading.Lock() + self._closed = False + + def __repr__(self) -> str: + return f"MemoryBackend(<{len(self._values)} keys>)" + + def __getstate__(self) -> _State: + with self._lock: + return { + "values": dict(self._values), + "next_generation": self._next_generation, + "closed": self._closed, + } + + def __setstate__(self, state: _State) -> None: + self._values = state["values"] + self._next_generation = state["next_generation"] + self._lock = threading.Lock() + self._closed = state["closed"] + + # --- Backend + + @property + def closed(self) -> bool: + return self._closed + + @property + def writable(self) -> bool: + return True + + def close(self) -> None: + self._closed = True + + # --- Readable + + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + value = self._values.get(path) + if value is None: + return Missing(time.time()) + start, end = resolve(byte_range, len(value.data)) + return Found(memoryview(value.data)[start:end], Generation(value.generation), time.time()) + + def get_if_changed( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: + value = self._values.get(path) + if value is not None and Generation(value.generation) == generation: + return Unchanged(generation, time.time()) + return self.get(path, byte_range) + + def get_ranges(self, path: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: + value = self._values.get(path) + if value is None: + return Missing(time.time()) + view = memoryview(value.data) + parts = tuple(view[slice(*resolve(r, len(value.data)))] for r in ranges) + return FoundRanges(parts, Generation(value.generation), time.time()) + + def head(self, path: str) -> ObjectMetadata | Missing: + value = self._values.get(path) + if value is None: + return Missing(time.time()) + return ObjectMetadata(len(value.data), Generation(value.generation), value.last_modified) + + def get_into(self, path: str, out: Buffer, byte_range: ByteRange | None) -> Copied | Missing: + view = byte_view(out, writable=True) + value = self._values.get(path) + if value is None: + return Missing(time.time()) + start, end = resolve(byte_range, len(value.data)) + n = end - start + if n > len(view): + raise ValueError(f"The output buffer holds {len(view)} bytes; the value has {n}.") + view[:n] = memoryview(value.data)[start:end] + return Copied(n, Generation(value.generation), time.time()) + + # --- Writable + + def _put( + self, path: str, value: memoryview, check: Callable[[Generation | None], R | None] + ) -> Written | R: + data = bytes(value) + with self._lock: + current = self._values.get(path) + refusal = check(None if current is None else Generation(current.generation)) + if refusal is not None: + return refusal + generation = self._next_generation + self._next_generation += 1 + self._values[path] = _Value(data, generation, time.time()) + return Written(Generation(generation)) + + def put(self, path: str, value: memoryview) -> Written: + return self._put(path, value, always) + + def put_if_absent(self, path: str, value: memoryview) -> Written | Rejected: + return self._put(path, value, if_absent) + + def put_if_match( + self, path: str, value: memoryview, generation: Generation + ) -> Written | Rejected | Missing: + return self._put(path, value, IfMatch(generation)) + + def _delete( + self, path: str, check: Callable[[Generation | None], R | None] + ) -> Deleted | Missing | R: + with self._lock: + current = self._values.get(path) + if current is None: + return Missing(time.time()) + generation = Generation(current.generation) + refusal = check(generation) + if refusal is not None: + return refusal + del self._values[path] + return Deleted(generation) + + def delete(self, path: str) -> Deleted | Missing: + return self._delete(path, always) + + def delete_if_match(self, path: str, generation: Generation) -> Deleted | Rejected | Missing: + return self._delete(path, IfMatch(generation)) + + def delete_dir(self, directory: str) -> None: + with self._lock: + for path in [p for p in self._values if strictly_within(p, directory)]: + del self._values[path] + + # --- Listable + + def list(self, directory: str) -> Iterator[str]: + with self._lock: + paths = list(self._values) + return (p for p in paths if strictly_within(p, directory)) + + def list_dir(self, directory: str) -> Iterator[ListItem]: + with self._lock: + paths = list(self._values) + return children(paths, directory) + + # --- async twins: memory never blocks, so they run inline + + async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + return self.get(path, byte_range) + + async def get_if_changed_async( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: + return self.get_if_changed(path, generation, byte_range) + + async def get_ranges_async( + self, path: str, ranges: Sequence[ByteRange] + ) -> FoundRanges | Missing: + return self.get_ranges(path, ranges) + + async def head_async(self, path: str) -> ObjectMetadata | Missing: + return self.head(path) + + async def get_into_async( + self, path: str, out: Buffer, byte_range: ByteRange | None + ) -> Copied | Missing: + return self.get_into(path, out, byte_range) + + async def put_async(self, path: str, value: memoryview) -> Written: + return self.put(path, value) + + async def put_if_absent_async(self, path: str, value: memoryview) -> Written | Rejected: + return self.put_if_absent(path, value) + + async def put_if_match_async( + self, path: str, value: memoryview, generation: Generation + ) -> Written | Rejected | Missing: + return self.put_if_match(path, value, generation) + + async def delete_async(self, path: str) -> Deleted | Missing: + return self.delete(path) + + async def delete_if_match_async( + self, path: str, generation: Generation + ) -> Deleted | Rejected | Missing: + return self.delete_if_match(path, generation) + + async def delete_dir_async(self, directory: str) -> None: + self.delete_dir(directory) + + async def list_async(self, directory: str) -> AsyncIterator[str]: + for path in self.list(directory): + yield path + + async def list_dir_async(self, directory: str) -> AsyncIterator[ListItem]: + for item in self.list_dir(directory): + yield item diff --git a/packages/zarr-storage/src/zarr_storage/backends/_obstore.py b/packages/zarr-storage/src/zarr_storage/backends/_obstore.py new file mode 100644 index 0000000000..0bdb8fbae4 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/backends/_obstore.py @@ -0,0 +1,850 @@ +"""A backend over an obstore store (S3, GCS, Azure, HTTP, local, memory). + +Each operation is written once as plain logic around obstore calls; the sync method and its async +twin differ only in which obstore function they await. The decisions (which put mode, whether an +error means "absent", whether a read is consistent) live in module functions both share. +""" + +from __future__ import annotations + +import asyncio +import contextlib +import errno +import os +import time +from pathlib import Path +from typing import TYPE_CHECKING, TypeVar, cast + +import obstore +from obstore.exceptions import ( + AlreadyExistsError, + GenericError, + NotModifiedError, + PreconditionError, +) +from obstore.store import HTTPStore, LocalStore + +from zarr_storage._batch import run_many_async +from zarr_storage._buffers import byte_view, copy_into +from zarr_storage._byte_range import resolve +from zarr_storage._errors import InvalidKeyError, KeyConflictError, UnsupportedOperationError +from zarr_storage._keys import check_key, relativize +from zarr_storage._types import ( + Deleted, + Found, + FoundRanges, + Generation, + Key, + Missing, + ObjectMetadata, + OffsetRequest, + Prefix, + RangeRequest, + Rejected, + SuffixRequest, + Unchanged, + Written, +) +from zarr_storage.backends._common import IfMatch, always, if_absent + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Callable, Iterable, Iterator, Sequence + + from obstore import GetOptions, GetResult, ListResult, ListStream, ObjectMeta, PutMode + from obstore.store import ObjectStore + from typing_extensions import Buffer + + from zarr_storage._types import ByteRange, Copied, ListItem + +__all__ = ["ObjectStoreBackend"] + +R = TypeVar("R") + +_EMPTY = memoryview(b"") + +_NOT_A_DIRECTORY = f"(os error {errno.ENOTDIR})" +"""How a filesystem store's refusal reads when a parent of the path is a file.""" + +_PUT_ATTEMPTS = 16 +"""How many times a write starts over when the store refused it for a state that then went away.""" + + +# ---------------------------------------------------------------------------- generations + + +def _generation(e_tag: str | None, version: str | None) -> Generation: + return Generation(("obstore", e_tag, version)) + + +def _metadata(meta: ObjectMeta) -> ObjectMetadata: + return ObjectMetadata( + meta["size"], + _generation(meta["e_tag"], meta["version"]), + meta["last_modified"].timestamp(), + ) + + +def _version_of(generation: object) -> tuple[str | None, str | None] | None: + """The `(e_tag, version)` in a generation this backend minted, or `None` for any other.""" + match generation: + case ("obstore", str() | None as e_tag, str() | None as version): + return e_tag, version + case _: + return None + + +def _if_match_mode(generation: Generation) -> PutMode | None: + """The put mode that writes only over `generation`; `None` for a generation this backend did + not mint, which no stored value has.""" + version = _version_of(generation) + if version is None: + return None + e_tag, object_version = version + if e_tag is None and object_version is None: + raise UnsupportedOperationError( + "The store reported neither an ETag nor a version, so writes can't be conditional." + ) + mode = {k: v for k, v in (("e_tag", e_tag), ("version", object_version)) if v is not None} + return cast("PutMode", mode) + + +def _refuse(current: ObjectMetadata | Missing) -> Rejected | Missing: + """The outcome of a write conditioned on a generation this backend did not mint.""" + return current if isinstance(current, Missing) else Rejected(current.generation) + + +def _found(result: GetResult, data: memoryview) -> Found: + return Found(data, _generation(result.meta["e_tag"], result.meta["version"]), time.time()) + + +def _if_none_match(generation: Generation) -> GetOptions: + """Read options that skip the read if the value has `generation`, where the store can tell.""" + version = _version_of(generation) + if version is None or version[0] is None: + return {} + return {"if_none_match": version[0]} + + +# ---------------------------------------------------------------------------- reads + + +def _get_options(byte_range: ByteRange | None) -> GetOptions: + match byte_range: + case None: + return {} + case RangeRequest(start=start, end=end): + return {"range": (start, end)} + case OffsetRequest(start=start): + return {"range": {"offset": start}} + case SuffixRequest(length=length): + return {"range": {"suffix": length}} + + +def _is_empty_request(byte_range: ByteRange | None) -> bool: + """Whether the range selects no bytes regardless of the value (object stores refuse those).""" + match byte_range: + case RangeRequest(start=start, end=end): + return start == end + case SuffixRequest(length=0): + return True + case _: + return False + + +def _empty_read_or_raise( + meta: ObjectMetadata | Missing, byte_range: ByteRange | None, error: BaseException | None +) -> Found | Missing: + """The result of a read the object store refused or that selects nothing. + + Object stores refuse a range that starts at the end of the value; the contract reads that as + empty. A range past the end raises `InvalidByteRangeError` (from `resolve`). Any other refusal + re-raises the store's error. + """ + if isinstance(meta, Missing): + return meta + start, end = resolve(byte_range, meta.size) + if start == end: + return Found(_EMPTY, meta.generation, time.time()) + if error is not None: + raise error + raise AssertionError("A non-empty range was treated as empty.") + + +def _pinned(generation: Generation) -> GetOptions | None: + """Read options that make a read fail unless the value still has `generation`. + + `None` when the store reported neither an ETag nor a version, so reads can't be pinned. + """ + version = _version_of(generation) + if version is None: + return None + e_tag, object_version = version + if e_tag is not None: + return {"if_match": e_tag} + if object_version is not None: + return {"version": object_version} + return None + + +def _slices(data: memoryview, ranges: Sequence[ByteRange]) -> tuple[memoryview, ...]: + return tuple(data[slice(*resolve(r, len(data)))] for r in ranges) + + +# ---------------------------------------------------------------------------- listing + +# obstore's `list` overloads mention Arrow types from an optional dependency, which pyright can't +# resolve without it; these wrappers pin the non-Arrow overload in one place. + + +def _list(store: ObjectStore, prefix: str | None) -> ListStream[Sequence[ObjectMeta]]: + return obstore.list(store, prefix, return_arrow=False) # pyright: ignore[reportUnknownMemberType] + + +def _list_with_delimiter( + store: ObjectStore, prefix: str | None +) -> ListResult[Sequence[ObjectMeta]]: + return obstore.list_with_delimiter(store, prefix, return_arrow=False) # pyright: ignore[reportUnknownMemberType] + + +async def _list_with_delimiter_async( + store: ObjectStore, prefix: str | None +) -> ListResult[Sequence[ObjectMeta]]: + return await obstore.list_with_delimiter_async(store, prefix, return_arrow=False) # pyright: ignore[reportUnknownMemberType] + + +def _prefix(directory: str) -> str | None: + return None if directory == "" else directory + + +def _valid_keys(paths: Iterable[str], directory: str) -> Iterator[str]: + """The paths that follow the key grammar and lie strictly under `directory`.""" + for path in paths: + if path == directory: + continue + try: + yield check_key(path) + except InvalidKeyError: + continue + + +def _children( + objects: Iterable[ObjectMeta], prefixes: Iterable[str], directory: str +) -> list[ListItem]: + items: list[ListItem] = [ + Key(relativize(path, directory)) + for path in _valid_keys((m["path"] for m in objects), directory) + ] + items.extend(Prefix(relativize(p.removesuffix("/"), directory)) for p in prefixes) + return items + + +def _ancestors(path: str) -> list[str]: + """`path`'s parent directories, nearest last: `a/b/c` gives `a`, `a/b`.""" + segments = path.split("/") + return ["/".join(segments[:i]) for i in range(1, len(segments))] + + +class ObjectStoreBackend: + """Keys stored in an obstore store. + + Generations are the store's ETag and version. `put_if_match` needs a store that supports it (S3, + GCS, Azure and obstore's `MemoryStore` do; obstore's `LocalStore` doesn't); otherwise it raises + `UnsupportedOperationError`. Object stores can't delete conditionally, so this backend has no + `delete_if_match`. `get_if_changed` sends `If-None-Match` with the generation's ETag. + + `get_ranges` reads the first range, then the others pinned (`If-Match`) to the generation and + size that first response reported. If the value changed in between, or the store reports + neither ETags nor versions, it reads the value once, whole, and slices it. + + obstore's `LocalStore` refuses paths that lie under a file; for it alone, such a refusal reads + as an absent key. Every other store's errors are raised. obstore's `LocalStore` has no locking, + so concurrent writes that create and remove the same directories can still surface its + errors; `LocalBackend` is the backend for that. + + Parameters + ---------- + store + The obstore store. + writable + Whether to accept writes. Defaults to `False` for `HTTPStore`, `True` otherwise. + concurrency + The number of requests an async batch (`get_many_async`, `put_many_async`, ...) keeps in + flight. Sync batches run one request at a time. + """ + + def __init__( + self, store: ObjectStore, *, writable: bool | None = None, concurrency: int = 32 + ) -> None: + if concurrency < 1: + raise ValueError(f"concurrency must be >= 1; got {concurrency}.") + self._store = store + self._concurrency = concurrency + self._writable = not isinstance(store, HTTPStore) if writable is None else writable + # Only a filesystem refuses a path that lies under a file; elsewhere a refusal is an error. + self._filesystem = isinstance(store, LocalStore) + self._closed = False + + @property + def store(self) -> ObjectStore: + """The obstore store.""" + return self._store + + def __repr__(self) -> str: + return f"ObjectStoreBackend({self._store!r})" + + # --- Backend + + @property + def closed(self) -> bool: + return self._closed + + @property + def writable(self) -> bool: + return self._writable + + def close(self) -> None: + self._closed = True + + # --- "absent" for filesystem-backed stores, which refuse paths under a file + + def _under_a_key(self, path: str) -> bool: + """Whether this is a filesystem store and a parent of `path` is a key.""" + return self._filesystem and any(self._head_raw(p) is not None for p in _ancestors(path)) + + async def _under_a_key_async(self, path: str) -> bool: + if not self._filesystem: + return False + for p in _ancestors(path): + if await self._head_raw_async(p) is not None: + return True + return False + + def _refused_under_a_key(self, error: GenericError, path: str) -> bool: + """Whether a filesystem store refused `path` because a parent of it is a key. + + `ENOTDIR` in the refusal says so as of the refusal. Looking at the parents afterwards can + race a concurrent delete of that parent, so that is only the fallback. + """ + return self._filesystem and (_NOT_A_DIRECTORY in str(error) or self._under_a_key(path)) + + async def _refused_under_a_key_async(self, error: GenericError, path: str) -> bool: + return self._filesystem and ( + _NOT_A_DIRECTORY in str(error) or await self._under_a_key_async(path) + ) + + def _blocked_by_a_key(self, directory: str) -> bool: + """Whether a filesystem store refused to list `directory` because it is, or lies under, a key.""" + return self._filesystem and ( + self._head_raw(directory) is not None or self._under_a_key(directory) + ) + + async def _blocked_by_a_key_async(self, directory: str) -> bool: + return self._filesystem and ( + await self._head_raw_async(directory) is not None + or await self._under_a_key_async(directory) + ) + + def _head_raw(self, path: str) -> ObjectMetadata | None: + try: + return _metadata(obstore.head(self._store, path)) + except (FileNotFoundError, GenericError): + return None + + async def _head_raw_async(self, path: str) -> ObjectMetadata | None: + try: + return _metadata(await obstore.head_async(self._store, path)) + except (FileNotFoundError, GenericError): + return None + + def _is_directory(self, path: str) -> bool: + """Whether `path` is a directory, on a filesystem store (even one holding only uploads).""" + if not isinstance(self._store, LocalStore) or self._store.prefix is None: + return False + return (Path(self._store.prefix) / path).is_dir() + + def _reclaim_empty_directory(self, path: str) -> bool: + """Remove a directory at `path` that holds no files, on a filesystem store. + + obstore's `LocalStore` leaves the directories of deleted keys behind, and one in the way of a + key would otherwise make the write fail. Only empty directories are removed, with `rmdir`, + deepest first; `rmdir` refuses a directory a writer has just added to, so this never removes + a key. Returns whether the directory is gone. + """ + if not isinstance(self._store, LocalStore) or self._store.prefix is None: + return False + return _remove_empty_directories(Path(self._store.prefix) / path) + + def _conflicts(self, path: str) -> bool: + """Whether this is a filesystem store and `path` collides with a key or a directory. + + Answers `False` if the check itself fails, so the caller raises the write's own error. + """ + if not self._filesystem: + return False + try: + if self._under_a_key(path): + return True + listed = _list_with_delimiter(self._store, path) + except Exception: # noqa: BLE001 -- only a diagnosis; the write's error is raised instead + return False + return len(listed["objects"]) > 0 or len(listed["common_prefixes"]) > 0 + + async def _conflicts_async(self, path: str) -> bool: + if not self._filesystem: + return False + try: + if await self._under_a_key_async(path): + return True + listed = await _list_with_delimiter_async(self._store, path) + except Exception: # noqa: BLE001 -- only a diagnosis; the write's error is raised instead + return False + return len(listed["objects"]) > 0 or len(listed["common_prefixes"]) > 0 + + # --- Readable + + def head(self, path: str) -> ObjectMetadata | Missing: + try: + return _metadata(obstore.head(self._store, path)) + except FileNotFoundError: + return Missing(time.time()) + except GenericError as e: + if self._refused_under_a_key(e, path): + return Missing(time.time()) + raise + + async def head_async(self, path: str) -> ObjectMetadata | Missing: + try: + return _metadata(await obstore.head_async(self._store, path)) + except FileNotFoundError: + return Missing(time.time()) + except GenericError as e: + if await self._refused_under_a_key_async(e, path): + return Missing(time.time()) + raise + + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + return self._get(path, byte_range, {}) + + async def get_async(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + return await self._get_async(path, byte_range, {}) + + def get_into(self, path: str, out: Buffer, byte_range: ByteRange | None) -> Copied | Missing: + """A read, copied into `out`: obstore can't read into a caller's buffer.""" + view = byte_view(out, writable=True) + return copy_into(self.get(path, byte_range), view) + + async def get_into_async( + self, path: str, out: Buffer, byte_range: ByteRange | None + ) -> Copied | Missing: + view = byte_view(out, writable=True) + return copy_into(await self.get_async(path, byte_range), view) + + def get_if_changed( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: + try: + result = self._get(path, byte_range, _if_none_match(generation)) + except NotModifiedError: + return Unchanged(generation, time.time()) + return _unless_unchanged(result, generation) + + async def get_if_changed_async( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: + try: + result = await self._get_async(path, byte_range, _if_none_match(generation)) + except NotModifiedError: + return Unchanged(generation, time.time()) + return _unless_unchanged(result, generation) + + def _get(self, path: str, byte_range: ByteRange | None, options: GetOptions) -> Found | Missing: + """A read with extra `options`; raises `NotModifiedError` if they include a matching + `if_none_match`.""" + if _is_empty_request(byte_range): + return _empty_read_or_raise(self.head(path), byte_range, None) + try: + result = obstore.get(self._store, path, options={**options, **_get_options(byte_range)}) + except FileNotFoundError: + return Missing(time.time()) + except GenericError as e: + return _empty_read_or_raise(self.head(path), byte_range, e) + return _found(result, memoryview(result.bytes())) + + async def _get_async( + self, path: str, byte_range: ByteRange | None, options: GetOptions + ) -> Found | Missing: + if _is_empty_request(byte_range): + return _empty_read_or_raise(await self.head_async(path), byte_range, None) + try: + result = await obstore.get_async( + self._store, path, options={**options, **_get_options(byte_range)} + ) + except FileNotFoundError: + return Missing(time.time()) + except GenericError as e: + return _empty_read_or_raise(await self.head_async(path), byte_range, e) + return _found(result, memoryview(await result.bytes_async())) + + def get_ranges(self, path: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: + if len(ranges) == 0 or _is_empty_request(ranges[0]): + return self._read_whole_and_slice(path, ranges) + try: + first = obstore.get(self._store, path, options=_get_options(ranges[0])) + except FileNotFoundError: + return Missing(time.time()) + except GenericError: + # The first range starts at the end of the value; one whole read answers every range. + return self._read_whole_and_slice(path, ranges) + meta = first.meta + generation = _generation(meta["e_tag"], meta["version"]) + pinned = _pinned(generation) + if pinned is None: + return self._read_whole_and_slice(path, ranges) + try: + rest = [self._read_pinned(path, pinned, *resolve(r, meta["size"])) for r in ranges[1:]] + except (PreconditionError, FileNotFoundError): + # The value changed after the first read; one whole read is consistent by itself. + return self._read_whole_and_slice(path, ranges) + return FoundRanges((memoryview(first.bytes()), *rest), generation, time.time()) + + def _read_pinned(self, path: str, pinned: GetOptions, start: int, end: int) -> memoryview: + if start == end: + return _EMPTY + result = obstore.get(self._store, path, options={**pinned, "range": (start, end)}) + _check_pinned(result.meta, pinned) + return memoryview(result.bytes()) + + def _read_whole_and_slice( + self, path: str, ranges: Sequence[ByteRange] + ) -> FoundRanges | Missing: + try: + result = obstore.get(self._store, path) + except FileNotFoundError: + return Missing(time.time()) + except GenericError as e: + if self._refused_under_a_key(e, path): + return Missing(time.time()) + raise + generation = _generation(result.meta["e_tag"], result.meta["version"]) + return FoundRanges(_slices(memoryview(result.bytes()), ranges), generation, time.time()) + + async def get_ranges_async( + self, path: str, ranges: Sequence[ByteRange] + ) -> FoundRanges | Missing: + if len(ranges) == 0 or _is_empty_request(ranges[0]): + return await self._read_whole_and_slice_async(path, ranges) + try: + first = await obstore.get_async(self._store, path, options=_get_options(ranges[0])) + except FileNotFoundError: + return Missing(time.time()) + except GenericError: + return await self._read_whole_and_slice_async(path, ranges) + meta = first.meta + generation = _generation(meta["e_tag"], meta["version"]) + pinned = _pinned(generation) + if pinned is None: + return await self._read_whole_and_slice_async(path, ranges) + try: + rest = await asyncio.gather( + *( + _read_pinned_async(self._store, path, pinned, *resolve(r, meta["size"])) + for r in ranges[1:] + ) + ) + except (PreconditionError, FileNotFoundError): + return await self._read_whole_and_slice_async(path, ranges) + return FoundRanges((memoryview(await first.bytes_async()), *rest), generation, time.time()) + + async def _read_whole_and_slice_async( + self, path: str, ranges: Sequence[ByteRange] + ) -> FoundRanges | Missing: + try: + result = await obstore.get_async(self._store, path) + except FileNotFoundError: + return Missing(time.time()) + except GenericError as e: + if await self._refused_under_a_key_async(e, path): + return Missing(time.time()) + raise + generation = _generation(result.meta["e_tag"], result.meta["version"]) + data = memoryview(await result.bytes_async()) + return FoundRanges(_slices(data, ranges), generation, time.time()) + + # --- Writable + + def put(self, path: str, value: memoryview) -> Written: + return self._put(path, value, None, always) + + async def put_async(self, path: str, value: memoryview) -> Written: + return await self._put_async(path, value, None, always) + + def put_if_absent(self, path: str, value: memoryview) -> Written | Rejected: + return self._put(path, value, "create", if_absent) + + async def put_if_absent_async(self, path: str, value: memoryview) -> Written | Rejected: + return await self._put_async(path, value, "create", if_absent) + + def put_if_match( + self, path: str, value: memoryview, generation: Generation + ) -> Written | Rejected | Missing: + mode = _if_match_mode(generation) + if mode is None: + return _refuse(self.head(path)) + return self._put(path, value, mode, IfMatch(generation)) + + async def put_if_match_async( + self, path: str, value: memoryview, generation: Generation + ) -> Written | Rejected | Missing: + mode = _if_match_mode(generation) + if mode is None: + return _refuse(await self.head_async(path)) + return await self._put_async(path, value, mode, IfMatch(generation)) + + def _put( + self, + path: str, + value: memoryview, + mode: PutMode | None, + check: Callable[[Generation | None], R | None], + ) -> Written | R: + """Write with `mode`. When the store refuses, `check` the key's current generation: its + refusal is the outcome, and if it has none, the state that made the store refuse went away + and the write starts over.""" + for _ in range(_PUT_ATTEMPTS): + try: + # obstore keeps a reference to the buffer it's given; copy so later changes to the + # caller's buffer don't reach the stored value. + result = obstore.put(self._store, path, bytes(value), mode=mode) + except (AlreadyExistsError, PreconditionError): + current = self.head(path) + if isinstance(current, Missing): + if self._reclaim_empty_directory(path): + continue + if self._conflicts(path): + raise _conflict(path, self._store) from None + refusal = check(None if isinstance(current, Missing) else current.generation) + if refusal is not None: + return refusal + continue + except NotImplementedError as e: + raise UnsupportedOperationError( + f"{self._store!r} does not support this conditional write." + ) from e + except GenericError as e: + if self._reclaim_empty_directory(path): + continue + if self._conflicts(path) or self._is_directory(path): + raise _conflict(path, self._store) from e + raise + return Written(_generation(result["e_tag"], result["version"])) + raise OSError(f"Gave up writing {path!r}: the store kept refusing it.") + + async def _put_async( + self, + path: str, + value: memoryview, + mode: PutMode | None, + check: Callable[[Generation | None], R | None], + ) -> Written | R: + for _ in range(_PUT_ATTEMPTS): + try: + result = await obstore.put_async(self._store, path, bytes(value), mode=mode) + except (AlreadyExistsError, PreconditionError): + current = await self.head_async(path) + if isinstance(current, Missing): + if await asyncio.to_thread(self._reclaim_empty_directory, path): + continue + if await self._conflicts_async(path): + raise _conflict(path, self._store) from None + refusal = check(None if isinstance(current, Missing) else current.generation) + if refusal is not None: + return refusal + continue + except NotImplementedError as e: + raise UnsupportedOperationError( + f"{self._store!r} does not support this conditional write." + ) from e + except GenericError as e: + if await asyncio.to_thread(self._reclaim_empty_directory, path): + continue + if await self._conflicts_async(path) or await asyncio.to_thread( + self._is_directory, path + ): + raise _conflict(path, self._store) from e + raise + return Written(_generation(result["e_tag"], result["version"])) + raise OSError(f"Gave up writing {path!r}: the store kept refusing it.") + + def delete(self, path: str) -> Deleted | Missing: + meta = self.head(path) + if isinstance(meta, Missing): + return meta + try: + obstore.delete(self._store, path) + except FileNotFoundError: + return Missing(time.time()) + return Deleted(meta.generation) + + async def delete_async(self, path: str) -> Deleted | Missing: + meta = await self.head_async(path) + if isinstance(meta, Missing): + return meta + try: + await obstore.delete_async(self._store, path) + except FileNotFoundError: + return Missing(time.time()) + return Deleted(meta.generation) + + def delete_dir(self, directory: str) -> None: + paths = list(self._paths(directory)) + try: + if paths: + obstore.delete(self._store, paths) + except FileNotFoundError: + # A concurrent delete removed some first; delete the rest one at a time. + for path in paths: + with contextlib.suppress(FileNotFoundError): + obstore.delete(self._store, path) + + async def delete_dir_async(self, directory: str) -> None: + paths = [path async for path in self._paths_async(directory)] + try: + if paths: + await obstore.delete_async(self._store, paths) + except FileNotFoundError: + for path in paths: + with contextlib.suppress(FileNotFoundError): + await obstore.delete_async(self._store, path) + + # --- async batches: the async operations, `concurrency` at once + + def get_many_async( + self, requests: Sequence[tuple[str, ByteRange | None]] + ) -> AsyncIterator[tuple[int, Found | Missing]]: + return run_many_async(self.get_async, requests, self._concurrency) + + def get_into_many_async( + self, requests: Sequence[tuple[str, Buffer, ByteRange | None]] + ) -> AsyncIterator[tuple[int, Copied | Missing]]: + return run_many_async(self.get_into_async, requests, self._concurrency) + + def put_many_async( + self, requests: Sequence[tuple[str, memoryview]] + ) -> AsyncIterator[tuple[int, Written]]: + return run_many_async(self.put_async, requests, self._concurrency) + + def delete_many_async( + self, paths: Sequence[str] + ) -> AsyncIterator[tuple[int, Deleted | Missing]]: + return run_many_async(self.delete_async, [(path,) for path in paths], self._concurrency) + + # --- Listable + + def _paths(self, directory: str) -> Iterator[str]: + """Every stored path under `directory`, whether or not it follows the key grammar.""" + try: + for chunk in _list(self._store, _prefix(directory)): + yield from (meta["path"] for meta in chunk if meta["path"] != directory) + except GenericError: + # A filesystem store can't walk a path that is, or lies under, a file; such a + # directory has no children. + if not self._blocked_by_a_key(directory): + raise + + async def _paths_async(self, directory: str) -> AsyncIterator[str]: + try: + async for chunk in _list(self._store, _prefix(directory)): + for meta in chunk: + if meta["path"] != directory: + yield meta["path"] + except GenericError: + if not await self._blocked_by_a_key_async(directory): + raise + + def list(self, directory: str) -> Iterator[str]: + return _valid_keys(self._paths(directory), directory) + + async def list_async(self, directory: str) -> AsyncIterator[str]: + async for path in self._paths_async(directory): + for key in _valid_keys([path], directory): + yield key + + def list_dir(self, directory: str) -> Iterator[ListItem]: + try: + result = _list_with_delimiter(self._store, _prefix(directory)) + except GenericError: + if not self._blocked_by_a_key(directory): + raise + return iter(()) + return iter(_children(result["objects"], result["common_prefixes"], directory)) + + async def list_dir_async(self, directory: str) -> AsyncIterator[ListItem]: + try: + result = await _list_with_delimiter_async(self._store, _prefix(directory)) + except GenericError: + if not await self._blocked_by_a_key_async(directory): + raise + return + for item in _children(result["objects"], result["common_prefixes"], directory): + yield item + + +async def _read_pinned_async( + store: ObjectStore, path: str, pinned: GetOptions, start: int, end: int +) -> memoryview: + if start == end: + return _EMPTY + result = await obstore.get_async(store, path, options={**pinned, "range": (start, end)}) + _check_pinned(result.meta, pinned) + return memoryview(await result.bytes_async()) + + +def _check_pinned(meta: ObjectMeta, pinned: GetOptions) -> None: + """Treat a response from another generation as a failed precondition. + + A store that ignored `If-Match` would otherwise mix parts of two values. + """ + if meta["e_tag"] != pinned.get("if_match", meta["e_tag"]) or meta["version"] != pinned.get( + "version", meta["version"] + ): + raise PreconditionError("The value changed between reads.") + + +def _remove_empty_directories(target: Path) -> bool: + """Remove `target` and the directories under it if none holds a file; whether it is gone.""" + if not target.is_dir(): + return False + try: + for directory, _, files in os.walk(target, topdown=False): + if files: + return False + os.rmdir(directory) + except FileNotFoundError: + return not target.exists() + except NotADirectoryError: + return False # A key was written in the directory's place meanwhile. + except OSError as e: + if e.errno == errno.ENOTEMPTY: + return False + raise + return True + + +def _unless_unchanged( + result: Found | Missing, generation: Generation +) -> Found | Missing | Unchanged: + """`Unchanged` for a value read whole although it has `generation`. + + That happens when the store can't skip the read (no ETag to send, or it ignores + `If-None-Match`); the bytes are discarded so the outcome doesn't depend on the store. + """ + if isinstance(result, Found) and result.generation == generation: + return Unchanged(generation, result.read_at) + return result + + +def _conflict(path: str, store: ObjectStore) -> KeyConflictError: + return KeyConflictError( + f"{path!r} collides with a directory or key of the same name, which {store!r} can't hold " + "side by side." + ) diff --git a/packages/zarr-storage/src/zarr_storage/py.typed b/packages/zarr-storage/src/zarr_storage/py.typed new file mode 100644 index 0000000000..e69de29bb2 diff --git a/packages/zarr-storage/src/zarr_storage/testing/__init__.py b/packages/zarr-storage/src/zarr_storage/testing/__init__.py new file mode 100644 index 0000000000..1e85817714 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/testing/__init__.py @@ -0,0 +1,38 @@ +"""The conformance suite every backend runs. + +A backend's test class subclasses `StoreSuite` and overrides `make_store` to return an empty, +writable `Store` over a fresh backend; it adds the mixin for each optional capability the backend has +(`ConditionalPutSuite`, `ConditionalDeleteSuite`, `ConditionalRaceSuite`, `KeyAndDirectorySuite`, +`PickleSuite`, `UniqueGenerationsSuite`) or the one +that checks it refuses it cleanly (`NoConditionalPutSuite`, +`KeyDirectoryConflictSuite`). Every test runs through the sync methods and again through the async +methods. Install the `testing` extra for pytest and Hypothesis. +""" + +from zarr_storage.testing._api import AsyncAPI, FullBackend, StoreAPI +from zarr_storage.testing._suite import ( + ConditionalDeleteSuite, + ConditionalPutSuite, + ConditionalRaceSuite, + KeyAndDirectorySuite, + KeyDirectoryConflictSuite, + NoConditionalPutSuite, + PickleSuite, + StoreSuite, + UniqueGenerationsSuite, +) + +__all__ = [ + "AsyncAPI", + "ConditionalDeleteSuite", + "ConditionalPutSuite", + "ConditionalRaceSuite", + "FullBackend", + "KeyAndDirectorySuite", + "KeyDirectoryConflictSuite", + "NoConditionalPutSuite", + "PickleSuite", + "StoreAPI", + "StoreSuite", + "UniqueGenerationsSuite", +] diff --git a/packages/zarr-storage/src/zarr_storage/testing/_api.py b/packages/zarr-storage/src/zarr_storage/testing/_api.py new file mode 100644 index 0000000000..49ca4415e6 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/testing/_api.py @@ -0,0 +1,268 @@ +"""The store surface the suite tests, and an adapter that drives it through `AsyncStore`. + +Every suite test runs twice: once against `Store`, once against `AsyncAPI`, which gives the same +sync-looking surface but performs every call through the `AsyncStore` method of the same name on +one event loop. The sync and async surfaces therefore meet one contract, checked by one set of tests. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Protocol, Self, TypeVar, cast + +from typing_extensions import Buffer + +from zarr_storage._protocols import ( + Listable, + ListableAsync, + Readable, + ReadableAsync, + ReadableInto, + ReadableIntoAsync, + Writable, + WritableAsync, + WritableIfMatch, + WritableIfMatchAsync, +) +from zarr_storage._store import AsyncStore + +if TYPE_CHECKING: + import asyncio + from collections.abc import AsyncIterator, Iterable, Iterator, Sequence + from types import TracebackType + + from zarr_storage._protocols import DeletableIfMatch, DeletableIfMatchAsync + from zarr_storage._store import Store + from zarr_storage._types import ( + ByteRange, + Copied, + Deleted, + Found, + FoundRanges, + Generation, + ListItem, + Missing, + ObjectMetadata, + Rejected, + Unchanged, + Written, + ) + +__all__ = ["AsyncAPI", "FullBackend", "StoreAPI", "async_api", "delete_if_match", "unwrap"] + +T = TypeVar("T") +Q = TypeVar("Q", bound="str | tuple[str, ByteRange | None]") + + +class FullBackend( + Readable, + ReadableInto[Buffer], + Writable, + WritableIfMatch, + Listable, + ReadableAsync, + ReadableIntoAsync[Buffer], + WritableAsync, + WritableIfMatchAsync, + ListableAsync, + Protocol, +): + """A backend with every capability the suite needs of all backends, sync and async. + + `DeletableIfMatch` is left out, since object stores lack it; `delete_if_match` reaches it. + """ + + +class StoreAPI(Protocol): + """The operations the suite calls; `Store` and `AsyncAPI` both provide them.""" + + @property + def root(self) -> str: ... + @property + def read_only(self) -> bool: ... + @property + def closed(self) -> bool: ... + + def get(self, key: str, *, byte_range: ByteRange | None = None) -> Found | Missing: ... + def get_if_changed( + self, key: str, generation: Generation, *, byte_range: ByteRange | None = None + ) -> Found | Missing | Unchanged: ... + def get_ranges(self, key: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: ... + def get_into( + self, key: str, out: Buffer, *, byte_range: ByteRange | None = None + ) -> Copied | Missing: ... + def head(self, key: str) -> ObjectMetadata | Missing: ... + def exists(self, key: str) -> bool: ... + def put(self, key: str, value: Buffer) -> Written: ... + def put_if_absent(self, key: str, value: Buffer) -> Written | Rejected: ... + def put_if_match( + self, key: str, value: Buffer, generation: Generation + ) -> Written | Rejected | Missing: ... + def delete(self, key: str) -> Deleted | Missing: ... + def delete_dir(self, directory: str = "") -> None: ... + def get_many(self, requests: Iterable[Q]) -> Iterator[tuple[Q, Found | Missing]]: ... + def get_into_many( + self, requests: Iterable[tuple[str, Buffer] | tuple[str, Buffer, ByteRange | None]] + ) -> Iterator[ + tuple[tuple[str, Buffer] | tuple[str, Buffer, ByteRange | None], Copied | Missing] + ]: ... + def put_many(self, items: Iterable[tuple[str, Buffer]]) -> list[tuple[str, Written]]: ... + def delete_many(self, keys: Iterable[str]) -> list[tuple[str, Deleted | Missing]]: ... + def list(self, directory: str = "") -> Iterator[str]: ... + def list_dir(self, directory: str = "") -> Iterator[ListItem]: ... + def __truediv__(self, directory: str) -> Self: ... + def with_read_only(self, read_only: bool = True) -> Self: ... + def close(self) -> None: ... + def __enter__(self) -> Self: ... + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: ... + + +def unwrap(store: StoreAPI) -> Store[FullBackend]: + """The `Store` behind a `StoreAPI`: a sync view of an `AsyncAPI`'s store.""" + if isinstance(store, AsyncAPI): + return store.store.as_sync() + return store # pyright: ignore[reportReturnType] + + +def delete_if_match( + store: StoreAPI, key: str, generation: Generation +) -> Deleted | Rejected | Missing: + """`delete_if_match` through `store`'s surface, for a backend that has it.""" + if isinstance(store, AsyncAPI): + return store.delete_if_match(key, generation) + return cast("Store[DeletableIfMatch]", store).delete_if_match(key, generation) + + +async def _next(items: AsyncIterator[T]) -> T: + return await anext(items) + + +def async_api(store: Store[FullBackend], runner: asyncio.Runner) -> AsyncAPI: + """An `AsyncAPI` over an `AsyncStore` that owns `store`'s backend, at its root.""" + return AsyncAPI(AsyncStore(store.backend, root=store.root, read_only=store.read_only), runner) + + +class AsyncAPI: + """`Store`'s surface, performed through an `AsyncStore` on one event loop. + + Each method runs the `AsyncStore` method of the same name to completion; listings and lazy + batches are taken one item per run. + """ + + def __init__(self, store: AsyncStore[FullBackend], runner: asyncio.Runner) -> None: + self._store = store + self._runner = runner + + @property + def store(self) -> AsyncStore[FullBackend]: + return self._store + + @property + def root(self) -> str: + return self._store.root + + @property + def read_only(self) -> bool: + return self._store.read_only + + @property + def closed(self) -> bool: + return self._store.closed + + def get(self, key: str, *, byte_range: ByteRange | None = None) -> Found | Missing: + return self._runner.run(self._store.get(key, byte_range=byte_range)) + + def get_if_changed( + self, key: str, generation: Generation, *, byte_range: ByteRange | None = None + ) -> Found | Missing | Unchanged: + return self._runner.run(self._store.get_if_changed(key, generation, byte_range=byte_range)) + + def get_ranges(self, key: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: + return self._runner.run(self._store.get_ranges(key, ranges)) + + def get_into( + self, key: str, out: Buffer, *, byte_range: ByteRange | None = None + ) -> Copied | Missing: + return self._runner.run(self._store.get_into(key, out, byte_range=byte_range)) + + def head(self, key: str) -> ObjectMetadata | Missing: + return self._runner.run(self._store.head(key)) + + def exists(self, key: str) -> bool: + return self._runner.run(self._store.exists(key)) + + def put(self, key: str, value: Buffer) -> Written: + return self._runner.run(self._store.put(key, value)) + + def put_if_absent(self, key: str, value: Buffer) -> Written | Rejected: + return self._runner.run(self._store.put_if_absent(key, value)) + + def put_if_match( + self, key: str, value: Buffer, generation: Generation + ) -> Written | Rejected | Missing: + return self._runner.run(self._store.put_if_match(key, value, generation)) + + def delete(self, key: str) -> Deleted | Missing: + return self._runner.run(self._store.delete(key)) + + def delete_if_match(self, key: str, generation: Generation) -> Deleted | Rejected | Missing: + store = cast("AsyncStore[DeletableIfMatchAsync]", self._store) + return self._runner.run(store.delete_if_match(key, generation)) + + def delete_dir(self, directory: str = "") -> None: + self._runner.run(self._store.delete_dir(directory)) + + def get_many(self, requests: Iterable[Q]) -> Iterator[tuple[Q, Found | Missing]]: + # `get_many` checks its requests when called; the results come one at a time. + return self._iterate(self._store.get_many(requests)) + + def get_into_many( + self, requests: Iterable[tuple[str, Buffer] | tuple[str, Buffer, ByteRange | None]] + ) -> Iterator[ + tuple[tuple[str, Buffer] | tuple[str, Buffer, ByteRange | None], Copied | Missing] + ]: + return self._iterate(self._store.get_into_many(requests)) + + def put_many(self, items: Iterable[tuple[str, Buffer]]) -> list[tuple[str, Written]]: + return self._runner.run(self._store.put_many(items)) + + def delete_many(self, keys: Iterable[str]) -> list[tuple[str, Deleted | Missing]]: + return self._runner.run(self._store.delete_many(keys)) + + def list(self, directory: str = "") -> Iterator[str]: + # `list` checks its argument when called; the items come one at a time. + return self._iterate(self._store.list(directory)) + + def list_dir(self, directory: str = "") -> Iterator[ListItem]: + return self._iterate(self._store.list_dir(directory)) + + def _iterate(self, items: AsyncIterator[T]) -> Iterator[T]: + while True: + try: + yield self._runner.run(_next(items)) + except StopAsyncIteration: + return + + def __truediv__(self, directory: str) -> Self: + return type(self)(self._store / directory, self._runner) + + def with_read_only(self, read_only: bool = True) -> Self: + return type(self)(self._store.with_read_only(read_only), self._runner) + + def close(self) -> None: + self._runner.run(self._store.aclose()) + + def __enter__(self) -> Self: + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self.close() diff --git a/packages/zarr-storage/src/zarr_storage/testing/_stateful.py b/packages/zarr-storage/src/zarr_storage/testing/_stateful.py new file mode 100644 index 0000000000..627e880e43 --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/testing/_stateful.py @@ -0,0 +1,357 @@ +"""A Hypothesis state machine that checks a store against a dict model. + +Each step reads or writes through a view at a random root, so roots, views and the key grammar are +exercised together with the operations. Half the time a step targets a key the model holds, so +deletes, conditional writes and batches meet present keys, not just absent ones. Generations that a write or +delete retired are kept and replayed as preconditions: a backend must never hand one out again. +After every step the machine checks that listing, reading and metadata agree with the model. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from hypothesis import strategies as st +from hypothesis.stateful import RuleBasedStateMachine, initialize, invariant, precondition, rule + +from zarr_storage._errors import BatchError, KeyConflictError +from zarr_storage._keys import join +from zarr_storage._types import ( + Deleted, + Found, + Generation, + Key, + Missing, + ObjectMetadata, + Prefix, + Rejected, + Unchanged, + Written, +) +from zarr_storage.testing._api import delete_if_match + +if TYPE_CHECKING: + from collections.abc import Callable + + from zarr_storage._types import ListItem + from zarr_storage.testing._api import StoreAPI + +__all__ = ["Capabilities", "StoreMachine"] + +SEGMENTS = ["a", "b", "ab"] +ROOT_CHOICES = ["", "a", "a/b", "b"] +KEYS = st.lists(st.sampled_from(SEGMENTS), min_size=1, max_size=3).map("/".join) +ROOTS = st.sampled_from(ROOT_CHOICES) +DIRECTORIES = st.lists(st.sampled_from(SEGMENTS), max_size=2).map("/".join) +VALUES = st.binary(max_size=6) + + +@dataclass(frozen=True) +class Capabilities: + """What the backend under test supports; the machine expects refusals for the rest.""" + + conditional_put: bool + conditional_delete: bool + holds_key_and_directory: bool + leaves_empty_directories: bool + """Whether `list_dir` may report a `Prefix` for a directory a delete left empty.""" + unique_generations: bool + """Whether the backend never hands out a generation twice, even after a delete.""" + + +def _under(path: str, directory: str) -> bool: + return directory == "" or path.startswith(f"{directory}/") + + +def _nested(a: str, b: str) -> bool: + """Whether one of two distinct keys lies under the other.""" + return _under(a, b) or _under(b, a) + + +def _relative(path: str, root: str) -> str: + return path.removeprefix(f"{root}/") if root else path + + +class StoreMachine(RuleBasedStateMachine): + """Operations on one store, compared with a dict from key to value and generation.""" + + def __init__(self, make_store: Callable[[], StoreAPI], capabilities: Capabilities) -> None: + super().__init__() + self._make_store = make_store + self._capabilities = capabilities + self.values: dict[str, bytes] = {} + self.generations: dict[str, Generation] = {} + self.retired: list[Generation] = [] + + @initialize() + def open_store(self) -> None: + self.store = self._make_store() + + # --- model helpers + + def _target(self, data: st.DataObject) -> tuple[str, str]: + """A `(root, key)` to operate on: half the time a key the model holds.""" + if self.values and data.draw(st.booleans(), label="existing key"): + path = data.draw(st.sampled_from(sorted(self.values)), label="path") + roots = [r for r in ROOT_CHOICES if r == "" or path.startswith(f"{r}/")] + root = data.draw(st.sampled_from(roots), label="root") + return root, _relative(path, root) + return data.draw(ROOTS, label="root"), data.draw(KEYS, label="key") + + def _conflicts(self, path: str) -> bool: + """Whether writing `path` collides with a stored key, on a backend that can't hold both.""" + if self._capabilities.holds_key_and_directory: + return False + return any(_under(path, k) or _under(k, path) for k in self.values if k != path) + + def _expect_conflict(self, root: str, key: str, value: bytes) -> None: + """A write to a path that collides with a stored key raises, whatever its preconditions.""" + try: + (self.store / root).put_if_match(key, value, Generation(object())) + except KeyConflictError: + return + raise AssertionError(f"put({join(root, key)!r}) should have raised KeyConflictError") + + def _refusal(self, path: str) -> Rejected | Missing: + """What a conditional write or delete whose precondition fails returns for `path`.""" + current = self.generations.get(path) + return Missing() if current is None else Rejected(current) + + def _check_refused(self, result: object, path: str) -> None: + expected = self._refusal(path) + if isinstance(expected, Missing): + assert isinstance(result, Missing), result + else: + assert result == expected + + def _batch_keys(self, data: st.DataObject, root: str) -> list[str]: + """Distinct keys under `root`, each half the time one the model holds.""" + held = [_relative(p, root) for p in sorted(self.values) if _under(p, root) and p != root] + keys = st.one_of(KEYS, st.sampled_from(held)) if held else KEYS + return data.draw(st.lists(keys, max_size=4, unique=True), label="keys") + + def _record(self, path: str, value: bytes, generation: Generation) -> None: + if self._capabilities.unique_generations: + assert generation not in self.retired, "a retired generation was handed out again" + assert generation != self.generations.get(path) + if path in self.generations: + self.retired.append(self.generations[path]) + self.values[path] = value + self.generations[path] = generation + + def _forget(self, path: str) -> None: + self.retired.append(self.generations.pop(path)) + del self.values[path] + + # --- writes + + @rule(data=st.data(), value=VALUES) + def put(self, data: st.DataObject, value: bytes) -> None: + root, key = self._target(data) + path = join(root, key) + if self._conflicts(path): + try: + (self.store / root).put(key, value) + except KeyConflictError: + return + raise AssertionError(f"put({path!r}) should have raised KeyConflictError") + result = (self.store / root).put(key, value) + self._record(path, value, result.generation) + + @rule(data=st.data(), value=VALUES) + def put_if_absent(self, data: st.DataObject, value: bytes) -> None: + root, key = self._target(data) + path = join(root, key) + if self._conflicts(path): + try: + (self.store / root).put_if_absent(key, value) + except KeyConflictError: + return + raise AssertionError(f"put_if_absent({path!r}) should have raised KeyConflictError") + result = (self.store / root).put_if_absent(key, value) + if path in self.values: + assert result == Rejected(self.generations[path]) + else: + assert isinstance(result, Written) + self._record(path, value, result.generation) + + @precondition(lambda self: self._capabilities.conditional_put) + @rule(data=st.data(), value=VALUES, stale=st.booleans()) + def put_if_match(self, data: st.DataObject, value: bytes, stale: bool) -> None: + root, key = self._target(data) + path = join(root, key) + if self._conflicts(path): + self._expect_conflict(root, key, value) + return + current = self.generations.get(path) + expected = Generation(object()) if stale or current is None else current + result = (self.store / root).put_if_match(key, value, expected) + if not stale and current is not None: + assert isinstance(result, Written) + self._record(path, value, result.generation) + else: + self._check_refused(result, path) + + @precondition( + lambda self: ( + self._capabilities.conditional_put + and self._capabilities.unique_generations + and len(self.retired) > 0 + ) + ) + @rule(data=st.data(), value=VALUES) + def put_if_match_retired(self, data: st.DataObject, value: bytes) -> None: + """A generation a write or delete retired never matches again.""" + root, key = self._target(data) + if self._conflicts(join(root, key)): + self._expect_conflict(root, key, value) + return + retired = data.draw(st.sampled_from(self.retired), label="retired generation") + result = (self.store / root).put_if_match(key, value, retired) + self._check_refused(result, join(root, key)) + + @rule(data=st.data()) + def put_many(self, data: st.DataObject) -> None: + root = data.draw(ROOTS, label="root") + keys = self._batch_keys(data, root) + if not self._capabilities.holds_key_and_directory: + # Which of two nested keys in one batch lands is a race between threads, and a model + # that depends on it breaks Hypothesis's replays; `KeyDirectoryConflictSuite` checks it. + keys = [k for i, k in enumerate(keys) if not any(_nested(k, o) for o in keys[:i])] + items = [(key, data.draw(VALUES, label="value")) for key in keys] + paths = [join(root, key) for key in keys] + may_conflict = any(self._conflicts(path) for path in paths) + failure: BatchError | None = None + try: + written = (self.store / root).put_many(items) + except BatchError as error: + failure = error + written = [(str(key), result) for key, result in error.completed] + if failure is None: + assert [key for key, _ in written] == keys + else: + # Only a collision can fail a write here; the writes that landed are kept. + assert may_conflict, failure + assert all(isinstance(e, KeyConflictError) for e in failure.exceptions), failure + assert len(failure.completed) + len(failure.remaining) == len(items) + values = dict(items) + for key, result in written: + assert isinstance(result, Written) + self._record(join(root, key), values[key], result.generation) + + @rule(data=st.data()) + def delete_many(self, data: st.DataObject) -> None: + root = data.draw(ROOTS, label="root") + keys = self._batch_keys(data, root) + results = (self.store / root).delete_many(keys) + assert [key for key, _ in results] == keys + for key, result in results: + path = join(root, key) + if path in self.values: + assert result == Deleted(self.generations[path]) + self._forget(path) + else: + assert isinstance(result, Missing) + + @rule(data=st.data()) + def delete(self, data: st.DataObject) -> None: + root, key = self._target(data) + path = join(root, key) + result = (self.store / root).delete(key) + if path in self.values: + assert result == Deleted(self.generations[path]) + self._forget(path) + else: + assert isinstance(result, Missing) + + @precondition(lambda self: self._capabilities.conditional_delete) + @rule(data=st.data(), stale=st.booleans()) + def delete_if_match(self, data: st.DataObject, stale: bool) -> None: + root, key = self._target(data) + path = join(root, key) + current = self.generations.get(path) + expected = Generation(object()) if stale or current is None else current + result = delete_if_match(self.store / root, key, expected) + if not stale and current is not None: + assert result == Deleted(current) + self._forget(path) + else: + self._check_refused(result, path) + + @precondition( + lambda self: ( + self._capabilities.conditional_delete + and self._capabilities.unique_generations + and len(self.retired) > 0 + ) + ) + @rule(data=st.data()) + def delete_if_match_retired(self, data: st.DataObject) -> None: + root, key = self._target(data) + retired = data.draw(st.sampled_from(self.retired), label="retired generation") + result = delete_if_match(self.store / root, key, retired) + self._check_refused(result, join(root, key)) + + @rule(root=ROOTS, directory=DIRECTORIES) + def delete_dir(self, root: str, directory: str) -> None: + (self.store / root).delete_dir(directory) + target = join(root, directory) + for path in [p for p in self.values if p != target and _under(p, target)]: + self._forget(path) + + # --- reads, checked against the model + + @rule(root=ROOTS, directory=DIRECTORIES) + def list_view(self, root: str, directory: str) -> None: + view = self.store / root + under = join(root, directory) + expected = {_relative(p, root) for p in self.values if p != under and _under(p, under)} + listed = list(view.list(directory)) + assert len(listed) == len(set(listed)), "list reported a key twice" + assert set(listed) == expected + + @rule(root=ROOTS, directory=DIRECTORIES) + def list_dir_view(self, root: str, directory: str) -> None: + under = join(root, directory) + expected: set[ListItem] = set() + for path in self.values: + if path == under or not _under(path, under): + continue + name, sep, _ = _relative(path, under).partition("/") + expected.add(Prefix(name) if sep else Key(name)) + listed = list((self.store / root).list_dir(directory)) + assert len(listed) == len(set(listed)), "list_dir reported a child twice" + found = set(listed) + if self._capabilities.leaves_empty_directories: + found -= {item for item in found - expected if isinstance(item, Prefix)} + assert found == expected + + @invariant() + def reads_match_the_model(self) -> None: + if not hasattr(self, "store"): + return + assert set(self.store.list()) == set(self.values) + batch = dict(self.store.get_many(self.values)) + assert set(batch) == set(self.values) + for path, result in batch.items(): + assert isinstance(result, Found) + assert (bytes(result.value), result.generation) == ( + self.values[path], + self.generations[path], + ) + for path, value in self.values.items(): + generation = self.generations[path] + result = self.store.get(path) + assert isinstance(result, Found) + assert (bytes(result.value), result.generation) == (value, generation) + meta = self.store.head(path) + assert isinstance(meta, ObjectMetadata) + assert (meta.size, meta.generation) == (len(value), generation) + unchanged = self.store.get_if_changed(path, generation) + assert isinstance(unchanged, Unchanged) + assert unchanged.generation == generation + + def teardown(self) -> None: + if hasattr(self, "store"): + self.store.close() diff --git a/packages/zarr-storage/src/zarr_storage/testing/_suite.py b/packages/zarr-storage/src/zarr_storage/testing/_suite.py new file mode 100644 index 0000000000..7a36a9a7ce --- /dev/null +++ b/packages/zarr-storage/src/zarr_storage/testing/_suite.py @@ -0,0 +1,1209 @@ +"""The storage contract, as tests. + +`StoreSuite` holds what every backend must do. Capabilities a backend may lack come as mixins: a +backend's test class combines `StoreSuite` with the mixin for each capability it has, or with the +mixin that checks it refuses that capability cleanly. Every test that takes the `store` fixture runs +twice, through `Store` and through `AsyncStore` (see `AsyncAPI`). + +Each behavior has one table-driven test for the expected outputs and one test per error case. +""" + +from __future__ import annotations + +import array +import asyncio +import pickle +import threading +import time +from collections import Counter +from dataclasses import dataclass +from functools import partial +from typing import TYPE_CHECKING, Literal + +import pytest +from hypothesis import HealthCheck, given, settings +from hypothesis import strategies as st +from hypothesis.stateful import run_state_machine_as_test + +from zarr_storage._errors import ( + BatchError, + InvalidByteRangeError, + InvalidKeyError, + KeyConflictError, + ReadOnlyError, + StoreClosedError, + UnsupportedOperationError, +) +from zarr_storage._types import ( + Copied, + Deleted, + Found, + Generation, + Key, + Missing, + ObjectMetadata, + OffsetRequest, + Prefix, + RangeRequest, + Rejected, + SuffixRequest, + Unchanged, + Written, +) +from zarr_storage.testing._api import FullBackend, StoreAPI, async_api, delete_if_match, unwrap +from zarr_storage.testing._stateful import Capabilities, StoreMachine + +if TYPE_CHECKING: + from collections.abc import Callable, Iterable, Iterator + from pathlib import Path + + from typing_extensions import Buffer + + from zarr_storage._store import Store + from zarr_storage._types import ByteRange, ListItem + +__all__ = [ + "ConditionalDeleteSuite", + "ConditionalPutSuite", + "ConditionalRaceSuite", + "KeyAndDirectorySuite", + "KeyDirectoryConflictSuite", + "NoConditionalPutSuite", + "PickleSuite", + "StoreSuite", + "UniqueGenerationsSuite", +] + +VALUE = b"0123456789" + +# ---------------------------------------------------------------------------- tables + +VALID_KEYS = ["a", "a/b", "a/b/c", ".a", "a.b", "..a", "a..x", "__a", "zarr.json", "c/0/0", "a b"] +INVALID_KEYS = ["", "/", "/a", "a/", "a//b", ".", "..", "./a", "a/.", "a/../b", "a\x00b"] +INVALID_DIRECTORIES = ["//", "/a", "a//b", "..", "a/./b", "a\x00"] + +VALUES: list[tuple[str, Buffer, bytes]] = [ + ("bytes", VALUE, VALUE), + ("empty", b"", b""), + ("bytearray", bytearray(VALUE), VALUE), + ("memoryview", memoryview(VALUE), VALUE), + ("int-array", array.array("i", [1, 2]), array.array("i", [1, 2]).tobytes()), + ("2d-view", memoryview(VALUE).cast("B", (2, 5)), VALUE), +] + +BYTE_RANGES: list[tuple[ByteRange | None, bytes]] = [ + (None, VALUE), + (RangeRequest(0, 10), VALUE), + (RangeRequest(2, 5), b"234"), + (RangeRequest(5, 5), b""), + (RangeRequest(8, 100), b"89"), + (RangeRequest(10, 12), b""), + (OffsetRequest(0), VALUE), + (OffsetRequest(7), b"789"), + (OffsetRequest(10), b""), + (SuffixRequest(3), b"789"), + (SuffixRequest(0), b""), + (SuffixRequest(100), VALUE), +] + + +@dataclass(frozen=True) +class PutCase: + present: bool + operation: Literal["put", "put_if_absent", "put_if_match_current", "put_if_match_stale"] + outcome: type[Written | Rejected | Missing] + + +PUT_CASES = [ + PutCase(present=False, operation="put", outcome=Written), + PutCase(present=True, operation="put", outcome=Written), + PutCase(present=False, operation="put_if_absent", outcome=Written), + PutCase(present=True, operation="put_if_absent", outcome=Rejected), +] +PUT_IF_MATCH_CASES = [ + PutCase(present=True, operation="put_if_match_current", outcome=Written), + PutCase(present=True, operation="put_if_match_stale", outcome=Rejected), + PutCase(present=False, operation="put_if_match_stale", outcome=Missing), +] + + +@dataclass(frozen=True) +class DeleteCase: + present: bool + operation: Literal["delete", "delete_if_match_current", "delete_if_match_stale"] + outcome: type[Deleted | Rejected | Missing] + + +DELETE_CASES = [ + DeleteCase(present=False, operation="delete", outcome=Missing), + DeleteCase(present=True, operation="delete", outcome=Deleted), +] +DELETE_IF_MATCH_CASES = [ + DeleteCase(present=True, operation="delete_if_match_current", outcome=Deleted), + DeleteCase(present=True, operation="delete_if_match_stale", outcome=Rejected), + DeleteCase(present=False, operation="delete_if_match_stale", outcome=Missing), +] + + +@dataclass(frozen=True) +class IfChangedCase: + present: bool + generation: Literal["current", "stale"] + byte_range: ByteRange | None + outcome: type[Found | Missing | Unchanged] + expected: bytes | None = None + + +IF_CHANGED_CASES = [ + IfChangedCase(present=True, generation="current", byte_range=None, outcome=Unchanged), + IfChangedCase( + present=True, generation="current", byte_range=RangeRequest(2, 5), outcome=Unchanged + ), + IfChangedCase(present=True, generation="stale", byte_range=None, outcome=Found, expected=VALUE), + IfChangedCase( + present=True, + generation="stale", + byte_range=RangeRequest(2, 5), + outcome=Found, + expected=b"234", + ), + IfChangedCase(present=False, generation="stale", byte_range=None, outcome=Missing), +] + +BATCH_VALUES = {"a": VALUE, "b/c": b"xy"} + +GET_MANY_CASES: list[tuple[list[str | tuple[str, ByteRange | None]], list[bytes | None]]] = [ + ([], []), + (["a"], [VALUE]), + (["a", "b/c", "missing", "b"], [VALUE, b"xy", None, None]), + (["a", "a"], [VALUE, VALUE]), + ( + [("a", RangeRequest(2, 5)), ("a", SuffixRequest(2)), ("b/c", None), ("x", None)], + [b"234", b"89", b"xy", None], + ), +] +"""Requests to `get_many` over `BATCH_VALUES`, and what each reads (`None`: `Missing`).""" + + +TREE = [ + "zarr.json", + "a/zarr.json", + "a/c/0", + "a/c/1", + "ab/zarr.json", + "lat/zarr.json", + "lat_bnds/zarr.json", +] + + +def _without(*removed: str) -> set[str]: + return set(TREE) - set(removed) + + +DELETE_DIR_CASES: list[tuple[str, set[str]]] = [ + ("a", _without("a/zarr.json", "a/c/0", "a/c/1")), + ("a/", _without("a/zarr.json", "a/c/0", "a/c/1")), + ("lat", _without("lat/zarr.json")), + ("a/c", _without("a/c/0", "a/c/1")), + ("a/c/0", _without()), + ("missing", _without()), + ("", set()), +] + +LIST_DIRECTORIES = ["", "a", "a/", "a/c", "lat", "lat/", "missing", "a/c/0"] + +VIEW_ROOTS: list[tuple[list[str], str]] = [ + ([], ""), + ([""], ""), + (["a"], "a"), + (["a/"], "a"), + (["a", "b"], "a/b"), + (["a/b", ""], "a/b"), + (["a", "b/c/"], "a/b/c"), + (["/"], ""), + (["a", "/"], "a"), +] + +KEY_SETS = st.sets( + st.lists(st.sampled_from(["a", "b", "ab", "c.d", "_"]), min_size=1, max_size=3).map("/".join), + max_size=8, +) +LISTED_DIRECTORIES = st.sampled_from(["", "a", "b", "a/a", "ab", "a/b/"]) + + +def _model_list(keys: Iterable[str], directory: str) -> set[str]: + directory = directory.removesuffix("/") + if directory == "": + return set(keys) + return {k for k in keys if k.startswith(f"{directory}/")} + + +def _model_list_dir(keys: Iterable[str], directory: str) -> set[ListItem]: + directory = directory.removesuffix("/") + items: set[ListItem] = set() + for key in _model_list(keys, directory): + rest = key if directory == "" else key.removeprefix(f"{directory}/") + name, sep, _ = rest.partition("/") + items.add(Prefix(name) if sep else Key(name)) + return items + + +def _without_key_directories(keys: set[str]) -> set[str]: + """`keys` minus those that are also another key's directory.""" + return {k for k in keys if not any(other.startswith(f"{k}/") for other in keys)} + + +def _stale() -> Generation: + return Generation(object()) + + +# ---------------------------------------------------------------------------- operations + + +def _get_key(store: StoreAPI, key: str) -> object: + return store.get(key) + + +def _head_key(store: StoreAPI, key: str) -> object: + return store.head(key) + + +def _put_key(store: StoreAPI, key: str) -> object: + return store.put(key, VALUE) + + +def _delete_key(store: StoreAPI, key: str) -> object: + return store.delete(key) + + +KEY_OPERATIONS: list[Callable[[StoreAPI, str], object]] = [ + _get_key, + _head_key, + _put_key, + _delete_key, +] +KEY_OPERATION_IDS = ["get", "head", "put", "delete"] + + +def _view_of(store: StoreAPI, directory: str) -> object: + return store / directory + + +def _list_of(store: StoreAPI, directory: str) -> object: + return store.list(directory) + + +def _list_dir_of(store: StoreAPI, directory: str) -> object: + return store.list_dir(directory) + + +def _delete_dir_of(store: StoreAPI, directory: str) -> object: + store.delete_dir(directory) + return None + + +DIRECTORY_OPERATIONS: list[Callable[[StoreAPI, str], object]] = [ + _view_of, + _list_of, + _list_dir_of, + _delete_dir_of, +] +DIRECTORY_OPERATION_IDS = ["view", "list", "list_dir", "delete_dir"] + + +def _missing_get(store: StoreAPI) -> object: + return store.get("missing") + + +def _missing_get_directory(store: StoreAPI) -> object: + return store.get("a") + + +def _missing_get_range(store: StoreAPI) -> object: + return store.get("missing", byte_range=RangeRequest(0, 1)) + + +def _missing_get_ranges(store: StoreAPI) -> object: + return store.get_ranges("missing", [RangeRequest(0, 1)]) + + +def _missing_get_into(store: StoreAPI) -> object: + return store.get_into("missing", bytearray(1)) + + +def _missing_head(store: StoreAPI) -> object: + return store.head("a") + + +def _missing_get_if_changed(store: StoreAPI) -> object: + return store.get_if_changed("missing", _stale()) + + +def _missing_get_many(store: StoreAPI) -> object: + ((_, result),) = store.get_many(["missing"]) + return result + + +def _missing_get_into_many(store: StoreAPI) -> object: + ((_, result),) = store.get_into_many([("missing", bytearray(1))]) + return result + + +MISSING_READS: list[Callable[[StoreAPI], object]] = [ + _missing_get, + _missing_get_directory, + _missing_get_range, + _missing_get_if_changed, + _missing_get_ranges, + _missing_get_into, + _missing_get_many, + _missing_get_into_many, + _missing_head, +] +MISSING_READ_IDS = [ + "get", + "get-directory", + "get-range", + "get_if_changed", + "get_ranges", + "get_into", + "get_many", + "get_into_many", + "head", +] + + +def _put(store: StoreAPI) -> object: + return store.put("k", VALUE) + + +def _delete(store: StoreAPI) -> object: + return store.delete("k") + + +def _delete_dir(store: StoreAPI) -> object: + store.delete_dir("") + return None + + +def _put_many(store: StoreAPI) -> object: + return store.put_many([("k", VALUE)]) + + +def _delete_many(store: StoreAPI) -> object: + return store.delete_many(["k"]) + + +def _put_many_empty(store: StoreAPI) -> object: + return store.put_many([]) + + +WRITES: list[Callable[[StoreAPI], object]] = [ + _put, + _delete, + _delete_dir, + _put_many, + _delete_many, + _put_many_empty, +] +WRITE_IDS = ["put", "delete", "delete_dir", "put_many", "delete_many", "put_many-empty"] + + +def _get(store: StoreAPI) -> object: + return store.get("k") + + +def _get_ranges(store: StoreAPI) -> object: + return store.get_ranges("k", [SuffixRequest(1)]) + + +def _get_into(store: StoreAPI) -> object: + return store.get_into("k", bytearray(1)) + + +def _head(store: StoreAPI) -> object: + return store.head("k") + + +def _get_many(store: StoreAPI) -> object: + return store.get_many(["k"]) + + +def _get_into_many(store: StoreAPI) -> object: + return store.get_into_many([("k", bytearray(1))]) + + +def _list(store: StoreAPI) -> object: + return store.list() + + +def _list_dir(store: StoreAPI) -> object: + return store.list_dir() + + +CLOSED_OPERATIONS: list[Callable[[StoreAPI], object]] = [ + _get, + _get_ranges, + _get_into, + _head, + _put, + _delete, + _delete_dir, + _get_many, + _get_into_many, + _put_many, + _delete_many, + _list, + _list_dir, +] +CLOSED_IDS = [ + "get", + "get_ranges", + "get_into", + "head", + "put", + "delete", + "delete_dir", + "get_many", + "get_into_many", + "put_many", + "delete_many", + "list", + "list_dir", +] + + +def _get_many_invalid(store: StoreAPI) -> object: + return store.get_many(["x", "/bad"]) + + +def _get_into_many_invalid(store: StoreAPI) -> object: + return store.get_into_many([("x", bytearray(1)), ("/bad", bytearray(1))]) + + +def _put_many_invalid(store: StoreAPI) -> object: + return store.put_many([("y", VALUE), ("/bad", VALUE)]) + + +def _delete_many_invalid(store: StoreAPI) -> object: + return store.delete_many(["x", "/bad"]) + + +INVALID_BATCHES: list[Callable[[StoreAPI], object]] = [ + _get_many_invalid, + _get_into_many_invalid, + _put_many_invalid, + _delete_many_invalid, +] +INVALID_BATCH_IDS = ["get_many", "get_into_many", "put_many", "delete_many"] + + +def _put_many_twice(store: StoreAPI) -> object: + return store.put_many([("x", b"1"), ("y", b"2"), ("x", b"3")]) + + +def _delete_many_twice(store: StoreAPI) -> object: + return store.delete_many(["x", "x"]) + + +def _write_repeatedly( + store: Store[FullBackend], stop: threading.Event, written: dict[Generation, int] +) -> None: + fill = 0 + while not stop.is_set(): + fill = (fill + 1) % 256 + written[store.put("k", bytes([fill]) * 1000).generation] = fill + + +def _create( + store: Store[FullBackend], key: str, barrier: threading.Barrier, results: list[bool] +) -> None: + barrier.wait() + results.append(isinstance(store.put_if_absent(key, VALUE), Written)) + + +def _check_put(store: StoreAPI, case: PutCase) -> None: + current = store.put("k", VALUE).generation if case.present else None + match case.operation: + case "put": + result = store.put("k", b"new") + case "put_if_absent": + result = store.put_if_absent("k", b"new") + case "put_if_match_current": + assert current is not None + result = store.put_if_match("k", b"new", current) + case "put_if_match_stale": + result = store.put_if_match("k", b"new", _stale()) + assert isinstance(result, case.outcome) + stored = store.get("k") + match result: + case Written(generation=generation): + assert generation != current + assert isinstance(stored, Found) + assert (bytes(stored.value), stored.generation) == (b"new", generation) + case Rejected(generation=generation): + assert generation == current + assert isinstance(stored, Found) + assert bytes(stored.value) == VALUE + case Missing(): + assert isinstance(stored, Missing) + + +def _check_delete(store: StoreAPI, case: DeleteCase) -> None: + current = store.put("k", VALUE).generation if case.present else None + match case.operation: + case "delete": + result = store.delete("k") + case "delete_if_match_current": + assert current is not None + result = delete_if_match(store, "k", current) + case "delete_if_match_stale": + result = delete_if_match(store, "k", _stale()) + assert isinstance(result, case.outcome) + match result: + case Deleted(generation=generation) | Rejected(generation=generation): + assert generation == current + case Missing(): + pass + assert store.exists("k") is isinstance(result, Rejected) + + +def _check_listing(store: Store[FullBackend], keys: set[str], directory: str) -> None: + for key in keys: + store.put(key, VALUE) + assert set(store.list(directory)) == _model_list(keys, directory) + assert set(store.list_dir(directory)) == _model_list_dir(keys, directory) + + +def _machine_store( + suite: StoreSuite, + directories: pytest.TempPathFactory, + api: Literal["sync", "async"], + runner: asyncio.Runner, +) -> StoreAPI: + store = suite.make_store(directories.mktemp("machine")) + return store if api == "sync" else async_api(store, runner) + + +# ---------------------------------------------------------------------------- the core suite + + +class StoreSuite: + """What every backend must do. + + Override `make_store` to return an empty, writable store over a fresh backend. `directory` is a + new empty directory a filesystem backend can use; other backends ignore it. + """ + + def make_store(self, directory: Path) -> Store[FullBackend]: + raise NotImplementedError("Override make_store to return an empty store.") + + @pytest.fixture(params=["sync", "async"]) + def store(self, request: pytest.FixtureRequest, tmp_path: Path) -> Iterator[StoreAPI]: + store = self.make_store(tmp_path) + if request.param == "sync": + yield store + return + with asyncio.Runner() as runner: + yield async_api(store, runner) + + # ------------------------------------------------------------------ everything at once + + @pytest.mark.parametrize("api", ["sync", "async"]) + def test_state_machine( + self, tmp_path_factory: pytest.TempPathFactory, api: Literal["sync", "async"] + ) -> None: + """Random sequences of operations through random views agree with a dict model.""" + capabilities = Capabilities( + conditional_put=isinstance(self, ConditionalPutSuite), + conditional_delete=isinstance(self, ConditionalDeleteSuite), + holds_key_and_directory=isinstance(self, KeyAndDirectorySuite), + leaves_empty_directories=isinstance(self, KeyDirectoryConflictSuite), + unique_generations=isinstance(self, UniqueGenerationsSuite), + ) + with asyncio.Runner() as runner: + make_store = partial(_machine_store, self, tmp_path_factory, api, runner) + run_state_machine_as_test( + # Hypothesis calls the factory with no arguments; a partial is a fine factory. + partial(StoreMachine, make_store, capabilities), # pyright: ignore[reportArgumentType] + settings=settings( + max_examples=40, + stateful_step_count=30, + deadline=None, + suppress_health_check=[HealthCheck.differing_executors], + ), + ) + + # ------------------------------------------------------------------ keys + + @pytest.mark.parametrize("key", VALID_KEYS) + def test_valid_key_round_trips(self, store: StoreAPI, key: str) -> None: + store.put(key, VALUE) + result = store.get(key) + assert isinstance(result, Found) + assert bytes(result.value) == VALUE + assert key in set(store.list()) + + @pytest.mark.parametrize("key", INVALID_KEYS) + @pytest.mark.parametrize("operation", KEY_OPERATIONS, ids=KEY_OPERATION_IDS) + def test_invalid_key_raises( + self, store: StoreAPI, key: str, operation: Callable[[StoreAPI, str], object] + ) -> None: + with pytest.raises(InvalidKeyError): + operation(store, key) + + @pytest.mark.parametrize("directory", INVALID_DIRECTORIES) + @pytest.mark.parametrize("operation", DIRECTORY_OPERATIONS, ids=DIRECTORY_OPERATION_IDS) + def test_invalid_directory_raises( + self, store: StoreAPI, directory: str, operation: Callable[[StoreAPI, str], object] + ) -> None: + with pytest.raises(InvalidKeyError): + operation(store, directory) + + # ------------------------------------------------------------------ get / put + + @pytest.mark.parametrize( + ("value", "expected"), [v[1:] for v in VALUES], ids=[v[0] for v in VALUES] + ) + def test_put_then_get(self, store: StoreAPI, value: Buffer, expected: bytes) -> None: + written = store.put("k", value) + result = store.get("k") + assert isinstance(result, Found) + assert (bytes(result.value), result.generation) == (expected, written.generation) + assert result.read_at is None or abs(result.read_at - time.time()) < 3600 + meta = store.head("k") + assert isinstance(meta, ObjectMetadata) + assert (meta.size, meta.generation) == (len(expected), written.generation) + assert meta.last_modified is None or abs(meta.last_modified - time.time()) < 3600 + assert store.exists("k") + + def test_put_copies_the_value(self, store: StoreAPI) -> None: + value = bytearray(VALUE) + store.put("k", value) + value[0:1] = b"X" + result = store.get("k") + assert isinstance(result, Found) + assert bytes(result.value) == VALUE + + @pytest.mark.parametrize("read", MISSING_READS, ids=MISSING_READ_IDS) + def test_reads_of_missing_key( + self, store: StoreAPI, read: Callable[[StoreAPI], object] + ) -> None: + store.put("a/k", VALUE) + assert isinstance(read(store), Missing) + assert not store.exists("a") + + @pytest.mark.parametrize("case", IF_CHANGED_CASES, ids=repr) + def test_get_if_changed(self, store: StoreAPI, case: IfChangedCase) -> None: + current = store.put("k", VALUE).generation if case.present else None + generation = current if case.generation == "current" and current is not None else _stale() + result = store.get_if_changed("k", generation, byte_range=case.byte_range) + assert isinstance(result, case.outcome) + match result: + case Unchanged(): + assert result.generation == current + case Found(): + assert (bytes(result.value), result.generation) == (case.expected, current) + case Missing(): + pass + + # ------------------------------------------------------------------ byte ranges + + @pytest.mark.parametrize(("byte_range", "expected"), BYTE_RANGES) + def test_byte_range( + self, store: StoreAPI, byte_range: ByteRange | None, expected: bytes + ) -> None: + store.put("k", VALUE) + result = store.get("k", byte_range=byte_range) + assert isinstance(result, Found) + assert bytes(result.value) == expected + out = bytearray(len(VALUE) + 1) + copied = store.get_into("k", out, byte_range=byte_range) + assert isinstance(copied, Copied) + assert (copied.size, copied.generation) == (len(expected), result.generation) + assert bytes(out[: copied.size]) == expected + if byte_range is not None: + ranges = store.get_ranges("k", [byte_range, SuffixRequest(1)]) + assert not isinstance(ranges, Missing) + assert [bytes(v) for v in ranges.values] == [expected, b"9"] + assert ranges.generation == result.generation + + @pytest.mark.parametrize("byte_range", [RangeRequest(11, 12), OffsetRequest(11)]) + def test_byte_range_past_end_raises(self, store: StoreAPI, byte_range: ByteRange) -> None: + store.put("k", VALUE) + with pytest.raises(InvalidByteRangeError): + store.get("k", byte_range=byte_range) + + def test_get_into_short_buffer_raises(self, store: StoreAPI) -> None: + store.put("k", VALUE) + with pytest.raises(ValueError, match="holds"): + store.get_into("k", bytearray(len(VALUE) - 1)) + + def test_get_into_read_only_buffer_raises(self, store: StoreAPI) -> None: + store.put("k", VALUE) + with pytest.raises(ValueError, match="writable"): + store.get_into("k", memoryview(bytes(len(VALUE)))) + + def test_get_into_non_contiguous_buffer_raises(self, store: StoreAPI) -> None: + store.put("k", VALUE) + with pytest.raises(ValueError, match="contiguous"): + store.get_into("k", memoryview(bytearray(2 * len(VALUE)))[::2]) + + @pytest.mark.parametrize("api", ["sync", "async"]) + def test_get_ranges_reads_one_generation(self, tmp_path: Path, api: str) -> None: + """Ranges read while another thread rewrites the value all come from the write whose + generation the result reports.""" + store = self.make_store(tmp_path) + written: dict[Generation, int] = {} + written[store.put("k", bytes(1000)).generation] = 0 + stop = threading.Event() + writer = threading.Thread(target=_write_repeatedly, args=(store, stop, written)) + seen: list[tuple[Generation, bytes, bytes]] = [] + writer.start() + try: + with asyncio.Runner() as runner: + reader: StoreAPI = store if api == "sync" else async_api(store, runner) + for _ in range(200): + result = reader.get_ranges("k", [RangeRequest(0, 1), SuffixRequest(1)]) + assert not isinstance(result, Missing) + first, last = (bytes(v) for v in result.values) + seen.append((result.generation, first, last)) + finally: + stop.set() + writer.join() + for generation, first, last in seen: + assert first == last + if isinstance(self, UniqueGenerationsSuite): + assert first == bytes([written[generation]]) + + # ------------------------------------------------------------------ writes + + @pytest.mark.parametrize("case", PUT_CASES, ids=repr) + def test_put(self, store: StoreAPI, case: PutCase) -> None: + _check_put(store, case) + + def test_concurrent_creates_apply_once(self, tmp_path: Path) -> None: + """Of many racing `put_if_absent` writes to one key, exactly one applies.""" + store = self.make_store(tmp_path) + for round_ in range(20): + barrier = threading.Barrier(8) + results: list[bool] = [] + threads = [ + threading.Thread(target=_create, args=(store, f"k{round_}", barrier, results)) + for _ in range(8) + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + assert results.count(True) == 1 + + @pytest.mark.parametrize("case", DELETE_CASES, ids=repr) + def test_delete(self, store: StoreAPI, case: DeleteCase) -> None: + _check_delete(store, case) + + def test_delete_of_a_directory_deletes_nothing(self, store: StoreAPI) -> None: + store.put("a/b", VALUE) + assert isinstance(store.delete("a"), Missing) + assert store.exists("a/b") + + @pytest.mark.parametrize(("directory", "remaining"), DELETE_DIR_CASES) + def test_delete_dir(self, store: StoreAPI, directory: str, remaining: set[str]) -> None: + for key in TREE: + store.put(key, VALUE) + store.delete_dir(directory) + assert set(store.list()) == remaining + + # ------------------------------------------------------------------ batches + + @pytest.mark.parametrize(("requests", "expected"), GET_MANY_CASES) + def test_get_many( + self, + store: StoreAPI, + requests: list[str | tuple[str, ByteRange | None]], + expected: list[bytes | None], + ) -> None: + written = dict(store.put_many(BATCH_VALUES.items())) + read: Counter[tuple[object, bytes | None]] = Counter() + for request, result in store.get_many(requests): + key = request if isinstance(request, str) else request[0] + match result: + case Found(value=value, generation=generation): + assert generation == written[key].generation + read[request, bytes(value)] += 1 + case Missing(): + read[request, None] += 1 + assert read == Counter(zip(requests, expected, strict=True)) + + def test_get_into_many(self, store: StoreAPI) -> None: + written = dict(store.put_many(BATCH_VALUES.items())) + whole, part, absent = bytearray(len(VALUE)), bytearray(2), bytearray(1) + results = { + request[0]: result + for request, result in store.get_into_many( + [("a", whole), ("b/c", part, OffsetRequest(1)), ("missing", absent)] + ) + } + assert results["a"] == Copied(len(VALUE), written["a"].generation, results["a"].read_at) + assert results["b/c"] == Copied(1, written["b/c"].generation, results["b/c"].read_at) + assert isinstance(results["missing"], Missing) + assert (bytes(whole), bytes(part), bytes(absent)) == (VALUE, b"y\x00", b"\x00") + + def test_put_many_and_delete_many(self, store: StoreAPI) -> None: + store.put("b/c", b"old") + items = [("b/c", b"xy"), ("a", VALUE), ("d", b"")] + written = store.put_many(items) + assert [key for key, _ in written] == ["b/c", "a", "d"] + for (key, value), (_, result) in zip(items, written, strict=True): + stored = store.get(key) + assert isinstance(stored, Found) + assert (bytes(stored.value), stored.generation) == (value, result.generation) + deleted = store.delete_many(["a", "missing", "b/c"]) + assert [(key, type(result)) for key, result in deleted] == [ + ("a", Deleted), + ("missing", Missing), + ("b/c", Deleted), + ] + assert deleted[0][1] == Deleted(written[1][1].generation) + assert set(store.list()) == {"d"} + assert store.put_many([]) == [] + assert store.delete_many([]) == [] + + def test_get_many_taken_in_part(self, store: StoreAPI) -> None: + """A caller may stop taking results; the store stays usable.""" + store.put_many([(f"k{i}", VALUE) for i in range(40)]) + results = store.get_many(f"k{i}" for i in range(40)) + assert isinstance(next(iter(results))[1], Found) + del results + assert len(list(store.get_many(f"k{i}" for i in range(40)))) == 40 + + def test_batches_through_a_view(self, store: StoreAPI) -> None: + view = store / "r" + view.put_many([("k", VALUE)]) + assert set(store.list()) == {"r/k"} + assert [key for key, _ in (store / "r").get_many(["k"])] == ["k"] + assert view.delete_many(["k"])[0][1] != Missing() + assert set(store.list()) == set() + + @pytest.mark.parametrize("batch", INVALID_BATCHES, ids=INVALID_BATCH_IDS) + def test_batch_with_an_invalid_key_raises_before_any_operation( + self, store: StoreAPI, batch: Callable[[StoreAPI], object] + ) -> None: + store.put("x", VALUE) + with pytest.raises(InvalidKeyError): + batch(store) + assert set(store.list()) == {"x"} + + @pytest.mark.parametrize("batch", [_put_many_twice, _delete_many_twice], ids=["put", "delete"]) + def test_write_batch_naming_a_key_twice_raises( + self, store: StoreAPI, batch: Callable[[StoreAPI], object] + ) -> None: + store.put("x", VALUE) + with pytest.raises(ValueError, match="more than once"): + batch(store) + assert set(store.list()) == {"x"} + result = store.get("x") + assert isinstance(result, Found) + assert bytes(result.value) == VALUE + + # ------------------------------------------------------------------ listing + + @pytest.mark.parametrize("directory", LIST_DIRECTORIES) + def test_list(self, store: StoreAPI, directory: str) -> None: + for key in TREE: + store.put(key, VALUE) + listed, children = list(store.list(directory)), list(store.list_dir(directory)) + assert len(listed) == len(set(listed)) + assert len(children) == len(set(children)) + assert set(listed) == _model_list(TREE, directory) + assert set(children) == _model_list_dir(TREE, directory) + + @given(keys=KEY_SETS, directory=LISTED_DIRECTORIES) + # Each backend's subclass runs this test; that is the point of the suite, not a flaky executor. + @settings( + max_examples=50, deadline=None, suppress_health_check=[HealthCheck.differing_executors] + ) + def test_listing_matches_a_model( + self, tmp_path_factory: pytest.TempPathFactory, keys: set[str], directory: str + ) -> None: + store = self.make_store(tmp_path_factory.mktemp("store")) + _check_listing(store, _without_key_directories(keys), directory) + + # ------------------------------------------------------------------ views + + @pytest.mark.parametrize(("path", "root"), VIEW_ROOTS) + def test_view_root(self, store: StoreAPI, path: list[str], root: str) -> None: + view = store + for directory in path: + view = view / directory + assert view.root == root + + def test_views_share_the_backend(self, store: StoreAPI) -> None: + child = store / "a" + child.put("k", VALUE) + assert store.exists("a/k") + store.put("a/b/c", VALUE) + assert set(child.list()) == {"k", "b/c"} + assert set(child.list("b")) == {"b/c"} + assert set(child.list_dir()) == {Key("k"), Prefix("b")} + result = child.get_ranges("b/c", [SuffixRequest(1)]) + assert not isinstance(result, Missing) + assert [bytes(v) for v in result.values] == [b"9"] + meta = child.head("b/c") + assert isinstance(meta, ObjectMetadata) + assert meta.size == len(VALUE) + assert set((store / "lat").list()) == set() + child.delete_dir() + assert set(store.list()) == set() + + def test_identity(self, store: StoreAPI, tmp_path_factory: pytest.TempPathFactory) -> None: + other = self.make_store(tmp_path_factory.mktemp("other")) + base = unwrap(store) + a, ab, lat, lat_bnds = base / "a", base / "a/b", base / "lat", base / "lat_bnds" + table = [ + (base, base / "", True, True), + (a, ab, True, True), + (ab, a, True, True), + (base, lat, True, True), + (lat, lat_bnds, True, False), + (a, lat, True, False), + (a, a.with_read_only(), True, True), + (base, other, False, False), + (a, other / "a", False, False), + ] + for left, right, shares, overlaps in table: + assert left.shares_backend(right) is shares + assert left.overlaps(right) is overlaps + assert base != base / "" + + def test_read_only_views(self, store: StoreAPI) -> None: + store.put("k", VALUE) + reader = store.with_read_only() + assert (store.read_only, reader.read_only, (reader / "x").read_only) == (False, True, True) + assert reader.exists("k") + assert not store.with_read_only(False).read_only + + def test_writable_view_of_read_only_store_raises(self, store: StoreAPI) -> None: + with pytest.raises(ReadOnlyError): + store.with_read_only().with_read_only(False) + + @pytest.mark.parametrize("write", WRITES, ids=WRITE_IDS) + def test_write_through_read_only_view_raises( + self, store: StoreAPI, write: Callable[[StoreAPI], object] + ) -> None: + store.put("k", VALUE) + with pytest.raises(ReadOnlyError): + write(store.with_read_only()) + assert store.exists("k") + + # ------------------------------------------------------------------ lifecycle + + def test_closing_a_derived_view_leaves_the_backend_open(self, store: StoreAPI) -> None: + store.put("k", VALUE) + (store / "a").close() + store.with_read_only().close() + assert not store.closed + assert store.exists("k") + + @pytest.mark.parametrize("operation", CLOSED_OPERATIONS, ids=CLOSED_IDS) + def test_use_after_owner_closes_raises( + self, store: StoreAPI, operation: Callable[[StoreAPI], object] + ) -> None: + view = store / "a" + with store: + store.put("a/k", VALUE) + assert view.closed + with pytest.raises(StoreClosedError): + operation(view) + + @pytest.mark.parametrize("listing", ["list", "list_dir", "get_many", "get_into_many"]) + def test_iterating_after_close_raises(self, store: StoreAPI, listing: str) -> None: + store.put("a", VALUE) + store.put("b", VALUE) + items = { + "list": partial(store.list), + "list_dir": partial(store.list_dir), + "get_many": partial(store.get_many, ["a", "b"]), + "get_into_many": partial(store.get_into_many, [("a", bytearray(len(VALUE)))]), + }[listing]() + store.close() + with pytest.raises(StoreClosedError): + next(iter(items)) + + +# ---------------------------------------------------------------------------- capability mixins + + +class UniqueGenerationsSuite: + """For backends that never hand out a generation twice, even after a delete. + + obstore's `LocalStore` doesn't qualify: its ETag comes from the inode and mtime, which a + filesystem such as ext4 can repeat. + """ + + def make_store(self, directory: Path) -> Store[FullBackend]: + raise NotImplementedError("Override make_store to return an empty store.") + + def test_rewrite_changes_generation(self, store: StoreAPI) -> None: + generations = [store.put("k", value).generation for value in [VALUE, VALUE, b"x" * 10]] + assert len(set(generations)) == 3 + + def test_generations_are_not_reused_after_deletes(self, tmp_path: Path) -> None: + store = self.make_store(tmp_path) + seen: set[Generation] = set() + for i in range(200): + generation = store.put(f"k{i % 3}", VALUE).generation + assert generation not in seen + seen.add(generation) + store.delete(f"k{(i + 1) % 3}") + + +class ConditionalPutSuite: + """For backends whose `put_if_match` works.""" + + @pytest.mark.parametrize("case", PUT_IF_MATCH_CASES, ids=repr) + def test_put_if_match(self, store: StoreAPI, case: PutCase) -> None: + _check_put(store, case) + + +class NoConditionalPutSuite: + """For backends whose `put_if_match` raises, because the store underneath can't do it.""" + + def test_put_if_match_raises(self, store: StoreAPI) -> None: + current = store.put("k", VALUE).generation + with pytest.raises(UnsupportedOperationError): + store.put_if_match("k", b"new", current) + + +class ConditionalDeleteSuite: + """For backends with `delete_if_match` (`DeletableIfMatch`).""" + + @pytest.mark.parametrize("case", DELETE_IF_MATCH_CASES, ids=repr) + def test_delete_if_match(self, store: StoreAPI, case: DeleteCase) -> None: + _check_delete(store, case) + + +def _race_put( + store: Store[FullBackend], + generation: Generation, + barrier: threading.Barrier, + results: list[str], +) -> None: + barrier.wait() + if isinstance(store.put_if_match("d/k", b"new", generation), Written): + results.append("put") + + +def _race_delete( + store: Store[FullBackend], + generation: Generation, + barrier: threading.Barrier, + results: list[str], +) -> None: + barrier.wait() + if isinstance(delete_if_match(store, "d/k", generation), Deleted): + results.append("delete") + + +class ConditionalRaceSuite: + """For backends with conditional puts and deletes: racing ones from one generation apply once.""" + + def make_store(self, directory: Path) -> Store[FullBackend]: + raise NotImplementedError("Override make_store to return an empty store.") + + def test_conditional_put_and_delete_apply_once(self, tmp_path: Path) -> None: + store = self.make_store(tmp_path) + for _ in range(100): + generation = store.put("d/k", VALUE).generation + barrier = threading.Barrier(4) + results: list[str] = [] + threads = [ + threading.Thread(target=target, args=(store, generation, barrier, results)) + for target in (_race_put, _race_delete, _race_put, _race_delete) + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + assert len(results) == 1 + + +class KeyAndDirectorySuite: + """For backends that hold a key `a` and a key `a/b` at once (object stores, memory).""" + + def make_store(self, directory: Path) -> Store[FullBackend]: + raise NotImplementedError("Override make_store to return an empty store.") + + def test_list_dir_reports_a_name_that_is_both_key_and_prefix(self, store: StoreAPI) -> None: + store.put("a/b", VALUE) + store.put("a", VALUE) + assert set(store.list_dir()) == {Key("a"), Prefix("a")} + assert set(store.list()) == {"a", "a/b"} + + def test_delete_dir_keeps_a_key_with_the_directory_name(self, store: StoreAPI) -> None: + store.put("a", VALUE) + store.put("a/b", VALUE) + store.delete_dir("a") + assert set(store.list()) == {"a"} + + @given(keys=KEY_SETS, directory=LISTED_DIRECTORIES) + @settings( + max_examples=50, deadline=None, suppress_health_check=[HealthCheck.differing_executors] + ) + def test_listing_with_key_directories_matches_a_model( + self, tmp_path_factory: pytest.TempPathFactory, keys: set[str], directory: str + ) -> None: + _check_listing(self.make_store(tmp_path_factory.mktemp("store")), keys, directory) + + +class KeyDirectoryConflictSuite: + """For backends that can't hold a key and a directory with one name (filesystems).""" + + def test_key_over_a_directory_raises(self, store: StoreAPI) -> None: + store.put("a/b", VALUE) + with pytest.raises(KeyConflictError): + store.put("a", VALUE) + + def test_directory_under_a_key_raises(self, store: StoreAPI) -> None: + store.put("a", VALUE) + with pytest.raises(KeyConflictError): + store.put("a/b", VALUE) + + def test_put_many_with_a_conflict_raises_batch_error(self, store: StoreAPI) -> None: + with pytest.raises(BatchError) as caught: + store.put_many([("a", VALUE), ("a/b", VALUE)]) + error = caught.value + assert [type(e) for e in error.exceptions] == [KeyConflictError] + ((key, result),) = error.completed + assert isinstance(result, Written) + assert error.remaining == (({"a", "a/b"} - {key}).pop(),) + assert set(store.list()) == {key} + + def test_reads_under_a_key_find_nothing(self, store: StoreAPI) -> None: + store.put("a", VALUE) + assert isinstance(store.get("a/b"), Missing) + assert isinstance(store.get_if_changed("a/b", _stale()), Missing) + assert isinstance(store.get_into("a/b", bytearray(len(VALUE))), Missing) + assert isinstance(store.head("a/b"), Missing) + assert not store.exists("a/b") + assert isinstance(store.get_ranges("a/b", [SuffixRequest(1)]), Missing) + assert isinstance(store.delete("a/b"), Missing) + assert set(store.list("a")) == set() + assert set(store.list_dir("a")) == set() + assert set(store.list_dir("a/b")) == set() + + +class PickleSuite: + """For backends whose stores can be pickled.""" + + @pytest.mark.parametrize(("root", "read_only"), [("", False), ("a", True), ("a/b", False)]) + def test_pickle_round_trip(self, store: StoreAPI, root: str, read_only: bool) -> None: + store.put("a/b/k", VALUE) + view = unwrap((store / root).with_read_only(read_only)) + restored = pickle.loads(pickle.dumps(view)) + assert (restored.root, restored.read_only, restored.closed) == (root, read_only, False) + assert set(restored.list()) == set(view.list()) + + def test_pickle_keeps_closed(self, store: StoreAPI) -> None: + store.close() + assert pickle.loads(pickle.dumps(unwrap(store))).closed diff --git a/packages/zarr-storage/tests/__init__.py b/packages/zarr-storage/tests/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/packages/zarr-storage/tests/test_batch.py b/packages/zarr-storage/tests/test_batch.py new file mode 100644 index 0000000000..40b6b43c9b --- /dev/null +++ b/packages/zarr-storage/tests/test_batch.py @@ -0,0 +1,111 @@ +"""The runners behind every batch: windows, ordering and the failure rule.""" + +from __future__ import annotations + +import asyncio +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from typing import TYPE_CHECKING + +import pytest + +from zarr_storage import BatchError +from zarr_storage._batch import run_many, run_many_async + +if TYPE_CHECKING: + from collections.abc import AsyncIterator + + +class _Tracker: + """Squares its argument, recording what started and the most calls ever running at once.""" + + def __init__(self, fails: int | None = None) -> None: + self._fails = fails + self._lock = threading.Lock() + self._running = 0 + self.most = 0 + self.started: set[int] = set() + + def _enter(self, x: int) -> None: + with self._lock: + self.started.add(x) + self._running += 1 + self.most = max(self.most, self._running) + + def _exit(self) -> None: + with self._lock: + self._running -= 1 + + def __call__(self, x: int) -> int: + self._enter(x) + try: + if x == self._fails: + raise ValueError(x) + time.sleep(0.01) + return x * x + finally: + self._exit() + + async def call_async(self, x: int) -> int: + self._enter(x) + try: + if x == self._fails: + raise ValueError(x) + await asyncio.sleep(0.01) + return x * x + finally: + self._exit() + + +def _run(tracker: _Tracker, mode: str, window: int, n: int, into: list[tuple[int, int]]) -> None: + """Run `tracker` over `range(n)`, appending each result to `into` as it is yielded.""" + requests = [(x,) for x in range(n)] + if mode == "serial": + into.extend(run_many(tracker, requests)) + elif mode == "threads": + with ThreadPoolExecutor(8) as executor: + into.extend(run_many(tracker, requests, executor, window)) + else: + asyncio.run(_collect(run_many_async(tracker.call_async, requests, window), into)) + + +async def _collect(results: AsyncIterator[tuple[int, int]], into: list[tuple[int, int]]) -> None: + async for pair in results: + into.append(pair) # noqa: PERF401 -- a failure keeps the pairs before it + + +@pytest.mark.parametrize( + ("mode", "window"), + [("serial", 1), ("threads", 1), ("threads", 3), ("async", 1), ("async", 3)], +) +@pytest.mark.parametrize("n", [0, 1, 7]) +def test_runner(mode: str, window: int, n: int) -> None: + """Every request runs once, its result keyed by its index, never more than `window` at once.""" + tracker = _Tracker() + results: list[tuple[int, int]] = [] + _run(tracker, mode, window, n, results) + assert sorted(results) == [(x, x * x) for x in range(n)] + assert tracker.most <= window + + +@pytest.mark.parametrize(("mode", "window"), [("serial", 1), ("threads", 3), ("async", 3)]) +def test_runner_stops_starting_after_a_failure(mode: str, window: int) -> None: + """The failure's peers already running finish and are yielded; nothing new starts.""" + tracker = _Tracker(fails=0) + results: list[tuple[int, int]] = [] + with pytest.raises(ExceptionGroup) as caught: + _run(tracker, mode, window, 10, results) + assert [type(e) for e in caught.value.exceptions] == [ValueError] + assert tracker.started == set(range(window)) + assert sorted(results) == [(x, x * x) for x in range(1, window)] + + +def test_batch_error_keeps_its_report_when_split() -> None: + """`except*` splits a group; each part is still a `BatchError` with the batch's report.""" + error = BatchError("x", [ValueError(), KeyError()], completed=(("a", 1),), remaining=("b",)) + values, rest = error.split(ValueError) + for part, kind in [(values, ValueError), (rest, KeyError)]: + assert isinstance(part, BatchError) + assert [type(e) for e in part.exceptions] == [kind] + assert (part.completed, part.remaining) == ((("a", 1),), ("b",)) diff --git a/packages/zarr-storage/tests/test_local.py b/packages/zarr-storage/tests/test_local.py new file mode 100644 index 0000000000..a041d7a058 --- /dev/null +++ b/packages/zarr-storage/tests/test_local.py @@ -0,0 +1,392 @@ +from __future__ import annotations + +import errno +import multiprocessing +import os +import threading +import time +from typing import TYPE_CHECKING + +import pytest + +from zarr_storage import ( + Found, + Generation, + InvalidKeyError, + Key, + KeyConflictError, + Prefix, + Rejected, + Store, + UnsupportedOperationError, + Written, +) +from zarr_storage.backends import LocalBackend +from zarr_storage.backends._local import RESERVED_SUFFIX, _directory_lock, segment_problem +from zarr_storage.testing import ( + ConditionalDeleteSuite, + ConditionalPutSuite, + ConditionalRaceSuite, + KeyDirectoryConflictSuite, + PickleSuite, + StoreSuite, + UniqueGenerationsSuite, +) + +if TYPE_CHECKING: + from collections.abc import Callable + from pathlib import Path + + +class TestLocalBackend( + StoreSuite, + UniqueGenerationsSuite, + ConditionalPutSuite, + ConditionalDeleteSuite, + ConditionalRaceSuite, + KeyDirectoryConflictSuite, + PickleSuite, +): + def make_store(self, directory: Path) -> Store[LocalBackend]: + return Store(LocalBackend(directory)) + + +def test_listing_skips_backend_files(tmp_path: Path) -> None: + """Lock files and temporary files are never listed.""" + store = Store(LocalBackend(tmp_path)) + store.put("a/k", b"x") + (tmp_path / "a" / f"k.123{RESERVED_SUFFIX}").write_bytes(b"partial") + assert (tmp_path / "a" / f"lock{RESERVED_SUFFIX}").exists() + assert set(store.list()) == {"a/k"} + assert set(store.list_dir("a")) == {Key("k")} + + +def test_delete_leaves_the_directory(tmp_path: Path) -> None: + """Deleting a key leaves its directory, which `list_dir` reports as an empty `Prefix`.""" + store = Store(LocalBackend(tmp_path)) + store.put("a/b/c", b"x") + store.delete("a/b/c") + assert set(store.list()) == set() + assert set(store.list_dir("a")) == {Prefix("b")} + + +def test_failed_put_leaves_no_staged_files(tmp_path: Path) -> None: + store = Store(LocalBackend(tmp_path)) + store.put("k", b"x") + assert isinstance(store.put_if_absent("k", b"y"), Rejected) + assert list((tmp_path / f"staging{RESERVED_SUFFIX}").iterdir()) == [] + + +def test_delete_dir_of_the_root_keeps_backend_files(tmp_path: Path) -> None: + store = Store(LocalBackend(tmp_path)) + store.put("k", b"x") + store.delete_dir("") + assert {p.name for p in tmp_path.iterdir()} == { + f"lock{RESERVED_SUFFIX}", + f"staging{RESERVED_SUFFIX}", + } + assert isinstance(store.put_if_absent("k", b"y"), Written) + + +def test_put_recreates_a_removed_root(tmp_path: Path) -> None: + """An outer backend can remove a nested backend's root; the nested one recreates it.""" + outer = Store(LocalBackend(tmp_path)) + inner = Store(LocalBackend(tmp_path / "sub")) + outer.delete_dir("sub") + inner.put("k", b"x") + assert outer.exists("sub/k") + + +@pytest.mark.parametrize( + ("segment", "windows", "problem"), + [ + ("a", False, None), + ("a", True, None), + ("a.b", True, None), + ("con", False, None), + ("a:b", False, None), + ("a\\b", False, None), + (f"a{RESERVED_SUFFIX}", False, "reserved"), + (f"a{RESERVED_SUFFIX}", True, "reserved"), + ("a:b", True, "forbids"), + ("a\\b", True, "forbids"), + ("a?", True, "forbids"), + ("a\x01", True, "forbids"), + ("NUL", True, "device"), + ("con", True, "device"), + ("com1.txt", True, "device"), + ("LPT9", True, "device"), + ("COM¹", True, "device"), + ("a.", True, "strips"), + ("a ", True, "strips"), + ], +) +def test_segment_problem(segment: str, windows: bool, problem: str | None) -> None: + found = segment_problem(segment, windows=windows) + if problem is None: + assert found is None + else: + assert found is not None + assert problem in found + + +def test_missing_root_without_create_raises(tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError): + LocalBackend(tmp_path / "missing", create=False) + + +def test_reserved_key_raises(tmp_path: Path) -> None: + with pytest.raises(InvalidKeyError, match="reserved"): + Store(LocalBackend(tmp_path)).put(f"a{RESERVED_SUFFIX}/b", b"x") + + +def test_overlong_key_raises(tmp_path: Path) -> None: + with pytest.raises(InvalidKeyError, match="too long"): + Store(LocalBackend(tmp_path)).put("a" * 1000, b"x") + + +def test_zero_concurrency_raises(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="concurrency"): + LocalBackend(tmp_path, concurrency=0) + + +def _create( + store: Store[LocalBackend], key: str, barrier: threading.Barrier, results: list[bool] +) -> None: + barrier.wait() + results.append(isinstance(store.put_if_absent(key, b"x"), Written)) + + +@pytest.mark.parametrize("overlapping_roots", [False, True]) +def test_concurrent_creates_through_overlapping_roots( + tmp_path: Path, overlapping_roots: bool +) -> None: + """Of many racing `put_if_absent` writes to one key, exactly one applies, even through + backends whose roots overlap.""" + outer = Store(LocalBackend(tmp_path)) + inner = Store(LocalBackend(tmp_path / "sub")) if overlapping_roots else outer / "sub" + for round_ in range(50): + key = f"k{round_}" + barrier = threading.Barrier(8) + results: list[bool] = [] + threads = [ + threading.Thread(target=_create, args=(store, path, barrier, results)) + for store, path in [(outer, f"sub/{key}"), (inner, key)] * 4 + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + assert results.count(True) == 1 + + +def _create_in_process(root: str, key: str, start: threading.Event) -> bool: + start.wait() + return isinstance(Store(LocalBackend(root)).put_if_absent(key, b"x"), Written) + + +def test_concurrent_creates_apply_once_across_processes(tmp_path: Path) -> None: + ctx = multiprocessing.get_context("spawn") + with ctx.Manager() as manager, ctx.Pool(4) as pool: + for round_ in range(5): + start = manager.Event() + pending = [ + pool.apply_async(_create_in_process, (str(tmp_path), f"k{round_}", start)) + for _ in range(4) + ] + start.set() + assert [p.get(timeout=60) for p in pending].count(True) == 1 + + +def _put_many(store: Store[LocalBackend], errors: list[BaseException]) -> None: + try: + for i in range(200): + store.put(f"d/k{i}", b"x") + except BaseException as e: # noqa: BLE001 -- the test reports any failure + errors.append(e) + + +def test_delete_dir_racing_put_never_breaks_the_put(tmp_path: Path) -> None: + """A put whose directory a concurrent `delete_dir` removes starts over instead of failing.""" + store = Store(LocalBackend(tmp_path)) + errors: list[BaseException] = [] + writer = threading.Thread(target=_put_many, args=(store, errors)) + writer.start() + while writer.is_alive(): + store.delete_dir("d") + time.sleep(0.0001) + writer.join() + assert errors == [] + + +def _hold_lock(directory: Path, held: threading.Event, release: threading.Event) -> None: + with _directory_lock(directory): + held.set() + release.wait() + + +def _record_lock_file(directory: Path, entered: threading.Event, saw: list[bool]) -> None: + with _directory_lock(directory): + saw.append((directory / f"lock{RESERVED_SUFFIX}").exists()) + entered.set() + + +def test_lock_waiter_retries_when_the_lock_file_is_removed(tmp_path: Path) -> None: + """A waiter that wakes holding a lock file someone removed takes the new lock file instead.""" + held, release, entered = threading.Event(), threading.Event(), threading.Event() + saw: list[bool] = [] + holder = threading.Thread(target=_hold_lock, args=(tmp_path, held, release)) + holder.start() + held.wait() + waiter = threading.Thread(target=_record_lock_file, args=(tmp_path, entered, saw)) + waiter.start() + time.sleep(0.1) + assert not entered.is_set() + (tmp_path / f"lock{RESERVED_SUFFIX}").unlink() + release.set() + holder.join() + waiter.join(timeout=5) + assert saw == [True] + + +class _FlakyReplace: + """`os.replace` that fails with the given errors first, then renames.""" + + def __init__(self, failures: list[OSError]) -> None: + self._failures = failures + self._real = os.replace + + def __call__(self, src: str | os.PathLike[str], dst: str | os.PathLike[str]) -> None: + if self._failures: + raise self._failures.pop() + self._real(src, dst) + + +def test_put_retries_a_rename_into_a_removed_directory( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + store = Store(LocalBackend(tmp_path)) + failures = [OSError(errno.ENOENT, "removed"), OSError(errno.EINVAL, "removed")] + monkeypatch.setattr(os, "replace", _FlakyReplace(failures)) + store.put("a/k", b"x") + result = store.get("a/k") + assert isinstance(result, Found) + assert bytes(result.value) == b"x" + + +class _PausedReplace: + """`os.replace` that, for one destination name, waits for a signal before renaming.""" + + def __init__(self, name: str) -> None: + self._name = name + self._real = os.replace + self.reached = threading.Event() + self.release = threading.Event() + + def __call__(self, src: str | os.PathLike[str], dst: str | os.PathLike[str]) -> None: + if os.fspath(dst).endswith(self._name): + self.reached.set() + self.release.wait() + self._real(src, dst) + + +@pytest.mark.parametrize("ancestor", ["", "P", "P/D"]) +def test_delete_dir_waits_for_a_write_in_progress( + tmp_path: Path, ancestor: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """`delete_dir` of any ancestor waits for a write that holds its directory's lock, then + removes what it wrote; a create after the delete applies.""" + store = Store(LocalBackend(tmp_path)) + store.put("P/D/other", b"x") + paused = _PausedReplace("/k") + monkeypatch.setattr(os, "replace", paused) + outcomes: dict[str, bool] = {} + barrier = threading.Barrier(1) + writer = threading.Thread(target=_put_outcome, args=(store, "P/D/k", barrier, outcomes)) + writer.start() + assert paused.reached.wait(5) + deleter = threading.Thread(target=store.delete_dir, args=(ancestor,)) + deleter.start() + time.sleep(0.2) + assert deleter.is_alive() + paused.release.set() + writer.join(5) + deleter.join(5) + assert outcomes == {"P/D/k": True} + assert not store.exists("P/D/k") + assert isinstance(store.put_if_absent("P/D/k", b"y"), Written) + + +def test_delete_dir_leaves_a_symlinked_directory_alone(tmp_path: Path) -> None: + outside = tmp_path / "outside" + outside.mkdir() + (outside / "keep").write_bytes(b"x") + store = Store(LocalBackend(tmp_path / "root")) + (tmp_path / "root" / "link").symlink_to(outside) + store.delete_dir("link") + assert (outside / "keep").exists() + + +def test_delete_dir_removes_a_nested_backends_staging(tmp_path: Path) -> None: + outer = Store(LocalBackend(tmp_path)) + inner = Store(LocalBackend(tmp_path / "sub")) + inner.put("k", b"x") + outer.delete_dir("sub") + assert not (tmp_path / "sub").exists() + + +def _put_outcome( + store: Store[LocalBackend], key: str, barrier: threading.Barrier, outcomes: dict[str, bool] +) -> None: + barrier.wait() + try: + store.put(key, b"x") + outcomes[key] = True + except KeyConflictError: + outcomes[key] = False + + +def test_reclaim_never_removes_an_acknowledged_key(tmp_path: Path) -> None: + """A put of `a` racing a put of `a/b`, over a directory `a` a delete left empty: exactly one + applies, and the one that applied is stored.""" + store = Store(LocalBackend(tmp_path)) + for _ in range(300): + store.delete_dir("") + store.put("a/x", b"x") + store.delete("a/x") + barrier = threading.Barrier(2) + outcomes: dict[str, bool] = {} + threads = [ + threading.Thread(target=_put_outcome, args=(store, key, barrier, outcomes)) + for key in ("a", "a/b") + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + stored = set(store.list()) + assert sum(outcomes.values()) == 1 + assert stored == {key for key, applied in outcomes.items() if applied} + + +def _coarse(self: LocalBackend) -> bool: + return False + + +@pytest.mark.parametrize( + "write", + [ + lambda s: s.put_if_match("k", b"x", Generation(object())), + lambda s: s.delete_if_match("k", Generation(object())), + ], + ids=["put_if_match", "delete_if_match"], +) +def test_if_match_on_a_filesystem_that_rounds_times_raises( + tmp_path: Path, write: Callable[[Store[LocalBackend]], object], monkeypatch: pytest.MonkeyPatch +) -> None: + """Generations need exact times only to compare against; `put_if_absent` still works.""" + monkeypatch.setattr(LocalBackend, "_probe_exact_times", _coarse) + store = Store(LocalBackend(tmp_path)) + with pytest.raises(UnsupportedOperationError, match="rounds file times"): + write(store) + assert isinstance(store.put_if_absent("k", b"x"), Written) diff --git a/packages/zarr-storage/tests/test_memory.py b/packages/zarr-storage/tests/test_memory.py new file mode 100644 index 0000000000..694018ad4a --- /dev/null +++ b/packages/zarr-storage/tests/test_memory.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +import pickle +from typing import TYPE_CHECKING + +from zarr_storage import Store +from zarr_storage.backends import MemoryBackend +from zarr_storage.testing import ( + ConditionalDeleteSuite, + ConditionalPutSuite, + ConditionalRaceSuite, + KeyAndDirectorySuite, + PickleSuite, + StoreSuite, + UniqueGenerationsSuite, +) + +if TYPE_CHECKING: + from pathlib import Path + + +class TestMemoryBackend( + StoreSuite, + UniqueGenerationsSuite, + ConditionalPutSuite, + ConditionalDeleteSuite, + ConditionalRaceSuite, + KeyAndDirectorySuite, + PickleSuite, +): + def make_store(self, directory: Path) -> Store[MemoryBackend]: + return Store(MemoryBackend()) + + +def test_pickled_copy_is_independent() -> None: + """A pickled memory store carries a copy of its values; writes to either don't reach the other.""" + store = Store(MemoryBackend()) + store.put("a", b"1") + copy = pickle.loads(pickle.dumps(store)) + copy.put("b", b"2") + store.put("c", b"3") + assert set(store.list()) == {"a", "c"} + assert set(copy.list()) == {"a", "b"} diff --git a/packages/zarr-storage/tests/test_obstore.py b/packages/zarr-storage/tests/test_obstore.py new file mode 100644 index 0000000000..213203297a --- /dev/null +++ b/packages/zarr-storage/tests/test_obstore.py @@ -0,0 +1,194 @@ +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING + +import obstore +import pytest +from obstore.exceptions import GenericError +from obstore.store import LocalStore, MemoryStore + +from zarr_storage import Found, ObjectMetadata, RangeRequest, Store, Unchanged +from zarr_storage.backends import ObjectStoreBackend +from zarr_storage.backends import _obstore as obstore_backend +from zarr_storage.backends._obstore import _empty_read_or_raise, _generation +from zarr_storage.testing import ( + ConditionalPutSuite, + KeyAndDirectorySuite, + KeyDirectoryConflictSuite, + NoConditionalPutSuite, + PickleSuite, + StoreSuite, + UniqueGenerationsSuite, +) + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Callable + from pathlib import Path + + +_SYNC_CALLS = ["get", "get_ranges", "head", "put", "delete", "list_with_delimiter", "copy"] + + +class _NotOnTheEventLoop: + """An obstore function that refuses to run on a thread that is running an event loop.""" + + def __init__(self, real: Callable[..., object]) -> None: + self._real = real + + def __call__(self, *args: object, **kwargs: object) -> object: + try: + asyncio.get_running_loop() + except RuntimeError: + return self._real(*args, **kwargs) + raise AssertionError("A blocking obstore call ran inside an async method.") + + +class NoBlockingCallsInAsyncRuns: + """Fail any async run of a test that makes a blocking obstore call.""" + + @pytest.fixture(autouse=True) + def _forbid_blocking_calls( + self, request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch + ) -> None: + callspec = getattr(request.node, "callspec", None) + params = {} if callspec is None else callspec.params + if "async" in (params.get("store"), params.get("api")): + for name in _SYNC_CALLS: + monkeypatch.setattr(obstore, name, _NotOnTheEventLoop(getattr(obstore, name))) + + +class TestObjectStoreMemory( + StoreSuite, + UniqueGenerationsSuite, + ConditionalPutSuite, + KeyAndDirectorySuite, + NoBlockingCallsInAsyncRuns, +): + def make_store(self, directory: Path) -> Store[ObjectStoreBackend]: + return Store(ObjectStoreBackend(MemoryStore())) + + +class TestObjectStoreLocal( + StoreSuite, + NoConditionalPutSuite, + KeyDirectoryConflictSuite, + PickleSuite, + NoBlockingCallsInAsyncRuns, +): + def make_store(self, directory: Path) -> Store[ObjectStoreBackend]: + return Store(ObjectStoreBackend(LocalStore(directory))) + + +_META = ObjectMetadata(10, _generation("e", None), None) + + +def test_refused_empty_read_is_empty() -> None: + result = _empty_read_or_raise(_META, RangeRequest(10, 12), GenericError("refused")) + assert isinstance(result, Found) + assert (bytes(result.value), result.generation) == (b"", _META.generation) + + +def test_refused_readable_range_reraises() -> None: + error = GenericError("refused") + with pytest.raises(GenericError, match="refused"): + _empty_read_or_raise(_META, RangeRequest(2, 5), error) + + +class _FailFor: + """An obstore function that raises a transient error for one path and works for the rest.""" + + def __init__(self, real: Callable[..., object], path: str) -> None: + self._real = real + self._path = path + + def __call__( + self, store: object, path: str | None = None, *args: object, **kwargs: object + ) -> object: + if path == self._path: + raise GenericError("injected: 503 Service Unavailable") + return self._real(store, path, *args, **kwargs) + + +def _list_a(store: Store[ObjectStoreBackend]) -> object: + return list(store.list("a")) + + +def _list_dir_a(store: Store[ObjectStoreBackend]) -> object: + return list(store.list_dir("a")) + + +def _get_ab(store: Store[ObjectStoreBackend]) -> object: + return store.get("a/b") + + +def _head_ab(store: Store[ObjectStoreBackend]) -> object: + return store.head("a/b") + + +async def _collect(items: AsyncIterator[object]) -> list[object]: + return [item async for item in items] + + +def _list_a_async(store: Store[ObjectStoreBackend]) -> object: + return asyncio.run(_collect(store.as_async().list("a"))) + + +def _list_dir_a_async(store: Store[ObjectStoreBackend]) -> object: + return asyncio.run(_collect(store.as_async().list_dir("a"))) + + +def _get_ab_async(store: Store[ObjectStoreBackend]) -> object: + return asyncio.run(store.as_async().get("a/b")) + + +@pytest.mark.parametrize( + ("operation", "failing", "path"), + [ + (_list_a, "list", "a"), + (_list_dir_a, "list_with_delimiter", "a"), + (_get_ab, "get", "a/b"), + (_head_ab, "head", "a/b"), + (_list_a_async, "list", "a"), + (_list_dir_a_async, "list_with_delimiter_async", "a"), + (_get_ab_async, "get_async", "a/b"), + ], + ids=["list", "list_dir", "get", "head", "list_async", "list_dir_async", "get_async"], +) +def test_errors_on_object_stores_are_not_read_as_absent( + operation: Callable[[Store[ObjectStoreBackend]], object], + failing: str, + path: str, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Only a filesystem store's refusals mean "under a key"; elsewhere an error is an error. + + Only the operation's own call fails, so a check of the path's parents would succeed and could + wrongly turn the error into "absent". + """ + store = Store(ObjectStoreBackend(MemoryStore())) + store.put("a", b"x") + store.put("a/b", b"x") + monkeypatch.setattr(obstore, failing, _FailFor(getattr(obstore, failing), path)) + with pytest.raises(GenericError): + operation(store) + + +def _no_if_none_match(generation: object) -> dict[str, str]: + return {} + + +@pytest.mark.parametrize("api", ["sync", "async"]) +def test_get_if_changed_without_if_none_match_is_unchanged( + api: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """A store that can't skip the read (no ETag, or it ignores `If-None-Match`) still reports + `Unchanged` for a value that has the generation.""" + monkeypatch.setattr(obstore_backend, "_if_none_match", _no_if_none_match) + store = Store(ObjectStoreBackend(MemoryStore())) + generation = store.put("k", b"x").generation + if api == "sync": + result = store.get_if_changed("k", generation) + else: + result = asyncio.run(store.as_async().get_if_changed("k", generation)) + assert result == Unchanged(generation, result.read_at) diff --git a/packages/zarr-storage/tests/test_store.py b/packages/zarr-storage/tests/test_store.py new file mode 100644 index 0000000000..63582e5066 --- /dev/null +++ b/packages/zarr-storage/tests/test_store.py @@ -0,0 +1,337 @@ +"""`Store` behavior that doesn't depend on a backend's own implementation.""" + +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING + +import pytest + +from zarr_storage import ( + AsyncStore, + BatchError, + Copied, + Found, + FoundRanges, + Generation, + Missing, + ObjectMetadata, + OffsetRequest, + RangeRequest, + Store, + SuffixRequest, + Unchanged, + Written, +) +from zarr_storage._byte_range import resolve +from zarr_storage.backends import MemoryBackend + +if TYPE_CHECKING: + from collections.abc import Iterator, Sequence + + from zarr_storage import ByteRange + + +class _ReadOnlyBackend: + """A backend with only the `Readable` capability, over a fixed dict.""" + + def __init__(self, values: dict[str, bytes]) -> None: + self._values = values + + @property + def closed(self) -> bool: + return False + + @property + def writable(self) -> bool: + return False + + def close(self) -> None: + pass + + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + data = self._values.get(path) + if data is None: + return Missing() + start, end = resolve(byte_range, len(data)) + return Found(memoryview(data)[start:end], Generation(0)) + + def get_if_changed( + self, path: str, generation: Generation, byte_range: ByteRange | None + ) -> Found | Missing | Unchanged: + raise NotImplementedError + + def get_ranges(self, path: str, ranges: Sequence[ByteRange]) -> FoundRanges | Missing: + raise NotImplementedError + + def head(self, path: str) -> ObjectMetadata | Missing: + data = self._values.get(path) + return Missing() if data is None else ObjectMetadata(len(data), Generation(0), None) + + +class _DeviceBuffer: + """Stands in for device memory (a GPU array): no buffer protocol, so host code can't view it.""" + + def __init__(self, size: int) -> None: + self._data = bytearray(size) + + def __len__(self) -> int: + return len(self._data) + + def upload(self, data: memoryview) -> None: + self._data[: len(data)] = data + + def download(self, n: int) -> bytes: + return bytes(self._data[:n]) + + +class _DeviceBackend(_ReadOnlyBackend): + """A backend that reads into `_DeviceBuffer`s only, as a GPU direct storage backend would.""" + + def get_into( + self, path: str, out: _DeviceBuffer, byte_range: ByteRange | None + ) -> Copied | Missing: + found = self.get(path, byte_range) + if isinstance(found, Missing): + return found + if len(found.value) > len(out): + raise ValueError("The output buffer is too short.") + out.upload(found.value) + return Copied(len(found.value), found.generation) + + +@pytest.mark.parametrize( + ("byte_range", "expected"), + [(None, b"0123"), (RangeRequest(1, 3), b"12"), (SuffixRequest(1), b"3")], +) +def test_get_into_a_device_buffer(byte_range: ByteRange | None, expected: bytes) -> None: + """`get_into` takes the buffer type the backend reads into, and applies the store's root.""" + store = Store(_DeviceBackend({"a/k": b"0123"})) / "a" + out = _DeviceBuffer(4) + assert store.get_into("k", out, byte_range=byte_range) == Copied(len(expected), Generation(0)) + assert out.download(len(expected)) == expected + assert store.get_into("missing", out) == Missing() + + +def test_get_into_many_device_buffers() -> None: + """Without a batch method of its own, the backend's `get_into` serves `get_into_many`.""" + store = Store(_DeviceBackend({"a/k": b"0123", "a/j": b"45"})) / "a" + whole, part = _DeviceBuffer(4), _DeviceBuffer(1) + results = [ + (request[0], result) + for request, result in store.get_into_many( + [("k", whole), ("j", part, SuffixRequest(1)), ("missing", _DeviceBuffer(1))] + ) + ] + assert results == [ + ("k", Copied(4, Generation(0))), + ("j", Copied(1, Generation(0))), + ("missing", Missing()), + ] + assert (whole.download(4), part.download(1)) == (b"0123", b"5") + + +class _BatchBackend(_ReadOnlyBackend): + """A backend with its own `get_many`, which answers in reverse and records what it was asked.""" + + def __init__(self, values: dict[str, bytes]) -> None: + super().__init__(values) + self.asked: list[tuple[str, ByteRange | None]] = [] + + def get_many( + self, requests: Sequence[tuple[str, ByteRange | None]] + ) -> Iterator[tuple[int, Found | Missing]]: + self.asked.extend(requests) + for index in reversed(range(len(requests))): + yield index, self.get(*requests[index]) + + +def test_get_many_uses_the_backends_batch_method() -> None: + backend = _BatchBackend({"r/a": b"0123"}) + results = list((Store(backend) / "r").get_many(["a", ("a", RangeRequest(1, 2)), "b"])) + assert backend.asked == [("r/a", None), ("r/a", RangeRequest(1, 2)), ("r/b", None)] + assert [(request, type(result)) for request, result in results] == [ + ("b", Missing), + (("a", RangeRequest(1, 2)), Found), + ("a", Found), + ] + + +class _FailingBackend(MemoryBackend): + """A memory backend whose `put` and `get` raise for the path `"r/bad"`.""" + + def __init__(self) -> None: + super().__init__() + self.attempted: list[str] = [] + + def put(self, path: str, value: memoryview) -> Written: + self.attempted.append(path) + if path == "r/bad": + raise OSError("refused") + return super().put(path, value) + + def get(self, path: str, byte_range: ByteRange | None) -> Found | Missing: + self.attempted.append(path) + if path == "r/bad": + raise OSError("refused") + return super().get(path, byte_range) + + +def test_put_many_failure_stops_the_batch_and_reports_it() -> None: + backend = _FailingBackend() + store = Store(backend) / "r" + with pytest.raises(BatchError) as caught: + store.put_many([("ok", b"1"), ("bad", b"2"), ("later", b"3")]) + error = caught.value + assert [type(e) for e in error.exceptions] == [OSError] + assert [(key, type(result)) for key, result in error.completed] == [("ok", Written)] + assert error.remaining == ("bad", "later") + assert backend.attempted == ["r/ok", "r/bad"] + assert set(store.list()) == {"ok"} + + +def test_get_many_failure_raises_after_the_reads_before_it() -> None: + backend = _FailingBackend() + store = Store(backend) / "r" + store.put("ok", b"1") + backend.attempted.clear() + results = store.get_many(["ok", "bad", "later"]) + assert next(results)[0] == "ok" + with pytest.raises(BatchError) as caught: + next(results) + assert (caught.value.completed, caught.value.remaining) == ((), ("bad", "later")) + assert backend.attempted == ["r/ok", "r/bad"] + + +def _static_buffer_checks(host: Store[MemoryBackend], device: Store[_DeviceBackend]) -> None: + """Never run; pyright checks that each marked call is a type error.""" + host.get_into("k", bytearray(4)) + device.get_into("k", _DeviceBuffer(4)) + device.get_into("k", bytearray(4)) # pyright: ignore[reportArgumentType] + host.get_into("k", _DeviceBuffer(4)) # pyright: ignore[reportArgumentType] + Store(_ReadOnlyBackend({})).get_into("k", bytearray(4)) # pyright: ignore[reportAttributeAccessIssue] + device.get_into_many([("k", _DeviceBuffer(4))]) + device.get_into_many([("k", bytearray(4))]) # pyright: ignore[reportArgumentType] + host.get_into_many([("k", _DeviceBuffer(4), None)]) # pyright: ignore[reportArgumentType] + Store(_ReadOnlyBackend({})).put_many([("k", b"")]) # pyright: ignore[reportAttributeAccessIssue] + + +def test_store_over_unwritable_backend_is_read_only() -> None: + store = Store(_ReadOnlyBackend({})) + assert store.read_only + assert (store / "a").read_only + + +@pytest.mark.parametrize( + "make", + [ + lambda: RangeRequest(1.5, 3), # pyright: ignore[reportArgumentType] + lambda: RangeRequest(True, 3), + lambda: OffsetRequest("1"), # pyright: ignore[reportArgumentType] + lambda: SuffixRequest(2.0), # pyright: ignore[reportArgumentType] + ], + ids=["float", "bool", "str", "float-suffix"], +) +def test_byte_range_of_non_int_raises(make: object) -> None: + with pytest.raises(TypeError, match="must be an int"): + make() # pyright: ignore[reportCallIssue] + + +@pytest.mark.parametrize( + "make", + [ + lambda: RangeRequest(-1, 3), + lambda: RangeRequest(3, 1), + lambda: OffsetRequest(-1), + lambda: SuffixRequest(-1), + ], + ids=["negative-start", "end-before-start", "negative-offset", "negative-suffix"], +) +def test_byte_range_out_of_order_raises(make: object) -> None: + with pytest.raises(ValueError, match="<=|>="): + make() # pyright: ignore[reportCallIssue] + + +def test_store_is_not_value_equal() -> None: + backend = MemoryBackend() + assert Store(backend) != Store(backend) + + +@pytest.mark.parametrize("read_only", [False, True]) +@pytest.mark.parametrize("root", ["", "a/b"]) +def test_views_across_sync_and_async(root: str, read_only: bool) -> None: + """`as_async` and `as_sync` keep the backend, root and read-only flag, and don't own.""" + backend = MemoryBackend() + store = Store(backend, root=root, read_only=read_only) + async_view = store.as_async() + sync_view = async_view.as_sync() + for view in (async_view, sync_view): + assert (view.backend, view.root, view.read_only) == (backend, root, read_only) + assert view.shares_backend(store) + assert view.overlaps(store) + assert isinstance(async_view, AsyncStore) + assert isinstance(sync_view, Store) + asyncio.run(async_view.aclose()) + sync_view.close() + assert not backend.closed + store.close() + assert backend.closed + + +class _AsyncClosingBackend(MemoryBackend): + """A backend whose cleanup is a coroutine; records which close ran.""" + + def __init__(self) -> None: + super().__init__() + self.closed_by: list[str] = [] + + def close(self) -> None: + self.closed_by.append("close") + super().close() + + async def aclose(self) -> None: + self.closed_by.append("aclose") + super().close() + + +class _SyncClosingBackend(MemoryBackend): + def __init__(self) -> None: + super().__init__() + self.closed_by: list[str] = [] + + def close(self) -> None: + self.closed_by.append("close") + super().close() + + +async def _close_async(store: AsyncStore[_AsyncClosingBackend | _SyncClosingBackend]) -> None: + async with store: + pass + + +@pytest.mark.parametrize( + ("backend", "expected"), + [(_AsyncClosingBackend(), ["aclose"]), (_SyncClosingBackend(), ["close"])], + ids=["closable-async", "sync-close"], +) +def test_async_store_closes_its_backend( + backend: _AsyncClosingBackend | _SyncClosingBackend, expected: list[str] +) -> None: + """`AsyncStore` awaits a backend's `aclose` where it has one, and calls `close` otherwise.""" + asyncio.run(_close_async(AsyncStore(backend) / "derived")) + assert backend.closed_by == [] + asyncio.run(_close_async(AsyncStore(backend))) + assert backend.closed_by == expected + assert backend.closed + + +async def _static_surface_checks( + store: Store[MemoryBackend], astore: AsyncStore[MemoryBackend] +) -> None: + """Never run; pyright checks that each marked line is a type error.""" + store.get("k") + await astore.get("k") + store.get_async("k") # pyright: ignore[reportAttributeAccessIssue] + store.aclose() # pyright: ignore[reportAttributeAccessIssue] + astore.close() # pyright: ignore[reportAttributeAccessIssue] + AsyncStore(_ReadOnlyBackend({})).get("k") # pyright: ignore[reportAttributeAccessIssue] diff --git a/packages/zarr-storage/towncrier-template.md b/packages/zarr-storage/towncrier-template.md new file mode 100644 index 0000000000..d3616fe8c5 --- /dev/null +++ b/packages/zarr-storage/towncrier-template.md @@ -0,0 +1,130 @@ +{#- +This package's copy of towncrier's default Markdown template. The one +change: each entry's continuation lines are indented by four spaces, not +two, and entries are separated by a blank line, so an entry with a +second paragraph or a nested list stays one list item in Python-Markdown, +which renders the docs' release notes page, as well as in CommonMark. +Without both, Python-Markdown ends the list at the second paragraph and +runs the later entries together as one paragraph. +-#} +{#- +══════════════════════════════════════════════════════════════════════════════ +TOWNCRIER MARKDOWN TEMPLATE +══════════════════════════════════════════════════════════════════════════════ + +─── Macro: heading ───────────────────────────────────────────────────────── +Purpose: + Generates Markdown headings with the appropriate number of # characters. + Based on header_prefix (default: "#") and the level argument. + +Arguments: + level The relative heading level (1=#, 2=##, 3=###, etc.) +-#} +{%- macro heading(level) -%} + {{- "#" * ( header_prefix | length + level -1 ) }} +{%- endmacro -%} + +{#- +─── Variable: newline ────────────────────────────────────────────────────── +Purpose: + Consistent newline handling. -#} +{%- set newline = "\n" -%} + +{#- ════════════════════════ TEMPLATE GENERATION ════════════════════════ -#} +{#- ─── TITLE HEADING ─── #} +{#- render_title is false when title_format is specified in the config #} +{%- if render_title %} + {%- if versiondata.name %} + {{- heading(1) ~ " " ~ versiondata.name ~ " " ~ versiondata.version ~ " (" ~ versiondata.date ~ ")" ~ newline }} + {%- else %} + {{- heading(1) ~ " " ~ versiondata.version ~ " (" ~ versiondata.date ~ ")" ~ newline }} + {%- endif %} +{%- endif %} +{#- If title_format is specified, we start with a new line #} +{{- newline }} + +{%- for section, _ in sections.items() %} + {#- ─── SECTION HEADING ─── #} + {%- if section %} + {{- newline }} + {{- heading(2) ~ " " ~ section ~ newline }} + {{- newline }} + {%- endif %} + + {%- if sections[section] %} + + {%- for category, val in definitions.items() if category in sections[section] %} + {%- set issue_pks = [] %} + {#- ─── CATEGORY HEADING ─── #} + {#- Increase heading level if section is not present #} + {{- heading(3 if section else 2) ~" " ~ definitions[category]['name'] ~ newline }} + {{- newline }} + + {#- ─── RENDER ENTRIES ─── #} + {%- for text, values in sections[section][category].items() %} + {#- A blank line between entries: Python-Markdown starts a new list + item after an entry of several paragraphs only after one. #} + {%- if not loop.first %} + {{- newline }} + {%- endif %} + {#- Prepare the string of issue numbers (e.g., "#1, #9, #142") #} + {%- set issue_pks = [] %} + {%- for v_issue in values %} + {%- set _= issue_pks.append(v_issue.split(": ", 1)[0]) %} + {%- endfor %} + {%- set issues_list = issue_pks | join(", ") %} + + {#- Check if text contains a sublist #} + {%- set text_has_sublist = (("\n - " in text) or ("\n * " in text)) %} + + {#- CASE 1: No text, only issues #} + {#- Output: - #1, #9, #142 #} + {%- if not text and issues_list %} + {{- "- " ~ issues_list ~ newline }} + + {#- Cases where both text and issues exist #} + {%- elif text and issues_list %} + {%- if text_has_sublist %} + {#- CASE 3: Text with sublist #} + {#- Output: - TEXT\n\n (#1, #9, #142) #} + {{- "- " ~ (text | indent(2)) ~ newline ~ newline ~ " (" ~ issues_list ~ ")" ~ newline }} + {%- else %} + {#- CASE 2: Text, no sublist #} + {#- Output: - TEXT (#1, #9, #142) #} + {{- "- " ~ (text | indent(2)) ~ " (" ~ issues_list ~ ")" ~ newline }} + {%- endif %} + + {%- elif text %} + {#- Implicit Case: Text, but no issues #} + {#- Output: - TEXT #} + {{- "- " ~ (text | indent(2)) ~ newline }} + {%- endif %} + {%- endfor %} + + {#- New line between list and link references #} + {{- newline }} + + {#- Link references #} + {%- if issues_by_category[section][category] and "]: " in issues_by_category[section][category][0] %} + {%- for issue in issues_by_category[section][category] %} + {{- issue ~ newline }} + {%- endfor %} + {{- newline }} + {%- endif %} + + {#- No changes in this category #} + {%- if sections[section][category]|length == 0 %} + {{- newline }} + {{- "No significant changes." ~ newline * 2 }} + {%- endif %} + {%- endfor %} + {%- else %} + {#- No changes in this section #} + {{- "No significant changes." ~ newline * 2 }} + {%- endif %} +{%- endfor %} +{#- +Newline at the end of the rendered newsfile content. +In this way the there are 2 newlines between the latest release and the previous release content. +-#} +{{- newline -}} diff --git a/packages/zarr-storage/uv.lock b/packages/zarr-storage/uv.lock new file mode 100644 index 0000000000..4c68fc2367 --- /dev/null +++ b/packages/zarr-storage/uv.lock @@ -0,0 +1,275 @@ +version = 1 +revision = 5 +requires-python = ">=3.11" + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "hypothesis" +version = "6.170.15" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "sortedcontainers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/96/ec/59f14eb701087710888b66bfef2dc8bee9dd4a364c69a732db0cc1d3d2dc/hypothesis-6.170.15.tar.gz", hash = "sha256:a4a57e5fb106ecb29cad37e22e411d957e20d933bfb9fb029e1999b1deb8a0b6", size = 521997, upload-time = "2026-11-06T09:54:57.339Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/84/cf/3b56a81685915f6bf0d67b3a6c1139fd9bd4583066161823834e42b9eaa7/hypothesis-6.170.15-cp311-abi3-macosx_10_12_x86_64.whl", hash = "sha256:db8211e0a56fa29bf9e5d9ff9cd969a91ff4e5ffa8fb4983d3dd788253aeee2c", size = 800537, upload-time = "2026-11-06T09:53:36.823Z" }, + { url = "https://files.pythonhosted.org/packages/b6/09/477b3233cca8c447a6dc5fcc076a7294a4fdacb615d960695f6970d706dc/hypothesis-6.170.15-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:34e0859a53f20200f39db3f9cd92112eb6034c31dc670e88d56d8725f453562d", size = 795693, upload-time = "2026-11-06T09:54:14.601Z" }, + { url = "https://files.pythonhosted.org/packages/7c/7a/34e042f1545472abe29ecf9eebac6f748096580f6fb831817f097addbe29/hypothesis-6.170.15-cp311-abi3-manylinux1_i686.manylinux_2_5_i686.whl", hash = "sha256:5c5980821384206939c1c9d519b7adfb80b54d07e669ab0d36fe877f08cff07a", size = 1162432, upload-time = "2026-11-06T09:53:55.461Z" }, + { url = "https://files.pythonhosted.org/packages/3a/7f/95b7b04d2b7e45669db446ee10d97f5d04b143bba380559a7f3a7e0c5b1f/hypothesis-6.170.15-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f29339b5cd4c2a221885e8b7ddbd447b4dc7ee14f763a6f5aaff726d4d3fd5ed", size = 1124598, upload-time = "2026-11-06T09:54:26.015Z" }, + { url = "https://files.pythonhosted.org/packages/33/8e/896172d9706bfe2e09c9154fecb11c7b76e50014442783dd9d5b404c8c84/hypothesis-6.170.15-cp311-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl", hash = "sha256:d836a61c3dd94c6fc4677f4d0d95d30b92eb0ceab6f2b5c58e35c32545583170", size = 1155097, upload-time = "2026-11-06T09:54:47.123Z" }, + { url = "https://files.pythonhosted.org/packages/29/33/b8791c6a9b4232c3cc8f592d3e36175bcb4bad6203f7e461db0a4f5fa11b/hypothesis-6.170.15-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:d18ffd069360589a66ed125db62f12b490b3b9f6d47e5fd2505f1089d63a0f17", size = 1148988, upload-time = "2026-11-06T09:54:33.101Z" }, + { url = "https://files.pythonhosted.org/packages/87/2c/3ee87dd6fdec16ce55a4aeb5f72ae24756855cdc4ee87637fb68c4daeadc/hypothesis-6.170.15-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:7636424c7b1477fdd279fce5a210b67687a3024ac7759fe2e46a767a76f8761c", size = 1201778, upload-time = "2026-11-06T09:53:49.736Z" }, + { url = "https://files.pythonhosted.org/packages/6b/6a/f2757fde7d1b3b5dda6df3441d346b59c60627f2c216d966739289149903/hypothesis-6.170.15-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:99a1a4711b5276e5bfc3aa0cca034d0e45f8696a46ac8dce008308f580c18234", size = 1168007, upload-time = "2026-11-06T09:54:34.331Z" }, + { url = "https://files.pythonhosted.org/packages/8d/79/b6ff2a2840e4bcdea5c4cf1bcdd3803954a632beec16cb9d77a9ee8ddbb8/hypothesis-6.170.15-cp311-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:f545a04342a53008fa696a0592f0a026278ee61fab04f3e9e395d0c7eba42fef", size = 1124154, upload-time = "2026-11-06T09:53:24.354Z" }, + { url = "https://files.pythonhosted.org/packages/52/d0/19bc4a98e9583936dbe6e754014ad1133d3a43c9068a880c4e91ce384fad/hypothesis-6.170.15-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1f11d685322a56ee73c2a7566ec9d1195740313c3b965bad3121c340ec0e657b", size = 1299503, upload-time = "2026-11-06T09:53:12.86Z" }, + { url = "https://files.pythonhosted.org/packages/ee/06/fad2d1546c08eb4133f9478716bf6bc5f881e0d06b5b4f1faf34fd515a5a/hypothesis-6.170.15-cp311-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:f6d5d34fa6b56f236ad9659043123efb85391a8a5640f067c9de82b0be0bff73", size = 1428191, upload-time = "2026-11-06T09:54:44.43Z" }, + { url = "https://files.pythonhosted.org/packages/ca/cd/71d43284cca9c5145ceae04ddac1adfc57e9da57cf8efd30c0876844c9bf/hypothesis-6.170.15-cp311-abi3-musllinux_1_2_i686.whl", hash = "sha256:2ff9983a58647b03754fa3d1005e59a9c563e15256340fc062b4a34c7969d52a", size = 1382322, upload-time = "2026-11-06T09:54:06.405Z" }, + { url = "https://files.pythonhosted.org/packages/0f/56/5754189cd709d4eb34601f8ebb57f1d2381b2dd74f62352ba6b84bce4a12/hypothesis-6.170.15-cp311-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:6e461d7b392a630593134611ebb1fce6baba88dbbf4f07661a37ff58ff4e1b6a", size = 1278225, upload-time = "2026-11-06T09:52:59.772Z" }, + { url = "https://files.pythonhosted.org/packages/7e/a9/5224ea74356981f01222c42f0acc55821fb5aadf3f1f15adc797c7eefbd2/hypothesis-6.170.15-cp311-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:62c0675a837bbe0f3f7c7a89b701bee159759840c23b9bec9d565d3434fb0b3a", size = 1294018, upload-time = "2026-11-06T09:54:29.127Z" }, + { url = "https://files.pythonhosted.org/packages/9d/c5/297f16660b15eacb2872af9fb127e7fe83f48f6f9caf223cb5abd5a25bd8/hypothesis-6.170.15-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:124b0b9962ff6b25786692b7b6cb0065f8bb7870081ddd9baf4745b0ff4a5c7a", size = 1333737, upload-time = "2026-11-06T09:54:03.427Z" }, + { url = "https://files.pythonhosted.org/packages/19/18/7e35ba1bc1f0e3ec55d9d97864db42b5ffeea92766ae4bd7ced0f77cc930/hypothesis-6.170.15-cp311-abi3-win32.whl", hash = "sha256:d34e54f485777eb217c85c425ddbe7b278ef56540feedb5de4097a5abc9f5719", size = 687717, upload-time = "2026-11-06T09:54:40.259Z" }, + { url = "https://files.pythonhosted.org/packages/fa/d6/c16f5abdb56baa2d401de38c2f758090796c1716fed9aa690179772e6adf/hypothesis-6.170.15-cp311-abi3-win_amd64.whl", hash = "sha256:de9fe573a4955b0cf0354822fbfe9b79882e8de02f26287313fe1ad47ba3e69f", size = 693782, upload-time = "2026-11-06T09:53:17.157Z" }, + { url = "https://files.pythonhosted.org/packages/13/7a/ae3b7fdd44c30b662f27344ccb810d080d11b7f557c2cffa88d01e677295/hypothesis-6.170.15-cp311-abi3-win_arm64.whl", hash = "sha256:de5e11c568fd411a3cf07e4e34d4f8cf07d638de2f254c8bd87c1879679cdd9a", size = 692109, upload-time = "2026-11-06T09:53:52.599Z" }, + { url = "https://files.pythonhosted.org/packages/22/fa/4f61c0e5487d75d9ef20bb6d7b9e13b5aabf89d249896d9af0a648578be7/hypothesis-6.170.15-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:679635cd52626790214f879a8bdbb877923333fbd18905c591af41bd74d2f8db", size = 801251, upload-time = "2026-11-06T09:53:45.982Z" }, + { url = "https://files.pythonhosted.org/packages/51/c2/8eb51b3b243248fc7289b37aa78caeff847b98e7a35ba6f6e12a846d68ec/hypothesis-6.170.15-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f95b532f23bed43ef3c45fbd19baf4717687a4993fa6f42fc704fbde18bd744c", size = 796937, upload-time = "2026-11-06T09:54:31.637Z" }, + { url = "https://files.pythonhosted.org/packages/af/fe/80d94b0366d3408078c32758f7149592fdf963d431e52188728e5380d806/hypothesis-6.170.15-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:9f5937e70c636ee46bd2242d9ef120d19744e63ce7d17a92fd4c9f57ef62b1c3", size = 1125542, upload-time = "2026-11-06T09:54:16.018Z" }, + { url = "https://files.pythonhosted.org/packages/fd/b4/c9dfacdfa074ebcee35ad8d9e6436480a45b543bd42ab92fa556e3bee113/hypothesis-6.170.15-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ecbd6294a636594580689debc3011a0ca3eb76577770f661224314202f107e14", size = 1168312, upload-time = "2026-11-06T09:53:22.993Z" }, + { url = "https://files.pythonhosted.org/packages/66/dd/29996ba7a8ef7eaea1493dffda4bfb04e85180a2f45d3d9ab7793ada3722/hypothesis-6.170.15-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:efaee8533a06b6a2382ff7c336ea5a747c1b22a4e97c19bfe825a02388f56d2d", size = 1300279, upload-time = "2026-11-06T09:54:30.502Z" }, + { url = "https://files.pythonhosted.org/packages/57/2d/b4be9bbb76ac5079af9cd5cf484e3e0ad91f60a7e98c69d56ac623d55573/hypothesis-6.170.15-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:4af2d31c00306707166b4f39e5208d2585f9ef4accb10586534101eb32f243fb", size = 1334296, upload-time = "2026-11-06T09:53:08.948Z" }, + { url = "https://files.pythonhosted.org/packages/69/55/f0677edf64249c5d91ae9387678bfd43ae0f54905fd2a4dbede9cbaf1e0e/hypothesis-6.170.15-cp311-cp311-win_amd64.whl", hash = "sha256:b48cb85180255ec3eb3b3e5d49068c9ad447b25e34fccafdabae63d2c3d854ad", size = 693893, upload-time = "2026-11-06T09:54:49.687Z" }, + { url = "https://files.pythonhosted.org/packages/91/db/84d418c965fe889d21e7791f946f7c893bf492bcedb38f3f5b8eb405a035/hypothesis-6.170.15-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:29223a2d180e463f1205b7e5eb397db0c4827de06f2ae571b00c392d355c410e", size = 802327, upload-time = "2026-11-06T09:54:20.994Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8f/63d8644162728ba8b2c8a8a8b6a34476663c964cc0f6885c19f84ead40d6/hypothesis-6.170.15-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7953ad8f123b4289801b833666aaab757ba52e91ed687d954eb26f4af7328e5b", size = 794302, upload-time = "2026-11-06T09:54:24.828Z" }, + { url = "https://files.pythonhosted.org/packages/ec/2e/d479c69c08d5aabd7e6499f3dddc22b241db42922d746d05df9548213229/hypothesis-6.170.15-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b1f37920547eafa77814bc9b638e47299562123023bdf3c9cb002795bb90fc92", size = 1123978, upload-time = "2026-11-06T09:54:36.215Z" }, + { url = "https://files.pythonhosted.org/packages/ee/e2/824c524d04152e5c26111247d2d4dbf210d43df7c6612f7ff22c20de5abc/hypothesis-6.170.15-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e9e20ec17d2095e9314ea232b4e34b5510def0d9e7a7e764cc70cae88fe4b0c4", size = 1167677, upload-time = "2026-11-06T09:54:55.123Z" }, + { url = "https://files.pythonhosted.org/packages/ce/49/4aab201f1fd0109560d985076a4c852c1abda14984470369bf29ea89ade3/hypothesis-6.170.15-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:11c541cf103c784306f6037e5ab962cbe3d5891d36fd98df993f38d7ddbcd360", size = 1298862, upload-time = "2026-11-06T09:53:51.237Z" }, + { url = "https://files.pythonhosted.org/packages/db/a7/83fc4ef9484deefd955949a9a72a14d9435b6ace339feb27de25bf286e29/hypothesis-6.170.15-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:eecdcd38395facf6ea3af90320a574f5e63c310c194d2a0eb75f121223942386", size = 1333300, upload-time = "2026-11-06T09:53:39.58Z" }, + { url = "https://files.pythonhosted.org/packages/d5/18/5f1a5ce07a9e6b7a2cbfb6a8bceb9b5161de5477abb8d99d659a898b7554/hypothesis-6.170.15-cp312-cp312-win_amd64.whl", hash = "sha256:7dfb3d92df4faa67d3c975d1ecdefabc5a2481b789048f52cd5fb775c6c517b8", size = 691422, upload-time = "2026-11-06T09:54:05.043Z" }, + { url = "https://files.pythonhosted.org/packages/16/16/d75485989f250dd5017d1627902961c2f985686355f50d10170a2490d970/hypothesis-6.170.15-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:e3f478d4d9e13d528e352a4d938dea10b25793663ac752d774d7d35b008a1b45", size = 802290, upload-time = "2026-11-06T09:53:33.892Z" }, + { url = "https://files.pythonhosted.org/packages/24/45/54e42ac718e1164090c04b588024e4818a8e2a28b9002faa21e9fd59ab67/hypothesis-6.170.15-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:9c92af3f0c8844d9eba793a4371035a45e88bb59579a1408ce5d3cceb9e2b1da", size = 794293, upload-time = "2026-11-06T09:54:09.14Z" }, + { url = "https://files.pythonhosted.org/packages/b8/ee/1a81a7d544e321165159a07e666f6c302ec109beccca3b37f8d0de54e0cd/hypothesis-6.170.15-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:dce4d989f6b9c0a589b944591f13de3dff34e186c14401e11b819fed81d70a5b", size = 1123881, upload-time = "2026-11-06T09:53:01.454Z" }, + { url = "https://files.pythonhosted.org/packages/c9/07/4d506b2e37601ec8b38a0c4b8f547dd344bd8eccd577dbd0543a744fce03/hypothesis-6.170.15-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f25261d1656f830a78863844be5fb55cd0208154dc9ddec85d44fe691488c557", size = 1167605, upload-time = "2026-11-06T09:53:32.309Z" }, + { url = "https://files.pythonhosted.org/packages/a5/77/a88824d02272f149870e36aaad0197dc5a186f53a1896ea3117f250b8414/hypothesis-6.170.15-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5ae92fa40b57c1cd331d63fe6f95aecfdb6aa32f15b4c418c108c41fcd4aef84", size = 1298841, upload-time = "2026-11-06T09:53:15.809Z" }, + { url = "https://files.pythonhosted.org/packages/68/b9/6e1b55b6b32926cebb7ef37076d8fda9d0f87045e90e4b75e93eeb9a56ac/hypothesis-6.170.15-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b829dc3cbd73fe08c02151afc438448180907da62c154cfa68abd178eafd4d48", size = 1333319, upload-time = "2026-11-06T09:52:57.279Z" }, + { url = "https://files.pythonhosted.org/packages/c8/77/7d2009c663eb26b329c622091f7593ee6c90eac663f389fdf7311d44b035/hypothesis-6.170.15-cp313-cp313-win_amd64.whl", hash = "sha256:1f7ede44b875dddd8f3e7562e6702c363e82c6fb680b0fe079536bae662987f8", size = 691393, upload-time = "2026-11-06T09:54:56.287Z" }, + { url = "https://files.pythonhosted.org/packages/96/85/977d43b789fe4ca530e76020bbd2db07a9af3c17a4bf27a276416b6540e5/hypothesis-6.170.15-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:ffac67d06f765fd000a132e71117ee92dbd59863cea18403220ca4e39b50397f", size = 802549, upload-time = "2026-11-06T09:54:17.455Z" }, + { url = "https://files.pythonhosted.org/packages/cb/1a/51e637155a8c897774463389f1341ec8d466a9860cf5e004689866c557b2/hypothesis-6.170.15-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:82b613ee882068e781d810eb73a8195b59edd6cc47e8acbe4971562044a99c83", size = 794333, upload-time = "2026-11-06T09:54:27.915Z" }, + { url = "https://files.pythonhosted.org/packages/24/c1/ea255465b482ba5fc5f678e4347d6c6bbe878c827ca652d634328be96534/hypothesis-6.170.15-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb6fe3d1cb91d4f14fd307f10ae969ef7e866c16e4010a750d1cade145dfd764", size = 1124402, upload-time = "2026-11-06T09:53:02.773Z" }, + { url = "https://files.pythonhosted.org/packages/7a/69/32a05f0ae41f699f30ff872be74fe46ee1d45ead7abfb114afe3a4b4a3ee/hypothesis-6.170.15-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f91f27a89e8d55c7f043cb1e10705096236d4f7bcb35f3ac4f821e0c9b9bcd05", size = 1167795, upload-time = "2026-11-06T09:54:00.357Z" }, + { url = "https://files.pythonhosted.org/packages/c2/fb/c16886cea3a7ae3045093480775eb058eed549f41aa0e21a8d96aa7d51b1/hypothesis-6.170.15-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e0e4ab687c0f2bd62b7963dc8c3c8d29211cdc6db7ce05d37ef2a20d21219c18", size = 1299361, upload-time = "2026-11-06T09:53:04.441Z" }, + { url = "https://files.pythonhosted.org/packages/b7/f3/ad57518f76503276c896388affeccca1ffb1cccc529d063dac588c4bb652/hypothesis-6.170.15-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b01d130c103fdf78a4a748cae7d095edfe5d37534828413a509936fb3567db61", size = 1333515, upload-time = "2026-11-06T09:53:18.642Z" }, + { url = "https://files.pythonhosted.org/packages/2b/08/500fbe05990d886d4502fa334f6cf03607be473b2a2efffb660238135f05/hypothesis-6.170.15-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:1138c03edf4c4401afb28e50255644b4ab4e1ad52864456cd4e84458ba8c671a", size = 634104, upload-time = "2026-11-06T09:54:22.317Z" }, + { url = "https://files.pythonhosted.org/packages/41/53/11e3260a72024f41246061ad204627f8336bfbf0ec2b180648d01a05491f/hypothesis-6.170.15-cp314-cp314-win_amd64.whl", hash = "sha256:57ae2617424f5bb55af0146b47e0aff64a9d6677a4d5216cf2a66bd3bda0d457", size = 691115, upload-time = "2026-11-06T09:53:41.223Z" }, + { url = "https://files.pythonhosted.org/packages/ba/41/655fca0f45b7ea6df6b5bb70ab52b740cc6bc30b971366036eddd36a77c9/hypothesis-6.170.15-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:c0514f3f8ab5d56103b756317dedc8476945b0633e51112312247e3caecd0eb2", size = 800640, upload-time = "2026-11-06T09:53:05.61Z" }, + { url = "https://files.pythonhosted.org/packages/a5/89/6bf52da8b15846c7263eac9fcffb89cc1d1c82f96ca5b839a0d00bb3aa3a/hypothesis-6.170.15-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:cc5434cde3b22c900dfe01f891804b48f6e2aa93e891ca98a14c07dd04fa6d54", size = 792915, upload-time = "2026-11-06T09:53:11.508Z" }, + { url = "https://files.pythonhosted.org/packages/11/f9/a0c2a356c7a3a4b65ec2f80382c9ea5acc1a86247ac64ecb1b7dfc38ee95/hypothesis-6.170.15-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f22900eec7df2696073c05f60c666654eaa93f17aebe8964f751d75b131dcd90", size = 1122841, upload-time = "2026-11-06T09:53:21.626Z" }, + { url = "https://files.pythonhosted.org/packages/ef/f1/d4fa867fae5a733e4041720622417804bcfb79c00d265c5451243a5aba20/hypothesis-6.170.15-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7288412dfba6b5a6f875279a25409eda6b25d097c26ea710540f904287c55d06", size = 1166715, upload-time = "2026-11-06T09:54:42.908Z" }, + { url = "https://files.pythonhosted.org/packages/04/5d/cbf8acf14caa5a2e4d34dd088a507cc84da49a660b6f532104d8afdb2903/hypothesis-6.170.15-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7d546a5d6a5549a6b34304f7315d20aff7afd206996592329877da529b9d9153", size = 1297608, upload-time = "2026-11-06T09:53:29.22Z" }, + { url = "https://files.pythonhosted.org/packages/9c/a0/912a0ebf3ab19b39235233f9e1664147f6e842aa6915828d68c1d1e13f27/hypothesis-6.170.15-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:bd8b36d0403eff6473686c0e4d8b7704780ebf548b0b2cc86cb9cbbbe64f8c2f", size = 1332521, upload-time = "2026-11-06T09:53:47.664Z" }, + { url = "https://files.pythonhosted.org/packages/37/36/d9286c792272651db4059a3497a19cc7f4f60e0452c862274d506b0132fd/hypothesis-6.170.15-cp314-cp314t-win_amd64.whl", hash = "sha256:67eb006925a3a49ddc4a48d9af8314349291a7eee9b2baea6beb93bc48211ead", size = 691229, upload-time = "2026-11-06T09:54:45.888Z" }, + { url = "https://files.pythonhosted.org/packages/91/42/b152b66db2061b47ff928efc7162c2dcf4591d92dab19b3c574af3552220/hypothesis-6.170.15-cp315-abi3.abi3t-macosx_10_12_x86_64.whl", hash = "sha256:e007ff1dfb8bfa228b97ed6695325cfa1ba6487c7e1175c884e8795e71fb41a5", size = 800291, upload-time = "2026-11-06T09:54:07.899Z" }, + { url = "https://files.pythonhosted.org/packages/67/6f/49fe04f0b4cfd142d474e08a1855ea093c7367e6ff33e7c5d4a64592f743/hypothesis-6.170.15-cp315-abi3.abi3t-macosx_11_0_arm64.whl", hash = "sha256:742846b0e2e31640acf8bba7c2b8b6eb9cecd6817a906af6d4b5ced9db697660", size = 792763, upload-time = "2026-11-06T09:54:51.025Z" }, + { url = "https://files.pythonhosted.org/packages/88/e6/61ec7fb3b84dd8d8644d6a322d5dba4501e2f32d0b10b8b5467e7d6defd4/hypothesis-6.170.15-cp315-abi3.abi3t-manylinux1_i686.manylinux_2_5_i686.whl", hash = "sha256:508488dd4e92e03ecd16ea086f33d766855ed1febd322a439ec231c920b1740a", size = 1158761, upload-time = "2026-11-06T09:54:02.105Z" }, + { url = "https://files.pythonhosted.org/packages/8d/ae/c9eb63eb1b2d935551494aa42ecf91bb214d89c52631854dfbb09ccb6d39/hypothesis-6.170.15-cp315-abi3.abi3t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ccc6a9ce18dced7f0d2cba36f334f44697c1c6e692949c1bbd0bc67a7065c86a", size = 1121659, upload-time = "2026-11-06T09:54:12.041Z" }, + { url = "https://files.pythonhosted.org/packages/0a/44/b99e644480a31153de52489dc4c062df5f82a10fbb2b699d0f17ecb08171/hypothesis-6.170.15-cp315-abi3.abi3t-manylinux2014_armv7l.manylinux_2_17_armv7l.whl", hash = "sha256:b8a0b5b3e21a1d2e235f42a813fbf04872196ad53151fdd4cfe02723db699f8d", size = 1152560, upload-time = "2026-11-06T09:54:41.478Z" }, + { url = "https://files.pythonhosted.org/packages/f8/0a/bfc1c697f34e395d635f49d2a29f1bfc41a0eccdcf9fba32fb33601325de/hypothesis-6.170.15-cp315-abi3.abi3t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:b59187094929ce6d5351a6f92e7f3831cd463fe69591def6ebd0023a3073af2e", size = 1146287, upload-time = "2026-11-06T09:53:42.894Z" }, + { url = "https://files.pythonhosted.org/packages/bb/69/90fac02af55ccbea1d1b3bbc72512c4f998e3fabc6b375c8c0afd0e6003f/hypothesis-6.170.15-cp315-abi3.abi3t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:0ecf4e2948c3de57ea12d66f6cf6c0e3ad88b5eae0d216cfa2922800ba5522a7", size = 1200042, upload-time = "2026-11-06T09:54:37.409Z" }, + { url = "https://files.pythonhosted.org/packages/f8/eb/6810206ee4864b39da6e81b2318516773d5a89b0e2132c00dfc9aa760179/hypothesis-6.170.15-cp315-abi3.abi3t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:55e3dcd5d8fbfd5a97018ff2712f375148b62ef537490c281a198977f3588578", size = 1165971, upload-time = "2026-11-06T09:53:25.72Z" }, + { url = "https://files.pythonhosted.org/packages/e1/c8/ebfb57d277e571c207d690efa56462f1f582721e5170bb86af42c3f97b07/hypothesis-6.170.15-cp315-abi3.abi3t-manylinux_2_31_riscv64.whl", hash = "sha256:417fe54775f846fa4903bb2094ce3092a0ecc107db6676b21fd8069fcc328cb9", size = 1120269, upload-time = "2026-11-06T09:53:56.767Z" }, + { url = "https://files.pythonhosted.org/packages/8a/2b/69dff10b56fbc21dc0d240de503d741ec14cb6819f67530bca0b6ff7c5af/hypothesis-6.170.15-cp315-abi3.abi3t-musllinux_1_2_aarch64.whl", hash = "sha256:47438186039fcfba3a245bb407490d442ce2c9cce4370565f1bfcdcb887e91e2", size = 1296436, upload-time = "2026-11-06T09:53:54.24Z" }, + { url = "https://files.pythonhosted.org/packages/5f/43/42b037bb17dca205f5953b11792ecc2c73dbf7ec743d5dc490d010edb9b0/hypothesis-6.170.15-cp315-abi3.abi3t-musllinux_1_2_armv7l.whl", hash = "sha256:372b05075d4a259c5ebf3fcf43dc689ef5d4335bed9d0c9dd25d14997842c0c0", size = 1425535, upload-time = "2026-11-06T09:53:27.106Z" }, + { url = "https://files.pythonhosted.org/packages/a8/29/bb3050f958b0e3156909011afdcc07812b58a47eba50f53a39d931c1256b/hypothesis-6.170.15-cp315-abi3.abi3t-musllinux_1_2_i686.whl", hash = "sha256:2161fb65416bf8da3b1c3119aa92ea63cc177fd88249bf7dc4ca64b4872e0bd1", size = 1379162, upload-time = "2026-11-06T09:54:23.523Z" }, + { url = "https://files.pythonhosted.org/packages/a5/db/30a2291c92cd437b0c8184bcead204add93d174a1317894dc50ddd3a681d/hypothesis-6.170.15-cp315-abi3.abi3t-musllinux_1_2_ppc64le.whl", hash = "sha256:b3db4c8aecd13e043c419d90ccb683fdf19711b547ca9d23e685a35157d4657f", size = 1274757, upload-time = "2026-11-06T09:54:38.796Z" }, + { url = "https://files.pythonhosted.org/packages/a5/0a/5e1bf0060a1f9486f6cad2b512b5101526dc376565eb669c20964ccee730/hypothesis-6.170.15-cp315-abi3.abi3t-musllinux_1_2_riscv64.whl", hash = "sha256:3b81f08d6052ac74f479215029e253d88fb80e9867e8bc034252d345abfa94ce", size = 1292003, upload-time = "2026-11-06T09:53:14.207Z" }, + { url = "https://files.pythonhosted.org/packages/0b/b9/9d18977c7ec9fa0d0fb22f0e52b3a270fef7b8251579d46660fca285e289/hypothesis-6.170.15-cp315-abi3.abi3t-musllinux_1_2_x86_64.whl", hash = "sha256:a51297c26ab637a97ec3283a39de163efce0a926d0ccd4378faabf5994adb01f", size = 1331925, upload-time = "2026-11-06T09:54:53.632Z" }, + { url = "https://files.pythonhosted.org/packages/ff/af/a86d51e4550c92a794777a59a9332d135c975c6f6c25b7506c447c3be27e/hypothesis-6.170.15-cp315-abi3.abi3t-win32.whl", hash = "sha256:51246e702c458b42a4b5079c3e6904c9dfb5193de920ed91fdbfa09fa694c55b", size = 685060, upload-time = "2026-11-06T09:53:35.48Z" }, + { url = "https://files.pythonhosted.org/packages/dc/3d/ef64b33ef8c5eceb7448860b77bc1d28e3e3b3e92b7d662456fee227f4b0/hypothesis-6.170.15-cp315-abi3.abi3t-win_amd64.whl", hash = "sha256:a951de05b61b7c1235a732543dc6c3b105b6c894e736f21cb891449589aaeba7", size = 691031, upload-time = "2026-11-06T09:53:10.201Z" }, + { url = "https://files.pythonhosted.org/packages/e8/3d/f875573e232550be631b6932f93bd6b2234d880891394fe4d70d57e292b7/hypothesis-6.170.15-cp315-abi3.abi3t-win_arm64.whl", hash = "sha256:9dc31166ccb95928c6eb05817684b2dd1c2a3698a6daad0450e5f449e44bb633", size = 689008, upload-time = "2026-11-06T09:54:13.194Z" }, + { url = "https://files.pythonhosted.org/packages/4d/f4/34e1ca38f88cbf478f970b7743ea022e0d70d5b4e943eea7c06532aea6b0/hypothesis-6.170.15-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:bfe10cd8c8138ba84cdd0f72d72e0a85f0f9adea8bc730f4ab4676cd41a730af", size = 801079, upload-time = "2026-11-06T09:53:19.956Z" }, + { url = "https://files.pythonhosted.org/packages/6d/8b/3438b357b3f6e076ff3f68aee5a62c1319573c4858b9f9db6e12e2893fca/hypothesis-6.170.15-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:5c1331070d8688d31b9ce99a100593bc6afdada964689b5e1a5f2ecffefa2455", size = 795993, upload-time = "2026-11-06T09:54:52.505Z" }, + { url = "https://files.pythonhosted.org/packages/fb/a7/f25cc6ec59659ef59093ffa743a2494cc153bb8a8eeeb78e1c1e58720888/hypothesis-6.170.15-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b56ea17542eabcb9e2c1d574fcabc9df7e14a7376cfce324e2fa05e9c836fff8", size = 1125066, upload-time = "2026-11-06T09:54:19.213Z" }, + { url = "https://files.pythonhosted.org/packages/a9/c3/4b58275fe5771859bd03fe5ceccee64968874fdb999c549abf7ae7f716b9/hypothesis-6.170.15-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c1c3b98713318639a6c852f3fff60bddd466096528eb2cd5853ab077ef04f74c", size = 1168451, upload-time = "2026-11-06T09:53:38.214Z" }, + { url = "https://files.pythonhosted.org/packages/dd/b8/c935e11303f94a6e280e059aedc7c5a6458207f7039aa241d2504e178c53/hypothesis-6.170.15-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:7ea75bc4c55f0ebb80179ae7e8fe1c8ab09b553005d1aadcec14e97e1306a82a", size = 694365, upload-time = "2026-11-06T09:53:59.021Z" }, + { url = "https://files.pythonhosted.org/packages/3b/54/ab7c9a7581b0acaeb58850629f4772aab054f2fd6a77bd3264262dfb1156/hypothesis-6.170.15-pp312-pypy312_pp80-macosx_10_12_x86_64.whl", hash = "sha256:45c267cb5bcd7b73b1ca6cedb039de423b8695b34a31cc87c29a7787058e7e59", size = 801399, upload-time = "2026-11-06T09:54:10.562Z" }, + { url = "https://files.pythonhosted.org/packages/fe/a6/a7dfce075d7371c9951e2a69e78d93bff5099d1b7b0ca207497eed68428b/hypothesis-6.170.15-pp312-pypy312_pp80-macosx_11_0_arm64.whl", hash = "sha256:3eb9428e0ba92993b61cab0289b8450f469b1e4c56cbdbdeaad258f0fb4f107e", size = 793309, upload-time = "2026-11-06T09:53:07.629Z" }, + { url = "https://files.pythonhosted.org/packages/6f/b4/ce76691a588dca99c73116593b2514af2d0095e7e72d3bcdfb7dcecf2170/hypothesis-6.170.15-pp312-pypy312_pp80-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:9c30c911be51aa4fa84706f1d79dcd9ad6b9728207e69667a3fed2d4ab711434", size = 1123240, upload-time = "2026-11-06T09:53:44.286Z" }, + { url = "https://files.pythonhosted.org/packages/52/a7/998ba78de4842158178f6d96eca52ec1c786548b3e08902d527469329c04/hypothesis-6.170.15-pp312-pypy312_pp80-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d2b68004bd5a238320569f7c03322b22815c2c145b1f7bbf52a319662570ee5c", size = 1167454, upload-time = "2026-11-06T09:54:48.426Z" }, + { url = "https://files.pythonhosted.org/packages/03/f5/410ff24a49dd645f651732f45e466cbb73f4bb73d1c54eaa5a07b832b0a2/hypothesis-6.170.15-pp312-pypy312_pp80-win_amd64.whl", hash = "sha256:d25e9a3f92e9f824bb110679506ece2b8fd7d093f1fa7d819ca46278a681a1f1", size = 692539, upload-time = "2026-11-06T09:53:30.812Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/e1/2069291243c926a2ff1cd706c7f3eeb9b62144bf60f77c9fb9ff2fb26bd3/iniconfig-2.3.1.tar.gz", hash = "sha256:67f4b9c50da0dedf52af349e7749a80a9057a5031199791b906c3bb3ae878960", size = 21209, upload-time = "2026-10-06T22:48:38.076Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/43/4ca9e49d27a1fcf6bece6f6aec0ea46bb9112489b93d4b688fb415457bdb/iniconfig-2.3.1-py3-none-any.whl", hash = "sha256:9121e2c1fdb355232495be3194c8dfe87ccc2d5dee45947b78e68f499790d7a7", size = 7552, upload-time = "2026-10-06T22:48:36.959Z" }, +] + +[[package]] +name = "obstore" +version = "0.12.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/76/c5/9fbb70a0914f4cd5ee7f0fd3e39e0812c29cdb4b4a8120ac7870f379e8a0/obstore-0.12.1.tar.gz", hash = "sha256:80549c9ea5478fd5d5bd75a3c9178841c7108a76d8139db05407a25a55a6a476", size = 138207, upload-time = "2026-10-27T22:51:02.613Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f7/ed/9667a1b1c6022b212460210929cc2b89bd72883c1182c9ea513a228158ad/obstore-0.12.1-cp311-abi3-macosx_10_12_x86_64.whl", hash = "sha256:8c97afecce59224b82fcb4588a3d6fbef479bd6cc2616b8f4d3d3ddba43fa6b7", size = 5431853, upload-time = "2026-10-27T22:49:38.938Z" }, + { url = "https://files.pythonhosted.org/packages/c9/bf/78437ef20196862b6f77f112184ca6c14783ddb98eb601fea88ee07c7acb/obstore-0.12.1-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:60af5d92da9332014bba8d34f985b9dc692e73d1141d1775dd3a328db476f219", size = 4592428, upload-time = "2026-10-27T22:49:40.799Z" }, + { url = "https://files.pythonhosted.org/packages/f5/30/80f63a19ace4d55ee94b7572dacf5435aeaa835e32550277c9b630fa6974/obstore-0.12.1-cp311-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:cf96279d243efd65d005c992e20476b3f343a54f71416d04af5d29d9998e739b", size = 5029909, upload-time = "2026-10-27T22:49:42.497Z" }, + { url = "https://files.pythonhosted.org/packages/07/43/4943fd0e5f89db060511a7e96ac887502f8d682b044e70b8e96f7f5bbf41/obstore-0.12.1-cp311-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:5b22ca0d66af4c3dd130ed492f966eb7d775529ea129bf379d002dd2c4093971", size = 5276990, upload-time = "2026-10-27T22:49:44.028Z" }, + { url = "https://files.pythonhosted.org/packages/39/85/374b2f8356f052c058be845dbd4e5930d952636bb54dde67e3219ad52689/obstore-0.12.1-cp311-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b9ad04de828d3726da251b95ac7cf5bf4397c026243bb88b2d087e4ca1d7506c", size = 5445620, upload-time = "2026-10-27T22:49:45.634Z" }, + { url = "https://files.pythonhosted.org/packages/df/b7/1b8c950cc7cdadb65eace8d8c7a3e53b45341b726a28a3f93d2e2db98e75/obstore-0.12.1-cp311-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:437831889a617deb632b54be8e978b59db5c34e57f7329dd5ced1f2ee2895473", size = 5355053, upload-time = "2026-10-27T22:49:47.206Z" }, + { url = "https://files.pythonhosted.org/packages/9f/de/dc6eb84ef8d1f96af626c83b3da436b1a4d6b89aa52b560d4451f9ca8ef8/obstore-0.12.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d31b255aec88d37636d374a651f5efc684c89716ab64dea6ac89d75801933af2", size = 5591012, upload-time = "2026-10-27T22:49:48.912Z" }, + { url = "https://files.pythonhosted.org/packages/81/b5/f327b606c3aa1c7cc78f7595da7df96d8751e69912afa6331b48234098b6/obstore-0.12.1-cp311-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:d7ee9cde74dbe11a4630c90d7245543993f3cebc38a70aceaedcd3b3d6192807", size = 5376219, upload-time = "2026-10-27T22:49:50.393Z" }, + { url = "https://files.pythonhosted.org/packages/22/f2/54890109d9c1c8ed16d3ad8b17f53953e04006d6b1bf013e13c56c7ef979/obstore-0.12.1-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:4240e5cb60d091e4a612499630eebede5164c821196322bc8a5ecb4861dc7b1f", size = 5570174, upload-time = "2026-10-27T22:49:51.889Z" }, + { url = "https://files.pythonhosted.org/packages/1a/e4/8573194930c7a8cf4354b894dfd3a40ad94e9c33b5e6c88213da7038a630/obstore-0.12.1-cp311-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:7b8e376fb425cd3963bbc2d51c0e0f19bebdaf3f824ffdac9bfc8dc13a782b5b", size = 5257775, upload-time = "2026-10-27T22:49:53.573Z" }, + { url = "https://files.pythonhosted.org/packages/6d/0d/9b3ec1f0c054b104ebf00333dc8f69f56c023c4f466fb09c52ccd25501a7/obstore-0.12.1-cp311-abi3-musllinux_1_2_i686.whl", hash = "sha256:2fcec4ccfeac0c5ad43c184692f60d15b79e8db594b075c45a40c5f7eeab9276", size = 5403147, upload-time = "2026-10-27T22:49:55.245Z" }, + { url = "https://files.pythonhosted.org/packages/b9/57/834fe762603c42d70957699d5fa84ccb51c812abc60f22a70cd56ac1a0fb/obstore-0.12.1-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:656547972c57aee8054fedd2585f4280d50bc8fee192269367547c4a01759d2f", size = 5821292, upload-time = "2026-10-27T22:49:56.895Z" }, + { url = "https://files.pythonhosted.org/packages/6d/a8/1e71da6cc12d4df8da701defd2ee78edfa5298f8eddfceb6b491a9c7b4aa/obstore-0.12.1-cp311-abi3-win_amd64.whl", hash = "sha256:d84cdaeeb5f45bfa685fef3ec87265b48af13f8d006ca7c5ed1424c91e448dd5", size = 5343080, upload-time = "2026-10-27T22:49:58.631Z" }, + { url = "https://files.pythonhosted.org/packages/3e/39/6bb0099a1002498743a9aff82b6e5abbf07d0ac8ed672080374a3528b16a/obstore-0.12.1-cp311-abi3-win_arm64.whl", hash = "sha256:6eaf07332b1121dcb3b401da9220b22f94d8e3823f2cdc69e4d8b13c4adba0a6", size = 5155426, upload-time = "2026-10-27T22:50:00.462Z" }, + { url = "https://files.pythonhosted.org/packages/b7/cd/ca597f403114ca8cf0f46fb15db1db68b2c04bd14203a21d8eb5491688d5/obstore-0.12.1-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:5a8dec8c89d710ba0debc7a9e7a3d56bb207f32d6acb0dc95f786c19aa4a811f", size = 5443755, upload-time = "2026-10-27T22:50:02.018Z" }, + { url = "https://files.pythonhosted.org/packages/d4/36/42c26efb568b1a386f279dcb26d05df04243dacc5f4123108790390a3c00/obstore-0.12.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:850062a27e9162c6bb9c4e18f539a693531df741b6c5fd5d8b2a17a8c4681dda", size = 4562659, upload-time = "2026-10-27T22:50:04.262Z" }, + { url = "https://files.pythonhosted.org/packages/d2/29/341473d040251cd4918204315cbc00b0bdc9ed94e89eb3ceacd4b4b75ad9/obstore-0.12.1-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:de9eb44136babc4eccdb96e5d04f195612b976f6e708573a127b2ac46eeb0b49", size = 5423989, upload-time = "2026-10-27T22:50:05.848Z" }, + { url = "https://files.pythonhosted.org/packages/13/5e/febe7b32fc416128d61e18c86fd5da7adf8be22b1bf0be1d94e8622a197e/obstore-0.12.1-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ee8a1d6619ad9db9a84b1e7b769f639d16f7122255e48fec475d93f00d1206f3", size = 5353082, upload-time = "2026-10-27T22:50:07.416Z" }, + { url = "https://files.pythonhosted.org/packages/47/55/926becb2372bc22af0aa0c9aad61dcdd9af29bae942a36951c622f167c63/obstore-0.12.1-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8806b2e4a46a8ae69b1af7187d5bb9a0f1c717583c5525e7eccdf9b7e76da475", size = 5575137, upload-time = "2026-10-27T22:50:09.169Z" }, + { url = "https://files.pythonhosted.org/packages/f6/94/c7050bf30060fe8ea9523dc55a2d57b73dd5567abe62becaac7deba6b102/obstore-0.12.1-cp314-cp314t-manylinux_2_24_aarch64.whl", hash = "sha256:7880d391c3edf7abd6b1cb4c9071acd2c4ea108e4af83f57df4f351f4a5816bc", size = 5363584, upload-time = "2026-10-27T22:50:10.833Z" }, + { url = "https://files.pythonhosted.org/packages/e0/85/868d3589d761d5250724566f0dd942ac4b341279821cb8bddd62fe87416f/obstore-0.12.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bf0131b338f5e23224d9009dd6f67b6088aeeeac80259ae32543f5653bce9ab1", size = 5561341, upload-time = "2026-10-27T22:50:12.609Z" }, + { url = "https://files.pythonhosted.org/packages/cb/78/ee95ddd84ecd492f132bd657f6d23761373eef496247fc1806bebfa02c3f/obstore-0.12.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:8378a242379dc8f3955bf17847f22987348f2d978d0e5a43c15fff97a3c60989", size = 5804292, upload-time = "2026-10-27T22:50:14.14Z" }, + { url = "https://files.pythonhosted.org/packages/35/b1/1197446a6376c4297e162c6242dd01de7ea3419fa627cce6239cb01a8f6b/obstore-0.12.1-cp314-cp314t-win_amd64.whl", hash = "sha256:13f5a95b9722e4d489b9ac0a9d75d3e9d097ccde241cdfd6ca0150a3dd742619", size = 5320383, upload-time = "2026-10-27T22:50:16.004Z" }, + { url = "https://files.pythonhosted.org/packages/0e/da/34f86122a19ae305e58d2913b0eee7e777cc69283d5bc068590b6a8f68c6/obstore-0.12.1-cp314-cp314t-win_arm64.whl", hash = "sha256:31b5d33132e28f0933c3d35e9a1436a3b32ed68fd8bfa6e1d1b8ad69f7aa8d82", size = 5136428, upload-time = "2026-10-27T22:50:17.72Z" }, + { url = "https://files.pythonhosted.org/packages/00/2d/8c472ebe021c474c55ad79a8dfe3719281d1ac0592870d7fca8d92361461/obstore-0.12.1-cp315-abi3.abi3t-macosx_10_12_x86_64.whl", hash = "sha256:2da906fb0b5c28f6389a9561b96a5cc4da92f5eb9c2aeb3e2bd3768ab87e3717", size = 5491955, upload-time = "2026-10-27T22:50:19.814Z" }, + { url = "https://files.pythonhosted.org/packages/ce/c5/7a6cce169714f2a4c4aa90831f2c4015b07727eb8e17d858820ba70e4a81/obstore-0.12.1-cp315-abi3.abi3t-macosx_11_0_arm64.whl", hash = "sha256:4095f4e70ca24e8e7b1efeb87912cfc5142e2ec82888b1a63c0caa94fe875d22", size = 4608411, upload-time = "2026-10-27T22:50:21.49Z" }, + { url = "https://files.pythonhosted.org/packages/ab/9e/c0a2ab434103f89cfb0dcb3218e489273295936fdaa0b58fdef0d932aa77/obstore-0.12.1-cp315-abi3.abi3t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ab20d8a422a11b4236d4041861f52fa2e7e5b53fc80f2629c21fa3dd1e7fd8a1", size = 5059569, upload-time = "2026-10-27T22:50:23.035Z" }, + { url = "https://files.pythonhosted.org/packages/58/e9/d0c6db93119f0b18730d11f43ee46730c1c1031f35a742d99955fe8114c5/obstore-0.12.1-cp315-abi3.abi3t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:cf2c08fca6766f70a4d693e752fc279b57481e0a9fbb6d86770f3eeea67ee7e8", size = 5289087, upload-time = "2026-10-27T22:50:24.627Z" }, + { url = "https://files.pythonhosted.org/packages/8a/13/f8ecff794eb93e13a749c4738cc39ffde2a227780676fc6b3e3e1874b953/obstore-0.12.1-cp315-abi3.abi3t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8a357ef92897cec37c6c4f630c71a8b5c3b602439bfeb7201c00682527526434", size = 5469409, upload-time = "2026-10-27T22:50:26.095Z" }, + { url = "https://files.pythonhosted.org/packages/1e/26/ad477c01bb505ba6ad648286e8ec89aab7290a089fc761d600c65609de33/obstore-0.12.1-cp315-abi3.abi3t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e422bf5821e5ceb8577154b62389b5b44136718f345db663bdd592d301aab1c0", size = 5390957, upload-time = "2026-10-27T22:50:28.009Z" }, + { url = "https://files.pythonhosted.org/packages/ab/cc/54242ed5d9f0b94631f686118292c639ba5824c7eb6fed7321fb52267523/obstore-0.12.1-cp315-abi3.abi3t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd80719105c062b941b172ffe758b8cebee7ab0526562793fb2e9d57961f8072", size = 5625205, upload-time = "2026-10-27T22:50:29.604Z" }, + { url = "https://files.pythonhosted.org/packages/a5/18/94263560896c75216c24295d1f6292180d9a006b94df9cfd45640af9fab5/obstore-0.12.1-cp315-abi3.abi3t-manylinux_2_24_aarch64.whl", hash = "sha256:fc673c165955d3b91fac276c8720daa10276f31d685f1e12fa16e4bb0144e82f", size = 5407054, upload-time = "2026-10-27T22:50:31.226Z" }, + { url = "https://files.pythonhosted.org/packages/fc/45/497e3a6de4cf78148fcffbd3bd3bc1272d9229feeece22e46db7927debfe/obstore-0.12.1-cp315-abi3.abi3t-musllinux_1_2_aarch64.whl", hash = "sha256:6a019c276c606d930f5a7c538865186f9baa102019be0e24bfc94a8289a571eb", size = 5601269, upload-time = "2026-10-27T22:50:32.738Z" }, + { url = "https://files.pythonhosted.org/packages/4e/4f/6c0bc2a08bf6615417ad03cc3bf6a36da2e4e456ec7a0577803a8e934189/obstore-0.12.1-cp315-abi3.abi3t-musllinux_1_2_armv7l.whl", hash = "sha256:a83b1ee193c32f4da0473606ef32b9e4de924e00754c404e0ce9ff82f3076797", size = 5285684, upload-time = "2026-10-27T22:50:34.304Z" }, + { url = "https://files.pythonhosted.org/packages/87/1f/b21fa2caaed5050c0622dc0ae5ccf5dc907a4ea33cf03d1f921734cc5828/obstore-0.12.1-cp315-abi3.abi3t-musllinux_1_2_i686.whl", hash = "sha256:8ed7826686178130f5f131e7b2d435f24f1256bf44029d4fb29ff1fca441c008", size = 5412047, upload-time = "2026-10-27T22:50:35.768Z" }, + { url = "https://files.pythonhosted.org/packages/ab/a0/989bb53ab9371959aa8923598520ff17f30e43b2e790d0a1e0cc5afb50f0/obstore-0.12.1-cp315-abi3.abi3t-musllinux_1_2_x86_64.whl", hash = "sha256:6e8b45d1ceed62706c0e19680a6cb705d9094d550e7f1287e230a86694a7c5c6", size = 5853753, upload-time = "2026-10-27T22:50:37.568Z" }, + { url = "https://files.pythonhosted.org/packages/4a/95/86c53a2d63dd3521fd895bf02a312893517f1489eaee837e7c1919b92f12/obstore-0.12.1-cp315-abi3.abi3t-win_amd64.whl", hash = "sha256:1b902e538809fd01e938707cc785e2efdfedc17600223323ef7b6a3d5c0874d9", size = 5365259, upload-time = "2026-10-27T22:50:39.099Z" }, + { url = "https://files.pythonhosted.org/packages/f1/6c/8a464c4d4a8ebce693e549777be90625cf01d0ba75daea99330d909d54a0/obstore-0.12.1-cp315-abi3.abi3t-win_arm64.whl", hash = "sha256:7c0497355e6fe55d98ef3fa94429488be9f2c4d26a786969f90a1e179efaa93a", size = 5179055, upload-time = "2026-10-27T22:50:41.339Z" }, + { url = "https://files.pythonhosted.org/packages/0f/46/db83f5e902acc4d7188e828b52ff72e1072fcc92d56af1849fc3e3894350/obstore-0.12.1-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:019af6060d0143975594a3f1efda0d15c0098702b889d048a2c085018a5c65e8", size = 5027315, upload-time = "2026-10-27T22:50:42.851Z" }, + { url = "https://files.pythonhosted.org/packages/ee/37/335aa42c62ce365b2ec37355f06eb6f57a8b53aa99160f0babec7c593e63/obstore-0.12.1-pp311-pypy311_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:8f7aeba8901b506056b01765e72d13af8a295ef1b64ad73251895364a843afef", size = 5278226, upload-time = "2026-10-27T22:50:44.413Z" }, + { url = "https://files.pythonhosted.org/packages/a4/e0/b3d447736a8e985ef10ffec4061bc895f2b410fe983b683213fec404ff0f/obstore-0.12.1-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:bdbd3db63d7f97ada70ff9351948db6e4ba883ad189e422ffa03ff02dc4c34c4", size = 5440705, upload-time = "2026-10-27T22:50:45.952Z" }, + { url = "https://files.pythonhosted.org/packages/db/7b/cb10055c1f4cd78e3c52c5a43d912b3c1f5a6601d2989ea3656515061057/obstore-0.12.1-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:96ce72f778133851d27602a429e744620ea6e762630a5f58140bb4f71fdaf8be", size = 5353724, upload-time = "2026-10-27T22:50:47.409Z" }, + { url = "https://files.pythonhosted.org/packages/6c/bd/bb2a98c22191b29f02c042e98be026a13e7eb284531f64d1a25a119ea6b4/obstore-0.12.1-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ca9a0399a5b43de54a31777cc56f8f2f7893ac444cac2b7d834f2bea299cad61", size = 5586399, upload-time = "2026-10-27T22:50:48.973Z" }, + { url = "https://files.pythonhosted.org/packages/e7/48/e0d510abfba83e571529b892e0d54280828b2761e6409af34b811cca58aa/obstore-0.12.1-pp311-pypy311_pp73-manylinux_2_24_aarch64.whl", hash = "sha256:5c1fe53af794af798ac1e0ec509a0cbfc66fe68df98d3cc1b557d4602dd39b50", size = 5375790, upload-time = "2026-10-27T22:50:50.884Z" }, + { url = "https://files.pythonhosted.org/packages/0f/0a/c5999ce2d5d70071d2e6b45b10c98bc22bb5427d84120f02ca94b1716c7d/obstore-0.12.1-pp311-pypy311_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:f7e02534dfcc2fd6188a4aa5901bafcf032142fbf8836caab80db7b6ad25e369", size = 5569722, upload-time = "2026-10-27T22:50:52.507Z" }, + { url = "https://files.pythonhosted.org/packages/f6/61/b899283be0df446f3f9cb2815a64d9faf8a8b30487954795b92471d3e90b/obstore-0.12.1-pp311-pypy311_pp73-musllinux_1_2_armv7l.whl", hash = "sha256:eea60451265e609c48510787080a403b79144a45e69f59b330f7e3901e11236f", size = 5252932, upload-time = "2026-10-27T22:50:54.126Z" }, + { url = "https://files.pythonhosted.org/packages/04/36/daed807520fdab5ac0031e0d7a459b3ce5aedde21f2f315b278c369599dc/obstore-0.12.1-pp311-pypy311_pp73-musllinux_1_2_i686.whl", hash = "sha256:f4cedcc164644f2923b19a763aa07a04134072bbf7a190637c267072cd8f6e4d", size = 5398228, upload-time = "2026-10-27T22:50:55.92Z" }, + { url = "https://files.pythonhosted.org/packages/b5/e0/6e38c19d1fd4dc40f087ad953c2c0dea1203994d359f253e1123ae175351/obstore-0.12.1-pp311-pypy311_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:94fba14cd7d64f7a0b4da0224dcaec9c379ba7a1ddcad2c5e4c35a0b56977b61", size = 5815556, upload-time = "2026-10-27T22:50:57.387Z" }, + { url = "https://files.pythonhosted.org/packages/d8/c6/f0b45052e07d8eb2f5a6836b45babf18b06dae2019404f9af4662db6666c/obstore-0.12.1-pp311-pypy311_pp80-macosx_10_12_x86_64.whl", hash = "sha256:2edaecb8b06e6f1674f8e0375f0e25df00c4a2162c489769cd4ffc182e8069ba", size = 5424080, upload-time = "2026-10-27T22:50:58.899Z" }, + { url = "https://files.pythonhosted.org/packages/d3/b5/91046acc476a58c02731140d987cc3faed8a4dd073154b9c3eaf34965107/obstore-0.12.1-pp311-pypy311_pp80-macosx_11_0_arm64.whl", hash = "sha256:eb4fcec7658aa6929f77502edef424213db8dfe173276ae6bff67df8f95e5aff", size = 4591674, upload-time = "2026-10-27T22:51:00.972Z" }, +] + +[[package]] +name = "packaging" +version = "26.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412, upload-time = "2026-08-04T18:15:28.737Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" }, +] + +[[package]] +name = "pluggy" +version = "1.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bf/db/7fc19e6f2dc92a966727031389fc2e08b558f0f25eb7403c1119ad4713cd/pluggy-1.7.0.tar.gz", hash = "sha256:d1eaa46ebb595891b860ab086b4d09c8588af65ebd4361b8e8f4bb8920b90ba8", size = 123304, upload-time = "2026-10-15T09:50:58.343Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/40/9e/2b38731e0fc536806f16490e1a12d7f0dc2a1235aa8cc07bcc75416a7daa/pluggy-1.7.0-py3-none-any.whl", hash = "sha256:7dd7b0d8832ba3cb632c306926ded123429211b83641b35dc5c41ad2d34f9bec", size = 27082, upload-time = "2026-10-15T09:50:56.808Z" }, +] + +[[package]] +name = "pygments" +version = "2.21.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/49/2e/ced460408999b33da6b31b0021b0f37d329e202d4169aeb164493778f25b/pygments-2.21.0.tar.gz", hash = "sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c", size = 5005329, upload-time = "2026-08-17T08:02:48.824Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/71/46/17f022dd3e953bf20a04a028a21ec746d942f8d2af30fa0f124fa0e6a684/pygments-2.21.0-py3-none-any.whl", hash = "sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9", size = 1250147, upload-time = "2026-08-17T08:02:44.912Z" }, +] + +[[package]] +name = "pytest" +version = "9.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/79/42/bb4641652e58dedeac2ec51abc1f8cae4fee4ec63899a0999016229a0f2a/pytest-9.2.0.tar.gz", hash = "sha256:97c4f3b1f3c7414aa36465e09b8220f1d03627993bd51b1282e193753c80557b", size = 1797555, upload-time = "2026-10-27T21:30:20.728Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/39/1f5d8c8877a0fa327db959c19341d4be61660de1ab6b7391f9eb805bcd67/pytest-9.2.0-py3-none-any.whl", hash = "sha256:07acdcf88ec174f907dc01af70bf52d49b014bbab03faeeade31c96b18c2e588", size = 411776, upload-time = "2026-10-27T21:30:19.131Z" }, +] + +[[package]] +name = "sortedcontainers" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e8/c4/ba2f8066cceb6f23394729afe52f3bf7adec04bf9ed2c820b39e19299111/sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88", size = 30594, upload-time = "2021-05-16T22:03:42.897Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575, upload-time = "2021-05-16T22:03:41.177Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "zarr-storage" +source = { editable = "." } +dependencies = [ + { name = "typing-extensions" }, +] + +[package.optional-dependencies] +obstore = [ + { name = "obstore" }, +] +testing = [ + { name = "hypothesis" }, + { name = "pytest" }, +] + +[package.dev-dependencies] +test = [ + { name = "hypothesis" }, + { name = "obstore" }, + { name = "pytest" }, +] + +[package.metadata] +requires-dist = [ + { name = "hypothesis", marker = "extra == 'testing'", specifier = ">=6.85" }, + { name = "obstore", marker = "extra == 'obstore'", specifier = ">=0.12" }, + { name = "pytest", marker = "extra == 'testing'", specifier = ">=8" }, + { name = "typing-extensions", specifier = ">=4.6" }, +] +provides-extras = ["obstore", "testing"] + +[package.metadata.requires-dev] +test = [ + { name = "hypothesis", specifier = ">=6.85" }, + { name = "obstore", specifier = ">=0.12" }, + { name = "pytest", specifier = ">=8" }, +]