Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/zarr-storage/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Release notes

<!-- towncrier release notes start -->
234 changes: 234 additions & 0 deletions packages/zarr-storage/DESIGN.md

Large diffs are not rendered by default.

21 changes: 21 additions & 0 deletions packages/zarr-storage/LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The MIT License (MIT)

Copyright (c) 2015-2025 Zarr Developers <https://github.com/zarr-developers>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
3 changes: 3 additions & 0 deletions packages/zarr-storage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# zarr-storage

A Zarr-agnostic key-value storage layer. Pre-alpha; see `DESIGN.md`.
1 change: 1 addition & 0 deletions packages/zarr-storage/changes/460.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add `zarr-storage`, a new package with a Zarr-agnostic key-value storage API: `Store` and `AsyncStore` views over memory, local-filesystem and obstore backends, with the same operations as sync methods and as coroutines, conditional reads and writes (`get_if_changed`, `put_if_absent`, `put_if_match`, `delete_if_match`), byte-range reads, reads into caller buffers, and batches (`get_many`, `get_into_many`, `put_many`, `delete_many`) that run concurrently where the backend can. It ships a conformance suite in `zarr_storage.testing` that any backend can run.
25 changes: 25 additions & 0 deletions packages/zarr-storage/changes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
Writing a changelog entry for `zarr-storage`
---------------------------------------------

Fragments in **this** directory are released notes for the `zarr-storage`
package only — kept separate from the parent zarr-python `changes/`
directory so a PR touching only `packages/zarr-storage/` produces a
release note for this package only.

Please put a new file in this directory named `xxxx.<type>.md`, where

- `xxxx` is the pull request number associated with this entry
- `<type>` is one of:
- feature
- bugfix
- doc
- removal
- misc

Inside the file, please write a short description of what you have
changed, and how it impacts users of `zarr-storage`.

A `zarr-storage` release runs `towncrier build` in `packages/zarr-storage/`,
which consumes the fragments here and updates `CHANGELOG.md`. Fragments
that describe parent zarr-python changes (not the storage package)
belong in the top-level `changes/` directory, not here.
59 changes: 59 additions & 0 deletions packages/zarr-storage/justfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Development verbs for the zarr-storage package. Recipes run with this
# directory as the working directory regardless of where `just` is invoked.

# List available recipes
# Quoted arguments must survive delegation from the root Justfile.
set positional-arguments

default:
@just --list

# Run the test suite on the project interpreter; extra args are passed to pytest
test *args:
uv run --group test pytest tests "$@"

# The oldest and newest interpreters the package is declared for: the floor is
# `requires-python`, the ceiling the newest classifier, both read from
# pyproject.toml. CI's matrix is written out in its workflow, so a change to
# the declared range is made there too.
# uv fetches an interpreter it does not have. A difference in how a version
# materialises annotations or resolves a stub shows up here, not only in CI.
# Run the test suite on the oldest and newest supported interpreters
test-versions *args:
#!/usr/bin/env bash
set -euo pipefail
read -r floor ceiling < <(uv run python - <<'EOF'
import re
import tomllib

project = tomllib.load(open("pyproject.toml", "rb"))["project"]
floor = re.fullmatch(r">=\s*(\d+\.\d+)", project["requires-python"]).group(1)
declared = [
classifier.rsplit(" ", 1)[1]
for classifier in project["classifiers"]
if classifier.startswith("Programming Language :: Python :: 3.")
]
print(floor, max(declared, key=lambda version: tuple(map(int, version.split(".")))))
EOF
)
for version in "$floor" "$ceiling"; do
echo "== python $version =="
uv run --python "$version" --group test pytest tests "$@"
done

# Lint with the same invocation CI uses
lint:
uvx ruff check .

# Run under the interpreter CI runs pyright under, so a version-dependent
# stdlib or stub difference shows up here rather than only in CI.
# Type-check the package, sources and tests alike
typecheck:
uv run --python 3.11 --group test --with pyright pyright

# Run everything CI runs for this package, on both ends of the version range
check: lint typecheck test-versions

# Preview the changelog that the next release would generate
changelog-draft:
uvx towncrier build --draft --version Unreleased
156 changes: 156 additions & 0 deletions packages/zarr-storage/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
[build-system]
requires = ["hatchling>=1.29.0", "hatch-vcs", "setuptools-scm>=9"]
build-backend = "hatchling.build"

[project]
name = "zarr-storage"
dynamic = ["version"]
description = "A Zarr-agnostic key-value storage layer for Zarr implementations."
readme = "README.md"
requires-python = ">=3.11"
license = "MIT"
license-files = ["LICENSE.txt"]
authors = [
{ name = "Davis Bennett", email = "davis.v.bennett@gmail.com" },
]
classifiers = [
"Development Status :: 2 - Pre-Alpha",
"Intended Audience :: Developers",
"Intended Audience :: Information Technology",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Topic :: Scientific/Engineering",
"Topic :: Software Development :: Libraries :: Python Modules",
"Typing :: Typed",
]
keywords = ["zarr", "storage", "key-value"]
dependencies = [
# `typing_extensions.Buffer`, the buffer-protocol type, before Python 3.12's
# `collections.abc.Buffer`.
"typing_extensions>=4.6",
]

[project.optional-dependencies]
# `zarr_storage.backends.ObjectStoreBackend`.
obstore = ["obstore>=0.12"]
# The `zarr_storage.testing` conformance suite.
testing = ["pytest>=8", "hypothesis>=6.85"]

[project.urls]
Homepage = "https://github.com/zarr-developers/zarr-python"
Source = "https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-storage"
Issues = "https://github.com/zarr-developers/zarr-python/issues"
Changelog = "https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-storage/CHANGELOG.md"

[dependency-groups]
test = [
"pytest>=8",
"hypothesis>=6.85",
"obstore>=0.12",
]

[tool.hatch.version]
source = "vcs"
tag-pattern = '^zarr_storage-v(?P<version>.+)$'
raw-options = { root = "../..", scm = { git = { describe_command = "git describe --dirty --tags --long --match zarr_storage-v*" } }, local_scheme = "no-local-version", fallback_version = "0.0.0" }

[tool.hatch.build.targets.wheel]
packages = ["src/zarr_storage"]

# An allowlist, so nothing that merely happens to sit in the package directory
# can ride along in a release.
[tool.hatch.build.targets.sdist]
include = [
"/src",
"/tests",
"/justfile",
"/CHANGELOG.md",
]

[tool.ruff]
extend = "../../pyproject.toml"
target-version = "py311"

[tool.ruff.lint]
# Use `object` for "any value"; a genuinely dynamic annotation needs a noqa
# saying why.
extend-select = ["ANN401"]

[tool.pytest.ini_options]
minversion = "8"
testpaths = ["tests"]
xfail_strict = true
addopts = ["-ra", "--strict-config", "--strict-markers"]
filterwarnings = ["error"]

[tool.numpydoc_validation]
checks = [
"GL10",
"SS04",
"PR02",
"PR03",
"PR05",
"PR06",
]

# Pyright is the only checker that runs on this package.
[tool.pyright]
include = ["src", "tests"]
enableTypeIgnoreComments = false
reportUnnecessaryTypeIgnoreComment = "error"
extraPaths = ["."]
typeCheckingMode = "strict"
pythonVersion = "3.11"

[[tool.pyright.executionEnvironments]]
root = "tests"
reportMissingTypeArgument = false
reportPrivateUsage = false
reportUnknownArgumentType = false
reportUnknownLambdaType = false
reportUnknownMemberType = false
reportUnknownParameterType = false
reportUnknownVariableType = false
reportUnusedFunction = false

[tool.towncrier]
directory = "changes"
filename = "CHANGELOG.md"
package = "zarr_storage"
underlines = ["", "", ""]
title_format = "## {version} ({project_date})"
issue_format = "[#{issue}](https://github.com/zarr-developers/zarr-python/pull/{issue})"
start_string = "<!-- towncrier release notes start -->\n"
template = "towncrier-template.md"

[[tool.towncrier.type]]
directory = "feature"
name = "Features"
showcontent = true

[[tool.towncrier.type]]
directory = "bugfix"
name = "Bugfixes"
showcontent = true

[[tool.towncrier.type]]
directory = "doc"
name = "Improved Documentation"
showcontent = true

[[tool.towncrier.type]]
directory = "removal"
name = "Deprecations and Removals"
showcontent = true

[[tool.towncrier.type]]
directory = "misc"
name = "Misc"
showcontent = true
113 changes: 113 additions & 0 deletions packages/zarr-storage/src/zarr_storage/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
"""A Zarr-agnostic key-value storage layer.

A `Store` is a view of a backend at a root directory, used through sync methods; an `AsyncStore`
is the same view used through coroutines, with the same method names. Backends implement capability
protocols (`Readable`, `Writable`, `Listable`, their batch variants, and async twins) over backend
paths; the stores validate keys, apply their root, enforce read-only access, and are the only
objects most callers need.
"""

from zarr_storage._errors import (
BatchError,
InvalidByteRangeError,
InvalidKeyError,
KeyConflictError,
ReadOnlyError,
StorageError,
StoreClosedError,
UnsupportedOperationError,
)
from zarr_storage._protocols import (
Backend,
ClosableAsync,
DeletableIfMatch,
DeletableIfMatchAsync,
Listable,
ListableAsync,
Readable,
ReadableAsync,
ReadableInto,
ReadableIntoAsync,
ReadableIntoMany,
ReadableIntoManyAsync,
ReadableMany,
ReadableManyAsync,
Writable,
WritableAsync,
WritableIfMatch,
WritableIfMatchAsync,
WritableMany,
WritableManyAsync,
)
from zarr_storage._store import AsyncStore, Store
from zarr_storage._types import (
ByteRange,
Copied,
Deleted,
Entry,
Found,
FoundRanges,
Generation,
Key,
ListItem,
Missing,
ObjectMetadata,
OffsetRequest,
Prefix,
RangeRequest,
Rejected,
SuffixRequest,
Unchanged,
Written,
)

__all__ = [
"AsyncStore",
"Backend",
"BatchError",
"ByteRange",
"ClosableAsync",
"Copied",
"DeletableIfMatch",
"DeletableIfMatchAsync",
"Deleted",
"Entry",
"Found",
"FoundRanges",
"Generation",
"InvalidByteRangeError",
"InvalidKeyError",
"Key",
"KeyConflictError",
"ListItem",
"Listable",
"ListableAsync",
"Missing",
"ObjectMetadata",
"OffsetRequest",
"Prefix",
"RangeRequest",
"ReadOnlyError",
"Readable",
"ReadableAsync",
"ReadableInto",
"ReadableIntoAsync",
"ReadableIntoMany",
"ReadableIntoManyAsync",
"ReadableMany",
"ReadableManyAsync",
"Rejected",
"StorageError",
"Store",
"StoreClosedError",
"SuffixRequest",
"Unchanged",
"UnsupportedOperationError",
"Writable",
"WritableAsync",
"WritableIfMatch",
"WritableIfMatchAsync",
"WritableMany",
"WritableManyAsync",
"Written",
]
Loading
Loading