Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 46 additions & 46 deletions packages/zarr-http-server/uv.lock

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions packages/zarr-metadata/changes/4635.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Strategies in `zarr_metadata.testing.strategies` that draw codec pipelines, such as `array_documents` and `pipelines`, now draw the same examples on every derandomized Hypothesis run. A codec was sampled from options that hold strategies, which Hypothesis labels by their memory address, so the examples its mutator produced varied from one process to the next.
1 change: 1 addition & 0 deletions packages/zarr-metadata/changes/4636.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The `rectilinear` chunk grid now refuses an empty list of chunk edge lengths, such as `"chunk_shapes": [[]]`, even along a dimension of length 0. zarr-extensions#78 made the list form non-empty; a dimension with no chunks writes a bare integer instead. The JSON Schema says `minItems: 1` for the list, and `chunk_grid_configurations` no longer draws an empty list.
1 change: 1 addition & 0 deletions packages/zarr-metadata/changes/4636.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`zarr_metadata.typed_json.check` and definitions' configurations now read more annotated-types constraints, as pydantic does: `MinLen`, `MaxLen` and `Len` on an array or a string, and `MultipleOf` on an integer. A value outside them is an `invalid_value` problem whose `ctx` uses pydantic's names (`min_length`, `multiple_of`), and the JSON Schema writes the matching keyword, such as `minItems`, `minLength` or `multipleOf`. A constraint on a type it does not apply to raises `TypeError`. Constant rules now live on the types, so the JSON Schemas say them too: non-empty `codecs`, sharding `codecs` and `index_codecs`, rectilinear edge lists, `struct` fields and v3 `struct` field names; `length_bytes` of at least 1 for `null_terminated_bytes` and zarr-python `raw_bytes`, and a positive multiple of 4 for `fixed_length_utf32`. Raw `r<N>` declares its bits a positive multiple of 8 on its type, and a refused name's message now names them: `bits in "r12": ...`. `check` now also refuses `"codecs": []`.
8 changes: 8 additions & 0 deletions packages/zarr-metadata/changes/4638.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Add `ZarrV3NodeType` and `ZARR_V3_NODE_TYPE` for the `node_type` member of a
v3 document, and constants for the separator the spec assumes when one is
absent: `ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR` (`"."`),
`DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` (`"/"`) and
`V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` (`"."`). The `TypeIs` functions
`is_zarr_v2_array_order`, `is_zarr_v2_array_dimension_separator` (in
`zarr_metadata.v2.array`) and `is_zarr_v3_node_type` (in `zarr_metadata.v3`)
narrow any value to the matching `Literal` type, under every type checker.
6 changes: 4 additions & 2 deletions packages/zarr-metadata/docs/extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,10 @@ names `acme.lz4` is then checked against this definition.
- a `NewType` or a type alias.

A number may carry bounds from annotated-types: `Gt`, `Ge`, `Lt`, `Le` or
`Interval`. Any other `Annotated` metadata is refused when the definition
is built.
`Interval`, and an integer a step, `MultipleOf`. An array or a string may
bound its length with `MinLen`, `MaxLen` or `Len`, so
`Annotated[tuple[int, ...], MinLen(1)]` is never empty. Any other
`Annotated` metadata is refused when the definition is built.

**The rules function**

Expand Down
16 changes: 15 additions & 1 deletion packages/zarr-metadata/src/zarr_metadata/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
from zarr_metadata.problems import MetadataValidationError, ProblemKind, ValidationProblem
from zarr_metadata.typed_json import JSONValue
from zarr_metadata.v2.array import (
ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR,
ZARR_V2_ARRAY_DIMENSION_SEPARATOR,
ZARR_V2_ARRAY_METADATA_STORE_KEY,
ZARR_V2_ARRAY_ORDER,
Expand Down Expand Up @@ -65,7 +66,13 @@
ZarrV2GroupMetadataStoreKey,
ZarrV2ZGroupJSON,
)
from zarr_metadata.v3 import ZarrV3DeclarationJSON, ZarrV3ExtensionField, ZarrV3NamedConfigJSON
from zarr_metadata.v3 import (
ZARR_V3_NODE_TYPE,
ZarrV3DeclarationJSON,
ZarrV3ExtensionField,
ZarrV3NamedConfigJSON,
ZarrV3NodeType,
)
from zarr_metadata.v3.array import (
ZARR_V3_ARRAY_METADATA_STORE_KEY,
ZarrV3ArrayMetadataJSON,
Expand All @@ -82,13 +89,15 @@
RegularChunkGridName,
)
from zarr_metadata.v3.chunk_key_encoding.default import (
DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR,
DEFAULT_CHUNK_KEY_ENCODING_NAME,
DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR,
DefaultChunkKeyEncodingMetadata,
DefaultChunkKeyEncodingName,
DefaultChunkKeyEncodingSeparator,
)
from zarr_metadata.v3.chunk_key_encoding.v2 import (
V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR,
V2_CHUNK_KEY_ENCODING_NAME,
V2_CHUNK_KEY_ENCODING_SEPARATOR,
V2ChunkKeyEncodingMetadata,
Expand Down Expand Up @@ -281,6 +290,7 @@
"COMPLEX64_DATA_TYPE_NAME",
"COMPLEX128_DATA_TYPE_NAME",
"CRC32C_CODEC_NAME",
"DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR",
"DEFAULT_CHUNK_KEY_ENCODING_NAME",
"DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR",
"ENDIANNESS",
Expand Down Expand Up @@ -309,10 +319,12 @@
"UINT32_DATA_TYPE_NAME",
"UINT64_DATA_TYPE_NAME",
"UNSET",
"V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR",
"V2_CHUNK_KEY_ENCODING_NAME",
"V2_CHUNK_KEY_ENCODING_SEPARATOR",
"VLEN_BYTES_CODEC_NAME",
"VLEN_UTF8_CODEC_NAME",
"ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR",
"ZARR_V2_ARRAY_DIMENSION_SEPARATOR",
"ZARR_V2_ARRAY_METADATA_STORE_KEY",
"ZARR_V2_ARRAY_ORDER",
Expand All @@ -322,6 +334,7 @@
"ZARR_V3_ARRAY_METADATA_STORE_KEY",
"ZARR_V3_CONSOLIDATED_METADATA_KEY",
"ZARR_V3_GROUP_METADATA_STORE_KEY",
"ZARR_V3_NODE_TYPE",
"ZSTD_CODEC_NAME",
"BloscCName",
"BloscCodecMetadata",
Expand Down Expand Up @@ -444,6 +457,7 @@
"ZarrV3GroupMetadataUpdate",
"ZarrV3NamedConfigJSON",
"ZarrV3NodeMetadataInput",
"ZarrV3NodeType",
"ZstdCodecMetadata",
"ZstdCodecName",
"__version__",
Expand Down
8 changes: 5 additions & 3 deletions packages/zarr-metadata/src/zarr_metadata/_json.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,9 +127,11 @@ class ValidationProblem:
document, `input` shows the change. `ctx` holds what was expected,
when that is more than a type:

- `gt`, `ge`, `lt` and `le`: the bounds from the value's type or from
a rule, with the names pydantic uses. For example `{"ge": 0, "le": 9}`
for a gzip `level`, whose type is `Annotated[int, Interval(ge=0, le=9)]`.
- `gt`, `ge`, `lt`, `le`, `multiple_of`, `min_length` and
`max_length`: the constraints from the value's type or from a rule,
with the names pydantic uses. For example `{"ge": 0, "le": 9}` for a
gzip `level`, whose type is `Annotated[int, Interval(ge=0, le=9)]`,
or `{"min_length": 1}` for an array document's `codecs`.
- `expected`: the values of a closed set, like zod's `values`, in the
order the message lists them. For example the values of a `Literal`,
of `node_type`, or of `zarr_format`.
Expand Down
7 changes: 4 additions & 3 deletions packages/zarr-metadata/src/zarr_metadata/_pydantic_schema.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
"""Private input types used only to generate accurate Pydantic JSON schemas.

These repeat the document TypedDicts in a stricter form: extension names
match the spec's pattern, `must_understand` is `true` at an extension
point, and `codecs` is not empty. They exist because pydantic builds a
match the spec's pattern, and `must_understand` is `true` at an extension
point. They exist because pydantic builds a
schema from Python types and cannot embed a JSON Schema that has its own
`$defs`, such as the ones `zarr_metadata.json_schema` writes; the v3
document schemas are recursive, so they cannot be inlined either.
Expand All @@ -16,6 +16,7 @@
from collections.abc import Mapping # resolved by Pydantic at runtime
from typing import Annotated, Literal, NotRequired

from annotated_types import MinLen
from pydantic import Field
from typing_extensions import TypedDict

Expand Down Expand Up @@ -57,7 +58,7 @@ class ZarrV3MandatoryNamedConfigJSON(TypedDict, closed=True):

ZarrV3DeclarationJSON = ExtensionName | ZarrV3NamedConfigJSON
ZarrV3MandatoryDeclarationJSON = ExtensionName | ZarrV3MandatoryNamedConfigJSON
ZarrV3CodecPipelineJSON = Annotated[tuple[ZarrV3MandatoryDeclarationJSON, ...], Field(min_length=1)]
ZarrV3CodecPipelineJSON = Annotated[tuple[ZarrV3MandatoryDeclarationJSON, ...], MinLen(1)]
ZarrV2FilterPipelineJSON = tuple[ZarrV2CodecMetadata, ...]


Expand Down
Loading
Loading