From 22c87418550ca1c5139afe8824030e72aa2b3acc Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Sat, 14 Nov 2026 22:05:12 +0100 Subject: [PATCH 1/7] feat(zarr-metadata): read length and step constraints from annotated-types (#4636) * fix(zarr-metadata): draw the same codecs on every derandomized run `_codec` drew from `st.sampled_from` over `(definition, configurations)` pairs whose configurations are strategies built during the draw. Hypothesis labels such a `sampled_from` by hashing its elements, and a strategy hashes by address, so these spans were labeled by memory layout. Hypothesis's mutator copies choices between spans that share a label, so which spans it grouped, and so which examples a derandomized run drew, changed with whatever moved the allocator: another process, machine load. `_codec` now samples an index into the options, labeled by a `range`. The share tests measured this drift. `examples_of` now draws twice and fails if the runs differ, which catches an address-labeled strategy in the same process; the array-document targets share one draw. Bounds are reset from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the seed minimum (0.06). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * docs(zarr-metadata): add changelog fragment for #4635 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * fix(zarr-metadata): refuse an empty rectilinear edge list zarr-extensions#78 (cadd1dab6) made the array form of a dimension's chunk edge lengths non-empty, even for an axis of length 0, which writes a bare integer to have no chunks. The rectilinear chunk grid's rules now report `chunk_shapes: [[]]`, its JSON Schema says `minItems: 1` for the list, and `chunk_grid_configurations` always draws at least one edge. Spec links are re-pinned to cadd1dab6. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * feat(zarr-metadata): read MinLen and MaxLen on arrays in the TypedDict checker The checker, the JSON Schema writer and the pydantic schema now read annotated-types' `MinLen`, `MaxLen` and `Len` on an array, as pydantic does. One table of constraints now drives parsing, the shape check and the schema keywords, in place of five parallel tables. An array outside its bounds is an `invalid_value` problem whose `ctx` holds `min_length` or `max_length`. A length bound on anything but an array raises `TypeError`. `codecs` and each rectilinear dimension's edge list now declare `MinLen(1)` on their types. This removes the two hand-patched `minItems: 1` schema fragments and the rectilinear `_rules` function. `chunk_shapes: [[]]` and `codecs: []` are still refused at the same `loc`. `check` now also refuses `codecs: []`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * feat(zarr-metadata): declare sharding's codec pipelines non-empty The `sharding_indexed` codec's `codecs` and `index_codecs` each hold exactly one array -> bytes codec, so neither is ever empty. Both now declare `MinLen(1)` on their types, as the array document's `codecs` does. An empty one is reported by the type check at the same `loc` as before, with `ctx={"min_length": 1}`. Before, the pipeline's order check reported it as "expected an array -> bytes codec, got none". Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * refactor(zarr-metadata): move constant data type rules onto their types `null_terminated_bytes` and zarr-python's `raw_bytes` checked `length_bytes >= 1` in a rule, and v3 `struct` checked that it has a field. The types now declare `Ge(1)` and `MinLen(1)`, the two rules that did only this are gone, and the struct rule keeps its name and field-type checks. The problems keep their `loc` and kind. The JSON Schema of each type now says so too: it accepted `length_bytes: 0` and `fields: []`, which the reader refuses. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * feat(zarr-metadata): read MultipleOf on integers in the TypedDict checker The checker, the JSON Schema writer and the pydantic schema now read annotated-types' `MultipleOf` on an integer, as pydantic does under `multiple_of`. JSON Schema's keyword is `multipleOf`. The step must be a positive integer, and a type takes one. Two steps on a NewType and its annotation merge to their least common multiple. A float is refused, because validators compare floats inexactly. Each constraint in the table now names the shapes it applies to, with the JSON Schema keyword for each, and a slot (below, above or step) in place of a side. `fixed_length_utf32`'s `length_bytes` and raw `r` `bits` now declare `Gt(0), MultipleOf(8 or 4)` on their types. The two rules that checked only this are gone. A test whose own `r*` definition accepts any size now uses its own configuration type. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * feat(zarr-metadata): read MinLen and MaxLen on strings in the TypedDict checker `MinLen`, `MaxLen` and `Len` now also bound a string, as pydantic's `min_length` does. The JSON Schema writes `minLength` and `maxLength`. Python and JSON Schema both count code points, so the checker and the schema agree on what a string's length is. Messages count characters: "expected a string with at least 1 character, got \"\"". A length bound on an object still raises `TypeError`. The v3 `struct` field name now declares `MinLen(1)` on its type, and the struct rule no longer checks for an empty name. The problem keeps its `loc` and kind. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * chore(zarr-metadata): number the changelog fragments by this PR Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * test(zarr-metadata): loosen the floor on documents with an axis past 32 bits The derandomized share stream changes with any change to what the strategies draw, and with machine load: base main measures 0.103 against a floor of 0.1 under load, and the rectilinear fix measures 0.047 to 0.133 across runs. Over seeds 0-5 at 500 examples both trees average about 0.17, from 0.07 to 0.29. A floor of 0.03 still asks that a few hundred examples meet several such axes. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * fix(zarr-metadata): hold a length bound beside an unknown key `constrained` skipped a type's constraints whenever the inner parse reported any problem. An `unknown_key` problem leaves the value readable, because the key is dropped and the value is still returned. So `check` returned an array of closed TypedDicts longer than its `MaxLen`, and a definition's rules ran on it, although the rules run only on a configuration within its bounds. The constraints are now skipped only for a problem other than an unknown key. Number bounds were never affected, because a number holds no keys. The docs now list `multiple_of`, `min_length` and `max_length` among the `ctx` keys, and `constraints_of` says a type takes one step too. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * fix(zarr-metadata): name the member a carried name's type check refuses A raw `r` name's size is now checked by its type. For `"data_type": "r12"` that gave "expected an integer > 0 that is a multiple of 8, got 12", about a string the document wrote, with no mention of bits. A type-check problem with a configuration that a name encodes now names the member and the name: `bits in "r12": ...`. A rule's message is left as written, because such rules already describe the name, as the v2 typestr rules do. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * test(zarr-metadata): pin that the types refuse empty pipelines and edge lists The sharding tests for an empty `codecs` or `index_codecs` checked only the `loc` and kind, which the pipeline's order check also reports. They now also assert `ctx={"min_length": 1}`, which only the type check gives. The rectilinear strategy test checked the grid's rules, which no longer look at empty lists. It now reads the configuration with the definition, which checks the type. The changelog no longer says the JSON Schema of a raw `r` name carries its multiple of 8: that schema is the name pattern. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- packages/zarr-metadata/changes/4635.bugfix.md | 1 + packages/zarr-metadata/changes/4636.bugfix.md | 1 + .../zarr-metadata/changes/4636.feature.md | 1 + packages/zarr-metadata/docs/extensions.md | 6 +- .../zarr-metadata/src/zarr_metadata/_json.py | 8 +- .../src/zarr_metadata/_pydantic_schema.py | 7 +- .../src/zarr_metadata/_typed_json.py | 346 +++++++++++++----- .../src/zarr_metadata/definition/__init__.py | 9 +- .../zarr_metadata/definition/_definition.py | 21 +- .../src/zarr_metadata/model/_array_v3.py | 8 +- .../src/zarr_metadata/testing/_chunk_grid.py | 11 +- .../src/zarr_metadata/testing/_draw.py | 7 +- .../src/zarr_metadata/typed_json.py | 10 +- .../src/zarr_metadata/v3/_json_schema.py | 10 +- .../src/zarr_metadata/v3/_types/chunk_grid.py | 25 +- .../src/zarr_metadata/v3/_types/codec.py | 6 +- .../src/zarr_metadata/v3/_types/data_type.py | 35 +- .../src/zarr_metadata/v3/array.py | 4 +- .../v3/chunk_grid/rectilinear.py | 6 +- .../v3/data_type/fixed_length_utf32.py | 14 - .../v3/data_type/null_terminated_bytes.py | 12 - .../src/zarr_metadata/v3/data_type/raw.py | 12 - .../src/zarr_metadata/v3/data_type/struct.py | 8 +- .../v3/data_type/zarr_python_raw_bytes.py | 12 - .../zarr-metadata/tests/test_json_schema.py | 41 ++- .../zarr-metadata/tests/test_problem_data.py | 5 +- .../zarr-metadata/tests/test_typed_json.py | 305 ++++++++++++--- .../zarr-metadata/tests/testing/_shares.py | 68 +++- .../tests/testing/test_documents.py | 26 +- .../tests/testing/test_strategies.py | 21 +- .../tests/v3/test_every_definition.py | 28 +- .../tests/v3/test_grid_shapes.py | 16 +- .../zarr-metadata/tests/v3/test_sharding.py | 15 +- 33 files changed, 795 insertions(+), 310 deletions(-) create mode 100644 packages/zarr-metadata/changes/4635.bugfix.md create mode 100644 packages/zarr-metadata/changes/4636.bugfix.md create mode 100644 packages/zarr-metadata/changes/4636.feature.md diff --git a/packages/zarr-metadata/changes/4635.bugfix.md b/packages/zarr-metadata/changes/4635.bugfix.md new file mode 100644 index 0000000000..d92c9b5193 --- /dev/null +++ b/packages/zarr-metadata/changes/4635.bugfix.md @@ -0,0 +1 @@ +Strategies in `zarr_metadata.testing.strategies` that draw codec pipelines, such as `array_documents` and `pipelines`, now draw the same examples on every derandomized Hypothesis run. A codec was sampled from options that hold strategies built during the draw, which Hypothesis labels by memory address, so the examples its mutator produced varied from one process to the next. diff --git a/packages/zarr-metadata/changes/4636.bugfix.md b/packages/zarr-metadata/changes/4636.bugfix.md new file mode 100644 index 0000000000..ac16d99ecc --- /dev/null +++ b/packages/zarr-metadata/changes/4636.bugfix.md @@ -0,0 +1 @@ +The `rectilinear` chunk grid now refuses an empty list of chunk edge lengths, such as `"chunk_shapes": [[]]`, even along a dimension of length 0. zarr-extensions#78 made the list form non-empty; a dimension with no chunks writes a bare integer instead. The JSON Schema says `minItems: 1` for the list, and `chunk_grid_configurations` no longer draws an empty list. diff --git a/packages/zarr-metadata/changes/4636.feature.md b/packages/zarr-metadata/changes/4636.feature.md new file mode 100644 index 0000000000..aed68bd9d0 --- /dev/null +++ b/packages/zarr-metadata/changes/4636.feature.md @@ -0,0 +1 @@ +`zarr_metadata.typed_json.check` and definitions' configurations now read more annotated-types constraints, as pydantic does: `MinLen`, `MaxLen` and `Len` on an array or a string, and `MultipleOf` on an integer. A value outside them is an `invalid_value` problem whose `ctx` uses pydantic's names (`min_length`, `multiple_of`), and the JSON Schema writes the matching keyword, such as `minItems`, `minLength` or `multipleOf`. A constraint on a type it does not apply to raises `TypeError`. Constant rules now live on the types, so the JSON Schemas say them too: non-empty `codecs`, sharding `codecs` and `index_codecs`, rectilinear edge lists, `struct` fields and v3 `struct` field names; `length_bytes` of at least 1 for `null_terminated_bytes` and zarr-python `raw_bytes`, and a positive multiple of 4 for `fixed_length_utf32`. Raw `r` declares its bits a positive multiple of 8 on its type, and a refused name's message now names them: `bits in "r12": ...`. `check` now also refuses `"codecs": []`. diff --git a/packages/zarr-metadata/docs/extensions.md b/packages/zarr-metadata/docs/extensions.md index 57acdeffbd..fc314760c8 100644 --- a/packages/zarr-metadata/docs/extensions.md +++ b/packages/zarr-metadata/docs/extensions.md @@ -82,8 +82,10 @@ names `acme.lz4` is then checked against this definition. - a `NewType` or a type alias. A number may carry bounds from annotated-types: `Gt`, `Ge`, `Lt`, `Le` or -`Interval`. Any other `Annotated` metadata is refused when the definition -is built. +`Interval`, and an integer a step, `MultipleOf`. An array or a string may +bound its length with `MinLen`, `MaxLen` or `Len`, so +`Annotated[tuple[int, ...], MinLen(1)]` is never empty. Any other +`Annotated` metadata is refused when the definition is built. **The rules function** diff --git a/packages/zarr-metadata/src/zarr_metadata/_json.py b/packages/zarr-metadata/src/zarr_metadata/_json.py index d5de1537a2..f55081b1f6 100644 --- a/packages/zarr-metadata/src/zarr_metadata/_json.py +++ b/packages/zarr-metadata/src/zarr_metadata/_json.py @@ -127,9 +127,11 @@ class ValidationProblem: document, `input` shows the change. `ctx` holds what was expected, when that is more than a type: - - `gt`, `ge`, `lt` and `le`: the bounds from the value's type or from - a rule, with the names pydantic uses. For example `{"ge": 0, "le": 9}` - for a gzip `level`, whose type is `Annotated[int, Interval(ge=0, le=9)]`. + - `gt`, `ge`, `lt`, `le`, `multiple_of`, `min_length` and + `max_length`: the constraints from the value's type or from a rule, + with the names pydantic uses. For example `{"ge": 0, "le": 9}` for a + gzip `level`, whose type is `Annotated[int, Interval(ge=0, le=9)]`, + or `{"min_length": 1}` for an array document's `codecs`. - `expected`: the values of a closed set, like zod's `values`, in the order the message lists them. For example the values of a `Literal`, of `node_type`, or of `zarr_format`. diff --git a/packages/zarr-metadata/src/zarr_metadata/_pydantic_schema.py b/packages/zarr-metadata/src/zarr_metadata/_pydantic_schema.py index c7b2074ea2..6e97d43d8d 100644 --- a/packages/zarr-metadata/src/zarr_metadata/_pydantic_schema.py +++ b/packages/zarr-metadata/src/zarr_metadata/_pydantic_schema.py @@ -1,8 +1,8 @@ """Private input types used only to generate accurate Pydantic JSON schemas. These repeat the document TypedDicts in a stricter form: extension names -match the spec's pattern, `must_understand` is `true` at an extension -point, and `codecs` is not empty. They exist because pydantic builds a +match the spec's pattern, and `must_understand` is `true` at an extension +point. They exist because pydantic builds a schema from Python types and cannot embed a JSON Schema that has its own `$defs`, such as the ones `zarr_metadata.json_schema` writes; the v3 document schemas are recursive, so they cannot be inlined either. @@ -16,6 +16,7 @@ from collections.abc import Mapping # resolved by Pydantic at runtime from typing import Annotated, Literal, NotRequired +from annotated_types import MinLen from pydantic import Field from typing_extensions import TypedDict @@ -57,7 +58,7 @@ class ZarrV3MandatoryNamedConfigJSON(TypedDict, closed=True): ZarrV3DeclarationJSON = ExtensionName | ZarrV3NamedConfigJSON ZarrV3MandatoryDeclarationJSON = ExtensionName | ZarrV3MandatoryNamedConfigJSON -ZarrV3CodecPipelineJSON = Annotated[tuple[ZarrV3MandatoryDeclarationJSON, ...], Field(min_length=1)] +ZarrV3CodecPipelineJSON = Annotated[tuple[ZarrV3MandatoryDeclarationJSON, ...], MinLen(1)] ZarrV2FilterPipelineJSON = tuple[ZarrV2CodecMetadata, ...] diff --git a/packages/zarr-metadata/src/zarr_metadata/_typed_json.py b/packages/zarr-metadata/src/zarr_metadata/_typed_json.py index e9574b46c4..b774ddd586 100644 --- a/packages/zarr-metadata/src/zarr_metadata/_typed_json.py +++ b/packages/zarr-metadata/src/zarr_metadata/_typed_json.py @@ -9,9 +9,11 @@ `Mapping[str, V]`, a `NewType` (parsed as its supertype), and a type alias (parsed as the type it stands for). This short list keeps the module small. Any other annotation has no parser. A number type may carry bounds, -written with annotated-types as pydantic reads them, for example -`Annotated[int, Interval(ge=0, le=9)]`. A value outside the bounds is a -problem. `constraints_of` lists the bounds the checker supports. +an integer type a step, and an array or a string bounds on its length, written +with annotated-types as pydantic reads them, for example +`Annotated[int, Interval(ge=0, le=9)]`, `Annotated[int, MultipleOf(4)]` or +`Annotated[tuple[int, ...], MinLen(1)]`. A value outside the bounds is a +problem. `constraints_of` lists the constraints the checker supports. A TypedDict is read as the typing spec defines it. This is not always what its runtime attributes say. `typeddict_keys` reads which keys the @@ -86,6 +88,8 @@ ) if TYPE_CHECKING: + from collections.abc import Sized + from zarr_metadata._json import ProblemKind @@ -838,47 +842,118 @@ def parse(candidate: object, loc: Loc) -> Parsed: # --- constraints --------------------------------------------------------- Constraints: TypeAlias = Mapping[str, int | float] -"""The bounds a type carries, keyed by the names pydantic uses: `{"ge": 0, "le": 9}`.""" +"""The constraints a type carries, keyed by the names pydantic uses: `{"ge": 0, "le": 9}`, `{"min_length": 1}`.""" -_BOUNDS: Final[tuple[tuple[type, str], ...]] = ( - (annotated_types.Gt, "gt"), - (annotated_types.Ge, "ge"), - (annotated_types.Lt, "lt"), - (annotated_types.Le, "le"), -) -"""The annotated-types constraints the checker supports, each with its name.""" -_FROM_BELOW: Final = frozenset({"gt", "ge"}) -"""The lower bounds.""" +@dataclass(frozen=True, slots=True) +class _Constraint: + """A constraint the checker reads: its annotated-types marker, and how it is checked and written.""" + + marker: type + """The annotated-types class. Its attribute named as the constraint holds the bound.""" + keywords: Mapping[str | None, str] + """For each shape (from `shape_of`) the constraint applies to, JSON Schema's keyword for it there.""" + holds: Callable[[float, float], bool] + """Whether a measured value satisfies the bound.""" + slot: Literal["below", "above", "step"] + """Which of a type's constraints it is: a lower bound, an upper bound, or a step. A type takes one of each.""" + of: Literal["number", "integer", "length"] + """What the bound limits: a number, an integer, or the length of an array or a string.""" + + +_CONSTRAINTS: Final[Mapping[str, _Constraint]] = { + "gt": _Constraint( + annotated_types.Gt, + {"int": "exclusiveMinimum", "number": "exclusiveMinimum"}, + operator.gt, + "below", + "number", + ), + "ge": _Constraint( + annotated_types.Ge, {"int": "minimum", "number": "minimum"}, operator.ge, "below", "number" + ), + "lt": _Constraint( + annotated_types.Lt, + {"int": "exclusiveMaximum", "number": "exclusiveMaximum"}, + operator.lt, + "above", + "number", + ), + "le": _Constraint( + annotated_types.Le, {"int": "maximum", "number": "maximum"}, operator.le, "above", "number" + ), + "multiple_of": _Constraint( + annotated_types.MultipleOf, + {"int": "multipleOf"}, + lambda value, step: value % step == 0, + "step", + "integer", + ), + "min_length": _Constraint( + annotated_types.MinLen, + {"tuple": "minItems", "str": "minLength"}, + operator.ge, + "below", + "length", + ), + "max_length": _Constraint( + annotated_types.MaxLen, + {"tuple": "maxItems", "str": "maxLength"}, + operator.le, + "above", + "length", + ), +} +"""The annotated-types constraints the checker reads, by the name pydantic uses, which is also the attribute of the marker that holds the bound.""" -_FROM_ABOVE: Final = frozenset({"lt", "le"}) -"""The upper bounds.""" +_NAMES: Final = tuple(each.marker.__name__ for each in _CONSTRAINTS.values()) +"""The names of the markers the checker reads, in the order of `_CONSTRAINTS`.""" -_EXCLUSIVE: Final = frozenset({"gt", "lt"}) -"""The exclusive bounds: a value may not equal them.""" +_MARKERS: Final = f"{', '.join(_NAMES[:-1])} and {_NAMES[-1]}" +"""The markers the checker reads, as an error message lists them: "Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen".""" -_HOLDS: Final[Mapping[str, Callable[[float, float], bool]]] = { - "gt": operator.gt, - "ge": operator.ge, - "lt": operator.lt, - "le": operator.le, +_MEASURED: Final[Mapping[str, str]] = { + "number": "a bound is on a number", + "integer": "a step is on an integer", + "length": "a bound is on an array or a string", } -"""For each bound name, a test of whether a number satisfies that bound.""" +"""For what each constraint limits, what an error message says it applies to.""" +_SECOND: Final[Mapping[str, str]] = { + "below": "bound from below", + "above": "bound from above", + "step": "step", +} +"""For each slot, what an error message calls a second constraint in it.""" -def constraints_of(metadata: Sequence[object]) -> dict[str, int | float]: - """The bounds in an `Annotated` type's metadata, by name: at most one lower and one upper bound. +_STRICTER: Final[Mapping[str, Callable[[int | float, int | float], int | float]]] = { + "below": max, + "above": min, + "step": lambda one, other: math.lcm(int(one), int(other)), +} +"""For each slot, the constraint a value meets exactly when it meets both of two: the stricter bound, or the least common multiple of two steps.""" - The supported bounds are annotated-types' `Gt`, `Ge`, `Lt` and `Le`, - and an `Interval`, which unpacks to them, as pydantic reads them. A - note (a string or a `Doc`) does not constrain the value and is - skipped. Anything else raises `TypeError`: +_EXCLUSIVE: Final = frozenset({"gt", "lt"}) +"""The exclusive bounds: a value may not equal them.""" - - a constraint the checker does not support, such as `MinLen`, - `Predicate`, or pydantic's `Field`. Otherwise the type would declare - a rule that the checker does not enforce. - - a second bound on the same side. pydantic uses the last one. - - a bound that is not a finite number. + +def constraints_of(metadata: Sequence[object]) -> dict[str, int | float]: + """The constraints in an `Annotated` type's metadata, by name: at most one lower bound, one upper bound and one step. + + The supported constraints are annotated-types' `Gt`, `Ge`, `Lt` and + `Le`, which bound a number, `MultipleOf`, which is a step an integer + is a multiple of, `MinLen` and `MaxLen`, which bound the length of an + array or a string, and an `Interval` or a `Len`, which unpack to them, as pydantic + reads them. A note (a string or a `Doc`) does not constrain the value + and is skipped. Anything else raises `TypeError`: + + - a constraint the checker does not support, such as `Predicate` or + pydantic's `Field`. Otherwise the type would declare a rule that the + checker does not enforce. + - a second bound on the same side, or a second step. pydantic uses the + last one. + - a bound that is not a finite number, a step that is not a positive + integer, or a length bound that is not a non-negative integer. A bound is stored as the number it equals, as an integer when it is one. `Ge(True)`, `Ge(1.0)` and `Ge(1)` are the same bound, because @@ -889,16 +964,20 @@ def constraints_of(metadata: Sequence[object]) -> dict[str, int | float]: for item in _unpacked(metadata): if isinstance(item, (str, typing_extensions.Doc)): continue - name = next((name for kind, name in _BOUNDS if isinstance(item, kind)), None) + name = next( + (name for name, each in _CONSTRAINTS.items() if isinstance(item, each.marker)), None + ) if name is None: - msg = f"{item!r} is not a constraint the checker reads, which are Gt, Ge, Lt and Le" + msg = f"{item!r} is not a constraint the checker reads, which are {_MARKERS}" raise UnreadConstraintError(msg) - side = _FROM_BELOW if name in _FROM_BELOW else _FROM_ABOVE - if not side.isdisjoint(said): - where = "below" if side is _FROM_BELOW else "above" - msg = f"{item!r} is a second bound from {where}; a type takes one from each side" + constraint = _CONSTRAINTS[name] + if any( + _CONSTRAINTS[held].slot == constraint.slot and _CONSTRAINTS[held].of == constraint.of + for held in said + ): + msg = f"{item!r} is a second {_SECOND[constraint.slot]}; a type takes one of each" raise TypeError(msg) - said[name] = _bound(item, name) + said[name] = _read_bound(item, name, constraint.of) return said @@ -922,51 +1001,129 @@ def _bound(item: object, name: str) -> int | float: raise TypeError(msg) -def _constrainable(inner: object, said: Constraints) -> None: - """Raise `TypeError` for bounds on a type that is not a number: a string, an array, a value of any type.""" - if len(said) != 0 and shape_of(inner) not in ("int", "number"): - msg = f"{', '.join(said)}: a bound is on a number, and {describe(inner)} is not one" - raise TypeError(msg) +def _read_bound(item: object, name: str, of: str) -> int | float: + """The bound of `item` named `name`, as a constraint on what `of` names reads it; `TypeError` if it cannot be one.""" + if of == "number": + return _bound(item, name) + if of == "integer": + return _step(item, name) + return _length(item, name) -def constrained(parse: Parser, description: str, said: Constraints) -> Parser: - """Wrap `parse` to also check the bounds its type carries: one `invalid_value` problem if any bound fails. +def _step(item: object, name: str) -> int: + """The step of `item` named `name`; `TypeError` if it is not a positive integer, as JSON Schema requires of `multipleOf`.""" + bound = _bound(item, name) + if isinstance(bound, int) and bound > 0: + return bound + msg = f"{item!r}: a step is a positive integer" + raise TypeError(msg) + + +def _length(item: object, name: str) -> int: + """The length of length bound `item` named `name`; `TypeError` if it is not a non-negative integer.""" + bound = _bound(item, name) + if isinstance(bound, int) and bound >= 0: + return bound + msg = f"{item!r}: a length bound is a non-negative integer" + raise TypeError(msg) + + +def _constrainable(inner: object, said: Constraints) -> None: + """Raise `TypeError` for a constraint on a type it does not limit: a bound on what is not a number, a step on what is not an integer, or a length bound on what is not an array or a string.""" + shape = shape_of(inner) + for of, applies in _MEASURED.items(): + names = [name for name in said if _CONSTRAINTS[name].of == of] + if any(shape not in _CONSTRAINTS[name].keywords for name in names): + msg = f"{', '.join(names)}: {applies}, and {describe(inner)} is not one" + raise TypeError(msg) + - The message states what the type allows, for example "expected an - integer in [0, 9], got 12", and the problem's `ctx` holds the bounds. - The bounds are checked only when `parse` reports no problem, in the - same way that rules run only on a value of their type. +def constrained( + parse: Parser, description: str, said: Constraints, unit: str = "element" +) -> Parser: + """Wrap `parse` to also check the constraints its type carries: one `invalid_value` problem if any fails. + + A bound limits a number, a step what an integer is a multiple of, and + a length bound the number of `unit`s in the value: the elements of an + array, or the characters of a string, which are its code points, as + JSON Schema counts them. The message states what the type allows, for + example "expected an integer in [0, 9], got 12", "expected an integer + > 0 that is a multiple of 4, got 6" or "expected an array of integers + with at least 1 element, got []", and the problem's `ctx` holds the + constraints, as pydantic names them. The constraints are checked only + on a value of the type, in the same way that rules are: when `parse` + reports no problem, or only keys that a closed TypedDict does not + declare. Such a key is left out and the value is still returned, so + the value it returns is held to the constraints too. """ - expected = f"expected {description} {_admitted(said)}" + expected = f"expected {description} {_admitted(said, unit)}" ctx: dict[str, JSONValue] = dict(said) - holds = tuple((_HOLDS[name], bound) for name, bound in said.items()) + checks = tuple( + (_CONSTRAINTS[name].holds, bound, _CONSTRAINTS[name].of == "length") + for name, bound in said.items() + ) def parse_constrained(value: object, loc: Loc) -> Parsed: typed, found = parse(value, loc) - if len(found) != 0: + if any(each.kind != "unknown_key" for each in found): return typed, found - number = cast("float", typed) - for keeps, bound in holds: - if not keeps(number, bound): + for keeps, bound, length in checks: + measured = len(cast("Sized", typed)) if length else cast("float", typed) + if not keeps(measured, bound): message = f"{expected}, got {shown(value)}" - return typed, (ValidationProblem(loc, message, "invalid_value", ctx=ctx),) - return typed, () + return typed, (*found, ValidationProblem(loc, message, "invalid_value", ctx=ctx)) + return typed, found return parse_constrained -def _admitted(said: Constraints) -> str: - """The range the bounds allow, as written after the type in a message: "in [0, 9]", "in (0, 1)", ">= 1".""" - low = next(((name, bound) for name, bound in said.items() if name in _FROM_BELOW), None) - high = next(((name, bound) for name, bound in said.items() if name in _FROM_ABOVE), None) +def _admitted(said: Constraints, unit: str) -> str: + """What the constraints allow, as written after the type in a message: "in [0, 9]", ">= 1", "> 0 that is a multiple of 4", "with at least 1 element".""" + if any(_CONSTRAINTS[name].of == "length" for name in said): + return _admitted_lengths(said.get("min_length"), said.get("max_length"), unit) + low = next( + ((name, bound) for name, bound in said.items() if _CONSTRAINTS[name].slot == "below"), None + ) + high = next( + ((name, bound) for name, bound in said.items() if _CONSTRAINTS[name].slot == "above"), None + ) + bounded = _admitted_range(low, high) + step = said.get("multiple_of") + if step is None: + return bounded + multiple = f"that is a multiple of {shown(step)}" + return multiple if bounded == "" else f"{bounded} {multiple}" + + +def _admitted_range( + low: tuple[str, int | float] | None, high: tuple[str, int | float] | None +) -> str: + """The range a lower and an upper bound allow, each named, as written after the type in a message: "in [0, 9]", "in (0, 1)", ">= 1"; empty without either.""" if low is not None and high is not None: opening = "(" if low[0] in _EXCLUSIVE else "[" closing = ")" if high[0] in _EXCLUSIVE else "]" return f"in {opening}{shown(low[1])}, {shown(high[1])}{closing}" if low is not None: return f"{'>' if low[0] in _EXCLUSIVE else '>='} {shown(low[1])}" - name, bound = cast("tuple[str, int | float]", high) - return f"{'<' if name in _EXCLUSIVE else '<='} {shown(bound)}" + if high is not None: + return f"{'<' if high[0] in _EXCLUSIVE else '<='} {shown(high[1])}" + return "" + + +def _admitted_lengths(low: float | None, high: float | None, unit: str) -> str: + """The lengths that length bounds allow, counted in `unit`s, as written after the type in a message: "with at least 1 element", "with 1 to 3 characters".""" + if low is not None and high is not None: + if low == high: + return f"with {_counted(low, unit)}" + return f"with {shown(low)} to {_counted(high, unit)}" + if low is not None: + return f"with at least {_counted(low, unit)}" + return f"with at most {_counted(cast('float', high), unit)}" + + +def _counted(count: float, unit: str) -> str: + """A number of `unit`s, as a message writes it: "1 element", "3 characters".""" + return f"{shown(count)} {unit}{'' if count == 1 else 's'}" # --- the compiler -------------------------------------------------------- @@ -1137,7 +1294,8 @@ def _compile(annotation: object, leaf: Leaf, building: _Building) -> Parser | No if len(said) == 0: return parse _constrainable(inner, said) - return constrained(parse, describe(inner), said) + unit = "character" if shape_of(inner) == "str" else "element" + return constrained(parse, describe(inner), said, unit) def _compile_type(inner: object, leaf: Leaf, building: _Building) -> Parser | None: @@ -1314,7 +1472,7 @@ def _typeddicts_in(annotation: object, seen: frozenset[object]) -> tuple[type, . def _metadata_problem(inner: object, metadata: tuple[object, ...]) -> TypeError | None: - """The error the checker raises for `metadata` on a member of type `inner`: a constraint it does not read, or a bound on a type that is not a number; None when it reads the metadata, as it does a note.""" + """The error the checker raises for `metadata` on a member of type `inner`: a constraint it does not read, or a constraint on a type it does not limit; None when it reads the metadata, as it does a note.""" try: _constrainable(inner, constraints_of(metadata)) except TypeError as error: @@ -1382,22 +1540,6 @@ def check( DIALECT: Final = "https://json-schema.org/draft/2020-12/schema" """The dialect of every schema: JSON Schema draft 2020-12, the same dialect pydantic and zod use.""" -_KEYWORDS: Final[Mapping[str, str]] = { - "gt": "exclusiveMinimum", - "ge": "minimum", - "lt": "exclusiveMaximum", - "le": "maximum", -} -"""JSON Schema's keyword for each bound.""" - -_STRICTER: Final[Mapping[str, Callable[[float, float], float]]] = { - "exclusiveMinimum": max, - "minimum": max, - "exclusiveMaximum": min, - "maximum": min, -} -"""For each keyword, the function that picks the stricter of two bounds.""" - def no_schema_leaf(annotation: object, schemas: Schemas) -> JSONSchema | None: """The schema leaf for a caller with no shapes of its own.""" @@ -1424,15 +1566,17 @@ def __init__(self, leaf: SchemaLeaf = no_schema_leaf) -> None: self._uses: dict[str, int] = {} def of(self, annotation: object) -> JSONSchema: - """The schema of `annotation` as the checker reads it: its type, with the bounds and notes from `Annotated`. - - Each bound becomes the matching JSON Schema keyword (`Ge(0)` becomes - `minimum`), and a `Doc` becomes the `description`. The schema keeps - both the type's bounds and the bounds of its `NewType`, because the - checker applies both. When both set the same keyword, the stricter - bound is kept. An annotation the checker supports is written. Any - other raises `TypeError`, which cannot happen if the caller already - checked the annotation with `parser`. + """The schema of `annotation` as the checker reads it: its type, with the constraints and notes from `Annotated`. + + Each constraint becomes the matching JSON Schema keyword (`Ge(0)` + becomes `minimum`, `MinLen(1)` becomes `minItems` on an array and + `minLength` on a string), and a `Doc` + becomes the `description`. The schema keeps both the type's bounds + and the bounds of its `NewType`, or of the fixed length of its + tuple, because the checker applies both. When both set the same + keyword, the stricter bound is kept. An annotation the checker + supports is written. Any other raises `TypeError`, which cannot + happen if the caller already checked the annotation with `parser`. """ inner, metadata = strip_annotation(annotation) schema = self._type(inner) @@ -1445,10 +1589,15 @@ def of(self, annotation: object) -> JSONSchema: ] if len(notes) != 0: schema = {**schema, "description": "\n\n".join(notes)} - for name, bound in constraints_of(metadata).items(): - keyword = _KEYWORDS[name] + said = constraints_of(metadata) + _constrainable(inner, said) + shape = shape_of(inner) + for name, bound in said.items(): + constraint = _CONSTRAINTS[name] + keyword = constraint.keywords[shape] held = cast("int | float | None", schema.get(keyword)) - schema = {**schema, keyword: bound if held is None else _STRICTER[keyword](held, bound)} + kept = bound if held is None else _STRICTER[constraint.slot](held, bound) + schema = {**schema, keyword: kept} return schema def object_of(self, typeddict: type) -> JSONSchema: @@ -1591,8 +1740,9 @@ def json_schema(shape: type) -> JSONSchema: - A TypedDict becomes an object with its keys, its required keys, and what any other key may hold (nothing, for a closed TypedDict). - - A bound becomes the matching JSON Schema keyword: `Ge(0)` becomes - `minimum`. + - A constraint becomes the matching JSON Schema keyword: `Ge(0)` + becomes `minimum`, and `MinLen(1)` becomes `minItems` on an array + and `minLength` on a string. - A `Doc` becomes the `description`. Only a `Doc` produces a description; docstrings are for Python readers and are not used. This matches zod, which writes only what `.describe()` sets. diff --git a/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py b/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py index 4dc31e9162..f259f3d7fa 100644 --- a/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py @@ -163,15 +163,18 @@ def acme_lz4_rules( type may carry bounds, written with annotated-types and read the way pydantic reads them: `Gt`, `Ge`, `Lt`, `Le` and `Interval`, at most one from each side, at any depth. For example, -`tuple[Annotated[int, Ge(1)], ...]` bounds each element. A value out of +`tuple[Annotated[int, Ge(1)], ...]` bounds each element. An integer's +type may carry a step, `MultipleOf`, at most one. An array's or a +string's type may bound its length with `MinLen`, `MaxLen` and `Len`, so +`Annotated[tuple[int, ...], MinLen(1)]` is never empty. A value out of bounds is an `invalid_value` problem. Its message says what the type admits ("expected an integer >= 1, got 0"), and its `ctx` holds the bounds. The rules run only on a configuration within its bounds, so a rule can rely on them, as pydantic's after-validators and zod's refinements do. Until a value out of bounds is fixed, it is the only problem reported for the configuration. `Annotated` may also carry a -note, as a string or a `Doc`. Any other metadata (a `MinLen`, a -`Predicate`, pydantic's `Field`) is rejected when the definition is +note, as a string or a `Doc`. Any other metadata (a `Predicate`, +pydantic's `Field`) is rejected when the definition is built, because a type whose constraint the checker does not enforce would give false information. diff --git a/packages/zarr-metadata/src/zarr_metadata/definition/_definition.py b/packages/zarr-metadata/src/zarr_metadata/definition/_definition.py index 933f2f7f55..05eefcf761 100644 --- a/packages/zarr-metadata/src/zarr_metadata/definition/_definition.py +++ b/packages/zarr-metadata/src/zarr_metadata/definition/_definition.py @@ -1717,6 +1717,10 @@ def _read_carried( declared = {key: value for key, value in carried.items() if key in members} undeclared = [key for key in carried if key not in members] configuration, judged = definition.read_configuration(declared) + # The type check describes a configuration member, which the document + # never wrote, so its messages name the member and the name it was read + # from. A rule of such a definition writes its messages for the name. + _, typed = definition._check_configuration(declared) # A problem with the configuration encoded in the name belongs to the # declaration. It is located at the declaration, where the name is, and the # `input` and `ctx` that described a configuration member do not apply @@ -1725,7 +1729,14 @@ def _read_carried( problems = ( *beside, *( - dataclasses.replace(found, loc=loc, input=UNSET, ctx={}, kind="invalid_value") + dataclasses.replace( + found, + loc=loc, + input=UNSET, + ctx={}, + kind="invalid_value", + message=_read_from(name, found) if found in typed else found.message, + ) for found in judged ), ) @@ -1741,6 +1752,14 @@ def _read_carried( ), problems +def _read_from(name: str, found: ValidationProblem) -> str: + """The message of `found`, a problem the type check found in the configuration that `name` encodes, naming the member it is about and the name: `bits in "r12": expected ...`.""" + if len(found.loc) == 0: + return f"{shown(name)}: {found.message}" + member = ".".join(str(part) for part in found.loc) + return f"{member} in {shown(name)}: {found.message}" + + def _sized(declaration: _NestedDeclaration, inner: ResolvedDeclaration[Any]) -> Problems: """The problems with the definition that read `inner` in the member that holds `declaration`, as the kind of `declaration` reports them for its alias. diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py b/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py index 22e4ecc466..627545859c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py @@ -27,7 +27,7 @@ within, ) from zarr_metadata._sentinel import UNSET -from zarr_metadata._typed_json import typeddict_keys +from zarr_metadata._typed_json import strip_annotation, typeddict_keys from zarr_metadata.definition._context import disagreements, scoped from zarr_metadata.definition._definition import ( AcceptedDeclaration, @@ -132,10 +132,10 @@ def members_past_the_levels( _EXTENSION_LISTS_V3: Final[tuple[tuple[str, type[Definition[Any]]], ...]] = tuple( (key, kind) for key, (annotation, _) in _MEMBERS_V3.items() - if get_origin(annotation) is tuple - and (kind := declaration_kind(get_args(annotation)[0])) is not None + if get_origin(listed := strip_annotation(annotation)[0]) is tuple + and (kind := declaration_kind(get_args(listed)[0])) is not None ) -"""The lists of extension points of a v3 array document, and the kind each entry is read as: each member that the TypedDict annotates as a tuple of a declaration alias.""" +"""The lists of extension points of a v3 array document, and the kind each entry is read as: each member that the TypedDict annotates as a tuple of a declaration alias, whatever constraints the tuple carries.""" def no_extra_fields() -> Mapping[str, ResolvedDeclaration[Any]]: diff --git a/packages/zarr-metadata/src/zarr_metadata/testing/_chunk_grid.py b/packages/zarr-metadata/src/zarr_metadata/testing/_chunk_grid.py index a987f241c8..6a88f813ea 100644 --- a/packages/zarr-metadata/src/zarr_metadata/testing/_chunk_grid.py +++ b/packages/zarr-metadata/src/zarr_metadata/testing/_chunk_grid.py @@ -89,17 +89,14 @@ def _edges(draw: st.DrawFn, extent: int | None) -> tuple[int | tuple[int, int], can be long, since a pair costs the same however many chunks it counts, which is what the spelling is for. Its edges often share a factor, as they must for a shard to divide them into inner chunks - longer than 1; edges drawn apart rarely do. A - dimension of length zero, or of a length not known, may be covered by - no edges at all. + longer than 1; edges drawn apart rarely do. There is always at least + one edge, even along a dimension of length zero: the spec allows no + empty list, and a dimension with no chunks writes a bare length instead. """ - min_runs = 0 if extent is None or extent == 0 else 1 factor = draw(st.just(1) | st.integers(2, _MAX_FACTOR)) sizes = [ factor * size - for size in draw( - st.lists(st.integers(1, _MAX_LENGTH), min_size=min_runs, max_size=_MAX_RUNS) - ) + for size in draw(st.lists(st.integers(1, _MAX_LENGTH), min_size=1, max_size=_MAX_RUNS)) ] runs = [(size, draw(_count(extent, size))) for size in sizes] covered = sum(size * count for size, count in runs) diff --git a/packages/zarr-metadata/src/zarr_metadata/testing/_draw.py b/packages/zarr-metadata/src/zarr_metadata/testing/_draw.py index f2c0ec3c0b..e63d4afe1c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/testing/_draw.py +++ b/packages/zarr-metadata/src/zarr_metadata/testing/_draw.py @@ -278,7 +278,12 @@ def pipelines_for( def _codec( draw: st.DrawFn, options: Sequence[tuple[K, Configurations]] ) -> tuple[K, ZarrV3DeclarationJSON]: - definition, configurations = draw(st.sampled_from(options)) + # An index, not the options: Hypothesis labels a `sampled_from` by + # hashing its elements, and each option holds a strategy built for this + # draw, whose hash is its address. Labels that follow addresses send + # Hypothesis's mutator down different paths from one run to the next, so + # a derandomized run would not draw the same examples twice. + definition, configurations = options[draw(st.sampled_from(range(len(options))))] return definition, draw(any_spelling(definition, configurations)) diff --git a/packages/zarr-metadata/src/zarr_metadata/typed_json.py b/packages/zarr-metadata/src/zarr_metadata/typed_json.py index 982a3d4868..b98d392927 100644 --- a/packages/zarr-metadata/src/zarr_metadata/typed_json.py +++ b/packages/zarr-metadata/src/zarr_metadata/typed_json.py @@ -41,8 +41,14 @@ - A number type may carry bounds, written with annotated-types as pydantic reads them: `Gt`, `Ge`, `Lt`, `Le` and `Interval`, at any depth. For example, `tuple[Annotated[int, Ge(1)], ...]` bounds each - element. A value outside the bounds is an `invalid_value` problem. Its - message states what the type allows, and its `ctx` holds the bounds. + element. An integer type may also carry a step, `MultipleOf`: + `Annotated[int, Gt(0), MultipleOf(4)]` is a positive multiple of 4. An + array or a string type may bound its length the same way, with + `MinLen`, `MaxLen` and `Len`: `Annotated[tuple[int, ...], MinLen(1)]` + is a non-empty array. A string's length counts its code points, as + JSON Schema counts them. A value outside the bounds is an `invalid_value` + problem. Its message states what the type allows, and its `ctx` holds + the bounds, as pydantic names them (`ge`, `min_length`). `Annotated` may also carry a note (a string or a `Doc`). Any other metadata raises `TypeError`, because `check` would not enforce a constraint it does not support. diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_json_schema.py b/packages/zarr-metadata/src/zarr_metadata/v3/_json_schema.py index f647895154..f4dd1e6f03 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/_json_schema.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_json_schema.py @@ -274,16 +274,8 @@ def _member_object(schema: Mapping[str, JSONValue], key: str) -> Mapping[str, JS def _array(context: ZarrV3Context, schemas: Schemas) -> JSONSchema: - """An array document: its TypedDict, at least one codec, and a constraint on its fill value for each data type in scope, which applies when the document names that data type. - - The TypedDict cannot declare a minimum length, so the schema adds it: - a pipeline needs its array -> bytes codec, and the reader rejects an - empty `codecs`. - """ + """An array document: its TypedDict, and a constraint on its fill value for each data type in scope, which applies when the document names that data type.""" schema = schemas.object_of(ZarrV3ArrayMetadataJSON) - properties = _member_object(schema, "properties") - codecs = _member_object(properties, "codecs") - schema = {**schema, "properties": {**properties, "codecs": {**codecs, "minItems": 1}}} schema = _with_extra_fields(schema, context.array_extra_fields, schemas) held: list[JSONValue] = [] for definition in context.data_types.values(): diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_types/chunk_grid.py b/packages/zarr-metadata/src/zarr_metadata/v3/_types/chunk_grid.py index ba34cde633..16135ea84d 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/_types/chunk_grid.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_types/chunk_grid.py @@ -10,7 +10,7 @@ from typing import Annotated, Final, Literal, NotRequired -from annotated_types import Ge +from annotated_types import Ge, MinLen from typing_extensions import TypedDict # regular: https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#regular-grids @@ -67,7 +67,7 @@ class RegularChunkGridObject(TypedDict, closed=True): """ -# rectilinear: https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/chunk-grids/rectilinear/README.md +# rectilinear: https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/chunk-grids/rectilinear/README.md RECTILINEAR_CHUNK_GRID_NAME: Final = "rectilinear" """The `name` field value of the rectilinear chunk grid.""" @@ -77,12 +77,19 @@ class RegularChunkGridObject(TypedDict, closed=True): _Positive = Annotated[int, Ge(1)] -RectilinearDimSpec = _Positive | tuple[_Positive | tuple[_Positive, _Positive], ...] +RectilinearDimSpec = ( + _Positive | Annotated[tuple[_Positive | tuple[_Positive, _Positive], ...], MinLen(1)] +) """JSON shape for one dimension's rectilinear spec. Either a bare integer (uniform shorthand for a regular dimension within -a rectilinear grid), or a tuple of integers and/or `[value, count]` RLE -pairs. Every extent, and every run's length and count, is at least 1. +a rectilinear grid), or a non-empty tuple of integers and/or +`[value, count]` RLE pairs. Every extent, and every run's length and +count, is at least 1. The tuple is non-empty even for a dimension of +length 0, which writes a bare integer to have no chunks: "as a non-empty +array", and "For an axis of length 0, the array form still requires at +least one edge length" +(https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/chunk-grids/rectilinear/README.md?plain=1#L89-L95). """ @@ -95,9 +102,9 @@ class RectilinearChunkGridConfiguration(TypedDict, closed=True): """The chunk edge lengths along each dimension of the array, one entry per dimension. An entry is a bare integer, a step repeated until it covers the - dimension, or a list of edge lengths and `[length, count]` runs, whose - lengths must sum to at least the dimension's length. Every length and - count is at least 1. + dimension, or a non-empty list of edge lengths and `[length, count]` + runs, whose lengths must sum to at least the dimension's length. Every + length and count is at least 1. """ @@ -121,7 +128,7 @@ class RectilinearChunkGridObject(TypedDict, closed=True): `kind` and `chunk_shapes` are required, so only the object form is valid; the short-hand-name form is not permitted by the spec for this grid. - https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/chunk-grids/rectilinear/README.md#L59-L62 + https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/chunk-grids/rectilinear/README.md#L59-L62 https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1562-L1564 """ diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_types/codec.py b/packages/zarr-metadata/src/zarr_metadata/v3/_types/codec.py index 00792264ea..806ce06c29 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/_types/codec.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_types/codec.py @@ -12,7 +12,7 @@ from typing import Annotated, Final, Literal, NotRequired -from annotated_types import Ge, Interval +from annotated_types import Ge, Interval, MinLen from typing_extensions import TypedDict from zarr_metadata._common import EmptyConfiguration, JSONValue @@ -510,12 +510,12 @@ class ShardingIndexedCodecConfiguration(TypedDict, closed=True): One length, at least 1, per axis of the shard the codec is handed, each dividing the shard's length along that axis. """ - codecs: tuple[ZarrV3CodecDeclaration, ...] + codecs: Annotated[tuple[ZarrV3CodecDeclaration, ...], MinLen(1)] """The codec pipeline that encodes each inner chunk. Like an array's `codecs`, it holds exactly one array -> bytes codec. """ - index_codecs: tuple[ZarrV3StaticCodecDeclaration, ...] + index_codecs: Annotated[tuple[ZarrV3StaticCodecDeclaration, ...], MinLen(1)] """The codec pipeline that encodes the shard index. It holds exactly one array -> bytes codec, and only codecs whose output diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_types/data_type.py b/packages/zarr-metadata/src/zarr_metadata/v3/_types/data_type.py index 3c8b19d703..69f304bb0c 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/_types/data_type.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_types/data_type.py @@ -13,7 +13,7 @@ from collections.abc import Mapping from typing import Annotated, Final, Literal, NewType, NotRequired, get_args -from annotated_types import Interval +from annotated_types import Ge, Gt, Interval, MinLen, MultipleOf from typing_extensions import ReadOnly, TypedDict from zarr_metadata._common import JSONValue @@ -352,10 +352,11 @@ class RawBytesConfiguration(TypedDict, closed=True): back as the name. """ - bits: int + bits: Annotated[int, Gt(0), MultipleOf(8)] """The size in bits that the name writes, `16` for `r16`. - A positive multiple of 8. + A positive multiple of 8: "raw bits, variable size given by *, limited + to be a multiple of 8". Zero bits is not a valid type. """ @@ -411,9 +412,11 @@ class RawBytesConfiguration(TypedDict, closed=True): class FixedLengthUtf32Configuration(TypedDict, closed=True): """Configuration for the `fixed_length_utf32` data type.""" - length_bytes: ReadOnly[int] - """The size in bytes of every value: a positive multiple of 4, since - each code point takes 4 bytes.""" + length_bytes: ReadOnly[Annotated[int, Gt(0), MultipleOf(4)]] + """The size in bytes of every value: a "positive integer, multiple of `4`", since each code point takes 4 bytes. + + https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/data-types/fixed_length_utf32/README.md?plain=1#L41 + """ class FixedLengthUtf32(TypedDict, closed=True): @@ -447,8 +450,11 @@ class FixedLengthUtf32(TypedDict, closed=True): class NullTerminatedBytesConfiguration(TypedDict, closed=True): """Configuration for the `null_terminated_bytes` data type.""" - length_bytes: ReadOnly[int] - """The size in bytes of every value, at least 1.""" + length_bytes: ReadOnly[Annotated[int, Ge(1)]] + """The size in bytes of every value, at least 1. + + zarr-python refuses a length below 1, since such a type holds no data. + """ class NullTerminatedBytes(TypedDict, closed=True): @@ -482,8 +488,11 @@ class NullTerminatedBytes(TypedDict, closed=True): class ZarrPythonRawBytesConfiguration(TypedDict, closed=True): """Configuration for zarr-python's `raw_bytes` data type.""" - length_bytes: ReadOnly[int] - """The size in bytes of every value, at least 1.""" + length_bytes: ReadOnly[Annotated[int, Ge(1)]] + """The size in bytes of every value, at least 1. + + zarr-python refuses a length below 1, since such a type holds no data. + """ class ZarrPythonRawBytes(TypedDict, closed=True): @@ -613,8 +622,8 @@ class NumpyTimedelta64(TypedDict, closed=True): class StructField(TypedDict, closed=True): """A single field entry inside a structured dtype.""" - name: ReadOnly[str] - """The field name (must be unique within a struct and non-empty).""" + name: ReadOnly[Annotated[str, MinLen(1)]] + """The field name: non-empty, and unique within the struct.""" data_type: ReadOnly[ZarrV3DataTypeDeclaration] """The field's data type. @@ -626,7 +635,7 @@ class StructField(TypedDict, closed=True): class StructConfiguration(TypedDict, closed=True): """Configuration for the `struct` data type.""" - fields: ReadOnly[tuple[StructField, ...]] + fields: ReadOnly[Annotated[tuple[StructField, ...], MinLen(1)]] """The record's fields, in the order they are encoded. At least one. The order is significant: it is the order in which the diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/array.py b/packages/zarr-metadata/src/zarr_metadata/v3/array.py index 2f256e219b..1d13e60f64 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/array.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/array.py @@ -3,7 +3,7 @@ from collections.abc import Mapping from typing import Annotated, Final, Literal, NotRequired -from annotated_types import Ge +from annotated_types import Ge, MinLen from typing_extensions import TypedDict from zarr_metadata._common import JSONValue @@ -47,7 +47,7 @@ class ZarrV3ArrayMetadataJSON(TypedDict, extra_items=ZarrV3ExtensionField): Which JSON values it may take depends on `data_type`. """ - codecs: tuple[ZarrV3CodecDeclaration, ...] + codecs: Annotated[tuple[ZarrV3CodecDeclaration, ...], MinLen(1)] """The codecs that encode each chunk, in encoding order. Zero or more array -> array codecs, then exactly one array -> bytes codec, diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py index 0b6db11ac1..2d7b2808a6 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py @@ -1,7 +1,7 @@ """ Rectilinear chunk grid (zarr-extensions). -See https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/chunk-grids/rectilinear/README.md +See https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/chunk-grids/rectilinear/README.md """ from collections.abc import Iterator @@ -79,7 +79,7 @@ def _shape_rules( "The length of `chunk_shapes` MUST match the number of dimensions of the array", and "The sum of the edge lengths MUST equal or exceed `L`" - (https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/chunk-grids/rectilinear/README.md?plain=1#L62-L91). + (https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/chunk-grids/rectilinear/README.md?plain=1#L62-L95). A bare integer repeats until it covers the dimension, so it always passes this check. """ @@ -112,7 +112,7 @@ def _chunk_lengths( A bare integer is the length of every chunk along its axis. A list gives the length of each chunk, and a `[length, count]` pair gives `count` chunks of that length - (https://github.com/zarr-developers/zarr-extensions/blob/4da7b37a84f76e660902f6d3de3eaef0e0febae6/chunk-grids/rectilinear/README.md?plain=1#L62-L91). + (https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/chunk-grids/rectilinear/README.md?plain=1#L62-L95). Every listed length counts, including the length of a chunk past the array's edge, which the array grows into when it is resized. """ diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/fixed_length_utf32.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/fixed_length_utf32.py index 7bb8c135e6..9c585d20ba 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/fixed_length_utf32.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/fixed_length_utf32.py @@ -21,19 +21,6 @@ _CODE_POINT_BYTES: Final = 4 -def _rules( - configuration: FixedLengthUtf32Configuration, nested: Nested -) -> Iterator[ValidationProblem]: - """Check that `length_bytes` is a "positive integer, multiple of `4`" (https://github.com/zarr-developers/zarr-extensions/blob/cadd1dab6c9aebafb66080b78b583f6507466b3d/data-types/fixed_length_utf32/README.md?plain=1#L41).""" - length_bytes = configuration["length_bytes"] - if length_bytes <= 0 or length_bytes % _CODE_POINT_BYTES != 0: - yield ValidationProblem( - ("length_bytes",), - f"expected a positive multiple of {_CODE_POINT_BYTES}, got {length_bytes}", - "invalid_value", - ) - - def _fill_value_rules( configuration: FixedLengthUtf32Configuration, nested: Nested, value: FixedLengthUtf32FillValue ) -> Iterator[ValidationProblem]: @@ -73,7 +60,6 @@ def _fill_value_canonical( FIXED_LENGTH_UTF32_DATA_TYPE: Final = DataTypeDefinition( name=FIXED_LENGTH_UTF32_DATA_TYPE_NAME, configuration=FixedLengthUtf32Configuration, - rules=_rules, fill_value=FixedLengthUtf32FillValue, fill_value_default=ConstantFillValue(""), fill_value_rules=_fill_value_rules, diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/null_terminated_bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/null_terminated_bytes.py index 333f362cb5..7217f5eb1d 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/null_terminated_bytes.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/null_terminated_bytes.py @@ -25,17 +25,6 @@ from zarr_metadata.v3.data_type._byte import base64_decoded, base64_problems, base64_spelling -def _rules( - configuration: NullTerminatedBytesConfiguration, nested: Nested -) -> Iterator[ValidationProblem]: - """Check that `length_bytes` is positive. zarr-python refuses a length below 1, since such a type holds no data.""" - length_bytes = configuration["length_bytes"] - if length_bytes < 1: - yield ValidationProblem( - ("length_bytes",), f"expected a positive size, got {length_bytes}", "invalid_value" - ) - - def _fill_value_rules( configuration: NullTerminatedBytesConfiguration, nested: Nested, @@ -75,7 +64,6 @@ def _fill_value_canonical( NULL_TERMINATED_BYTES_DATA_TYPE: Final = DataTypeDefinition( name=NULL_TERMINATED_BYTES_DATA_TYPE_NAME, configuration=NullTerminatedBytesConfiguration, - rules=_rules, fill_value=NullTerminatedBytesFillValue, fill_value_default=ConstantFillValue(""), fill_value_rules=_fill_value_rules, diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py index 63fc15e878..8af68ee617 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py @@ -41,17 +41,6 @@ def raw_bytes_dtype_name(value: str) -> RawBytesDataTypeName: return RawBytesDataTypeName(value) -def _rules(configuration: RawBytesConfiguration, nested: Nested) -> Iterator[ValidationProblem]: - """Check the size: "raw bits, variable size given by *, limited to be a multiple of 8". Zero bits is not a valid type.""" - bits = configuration["bits"] - if bits == 0 or bits % 8 != 0: - yield ValidationProblem( - ("bits",), - f"expected a size in bits that is a positive multiple of 8, got {bits}", - "invalid_value", - ) - - def _fill_value_rules( configuration: RawBytesConfiguration, nested: Nested, value: RawBytesFillValue ) -> Iterator[ValidationProblem]: @@ -88,7 +77,6 @@ def _storage(configuration: RawBytesConfiguration, nested: Nested) -> StorageCla RAW_BYTES_DATA_TYPE: Final = DataTypeDefinition( name=RAW_BYTES_NAME, configuration=RawBytesConfiguration, - rules=_rules, fill_value=RawBytesFillValue, fill_value_default=_fill_value_default, fill_value_rules=_fill_value_rules, diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/struct.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/struct.py index af9fbc0038..9034b80443 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/struct.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/struct.py @@ -32,7 +32,7 @@ def _rules(configuration: StructConfiguration, nested: Nested) -> Iterator[ValidationProblem]: - """Check that there is at least one field, that field names are non-empty and distinct, and that field types have a fixed size. + """Check that field names are distinct, and that field types have a fixed size. Names must be distinct because a fill value addresses fields by name. "Variable-length data types @@ -43,15 +43,9 @@ def _rules(configuration: StructConfiguration, nested: Nested) -> Iterator[Valid whose storage is unknown is not checked. """ fields = configuration["fields"] - if len(fields) == 0: - yield ValidationProblem(("fields",), "expected at least one struct field", "invalid_value") seen: dict[str, int] = {} for index, member in enumerate(fields): name = member["name"] - if name == "": - yield ValidationProblem( - ("fields", index, "name"), "expected a non-empty field name", "invalid_value" - ) first = seen.setdefault(name, index) if first != index: yield ValidationProblem( diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/zarr_python_raw_bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/zarr_python_raw_bytes.py index e1f41efcc9..cc9aa3bccb 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/zarr_python_raw_bytes.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/zarr_python_raw_bytes.py @@ -29,17 +29,6 @@ from zarr_metadata.v3.data_type._byte import base64_decoded, base64_problems, base64_spelling -def _rules( - configuration: ZarrPythonRawBytesConfiguration, nested: Nested -) -> Iterator[ValidationProblem]: - """Check that `length_bytes` is positive. zarr-python refuses a length below 1, since such a type holds no data.""" - length_bytes = configuration["length_bytes"] - if length_bytes < 1: - yield ValidationProblem( - ("length_bytes",), f"expected a positive size, got {length_bytes}", "invalid_value" - ) - - def _fill_value_rules( configuration: ZarrPythonRawBytesConfiguration, nested: Nested, @@ -82,7 +71,6 @@ def _fill_value_default(configuration: ZarrPythonRawBytesConfiguration, nested: ZARR_PYTHON_RAW_BYTES_DATA_TYPE: Final = DataTypeDefinition( name=ZARR_PYTHON_RAW_BYTES_DATA_TYPE_NAME, configuration=ZarrPythonRawBytesConfiguration, - rules=_rules, fill_value=ZarrPythonRawBytesFillValue, fill_value_default=_fill_value_default, fill_value_rules=_fill_value_rules, diff --git a/packages/zarr-metadata/tests/test_json_schema.py b/packages/zarr-metadata/tests/test_json_schema.py index 48d4b6f2fb..5e7f50c240 100644 --- a/packages/zarr-metadata/tests/test_json_schema.py +++ b/packages/zarr-metadata/tests/test_json_schema.py @@ -15,7 +15,7 @@ from typing import Annotated, Any, Literal, NewType, NotRequired, cast import pytest -from annotated_types import Ge, Gt, Interval, Le, Lt, MinLen +from annotated_types import Ge, Gt, Interval, Le, Len, Lt, MaxLen, MinLen, MultipleOf, Predicate from hypothesis import HealthCheck, event, given, settings from hypothesis import strategies as st from jsonschema import Draft202012Validator @@ -170,6 +170,41 @@ def _held(member: dict[str, Any], defs: dict[str, Any] | None = None) -> dict[st _holding(Annotated[int, Ge(0), Doc("a count")]), _held({"type": "integer", "description": "a count", "minimum": 0}), ), + ( + "a-step", + _holding(Annotated[int, Gt(0), MultipleOf(4)]), + _held({"type": "integer", "exclusiveMinimum": 0, "multipleOf": 4}), + ), + ( + "non-empty-string", + _holding(Annotated[str, MinLen(1)]), + _held({"type": "string", "minLength": 1}), + ), + ( + "non-empty-array", + _holding(Annotated[tuple[int, ...], MinLen(1)]), + _held({"type": "array", "items": {"type": "integer"}, "minItems": 1}), + ), + ( + "length-range", + _holding(Annotated[tuple[int, ...], Len(1, 3)]), + _held({"type": "array", "items": {"type": "integer"}, "minItems": 1, "maxItems": 3}), + ), + # A length bound and the length of a fixed-length array set the same + # keyword, and the stricter is kept. + ( + "a-length-on-a-pair", + _holding(Annotated[tuple[int, str], MinLen(1), MaxLen(3)]), + _held( + { + "type": "array", + "prefixItems": [{"type": "integer"}, {"type": "string"}], + "items": False, + "minItems": 2, + "maxItems": 3, + } + ), + ), # Metadata that is not a `Doc` adds nothing to the schema. ("note", _holding(Annotated[str, "a note"]), _held({"type": "string"})), # A value must meet its type's bounds and its `NewType`'s bounds. When @@ -334,7 +369,7 @@ def test_error_a_schema_that_fails_to_write_leaves_nothing_behind() -> None: def test_json_schema_refuses_metadata_check_does_not_hold_a_value_to() -> None: with pytest.raises(TypeError, match="is not a constraint the checker reads"): - json_schema(_holding(Annotated[tuple[int, ...], MinLen(1)])) + json_schema(_holding(Annotated[int, Predicate(bool)])) _PROPERTY = settings(max_examples=300, deadline=None, suppress_health_check=[HealthCheck.too_slow]) @@ -950,6 +985,7 @@ def test_every_schema_is_a_json_schema_written_the_same_each_time(context: ZarrV (Le(9), Le(5), {"type": "integer", "maximum": 5}), (Ge(5), Ge(9), {"type": "integer", "minimum": 9}), (Ge(9), Ge(5), {"type": "integer", "minimum": 9}), + (MultipleOf(4), MultipleOf(6), {"type": "integer", "multipleOf": 12}), ], ids=[ "gt-looser-inside", @@ -960,6 +996,7 @@ def test_every_schema_is_a_json_schema_written_the_same_each_time(context: ZarrV "le-stricter-inside", "ge", "ge-stricter-inside", + "two-steps-are-their-least-common-multiple", ], ) def test_the_stricter_of_two_bounds_of_one_keyword_is_kept( diff --git a/packages/zarr-metadata/tests/test_problem_data.py b/packages/zarr-metadata/tests/test_problem_data.py index 91d9d01412..34d2d79799 100644 --- a/packages/zarr-metadata/tests/test_problem_data.py +++ b/packages/zarr-metadata/tests/test_problem_data.py @@ -315,7 +315,8 @@ def test_a_problem_with_what_a_name_carries_is_the_declaration_s() -> None: # `r24` carries `{"bits": 24}`, which this scope's raw-bits definition # refuses. The problem is located at the declaration, because the name is what # the document wrote there. The bound on `bits` is not a bound on the - # name, so the problem's `ctx` is empty. + # name, so the problem's `ctx` is empty, and its message names the + # member and the name it was read from. context = CORE_AND_EXTENSIONS_CONTEXT.copy( update={ "data_types": { @@ -326,7 +327,7 @@ def test_a_problem_with_what_a_name_carries_is_the_declaration_s() -> None: ) _, problems = resolve("r24", DataTypeDefinition, context, ("data_type",)) assert [(p.loc, p.input, dict(p.ctx)) for p in problems] == [(("data_type",), "r24", {})] - assert problems[0].message == "expected an integer <= 16, got 24" + assert problems[0].message == 'bits in "r24": expected an integer <= 16, got 24' class Ordered(TypedDict, closed=True): diff --git a/packages/zarr-metadata/tests/test_typed_json.py b/packages/zarr-metadata/tests/test_typed_json.py index f399e4cf30..7684d07287 100644 --- a/packages/zarr-metadata/tests/test_typed_json.py +++ b/packages/zarr-metadata/tests/test_typed_json.py @@ -14,6 +14,7 @@ import json import math import pkgutil +import re import sys import textwrap import types @@ -41,6 +42,7 @@ Le, Len, Lt, + MaxLen, MinLen, MultipleOf, Predicate, @@ -1012,6 +1014,10 @@ class Noted(TypedDict, extra_items=Annotated[object, "any JSON"]): level: int +class _Item(TypedDict, closed=True): + a: int + + @pytest.mark.parametrize( ("annotation", "value", "found"), [ @@ -1042,6 +1048,137 @@ class Noted(TypedDict, extra_items=Annotated[object, "any JSON"]): (Digit | str, 10, [((), "expected an integer in [0, 9], got 10", {"ge": 0, "le": 9})]), (Digit, "nine", [((), 'expected an integer, got "nine"', {})]), (Annotated[int, "a note", Doc("another")], -1, []), + (Annotated[int, MultipleOf(4)], 8, []), + ( + Annotated[int, MultipleOf(4)], + 6, + [((), "expected an integer that is a multiple of 4, got 6", {"multiple_of": 4})], + ), + ( + Annotated[int, Gt(0), MultipleOf(4)], + 0, + [ + ( + (), + "expected an integer > 0 that is a multiple of 4, got 0", + {"gt": 0, "multiple_of": 4}, + ) + ], + ), + ( + Annotated[int, Interval(ge=0, le=9), MultipleOf(2)], + 3, + [ + ( + (), + "expected an integer in [0, 9] that is a multiple of 2, got 3", + {"ge": 0, "le": 9, "multiple_of": 2}, + ) + ], + ), + (Annotated[tuple[int, ...], MinLen(1)], [0], []), + ( + Annotated[tuple[int, ...], MinLen(1)], + [], + [ + ( + (), + "expected an array of integers with at least 1 element, got []", + {"min_length": 1}, + ) + ], + ), + ( + Annotated[tuple[int, ...], MaxLen(2)], + [0, 0, 0], + [ + ( + (), + "expected an array of integers with at most 2 elements, got [0, 0, 0]", + {"max_length": 2}, + ) + ], + ), + ( + Annotated[tuple[int, ...], Len(1, 3)], + [], + [ + ( + (), + "expected an array of integers with 1 to 3 elements, got []", + {"min_length": 1, "max_length": 3}, + ) + ], + ), + ( + Annotated[tuple[int, ...], Len(2, 2)], + [0], + [ + ( + (), + "expected an array of integers with 2 elements, got [0]", + {"min_length": 2, "max_length": 2}, + ) + ], + ), + ( + Annotated[tuple[int, int], MinLen(1)], + [0, 0], + [], + ), + ( + Annotated[tuple[Digit, ...], MinLen(1)], + [10], + [((0,), "expected an integer in [0, 9], got 10", {"ge": 0, "le": 9})], + ), + ( + Digit | Annotated[tuple[Digit, ...], MinLen(1)], + [], + [ + ( + (), + "expected an array of integers with at least 1 element, got []", + {"min_length": 1}, + ) + ], + ), + (Annotated[str, MinLen(1)], "a", []), + ( + Annotated[str, MinLen(1)], + "", + [((), 'expected a string with at least 1 character, got ""', {"min_length": 1})], + ), + ( + Annotated[str, Len(2, 2)], + "abc", + [ + ( + (), + 'expected a string with 2 characters, got "abc"', + {"min_length": 2, "max_length": 2}, + ) + ], + ), + # A key the element's closed TypedDict does not declare is left out + # and the value is still returned, so its length is held too. + ( + Annotated[tuple[_Item, ...], MaxLen(1)], + [{"a": 1, "zzz": 0}, {"a": 2}], + [ + ((0, "zzz"), "unexpected key 'zzz'", {}), + ( + (), + ( + "expected an array of objects with at most 1 element, got " + '[{"a": 1, "zzz": 0}, {"a": 2}]' + ), + {"max_length": 1}, + ), + ], + ), + # One code point outside the Basic Multilingual Plane is one + # character, as JSON Schema's `maxLength` counts it. + (Annotated[str, MaxLen(1)], "\U0001d11e", []), ], ids=[ "within", @@ -1059,15 +1196,33 @@ class Noted(TypedDict, extra_items=Annotated[object, "any JSON"]): "the-branch-it-bounds", "a-value-of-another-type-is-that-problem-alone", "notes", + "a-multiple", + "not-a-multiple", + "a-multiple-out-of-bounds", + "bounded-and-not-a-multiple", + "a-length-within", + "a-length-below", + "a-length-above", + "a-length-range", + "an-exact-length", + "a-length-on-a-fixed-length-array", + "a-length-checked-only-once-the-elements-are-read", + "the-branch-it-bounds-the-length-of", + "a-string-length-within", + "a-string-length-below", + "an-exact-string-length", + "a-length-held-beside-an-unknown-key", + "a-string-length-counts-code-points", ], ) def test_a_value_is_held_to_the_bounds_its_type_carries( annotation: object, value: object, found: list[tuple[Loc, str, dict[str, JSONValue]]] ) -> None: - # annotated-types bounds, read as pydantic reads them. A value outside - # any bound gives one problem that says what the type allows, with the - # bounds in the problem's `ctx`. Bounds are checked only for a value of - # the type. A bound is compared as the number it equals, whichever of + # annotated-types bounds, read as pydantic reads them: on a number's + # value, on what an integer is a multiple of, or on an array's length. A value outside any bound gives one + # problem that says what the type allows, with the bounds in the + # problem's `ctx`, named as pydantic names them. Bounds are checked only + # for a value of the type. A bound is compared as the number it equals, whichever of # the equal values Python's `Annotated` cache returns. _, problems = _read(annotation, value) assert [(problem.loc, problem.message, dict(problem.ctx)) for problem in problems] == found @@ -1101,6 +1256,33 @@ def test_error_a_bound_on_what_is_not_a_number() -> None: parser(Annotated[str, Ge(0)], no_leaf) +@pytest.mark.parametrize( + ("annotation", "named"), + [ + ( + Annotated[bool, MinLen(1)], + "min_length: a bound is on an array or a string, and a boolean is not one", + ), + ( + Annotated[Mapping[str, int], MaxLen(1)], + "max_length: a bound is on an array or a string, and an object is not one", + ), + ( + Annotated[int, Len(1, 2)], + "min_length, max_length: a bound is on an array or a string, and an integer is not one", + ), + ], + ids=["min-len-on-a-boolean", "max-len-on-an-object", "len-on-a-number"], +) +def test_error_a_length_bound_on_what_is_not_an_array_or_a_string( + annotation: object, named: str +) -> None: + # pydantic also bounds the number of an object's keys; the checker does + # not, so the type cannot promise more than it checks. + with pytest.raises(TypeError, match=re.escape(named)): + parser(annotation, no_leaf) + + def test_error_a_bound_on_the_values_an_open_typeddict_takes() -> None: class Loose(TypedDict, extra_items=Annotated[object, Ge(0)]): level: int @@ -1117,9 +1299,6 @@ class Later(BaseMetadata): @pytest.mark.parametrize( "constraint", [ - MinLen(1), - Len(1, 3), - MultipleOf(2), Predicate(str.isdigit), Timezone(None), Later(), @@ -1127,9 +1306,6 @@ class Later(BaseMetadata): pydantic.StringConstraints(pattern="^[a-z]+$"), ], ids=[ - "min-len", - "len", - "multiple-of", "predicate", "timezone", "one-a-later-release-adds", @@ -1145,7 +1321,7 @@ def test_error_a_constraint_the_checker_does_not_read(constraint: object) -> Non class _ConstrainedMember(TypedDict, closed=True): - a: Annotated[int, MultipleOf(2)] + a: Annotated[int, Later()] class _HoldsAConstrainedMember(TypedDict, closed=True): @@ -1174,16 +1350,16 @@ class _HoldsAnUnresolvableTypedDict(TypedDict, closed=True): class _ConstraintOnAReadableMember(TypedDict, closed=True): - inner: Annotated[_Readable, MultipleOf(2)] + inner: Annotated[_Readable, Later()] class _ConstraintOnAConstrainedMember(TypedDict, closed=True): - inner: Annotated[_ConstrainedMember, MultipleOf(2)] + inner: Annotated[_ConstrainedMember, Later()] class _SelfReferentialWithAConstraint(TypedDict, closed=True): parent: NotRequired[_SelfReferentialWithAConstraint] - a: Annotated[int, MultipleOf(2)] + a: Annotated[int, Later()] class _SelfReferentialNotAJSONShape(TypedDict, closed=True): @@ -1192,7 +1368,7 @@ class _SelfReferentialNotAJSONShape(TypedDict, closed=True): class _SelfReferentialConstraintOnItself(TypedDict, closed=True): - parent: NotRequired[Annotated[_SelfReferentialConstraintOnItself, MultipleOf(2)]] + parent: NotRequired[Annotated[_SelfReferentialConstraintOnItself, Later()]] a: int @@ -1207,7 +1383,7 @@ class _CycleBack(TypedDict, closed=True): class _OuterWithAConstraintAfterACycle(TypedDict, closed=True): inner: _InnerLeadingBackToAConstraint - a: Annotated[int, MultipleOf(2)] + a: Annotated[int, Later()] class _InnerLeadingBackToAConstraint(TypedDict, closed=True): @@ -1224,7 +1400,7 @@ class _InnerLeadingBackToAShape(TypedDict, closed=True): class _OuterWithAConstraintOnTheCycle(TypedDict, closed=True): - inner: Annotated[_InnerLeadingBackToTheConstraint, MultipleOf(2)] + inner: Annotated[_InnerLeadingBackToTheConstraint, Later()] a: int @@ -1234,19 +1410,17 @@ class _InnerLeadingBackToTheConstraint(TypedDict, closed=True): class _SelfReferentialTwiceWithAConstraintOnTheSecond(TypedDict, closed=True): plain: NotRequired[_SelfReferentialTwiceWithAConstraintOnTheSecond] - constrained: NotRequired[ - Annotated[_SelfReferentialTwiceWithAConstraintOnTheSecond, MultipleOf(2)] - ] + constrained: NotRequired[Annotated[_SelfReferentialTwiceWithAConstraintOnTheSecond, Later()]] class _DocumentedThenConstrainedBackReferences(TypedDict, closed=True): documented: NotRequired[Annotated[_DocumentedThenConstrainedBackReferences, Doc("parent")]] - constrained: NotRequired[Annotated[_DocumentedThenConstrainedBackReferences, MultipleOf(2)]] + constrained: NotRequired[Annotated[_DocumentedThenConstrainedBackReferences, Later()]] class _OuterWithAConstraintAfterAnOptionalCycle(TypedDict, closed=True): inner: _InnerLeadingBackOptionally - a: Annotated[int, MultipleOf(2)] + a: Annotated[int, Later()] class _InnerLeadingBackOptionally(TypedDict, closed=True): @@ -1262,7 +1436,7 @@ class _HoldsATupleOfAConstrainedMember(TypedDict, closed=True): class _ConstraintOnANotAJSONShapeMember(TypedDict, closed=True): - inner: Annotated[_NotAJSONShape, MultipleOf(2)] + inner: Annotated[_NotAJSONShape, Later()] _Children = TypeAliasType("_Children", "tuple[_SelfReferentialThroughAnAlias, ...]") @@ -1313,16 +1487,16 @@ class _SelfReferentialReadable(TypedDict, closed=True): ( _ConstrainedMember, Unread( - "_ConstrainedMember.a: MultipleOf(multiple_of=2) is not a constraint the checker " - "reads, which are Gt, Ge, Lt and Le", + "_ConstrainedMember.a: Later() is not a constraint the checker " + "reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), ( _HoldsAConstrainedMember, Unread( - "_HoldsAConstrainedMember.inner: _ConstrainedMember.a: MultipleOf(multiple_of=2) is " - "not a constraint the checker reads, which are Gt, Ge, Lt and Le", + "_HoldsAConstrainedMember.inner: _ConstrainedMember.a: Later() is " + "not a constraint the checker reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), @@ -1354,8 +1528,8 @@ class _SelfReferentialReadable(TypedDict, closed=True): ( _ConstraintOnAReadableMember, Unread( - "_ConstraintOnAReadableMember.inner: MultipleOf(multiple_of=2) is not a " - "constraint the checker reads, which are Gt, Ge, Lt and Le", + "_ConstraintOnAReadableMember.inner: Later() is not a " + "constraint the checker reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), @@ -1363,16 +1537,16 @@ class _SelfReferentialReadable(TypedDict, closed=True): _ConstraintOnAConstrainedMember, Unread( "_ConstraintOnAConstrainedMember.inner: _ConstrainedMember.a: " - "MultipleOf(multiple_of=2) is not a constraint the checker reads, which are Gt, " - "Ge, Lt and Le", + "Later() is not a constraint the checker reads, which are Gt, " + "Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), ( _SelfReferentialWithAConstraint, Unread( - "_SelfReferentialWithAConstraint.a: MultipleOf(multiple_of=2) is not a constraint " - "the checker reads, which are Gt, Ge, Lt and Le", + "_SelfReferentialWithAConstraint.a: Later() is not a constraint " + "the checker reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), @@ -1387,8 +1561,8 @@ class _SelfReferentialReadable(TypedDict, closed=True): ( _SelfReferentialConstraintOnItself, Unread( - "_SelfReferentialConstraintOnItself.parent: MultipleOf(multiple_of=2) is not a " - "constraint the checker reads, which are Gt, Ge, Lt and Le", + "_SelfReferentialConstraintOnItself.parent: Later() is not a " + "constraint the checker reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), @@ -1403,8 +1577,8 @@ class _SelfReferentialReadable(TypedDict, closed=True): ( _OuterWithAConstraintAfterACycle, Unread( - "_OuterWithAConstraintAfterACycle.a: MultipleOf(multiple_of=2) is not a " - "constraint the checker reads, which are Gt, Ge, Lt and Le", + "_OuterWithAConstraintAfterACycle.a: Later() is not a " + "constraint the checker reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), @@ -1419,8 +1593,8 @@ class _SelfReferentialReadable(TypedDict, closed=True): ( _OuterWithAConstraintOnTheCycle, Unread( - "_OuterWithAConstraintOnTheCycle.inner: MultipleOf(multiple_of=2) is not a " - "constraint the checker reads, which are Gt, Ge, Lt and Le", + "_OuterWithAConstraintOnTheCycle.inner: Later() is not a " + "constraint the checker reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), @@ -1428,24 +1602,24 @@ class _SelfReferentialReadable(TypedDict, closed=True): _SelfReferentialTwiceWithAConstraintOnTheSecond, Unread( "_SelfReferentialTwiceWithAConstraintOnTheSecond.constrained: " - "MultipleOf(multiple_of=2) is not a constraint the checker reads, which are Gt, " - "Ge, Lt and Le", + "Later() is not a constraint the checker reads, which are Gt, " + "Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), ( _DocumentedThenConstrainedBackReferences, Unread( - "_DocumentedThenConstrainedBackReferences.constrained: MultipleOf(multiple_of=2) " - "is not a constraint the checker reads, which are Gt, Ge, Lt and Le", + "_DocumentedThenConstrainedBackReferences.constrained: Later() " + "is not a constraint the checker reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), ( _OuterWithAConstraintAfterAnOptionalCycle, Unread( - "_OuterWithAConstraintAfterAnOptionalCycle.a: MultipleOf(multiple_of=2) is not a " - "constraint the checker reads, which are Gt, Ge, Lt and Le", + "_OuterWithAConstraintAfterAnOptionalCycle.a: Later() is not a " + "constraint the checker reads, which are Gt, Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), @@ -1462,8 +1636,8 @@ class _SelfReferentialReadable(TypedDict, closed=True): _HoldsATupleOfAConstrainedMember, Unread( "_HoldsATupleOfAConstrainedMember.inner: _ConstrainedMember.a: " - "MultipleOf(multiple_of=2) is not a constraint the checker reads, which are Gt, " - "Ge, Lt and Le", + "Later() is not a constraint the checker reads, which are Gt, " + "Ge, Lt, Le, MultipleOf, MinLen and MaxLen", constraint=True, ), ), @@ -1562,12 +1736,14 @@ def test_unread_in_names_the_part_without_a_parser_and_whether_it_is_a_constrain Annotated[Digit, Ge(1)], Annotated[int, Ge(0), Gt(0)], Annotated[int, Interval(le=9), Lt(5)], + Annotated[tuple[int, ...], Len(1, 3), MinLen(2)], + Annotated[int, MultipleOf(2), MultipleOf(3)], ], - ids=["narrowing-an-alias", "gt-and-ge", "interval-and-lt"], + ids=["narrowing-an-alias", "gt-and-ge", "interval-and-lt", "len-and-min-len", "two-steps"], ) def test_error_a_second_bound_from_one_side(annotation: object) -> None: # Pydantic uses the last bound given; write only one. - with pytest.raises(TypeError, match="is a second bound from (below|above)"): + with pytest.raises(TypeError, match="is a second (bound from below|bound from above|step)"): parser(annotation, no_leaf) @@ -1579,6 +1755,27 @@ def test_error_a_bound_that_is_not_a_finite_number(bound: object) -> None: parser(Annotated[int, Ge(bound)], no_leaf) # pyright: ignore[reportArgumentType] +def test_error_a_step_on_what_is_not_an_integer() -> None: + # pydantic also takes a step on a float; JSON Schema validators compare + # floats inexactly, so the checker takes one on an integer only. + with pytest.raises( + TypeError, match="multiple_of: a step is on an integer, and a number is not one" + ): + parser(Annotated[float, MultipleOf(2)], no_leaf) + + +@pytest.mark.parametrize("step", [0, -2, 1.5], ids=["zero", "negative", "fractional"]) +def test_error_a_step_that_is_not_a_positive_integer(step: object) -> None: + with pytest.raises(TypeError, match="a step is a positive integer"): + parser(Annotated[int, MultipleOf(step)], no_leaf) # pyright: ignore[reportArgumentType] + + +@pytest.mark.parametrize("length", [-1, 1.5], ids=["negative", "fractional"]) +def test_error_a_length_bound_that_is_not_a_non_negative_integer(length: object) -> None: + with pytest.raises(TypeError, match="a length bound is a non-negative integer"): + parser(Annotated[tuple[int, ...], MinLen(length)], no_leaf) # pyright: ignore[reportArgumentType] + + # --- check: the public door ------------------------------------------------ _V3_DOCUMENT = dict(ZarrV3ArrayMetadata.create_default(shape=(4,)).to_json()) @@ -1671,6 +1868,16 @@ def test_error_check_refuses_a_value_of_another_type() -> None: assert [(problem.loc, problem.kind) for problem in problems] == [(("level",), "invalid_type")] +def test_error_check_refuses_an_array_document_without_codecs() -> None: + # The document's type says `codecs` is never empty, so `check` refuses + # an empty list at `codecs`, as the reader and the JSON Schema do. + typed, problems = check({**_V3_DOCUMENT, "codecs": []}, ZarrV3ArrayMetadataJSON) + assert typed is None + assert [(problem.loc, problem.kind, dict(problem.ctx)) for problem in problems] == [ + (("codecs",), "invalid_value", {"min_length": 1}) + ] + + _NOT_DOCUMENTS: Final = frozenset({"ZarrV3ContextSlots", "ZarrV2ContextSlots"}) """TypedDicts that describe no document: the keyword arguments of a scope class, which hold definitions, not JSON.""" diff --git a/packages/zarr-metadata/tests/testing/_shares.py b/packages/zarr-metadata/tests/testing/_shares.py index 27517e2653..3d70bf11aa 100644 --- a/packages/zarr-metadata/tests/testing/_shares.py +++ b/packages/zarr-metadata/tests/testing/_shares.py @@ -2,9 +2,15 @@ A reach test asks whether a target can be drawn at all, in as many examples as it takes, which says nothing of whether a test, at the 100 examples it -runs by default, ever meets it. Runs are derandomized, which keeps most shares -fixed, but a share can still move a few points between processes, so a bound -needs room: this is no place to bound what is drawn a few percent of the time. +runs by default, ever meets it. Runs are derandomized, so a share is a fixed +number for a given strategy. A change to the strategy draws other examples, +as another seed would, so a bound sits where the share stays across seeds: +this is no place to bound what is drawn a few percent of the time. + +A bound is set from the shares of 100 seeds, each run as `examples_of` +runs, but with `derandomize=False` under `hypothesis.seed`. It lies past +the most extreme share on its side by a quarter of that share's distance +from the median, rounded outward to a hundredth. """ from __future__ import annotations @@ -14,24 +20,58 @@ from hypothesis import HealthCheck, given, settings if TYPE_CHECKING: - from collections.abc import Callable + from collections.abc import Callable, Sequence from hypothesis.strategies import SearchStrategy T = TypeVar("T") +_DERANDOMIZED = settings( + database=None, + derandomize=True, + deadline=None, + suppress_health_check=[HealthCheck.filter_too_much, HealthCheck.too_slow], +) + + +def examples_of(strategy: SearchStrategy[T], *, examples: int = 300) -> list[T]: + """The examples a derandomized run of `strategy` draws, checked to be the same in a second run. + + Hypothesis's mutator copies spans between the parts of an example that + share a label, and a label can follow an object's address: a + `sampled_from` of elements that hold a strategy built for the draw is + labeled by that strategy's `id`. Such a strategy draws other examples + from run to run, and from machine to machine, however derandomized. A + second run in the same process lays objects out at other addresses, so + it catches this. + """ + first = _draw(strategy, examples) + second = _draw(strategy, examples) + if [repr(value) for value in first] != [repr(value) for value in second]: + raise AssertionError( + f"{strategy!r} drew other examples on a second derandomized run. " + "A strategy built during a draw is likely in a `sampled_from`, which " + "labels it by its address; sample an index into the options instead." + ) + return first + + +def share(values: Sequence[T], counts: Callable[[T], bool]) -> float: + """The share of `values` that `counts` holds of.""" + return sum(counts(value) for value in values) / len(values) + def share_of( strategy: SearchStrategy[T], counts: Callable[[T], bool], *, examples: int = 300 ) -> float: """The share of the examples of `strategy` that `counts` holds of.""" - tally: list[bool] = [] - run = settings( - max_examples=examples, - database=None, - derandomize=True, - deadline=None, - suppress_health_check=[HealthCheck.filter_too_much, HealthCheck.too_slow], - )(given(strategy)(lambda value: tally.append(counts(value)))) - run() - return sum(tally) / len(tally) + return share(examples_of(strategy, examples=examples), counts) + + +def _draw(strategy: SearchStrategy[T], examples: int) -> list[T]: + drawn: list[T] = [] + # A lambda, not `drawn.append`: Hypothesis cannot wrap a builtin method, + # and derandomizes from the test function's source, so this line is the seed. + run = given(strategy)(lambda value: drawn.append(value)) + settings(_DERANDOMIZED, max_examples=examples)(run)() + return drawn diff --git a/packages/zarr-metadata/tests/testing/test_documents.py b/packages/zarr-metadata/tests/testing/test_documents.py index 6c20a3c29f..53cd047724 100644 --- a/packages/zarr-metadata/tests/testing/test_documents.py +++ b/packages/zarr-metadata/tests/testing/test_documents.py @@ -8,6 +8,7 @@ from __future__ import annotations +import functools import json from typing import TYPE_CHECKING, Any, cast @@ -15,7 +16,7 @@ from hypothesis import find, given, settings from hypothesis import strategies as st -from tests.testing._shares import share_of +from tests.testing._shares import examples_of, share from zarr_metadata.definition import AcceptedDeclaration, Definition, resolve from zarr_metadata.definition._definition import canonicalize from zarr_metadata.model import ( @@ -340,19 +341,28 @@ def test_a_data_type_no_codec_takes_is_not_drawn(document: Mapping[str, Any]) -> _DOCUMENT_SHARES: dict[str, tuple[Callable[[Mapping[str, Any]], bool], float, float]] = { - "no axes": (lambda d: d["shape"] == (), 0.03, 0.25), - "more axes than five": (lambda d: len(d["shape"]) > 5, 0.1, 1), - "a zero-length axis": (lambda d: 0 in d["shape"], 0.1, 1), - "an axis past 32 bits": (lambda d: any(n >= 2**32 for n in d["shape"]), 0.1, 1), - "the first data type": (lambda d: _name(d["data_type"]) == "bool", 0, 0.12), + "no axes": (lambda d: d["shape"] == (), 0.04, 0.34), + "more axes than five": (lambda d: len(d["shape"]) > 5, 0.15, 1), + "a zero-length axis": (lambda d: 0 in d["shape"], 0.44, 1), + "an axis past 32 bits": (lambda d: any(n >= 2**32 for n in d["shape"]), 0.03, 1), + "the first data type": (lambda d: _name(d["data_type"]) == "bool", 0, 0.18), } -"""What a share of array documents holds, as a low and a high bound: what a test running only a few hundred examples meets.""" +"""What a share of array documents holds, as a low and a high bound: what a test running only a few hundred examples meets. + +Each bound is set as `_shares` says, from 100 seeds. +""" + + +@functools.cache +def _array_document_examples() -> list[Mapping[str, Any]]: + """Drawn once for every target, as each counts the same examples.""" + return examples_of(array_documents()) @pytest.mark.parametrize("target", _DOCUMENT_SHARES) def test_array_documents_share(target: str) -> None: counts, low, high = _DOCUMENT_SHARES[target] - assert low <= share_of(array_documents(), counts) <= high + assert low <= share(_array_document_examples(), counts) <= high def _empty_listings(document: Mapping[str, Any]) -> list[Any]: diff --git a/packages/zarr-metadata/tests/testing/test_strategies.py b/packages/zarr-metadata/tests/testing/test_strategies.py index 8ef6023265..6a7a66035e 100644 --- a/packages/zarr-metadata/tests/testing/test_strategies.py +++ b/packages/zarr-metadata/tests/testing/test_strategies.py @@ -670,7 +670,6 @@ def _is_pair(entry: object) -> bool: _RECTILINEAR_SPELLINGS: dict[str, tuple[tuple[int, ...] | None, Callable[[object], bool]]] = { "a bare length": (None, lambda dimension: isinstance(dimension, int)), - "no edges": (None, lambda dimension: dimension == ()), "only written-out lengths": ( None, lambda dimension: ( @@ -741,6 +740,18 @@ def test_rectilinear_dimensions_reach(spelling: str) -> None: ) +@given(data=st.data(), shape=st.none() | st.lists(st.integers(0, 2), max_size=3).map(tuple)) +def test_rectilinear_dimensions_are_never_empty( + data: st.DataObject, shape: tuple[int, ...] | None +) -> None: + """A list of edges has at least one, even along a dimension of length 0 or of a length not known.""" + configuration = data.draw(chunk_grid_configurations(RECTILINEAR_CHUNK_GRID, shape)) + # The type declares each list non-empty, so the definition's read of + # the configuration, which checks the type, is what would refuse one. + assert () not in configuration["chunk_shapes"] + assert RECTILINEAR_CHUNK_GRID.read_configuration(configuration)[1] == () + + @given(data=st.data(), shape=st.lists(st.integers(0, 2**63 - 1), max_size=3).map(tuple)) def test_chunk_grids_fit_a_long_shape(data: st.DataObject, shape: tuple[int, ...]) -> None: """However long an axis is, a grid covers it without a list as long as the axis.""" @@ -1057,7 +1068,7 @@ def _float_class(value: object) -> str | None: ) -@pytest.mark.parametrize(("kind", "at_least"), [("normal", 0.2), ("NaN", 0.1), ("infinity", 0.01)]) +@pytest.mark.parametrize(("kind", "at_least"), [("normal", 0.2), ("NaN", 0.26), ("infinity", 0.01)]) def test_hex_fill_values_are_not_only_small_numbers(kind: str, at_least: float) -> None: """Bits drawn as an integer alone are nearly all subnormal.""" assert share_of(_HEX_FILL_VALUES, lambda value: _float_class(value) == kind) >= at_least @@ -1097,7 +1108,7 @@ def _shard_inner_lengths(draw: st.DrawFn) -> tuple[int, ...]: def test_shards_of_a_rectilinear_grid_are_often_divided() -> None: """Edges drawn apart rarely share a factor, which leaves a shard only inner chunks of length 1.""" - assert share_of(_shard_inner_lengths(), lambda inner: any(n > 1 for n in inner)) >= 0.4 + assert share_of(_shard_inner_lengths(), lambda inner: any(n > 1 for n in inner)) >= 0.39 def _edge_factor(configuration: Mapping[str, Any]) -> int: @@ -1145,7 +1156,7 @@ def test_a_long_dimension_keeps_its_edge_factor() -> None: edges = chunk_grid_configurations(RECTILINEAR_CHUNK_GRID, (2**16 + 1,)).filter( lambda configuration: not isinstance(configuration["chunk_shapes"][0], int) ) - assert share_of(edges, lambda configuration: _edge_factor(configuration) > 1) >= 0.3 + assert share_of(edges, lambda configuration: _edge_factor(configuration) > 1) >= 0.32 def test_divisors_of_a_very_long_shard_are_found_quickly() -> None: @@ -1168,4 +1179,4 @@ def _field_types(declaration: Any) -> list[str]: # noqa: ANN401 def test_struct_fields_do_not_favor_the_first_data_type() -> None: """A type drawn after a field's name would be `bool` a fifth of the time.""" - assert share_of(declarations(STRUCT_DATA_TYPE), lambda d: _field_types(d)[0] == "bool") <= 0.12 + assert share_of(declarations(STRUCT_DATA_TYPE), lambda d: _field_types(d)[0] == "bool") <= 0.16 diff --git a/packages/zarr-metadata/tests/v3/test_every_definition.py b/packages/zarr-metadata/tests/v3/test_every_definition.py index 3090024c53..c72161d85c 100644 --- a/packages/zarr-metadata/tests/v3/test_every_definition.py +++ b/packages/zarr-metadata/tests/v3/test_every_definition.py @@ -12,6 +12,7 @@ from typing import Any import pytest +from typing_extensions import TypedDict from zarr_metadata import ValidationProblem from zarr_metadata._json import value_at @@ -26,7 +27,7 @@ ) from zarr_metadata.definition._definition import canonicalize from zarr_metadata.v3.codec.gzip import GZIP_CODEC -from zarr_metadata.v3.data_type.raw import RAW_BYTES_DATA_TYPE, RawBytesConfiguration +from zarr_metadata.v3.data_type.raw import RAW_BYTES_DATA_TYPE from zarr_metadata.v3.definition import ( CORE_AND_EXTENSIONS_CONTEXT, CORE_CONTEXT, @@ -385,11 +386,17 @@ def test_raw_bits_read_as_r_star_with_the_size_their_name_carries( ) +class AnyRawSize(TypedDict, closed=True): + """A raw-bits configuration of any size, which the package's own type does not allow.""" + + bits: int + + def test_a_reader_reads_raw_bits_its_own_way_by_defining_r_star() -> None: # This definition accepts any size. Filed under `r*` in place of the # package's own, it reads every raw-bits name. Filed back, the package's # definition reads them again. - mine = DataTypeDefinition(name="r*", configuration=RawBytesConfiguration) + mine = DataTypeDefinition(name="r*", configuration=AnyRawSize) context = CORE_AND_EXTENSIONS_CONTEXT.copy( update={"data_types": {**CORE_AND_EXTENSIONS_CONTEXT.data_types, "r*": mine}} ) @@ -900,12 +907,21 @@ def test_error_struct_field_type_is_judged_where_it_sits() -> None: ] -@pytest.mark.parametrize("name", ["r12", "r0"]) -def test_error_a_raw_bytes_width_is_not_a_positive_multiple_of_8(name: str) -> None: +@pytest.mark.parametrize( + ("name", "message"), + [ + ("r12", 'bits in "r12": expected an integer > 0 that is a multiple of 8, got 12'), + ("r0", 'bits in "r0": expected an integer > 0 that is a multiple of 8, got 0'), + ], +) +def test_error_a_raw_bytes_width_is_not_a_positive_multiple_of_8(name: str, message: str) -> None: # Raw bits of a size the spec does not allow are reported, not left # unchecked. The document wrote a name, so the problem is located at - # the declaration. - assert _problems("data_type:r*", name) == [((), "invalid_value")] + # the declaration, and its message names the size the name spells. + _, problems = resolve(name, DataTypeDefinition, ZARR_PYTHON_CONTEXT) + assert [(found.loc, found.kind, found.message) for found in problems] == [ + ((), "invalid_value", message) + ] def test_error_a_configuration_written_beside_a_raw_bits_name() -> None: diff --git a/packages/zarr-metadata/tests/v3/test_grid_shapes.py b/packages/zarr-metadata/tests/v3/test_grid_shapes.py index 40887f37bb..d56dc4d366 100644 --- a/packages/zarr-metadata/tests/v3/test_grid_shapes.py +++ b/packages/zarr-metadata/tests/v3/test_grid_shapes.py @@ -64,8 +64,9 @@ def _problems(grid: JSONValue, shape: tuple[int, ...]) -> list[tuple[tuple[str | (_rectilinear([4, 4, 4]), (10,), ({4},)), (_rectilinear([4, 4, 2, 7]), (10,), ({4, 2, 7},)), (_rectilinear([[4, 2], 2]), (10,), ({4, 2},)), - # A dimension of length 0 and no chunks. - (_rectilinear([]), (0,), (set(),)), + # A dimension of length 0: a bare integer gives it no chunks, and a + # list still lists at least one. + (_rectilinear(4, [3]), (0, 0), ({4}, {3})), (_rectilinear(2, [3, 3]), (7, 6), ({2}, {3})), # A grid that nothing in scope claims, or one that is not read, is # not checked, and its lengths are unknown along each axis. @@ -95,13 +96,22 @@ def test_error_a_rectilinear_grid_of_another_rank(grid: JSONValue, shape: tuple[ assert _problems(grid, shape) == [(("configuration", "chunk_shapes"), "invalid_value")] -@pytest.mark.parametrize("spec", [[4, 4], [[4, 2]], []]) +@pytest.mark.parametrize("spec", [[4, 4], [[4, 2]]]) def test_error_rectilinear_chunks_that_do_not_cover_their_dimension(spec: JSONValue) -> None: assert _problems(_rectilinear(2, spec), (4, 10)) == [ (("configuration", "chunk_shapes", 1), "invalid_value") ] +def test_error_rectilinear_empty_edge_list() -> None: + # The spec requires a non-empty list even along a dimension of length 0, + # which writes a bare integer to have no chunks; no shape is needed to tell. + _, found = resolve(_rectilinear(2, []), ChunkGridDefinition, CORE_AND_EXTENSIONS_CONTEXT) + assert [(problem.loc, problem.kind) for problem in found] == [ + (("configuration", "chunk_shapes", 1), "invalid_value") + ] + + def test_a_grid_that_says_nothing_of_the_shape_fits_every_one_its_lengths_unknown() -> None: lenient = ChunkGridDefinition(name="acme.grid", configuration=EmptyConfiguration) context = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(chunk_grids=[lenient])) diff --git a/packages/zarr-metadata/tests/v3/test_sharding.py b/packages/zarr-metadata/tests/v3/test_sharding.py index dc3e81e69b..ae6336a979 100644 --- a/packages/zarr-metadata/tests/v3/test_sharding.py +++ b/packages/zarr-metadata/tests/v3/test_sharding.py @@ -206,7 +206,6 @@ def test_error_an_inner_chunk_length_that_does_not_divide_the_shard( ("shard", "found"), [ # Each pipeline is read as the array's pipeline is, at its own location. - (_shard([4, 4], codecs=[]), [((0, "configuration", "codecs"), "invalid_value")]), ( _shard([4, 4], codecs=["crc32c", LITTLE]), [((0, "configuration", "codecs", 1), "invalid_value")], @@ -233,6 +232,20 @@ def test_error_a_pipeline_a_shard_holds_that_does_not_fit_what_it_is_handed( assert _problems([shard], _chunk({8}, {8})) == found +@pytest.mark.parametrize( + ("shard", "key"), + [(_shard([4, 4], codecs=[]), "codecs"), (_shard([4, 4], index=[]), "index_codecs")], + ids=["codecs", "index-codecs"], +) +def test_error_a_shard_s_empty_pipeline_is_refused_by_its_type(shard: JSONValue, key: str) -> None: + # Each pipeline's type declares it non-empty, so the type check reports + # an empty one, before the pipeline's order check would. + problems = _read([shard], _chunk({8}, {8}))[1] + assert [(p.loc, p.kind, dict(p.ctx)) for p in problems] == [ + ((0, "configuration", key), "invalid_value", {"min_length": 1}) + ] + + def test_error_an_array_document_s_shard_is_read_where_it_sits() -> None: document = dict(ZarrV3ArrayMetadata.create_default(shape=(16, 16)).to_json()) document["chunk_grid"] = {"name": "regular", "configuration": {"chunk_shape": [8, 6]}} From 1ab86a0129949eac855c377439881dc420af1912 Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Sat, 14 Nov 2026 23:03:44 +0100 Subject: [PATCH 2/7] fix(zarr-metadata): draw the same codecs on every derandomized run (#4635) * fix(zarr-metadata): draw the same codecs on every derandomized run `_codec` drew from `st.sampled_from` over `(definition, configurations)` pairs whose configurations are strategies built during the draw. Hypothesis labels such a `sampled_from` by hashing its elements, and a strategy hashes by address, so these spans were labeled by memory layout. Hypothesis's mutator copies choices between spans that share a label, so which spans it grouped, and so which examples a derandomized run drew, changed with whatever moved the allocator: another process, machine load. `_codec` now samples an index into the options, labeled by a `range`. The share tests measured this drift. `examples_of` now draws twice and fails if the runs differ, which catches an address-labeled strategy in the same process; the array-document targets share one draw. Bounds are reset from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the seed minimum (0.06). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * docs(zarr-metadata): add changelog fragment for #4635 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 * fix(zarr-metadata): sample codec names and make share bounds reproducible `_codec` samples definition names rather than an index. An index into options of the same length shares one label with every other, so the mutator could copy a choice between unrelated codec slots; names key the label on the option set, as the chunk grid and data type draws already do. `examples_of` is cached per strategy, so the share tests that count one strategy draw it once, and its docstring calls the second-run check what it is, a tripwire. `across_seeds` and `bounds` put the rule the bounds follow into code, so a bound can be reset rather than re-derived; every bound is reset by it. The two first-data-type checks run over 1000 examples: over 300, `bool`'s share across seeds reached 0.13 to 0.15, too near the 0.2 they guard against. The changelog fragment no longer says the options' strategies are built during the draw: some are cached. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- packages/zarr-metadata/changes/4635.bugfix.md | 2 +- .../src/zarr_metadata/testing/_draw.py | 16 +-- .../zarr-metadata/tests/testing/_shares.py | 99 +++++++++++++------ .../tests/testing/test_documents.py | 33 ++++--- .../tests/testing/test_shares.py | 26 +++++ .../tests/testing/test_strategies.py | 19 +++- 6 files changed, 137 insertions(+), 58 deletions(-) create mode 100644 packages/zarr-metadata/tests/testing/test_shares.py diff --git a/packages/zarr-metadata/changes/4635.bugfix.md b/packages/zarr-metadata/changes/4635.bugfix.md index d92c9b5193..791809568b 100644 --- a/packages/zarr-metadata/changes/4635.bugfix.md +++ b/packages/zarr-metadata/changes/4635.bugfix.md @@ -1 +1 @@ -Strategies in `zarr_metadata.testing.strategies` that draw codec pipelines, such as `array_documents` and `pipelines`, now draw the same examples on every derandomized Hypothesis run. A codec was sampled from options that hold strategies built during the draw, which Hypothesis labels by memory address, so the examples its mutator produced varied from one process to the next. +Strategies in `zarr_metadata.testing.strategies` that draw codec pipelines, such as `array_documents` and `pipelines`, now draw the same examples on every derandomized Hypothesis run. A codec was sampled from options that hold strategies, which Hypothesis labels by their memory address, so the examples its mutator produced varied from one process to the next. diff --git a/packages/zarr-metadata/src/zarr_metadata/testing/_draw.py b/packages/zarr-metadata/src/zarr_metadata/testing/_draw.py index e63d4afe1c..2c42d4f1b3 100644 --- a/packages/zarr-metadata/src/zarr_metadata/testing/_draw.py +++ b/packages/zarr-metadata/src/zarr_metadata/testing/_draw.py @@ -278,12 +278,16 @@ def pipelines_for( def _codec( draw: st.DrawFn, options: Sequence[tuple[K, Configurations]] ) -> tuple[K, ZarrV3DeclarationJSON]: - # An index, not the options: Hypothesis labels a `sampled_from` by - # hashing its elements, and each option holds a strategy built for this - # draw, whose hash is its address. Labels that follow addresses send - # Hypothesis's mutator down different paths from one run to the next, so - # a derandomized run would not draw the same examples twice. - definition, configurations = options[draw(st.sampled_from(range(len(options))))] + # A name, not an option: Hypothesis labels a `sampled_from` by hashing its + # elements, and an option holds a strategy, which hashes by its `id`. A + # label that follows where a strategy sits in memory sends Hypothesis's + # mutator down other paths from one run to the next, so a derandomized + # run would not draw the same examples twice. Names also keep one set of + # options apart from another, as an index into either would not. + named = { + definition.name: (definition, configurations) for definition, configurations in options + } + definition, configurations = named[draw(st.sampled_from(list(named)))] return definition, draw(any_spelling(definition, configurations)) diff --git a/packages/zarr-metadata/tests/testing/_shares.py b/packages/zarr-metadata/tests/testing/_shares.py index 3d70bf11aa..b69ee73a80 100644 --- a/packages/zarr-metadata/tests/testing/_shares.py +++ b/packages/zarr-metadata/tests/testing/_shares.py @@ -4,61 +4,55 @@ as it takes, which says nothing of whether a test, at the 100 examples it runs by default, ever meets it. Runs are derandomized, so a share is a fixed number for a given strategy. A change to the strategy draws other examples, -as another seed would, so a bound sits where the share stays across seeds: -this is no place to bound what is drawn a few percent of the time. +as another seed would, and so does a change to the line in `_draw` that +Hypothesis derandomizes from, which reseeds every share at once. So a bound +sits where the share stays across seeds: this is no place to bound what is +drawn a few percent of the time. -A bound is set from the shares of 100 seeds, each run as `examples_of` -runs, but with `derandomize=False` under `hypothesis.seed`. It lies past -the most extreme share on its side by a quarter of that share's distance -from the median, rounded outward to a hundredth. +A bound is reset with `bounds(across_seeds(strategy, statistic))`, which +runs 100 seeds and applies the rule `bounds` states. """ from __future__ import annotations -from typing import TYPE_CHECKING, TypeVar +import functools +import math +from typing import TYPE_CHECKING, Any, TypeVar, cast -from hypothesis import HealthCheck, given, settings +from hypothesis import HealthCheck, given, seed, settings if TYPE_CHECKING: - from collections.abc import Callable, Sequence + from collections.abc import Callable, Iterable, Sequence from hypothesis.strategies import SearchStrategy T = TypeVar("T") -_DERANDOMIZED = settings( +_SETTINGS = settings( database=None, - derandomize=True, deadline=None, suppress_health_check=[HealthCheck.filter_too_much, HealthCheck.too_slow], ) -def examples_of(strategy: SearchStrategy[T], *, examples: int = 300) -> list[T]: - """The examples a derandomized run of `strategy` draws, checked to be the same in a second run. +def examples_of(strategy: SearchStrategy[T], *, examples: int = 300) -> tuple[T, ...]: + """The examples a derandomized run of `strategy` draws, drawn once per strategy and checked against a second run. Hypothesis's mutator copies spans between the parts of an example that share a label, and a label can follow an object's address: a - `sampled_from` of elements that hold a strategy built for the draw is - labeled by that strategy's `id`. Such a strategy draws other examples - from run to run, and from machine to machine, however derandomized. A - second run in the same process lays objects out at other addresses, so - it catches this. + `sampled_from` of elements that hold a strategy is labeled by that + strategy's `id`. Such a strategy draws other examples from run to run, + and from machine to machine, however derandomized. A second run in the + same process often lays objects out at other addresses, so it often + catches this, but it is a tripwire, not a proof. """ - first = _draw(strategy, examples) - second = _draw(strategy, examples) - if [repr(value) for value in first] != [repr(value) for value in second]: - raise AssertionError( - f"{strategy!r} drew other examples on a second derandomized run. " - "A strategy built during a draw is likely in a `sampled_from`, which " - "labels it by its address; sample an index into the options instead." - ) - return first + return cast("tuple[T, ...]", _checked(strategy, examples)) -def share(values: Sequence[T], counts: Callable[[T], bool]) -> float: +def share(values: Iterable[T], counts: Callable[[T], bool]) -> float: """The share of `values` that `counts` holds of.""" - return sum(counts(value) for value in values) / len(values) + tally = [counts(value) for value in values] + return sum(tally) / len(tally) def share_of( @@ -68,10 +62,51 @@ def share_of( return share(examples_of(strategy, examples=examples), counts) -def _draw(strategy: SearchStrategy[T], examples: int) -> list[T]: +def across_seeds( + strategy: SearchStrategy[T], + statistic: Callable[[Sequence[T]], float], + *, + seeds: Iterable[int] = range(100), + examples: int = 300, +) -> list[float]: + """`statistic` of the examples of `strategy` under each of `seeds`, as a change to the strategy would move it.""" + return [statistic(_draw(strategy, examples, chosen)) for chosen in seeds] + + +def bounds(statistics: Sequence[float]) -> tuple[float, float]: + """A low and a high bound for a statistic measured across seeds. + + Each lies past the most extreme measure on its side by a quarter of that + measure's distance from the median, rounded outward to a hundredth. + """ + ordered = sorted(statistics) + median = ordered[len(ordered) // 2] + low = ordered[0] - (median - ordered[0]) / 4 + high = ordered[-1] + (ordered[-1] - median) / 4 + return math.floor(round(low * 100, 6)) / 100, math.ceil(round(high * 100, 6)) / 100 + + +@functools.cache +def _checked(strategy: SearchStrategy[Any], examples: int) -> tuple[Any, ...]: + first = _draw(strategy, examples) + second = _draw(strategy, examples) + if [repr(value) for value in first] != [repr(value) for value in second]: + raise AssertionError( + f"{strategy!r} drew other examples on a second derandomized run. " + "A strategy is likely in a `sampled_from`, which labels it by its " + "address; sample a name or an index instead." + ) + return first + + +def _draw(strategy: SearchStrategy[T], examples: int, chosen: int | None = None) -> tuple[T, ...]: + """Derandomized examples of `strategy`, or seeded with `chosen`.""" drawn: list[T] = [] # A lambda, not `drawn.append`: Hypothesis cannot wrap a builtin method, # and derandomizes from the test function's source, so this line is the seed. run = given(strategy)(lambda value: drawn.append(value)) - settings(_DERANDOMIZED, max_examples=examples)(run)() - return drawn + if chosen is None: + settings(_SETTINGS, max_examples=examples, derandomize=True)(run)() + else: + settings(_SETTINGS, max_examples=examples, derandomize=False)(seed(chosen)(run))() + return tuple(drawn) diff --git a/packages/zarr-metadata/tests/testing/test_documents.py b/packages/zarr-metadata/tests/testing/test_documents.py index 53cd047724..41bd8eda65 100644 --- a/packages/zarr-metadata/tests/testing/test_documents.py +++ b/packages/zarr-metadata/tests/testing/test_documents.py @@ -8,7 +8,6 @@ from __future__ import annotations -import functools import json from typing import TYPE_CHECKING, Any, cast @@ -16,7 +15,7 @@ from hypothesis import find, given, settings from hypothesis import strategies as st -from tests.testing._shares import examples_of, share +from tests.testing._shares import share_of from zarr_metadata.definition import AcceptedDeclaration, Definition, resolve from zarr_metadata.definition._definition import canonicalize from zarr_metadata.model import ( @@ -340,29 +339,35 @@ def test_a_data_type_no_codec_takes_is_not_drawn(document: Mapping[str, Any]) -> assert document["data_type"] == "int8" +_ARRAY_DOCUMENTS = array_documents() +"""One strategy for every share, so `examples_of` draws its examples once.""" + _DOCUMENT_SHARES: dict[str, tuple[Callable[[Mapping[str, Any]], bool], float, float]] = { - "no axes": (lambda d: d["shape"] == (), 0.04, 0.34), - "more axes than five": (lambda d: len(d["shape"]) > 5, 0.15, 1), + "no axes": (lambda d: d["shape"] == (), 0.07, 0.31), + "more axes than five": (lambda d: len(d["shape"]) > 5, 0.16, 1), "a zero-length axis": (lambda d: 0 in d["shape"], 0.44, 1), - "an axis past 32 bits": (lambda d: any(n >= 2**32 for n in d["shape"]), 0.03, 1), - "the first data type": (lambda d: _name(d["data_type"]) == "bool", 0, 0.18), + "an axis past 32 bits": (lambda d: any(n >= 2**32 for n in d["shape"]), 0.02, 1), } """What a share of array documents holds, as a low and a high bound: what a test running only a few hundred examples meets. -Each bound is set as `_shares` says, from 100 seeds. +Each bound is set as `_shares` says. """ -@functools.cache -def _array_document_examples() -> list[Mapping[str, Any]]: - """Drawn once for every target, as each counts the same examples.""" - return examples_of(array_documents()) - - @pytest.mark.parametrize("target", _DOCUMENT_SHARES) def test_array_documents_share(target: str) -> None: counts, low, high = _DOCUMENT_SHARES[target] - assert low <= share(_array_document_examples(), counts) <= high + assert low <= share_of(_ARRAY_DOCUMENTS, counts) <= high + + +def test_array_documents_do_not_favor_the_first_data_type() -> None: + """`bool`, first of the data types by name, is drawn no more often than the rest would allow. + + Over 1000 examples: over 300, its share moves too much across seeds to + tell a fair draw from one that draws it a fifth of the time. + """ + share = share_of(_ARRAY_DOCUMENTS, lambda d: _name(d["data_type"]) == "bool", examples=1000) + assert share <= 0.13 def _empty_listings(document: Mapping[str, Any]) -> list[Any]: diff --git a/packages/zarr-metadata/tests/testing/test_shares.py b/packages/zarr-metadata/tests/testing/test_shares.py new file mode 100644 index 0000000000..d49c20143c --- /dev/null +++ b/packages/zarr-metadata/tests/testing/test_shares.py @@ -0,0 +1,26 @@ +"""The helpers that measure and bound how often a strategy draws something.""" + +from __future__ import annotations + +import itertools + +import pytest +from hypothesis import strategies as st + +from tests.testing._shares import bounds, examples_of + +_COUNTER = itertools.count() + + +def test_bounds() -> None: + """Past each extreme by a quarter of its distance from the median, rounded outward.""" + assert bounds([0.1, 0.2, 0.3]) == (0.07, 0.33) + assert bounds([0.3, 0.1, 0.2]) == (0.07, 0.33) + assert bounds([0.5, 0.5, 0.5]) == (0.5, 0.5) + assert bounds([0.0, 0.4, 0.4, 1.0]) == (-0.1, 1.15) + + +def test_a_strategy_that_draws_otherwise_on_a_second_run_is_an_assertion_error() -> None: + drifting = st.builds(lambda: next(_COUNTER)) + with pytest.raises(AssertionError, match="second derandomized run"): + examples_of(drifting, examples=5) diff --git a/packages/zarr-metadata/tests/testing/test_strategies.py b/packages/zarr-metadata/tests/testing/test_strategies.py index 6a7a66035e..97dd2075ee 100644 --- a/packages/zarr-metadata/tests/testing/test_strategies.py +++ b/packages/zarr-metadata/tests/testing/test_strategies.py @@ -1068,7 +1068,9 @@ def _float_class(value: object) -> str | None: ) -@pytest.mark.parametrize(("kind", "at_least"), [("normal", 0.2), ("NaN", 0.26), ("infinity", 0.01)]) +@pytest.mark.parametrize( + ("kind", "at_least"), [("normal", 0.22), ("NaN", 0.25), ("infinity", 0.01)] +) def test_hex_fill_values_are_not_only_small_numbers(kind: str, at_least: float) -> None: """Bits drawn as an integer alone are nearly all subnormal.""" assert share_of(_HEX_FILL_VALUES, lambda value: _float_class(value) == kind) >= at_least @@ -1108,7 +1110,7 @@ def _shard_inner_lengths(draw: st.DrawFn) -> tuple[int, ...]: def test_shards_of_a_rectilinear_grid_are_often_divided() -> None: """Edges drawn apart rarely share a factor, which leaves a shard only inner chunks of length 1.""" - assert share_of(_shard_inner_lengths(), lambda inner: any(n > 1 for n in inner)) >= 0.39 + assert share_of(_shard_inner_lengths(), lambda inner: any(n > 1 for n in inner)) >= 0.36 def _edge_factor(configuration: Mapping[str, Any]) -> int: @@ -1156,7 +1158,7 @@ def test_a_long_dimension_keeps_its_edge_factor() -> None: edges = chunk_grid_configurations(RECTILINEAR_CHUNK_GRID, (2**16 + 1,)).filter( lambda configuration: not isinstance(configuration["chunk_shapes"][0], int) ) - assert share_of(edges, lambda configuration: _edge_factor(configuration) > 1) >= 0.32 + assert share_of(edges, lambda configuration: _edge_factor(configuration) > 1) >= 0.36 def test_divisors_of_a_very_long_shard_are_found_quickly() -> None: @@ -1178,5 +1180,12 @@ def _field_types(declaration: Any) -> list[str]: # noqa: ANN401 def test_struct_fields_do_not_favor_the_first_data_type() -> None: - """A type drawn after a field's name would be `bool` a fifth of the time.""" - assert share_of(declarations(STRUCT_DATA_TYPE), lambda d: _field_types(d)[0] == "bool") <= 0.16 + """A type drawn after a field's name would be `bool` a fifth of the time. + + Over 1000 examples: over 300, the share of `bool` moves too much across + seeds to tell a fair draw from one that favors it. + """ + first_is_bool = share_of( + declarations(STRUCT_DATA_TYPE), lambda d: _field_types(d)[0] == "bool", examples=1000 + ) + assert first_is_bool <= 0.13 From 883b15bdc748014ddaa98b65347a6138bd3d5fa8 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 14 Nov 2026 23:04:29 +0100 Subject: [PATCH 3/7] chore(deps): bump tornado (#4631) Bumps the uv group with 1 update in the /packages/zarr-http-server directory: [tornado](https://github.com/tornadoweb/tornado). Updates `tornado` from 6.6 to 6.6.1 - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.6.0...v6.6.1) --- updated-dependencies: - dependency-name: tornado dependency-version: 6.6.1 dependency-type: indirect dependency-group: uv ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- packages/zarr-http-server/uv.lock | 92 +++++++++++++++---------------- 1 file changed, 46 insertions(+), 46 deletions(-) diff --git a/packages/zarr-http-server/uv.lock b/packages/zarr-http-server/uv.lock index 9f6ed790da..dc1355ff5f 100644 --- a/packages/zarr-http-server/uv.lock +++ b/packages/zarr-http-server/uv.lock @@ -1985,52 +1985,52 @@ wheels = [ [[package]] name = "tornado" -version = "6.6" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/d0/02/5f549b1ad5da6267ed1e1fac2eef69257b1979cc812497002499b7895ff9/tornado-6.6.tar.gz", hash = "sha256:34c4aa916d7f2b2d5cadee2c6850f5a4a7f3d688cbf58399ea9bd3057af4abd5", size = 617998, upload-time = "2026-10-21T15:31:04.733Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/df/f5/d3e6f76b945f9812842db742d0708a3b6287e850e81be3ca10e290c103c6/tornado-6.6-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:01abf15429d64f04ccfafc238b191cc2c177108b7a331f4461707ad77198c8c2", size = 528326, upload-time = "2026-10-21T15:30:05.528Z" }, - { url = "https://files.pythonhosted.org/packages/30/69/b20cf3a8bf641ec792a5a6050aa153e8f120604f9c23471142de535113e5/tornado-6.6-cp311-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:11bb97c00981eb8254e55bd6a5ad960c983ae343fef8ae3ae5584cd1fb70236a", size = 529544, upload-time = "2026-10-21T15:30:07.126Z" }, - { url = "https://files.pythonhosted.org/packages/0f/f4/d9d8d3ac345aa9df3a132ad03bd443160c2ab24ad4003b4e8d6d5af06371/tornado-6.6-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e152a3b2d3eae113b836d979823f7bb2320f7e334a84449729964921f374a4c", size = 530523, upload-time = "2026-10-21T15:30:08.478Z" }, - { url = "https://files.pythonhosted.org/packages/ad/82/1296ade1c14dee5a4906e5ca092997622ece7c4dc768c68ed9caaf895d1b/tornado-6.6-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e912d4c4babd04db6f34b25b748174a4a176debe86c8610beedf1815f9af7fdd", size = 530739, upload-time = "2026-10-21T15:30:09.98Z" }, - { url = "https://files.pythonhosted.org/packages/66/30/6ac1eab66c81af46dafd93c9b24bd311f82e199d9d7b8ce2a64a91b6a200/tornado-6.6-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:33aabe8a03d426cef4e0fe5669f0643cadd983b54957f1a7ac2789ec326e9644", size = 529738, upload-time = "2026-10-21T15:30:11.498Z" }, - { url = "https://files.pythonhosted.org/packages/24/b4/6c1f6aa1fd148e4d0f500a10cd0c9949e7bf385e161ff497c81de5b3ae4f/tornado-6.6-cp311-abi3-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9aaf0e1c2fb9f046f30e2036b8d2060666399e8464d0a72769e43c754089567b", size = 527908, upload-time = "2026-10-21T15:30:13.298Z" }, - { url = "https://files.pythonhosted.org/packages/54/5b/67d3d452b1179bc9e265cebd381a56ab4cf2d7b2a54607aedd8d6e1ebc2e/tornado-6.6-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:61fc5513cea97720ab9bf54b04827ac0f4500c63f0b072b963d6145c798b4661", size = 530367, upload-time = "2026-10-21T15:30:15.183Z" }, - { url = "https://files.pythonhosted.org/packages/74/14/4e8e9a14f77bcafb2408634b2d4c8c676ad7a43bf8dac4ee2d48370698f9/tornado-6.6-cp311-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:31e1a0d5cd704d2fe8a008c1cc27a0e2b67269ca263630a27985dddc160b5537", size = 530456, upload-time = "2026-10-21T15:30:16.741Z" }, - { url = "https://files.pythonhosted.org/packages/56/12/e3ddef99a2089ccfc86f32bd3f39caaad87bee4868be1bacbf213a0b07a2/tornado-6.6-cp311-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:74416e41e9bdf70fbc2fb242f6fbbc7a58ec11e59d39f15fc5a421ecb6845118", size = 529610, upload-time = "2026-10-21T15:30:18.031Z" }, - { url = "https://files.pythonhosted.org/packages/35/3d/62a692343d46b37e852e5078e0f6513534d925f70a04580072e3c127b129/tornado-6.6-cp311-abi3-musllinux_1_2_s390x.whl", hash = "sha256:8cdd69104ae9d705b3f04d2563aa96e357043e267a455802daae9a1e633a37c8", size = 529803, upload-time = "2026-10-21T15:30:19.326Z" }, - { url = "https://files.pythonhosted.org/packages/49/04/6c59a9413adfe4810f648a4dd9fdf476f13b0495f9cc25ef856b71f9ce1e/tornado-6.6-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:fc7c8a3263501fbe97518cd699101bc3c5f499a7dd4f75fc6cbb3bd9ee1db1f1", size = 529751, upload-time = "2026-10-21T15:30:20.826Z" }, - { url = "https://files.pythonhosted.org/packages/31/36/a1b3b84cd347ae43f982d3bf95628de6c6365668dba2543af47c6a1c888e/tornado-6.6-cp311-abi3-win32.whl", hash = "sha256:b54ca69ff93d0374efb083a12a1e9ec2dd2036e8496d197e0e8ef77b8d9135c3", size = 530643, upload-time = "2026-10-21T15:30:22.428Z" }, - { url = "https://files.pythonhosted.org/packages/b7/50/dc813fe35a672e14ee8bed8b5c634eeb343a05c98eb455318a169de24567/tornado-6.6-cp311-abi3-win_amd64.whl", hash = "sha256:48f26ed9d7beb64e5f06adc864524e581ab6d62ba8ff46eaaa1ac2d0e6d753fd", size = 530886, upload-time = "2026-10-21T15:30:23.975Z" }, - { url = "https://files.pythonhosted.org/packages/0d/71/62a9b9e980026d38874740d0a7c162f53eb0563f6a441e3aff84c48db7e8/tornado-6.6-cp311-abi3-win_arm64.whl", hash = "sha256:29114ffd4e4de869d390393df9e9a0f191065aae1a96f38a58d35e464c04a867", size = 530899, upload-time = "2026-10-21T15:30:25.264Z" }, - { url = "https://files.pythonhosted.org/packages/ac/4c/633e3568eb574df555d6ddd2dac44b7793a5c26cd63b444e653fcfb1d6ec/tornado-6.6-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:63fc8d243b52fd8dd9ed4ad02d0df08f2024f9aea632a433a8a6c91075076d44", size = 528432, upload-time = "2026-10-21T15:30:26.578Z" }, - { url = "https://files.pythonhosted.org/packages/37/eb/b525d1870ad28a56f47b664dd8fd3fab4f7ddd237c87c14e97248bf4762f/tornado-6.6-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:8d4b85cac795bfc6a7f20ba81871abb8c2ae8fba019061b8a9ae1c2f97e29637", size = 532586, upload-time = "2026-10-21T15:30:28.027Z" }, - { url = "https://files.pythonhosted.org/packages/b7/35/909a33c253352346ca081a27ca66b45c1398468adf784b767597ca773c28/tornado-6.6-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e035c065edadfa696529d05eebcbeb25154c129b5aee6dbfb697a6abf5d4f697", size = 533641, upload-time = "2026-10-21T15:30:29.318Z" }, - { url = "https://files.pythonhosted.org/packages/6f/64/100202666d6e3d96288ab184827762415456c139798a15c962547f9fabf9/tornado-6.6-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5b94e589ab855090c448b5c0937b3a75f1c1a604b7b36207e8236d209f720b9c", size = 533869, upload-time = "2026-10-21T15:30:30.882Z" }, - { url = "https://files.pythonhosted.org/packages/b1/e0/af3ed62bc15830d60328d5b4a1a9076b35ea0ba142cb17b1c1772cb15c59/tornado-6.6-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1a394c8c9db9137f2a2b42869d9f7b91398fcf19da120d0dde4c4546172710f7", size = 532767, upload-time = "2026-10-21T15:30:32.273Z" }, - { url = "https://files.pythonhosted.org/packages/24/d7/1e2e1a6932eb62d63324b64dba2f62e6eaf79ad883dd209352c297b7848c/tornado-6.6-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0215d1df6e4b5e736ddad5adf0af4138cf7437fdd39c77c5e69768e37850a632", size = 530956, upload-time = "2026-10-21T15:30:33.746Z" }, - { url = "https://files.pythonhosted.org/packages/70/65/3b9a0e53d7a4c39f2ebe95486540fa8eeb67bfb6e8c437d603c00d35b128/tornado-6.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:4d711790e0a9576ad924d942c4607e4e401d697588ea6f30f6662824008a7add", size = 533372, upload-time = "2026-10-21T15:30:35.085Z" }, - { url = "https://files.pythonhosted.org/packages/44/48/90979cbfaa3ff941324a2605eb794af0f087c670cd99f18a69ab40498073/tornado-6.6-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:034a66bae0f55421d007803e2385d2a59707d8829b2f026373d42f08256e9922", size = 533425, upload-time = "2026-10-21T15:30:36.425Z" }, - { url = "https://files.pythonhosted.org/packages/bb/c4/ce743a86d91f85c255b77a4282eb5b2716b61e74430eb9d1df8578a539d2/tornado-6.6-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:5cb38483b077129056a94dd415a71d29ab4e42b46da894fee8b4657003f10b21", size = 532545, upload-time = "2026-10-21T15:30:37.789Z" }, - { url = "https://files.pythonhosted.org/packages/02/5e/a82131e35778540fc19139229d02aa8cf2453a8022e202be50c88f0f8ac3/tornado-6.6-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:46bbbd107efa37777ac4b83601866d19b4dbd5b770a7a9123582633d95959ad3", size = 532700, upload-time = "2026-10-21T15:30:39.336Z" }, - { url = "https://files.pythonhosted.org/packages/3a/09/6dac8c2c677c65d60b3d7dae94176dbfe392f314e7c8e2d709623edd6a26/tornado-6.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b85e9d0263cdbc027b95947c8b873a7bc7fd6a08d16bfb4f5047350b0808e464", size = 532712, upload-time = "2026-10-21T15:30:40.698Z" }, - { url = "https://files.pythonhosted.org/packages/b2/ac/71e5776cbcdc3264e81aa6db28c69717ba50607a9eb8d5b766f11a4b020e/tornado-6.6-cp314-cp314t-win32.whl", hash = "sha256:4276eef0880e5fe4488e6799c2842f59af3b06637e9313ab4fb506f9c9c2d6cb", size = 530491, upload-time = "2026-10-21T15:30:42.115Z" }, - { url = "https://files.pythonhosted.org/packages/d8/0f/0b6a6908a5b5aa8320009dfdb84434b1cf160b577a00456e59ab9e6ea598/tornado-6.6-cp314-cp314t-win_amd64.whl", hash = "sha256:10d4af98c6df09f8429b1340ccaea3a72887b86f695dfb8056bda517c8115a00", size = 530780, upload-time = "2026-10-21T15:30:43.414Z" }, - { url = "https://files.pythonhosted.org/packages/3c/39/c152e1cc5371f75f5f9f0a4f44bf2dc9396be9407d384de7428446fb4651/tornado-6.6-cp314-cp314t-win_arm64.whl", hash = "sha256:b1f0ef20a5278854f5fdac31637bafadb055405df28b7fc945e54add12b172e3", size = 530781, upload-time = "2026-10-21T15:30:44.741Z" }, - { url = "https://files.pythonhosted.org/packages/0c/c5/f96c7d7a8895745bb37dfb6e6107e1ee40ad02a4b162d7d6c143644455ab/tornado-6.6-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:8f8e382da30fe042f27f27a871e21aefc2613870e5a41648de18446c72d6ea59", size = 528446, upload-time = "2026-10-21T15:30:46.112Z" }, - { url = "https://files.pythonhosted.org/packages/c0/dd/ab8f6500d274b9c55b025798eb54e92591bcff27515799bec443bd54e35a/tornado-6.6-cp315-cp315t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:8c7a8a328396c2195454a2ecd9499512441bbce99505544a717e6a87d16ff6f9", size = 532774, upload-time = "2026-10-21T15:30:47.393Z" }, - { url = "https://files.pythonhosted.org/packages/87/23/b78e7d1bf329160b8157284005395421ea941864c16c7f04c3e48c98bcf9/tornado-6.6-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ca025bb0da81c294aef26f27427316df1525220820315f4bf32dc1e84a4d2232", size = 533872, upload-time = "2026-10-21T15:30:48.884Z" }, - { url = "https://files.pythonhosted.org/packages/e0/4f/ab66b614ad0bc9a219484be6bc2c002c8de9916a7b6c7b27fe6273838e5c/tornado-6.6-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:59c22ebc905ac2a267f8d9d8a878f86afbcbbde329d75c485f1422fc45a81b83", size = 534027, upload-time = "2026-10-21T15:30:50.157Z" }, - { url = "https://files.pythonhosted.org/packages/da/63/501fa15a36bace2f4b9158228cf922d606403261bf71c960c433ec3ca0a4/tornado-6.6-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f89802bc0bcd15f988d9325b4e6d2171bb04ea68d188732556335e10d3fd00bd", size = 532936, upload-time = "2026-10-21T15:30:51.45Z" }, - { url = "https://files.pythonhosted.org/packages/7f/b7/82d643f635f0300bda98613ca2660b4f3134b6fb594c6fddfa4de706806c/tornado-6.6-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:70043256d5958f9c655066153d23d60ea3e29bf9854f7fb75850e223c477758a", size = 531254, upload-time = "2026-10-21T15:30:52.865Z" }, - { url = "https://files.pythonhosted.org/packages/d4/53/92c40891a94750bc09257310af964bf25c59e3531d6199736c318baa864f/tornado-6.6-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:82b49495e3849d644b841ef33b35491160055997deaa93720ff647ef019421db", size = 533566, upload-time = "2026-10-21T15:30:54.15Z" }, - { url = "https://files.pythonhosted.org/packages/7e/88/b82f5c5efde731b88d839ad99cb6ace2a521f03bb6d7346d2de089c76da8/tornado-6.6-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:07b2bcaa4bf447bdbfb664e4c28344226540db3f0ebccd74886528338a240765", size = 533567, upload-time = "2026-10-21T15:30:55.451Z" }, - { url = "https://files.pythonhosted.org/packages/02/2c/e9a7f5d0fca0bdbcd48b2d9c7884dd0b32669c9945d98bc6d303a80ecfea/tornado-6.6-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:c36b921374c302e54909bae17d9f354061d80e7ae7d8c94f73abbcc217866460", size = 532849, upload-time = "2026-10-21T15:30:56.729Z" }, - { url = "https://files.pythonhosted.org/packages/d0/f5/4ade737340ca6ce9c677ea0c2843591c8ad065d07f0913c28d922c7e078e/tornado-6.6-cp315-cp315t-musllinux_1_2_s390x.whl", hash = "sha256:7412d9e37aa9f7048e4b591f520a95f7a097278ffab15f1ddf3f5c2320cf6648", size = 532873, upload-time = "2026-10-21T15:30:58.067Z" }, - { url = "https://files.pythonhosted.org/packages/8e/04/6acdc937d0bb6189ddb34ceded517d7bd1c7b3bded5d4e1b3442160e6b6f/tornado-6.6-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:15bc95112b2d24b0bfaee9a004e3f974da929e33b4244a7f695a22f45b08a5e5", size = 532883, upload-time = "2026-10-21T15:30:59.395Z" }, - { url = "https://files.pythonhosted.org/packages/ab/27/0a766fac6b1cc629d96346329a0485287fd2857c51a887a3a154f2a69938/tornado-6.6-cp315-cp315t-win32.whl", hash = "sha256:f661c21117b875d612371f3eb243d0c69f56d9bdd6aee38b7c24453419dfe554", size = 530488, upload-time = "2026-10-21T15:31:00.77Z" }, - { url = "https://files.pythonhosted.org/packages/5c/80/3e4142570883da27b3f60ae82c638ac96e60997eac684740613557519c96/tornado-6.6-cp315-cp315t-win_amd64.whl", hash = "sha256:ee8853ab35da510dc952879ebd9995c82bc10ffb7925fd87930dc0d6916f4b39", size = 530777, upload-time = "2026-10-21T15:31:02.087Z" }, - { url = "https://files.pythonhosted.org/packages/17/57/4a85c617b6349962320b4afbe1386d5b05fd57218ee8ad6e82e0110ec9ee/tornado-6.6-cp315-cp315t-win_arm64.whl", hash = "sha256:7de62e2dfdf559fd7154d2b9aa58ff3d9cff8304ae0519341feb082618bc2a33", size = 530780, upload-time = "2026-10-21T15:31:03.425Z" }, +version = "6.6.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/64/2b/8ff2b41620cbba0eeb0862fed21dcaa5a2ad36c6dd9bb465f9ff7a147be2/tornado-6.6.1.tar.gz", hash = "sha256:28e6311402af59c8b02e3fa0f5e7d8eb465b25081381803b127d97e6a4395d5a", size = 623388, upload-time = "2026-11-09T21:09:22.71Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/77/d3/ef3a4d38d1a72c52ad07483ed760dbb93027d28f19c3a2d98148abdd63e1/tornado-6.6.1-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:a8d3e6acde5a80d13b13b5e4c7c226ca682f31d0a62b6c33cea038714bc2de1b", size = 533249, upload-time = "2026-11-09T21:08:22.475Z" }, + { url = "https://files.pythonhosted.org/packages/38/4d/c65ea949a3cdc185f1afac8c337074d4f2d029b48293d2db4d2c3dea4849/tornado-6.6.1-cp311-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:1b6ff57cc69b339f8feb9e7b567b0e0eacb16bfaf234159496623ef7a67ba640", size = 534470, upload-time = "2026-11-09T21:08:24.27Z" }, + { url = "https://files.pythonhosted.org/packages/03/ff/20911f85dd399b56761496e8138810e8608ea25f0912377f4205d52af853/tornado-6.6.1-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:da6ce2da215810d56539fa6cac77d9e27f4d8139c6868c53436b2fe933f43c79", size = 535446, upload-time = "2026-11-09T21:08:25.625Z" }, + { url = "https://files.pythonhosted.org/packages/be/70/a4a159dd4f1fef39ff8eadae655b133a85722b39087a89e21ce463af9832/tornado-6.6.1-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8cd28b2cacbe63d53e28a38546a3492b488417eeb3f7ecf10954476fc2508882", size = 535664, upload-time = "2026-11-09T21:08:27.191Z" }, + { url = "https://files.pythonhosted.org/packages/ed/d5/db70d5d23d3b4eaea0b26b710f28d2ca387bb44e68ccfede60ba3bbc2096/tornado-6.6.1-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:25b2619274bca77d0eb7b4d96dcf2d1142bb9bf172ad6a7491a1e1f82e6e48fa", size = 534660, upload-time = "2026-11-09T21:08:28.78Z" }, + { url = "https://files.pythonhosted.org/packages/f7/f6/0528ee039266902e707edd9996ee3ce7364dc1baf08bed585d8e6fb3ccd0/tornado-6.6.1-cp311-abi3-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:77dd6e0c659fdc45824e0746988ac5c060c2a9fee9b8cbbcdb999b3f29d3cd10", size = 532834, upload-time = "2026-11-09T21:08:30.432Z" }, + { url = "https://files.pythonhosted.org/packages/a6/bb/b733df2d487a78283080663bd6d96955121b58b6d445d9cb832607da9786/tornado-6.6.1-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:b62fded687e55acaf9e36c1230b006066a35b3f6df093ba1fd3b2ff592070e1d", size = 535289, upload-time = "2026-11-09T21:08:31.692Z" }, + { url = "https://files.pythonhosted.org/packages/a7/7c/c81cb83f2bbb5740fcba1eb83d794b6d6fab2063b16a475cc9039266091e/tornado-6.6.1-cp311-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:ecef88054d8a77878f61061ff67dfc5e1c17d1f7334890ab508b310e8e68ee46", size = 535382, upload-time = "2026-11-09T21:08:33.282Z" }, + { url = "https://files.pythonhosted.org/packages/8a/ae/c499da4cc8be1c1f31a5d8df2328f2aae091de2db13302ad5c20ed704ec4/tornado-6.6.1-cp311-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:ca95e762df1ca761f330e3f2a034c15ac144569844b52527a5a9eca9f524c0f1", size = 534534, upload-time = "2026-11-09T21:08:34.57Z" }, + { url = "https://files.pythonhosted.org/packages/61/7c/6b163c9e58ca9df0c941bda9ba2dcabbd87ecb81c5fa546f54eeac689232/tornado-6.6.1-cp311-abi3-musllinux_1_2_s390x.whl", hash = "sha256:8cae3d8f2585be01670da9fd5010c61724c51b61ecd4c06820e95874739daf7c", size = 534728, upload-time = "2026-11-09T21:08:35.828Z" }, + { url = "https://files.pythonhosted.org/packages/18/11/b5107ac6a278387635d73d283e7ab5a5c76fb8ccf402efa57006ea25852f/tornado-6.6.1-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6596f3e939962f13099ce8d1a96b83226e908ca369b193370a6efa84585a1601", size = 534676, upload-time = "2026-11-09T21:08:37.209Z" }, + { url = "https://files.pythonhosted.org/packages/65/df/0574c4479c565d9233def796afc9179fa840f41f8c240f740609f746da67/tornado-6.6.1-cp311-abi3-win32.whl", hash = "sha256:0f0f596a0dab6826aa21d103c7bc242e3fb7cfc99c5c3d44a35ba8c83abbdca4", size = 535565, upload-time = "2026-11-09T21:08:38.518Z" }, + { url = "https://files.pythonhosted.org/packages/36/57/35439a779582e3b19c98e799b6a41519fb3bdaa3a21b7a94ce4dd7b3b732/tornado-6.6.1-cp311-abi3-win_amd64.whl", hash = "sha256:5cb82ca38596e1350fceaf994f287847f82f7b8436380d36ba0937a03107d0ee", size = 535806, upload-time = "2026-11-09T21:08:39.892Z" }, + { url = "https://files.pythonhosted.org/packages/89/b8/2cb42845af01ea7e70ac23e6f6fe2f26b9b158bce186c6991c057c9abe1d/tornado-6.6.1-cp311-abi3-win_arm64.whl", hash = "sha256:58d2f47e7b54fded3407dbbbe4e59d29ca627f85bf6c0f21f0b151c6c7fc9c53", size = 535825, upload-time = "2026-11-09T21:08:41.339Z" }, + { url = "https://files.pythonhosted.org/packages/10/de/cfeae9e75c144ca22018ee3b858e343c1658d6edd2184f35d810ec1ef789/tornado-6.6.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:ed0bc5a1a9f6346e7bdadc106fa2d4c6e79d71d8b2df4366260ab3baf36a8399", size = 533356, upload-time = "2026-11-09T21:08:42.603Z" }, + { url = "https://files.pythonhosted.org/packages/3d/d8/7812a729c797fb8c15748c9a69faeaaa13630075cf9b76ba9f7fbcfb7f42/tornado-6.6.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:54c80f9bed0bf1533ecf5a90dc59651645622d62b0bcffa2692d108a1f794917", size = 537511, upload-time = "2026-11-09T21:08:43.84Z" }, + { url = "https://files.pythonhosted.org/packages/70/47/ebef304487feea6d8efb6a7d2642964ecb1f5cc785f687628e1d7cac4326/tornado-6.6.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb67ba2cbf598e842f14684eab029a5a4ba7ab587e8c4e40940a77ed0596b31e", size = 538566, upload-time = "2026-11-09T21:08:45.203Z" }, + { url = "https://files.pythonhosted.org/packages/a3/6a/87532e6745c415eb2daa9bc1c9c89c69ce9e8255b1c0bde9938f3299440b/tornado-6.6.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d83bac141bd9360d8e995ec512097be19ead2e9213c3e7088510a036f3192b8d", size = 538793, upload-time = "2026-11-09T21:08:46.478Z" }, + { url = "https://files.pythonhosted.org/packages/e4/c7/26af60f71f7c32f19257bad58711500628f579edf1a2c3d8328ad15aae04/tornado-6.6.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:60aad449dbcebaa8d69bff50c6ba39858fab8d256e779a99bb30ebf6724b9b48", size = 537691, upload-time = "2026-11-09T21:08:47.975Z" }, + { url = "https://files.pythonhosted.org/packages/ba/9d/af233b8bf4311d3466099ab267eb595ec6f1655607d290dc7f706b6dbd6f/tornado-6.6.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b9ce0d6548bd73b1329075b247d3fdc61538922f558ee42059e08f5df53fc197", size = 535877, upload-time = "2026-11-09T21:08:49.76Z" }, + { url = "https://files.pythonhosted.org/packages/95/66/78e7583ce69f66e5177c65988a98373be2ab25c1d22405d274e037acbc58/tornado-6.6.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ce5a18654f5d56d11400ab5d87103660e14deceab41818a06a42acc8dbbe427d", size = 538295, upload-time = "2026-11-09T21:08:51.221Z" }, + { url = "https://files.pythonhosted.org/packages/a3/3a/ad336d4994547cf268a3bd6d10f6ef691e1e888fc776940c0bfe02e8d85c/tornado-6.6.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:3298fce72c35f836eab4c4db8b960104ff1ea718033a6b5e236f04efe3a1b6e1", size = 538349, upload-time = "2026-11-09T21:08:52.761Z" }, + { url = "https://files.pythonhosted.org/packages/cf/cf/563d24516afd266f6b3c3f1c7ac15ad39b8668966d2bccce917c169ab0c0/tornado-6.6.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:0223e827cedf4e35ebb035e57a0d9974563cd043e3a53f8bb72a859b6f53777e", size = 537469, upload-time = "2026-11-09T21:08:54.279Z" }, + { url = "https://files.pythonhosted.org/packages/c8/ea/1705ff4014c8bad5d7045022647156cfebd6c7014886da47e13512d27c08/tornado-6.6.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:533185c89be3ffb01656c9425da4ff42587a3af784b37d39fe762e8fce3b2dbe", size = 537624, upload-time = "2026-11-09T21:08:55.648Z" }, + { url = "https://files.pythonhosted.org/packages/50/5d/421168e62c8b80e3ef88f80d642e9a7e5b1d0c6e8cd136d99064f663951c/tornado-6.6.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:fbaa022a49a27b65668592cf43b898ff3978cbe4e6926fdfe24dbef59053bd26", size = 537632, upload-time = "2026-11-09T21:08:57.797Z" }, + { url = "https://files.pythonhosted.org/packages/6a/b4/13b2f98a3141c152915fa3e69531a04bdc1cd7dd203f442136f3792a486f/tornado-6.6.1-cp314-cp314t-win32.whl", hash = "sha256:4bc6c60faf86162df25ba2ff5611d27b6ef8226de9d61c07a0fab53c169630a5", size = 535424, upload-time = "2026-11-09T21:08:59.145Z" }, + { url = "https://files.pythonhosted.org/packages/84/83/6d54c37b3e437ab0ff2bddd71249c6ea2a17051104dd2995cb4e181e95a7/tornado-6.6.1-cp314-cp314t-win_amd64.whl", hash = "sha256:71c16b382b22d3234913a3f74a1ea84bede197d11eb03871df17bb123088dfa5", size = 535715, upload-time = "2026-11-09T21:09:00.456Z" }, + { url = "https://files.pythonhosted.org/packages/d4/38/75419d8cd577b7c4f335f4c5e75f127af286545ad49047a73afffdf877fe/tornado-6.6.1-cp314-cp314t-win_arm64.whl", hash = "sha256:510c7261ed986ecd6f74edc1a13412759e7abc2e8be3a6c85e15f6c53e1f3c84", size = 535718, upload-time = "2026-11-09T21:09:01.783Z" }, + { url = "https://files.pythonhosted.org/packages/91/8b/96e460653f3e2ad83d70328414394132d73d5655047c692ce7b69ae2f6c0/tornado-6.6.1-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:01dc73e3697eb94d78c42e03931ab621da97d12fdfb6339e64ff5a934b220fcf", size = 533376, upload-time = "2026-11-09T21:09:03.097Z" }, + { url = "https://files.pythonhosted.org/packages/1a/1e/644bace0700e533235f01ffa68a7b5e3d6cc9eb7380e3929039d04b648b9/tornado-6.6.1-cp315-cp315t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95bd4751357bef52aa655e236b9667ff054885de4b118e4ce9ba5c06e12ef5d0", size = 537696, upload-time = "2026-11-09T21:09:04.519Z" }, + { url = "https://files.pythonhosted.org/packages/e6/0a/afccc23d1ed71bcb1ce4a5b6e0f61f82c7a0b275f3b377ee7aee6680a565/tornado-6.6.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:29bce3190c2973fcd2171c95eeab8bf06d2f7a7eb107455ce3e4662c704aecdb", size = 538796, upload-time = "2026-11-09T21:09:05.956Z" }, + { url = "https://files.pythonhosted.org/packages/e5/24/69692f8ba851ed2738c2623eab592360d646f0a2e16b6deea16516aa8595/tornado-6.6.1-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b777fbe0bdb2c4dc764aadf5f70e9b23b2e536c339d61c799d3ea1b575a7c285", size = 538951, upload-time = "2026-11-09T21:09:07.298Z" }, + { url = "https://files.pythonhosted.org/packages/00/82/234306b41b33600a470da7e595b8064b85e2f509f64f5c1d39387c0abbb8/tornado-6.6.1-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:15443805d6b7aa11610cc6c5be7e71b4a93369aeb2ca5a13c2def53a3af5ba45", size = 537861, upload-time = "2026-11-09T21:09:08.857Z" }, + { url = "https://files.pythonhosted.org/packages/5b/87/6d13a60c320c0379cbfa4c2520b4ba076780c33e6a01ba3073fff74b71db/tornado-6.6.1-cp315-cp315t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b77ab23d1520c807d11e703e79b020d3bc3676a0f5ebc8531f46d3fac8aecb2f", size = 536178, upload-time = "2026-11-09T21:09:10.424Z" }, + { url = "https://files.pythonhosted.org/packages/96/c7/e27f2a21e05d7c5aa9886db7368436e6436880b85349072988b4f62422e3/tornado-6.6.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:872249311539281c1e8a5d8d6f27f478bf705a9bd931bf374b6b31b05177dfda", size = 538491, upload-time = "2026-11-09T21:09:11.968Z" }, + { url = "https://files.pythonhosted.org/packages/f3/c7/1f981336039c51d81ee6f895e3412178edc409bd709e38767365c55c0daa/tornado-6.6.1-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:08b808e8d3312c268f9fe1ffebfcfaaf342e7fe4ac40d219a7e1b58bac547d6f", size = 538491, upload-time = "2026-11-09T21:09:13.226Z" }, + { url = "https://files.pythonhosted.org/packages/5d/4f/7dafc65726778ac148b68c91920605892e84bf0ffcc02890aaf1a7d0ba8d/tornado-6.6.1-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:8d475f4053c693b5c1483eb4845824129b0b51c65712e736b3c2ea1c8f6080cd", size = 537775, upload-time = "2026-11-09T21:09:14.697Z" }, + { url = "https://files.pythonhosted.org/packages/a1/25/85c852307f9e49896b6dc5119f6a0131f5f6d97c478a687654c2c0c3f0f4/tornado-6.6.1-cp315-cp315t-musllinux_1_2_s390x.whl", hash = "sha256:e923e75c9d641cdb8bbac60070a2dccdb684fc555af2bff48b0eaded0f88c2dc", size = 537798, upload-time = "2026-11-09T21:09:15.946Z" }, + { url = "https://files.pythonhosted.org/packages/60/43/a37ec29b0f98c3c639da495e1d547d6c44c69a6727b9864cf149bf346433/tornado-6.6.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:a72f70b1aa630fa2106d39a025e2be75c272b7ec8233da94e2f407ea754cc32d", size = 537809, upload-time = "2026-11-09T21:09:17.265Z" }, + { url = "https://files.pythonhosted.org/packages/3b/d4/13973391071552e03acb308d93f6deb4fb535a769a6bbdafdb4cafcb2f91/tornado-6.6.1-cp315-cp315t-win32.whl", hash = "sha256:2fbfd2510b6163b189773155a8246c025cb260ccc7cd12ad18c36c54f5617bb9", size = 535422, upload-time = "2026-11-09T21:09:18.525Z" }, + { url = "https://files.pythonhosted.org/packages/19/94/c952b625904b63fdac5b747f2eeaca64ab5d36cdb8faff57effa57162d87/tornado-6.6.1-cp315-cp315t-win_amd64.whl", hash = "sha256:3c35091add5863feb9cf651851bb1b23592f1f9551b5dfed72837c83ea4e5b85", size = 535713, upload-time = "2026-11-09T21:09:20.061Z" }, + { url = "https://files.pythonhosted.org/packages/f8/62/8d0c0c42f3346f3f2455738d3bc689bbd8efba38abf7b90ab2b8294eae83/tornado-6.6.1-cp315-cp315t-win_arm64.whl", hash = "sha256:065bc741478b168df9baf754956a3010c04d1330bb45a84833d807c29e11b257", size = 535719, upload-time = "2026-11-09T21:09:21.317Z" }, ] [[package]] From d8b85431b703d21bb660100ae5dfb9f8bf0d3bbe Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat, 14 Nov 2026 23:13:20 +0100 Subject: [PATCH 4/7] chore(deps): bump joserfc from 1.6.8 to 1.7.6 (#4632) Bumps [joserfc](https://github.com/authlib/joserfc) from 1.6.8 to 1.7.6. - [Release notes](https://github.com/authlib/joserfc/releases) - [Changelog](https://github.com/authlib/joserfc/blob/main/docs/changelog.rst) - [Commits](https://github.com/authlib/joserfc/compare/1.6.8...1.7.6) --- updated-dependencies: - dependency-name: joserfc dependency-version: 1.7.6 dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- uv.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/uv.lock b/uv.lock index 8dbf06aab4..7efabdf939 100644 --- a/uv.lock +++ b/uv.lock @@ -1293,14 +1293,14 @@ wheels = [ [[package]] name = "joserfc" -version = "1.6.8" +version = "1.7.6" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "cryptography" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/5d/ac/d4fd5b30f82900eac60d765f179f0ba005825ac462cc8ced6e13ec685ab3/joserfc-1.6.8.tar.gz", hash = "sha256:878620c553a6ebdd76ccdc356782fee3f735f21a356d079a546b42a4670ace5f", size = 232930, upload-time = "2026-05-27T03:22:37.819Z" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/db/ec5b5fe07a7bda62cd2c2d619daf160eb7442b979c7cd61bbe0262108de9/joserfc-1.7.6.tar.gz", hash = "sha256:fb1cf6d2977318bef976c2ed2ef2167cc5f8ade8fc96f034e38f7abd656907c4", size = 236091, upload-time = "2026-10-18T23:12:18.43Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/98/8c/5cdce2cf3ce8155849baf9a5e2ce77e89dc87ec3bdb38259e5d85fbc45bd/joserfc-1.6.8-py3-none-any.whl", hash = "sha256:22fb31a69094a5e6f44632002a9df2c30c941fc6c8ce1b037e92c03de954cf9f", size = 70927, upload-time = "2026-05-27T03:22:35.796Z" }, + { url = "https://files.pythonhosted.org/packages/98/40/63886ba742c6fd9e6dd222e20b8d54012fbc933465350c7747c4ed5b31e4/joserfc-1.7.6-py3-none-any.whl", hash = "sha256:e2d72e3fa937ed8ddd42ad8c1cc337a43e7ffd02efa572a2aa15e2764a05b506", size = 71588, upload-time = "2026-10-18T23:12:17.085Z" }, ] [[package]] From 0127f0492b4ae98b0495527dd5faadf4efd64462 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 16 Nov 2026 13:13:38 +0100 Subject: [PATCH 5/7] feat(zarr-metadata): export the v3 node types, spec default separators and narrowing helpers Add `ZarrV3NodeType` and `ZARR_V3_NODE_TYPE`; the default separators `ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR`, `DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` and `V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR`; and the `TypeIs` helpers `is_zarr_v2_array_order`, `is_zarr_v2_array_dimension_separator` and `is_zarr_v3_node_type`. The v2 array reader, the v3 node reader and the chunk key encodings' canonicalization now use them in place of their own literals, which removes two casts from the v2 array reader. Assisted-by: ClaudeCode:claude-opus-5-5 --- .../src/zarr_metadata/__init__.py | 16 +++- .../src/zarr_metadata/model/_array_v2.py | 41 ++++++---- .../src/zarr_metadata/model/_group_v3.py | 19 ++--- .../src/zarr_metadata/v2/array.py | 23 +++++- .../src/zarr_metadata/v3/__init__.py | 15 +++- .../src/zarr_metadata/v3/_common.py | 29 ++++++- .../v3/_types/chunk_key_encoding.py | 16 ++++ .../v3/chunk_key_encoding/__init__.py | 4 + .../v3/chunk_key_encoding/default.py | 4 +- .../zarr_metadata/v3/chunk_key_encoding/v2.py | 4 +- .../zarr-metadata/tests/test_public_api.py | 7 ++ .../zarr-metadata/tests/test_spec_values.py | 79 +++++++++++++++++++ 12 files changed, 224 insertions(+), 33 deletions(-) create mode 100644 packages/zarr-metadata/tests/test_spec_values.py diff --git a/packages/zarr-metadata/src/zarr_metadata/__init__.py b/packages/zarr-metadata/src/zarr_metadata/__init__.py index 3cde0f7230..1838352c1b 100644 --- a/packages/zarr-metadata/src/zarr_metadata/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/__init__.py @@ -38,6 +38,7 @@ from zarr_metadata.problems import MetadataValidationError, ProblemKind, ValidationProblem from zarr_metadata.typed_json import JSONValue from zarr_metadata.v2.array import ( + ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR, ZARR_V2_ARRAY_DIMENSION_SEPARATOR, ZARR_V2_ARRAY_METADATA_STORE_KEY, ZARR_V2_ARRAY_ORDER, @@ -65,7 +66,13 @@ ZarrV2GroupMetadataStoreKey, ZarrV2ZGroupJSON, ) -from zarr_metadata.v3 import ZarrV3DeclarationJSON, ZarrV3ExtensionField, ZarrV3NamedConfigJSON +from zarr_metadata.v3 import ( + ZARR_V3_NODE_TYPE, + ZarrV3DeclarationJSON, + ZarrV3ExtensionField, + ZarrV3NamedConfigJSON, + ZarrV3NodeType, +) from zarr_metadata.v3.array import ( ZARR_V3_ARRAY_METADATA_STORE_KEY, ZarrV3ArrayMetadataJSON, @@ -82,6 +89,7 @@ RegularChunkGridName, ) from zarr_metadata.v3.chunk_key_encoding.default import ( + DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, DEFAULT_CHUNK_KEY_ENCODING_NAME, DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR, DefaultChunkKeyEncodingMetadata, @@ -89,6 +97,7 @@ DefaultChunkKeyEncodingSeparator, ) from zarr_metadata.v3.chunk_key_encoding.v2 import ( + V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, V2_CHUNK_KEY_ENCODING_NAME, V2_CHUNK_KEY_ENCODING_SEPARATOR, V2ChunkKeyEncodingMetadata, @@ -281,6 +290,7 @@ "COMPLEX64_DATA_TYPE_NAME", "COMPLEX128_DATA_TYPE_NAME", "CRC32C_CODEC_NAME", + "DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "DEFAULT_CHUNK_KEY_ENCODING_NAME", "DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR", "ENDIANNESS", @@ -309,10 +319,12 @@ "UINT32_DATA_TYPE_NAME", "UINT64_DATA_TYPE_NAME", "UNSET", + "V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "V2_CHUNK_KEY_ENCODING_NAME", "V2_CHUNK_KEY_ENCODING_SEPARATOR", "VLEN_BYTES_CODEC_NAME", "VLEN_UTF8_CODEC_NAME", + "ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR", "ZARR_V2_ARRAY_DIMENSION_SEPARATOR", "ZARR_V2_ARRAY_METADATA_STORE_KEY", "ZARR_V2_ARRAY_ORDER", @@ -322,6 +334,7 @@ "ZARR_V3_ARRAY_METADATA_STORE_KEY", "ZARR_V3_CONSOLIDATED_METADATA_KEY", "ZARR_V3_GROUP_METADATA_STORE_KEY", + "ZARR_V3_NODE_TYPE", "ZSTD_CODEC_NAME", "BloscCName", "BloscCodecMetadata", @@ -444,6 +457,7 @@ "ZarrV3GroupMetadataUpdate", "ZarrV3NamedConfigJSON", "ZarrV3NodeMetadataInput", + "ZarrV3NodeType", "ZstdCodecMetadata", "ZstdCodecName", "__version__", diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_array_v2.py b/packages/zarr-metadata/src/zarr_metadata/model/_array_v2.py index 58795e93a6..2e5ba2930e 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_array_v2.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_array_v2.py @@ -66,10 +66,15 @@ from zarr_metadata.v2._context import ZarrV2Context # noqa: TC001 from zarr_metadata.v2._declaration_json import ZarrV2DataTypeMetadata from zarr_metadata.v2.array import ( + ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR, + ZARR_V2_ARRAY_DIMENSION_SEPARATOR, ZARR_V2_ARRAY_METADATA_STORE_KEY, + ZARR_V2_ARRAY_ORDER, ZarrV2ArrayDimensionSeparator, ZarrV2ArrayMetadataJSON, ZarrV2ArrayOrder, + is_zarr_v2_array_dimension_separator, + is_zarr_v2_array_order, ) from zarr_metadata.v2.attributes import ZARR_V2_ATTRIBUTES_STORE_KEY from zarr_metadata.v2.codec import ZarrV2CodecMetadata @@ -239,13 +244,15 @@ def read_array_v2( doc["dtype"], refined["dtype"], found, ZarrV2DataTypeDefinition, context, ("dtype",) ) problems.extend(found) + order: ZarrV2ArrayOrder | None = None if "order" in doc: # Checked after refinement, so a value that only compares equal to # "C" is refused, like every other value outside the set. - order, found = refine_json(doc["order"], ("order",)) - refined["order"] = order - if len(found) != 0 or order not in ("C", "F"): - problems.append(outside_of(("order",), doc["order"], ("C", "F"))) + refined["order"], found = refine_json(doc["order"], ("order",)) + if len(found) == 0 and is_zarr_v2_array_order(refined["order"]): + order = refined["order"] + else: + problems.append(outside_of(("order",), doc["order"], ZARR_V2_ARRAY_ORDER)) compressor: ResolvedDeclaration[ZarrV2CodecDefinition[Any]] | UNSET | None = UNSET if "compressor" in doc: compressor = None @@ -297,12 +304,20 @@ def read_array_v2( problems.extend(found) filters = tuple(read) refined["filters"] = tuple(items) + dimension_separator: ZarrV2ArrayDimensionSeparator = ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR if "dimension_separator" in doc: - separator, found = refine_json(doc["dimension_separator"], ("dimension_separator",)) - refined["dimension_separator"] = separator - if len(found) != 0 or separator not in (".", "/"): + refined["dimension_separator"], found = refine_json( + doc["dimension_separator"], ("dimension_separator",) + ) + if len(found) == 0 and is_zarr_v2_array_dimension_separator(refined["dimension_separator"]): + dimension_separator = refined["dimension_separator"] + else: problems.append( - outside_of(("dimension_separator",), doc["dimension_separator"], (".", "/")) + outside_of( + ("dimension_separator",), + doc["dimension_separator"], + ZARR_V2_ARRAY_DIMENSION_SEPARATOR, + ) ) fill_value: JSONValue = None if "fill_value" in doc: @@ -325,16 +340,14 @@ def read_array_v2( filters=filters, problems=within(with_input(problems, doc), at), ) - if len(problems) != 0 or shape is None or chunks is None or attributes is None: + if len(problems) != 0 or shape is None or chunks is None or order is None or attributes is None: return reading, None members = ArrayMembersV2( shape=shape, chunks=chunks, fill_value=fill_value, - order=cast("ZarrV2ArrayOrder", refined["order"]), - dimension_separator=cast( - "ZarrV2ArrayDimensionSeparator", refined.get("dimension_separator", ".") - ), + order=order, + dimension_separator=dimension_separator, attributes=attributes, extra_fields=extra_fields, document={key: refined[key] for key in cast("Mapping[str, object]", doc)}, @@ -897,7 +910,7 @@ def from_key_value( "order": "C", "compressor": None, "filters": None, - "dimension_separator": ".", + "dimension_separator": ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR, } ) """The document of `ZarrV2ArrayMetadata.create_default` given nothing: a scalar `|u1` array, without its `zarr_format`.""" diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py b/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py index 0d0127ad01..9c5d9dc006 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py @@ -76,7 +76,12 @@ unclaimed_declarations_of, unexpected_keys, ) -from zarr_metadata.v3._common import ZarrV3ExtensionField +from zarr_metadata.v3._common import ( + ZARR_V3_NODE_TYPE, + ZarrV3ExtensionField, + ZarrV3NodeType, + is_zarr_v3_node_type, +) # A runtime import: `get_type_hints` resolves the public signatures. from zarr_metadata.v3._context import ZarrV3Context # noqa: TC001 @@ -872,12 +877,8 @@ def _read_node_v3( return ZarrV3UnknownNodeMetadataReading(problems), None -_NODE_TYPES: Final = ("array", "group") -"""The node types the spec defines.""" - - -def _node_type(value: object) -> tuple[str | None, tuple[ValidationProblem, ...]]: - """The node type `value` declares, one of `_NODE_TYPES`. None, with the problems, when it declares none of them or is not an object. +def _node_type(value: object) -> tuple[ZarrV3NodeType | None, tuple[ValidationProblem, ...]]: + """The node type `value` declares, one of `ZARR_V3_NODE_TYPE`. None, with the problems, when it declares none of them or is not an object. When a document declares no valid node type, its `zarr_format` is checked too, so a document in another format is reported as not v3. @@ -886,7 +887,7 @@ def _node_type(value: object) -> tuple[str | None, tuple[ValidationProblem, ...] return None, not_an_object(value) document = value node_type = document.get("node_type") - if isinstance(node_type, str) and node_type in _NODE_TYPES: + if is_zarr_v3_node_type(node_type): return node_type, () problems = [ *missing_keys(frozenset({"zarr_format"}), document), @@ -895,7 +896,7 @@ def _node_type(value: object) -> tuple[str | None, tuple[ValidationProblem, ...] if "node_type" not in document: problems.append(ValidationProblem(("node_type",), "missing required key", "missing_key")) else: - problems.append(outside_of(("node_type",), node_type, _NODE_TYPES)) + problems.append(outside_of(("node_type",), node_type, ZARR_V3_NODE_TYPE)) return None, with_input(problems, document) diff --git a/packages/zarr-metadata/src/zarr_metadata/v2/array.py b/packages/zarr-metadata/src/zarr_metadata/v2/array.py index d0eacd29eb..4c0665b6b9 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v2/array.py +++ b/packages/zarr-metadata/src/zarr_metadata/v2/array.py @@ -4,7 +4,7 @@ from typing import Annotated, Final, Literal, NotRequired from annotated_types import Ge -from typing_extensions import TypedDict +from typing_extensions import TypedDict, TypeIs from zarr_metadata._common import JSONValue from zarr_metadata.v2._declaration_json import ZarrV2CodecMetadata, ZarrV2DataTypeMetadata @@ -20,6 +20,12 @@ ZARR_V2_ARRAY_ORDER: Final = ("C", "F") """Tuple of permitted values for the `order` field of v2 array metadata.""" + +def is_zarr_v2_array_order(value: object) -> TypeIs[ZarrV2ArrayOrder]: + """Whether `value` is one of `ZARR_V2_ARRAY_ORDER`, narrowing it to `ZarrV2ArrayOrder`.""" + return value in ZARR_V2_ARRAY_ORDER + + ZarrV2ArrayDimensionSeparator = Literal[".", "/"] """Literal type of permitted values for the `dimension_separator` field of v2 array metadata. @@ -32,6 +38,18 @@ ZARR_V2_ARRAY_DIMENSION_SEPARATOR: Final = (".", "/") """Tuple of permitted values for the `dimension_separator` field of v2 array metadata.""" +ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR: Final[ZarrV2ArrayDimensionSeparator] = "." +"""The `dimension_separator` a v2 array's metadata means when it has none. + +"If the value is not set, then the default MUST be assumed to be `"."`" +(https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L83-L86). +""" + + +def is_zarr_v2_array_dimension_separator(value: object) -> TypeIs[ZarrV2ArrayDimensionSeparator]: + """Whether `value` is one of `ZARR_V2_ARRAY_DIMENSION_SEPARATOR`, narrowing it to `ZarrV2ArrayDimensionSeparator`.""" + return value in ZARR_V2_ARRAY_DIMENSION_SEPARATOR + class ZarrV2ZArrayJSON(TypedDict): """ @@ -148,6 +166,7 @@ class ZarrV2ArrayMetadataJSON(TypedDict, extra_items=JSONValue): __all__ = [ + "ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR", "ZARR_V2_ARRAY_DIMENSION_SEPARATOR", "ZARR_V2_ARRAY_METADATA_STORE_KEY", "ZARR_V2_ARRAY_ORDER", @@ -156,4 +175,6 @@ class ZarrV2ArrayMetadataJSON(TypedDict, extra_items=JSONValue): "ZarrV2ArrayMetadataStoreKey", "ZarrV2ArrayOrder", "ZarrV2ZArrayJSON", + "is_zarr_v2_array_dimension_separator", + "is_zarr_v2_array_order", ] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/__init__.py index c831630f34..07b943a844 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/__init__.py @@ -1,8 +1,8 @@ """Zarr v3 metadata types, and the front door to the v3 definitions. This module holds the JSON of any declaration (a bare name or a -`ZarrV3NamedConfigJSON` envelope) and the node name and path -types. Each document type is in the submodule for its node: `array`, +`ZarrV3NamedConfigJSON` envelope), the node type, and the node name and +path types. Each document type is in the submodule for its node: `array`, `group` and `consolidated`. The extensions are in `codec`, `data_type`, `chunk_grid` and `chunk_key_encoding`, and the v3 kinds and scopes are in `definition`. @@ -18,7 +18,13 @@ from zarr_metadata._common import ZarrV3NamedConfigJSON from zarr_metadata._renamed import renamed_scope -from zarr_metadata.v3._common import ZarrV3DeclarationJSON, ZarrV3ExtensionField +from zarr_metadata.v3._common import ( + ZARR_V3_NODE_TYPE, + ZarrV3DeclarationJSON, + ZarrV3ExtensionField, + ZarrV3NodeType, + is_zarr_v3_node_type, +) from zarr_metadata.v3._core import RENAMED_SCOPES from zarr_metadata.v3._hierarchy import NodeName, NodePath from zarr_metadata.v3.chunk_grid.rectilinear import RECTILINEAR_CHUNK_GRID @@ -111,6 +117,7 @@ "VLEN_UTF8_CODEC", "ZARR_PYTHON_CONTEXT", "ZARR_PYTHON_RAW_BYTES_DATA_TYPE", + "ZARR_V3_NODE_TYPE", "ZSTD_CODEC", "NodeName", "NodePath", @@ -118,6 +125,8 @@ "ZarrV3DeclarationJSON", "ZarrV3ExtensionField", "ZarrV3NamedConfigJSON", + "ZarrV3NodeType", + "is_zarr_v3_node_type", ] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_common.py b/packages/zarr-metadata/src/zarr_metadata/v3/_common.py index 29be28b612..da8f4f511a 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/_common.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_common.py @@ -1,18 +1,38 @@ -"""The v3 declaration: its JSON type, and the type aliases for members that hold a declaration. +"""The v3 node types, the v3 declaration's JSON type, and the type aliases for members that hold a declaration. This module is private. It sits below both code paths that read a declaration: the model, which checks the members of a document, and the definitions, which read a declaration's configuration. Public consumers import +`ZarrV3NodeType`, `ZARR_V3_NODE_TYPE`, `is_zarr_v3_node_type` and `ZarrV3DeclarationJSON` from `zarr_metadata.v3`, and the aliases from `zarr_metadata.v3.definition`. One declaration is read with `zarr_metadata.definition.resolve`. """ -from typing import TypeAlias +from typing import Final, TypeAlias -from typing_extensions import TypeAliasType +from typing_extensions import TypeAliasType, TypeIs from zarr_metadata._common import JSONValue, ZarrV3NamedConfigJSON +from zarr_metadata._hierarchy import NodeType + +ZarrV3NodeType = NodeType +"""Literal type of permitted values for the `node_type` field of v3 metadata. + +An array's document declares `"array"` +(https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L483-L487) +and a group's `"group"` +(https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L760-L764). +""" + +ZARR_V3_NODE_TYPE: Final = ("array", "group") +"""Tuple of permitted values for the `node_type` field of v3 metadata.""" + + +def is_zarr_v3_node_type(value: object) -> TypeIs[ZarrV3NodeType]: + """Whether `value` is one of `ZARR_V3_NODE_TYPE`, narrowing it to `ZarrV3NodeType`.""" + return value in ZARR_V3_NODE_TYPE + ZarrV3ExtensionField: TypeAlias = JSONValue """The JSON value of an unknown top-level v3 metadata field. @@ -59,12 +79,15 @@ __all__ = [ + "ZARR_V3_NODE_TYPE", "ZarrV3ChunkGridDeclaration", "ZarrV3ChunkKeyEncodingDeclaration", "ZarrV3CodecDeclaration", "ZarrV3DataTypeDeclaration", "ZarrV3DeclarationJSON", "ZarrV3ExtensionField", + "ZarrV3NodeType", "ZarrV3StaticCodecDeclaration", "ZarrV3StorageTransformerDeclaration", + "is_zarr_v3_node_type", ] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_types/chunk_key_encoding.py b/packages/zarr-metadata/src/zarr_metadata/v3/_types/chunk_key_encoding.py index f73b73f0cc..4113c6ee07 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/_types/chunk_key_encoding.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_types/chunk_key_encoding.py @@ -30,6 +30,13 @@ DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR: Final = ("/", ".") """Tuple of permitted values for the `separator` field of the default chunk key encoding.""" +DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR: Final[DefaultChunkKeyEncodingSeparator] = "/" +"""The `separator` the default chunk key encoding means when its configuration has none. + +"If not specified, `separator` defaults to `/`" +(https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/chunk-key-encodings/default/index.rst#L27-L29). +""" + class DefaultChunkKeyEncodingConfiguration(TypedDict, closed=True): """Configuration for the default chunk key encoding. @@ -88,6 +95,13 @@ class DefaultChunkKeyEncodingObject(TypedDict, closed=True): V2_CHUNK_KEY_ENCODING_SEPARATOR: Final = ("/", ".") """Tuple of permitted values for the `separator` field of the v2 chunk key encoding.""" +V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR: Final[V2ChunkKeyEncodingSeparator] = "." +"""The `separator` the v2 chunk key encoding means when its configuration has none. + +"If not specified, `separator` defaults to `.`" +(https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/chunk-key-encodings/v2/index.rst#L27-L29). +""" + class V2ChunkKeyEncodingConfiguration(TypedDict, closed=True): """Configuration for the v2 chunk key encoding. @@ -130,8 +144,10 @@ class V2ChunkKeyEncodingObject(TypedDict, closed=True): __all__ = [ + "DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "DEFAULT_CHUNK_KEY_ENCODING_NAME", "DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR", + "V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "V2_CHUNK_KEY_ENCODING_NAME", "V2_CHUNK_KEY_ENCODING_SEPARATOR", "DefaultChunkKeyEncodingConfiguration", diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/__init__.py index 7162840872..e8563fa624 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/__init__.py @@ -20,6 +20,7 @@ from zarr_metadata.v3.chunk_key_encoding.default import ( DEFAULT_CHUNK_KEY_ENCODING, + DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, DEFAULT_CHUNK_KEY_ENCODING_NAME, DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR, DefaultChunkKeyEncodingConfiguration, @@ -30,6 +31,7 @@ ) from zarr_metadata.v3.chunk_key_encoding.v2 import ( V2_CHUNK_KEY_ENCODING, + V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, V2_CHUNK_KEY_ENCODING_NAME, V2_CHUNK_KEY_ENCODING_SEPARATOR, V2ChunkKeyEncodingConfiguration, @@ -41,9 +43,11 @@ __all__ = [ "DEFAULT_CHUNK_KEY_ENCODING", + "DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "DEFAULT_CHUNK_KEY_ENCODING_NAME", "DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR", "V2_CHUNK_KEY_ENCODING", + "V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "V2_CHUNK_KEY_ENCODING_NAME", "V2_CHUNK_KEY_ENCODING_SEPARATOR", "DefaultChunkKeyEncodingConfiguration", diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/default.py b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/default.py index 693c4c4720..a92aad7fa4 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/default.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/default.py @@ -11,6 +11,7 @@ from zarr_metadata.v3._definition import ChunkKeyEncodingDefinition from zarr_metadata.v3._types.chunk_key_encoding import ( + DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, DEFAULT_CHUNK_KEY_ENCODING_NAME, DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR, DefaultChunkKeyEncodingConfiguration, @@ -30,7 +31,7 @@ def _canonical( (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/chunk-key-encodings/default/index.rst#L27-L29), so the two spellings describe the same encoding. """ - if configuration.get("separator") != "/": + if configuration.get("separator") != DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR: return configuration return {} @@ -45,6 +46,7 @@ def _canonical( __all__ = [ "DEFAULT_CHUNK_KEY_ENCODING", + "DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "DEFAULT_CHUNK_KEY_ENCODING_NAME", "DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR", "DefaultChunkKeyEncodingConfiguration", diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/v2.py b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/v2.py index 24c458d102..752e553053 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/v2.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/chunk_key_encoding/v2.py @@ -17,6 +17,7 @@ from zarr_metadata.v3._definition import ChunkKeyEncodingDefinition from zarr_metadata.v3._types.chunk_key_encoding import ( + V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, V2_CHUNK_KEY_ENCODING_NAME, V2_CHUNK_KEY_ENCODING_SEPARATOR, V2ChunkKeyEncodingConfiguration, @@ -34,7 +35,7 @@ def _canonical(configuration: V2ChunkKeyEncodingConfiguration) -> V2ChunkKeyEnco (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/chunk-key-encodings/v2/index.rst#L27-L29), so the two spellings describe the same encoding. """ - if configuration.get("separator") != ".": + if configuration.get("separator") != V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR: return configuration return {} @@ -49,6 +50,7 @@ def _canonical(configuration: V2ChunkKeyEncodingConfiguration) -> V2ChunkKeyEnco __all__ = [ "V2_CHUNK_KEY_ENCODING", + "V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "V2_CHUNK_KEY_ENCODING_NAME", "V2_CHUNK_KEY_ENCODING_SEPARATOR", "V2ChunkKeyEncodingConfiguration", diff --git a/packages/zarr-metadata/tests/test_public_api.py b/packages/zarr-metadata/tests/test_public_api.py index e09dc0a0a8..71f7b8064e 100644 --- a/packages/zarr-metadata/tests/test_public_api.py +++ b/packages/zarr-metadata/tests/test_public_api.py @@ -180,6 +180,8 @@ def _group_rank(s: str) -> int: "ZarrV2ArrayOrder", "ZARR_V2_ARRAY_DIMENSION_SEPARATOR", "ZarrV2ArrayDimensionSeparator", + "ZARR_V3_NODE_TYPE", + "ZarrV3NodeType", "ENDIANNESS", "Endianness", "BYTES_CODEC_NAME", @@ -227,6 +229,10 @@ def _group_rank(s: str) -> int: "V2_CHUNK_KEY_ENCODING_SEPARATOR", "V2ChunkKeyEncodingSeparator", "NUMPY_TIME_UNIT", + # Spec defaults: the value an absent member means. + "ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR", + "DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", + "V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR", "NumpyTimeUnit", ] @@ -448,6 +454,7 @@ def test_promoted_pairs_drift() -> None: (zm.ZARR_V2_ARRAY_DIMENSION_SEPARATOR, zm.ZarrV2ArrayDimensionSeparator), (zm.DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR, zm.DefaultChunkKeyEncodingSeparator), (zm.V2_CHUNK_KEY_ENCODING_SEPARATOR, zm.V2ChunkKeyEncodingSeparator), + (zm.ZARR_V3_NODE_TYPE, zm.ZarrV3NodeType), ] for const, lit in pairs: assert set(const) == set(get_args(lit)) diff --git a/packages/zarr-metadata/tests/test_spec_values.py b/packages/zarr-metadata/tests/test_spec_values.py new file mode 100644 index 0000000000..d064d4bfc7 --- /dev/null +++ b/packages/zarr-metadata/tests/test_spec_values.py @@ -0,0 +1,79 @@ +"""Test the spec's closed value sets and defaults, as the narrowing helpers and default constants spell them.""" + +from collections.abc import Callable + +import pytest + +from zarr_metadata import ( + DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, + DEFAULT_CHUNK_KEY_ENCODING_NAME, + DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR, + UNSET, + V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, + V2_CHUNK_KEY_ENCODING_NAME, + V2_CHUNK_KEY_ENCODING_SEPARATOR, + ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR, + ZARR_V2_ARRAY_DIMENSION_SEPARATOR, + ZarrV2ArrayMetadata, +) +from zarr_metadata.definition import canonical_of, resolve +from zarr_metadata.v2.array import is_zarr_v2_array_dimension_separator, is_zarr_v2_array_order +from zarr_metadata.v3 import is_zarr_v3_node_type +from zarr_metadata.v3.definition import CORE_CONTEXT, ChunkKeyEncodingDefinition + + +@pytest.mark.parametrize( + ("helper", "value", "expected"), + [ + (is_zarr_v2_array_order, "C", True), + (is_zarr_v2_array_order, "F", True), + (is_zarr_v2_array_order, "c", False), + (is_zarr_v2_array_order, "A", False), + (is_zarr_v2_array_order, None, False), + (is_zarr_v2_array_dimension_separator, ".", True), + (is_zarr_v2_array_dimension_separator, "/", True), + (is_zarr_v2_array_dimension_separator, "\\", False), + (is_zarr_v2_array_dimension_separator, None, False), + (is_zarr_v3_node_type, "array", True), + (is_zarr_v3_node_type, "group", True), + (is_zarr_v3_node_type, "Array", False), + (is_zarr_v3_node_type, ("array",), False), + (is_zarr_v3_node_type, 3, False), + ], +) +def test_narrowing_helpers(helper: Callable[[object], bool], value: object, expected: bool) -> None: + """Each helper accepts exactly the members of its value set.""" + assert helper(value) is expected + + +def test_v2_array_default_dimension_separator() -> None: + """An array document without a `dimension_separator` means the default one, which is a permitted separator.""" + assert ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR in ZARR_V2_ARRAY_DIMENSION_SEPARATOR + metadata = ZarrV2ArrayMetadata.create_default(dimension_separator=UNSET) + assert "dimension_separator" not in metadata.to_json() + assert metadata.dimension_separator == ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR + + +@pytest.mark.parametrize( + ("name", "default", "permitted"), + [ + ( + DEFAULT_CHUNK_KEY_ENCODING_NAME, + DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, + DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR, + ), + ( + V2_CHUNK_KEY_ENCODING_NAME, + V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, + V2_CHUNK_KEY_ENCODING_SEPARATOR, + ), + ], +) +def test_chunk_key_encoding_default_separator( + name: str, default: str, permitted: tuple[str, ...] +) -> None: + """A chunk key encoding's default separator is permitted, and its simplest spelling omits it.""" + assert default in permitted + declaration = {"name": name, "configuration": {"separator": default}} + resolved, problems = resolve(declaration, ChunkKeyEncodingDefinition, CORE_CONTEXT) + assert canonical_of(resolved, problems) == {"name": name} From bd7651ed44e27e5735961f0b39ca8df0f4bde392 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 16 Nov 2026 13:14:37 +0100 Subject: [PATCH 6/7] docs(zarr-metadata): add the changelog fragment for the spec constants Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-metadata/changes/448.feature.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 packages/zarr-metadata/changes/448.feature.md diff --git a/packages/zarr-metadata/changes/448.feature.md b/packages/zarr-metadata/changes/448.feature.md new file mode 100644 index 0000000000..759b4bac5f --- /dev/null +++ b/packages/zarr-metadata/changes/448.feature.md @@ -0,0 +1,8 @@ +Add `ZarrV3NodeType` and `ZARR_V3_NODE_TYPE` for the `node_type` member of a +v3 document, and constants for the separator the spec assumes when one is +absent: `ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR` (`"."`), +`DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` (`"/"`) and +`V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` (`"."`). The `TypeIs` functions +`is_zarr_v2_array_order`, `is_zarr_v2_array_dimension_separator` (in +`zarr_metadata.v2.array`) and `is_zarr_v3_node_type` (in `zarr_metadata.v3`) +narrow any value to the matching `Literal` type, under every type checker. From 63fa4054c1cb55bf0e0f7f29c980733a06fea9f5 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 16 Nov 2026 13:22:58 +0100 Subject: [PATCH 7/7] docs(zarr-metadata): name the changelog fragment after the upstream PR Assisted-by: ClaudeCode:claude-opus-5-5 --- .../zarr-metadata/changes/{448.feature.md => 4638.feature.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename packages/zarr-metadata/changes/{448.feature.md => 4638.feature.md} (100%) diff --git a/packages/zarr-metadata/changes/448.feature.md b/packages/zarr-metadata/changes/4638.feature.md similarity index 100% rename from packages/zarr-metadata/changes/448.feature.md rename to packages/zarr-metadata/changes/4638.feature.md