Skip to content

feat(zarr-metadata): export the v3 node types, spec default separators and narrowing helpers - #448

Closed
d-v-b wants to merge 7 commits into
mainfrom
feat/zarr-metadata-spec-constants
Closed

d-v-b wants to merge 7 commits into
mainfrom
feat/zarr-metadata-spec-constants

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Adds ZarrV3NodeType and ZARR_V3_NODE_TYPE, and constants for the separators the spec assumes when one is absent (ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR, DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR, V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR). Adds TypeIs helpers that narrow an object to a spec value set: is_zarr_v2_array_order, is_zarr_v2_array_dimension_separator and is_zarr_v3_node_type.

Why

pydantic-zarr hardcodes these spec values or re-derives them (get_args(MemoryOrder) followed by cast, node_type == "array" or node_type == "group", a hardcoded "." default). It should import them from zarr-metadata instead.

The agent checked whether a tuple constant alone is enough, using x in ZARR_V2_ARRAY_ORDER with x: object:

  • pyright and pyrefly narrow x to the Literal; mypy and ty do not.
  • A generic is_one_of(value, allowed: tuple[T, ...]) -> TypeIs[T] widens T to str under pyright and mypy.

So each value set gets its own helper.

What changed
  • zarr_metadata.v2.array: ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR, is_zarr_v2_array_order, is_zarr_v2_array_dimension_separator.
  • zarr_metadata.v3 (home: the private module v3/_common.py): ZarrV3NodeType (an alias of the format-neutral _hierarchy.NodeType), ZARR_V3_NODE_TYPE, is_zarr_v3_node_type.
  • zarr_metadata.v3.chunk_key_encoding.{default,v2}: DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR ("/") and V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR ("."), each citing the spec line.
  • The top-level zarr_metadata exports the constants and types. The helpers stay in their home modules, like is_node_name_v3.
  • The package now uses these names itself:
    • The v2 array reader drops ("C", "F"), (".", "/"), its "." default and two cast calls.
    • The v3 node reader replaces the private _NODE_TYPES and now returns ZarrV3NodeType | None instead of str | None.
    • Each chunk key encoding's _canonical compares against its default constant.
Considered and left out
  • A shared FloatSpecialFillValue / FLOAT_SPECIAL_FILL_VALUES export. It would need a single public home, and putting it in the float modules makes it value-tied with Float{16,32,64}SpecialFillValue. Those per-width aliases are already public.
  • Public document key sets. ZarrV3GroupMetadataJSON.__required_keys__ | __optional_keys__ already gives them at runtime.
  • A v2 node-name validator, and a codec-definition property for "passes on the data type it receives". These change behaviour, so they are better as separate PRs.
  • The default sharding index codecs. zarr-python picks them; the spec does not define them.
Tests
  • tests/test_spec_values.py: one parametrized test of the three helpers over members and non-members; one test that the v2 array model reads an absent dimension_separator as the default; one test that each chunk key encoding's simplest spelling drops its default separator.
  • test_public_api.py: the new front-door names, and the ZARR_V3_NODE_TYPE/ZarrV3NodeType drift pair.
  • Local runs: 3940 passed and 3 skipped on Python 3.13; pyright (Python 3.11) reports 0 errors over sources and tests; ruff passes; the pre-commit hooks pass; mkdocs build --strict printed no warnings.

🤖 Generated with Claude Code

d-v-b and others added 6 commits November 14, 2026 22:05
…types (zarr-developers#4636)

* fix(zarr-metadata): draw the same codecs on every derandomized run

`_codec` drew from `st.sampled_from` over `(definition, configurations)`
pairs whose configurations are strategies built during the draw. Hypothesis
labels such a `sampled_from` by hashing its elements, and a strategy hashes
by address, so these spans were labeled by memory layout. Hypothesis's
mutator copies choices between spans that share a label, so which spans it
grouped, and so which examples a derandomized run drew, changed with
whatever moved the allocator: another process, machine load. `_codec` now
samples an index into the options, labeled by a `range`.

The share tests measured this drift. `examples_of` now draws twice and
fails if the runs differ, which catches an address-labeled strategy in the
same process; the array-document targets share one draw. Bounds are reset
from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the
seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the
seed minimum (0.06).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-metadata): add changelog fragment for zarr-developers#4635

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(zarr-metadata): refuse an empty rectilinear edge list

zarr-extensions#78 (cadd1dab6) made the array form of a dimension's
chunk edge lengths non-empty, even for an axis of length 0, which
writes a bare integer to have no chunks. The rectilinear chunk grid's
rules now report `chunk_shapes: [[]]`, its JSON Schema says
`minItems: 1` for the list, and `chunk_grid_configurations` always
draws at least one edge. Spec links are re-pinned to cadd1dab6.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(zarr-metadata): read MinLen and MaxLen on arrays in the TypedDict checker

The checker, the JSON Schema writer and the pydantic schema now read
annotated-types' `MinLen`, `MaxLen` and `Len` on an array, as pydantic
does. One table of constraints now drives parsing, the shape check and
the schema keywords, in place of five parallel tables. An array outside
its bounds is an `invalid_value` problem whose `ctx` holds `min_length`
or `max_length`. A length bound on anything but an array raises
`TypeError`.

`codecs` and each rectilinear dimension's edge list now declare
`MinLen(1)` on their types. This removes the two hand-patched
`minItems: 1` schema fragments and the rectilinear `_rules` function.
`chunk_shapes: [[]]` and `codecs: []` are still refused at the same
`loc`. `check` now also refuses `codecs: []`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(zarr-metadata): declare sharding's codec pipelines non-empty

The `sharding_indexed` codec's `codecs` and `index_codecs` each hold
exactly one array -> bytes codec, so neither is ever empty. Both now
declare `MinLen(1)` on their types, as the array document's `codecs`
does. An empty one is reported by the type check at the same `loc` as
before, with `ctx={"min_length": 1}`. Before, the pipeline's order check
reported it as "expected an array -> bytes codec, got none".

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(zarr-metadata): move constant data type rules onto their types

`null_terminated_bytes` and zarr-python's `raw_bytes` checked
`length_bytes >= 1` in a rule, and v3 `struct` checked that it has a
field. The types now declare `Ge(1)` and `MinLen(1)`, the two rules that
did only this are gone, and the struct rule keeps its name and field-type
checks. The problems keep their `loc` and kind. The JSON Schema of each
type now says so too: it accepted `length_bytes: 0` and `fields: []`,
which the reader refuses.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(zarr-metadata): read MultipleOf on integers in the TypedDict checker

The checker, the JSON Schema writer and the pydantic schema now read
annotated-types' `MultipleOf` on an integer, as pydantic does under
`multiple_of`. JSON Schema's keyword is `multipleOf`. The step must be a
positive integer, and a type takes one. Two steps on a NewType and its
annotation merge to their least common multiple. A float is refused,
because validators compare floats inexactly.

Each constraint in the table now names the shapes it applies to, with
the JSON Schema keyword for each, and a slot (below, above or step) in
place of a side.

`fixed_length_utf32`'s `length_bytes` and raw `r<N>` `bits` now declare
`Gt(0), MultipleOf(8 or 4)` on their types. The two rules that checked
only this are gone. A test whose own `r*` definition accepts any size
now uses its own configuration type.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(zarr-metadata): read MinLen and MaxLen on strings in the TypedDict checker

`MinLen`, `MaxLen` and `Len` now also bound a string, as pydantic's
`min_length` does. The JSON Schema writes `minLength` and `maxLength`.
Python and JSON Schema both count code points, so the checker and the
schema agree on what a string's length is. Messages count characters:
"expected a string with at least 1 character, got \"\"". A length bound
on an object still raises `TypeError`.

The v3 `struct` field name now declares `MinLen(1)` on its type, and
the struct rule no longer checks for an empty name. The problem keeps
its `loc` and kind.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(zarr-metadata): number the changelog fragments by this PR

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(zarr-metadata): loosen the floor on documents with an axis past 32 bits

The derandomized share stream changes with any change to what the
strategies draw, and with machine load: base main measures 0.103
against a floor of 0.1 under load, and the rectilinear fix measures
0.047 to 0.133 across runs. Over seeds 0-5 at 500 examples both trees
average about 0.17, from 0.07 to 0.29. A floor of 0.03 still asks that
a few hundred examples meet several such axes.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(zarr-metadata): hold a length bound beside an unknown key

`constrained` skipped a type's constraints whenever the inner parse
reported any problem. An `unknown_key` problem leaves the value readable,
because the key is dropped and the value is still returned. So
`check` returned an array of closed TypedDicts longer than its `MaxLen`,
and a definition's rules ran on it, although the rules run only on a
configuration within its bounds. The constraints are now skipped only
for a problem other than an unknown key. Number bounds were never
affected, because a number holds no keys.

The docs now list `multiple_of`, `min_length` and `max_length` among the
`ctx` keys, and `constraints_of` says a type takes one step too.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(zarr-metadata): name the member a carried name's type check refuses

A raw `r<N>` name's size is now checked by its type. For
`"data_type": "r12"` that gave "expected an integer > 0 that is a
multiple of 8, got 12", about a string the document wrote, with no
mention of bits. A type-check problem with a configuration that a name
encodes now names the member and the name: `bits in "r12": ...`. A
rule's message is left as written, because such rules already describe
the name, as the v2 typestr rules do.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(zarr-metadata): pin that the types refuse empty pipelines and edge lists

The sharding tests for an empty `codecs` or `index_codecs` checked only
the `loc` and kind, which the pipeline's order check also reports. They
now also assert `ctx={"min_length": 1}`, which only the type check gives.
The rectilinear strategy test checked the grid's rules, which no longer
look at empty lists. It now reads the configuration with the definition,
which checks the type. The changelog no longer says the JSON Schema of a
raw `r<N>` name carries its multiple of 8: that schema is the name
pattern.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…arr-developers#4635)

* fix(zarr-metadata): draw the same codecs on every derandomized run

`_codec` drew from `st.sampled_from` over `(definition, configurations)`
pairs whose configurations are strategies built during the draw. Hypothesis
labels such a `sampled_from` by hashing its elements, and a strategy hashes
by address, so these spans were labeled by memory layout. Hypothesis's
mutator copies choices between spans that share a label, so which spans it
grouped, and so which examples a derandomized run drew, changed with
whatever moved the allocator: another process, machine load. `_codec` now
samples an index into the options, labeled by a `range`.

The share tests measured this drift. `examples_of` now draws twice and
fails if the runs differ, which catches an address-labeled strategy in the
same process; the array-document targets share one draw. Bounds are reset
from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the
seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the
seed minimum (0.06).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-metadata): add changelog fragment for zarr-developers#4635

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(zarr-metadata): sample codec names and make share bounds reproducible

`_codec` samples definition names rather than an index. An index into
options of the same length shares one label with every other, so the
mutator could copy a choice between unrelated codec slots; names key the
label on the option set, as the chunk grid and data type draws already do.

`examples_of` is cached per strategy, so the share tests that count one
strategy draw it once, and its docstring calls the second-run check what
it is, a tripwire. `across_seeds` and `bounds` put the rule the bounds
follow into code, so a bound can be reset rather than re-derived; every
bound is reset by it. The two first-data-type checks run over 1000
examples: over 300, `bool`'s share across seeds reached 0.13 to 0.15,
too near the 0.2 they guard against. The changelog fragment no longer
says the options' strategies are built during the draw: some are cached.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Bumps the uv group with 1 update in the /packages/zarr-http-server directory: [tornado](https://github.com/tornadoweb/tornado).


Updates `tornado` from 6.6 to 6.6.1
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](tornadoweb/tornado@v6.6.0...v6.6.1)

---
updated-dependencies:
- dependency-name: tornado
  dependency-version: 6.6.1
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [joserfc](https://github.com/authlib/joserfc) from 1.6.8 to 1.7.6.
- [Release notes](https://github.com/authlib/joserfc/releases)
- [Changelog](https://github.com/authlib/joserfc/blob/main/docs/changelog.rst)
- [Commits](authlib/joserfc@1.6.8...1.7.6)

---
updated-dependencies:
- dependency-name: joserfc
  dependency-version: 1.7.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…s and narrowing helpers

Add `ZarrV3NodeType` and `ZARR_V3_NODE_TYPE`; the default separators
`ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR`,
`DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` and
`V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR`; and the `TypeIs` helpers
`is_zarr_v2_array_order`, `is_zarr_v2_array_dimension_separator` and
`is_zarr_v3_node_type`. The v2 array reader, the v3 node reader and the
chunk key encodings' canonicalization now use them in place of their own
literals, which removes two casts from the v2 array reader.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b

d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Superseded by zarr-developers#4638, which is opened from the same branch. The agent closed this fork PR and kept the branch, because the upstream PR uses it.

@d-v-b d-v-b closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant