Skip to content

feat(zarr-metadata): read length and step constraints from annotated-types - #4636

Merged
d-v-b merged 14 commits into
zarr-developers:mainfrom
d-v-b:feat/zarr-metadata-length-and-step-constraints
Nov 14, 2026
Merged

d-v-b merged 14 commits into
zarr-developers:mainfrom
d-v-b:feat/zarr-metadata-length-and-step-constraints

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

zarr_metadata.typed_json.check and definitions' configurations now read more annotated-types constraints, as pydantic does: MinLen, MaxLen and Len on an array or a string, and MultipleOf on an integer. A value outside them is an invalid_value problem whose ctx uses pydantic's names (min_length, multiple_of), and the JSON Schema writes the matching keyword, such as minItems, minLength or multipleOf. A constraint on a type it does not apply to raises TypeError. Constant rules now live on the types, so the JSON Schemas say them too: non-empty codecs, sharding codecs and index_codecs, rectilinear edge lists, struct fields and v3 struct field names; length_bytes of at least 1 for null_terminated_bytes and zarr-python raw_bytes, and a positive multiple of 4 for fixed_length_utf32. Raw r<N> declares its bits a positive multiple of 8 on its type, and a refused name's message now names them: bits in "r12": .... check now also refuses "codecs": [].

The rectilinear chunk grid now refuses an empty list of chunk edge lengths, such as "chunk_shapes": [[]], even along a dimension of length 0. zarr-extensions#78 made the list form non-empty; a dimension with no chunks writes a bare integer instead. The JSON Schema says minItems: 1 for the list, and chunk_grid_configurations no longer draws an empty list.

What changed, commit by commit
  1. fix(zarr-metadata): refuse an empty rectilinear edge list: brings the rectilinear grid in line with zarr-extensions#78 (pinned at cadd1dab6). This first version used a hand-written rule and a hand-patched JSON Schema fragment, because the checker could not yet bound a length.
  2. feat(zarr-metadata): read MinLen and MaxLen on arrays in the TypedDict checker: the checker, the JSON Schema writer and the pydantic schema read MinLen/MaxLen/Len on arrays. The array document's codecs and each rectilinear edge list declare MinLen(1). This removes both hand-patched minItems: 1 fragments from v3/_json_schema.py and the rectilinear _rules from commit 1.
  3. feat(zarr-metadata): declare sharding's codec pipelines non-empty: codecs and index_codecs of sharding_indexed declare MinLen(1).
  4. refactor(zarr-metadata): move constant data type rules onto their types: Ge(1) on length_bytes of null_terminated_bytes and zarr-python raw_bytes, and MinLen(1) on v3 struct fields. Two rules are deleted.
  5. feat(zarr-metadata): read MultipleOf on integers in the TypedDict checker: fixed_length_utf32.length_bytes and raw r<N> bits declare Gt(0), MultipleOf(4 or 8). Two more rules are deleted.
  6. feat(zarr-metadata): read MinLen and MaxLen on strings in the TypedDict checker: the v3 struct field name declares MinLen(1), and the struct rule's empty-name check is deleted.

Every problem a deleted rule reported is still reported, at the same loc and with the same kind. The message is now the checker's generic one, for example "expected an integer > 0 that is a multiple of 4, got 6".

Design notes
  • One table, _CONSTRAINTS in _typed_json.py, now drives parsing, the "constraint on the wrong shape" check and the JSON Schema keywords. It replaces seven parallel tables. Each entry names its annotated-types marker, the shapes it applies to with the JSON Schema keyword for each (MinLen writes minItems on an array and minLength on a string), what it measures, and a slot (below, above or step). A type takes one constraint per slot.
  • Names follow pydantic: the ctx keys are min_length, max_length and multiple_of, and the problem kind stays invalid_value.
  • MultipleOf is accepted on integers only. JSON Schema validators compare floats inexactly, and no zarr type needs a float step. A step must be a positive integer, as multipleOf requires. When a type alias and the annotation on it each carry a step, the schema writes their least common multiple.
  • A string's length counts code points, as both Python's len and JSON Schema's minLength do, so the checker and the schema agree.
  • Still refused: a length bound on an object, Predicate, and pydantic's Field/StringConstraints. None of them is in annotated-types, has a JSON Schema keyword, or is needed today.
  • The array model's reader found its lists of extension points with get_origin(annotation) is tuple, which an Annotated wrapper hides. It now strips the annotation first.
  • A definition that reuses RawBytesConfiguration to accept any raw size can no longer do so, because the type now says the spec's rule. The test that showed a reader defining r* its own way now uses its own configuration type.
  • An empty struct field name is now a type problem, so the struct rules (duplicate names, variable-size field types) wait until it is fixed, as they do for any out-of-bounds member.
Testing

The agent ran just check in packages/zarr-metadata: ruff, pyright, and the suite on Python 3.11 (3920 passed, 3 skipped) and 3.14 (3923 passed), plus the docs build. The tests follow one table test for the expected results plus one test per error case: a length bound on a non-array or non-string, a step on a float, a step or length that is not a valid integer, and a second constraint in one slot.

Review

The agent ran two reviews of this PR: roborev on every commit, and a separate agent that saw only the PR. These findings held up and are fixed in this PR:

  • A length bound was skipped when an element reported an unknown_key. That problem leaves the value readable, so check could return an array of closed TypedDicts longer than its MaxLen, and a definition's rules ran on it. No type in the package could trigger it. Fixed in fix(zarr-metadata): hold a length bound beside an unknown key, with a table case.
  • "data_type": "r12" reported "expected an integer > 0 that is a multiple of 8, got 12", about a string the document wrote. A type-check problem with a configuration that a name encodes now names the member and the name: bits in "r12": .... A rule's message is left as written: v2 typestr rules already describe the typestr, and prefixing them would show internal keys such as byteorder.
  • The sharding empty-pipeline tests passed with or without MinLen(1). They now assert ctx={"min_length": 1}.
  • The rectilinear strategy test called the grid's rules, which no longer check for empty lists. It now reads the configuration through the definition.
  • The changelog said the JSON Schema of a raw r<N> name carries its multiple of 8. It does not, because that schema is the name pattern. The text now says only what is true.
  • Stale docs: the ValidationProblem ctx keys, and the first line of constraints_of.

Noted and left as is:

  • An array document's codecs: [] read through the model still reports the pipeline's "expected an array -> bytes codec, got none". check and a shard's pipelines report the min_length problem. The loc and kind are the same.
  • For a step on a NewType plus another step on top, the checker requires the least common multiple, as it applies both bounds of a bounded NewType. pydantic keeps the outer step. No type in the package stacks steps.
Branch notes

🤖 Generated with Claude Code

d-v-b and others added 9 commits November 14, 2026 18:17
`_codec` drew from `st.sampled_from` over `(definition, configurations)`
pairs whose configurations are strategies built during the draw. Hypothesis
labels such a `sampled_from` by hashing its elements, and a strategy hashes
by address, so these spans were labeled by memory layout. Hypothesis's
mutator copies choices between spans that share a label, so which spans it
grouped, and so which examples a derandomized run drew, changed with
whatever moved the allocator: another process, machine load. `_codec` now
samples an index into the options, labeled by a `range`.

The share tests measured this drift. `examples_of` now draws twice and
fails if the runs differ, which catches an address-labeled strategy in the
same process; the array-document targets share one draw. Bounds are reset
from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the
seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the
seed minimum (0.06).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zarr-extensions#78 (cadd1dab6) made the array form of a dimension's
chunk edge lengths non-empty, even for an axis of length 0, which
writes a bare integer to have no chunks. The rectilinear chunk grid's
rules now report `chunk_shapes: [[]]`, its JSON Schema says
`minItems: 1` for the list, and `chunk_grid_configurations` always
draws at least one edge. Spec links are re-pinned to cadd1dab6.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t checker

The checker, the JSON Schema writer and the pydantic schema now read
annotated-types' `MinLen`, `MaxLen` and `Len` on an array, as pydantic
does. One table of constraints now drives parsing, the shape check and
the schema keywords, in place of five parallel tables. An array outside
its bounds is an `invalid_value` problem whose `ctx` holds `min_length`
or `max_length`. A length bound on anything but an array raises
`TypeError`.

`codecs` and each rectilinear dimension's edge list now declare
`MinLen(1)` on their types. This removes the two hand-patched
`minItems: 1` schema fragments and the rectilinear `_rules` function.
`chunk_shapes: [[]]` and `codecs: []` are still refused at the same
`loc`. `check` now also refuses `codecs: []`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The `sharding_indexed` codec's `codecs` and `index_codecs` each hold
exactly one array -> bytes codec, so neither is ever empty. Both now
declare `MinLen(1)` on their types, as the array document's `codecs`
does. An empty one is reported by the type check at the same `loc` as
before, with `ctx={"min_length": 1}`. Before, the pipeline's order check
reported it as "expected an array -> bytes codec, got none".

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`null_terminated_bytes` and zarr-python's `raw_bytes` checked
`length_bytes >= 1` in a rule, and v3 `struct` checked that it has a
field. The types now declare `Ge(1)` and `MinLen(1)`, the two rules that
did only this are gone, and the struct rule keeps its name and field-type
checks. The problems keep their `loc` and kind. The JSON Schema of each
type now says so too: it accepted `length_bytes: 0` and `fields: []`,
which the reader refuses.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cker

The checker, the JSON Schema writer and the pydantic schema now read
annotated-types' `MultipleOf` on an integer, as pydantic does under
`multiple_of`. JSON Schema's keyword is `multipleOf`. The step must be a
positive integer, and a type takes one. Two steps on a NewType and its
annotation merge to their least common multiple. A float is refused,
because validators compare floats inexactly.

Each constraint in the table now names the shapes it applies to, with
the JSON Schema keyword for each, and a slot (below, above or step) in
place of a side.

`fixed_length_utf32`'s `length_bytes` and raw `r<N>` `bits` now declare
`Gt(0), MultipleOf(8 or 4)` on their types. The two rules that checked
only this are gone. A test whose own `r*` definition accepts any size
now uses its own configuration type.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ct checker

`MinLen`, `MaxLen` and `Len` now also bound a string, as pydantic's
`min_length` does. The JSON Schema writes `minLength` and `maxLength`.
Python and JSON Schema both count code points, so the checker and the
schema agree on what a string's length is. Messages count characters:
"expected a string with at least 1 character, got \"\"". A length bound
on an object still raises `TypeError`.

The v3 `struct` field name now declares `MinLen(1)` on its type, and
the struct rule no longer checks for an empty name. The problem keeps
its `loc` and kind.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package labels Nov 14, 2026
@read-the-docs-community

read-the-docs-community Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.90%. Comparing base (011d68a) to head (8ea0355).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4636   +/-   ##
=======================================
  Coverage   94.90%   94.90%           
=======================================
  Files          96       96           
  Lines       14051    14051           
=======================================
  Hits        13335    13335           
  Misses        716      716           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

d-v-b and others added 5 commits November 14, 2026 19:48
…32 bits

The derandomized share stream changes with any change to what the
strategies draw, and with machine load: base main measures 0.103
against a floor of 0.1 under load, and the rectilinear fix measures
0.047 to 0.133 across runs. Over seeds 0-5 at 500 examples both trees
average about 0.17, from 0.07 to 0.29. A floor of 0.03 still asks that
a few hundred examples meet several such axes.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`constrained` skipped a type's constraints whenever the inner parse
reported any problem. An `unknown_key` problem leaves the value readable,
because the key is dropped and the value is still returned. So
`check` returned an array of closed TypedDicts longer than its `MaxLen`,
and a definition's rules ran on it, although the rules run only on a
configuration within its bounds. The constraints are now skipped only
for a problem other than an unknown key. Number bounds were never
affected, because a number holds no keys.

The docs now list `multiple_of`, `min_length` and `max_length` among the
`ctx` keys, and `constraints_of` says a type takes one step too.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A raw `r<N>` name's size is now checked by its type. For
`"data_type": "r12"` that gave "expected an integer > 0 that is a
multiple of 8, got 12", about a string the document wrote, with no
mention of bits. A type-check problem with a configuration that a name
encodes now names the member and the name: `bits in "r12": ...`. A
rule's message is left as written, because such rules already describe
the name, as the v2 typestr rules do.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ge lists

The sharding tests for an empty `codecs` or `index_codecs` checked only
the `loc` and kind, which the pipeline's order check also reports. They
now also assert `ctx={"min_length": 1}`, which only the type check gives.
The rectilinear strategy test checked the grid's rules, which no longer
look at empty lists. It now reads the configuration with the definition,
which checks the type. The changelog no longer says the JSON Schema of a
raw `r<N>` name carries its multiple of 8: that schema is the name
pattern.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The share tests in `tests/testing` flaked on this branch: "an axis past
32 bits" below its floor, and "no axes" above its ceiling on Python
3.14. zarr-developers#4635 fixes the cause, a `sampled_from` that Hypothesis labeled by
memory address. It also resets the bounds from 100 seeds. The conflict
in `_DOCUMENT_SHARES` takes zarr-developers#4635's bounds.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b
d-v-b marked this pull request as ready for review November 14, 2026 21:05
@d-v-b
d-v-b merged commit 22c8741 into zarr-developers:main Nov 14, 2026
45 checks passed
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Nov 14, 2026
zarr-developers#4636 was stacked on this PR's first two commits and squash-merged, so
main holds their index-sampling version; each conflict keeps this
branch's newer version. zarr-developers#4636's rectilinear change (always at least one
edge) moves the array-document shares, so their bounds are re-measured
with `bounds(across_seeds(...))`: `bool`'s seed maximum over 1000
examples rose to 0.108, and its ceiling to 0.13.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant