Repository navigation
Conversation
…types (zarr-developers#4636) * fix(zarr-metadata): draw the same codecs on every derandomized run `_codec` drew from `st.sampled_from` over `(definition, configurations)` pairs whose configurations are strategies built during the draw. Hypothesis labels such a `sampled_from` by hashing its elements, and a strategy hashes by address, so these spans were labeled by memory layout. Hypothesis's mutator copies choices between spans that share a label, so which spans it grouped, and so which examples a derandomized run drew, changed with whatever moved the allocator: another process, machine load. `_codec` now samples an index into the options, labeled by a `range`. The share tests measured this drift. `examples_of` now draws twice and fails if the runs differ, which catches an address-labeled strategy in the same process; the array-document targets share one draw. Bounds are reset from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the seed minimum (0.06). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(zarr-metadata): add changelog fragment for zarr-developers#4635 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(zarr-metadata): refuse an empty rectilinear edge list zarr-extensions#78 (cadd1dab6) made the array form of a dimension's chunk edge lengths non-empty, even for an axis of length 0, which writes a bare integer to have no chunks. The rectilinear chunk grid's rules now report `chunk_shapes: [[]]`, its JSON Schema says `minItems: 1` for the list, and `chunk_grid_configurations` always draws at least one edge. Spec links are re-pinned to cadd1dab6. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(zarr-metadata): read MinLen and MaxLen on arrays in the TypedDict checker The checker, the JSON Schema writer and the pydantic schema now read annotated-types' `MinLen`, `MaxLen` and `Len` on an array, as pydantic does. One table of constraints now drives parsing, the shape check and the schema keywords, in place of five parallel tables. An array outside its bounds is an `invalid_value` problem whose `ctx` holds `min_length` or `max_length`. A length bound on anything but an array raises `TypeError`. `codecs` and each rectilinear dimension's edge list now declare `MinLen(1)` on their types. This removes the two hand-patched `minItems: 1` schema fragments and the rectilinear `_rules` function. `chunk_shapes: [[]]` and `codecs: []` are still refused at the same `loc`. `check` now also refuses `codecs: []`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(zarr-metadata): declare sharding's codec pipelines non-empty The `sharding_indexed` codec's `codecs` and `index_codecs` each hold exactly one array -> bytes codec, so neither is ever empty. Both now declare `MinLen(1)` on their types, as the array document's `codecs` does. An empty one is reported by the type check at the same `loc` as before, with `ctx={"min_length": 1}`. Before, the pipeline's order check reported it as "expected an array -> bytes codec, got none". Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(zarr-metadata): move constant data type rules onto their types `null_terminated_bytes` and zarr-python's `raw_bytes` checked `length_bytes >= 1` in a rule, and v3 `struct` checked that it has a field. The types now declare `Ge(1)` and `MinLen(1)`, the two rules that did only this are gone, and the struct rule keeps its name and field-type checks. The problems keep their `loc` and kind. The JSON Schema of each type now says so too: it accepted `length_bytes: 0` and `fields: []`, which the reader refuses. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(zarr-metadata): read MultipleOf on integers in the TypedDict checker The checker, the JSON Schema writer and the pydantic schema now read annotated-types' `MultipleOf` on an integer, as pydantic does under `multiple_of`. JSON Schema's keyword is `multipleOf`. The step must be a positive integer, and a type takes one. Two steps on a NewType and its annotation merge to their least common multiple. A float is refused, because validators compare floats inexactly. Each constraint in the table now names the shapes it applies to, with the JSON Schema keyword for each, and a slot (below, above or step) in place of a side. `fixed_length_utf32`'s `length_bytes` and raw `r<N>` `bits` now declare `Gt(0), MultipleOf(8 or 4)` on their types. The two rules that checked only this are gone. A test whose own `r*` definition accepts any size now uses its own configuration type. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(zarr-metadata): read MinLen and MaxLen on strings in the TypedDict checker `MinLen`, `MaxLen` and `Len` now also bound a string, as pydantic's `min_length` does. The JSON Schema writes `minLength` and `maxLength`. Python and JSON Schema both count code points, so the checker and the schema agree on what a string's length is. Messages count characters: "expected a string with at least 1 character, got \"\"". A length bound on an object still raises `TypeError`. The v3 `struct` field name now declares `MinLen(1)` on its type, and the struct rule no longer checks for an empty name. The problem keeps its `loc` and kind. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(zarr-metadata): number the changelog fragments by this PR Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(zarr-metadata): loosen the floor on documents with an axis past 32 bits The derandomized share stream changes with any change to what the strategies draw, and with machine load: base main measures 0.103 against a floor of 0.1 under load, and the rectilinear fix measures 0.047 to 0.133 across runs. Over seeds 0-5 at 500 examples both trees average about 0.17, from 0.07 to 0.29. A floor of 0.03 still asks that a few hundred examples meet several such axes. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(zarr-metadata): hold a length bound beside an unknown key `constrained` skipped a type's constraints whenever the inner parse reported any problem. An `unknown_key` problem leaves the value readable, because the key is dropped and the value is still returned. So `check` returned an array of closed TypedDicts longer than its `MaxLen`, and a definition's rules ran on it, although the rules run only on a configuration within its bounds. The constraints are now skipped only for a problem other than an unknown key. Number bounds were never affected, because a number holds no keys. The docs now list `multiple_of`, `min_length` and `max_length` among the `ctx` keys, and `constraints_of` says a type takes one step too. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(zarr-metadata): name the member a carried name's type check refuses A raw `r<N>` name's size is now checked by its type. For `"data_type": "r12"` that gave "expected an integer > 0 that is a multiple of 8, got 12", about a string the document wrote, with no mention of bits. A type-check problem with a configuration that a name encodes now names the member and the name: `bits in "r12": ...`. A rule's message is left as written, because such rules already describe the name, as the v2 typestr rules do. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(zarr-metadata): pin that the types refuse empty pipelines and edge lists The sharding tests for an empty `codecs` or `index_codecs` checked only the `loc` and kind, which the pipeline's order check also reports. They now also assert `ctx={"min_length": 1}`, which only the type check gives. The rectilinear strategy test checked the grid's rules, which no longer look at empty lists. It now reads the configuration with the definition, which checks the type. The changelog no longer says the JSON Schema of a raw `r<N>` name carries its multiple of 8: that schema is the name pattern. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…arr-developers#4635) * fix(zarr-metadata): draw the same codecs on every derandomized run `_codec` drew from `st.sampled_from` over `(definition, configurations)` pairs whose configurations are strategies built during the draw. Hypothesis labels such a `sampled_from` by hashing its elements, and a strategy hashes by address, so these spans were labeled by memory layout. Hypothesis's mutator copies choices between spans that share a label, so which spans it grouped, and so which examples a derandomized run drew, changed with whatever moved the allocator: another process, machine load. `_codec` now samples an index into the options, labeled by a `range`. The share tests measured this drift. `examples_of` now draws twice and fails if the runs differ, which catches an address-labeled strategy in the same process; the array-document targets share one draw. Bounds are reset from the shares of 100 seeds: the "no axes" ceiling (0.25) sat under the seed maximum (0.303), and the "axis past 32 bits" floor (0.1) over the seed minimum (0.06). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(zarr-metadata): add changelog fragment for zarr-developers#4635 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(zarr-metadata): sample codec names and make share bounds reproducible `_codec` samples definition names rather than an index. An index into options of the same length shares one label with every other, so the mutator could copy a choice between unrelated codec slots; names key the label on the option set, as the chunk grid and data type draws already do. `examples_of` is cached per strategy, so the share tests that count one strategy draw it once, and its docstring calls the second-run check what it is, a tripwire. `across_seeds` and `bounds` put the rule the bounds follow into code, so a bound can be reset rather than re-derived; every bound is reset by it. The two first-data-type checks run over 1000 examples: over 300, `bool`'s share across seeds reached 0.13 to 0.15, too near the 0.2 they guard against. The changelog fragment no longer says the options' strategies are built during the draw: some are cached. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Bumps the uv group with 1 update in the /packages/zarr-http-server directory: [tornado](https://github.com/tornadoweb/tornado). Updates `tornado` from 6.6 to 6.6.1 - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](tornadoweb/tornado@v6.6.0...v6.6.1) --- updated-dependencies: - dependency-name: tornado dependency-version: 6.6.1 dependency-type: indirect dependency-group: uv ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [joserfc](https://github.com/authlib/joserfc) from 1.6.8 to 1.7.6. - [Release notes](https://github.com/authlib/joserfc/releases) - [Changelog](https://github.com/authlib/joserfc/blob/main/docs/changelog.rst) - [Commits](authlib/joserfc@1.6.8...1.7.6) --- updated-dependencies: - dependency-name: joserfc dependency-version: 1.7.6 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…s and narrowing helpers Add `ZarrV3NodeType` and `ZARR_V3_NODE_TYPE`; the default separators `ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR`, `DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR` and `V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR`; and the `TypeIs` helpers `is_zarr_v2_array_order`, `is_zarr_v2_array_dimension_separator` and `is_zarr_v3_node_type`. The v2 array reader, the v3 node reader and the chunk key encodings' canonicalization now use them in place of their own literals, which removes two casts from the v2 array reader. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
Owner
Author
|
🤖 AI text below 🤖 Superseded by zarr-developers#4638, which is opened from the same branch. The agent closed this fork PR and kept the branch, because the upstream PR uses it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Adds
ZarrV3NodeTypeandZARR_V3_NODE_TYPE, and constants for the separators the spec assumes when one is absent (ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR,DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR,V2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR). AddsTypeIshelpers that narrow anobjectto a spec value set:is_zarr_v2_array_order,is_zarr_v2_array_dimension_separatorandis_zarr_v3_node_type.Why
pydantic-zarr hardcodes these spec values or re-derives them (
get_args(MemoryOrder)followed bycast,node_type == "array" or node_type == "group", a hardcoded"."default). It should import them from zarr-metadata instead.The agent checked whether a tuple constant alone is enough, using
x in ZARR_V2_ARRAY_ORDERwithx: object:xto the Literal; mypy and ty do not.is_one_of(value, allowed: tuple[T, ...]) -> TypeIs[T]widensTtostrunder pyright and mypy.So each value set gets its own helper.
What changed
zarr_metadata.v2.array:ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR,is_zarr_v2_array_order,is_zarr_v2_array_dimension_separator.zarr_metadata.v3(home: the private modulev3/_common.py):ZarrV3NodeType(an alias of the format-neutral_hierarchy.NodeType),ZARR_V3_NODE_TYPE,is_zarr_v3_node_type.zarr_metadata.v3.chunk_key_encoding.{default,v2}:DEFAULT_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR("/") andV2_CHUNK_KEY_ENCODING_DEFAULT_SEPARATOR("."), each citing the spec line.zarr_metadataexports the constants and types. The helpers stay in their home modules, likeis_node_name_v3.("C", "F"),(".", "/"), its"."default and twocastcalls._NODE_TYPESand now returnsZarrV3NodeType | Noneinstead ofstr | None._canonicalcompares against its default constant.Considered and left out
FloatSpecialFillValue/FLOAT_SPECIAL_FILL_VALUESexport. It would need a single public home, and putting it in the float modules makes it value-tied withFloat{16,32,64}SpecialFillValue. Those per-width aliases are already public.ZarrV3GroupMetadataJSON.__required_keys__ | __optional_keys__already gives them at runtime.Tests
tests/test_spec_values.py: one parametrized test of the three helpers over members and non-members; one test that the v2 array model reads an absentdimension_separatoras the default; one test that each chunk key encoding's simplest spelling drops its default separator.test_public_api.py: the new front-door names, and theZARR_V3_NODE_TYPE/ZarrV3NodeTypedrift pair.mkdocs build --strictprinted no warnings.🤖 Generated with Claude Code