Repository navigation
feat(zarr-metadata): compare each codec in its simplest spelling for the chunk it receives - #4640
Merged
d-v-b merged 2 commits intoNov 17, 2026
Conversation
…the chunk it receives A definition's `canonical` reduces two spellings of a configuration to one when they mean the same thing wherever the declaration is used. It receives the configuration alone, so it cannot reduce spellings that mean the same thing only for some chunks. The `endian` of a `bytes` codec is one: it means nothing when each value takes one byte, and is required when a value takes several. `ArrayCodecDefinition` gains `chunk_canonical`, a function of the configuration, the nested declarations and the chunk, and `codec_for_chunk` returns the codec of a `Stage` in that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. The result is a declaration, so `declaration_key`, `canonical_of` and `refines` work on it as on any other. Its configuration is checked again: the definition must allow it, the chunk rules must find no problem with it, and each nested declaration must stay as it is. `bytes` declares a `chunk_canonical` that drops `endian` for a data type stored in single bytes, and a v3 array model compares, hashes and refines by its codecs as `codec_for_chunk` returns them. So an `int8` array is one array with either `endian` or with none, as zarr-python 3.0 and 3.1 write it, while a `bytes` codec after a `cast_value` to a multi-byte data type is still compared by its `endian`. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
Documentation build overview
6 files changed ·
|
d-v-b
marked this pull request as ready for review
November 17, 2026 15:58
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 17, 2026
…canonical-documents Main now carries the chunk-aware canonical hook (zarr-developers#4640) and the nested declaration order fix (zarr-developers#4641). The one conflict was in model/_array_v3.py, where this branch adds canonical_array_document_v3 next to codecs_for_chunks; both are kept. Assisted-by: ClaudeCode:claude-fable-5-2
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Nov 17, 2026
… document it becomes A review of zarr-developers#4640 found gaps in how _for_chunk checks the result of chunk_canonical: - a result that added a nested declaration passed, and the respelled codec then held a declaration its nested mapping did not; - a result that was not JSON raised an error that did not name the definition; - an array written as a list was reported as a changed nested declaration; - a problem inside a nested declaration, which is the document's, was held against the result. The result is now refined as any JSON is, its declaration locations are compared with the codec's, and problems under a nested location are left out. respelled refuses a location that holds no nested declaration with the TypeError its docstring promises. Assisted-by: ClaudeCode:claude-fable-5-2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR handles the non-ambiguity between the various ways semantically identical
bytescodices can be configured in the codec pipeline.🤖 AI text below 🤖
A v3 codec that receives an array can declare its simplest spelling for the chunk it receives, with
chunk_canonicalonArrayCodecDefinition. It is a function of the configuration, the nested declarations and theChunk, for two spellings that mean the same thing for one chunk and not for another, whichcanonicalcannot tell from the configuration alone.codec_for_chunkinzarr_metadata.v3.definitionreturns the codec of aStagein that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. Thebytescodec declares one: it drops anendianwhen each value takes one byte. So twoZarrV3ArrayMetadatamodels that differ only in theendianof abytescodec that receives values of one byte (bool,int8,uint8orr8) are now equal, hash alike and refine each other. Abytescodec that receives values of several bytes, as it does after acast_valuetofloat32, is compared by itsendianas before.Why
pydantic-zarr drops the no-op
endianof abytescodec itself, so that a spec built by hand equals one read back from zarr-python (3.0 wrote theendianfor single-byte data types, 3.1 does not). To do that it re-derived the data type eachbytescodec receives, and got it wrong after acast_value(fixed in zarr-developers/pydantic-zarr#203). zarr-metadata already tracks that chunk for its rules, but a definition'scanonicalreceives the configuration alone, so the equivalence could not be declared where the codec is defined. The v2 models already treat<i1and|i1as one dtype, and v3 blosc drops atypesizethatnoshuffleignores; this is the same kind of equivalence, with the chunk as an input.Design notes
codec_for_chunk(stage)returns a declaration, not JSON. Each codec of a pipeline the codec contains is simplified first, for the chunk that one receives, and then the codec's own configuration bychunk_canonical. Because the result is an ordinaryAcceptedDeclaration, the existingdeclaration_key,canonical_ofandrefinesapply to it unchanged, and nothing else had to learn about chunks. When nothing is simplified the codec itself is returned.chunk_canonicalruns beforecanonical, which then simplifies the result as it does any configuration.chunk_canonicalis checked, as the result ofcanonicalis: the definition must allow it, the chunk rules must find no problem with it, and each nested declaration must stay as it is. A failure raisesValueError, a bug in the definition.to_jsonstill writes each codec as given, andAcceptedDeclaration.__eq__on a codec alone is unchanged: only the array model, which knows the chunk, compares codecs this way.definition/_definition.py:respelledbuilds the declaration another spelling reads as, anditem_atreads a location in a configuration.Tests and checks
tests/v3/test_pipelines.py: one parametrized test of the simplest spelling per chunk (single-byte and multi-byte data types, after a cast, inside shards and nested shards, index codecs, unknown data types and unknown chunks), one test that a codec with a problem is left alone, and one test per error case of a faultychunk_canonical.tests/model/test_array.py:==,hashandrefinesof array models, for pairs that are and are not one array.just test-versions),ruff0.17.3,pyright,mkdocs build --strict, the changelog check and the pre-commit hooks.Stack bookkeeping
mainat 95ec04a (zarr-metadata 0.9.0).canonicalizemethod on each model, which usescodec_for_chunkfor the codecs.🤖 Generated with Claude Code