Repository navigation
Conversation
…the chunk it receives A definition's `canonical` reduces two spellings of a configuration to one when they mean the same thing wherever the declaration is used. It receives the configuration alone, so it cannot reduce spellings that mean the same thing only for some chunks. The `endian` of a `bytes` codec is one: it means nothing when each value takes one byte, and is required when a value takes several. `ArrayCodecDefinition` gains `chunk_canonical`, a function of the configuration, the nested declarations and the chunk, and `codec_for_chunk` returns the codec of a `Stage` in that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. The result is a declaration, so `declaration_key`, `canonical_of` and `refines` work on it as on any other. Its configuration is checked again: the definition must allow it, the chunk rules must find no problem with it, and each nested declaration must stay as it is. `bytes` declares a `chunk_canonical` that drops `endian` for a data type stored in single bytes, and a v3 array model compares, hashes and refines by its codecs as `codec_for_chunk` returns them. So an `int8` array is one array with either `endian` or with none, as zarr-python 3.0 and 3.1 write it, while a `bytes` codec after a `cast_value` to a multi-byte data type is still compared by its `endian`. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
Owner
Author
|
🤖 AI text below 🤖 Closed by the agent in favour of zarr-developers#4640, which is the same branch opened upstream. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
A v3 codec that receives an array can declare its simplest spelling for the chunk it receives, with
chunk_canonicalonArrayCodecDefinition. It is a function of the configuration, the nested declarations and theChunk, for two spellings that mean the same thing for one chunk and not for another, whichcanonicalcannot tell from the configuration alone.codec_for_chunkinzarr_metadata.v3.definitionreturns the codec of aStagein that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. Thebytescodec declares one: it drops anendianwhen each value takes one byte. So twoZarrV3ArrayMetadatamodels that differ only in theendianof abytescodec that receives values of one byte (bool,int8,uint8orr8) are now equal, hash alike and refine each other. Abytescodec that receives values of several bytes, as it does after acast_valuetofloat32, is compared by itsendianas before.Why
pydantic-zarr drops the no-op
endianof abytescodec itself, so that a spec built by hand equals one read back from zarr-python (3.0 wrote theendianfor single-byte data types, 3.1 does not). To do that it re-derived the data type eachbytescodec receives, and got it wrong after acast_value(d-v-b/pydantic-zarr#21). zarr-metadata already tracks that chunk for its rules, but a definition'scanonicalreceives the configuration alone, so the equivalence could not be declared where the codec is defined. The v2 models already treat<i1and|i1as one dtype, and v3 blosc drops atypesizethatnoshuffleignores; this is the same kind of equivalence, with the chunk as an input.Design notes
codec_for_chunk(stage)returns a declaration, not JSON. Each codec of a pipeline the codec contains is simplified first, for the chunk that one receives, and then the codec's own configuration bychunk_canonical. Because the result is an ordinaryAcceptedDeclaration, the existingdeclaration_key,canonical_ofandrefinesapply to it unchanged, and nothing else had to learn about chunks. When nothing is simplified the codec itself is returned.chunk_canonicalruns beforecanonical, which then simplifies the result as it does any configuration.chunk_canonicalis checked, as the result ofcanonicalis: the definition must allow it, the chunk rules must find no problem with it, and each nested declaration must stay as it is. A failure raisesValueError, a bug in the definition.to_jsonstill writes each codec as given, andAcceptedDeclaration.__eq__on a codec alone is unchanged: only the array model, which knows the chunk, compares codecs this way.definition/_definition.py:respelledbuilds the declaration another spelling reads as, anditem_atreads a location in a configuration.Tests and checks
tests/v3/test_pipelines.py: one parametrized test of the simplest spelling per chunk (single-byte and multi-byte data types, after a cast, inside shards and nested shards, index codecs, unknown data types and unknown chunks), one test that a codec with a problem is left alone, and one test per error case of a faultychunk_canonical.tests/model/test_array.py:==,hashandrefinesof array models, for pairs that are and are not one array.just test-versions),ruff0.17.3,pyright,mkdocs build --strict, the changelog check and the pre-commit hooks.Stack bookkeeping
main, which matches upstreammainat 95ec04a (zarr-metadata 0.9.0).canonicalizemethod on each model that returns the document in its simplest equivalent spelling, which usescodec_for_chunkfor the codecs.🤖 Generated with Claude Code