Skip to content

feat(zarr-metadata): compare each codec in its simplest spelling for the chunk it receives - #450

Closed
d-v-b wants to merge 2 commits into
mainfrom
feat/zarr-metadata-chunk-canonical
Closed

d-v-b wants to merge 2 commits into
mainfrom
feat/zarr-metadata-chunk-canonical

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

A v3 codec that receives an array can declare its simplest spelling for the chunk it receives, with chunk_canonical on ArrayCodecDefinition. It is a function of the configuration, the nested declarations and the Chunk, for two spellings that mean the same thing for one chunk and not for another, which canonical cannot tell from the configuration alone. codec_for_chunk in zarr_metadata.v3.definition returns the codec of a Stage in that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. The bytes codec declares one: it drops an endian when each value takes one byte. So two ZarrV3ArrayMetadata models that differ only in the endian of a bytes codec that receives values of one byte (bool, int8, uint8 or r8) are now equal, hash alike and refine each other. A bytes codec that receives values of several bytes, as it does after a cast_value to float32, is compared by its endian as before.

Why

pydantic-zarr drops the no-op endian of a bytes codec itself, so that a spec built by hand equals one read back from zarr-python (3.0 wrote the endian for single-byte data types, 3.1 does not). To do that it re-derived the data type each bytes codec receives, and got it wrong after a cast_value (d-v-b/pydantic-zarr#21). zarr-metadata already tracks that chunk for its rules, but a definition's canonical receives the configuration alone, so the equivalence could not be declared where the codec is defined. The v2 models already treat <i1 and |i1 as one dtype, and v3 blosc drops a typesize that noshuffle ignores; this is the same kind of equivalence, with the chunk as an input.

Design notes
  • codec_for_chunk(stage) returns a declaration, not JSON. Each codec of a pipeline the codec contains is simplified first, for the chunk that one receives, and then the codec's own configuration by chunk_canonical. Because the result is an ordinary AcceptedDeclaration, the existing declaration_key, canonical_of and refines apply to it unchanged, and nothing else had to learn about chunks. When nothing is simplified the codec itself is returned.
  • chunk_canonical runs before canonical, which then simplifies the result as it does any configuration.
  • The result of chunk_canonical is checked, as the result of canonical is: the definition must allow it, the chunk rules must find no problem with it, and each nested declaration must stay as it is. A failure raises ValueError, a bug in the definition.
  • A codec the chunk rules find a problem with, a codec that receives bytes, and a codec nothing claims are left as they are. A chunk whose data type is unknown simplifies nothing.
  • to_json still writes each codec as given, and AcceptedDeclaration.__eq__ on a codec alone is unchanged: only the array model, which knows the chunk, compares codecs this way.
  • New internal helpers in definition/_definition.py: respelled builds the declaration another spelling reads as, and item_at reads a location in a configuration.
Tests and checks
  • tests/v3/test_pipelines.py: one parametrized test of the simplest spelling per chunk (single-byte and multi-byte data types, after a cast, inside shards and nested shards, index codecs, unknown data types and unknown chunks), one test that a codec with a problem is left alone, and one test per error case of a faulty chunk_canonical.
  • tests/model/test_array.py: ==, hash and refines of array models, for pairs that are and are not one array.
  • Run locally: the package suite on Python 3.11 and 3.14 (just test-versions), ruff 0.17.3, pyright, mkdocs build --strict, the changelog check and the pre-commit hooks.
Stack bookkeeping

🤖 Generated with Claude Code

…the chunk it receives

A definition's `canonical` reduces two spellings of a configuration to
one when they mean the same thing wherever the declaration is used. It
receives the configuration alone, so it cannot reduce spellings that
mean the same thing only for some chunks. The `endian` of a `bytes`
codec is one: it means nothing when each value takes one byte, and is
required when a value takes several.

`ArrayCodecDefinition` gains `chunk_canonical`, a function of the
configuration, the nested declarations and the chunk, and
`codec_for_chunk` returns the codec of a `Stage` in that spelling, with
each codec of the pipelines it contains simplified for the chunk that
one receives. The result is a declaration, so `declaration_key`,
`canonical_of` and `refines` work on it as on any other. Its
configuration is checked again: the definition must allow it, the chunk
rules must find no problem with it, and each nested declaration must
stay as it is.

`bytes` declares a `chunk_canonical` that drops `endian` for a data type
stored in single bytes, and a v3 array model compares, hashes and
refines by its codecs as `codec_for_chunk` returns them. So an `int8`
array is one array with either `endian` or with none, as zarr-python 3.0
and 3.1 write it, while a `bytes` codec after a `cast_value` to a
multi-byte data type is still compared by its `endian`.

Assisted-by: ClaudeCode:claude-fable-5-2
@d-v-b

d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Closed by the agent in favour of zarr-developers#4640, which is the same branch opened upstream.

@d-v-b d-v-b closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant