Skip to content

feat(zarr-metadata): compare each codec in its simplest spelling for the chunk it receives - #4640

Merged
d-v-b merged 2 commits into
zarr-developers:mainfrom
d-v-b:feat/zarr-metadata-chunk-canonical
Nov 17, 2026
Merged

d-v-b merged 2 commits into
zarr-developers:mainfrom
d-v-b:feat/zarr-metadata-chunk-canonical

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

This PR handles the non-ambiguity between the various ways semantically identical bytes codices can be configured in the codec pipeline.

🤖 AI text below 🤖

A v3 codec that receives an array can declare its simplest spelling for the chunk it receives, with chunk_canonical on ArrayCodecDefinition. It is a function of the configuration, the nested declarations and the Chunk, for two spellings that mean the same thing for one chunk and not for another, which canonical cannot tell from the configuration alone. codec_for_chunk in zarr_metadata.v3.definition returns the codec of a Stage in that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. The bytes codec declares one: it drops an endian when each value takes one byte. So two ZarrV3ArrayMetadata models that differ only in the endian of a bytes codec that receives values of one byte (bool, int8, uint8 or r8) are now equal, hash alike and refine each other. A bytes codec that receives values of several bytes, as it does after a cast_value to float32, is compared by its endian as before.

Why

pydantic-zarr drops the no-op endian of a bytes codec itself, so that a spec built by hand equals one read back from zarr-python (3.0 wrote the endian for single-byte data types, 3.1 does not). To do that it re-derived the data type each bytes codec receives, and got it wrong after a cast_value (fixed in zarr-developers/pydantic-zarr#203). zarr-metadata already tracks that chunk for its rules, but a definition's canonical receives the configuration alone, so the equivalence could not be declared where the codec is defined. The v2 models already treat <i1 and |i1 as one dtype, and v3 blosc drops a typesize that noshuffle ignores; this is the same kind of equivalence, with the chunk as an input.

Design notes
  • codec_for_chunk(stage) returns a declaration, not JSON. Each codec of a pipeline the codec contains is simplified first, for the chunk that one receives, and then the codec's own configuration by chunk_canonical. Because the result is an ordinary AcceptedDeclaration, the existing declaration_key, canonical_of and refines apply to it unchanged, and nothing else had to learn about chunks. When nothing is simplified the codec itself is returned.
  • chunk_canonical runs before canonical, which then simplifies the result as it does any configuration.
  • The result of chunk_canonical is checked, as the result of canonical is: the definition must allow it, the chunk rules must find no problem with it, and each nested declaration must stay as it is. A failure raises ValueError, a bug in the definition.
  • A codec the chunk rules find a problem with, a codec that receives bytes, and a codec nothing claims are left as they are. A chunk whose data type is unknown simplifies nothing.
  • to_json still writes each codec as given, and AcceptedDeclaration.__eq__ on a codec alone is unchanged: only the array model, which knows the chunk, compares codecs this way.
  • New internal helpers in definition/_definition.py: respelled builds the declaration another spelling reads as, and item_at reads a location in a configuration.
Tests and checks
  • tests/v3/test_pipelines.py: one parametrized test of the simplest spelling per chunk (single-byte and multi-byte data types, after a cast, inside shards and nested shards, index codecs, unknown data types and unknown chunks), one test that a codec with a problem is left alone, and one test per error case of a faulty chunk_canonical.
  • tests/model/test_array.py: ==, hash and refines of array models, for pairs that are and are not one array.
  • Run locally: the package suite on Python 3.11 and 3.14 (just test-versions), ruff 0.17.3, pyright, mkdocs build --strict, the changelog check and the pre-commit hooks.
Stack bookkeeping

🤖 Generated with Claude Code

…the chunk it receives

A definition's `canonical` reduces two spellings of a configuration to
one when they mean the same thing wherever the declaration is used. It
receives the configuration alone, so it cannot reduce spellings that
mean the same thing only for some chunks. The `endian` of a `bytes`
codec is one: it means nothing when each value takes one byte, and is
required when a value takes several.

`ArrayCodecDefinition` gains `chunk_canonical`, a function of the
configuration, the nested declarations and the chunk, and
`codec_for_chunk` returns the codec of a `Stage` in that spelling, with
each codec of the pipelines it contains simplified for the chunk that
one receives. The result is a declaration, so `declaration_key`,
`canonical_of` and `refines` work on it as on any other. Its
configuration is checked again: the definition must allow it, the chunk
rules must find no problem with it, and each nested declaration must
stay as it is.

`bytes` declares a `chunk_canonical` that drops `endian` for a data type
stored in single bytes, and a v3 array model compares, hashes and
refines by its codecs as `codec_for_chunk` returns them. So an `int8`
array is one array with either `endian` or with none, as zarr-python 3.0
and 3.1 write it, while a `bytes` codec after a `cast_value` to a
multi-byte data type is still compared by its `endian`.

Assisted-by: ClaudeCode:claude-fable-5-2
@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 zarr-metadata | 🛠️ Build #35931045 | 📁 Comparing 070840f against latest (95ec04a)

  🔍 Preview build  

6 files changed · ± 6 modified

± Modified

@d-v-b
d-v-b marked this pull request as ready for review November 17, 2026 15:58
@d-v-b
d-v-b merged commit 9831e99 into zarr-developers:main Nov 17, 2026
45 checks passed
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Nov 17, 2026
…canonical-documents

Main now carries the chunk-aware canonical hook (zarr-developers#4640) and the nested declaration order fix (zarr-developers#4641). The one conflict was in model/_array_v3.py, where this branch adds canonical_array_document_v3 next to codecs_for_chunks; both are kept.

Assisted-by: ClaudeCode:claude-fable-5-2
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Nov 17, 2026
… document it becomes

A review of zarr-developers#4640 found gaps in how _for_chunk checks the result of chunk_canonical:

- a result that added a nested declaration passed, and the respelled codec then held a declaration its nested mapping did not;
- a result that was not JSON raised an error that did not name the definition;
- an array written as a list was reported as a changed nested declaration;
- a problem inside a nested declaration, which is the document's, was held against the result.

The result is now refined as any JSON is, its declaration locations are compared with the codec's, and problems under a nested location are left out. respelled refuses a location that holds no nested declaration with the TypeError its docstring promises.

Assisted-by: ClaudeCode:claude-fable-5-2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant