Skip to content

feat(zarr-metadata): return each model with its document in its simplest equivalent spelling - #4642

Draft
d-v-b wants to merge 8 commits into
zarr-developers:mainfrom
d-v-b:feat/zarr-metadata-canonical-documents
Draft

d-v-b wants to merge 8 commits into
zarr-developers:mainfrom
d-v-b:feat/zarr-metadata-canonical-documents

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

Each model has a canonicalize method that returns the model with its document in its simplest equivalent spelling: ZarrV3ArrayMetadata, ZarrV3GroupMetadata, ZarrV3ConsolidatedMetadata, ZarrV2ArrayMetadata, ZarrV2GroupMetadata and ZarrV2ConsolidatedMetadata. The result equals the model. Every model equal to it, in the same scope, gives the same document, apart from the order of the members of an object, so to_key_value with JSONWriter(sort_keys=True) writes the same bytes for equal models. The document is written from the parts == compares: each declaration as canonical_of writes it, each v3 codec as codec_for_chunk returns it for the chunk it receives, and the fill value as canonical_fill_value writes it. Empty attributes, empty v3 storage_transformers and a v2 dimension_separator of "." are left out, and everything else is kept as written.

Why

The models already say when two documents mean the same thing, but to_json writes each as given, so "equal models may still write different documents". A consumer that needs one spelling had to rebuild it. pydantic-zarr does, in ArraySpec.canonicalize_codecs and canonicalize_fill_value, so that specs compare equal field by field, and its copy of the codec rules drifted from this package's (fixed in zarr-developers/pydantic-zarr#203). With this method it can ask for ZarrV3ArrayMetadata.from_json(document).canonicalize().to_json(). A canonical document also gives a content key for metadata, which nothing provided.

Design notes
  • A model, not a flag on to_json. canonicalize returns a model, so to_json, to_key_value and every typed view work on the result, and the canonical document is read again, which checks it as every document is checked.
  • Written from what == compares. Each model's canonical document is assembled from the same parts as its key (array_key_of for the v3 array), so the two cannot disagree about which spellings are one.
  • Member order is left to the writer. JSON gives the order of an object's members no meaning, and the models compare uninterpreted JSON as sorted text. The canonical document keeps the members that remain in the order they were written, and JSONWriter(sort_keys=True) gives the byte-level normal form.
  • What is left out. Only members whose absence means the same to ==: empty attributes, empty storage_transformers, a v2 dimension_separator of ".". dimension_names of nulls and v2 filters: [] are kept, because == tells them from an absent or null member.
  • Groups and consolidated metadata. A group does not read the documents its consolidated_metadata lists, and compares them as written, so its canonicalize keeps them as written. ZarrV3ConsolidatedMetadata.canonicalize canonicalizes each node, and group.replace(consolidated_metadata=...) writes the result back. ZarrV2ConsolidatedMetadata.canonicalize rewrites each node's .zarray or .zgroup entry, drops an empty .zattrs entry of a node, and keeps every other entry.
  • v2 codecs grow. The v2 definitions' canonical writes each numcodecs default a codec leaves out, so a canonical v2 document can be longer than its source. That is existing behaviour of canonical_of, and the guide says so.
  • Unclaimed names. The configuration of a declaration nothing in scope claims is kept as written. Its envelope is written in the shortest form, without an empty configuration or a must_understand of true, which is how == already compares it. A bytes codec after one keeps its endian, since the chunk it receives is unknown.
  • The Hypothesis strategies' spellings="canonical" still means each declaration in the spelling canonicalize gives it. A document drawn that way is not always a fixed point of the model's canonicalize (it can hold empty attributes, or an endian on single-byte values).
Tests and checks
  • tests/model/test_canonicalize.py:
    • one parametrized test of document to canonical document across the six models, read in scopes that are not the default ones, which also checks the class, the scope, equality, the hash and that a second canonicalize changes nothing;
    • one test that equal models give one document, and that a sorting writer writes the same bytes;
    • one test that unequal models keep different documents;
    • Hypothesis tests over array_documents (both spellings), group_documents and consolidated_group_documents: every drawn model equals its canonical model, which keeps the scope, is a fixed point, holds no empty attributes or storage_transformers, survives a round trip through JSON text, and is the canonical model of the same document with every object reordered;
    • one Hypothesis test that an array document, and the same document with each declaration in its own simplest spelling, are one array with one canonical document.
  • docs/models.md gains a "Canonical documents" section, whose examples run in the docs tests.
  • Run locally: the package suite on Python 3.11 and 3.14 (just test-versions), ruff 0.17.3, pyright, mkdocs build --strict, the changelog check and the pre-commit hooks.
Stack bookkeeping

🤖 Generated with Claude Code

…the chunk it receives

A definition's `canonical` reduces two spellings of a configuration to
one when they mean the same thing wherever the declaration is used. It
receives the configuration alone, so it cannot reduce spellings that
mean the same thing only for some chunks. The `endian` of a `bytes`
codec is one: it means nothing when each value takes one byte, and is
required when a value takes several.

`ArrayCodecDefinition` gains `chunk_canonical`, a function of the
configuration, the nested declarations and the chunk, and
`codec_for_chunk` returns the codec of a `Stage` in that spelling, with
each codec of the pipelines it contains simplified for the chunk that
one receives. The result is a declaration, so `declaration_key`,
`canonical_of` and `refines` work on it as on any other. Its
configuration is checked again: the definition must allow it, the chunk
rules must find no problem with it, and each nested declaration must
stay as it is.

`bytes` declares a `chunk_canonical` that drops `endian` for a data type
stored in single bytes, and a v3 array model compares, hashes and
refines by its codecs as `codec_for_chunk` returns them. So an `int8`
array is one array with either `endian` or with none, as zarr-python 3.0
and 3.1 write it, while a `bytes` codec after a `cast_value` to a
multi-byte data type is still compared by its `endian`.

Assisted-by: ClaudeCode:claude-fable-5-2
…est equivalent spelling

Two models are equal when their documents mean the same thing, and
`to_json` writes each document as it was given, so equal models can
write different documents. A consumer that needs one spelling for all of
them, to compare documents field by field or to hash them, had to
normalize each member itself.

Each model gains `canonicalize`: the v3 and v2 array, group and
consolidated metadata models. It returns the model read from its
document in its simplest equivalent spelling, written from the parts
that `==` compares: each declaration as `canonical_of` writes it, each
v3 codec as `codec_for_chunk` returns it for the chunk it receives, and
the fill value in the canonical spelling of its data type. Empty
`attributes`, empty v3 `storage_transformers` and a v2
`dimension_separator` of "." are left out, since a document without them
means the same. Everything else is kept as written.

The result equals the model, and equal models in one scope give the
same document apart from the order of the members of an object, which a
`JSONWriter(sort_keys=True)` sorts. A group keeps the documents its
`consolidated_metadata` lists as written, since it does not read them;
the consolidated metadata models canonicalize each node they list.

Assisted-by: ClaudeCode:claude-fable-5-2
@github-actions github-actions Bot added needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package labels Nov 16, 2026
@read-the-docs-community

read-the-docs-community Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 zarr-metadata | 🛠️ Build #35968357 | 📁 Comparing 34c3378 against latest (95ec04a)

  🔍 Preview build  

6 files changed · ± 6 modified

± Modified

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.90%. Comparing base (b0020f2) to head (34c3378).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4642   +/-   ##
=======================================
  Coverage   94.90%   94.90%           
=======================================
  Files          96       96           
  Lines       14055    14055           
=======================================
  Hits        13339    13339           
  Misses        716      716           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…canonical-documents

Main now carries the chunk-aware canonical hook (zarr-developers#4640) and the nested declaration order fix (zarr-developers#4641). The one conflict was in model/_array_v3.py, where this branch adds canonical_array_document_v3 next to codecs_for_chunks; both are kept.

Assisted-by: ClaudeCode:claude-fable-5-2
…at are not the default

A review found that the Hypothesis tests passed with a canonicalize that returned the model's own document, that no test noticed a model losing its scope, and that four behaviours had no case: a claimed extra field, a v2 fill value, a chunk grid and a storage transformer that are respelled.

The parametrized cases are now read in scopes that are not the default ones. The generated documents are compared with the same document reordered at every depth and, for arrays, with each declaration in its own simplest spelling. The reordering helper reverses every object, which the fix in zarr-developers#4641 allows.

Assisted-by: ClaudeCode:claude-fable-5-2
…g claims

Its configuration is kept, but its envelope is written in the shortest form: an empty configuration and a must_understand of true are left out.

Assisted-by: ClaudeCode:claude-fable-5-2

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs release notes Automatically applied to PRs which haven't added release notes zarr-metadata Specific to the zarr-metadata sub-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant