Skip to content

feat(zarr-metadata): return each model with its document in its simplest equivalent spelling - #452

Closed
d-v-b wants to merge 3 commits into
feat/zarr-metadata-chunk-canonicalfrom
feat/zarr-metadata-canonical-documents
Closed

d-v-b wants to merge 3 commits into
feat/zarr-metadata-chunk-canonicalfrom
feat/zarr-metadata-canonical-documents

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Each model has a canonicalize method that returns the model with its document in its simplest equivalent spelling: ZarrV3ArrayMetadata, ZarrV3GroupMetadata, ZarrV3ConsolidatedMetadata, ZarrV2ArrayMetadata, ZarrV2GroupMetadata and ZarrV2ConsolidatedMetadata. The result equals the model. Every model equal to it, in the same scope, gives the same document, apart from the order of the members of an object, so to_key_value with JSONWriter(sort_keys=True) writes the same bytes for equal models. The document is written from the parts == compares: each declaration as canonical_of writes it, each v3 codec as codec_for_chunk returns it for the chunk it receives, and the fill value as canonical_fill_value writes it. Empty attributes, empty v3 storage_transformers and a v2 dimension_separator of "." are left out, and everything else is kept as written.

Why

The models already say when two documents mean the same thing, but to_json writes each as given, so "equal models may still write different documents". A consumer that needs one spelling had to rebuild it. pydantic-zarr does, in ArraySpec.canonicalize_codecs and canonicalize_fill_value, so that specs compare equal field by field, and its copy of the codec rules drifted from this package's (d-v-b/pydantic-zarr#21). With this method it can ask for ZarrV3ArrayMetadata.from_json(document).canonicalize().to_json(). A canonical document also gives a content key for metadata, which nothing provided.

Design notes
  • A model, not a flag on to_json. canonicalize returns a model, so to_json, to_key_value and every typed view work on the result, and the canonical document is read again, which checks it as every document is checked.
  • Written from what == compares. Each model's canonical document is assembled from the same parts as its key (array_key_of for the v3 array), so the two cannot disagree about which spellings are one.
  • Member order is left to the writer. JSON gives the order of an object's members no meaning, and the models compare uninterpreted JSON as sorted text. The canonical document keeps the members that remain in the order they were written, and JSONWriter(sort_keys=True) gives the byte-level normal form.
  • What is left out. Only members whose absence means the same to ==: empty attributes, empty storage_transformers, a v2 dimension_separator of ".". dimension_names of nulls and v2 filters: [] are kept, because == tells them from an absent or null member.
  • Groups and consolidated metadata. A group does not read the documents its consolidated_metadata lists, and compares them as written, so its canonicalize keeps them as written. ZarrV3ConsolidatedMetadata.canonicalize canonicalizes each node, and group.replace(consolidated_metadata=...) writes the result back. ZarrV2ConsolidatedMetadata.canonicalize rewrites each node's .zarray or .zgroup entry, drops an empty .zattrs entry of a node, and keeps every other entry.
  • v2 codecs grow. The v2 definitions' canonical writes each numcodecs default a codec leaves out, so a canonical v2 document can be longer than its source. That is existing behaviour of canonical_of, and the guide says so.
  • Unclaimed names. The configuration of a declaration nothing in scope claims is kept as written. A bytes codec after one keeps its endian, since the chunk it receives is unknown.
  • The Hypothesis strategies' spellings="canonical" still means each declaration in the spelling canonicalize gives it. A document drawn that way is not always a fixed point of the model's canonicalize (it can hold empty attributes, or an endian on single-byte values).
Tests and checks
  • tests/model/test_canonicalize.py:
    • one parametrized test of document to canonical document across the six models, which also checks the class, the scope, equality, the hash and that a second canonicalize changes nothing;
    • one test that equal models give one document, and that a sorting writer writes the same bytes;
    • one test that unequal models keep different documents;
    • Hypothesis tests over array_documents (both spellings), group_documents and consolidated_group_documents: every drawn model equals its canonical model, which is a fixed point and survives a round trip through JSON text.
  • docs/models.md gains a "Canonical documents" section, whose examples run in the docs tests.
  • Run locally: the package suite on Python 3.11 and 3.14 (just test-versions), ruff 0.17.3, pyright, mkdocs build --strict, the changelog check and the pre-commit hooks.
Stack bookkeeping

🤖 Generated with Claude Code

…est equivalent spelling

Two models are equal when their documents mean the same thing, and
`to_json` writes each document as it was given, so equal models can
write different documents. A consumer that needs one spelling for all of
them, to compare documents field by field or to hash them, had to
normalize each member itself.

Each model gains `canonicalize`: the v3 and v2 array, group and
consolidated metadata models. It returns the model read from its
document in its simplest equivalent spelling, written from the parts
that `==` compares: each declaration as `canonical_of` writes it, each
v3 codec as `codec_for_chunk` returns it for the chunk it receives, and
the fill value in the canonical spelling of its data type. Empty
`attributes`, empty v3 `storage_transformers` and a v2
`dimension_separator` of "." are left out, since a document without them
means the same. Everything else is kept as written.

The result equals the model, and equal models in one scope give the
same document apart from the order of the members of an object, which a
`JSONWriter(sort_keys=True)` sorts. A group keeps the documents its
`consolidated_metadata` lists as written, since it does not read them;
the consolidated metadata models canonicalize each node they list.

Assisted-by: ClaudeCode:claude-fable-5-2
@d-v-b

d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Closed by the agent in favour of zarr-developers#4642, which is the same branch opened upstream.

@d-v-b d-v-b closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant