Repository navigation
Conversation
…the chunk it receives A definition's `canonical` reduces two spellings of a configuration to one when they mean the same thing wherever the declaration is used. It receives the configuration alone, so it cannot reduce spellings that mean the same thing only for some chunks. The `endian` of a `bytes` codec is one: it means nothing when each value takes one byte, and is required when a value takes several. `ArrayCodecDefinition` gains `chunk_canonical`, a function of the configuration, the nested declarations and the chunk, and `codec_for_chunk` returns the codec of a `Stage` in that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. The result is a declaration, so `declaration_key`, `canonical_of` and `refines` work on it as on any other. Its configuration is checked again: the definition must allow it, the chunk rules must find no problem with it, and each nested declaration must stay as it is. `bytes` declares a `chunk_canonical` that drops `endian` for a data type stored in single bytes, and a v3 array model compares, hashes and refines by its codecs as `codec_for_chunk` returns them. So an `int8` array is one array with either `endian` or with none, as zarr-python 3.0 and 3.1 write it, while a `bytes` codec after a `cast_value` to a multi-byte data type is still compared by its `endian`. Assisted-by: ClaudeCode:claude-fable-5-2
…est equivalent spelling Two models are equal when their documents mean the same thing, and `to_json` writes each document as it was given, so equal models can write different documents. A consumer that needs one spelling for all of them, to compare documents field by field or to hash them, had to normalize each member itself. Each model gains `canonicalize`: the v3 and v2 array, group and consolidated metadata models. It returns the model read from its document in its simplest equivalent spelling, written from the parts that `==` compares: each declaration as `canonical_of` writes it, each v3 codec as `codec_for_chunk` returns it for the chunk it receives, and the fill value in the canonical spelling of its data type. Empty `attributes`, empty v3 `storage_transformers` and a v2 `dimension_separator` of "." are left out, since a document without them means the same. Everything else is kept as written. The result equals the model, and equal models in one scope give the same document apart from the order of the members of an object, which a `JSONWriter(sort_keys=True)` sorts. A group keeps the documents its `consolidated_metadata` lists as written, since it does not read them; the consolidated metadata models canonicalize each node they list. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
…data-canonical-documents
Assisted-by: ClaudeCode:claude-fable-5-2
Documentation build overview
6 files changed ·
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4642 +/- ##
=======================================
Coverage 94.90% 94.90%
=======================================
Files 96 96
Lines 14055 14055
=======================================
Hits 13339 13339
Misses 716 716 🚀 New features to boost your workflow:
|
…canonical-documents Main now carries the chunk-aware canonical hook (zarr-developers#4640) and the nested declaration order fix (zarr-developers#4641). The one conflict was in model/_array_v3.py, where this branch adds canonical_array_document_v3 next to codecs_for_chunks; both are kept. Assisted-by: ClaudeCode:claude-fable-5-2
…at are not the default A review found that the Hypothesis tests passed with a canonicalize that returned the model's own document, that no test noticed a model losing its scope, and that four behaviours had no case: a claimed extra field, a v2 fill value, a chunk grid and a storage transformer that are respelled. The parametrized cases are now read in scopes that are not the default ones. The generated documents are compared with the same document reordered at every depth and, for arrays, with each declaration in its own simplest spelling. The reordering helper reverses every object, which the fix in zarr-developers#4641 allows. Assisted-by: ClaudeCode:claude-fable-5-2
…g claims Its configuration is kept, but its envelope is written in the shortest form: an empty configuration and a must_understand of true are left out. Assisted-by: ClaudeCode:claude-fable-5-2
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Each model has a
canonicalizemethod that returns the model with its document in its simplest equivalent spelling:ZarrV3ArrayMetadata,ZarrV3GroupMetadata,ZarrV3ConsolidatedMetadata,ZarrV2ArrayMetadata,ZarrV2GroupMetadataandZarrV2ConsolidatedMetadata. The result equals the model. Every model equal to it, in the same scope, gives the same document, apart from the order of the members of an object, soto_key_valuewithJSONWriter(sort_keys=True)writes the same bytes for equal models. The document is written from the parts==compares: each declaration ascanonical_ofwrites it, each v3 codec ascodec_for_chunkreturns it for the chunk it receives, and the fill value ascanonical_fill_valuewrites it. Emptyattributes, empty v3storage_transformersand a v2dimension_separatorof"."are left out, and everything else is kept as written.Why
The models already say when two documents mean the same thing, but
to_jsonwrites each as given, so "equal models may still write different documents". A consumer that needs one spelling had to rebuild it. pydantic-zarr does, inArraySpec.canonicalize_codecsandcanonicalize_fill_value, so that specs compare equal field by field, and its copy of the codec rules drifted from this package's (fixed in zarr-developers/pydantic-zarr#203). With this method it can ask forZarrV3ArrayMetadata.from_json(document).canonicalize().to_json(). A canonical document also gives a content key for metadata, which nothing provided.Design notes
to_json.canonicalizereturns a model, soto_json,to_key_valueand every typed view work on the result, and the canonical document is read again, which checks it as every document is checked.==compares. Each model's canonical document is assembled from the same parts as its key (array_key_offor the v3 array), so the two cannot disagree about which spellings are one.JSONWriter(sort_keys=True)gives the byte-level normal form.==: emptyattributes, emptystorage_transformers, a v2dimension_separatorof".".dimension_namesof nulls and v2filters: []are kept, because==tells them from an absent or null member.consolidated_metadatalists, and compares them as written, so itscanonicalizekeeps them as written.ZarrV3ConsolidatedMetadata.canonicalizecanonicalizes each node, andgroup.replace(consolidated_metadata=...)writes the result back.ZarrV2ConsolidatedMetadata.canonicalizerewrites each node's.zarrayor.zgroupentry, drops an empty.zattrsentry of a node, and keeps every other entry.canonicalwrites each numcodecs default a codec leaves out, so a canonical v2 document can be longer than its source. That is existing behaviour ofcanonical_of, and the guide says so.configurationor amust_understandoftrue, which is how==already compares it. Abytescodec after one keeps itsendian, since the chunk it receives is unknown.spellings="canonical"still means each declaration in the spellingcanonicalizegives it. A document drawn that way is not always a fixed point of the model'scanonicalize(it can hold emptyattributes, or anendianon single-byte values).Tests and checks
tests/model/test_canonicalize.py:canonicalizechanges nothing;array_documents(both spellings),group_documentsandconsolidated_group_documents: every drawn model equals its canonical model, which keeps the scope, is a fixed point, holds no emptyattributesorstorage_transformers, survives a round trip through JSON text, and is the canonical model of the same document with every object reordered;docs/models.mdgains a "Canonical documents" section, whose examples run in the docs tests.just test-versions),ruff0.17.3,pyright,mkdocs build --strict, the changelog check and the pre-commit hooks.Stack bookkeeping
codec_for_chunkand has merged.mainis merged into this branch (a merge commit, no rebase), so the diff shows this PR's changes alone._for_chunk, which merged with feat(zarr-metadata): compare each codec in its simplest spelling for the chunk it receives #4640: a custom codec whosecanonicaloutput itschunk_canonicalreduces further has no fixed point, socanonicalizeis not idempotent for it. The fix is a separate change, fix(zarr-metadata): simplify a codec for its chunk in its canonical spelling, and check the result d-v-b/zarr-python#462, not yet opened upstream.🤖 Generated with Claude Code