Repository navigation
Conversation
…est equivalent spelling Two models are equal when their documents mean the same thing, and `to_json` writes each document as it was given, so equal models can write different documents. A consumer that needs one spelling for all of them, to compare documents field by field or to hash them, had to normalize each member itself. Each model gains `canonicalize`: the v3 and v2 array, group and consolidated metadata models. It returns the model read from its document in its simplest equivalent spelling, written from the parts that `==` compares: each declaration as `canonical_of` writes it, each v3 codec as `codec_for_chunk` returns it for the chunk it receives, and the fill value in the canonical spelling of its data type. Empty `attributes`, empty v3 `storage_transformers` and a v2 `dimension_separator` of "." are left out, since a document without them means the same. Everything else is kept as written. The result equals the model, and equal models in one scope give the same document apart from the order of the members of an object, which a `JSONWriter(sort_keys=True)` sorts. A group keeps the documents its `consolidated_metadata` lists as written, since it does not read them; the consolidated metadata models canonicalize each node they list. Assisted-by: ClaudeCode:claude-fable-5-2
…data-canonical-documents
Assisted-by: ClaudeCode:claude-fable-5-2
Owner
Author
|
🤖 AI text below 🤖 Closed by the agent in favour of zarr-developers#4642, which is the same branch opened upstream. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Each model has a
canonicalizemethod that returns the model with its document in its simplest equivalent spelling:ZarrV3ArrayMetadata,ZarrV3GroupMetadata,ZarrV3ConsolidatedMetadata,ZarrV2ArrayMetadata,ZarrV2GroupMetadataandZarrV2ConsolidatedMetadata. The result equals the model. Every model equal to it, in the same scope, gives the same document, apart from the order of the members of an object, soto_key_valuewithJSONWriter(sort_keys=True)writes the same bytes for equal models. The document is written from the parts==compares: each declaration ascanonical_ofwrites it, each v3 codec ascodec_for_chunkreturns it for the chunk it receives, and the fill value ascanonical_fill_valuewrites it. Emptyattributes, empty v3storage_transformersand a v2dimension_separatorof"."are left out, and everything else is kept as written.Why
The models already say when two documents mean the same thing, but
to_jsonwrites each as given, so "equal models may still write different documents". A consumer that needs one spelling had to rebuild it. pydantic-zarr does, inArraySpec.canonicalize_codecsandcanonicalize_fill_value, so that specs compare equal field by field, and its copy of the codec rules drifted from this package's (d-v-b/pydantic-zarr#21). With this method it can ask forZarrV3ArrayMetadata.from_json(document).canonicalize().to_json(). A canonical document also gives a content key for metadata, which nothing provided.Design notes
to_json.canonicalizereturns a model, soto_json,to_key_valueand every typed view work on the result, and the canonical document is read again, which checks it as every document is checked.==compares. Each model's canonical document is assembled from the same parts as its key (array_key_offor the v3 array), so the two cannot disagree about which spellings are one.JSONWriter(sort_keys=True)gives the byte-level normal form.==: emptyattributes, emptystorage_transformers, a v2dimension_separatorof".".dimension_namesof nulls and v2filters: []are kept, because==tells them from an absent or null member.consolidated_metadatalists, and compares them as written, so itscanonicalizekeeps them as written.ZarrV3ConsolidatedMetadata.canonicalizecanonicalizes each node, andgroup.replace(consolidated_metadata=...)writes the result back.ZarrV2ConsolidatedMetadata.canonicalizerewrites each node's.zarrayor.zgroupentry, drops an empty.zattrsentry of a node, and keeps every other entry.canonicalwrites each numcodecs default a codec leaves out, so a canonical v2 document can be longer than its source. That is existing behaviour ofcanonical_of, and the guide says so.bytescodec after one keeps itsendian, since the chunk it receives is unknown.spellings="canonical"still means each declaration in the spellingcanonicalizegives it. A document drawn that way is not always a fixed point of the model'scanonicalize(it can hold emptyattributes, or anendianon single-byte values).Tests and checks
tests/model/test_canonicalize.py:canonicalizechanges nothing;array_documents(both spellings),group_documentsandconsolidated_group_documents: every drawn model equals its canonical model, which is a fixed point and survives a round trip through JSON text.docs/models.mdgains a "Canonical documents" section, whose examples run in the docs tests.just test-versions),ruff0.17.3,pyright,mkdocs build --strict, the changelog check and the pre-commit hooks.Stack bookkeeping
feat/zarr-metadata-chunk-canonical), which addscodec_for_chunk. Merge that first, then retarget this tomain.🤖 Generated with Claude Code