Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/zarr-metadata/changes/4642.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Each model has a `canonicalize` method that returns the model with its document in its simplest equivalent spelling: `ZarrV3ArrayMetadata`, `ZarrV3GroupMetadata`, `ZarrV3ConsolidatedMetadata`, `ZarrV2ArrayMetadata`, `ZarrV2GroupMetadata` and `ZarrV2ConsolidatedMetadata`. The result equals the model. Every model equal to it, in the same scope, gives the same document, apart from the order of the members of an object, so `to_key_value` with `JSONWriter(sort_keys=True)` writes the same bytes for equal models. The document is written from the parts `==` compares: each declaration as `canonical_of` writes it, each v3 codec as `codec_for_chunk` returns it for the chunk it receives, and the fill value as `canonical_fill_value` writes it. Empty `attributes`, empty v3 `storage_transformers` and a v2 `dimension_separator` of `"."` are left out, and everything else is kept as written.
75 changes: 75 additions & 0 deletions packages/zarr-metadata/docs/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ To change a model, use one of these methods:
- `refined_in` also reads the document in another context. The new context
must claim what this context left unclaimed and contradict nothing.
Otherwise it raises `ScopeConflictError`.
- `canonicalize` returns the model with its document in its simplest
equivalent spelling. See [Canonical documents](#canonical-documents).

## Writing store bytes

Expand Down Expand Up @@ -264,3 +266,76 @@ equal to itself.

Equal models have equal hashes. They may still write different documents,
because `to_json` writes each document as it was given.

## Canonical documents

`canonicalize` returns a model with its document in its simplest
equivalent spelling. The result equals the model it came from. Every model
equal to that one, read in the same context, gives the same document.

```python test="true" session="models"
from zarr_metadata.model import ZarrV3ArrayMetadata

written = ZarrV3ArrayMetadata.from_json(
{
"zarr_format": 3,
"node_type": "array",
"shape": [4],
"data_type": "uint8",
"chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [4]}},
"chunk_key_encoding": {"name": "default", "configuration": {"separator": "/"}},
"fill_value": 0,
"codecs": [
{"name": "bytes", "configuration": {"endian": "little"}},
{"name": "zstd", "configuration": {"level": 1, "checksum": False}},
],
"attributes": {},
}
)
canonical = written.canonicalize()
assert canonical == written
print(canonical.to_json()["chunk_key_encoding"])
#> {'name': 'default'}
print(canonical.to_json()["codecs"])
#> ({'name': 'bytes'}, {'name': 'zstd', 'configuration': {'level': 1}})
print("attributes" in canonical.to_json())
#> False
```

The canonical document is written from the same parts that `==` compares:

- Each declaration is written as `canonical_of` writes it.
- Each v3 codec is written for the chunk it receives, as `codec_for_chunk`
returns it. Above, each `uint8` value takes one byte, so the `endian`
says nothing.
- The fill value is written as `canonical_fill_value` writes it for the
data type.
- A member that says what its absence says is left out. These are empty
`attributes`, empty v3 `storage_transformers`, and a v2
`dimension_separator` of `"."`.
- Everything else is kept as written. This includes the attributes, and
the configuration of a declaration that nothing in the context claims,
because only its own definition could simplify it.

A v2 codec is written with each numcodecs default it leaves out, as its
definition's `canonical` writes it. So a canonical v2 document can be
longer than the one it came from.

The order of the members of an object is not part of the spelling, because
JSON gives it no meaning. Two equal models can give canonical documents
whose members are in different orders. A `JSONWriter` that sorts them
writes the same bytes for both, so those bytes are a key for what the
document means:

```python test="true" session="models"
import hashlib

stored = canonical.to_key_value({"sort_keys": True})["zarr.json"]
digest = hashlib.sha256(stored).hexdigest()
```

The other models work the same way. A v3 group does not read the documents
its `consolidated_metadata` lists, so its `canonicalize` keeps them as
written. `ZarrV3ConsolidatedMetadata.canonicalize` and
`ZarrV2ConsolidatedMetadata.canonicalize` write each node they list in its
simplest spelling.
5 changes: 4 additions & 1 deletion packages/zarr-metadata/src/zarr_metadata/model/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,10 @@
scope and raises `MetadataValidationError` with every problem, so an
invalid model is never built. `to_json` returns the document as written.
`replace` reads new members in the model's own scope. `with_context` and
`refined_in` read the document in another scope. `to_key_value` writes
`refined_in` read the document in another scope. `canonicalize` returns
the model with its document in its simplest equivalent spelling, the one
that every equal model gives, apart from the order of the members of an
object. `to_key_value` writes
the model as it is, with a `JSONWriter`, or a mapping of its settings, deciding how
the bytes are spelled (indentation, key order, ASCII escaping, trailing
newline); the default is that of `json.dumps`. A v3 group reads its
Expand Down
35 changes: 35 additions & 0 deletions packages/zarr-metadata/src/zarr_metadata/model/_array_v2.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
Derived,
StoreKey,
attributes_of,
canonical_members,
check_literal,
created,
dimension_lengths,
Expand All @@ -59,6 +60,7 @@
missing_keys,
other_members,
reading_of,
simplest,
unclaimed_declarations_of,
)

Expand Down Expand Up @@ -772,6 +774,39 @@ def __replace__(self, /, **members: Unpack[ZarrV2ArrayMetadataUpdate]) -> ZarrV2
"""Return `self.replace(**members)`, which `copy.replace(model, **members)` (Python 3.13) calls. It calls `replace` rather than being it, so a subclass that overrides `replace` changes what `copy.replace` does too."""
return self.replace(**members)

def canonicalize(self) -> ZarrV2ArrayMetadata:
"""Return this model with its document in its simplest equivalent spelling, read in this model's own scope.

The result equals this model. Every model equal to this one, in
this scope, gives the same document: `to_json` of each result is
the same JSON, apart from the order of the members of an object,
which `to_key_value` sorts with `JSONWriter(sort_keys=True)`. So
the bytes that writer writes are the same for equal models, and
are a key for what the document means.

The dtype, the compressor and each filter are written as
`canonical_of` writes them, and the fill value as
`canonical_fill_value` writes it for the dtype. A
`dimension_separator` of `"."` is left out, and so are empty
`attributes`, because a document without them means the same.
Everything else is kept as written: `shape`, `chunks`, `order`,
the attributes, and whatever nothing in scope claims.
"""
compressor, filters = self.compressor, self.filters
spelled: dict[str, JSONValue] = {
"dtype": simplest(self.dtype),
"fill_value": spelled_canonically(self.dtype, self._members.fill_value),
"compressor": None if compressor is None else simplest(compressor),
"filters": None if filters is None else tuple(simplest(entry) for entry in filters),
}
document = canonical_members(
self._document,
spelled,
empty=("attributes",),
defaults={"dimension_separator": ZARR_V2_ARRAY_DEFAULT_DIMENSION_SEPARATOR},
)
return type(self)(document, context=self._context)

def with_context(self, context: ZarrV2Context | None = None) -> ZarrV2ArrayMetadata:
"""Return this document read in `context`, whatever that changes: a gain, a loss, or a conflict.

Expand Down
50 changes: 50 additions & 0 deletions packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
Derived,
StoreKey,
attributes_of,
canonical_members,
check_literal,
created,
dimension_lengths,
Expand All @@ -60,6 +61,8 @@
member_changes,
missing_keys,
reading_of,
simplest,
simplest_members,
unclaimed_declarations_of,
)
from zarr_metadata.v3._common import ZarrV3DeclarationJSON, ZarrV3ExtensionField
Expand Down Expand Up @@ -734,6 +737,29 @@ def __replace__(self, /, **members: Unpack[ZarrV3ArrayMetadataUpdate]) -> ZarrV3
"""Return `self.replace(**members)`, which `copy.replace(model, **members)` (Python 3.13) calls. It calls `replace` rather than being it, so a subclass that overrides `replace` changes what `copy.replace` does too."""
return self.replace(**members)

def canonicalize(self) -> ZarrV3ArrayMetadata:
"""Return this model with its document in its simplest equivalent spelling, read in this model's own scope.

The result equals this model. Every model equal to this one, in
this scope, gives the same document: `to_json` of each result is
the same JSON, apart from the order of the members of an object,
which `to_key_value` sorts with `JSONWriter(sort_keys=True)`. So
the bytes that writer writes are the same for equal models, and
are a key for what the document means.

Each declaration is written as `canonical_of` writes it, each
codec as `codec_for_chunk` returns it for the chunk it receives,
and the fill value as `canonical_fill_value` writes it for the
data type. `attributes` and `storage_transformers` are left out
when empty, because a document without them means the same.
Everything else is kept as written: `shape`, `dimension_names`,
the attributes, and whatever nothing in scope claims, which only
its own definition could simplify. A document that is already in
this spelling is returned as it is, in a new model.
"""
document = canonical_array_document_v3(self._reading, self._members)
return type(self)(document, context=self._context)

def with_context(self, context: ZarrV3Context | None = None) -> ZarrV3ArrayMetadata:
"""Return this document read in `context`, whatever that changes: a gain, a loss, or a conflict.

Expand Down Expand Up @@ -951,6 +977,30 @@ def array_key_of(
)


def canonical_array_document_v3(
reading: ZarrV3ArrayMetadataReading, members: ArrayMembersV3
) -> dict[str, JSONValue]:
"""A v3 array document in its simplest equivalent spelling, from its `reading` and refined `members`, as `ZarrV3ArrayMetadata.canonicalize` reads it.

It is written from the parts that `array_key_of` compares, so two
documents with one key are written the same way, apart from the order
of the members of an object.
"""
data_type = held(reading.data_type)
spelled: dict[str, JSONValue] = {
"data_type": simplest(data_type),
"chunk_grid": simplest(reading.chunk_grid),
"chunk_key_encoding": simplest(reading.chunk_key_encoding),
"fill_value": spelled_canonically(data_type, members.fill_value),
"codecs": tuple(simplest(codec) for codec in codecs_for_chunks(reading)),
"storage_transformers": tuple(simplest(entry) for entry in reading.storage_transformers),
**simplest_members(reading.extra_fields),
}
return canonical_members(
members.document, spelled, empty=("attributes", "storage_transformers")
)


def codecs_for_chunks(
reading: ZarrV3ArrayMetadataReading,
) -> tuple[AcceptedDeclaration[CodecDefinition[Any]] | UnclaimedDeclaration, ...]:
Expand Down
31 changes: 31 additions & 0 deletions packages/zarr-metadata/src/zarr_metadata/model/_group_v2.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
from zarr_metadata.model._keyed import Keyed
from zarr_metadata.model._shared import (
StoreKey,
canonical_members,
check_literal,
load_store_json,
located_conflicts,
Expand Down Expand Up @@ -378,6 +379,11 @@ def __replace__(self, /, **members: Unpack[ZarrV2GroupMetadataUpdate]) -> ZarrV2
"""Return `self.replace(**members)`, which `copy.replace(model, **members)` (Python 3.13) calls. It calls `replace` rather than being it, so a subclass that overrides `replace` changes what `copy.replace` does too."""
return self.replace(**members)

def canonicalize(self) -> ZarrV2GroupMetadata:
"""Return this model with its document in its simplest equivalent spelling, read in this model's own scope: without `attributes` when they are empty, because a group with no `.zattrs` means the same. The result equals this model, and every model equal to this one gives the same document, apart from the order of the members of an object, which `to_key_value` sorts with `JSONWriter(sort_keys=True)`."""
document = canonical_members(self._document, {}, empty=("attributes",))
return type(self)(document, context=self._context)

def with_context(self, context: ZarrV2Context | None = None) -> ZarrV2GroupMetadata:
"""Return this document read in `context`: the same group, holding that scope."""
context = scoped(context, CORE_V2_CONTEXT)
Expand Down Expand Up @@ -586,6 +592,31 @@ def _located_declarations(self) -> Iterator[tuple[Loc, ResolvedDeclaration[Any]]
def __reduce__(self) -> tuple[type[ZarrV2ConsolidatedMetadata], tuple[object, ZarrV2Context]]:
return type(self), (self._document, self._context)

def canonicalize(self) -> ZarrV2ConsolidatedMetadata:
"""Return this document with each node in its simplest equivalent spelling, as that node's `canonicalize` writes it, read in this model's own scope.

Each node's `.zarray` or `.zgroup` entry is written without the
attributes, and its `.zattrs` entry holds them, or is left out
when they are empty. Every other entry is kept as written. The
result equals this model. Every model equal to this one, in this
scope, gives the same document, apart from the order of the
members of an object, which a `JSONWriter(sort_keys=True)` sorts.
"""
entries = dict(object_at(self._document, "metadata"))
for path, node in self._metadata.items():
names = self._files[path]
written = cast("dict[str, JSONValue]", node.canonicalize().to_json())
attributes = written.pop("attributes", None)
for name, key in names.items():
if name != ZARR_V2_ATTRIBUTES_STORE_KEY:
entries[key] = written
elif attributes is None:
del entries[key]
else:
entries[key] = attributes
document: dict[str, JSONValue] = {**self._document, "metadata": entries}
return type(self)(document, context=self._context)

def with_context(self, context: ZarrV2Context | None = None) -> ZarrV2ConsolidatedMetadata:
"""Return this document with every node read in `context`, whatever that changes. Raises `MetadataValidationError` when a node has a problem in `context`."""
context = scoped(context, CORE_V2_CONTEXT)
Expand Down
37 changes: 37 additions & 0 deletions packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,14 @@
from zarr_metadata.model._shared import (
StoreKey,
attributes_of,
canonical_members,
check_literal,
load_store_json,
located_conflicts,
member_changes,
missing_keys,
reading_of,
simplest_members,
unclaimed_declarations_of,
unexpected_keys,
)
Expand Down Expand Up @@ -392,6 +394,27 @@ def __replace__(self, /, **members: Unpack[ZarrV3GroupMetadataUpdate]) -> ZarrV3
"""Return `self.replace(**members)`, which `copy.replace(model, **members)` (Python 3.13) calls. It calls `replace` rather than being it, so a subclass that overrides `replace` changes what `copy.replace` does too."""
return self.replace(**members)

def canonicalize(self) -> ZarrV3GroupMetadata:
"""Return this model with its document in its simplest equivalent spelling, read in this model's own scope.

The result equals this model. Every model equal to this one, in
this scope, gives the same document: `to_json` of each result is
the same JSON, apart from the order of the members of an object,
which `to_key_value` sorts with `JSONWriter(sort_keys=True)`.

`attributes` is left out when empty, because a document without
it means the same, and each extra field a definition in scope
claims is written as `canonical_of` writes it. Everything else is
kept as written. That includes the documents that
`consolidated_metadata` lists, which a group does not read:
`consolidated().canonicalize()` returns them in their simplest
spelling, and `replace(consolidated_metadata=...)` writes them
back into the group.
"""
spelled = simplest_members(self._reading.extra_fields)
document = canonical_members(self._document, spelled, empty=("attributes",))
return type(self)(document, context=self._context)

def with_context(self, context: ZarrV3Context | None = None) -> ZarrV3GroupMetadata:
"""Return this document read in `context`, whatever that changes. Raises `MetadataValidationError` when the document has a problem in `context`. The existing reading is reused when `context` reads every claim identically."""
context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT)
Expand Down Expand Up @@ -595,6 +618,20 @@ def _key_of(self) -> tuple[object, ...]:
def __reduce__(self) -> tuple[type[ZarrV3ConsolidatedMetadata], tuple[object, ZarrV3Context]]:
return type(self), (self._document, self._context)

def canonicalize(self) -> ZarrV3ConsolidatedMetadata:
"""Return this member with each document it lists in its simplest equivalent spelling, as that node's `canonicalize` writes it, read in this model's own scope.

The result equals this model. Every model equal to this one, in
this scope, gives the same member, apart from the order of the
members of an object, which a `JSONWriter(sort_keys=True)` sorts.
"""
metadata: dict[str, JSONValue] = {
path: cast("JSONValue", node.canonicalize().to_json())
for path, node in self._metadata.items()
}
document: dict[str, JSONValue] = {**self._document, "metadata": metadata}
return type(self)(document, context=self._context)

def with_context(self, context: ZarrV3Context | None = None) -> ZarrV3ConsolidatedMetadata:
"""Return this member with every document read in `context`, whatever that changes. Raises `MetadataValidationError` when a document has a problem in `context`. The existing readings are reused when `context` reads every claim of every document identically."""
context = scoped(context, CORE_AND_EXTENSIONS_CONTEXT)
Expand Down
Loading
Loading