Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/zarr-metadata/changes/4640.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A v3 codec that receives an array can declare its simplest spelling for the chunk it receives, with `chunk_canonical` on `ArrayCodecDefinition`. It is a function of the configuration, the nested declarations and the `Chunk`, for two spellings that mean the same thing for one chunk and not for another, which `canonical` cannot tell from the configuration alone. `codec_for_chunk` in `zarr_metadata.v3.definition` returns the codec of a `Stage` in that spelling, with each codec of the pipelines it contains simplified for the chunk that one receives. The `bytes` codec declares one: it drops an `endian` when each value takes one byte. So two `ZarrV3ArrayMetadata` models that differ only in the `endian` of a `bytes` codec that receives values of one byte (`bool`, `int8`, `uint8` or `r8`) are now equal, hash alike and refine each other. A `bytes` codec that receives values of several bytes, as it does after a `cast_value` to `float32`, is compared by its `endian` as before.
4 changes: 3 additions & 1 deletion packages/zarr-metadata/docs/extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,9 @@ Other kinds declare more:
- A codec is defined by the class of its kind, `ArrayArrayCodecDefinition`,
`ArrayBytesCodecDefinition` or `BytesBytesCodecDefinition`. A codec that
receives an array declares what it checks about the chunk it receives
(`chunk_rules`) and any pipelines it contains (`pipelines`), and an
(`chunk_rules`), its simplest spelling for that chunk
(`chunk_canonical`, as `bytes` drops an `endian` when each value takes
one byte) and any pipelines it contains (`pipelines`), and an
array -> array codec also declares what it hands on (`transition`). A
bytes -> bytes codec receives bytes, so it declares none of these.

Expand Down
7 changes: 7 additions & 0 deletions packages/zarr-metadata/docs/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,13 @@ These are each declaration, and the fill value read against its data type.
- A v3 blosc codec with `noshuffle`
[ignores `typesize`](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/codecs/blosc/index.html#configuration-parameters).
So it is the same codec with or without one.
- A v3 `bytes` codec's `endian` orders the bytes of each value, and the
encoding of a value that takes one byte
[does not depend on it](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/codecs/bytes/index.html#configuration-parameters).
So in a `uint8` array, a `bytes` codec is the same codec with either
`endian`, or with none. The codec is compared for the chunk it
receives: after a `cast_value` to `float32`, the two `endian` values
are two codecs.

A v2 document's
[dtype](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#data-type-encoding),
Expand Down
10 changes: 7 additions & 3 deletions packages/zarr-metadata/src/zarr_metadata/definition/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,11 @@ def acme_lz4_rules(
equivalent spelling, and declarations are compared by it. Each nested declaration is
written in its own simplest spelling, then the definition's `canonical`
is applied (blosc drops a `typesize` that `noshuffle` ignores, and a
rectilinear grid run-length encodes its chunk shapes). The envelope
rectilinear grid run-length encodes its chunk shapes). A v3 codec that
receives an array can have a simpler spelling still for the chunk it
receives, which its `chunk_canonical` declares, and an array model
compares its codecs in that spelling (see `zarr_metadata.v3.definition`).
The envelope
uses the shortest form every reader accepts: a data type with nothing to
configure is its bare name, and any other declaration is an object,
`{"name": ...}`, because a Zarr v3.0 reader does not accept a short-hand
Expand Down Expand Up @@ -259,8 +263,8 @@ def acme_lz4_rules(
that has a member no checker reads (the error names the TypedDict that
contains it); a `name` that is not a string; an attribute declared as a
function (`rules`, `canonical`, `fill_value_rules`, `storage`,
`shape_rules`, `chunk_lengths`, `chunk_rules`, `transition`,
`pipelines`) that is not a function; a data type's `fill_value` that no
`shape_rules`, `chunk_lengths`, `chunk_rules`, `chunk_canonical`,
`transition`, `pipelines`) that is not a function; a data type's `fill_value` that no
checker reads, or that contains a declaration, which a value of the
data type never does; a codec `kind` that is not one of the three, or a
`size` that is not `"static"` or `"dynamic"`; a function that is never
Expand Down
48 changes: 48 additions & 0 deletions packages/zarr-metadata/src/zarr_metadata/definition/_definition.py
Original file line number Diff line number Diff line change
Expand Up @@ -1259,6 +1259,52 @@ def written_json(declaration: AcceptedDeclaration[Any] | UnclaimedDeclaration) -
return kind._envelope_json(declaration.name, declaration.configuration)


def respelled(
declaration: AcceptedDeclaration[D],
configuration: Mapping[str, JSONValue],
nested: Nested,
) -> AcceptedDeclaration[D]:
"""`declaration` with `configuration` as its configuration, and each declaration in `nested` in place of the nested declaration at its location.

This builds the declaration that another spelling of `declaration`
reads as, without reading it again. The caller answers for the
spelling: `configuration` must be one that the definition allows and
that means the same thing, and it must keep each nested declaration
where it is. `nested` holds only the nested declarations to replace,
each one accepted or unclaimed, and each is written into the
configuration as its `to_json` writes it. A location in `nested` that
does not lead into `configuration` raises `TypeError`. The result is
written as `to_json` writes it, since no document wrote it.
"""
spelled: JSONValue = dict(configuration)
for loc, inner in nested.items():
spelled = _replaced(spelled, loc, written_json(held(inner)))
kind = declaration.read_as
name = declaration.name
carried = kind._filed_under(name)[1] is not None
return AcceptedDeclaration(
json=kind._envelope_json(name, {} if carried else cast("Mapping[str, JSONValue]", spelled)),
name=name,
definition=declaration.definition,
configuration=cast("Mapping[str, JSONValue]", spelled),
nested={**declaration.nested, **nested},
)


def item_at(value: JSONValue, path: Loc) -> JSONValue | UNSET:
"""The item of `value` at `path`; `UNSET` when `path` does not lead into `value`."""
for step in path:
if isinstance(step, str):
if not isinstance(value, Mapping) or step not in value:
return UNSET
value = value[step]
else:
if not isinstance(value, tuple) or not 0 <= step < len(value):
return UNSET
value = value[step]
return value


Nested: TypeAlias = Mapping[Loc, ResolvedDeclaration[Any]]
"""The declarations in a configuration, each as the scope read it, keyed by location in the configuration."""

Expand Down Expand Up @@ -1902,13 +1948,15 @@ def _replaced(value: JSONValue, path: Loc, new: JSONValue) -> JSONValue:
"held",
"in_document",
"is_declaration",
"item_at",
"kind_of",
"named_configuration",
"no_rules",
"own_key",
"read_only",
"resolve",
"resolve_refined",
"respelled",
"ruled",
"spelled_canonically",
"unchanged",
Expand Down
19 changes: 14 additions & 5 deletions packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@
chunk_grid_lengths,
default_array_bytes_codec,
)
from zarr_metadata.v3._pipeline import Stage, read_pipeline
from zarr_metadata.v3._pipeline import Stage, codec_for_chunk, read_pipeline
from zarr_metadata.v3.array import (
ZARR_V3_ARRAY_METADATA_STORE_KEY,
ZarrV3ArrayMetadataJSON,
Expand Down Expand Up @@ -780,7 +780,7 @@ def refines(self, other: ZarrV3ArrayMetadata) -> bool:
(self.data_type, other.data_type),
(self.chunk_grid, other.chunk_grid),
(self.chunk_key_encoding, other.chunk_key_encoding),
*zip(self.codecs, other.codecs, strict=True),
*zip(codecs_for_chunks(self._reading), codecs_for_chunks(other._reading), strict=True),
*zip(self.storage_transformers, other.storage_transformers, strict=True),
*((self.extra_fields[key], other.extra_fields[key]) for key in self.extra_fields),
)
Expand Down Expand Up @@ -926,8 +926,10 @@ def array_key_of(

The key holds each declaration's `declaration_key`, and the fill value as JSON
text: in its canonical spelling when a definition in scope read the
data type, and as written otherwise. Every other member is included
as it is, with JSON members as text.
data type, and as written otherwise. A codec is compared as
`codec_for_chunk` returns it for the chunk it receives, so two
spellings that mean the same for that chunk are one codec. Every
other member is included as it is, with JSON members as text.
"""
data_type = held(reading.data_type)
fill_value = (
Expand All @@ -940,7 +942,7 @@ def array_key_of(
json_text(fill_value),
declaration_key(data_type),
declaration_key(held(reading.chunk_grid)),
tuple(declaration_key(held(stage.codec)) for stage in reading.pipeline),
tuple(declaration_key(codec) for codec in codecs_for_chunks(reading)),
declaration_key(held(reading.chunk_key_encoding)),
members.dimension_names,
json_text(members.attributes),
Expand All @@ -949,6 +951,13 @@ def array_key_of(
)


def codecs_for_chunks(
reading: ZarrV3ArrayMetadataReading,
) -> tuple[AcceptedDeclaration[CodecDefinition[Any]] | UnclaimedDeclaration, ...]:
"""The codecs of `reading`, a read that found no problems, in pipeline order: each as `codec_for_chunk` returns it for the chunk it receives."""
return tuple(held(codec_for_chunk(stage)) for stage in reading.pipeline)


def extra_fields_key(extra_fields: Mapping[str, ResolvedDeclaration[Any]]) -> tuple[object, ...]:
"""A key for what a document's extra fields mean: each key with its declaration's `declaration_key`, sorted by key."""
return tuple(
Expand Down
28 changes: 28 additions & 0 deletions packages/zarr-metadata/src/zarr_metadata/v3/_definition.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
Nested,
Problems,
ResolvedDeclaration,
T,
WithFillValue,
asked,
is_declaration,
Expand Down Expand Up @@ -94,6 +95,11 @@ def no_pipelines(*_: object) -> Mapping[str, Chunk]:
return {}


def unchanged_for_chunk(configuration: T, nested: object, chunk: object) -> T:
"""The simplest spelling of a codec whose configuration means the same for every chunk it receives: the configuration itself."""
return configuration


StorageClass = Literal["single_byte", "multi_byte", "variable_length"]
"""How a data type's values are stored: in single bytes, in several bytes at a time, or each value in as many bytes as it needs.

Expand Down Expand Up @@ -415,10 +421,31 @@ class ArrayCodecDefinition(CodecDefinition[C]):
inner codecs receive its inner chunks, and its index codecs receive
the shard index. It is called whatever the chunk rules found, and must
return only what is true in either case.

Two spellings of a configuration can mean the same thing for one
chunk and different things for another. For example, the `endian` of
a `bytes` codec changes nothing when each value takes one byte, and
is required when a value takes several. `canonical` cannot reduce
such spellings to one, because it receives the configuration alone.
`chunk_canonical` can: it receives the configuration, the nested
declarations as the scope read them, and the chunk, and returns the
simplest configuration that means the same for that chunk. It is
called only for a codec that the chunk rules found no problem with.
It simplifies only the codec's own members, and it must keep each
nested declaration as it is, because each codec of a pipeline the
codec contains is simplified for the chunk it receives in turn. What
does not depend on the chunk is left to `canonical`, which simplifies
the result as it simplifies any configuration. A chunk rule checks
only the parts of a chunk that are known, and `chunk_canonical`
simplifies only by them: a chunk with nothing known about it,
`Chunk()`, leaves every configuration as it is. `codec_for_chunk`
returns the codec of a `Stage` in that spelling.
"""

chunk_rules: Callable[[C, Nested, Chunk], Iterable[ValidationProblem]] = no_rules
"""The spec violations for this codec with a given input chunk, located in the configuration. A problem at `()` is about the codec as a whole, such as an input it cannot encode, and is reported at the codec."""
chunk_canonical: Callable[[C, Nested, Chunk], C] = unchanged_for_chunk
"""A configuration the chunk rules allow for a given input chunk, in the simplest spelling that means the same for that chunk."""
pipelines: Callable[[C, Nested, Chunk], Mapping[str, Chunk]] = no_pipelines
"""The chunk each contained pipeline receives, keyed by the configuration member that holds the pipeline."""

Expand Down Expand Up @@ -879,6 +906,7 @@ def chunk_grid_lengths(
"no_pipelines",
"single_byte",
"storage_of",
"unchanged_for_chunk",
"unknown_chunk",
"unknown_lengths",
"unknown_storage",
Expand Down
107 changes: 105 additions & 2 deletions packages/zarr-metadata/src/zarr_metadata/v3/_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,23 @@
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, TypeGuard, cast

from zarr_metadata._json import ValidationProblem, is_object, is_tuple, with_input
from zarr_metadata._json import (
ValidationProblem,
copied,
is_object,
is_tuple,
json_text,
with_input,
)
from zarr_metadata._sentinel import UNSET
from zarr_metadata.definition._definition import (
AcceptedDeclaration,
ResolvedDeclaration,
asked,
in_document,
item_at,
read_only,
respelled,
ruled,
)
from zarr_metadata.v3._definition import (
Expand All @@ -53,6 +63,7 @@
if TYPE_CHECKING:
from collections.abc import Iterator, Mapping, Sequence

from zarr_metadata._common import JSONValue
from zarr_metadata._typed_json import Loc
from zarr_metadata.definition._definition import (
Nested,
Expand Down Expand Up @@ -326,4 +337,96 @@ def _handed_on(
return given


__all__ = ["Stage", "read_pipeline"]
def codec_for_chunk(stage: Stage) -> ResolvedDeclaration[CodecDefinition[Any]]:
"""The codec of `stage`, in the simplest spelling that means the same for the chunk it receives.

A codec's own `canonical` reduces the spellings that mean the same
thing wherever the codec is used. Some spellings mean the same thing
only for some chunks. For example, a `bytes` codec with and without
`endian` is one codec for values that take one byte, and two for
values that take several. So a codec is compared, and written in its
simplest spelling, as the codec this function returns: first each
codec of each pipeline it contains, for the chunk that codec
receives, and then its own configuration, as its definition's
`chunk_canonical` simplifies it for the chunk in `stage`.

A codec the scope did not read is returned as it is. A codec's own
configuration is kept as it is when the codec receives bytes, and
when its chunk rules find a problem with it, because a codec with a
problem has no simpler spelling. When nothing is simplified, the
result is the codec itself. Otherwise it is a new declaration, read
by the same definition. `canonical_of` writes it in its simplest
spelling, and `==` compares it.

`chunk_canonical` is the extension author's code, so its result is
checked: the definition must allow it, the chunk rules must find no
problem with it, and each nested declaration must be where it was. A
result that fails a check raises `ValueError`, which indicates a bug
in the definition, not in the codec.
"""
codec = stage.codec
if not isinstance(codec, AcceptedDeclaration):
return codec
inner: dict[Loc, ResolvedDeclaration[Any]] = {}
for member, stages in stage.inner.items():
for index, each in enumerate(stages):
simpler = codec_for_chunk(each)
if simpler is not each.codec:
inner[(member, index)] = simpler
configuration = _for_chunk(codec, stage.incoming)
if configuration is None and len(inner) == 0:
return codec
return respelled(codec, codec.configuration if configuration is None else configuration, inner)


def _for_chunk(
codec: AcceptedDeclaration[CodecDefinition[Any]], chunk: Chunk | None
) -> Mapping[str, JSONValue] | None:
"""The configuration of `codec` as its definition's `chunk_canonical` simplifies it for `chunk`, checked; None when nothing is simplified."""
definition = codec.definition
if chunk is None or not isinstance(definition, ArrayCodecDefinition):
return None
configuration, nested = codec.configuration, codec.nested
if len(_chunk_problems(definition, configuration, nested, chunk, ())) != 0:
return None
given: object = asked(
definition,
"chunk_canonical",
lambda: cast("object", definition.chunk_canonical(read_only(configuration), nested, chunk)),
)
if given is configuration:
return None
if not is_object(given):
msg = f"{definition.name!r}: its chunk_canonical gives a configuration, got {given!r}"
raise TypeError(msg)
simplified = cast("dict[str, JSONValue]", copied(cast("JSONValue", dict(given))))
if json_text(simplified) == json_text(cast("JSONValue", configuration)):
return None
_, refused = definition.read_configuration(simplified)
found = (
*refused,
*_chunk_problems(definition, simplified, nested, chunk, ()),
*(
ValidationProblem(
loc, "expected the nested declaration to stay as it is", "invalid_value"
)
for loc in nested
if _text_at(simplified, loc) != _text_at(cast("JSONValue", configuration), loc)
),
)
if len(found) != 0:
msg = (
f"{definition.name!r}: its chunk_canonical gave {simplified!r}, which does not hold: "
f"{list(found)!r}"
)
raise ValueError(msg)
return simplified


def _text_at(configuration: JSONValue, loc: Loc) -> str | None:
"""The JSON text of the item of `configuration` at `loc`; None when `loc` does not lead into it."""
item = item_at(configuration, loc)
return None if item is UNSET else json_text(item)


__all__ = ["Stage", "codec_for_chunk", "read_pipeline"]
Loading
Loading