Skip to content

fix(zarr-metadata): compare codecs for the chunk the richer reading knows when judging a refinement - #463

Open
d-v-b wants to merge 2 commits into
fix/zarr-metadata-chunk-canonical-orderfrom
fix/zarr-metadata-refines-across-scopes
Open

d-v-b wants to merge 2 commits into
fix/zarr-metadata-chunk-canonical-orderfrom
fix/zarr-metadata-refines-across-scopes

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 17, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

ZarrV3ArrayMetadata.refines compares each codec for the chunk it receives in the model that refines, so a document read in a richer scope refines the same document read in a poorer one, whatever the richer scope learns of the chunks. Before, it did not when the richer scope simplified a codec for its chunk: refined_in of a uint8 array whose bytes codec writes an endian, read in a scope that claims no bytes codec, did not refine the model it came from. codec_refines in zarr_metadata.v3.definition makes the comparison for one codec.

Why

Two subagent reviews, of zarr-developers#4640 and zarr-developers#4641, found this independently. It is a regression from zarr-developers#4640, which is not in a release yet.

Since zarr-developers#4640 an array model's refines compared each side's codecs as simplified for that side's own chunks. A richer scope can know more of a chunk than a poorer one, and then drops a bytes codec's endian that the poorer reading keeps. There are three routes:

  • the richer scope claims the codec itself: the gain was judged by reading the unclaimed JSON again, as written, against the simplified codec;
  • it claims the data type, or an array -> array codec that comes before, so the poorer reading's bytes received a chunk with nothing known of it;
  • it claims a codec that holds a pipeline, whose inner codecs are simplified.

In each, model.refined_in(richer).refines(model) was False, which tests/model/test_pair.py states is always true. Its fixtures wrote no endian on a one-byte data type, so they missed it. ZarrV3ConsolidatedMetadata.refines inherited the failure.

Design notes
  • codec_refines(stage, other) is the order refines defines for declarations, walked along the stages of the model that refines. Both readings are simplified for the chunk that stage knows, which is at least what the other reading knew.
  • A gain is judged as before, by what the unclaimed codec reads as with the claims of the accepted one, and then both are simplified for the stage's chunks. read_as_claimed in definition/_scope.py is that read, taken out of _gained.
  • Simplifying only the other side's codec, or accepting either the written or the simplified spelling, would break transitivity through equality: two equal models would not refine the same poorer reading.
  • codec_for_chunk is now _for_stage(stage.codec, stage), so one function simplifies a codec for a stage's chunks, whichever scope read the codec.
  • Within one scope nothing changes: a fuzz over 97,240 pairs of pipelines still finds refines in both directions exactly where == holds.
Tests and checks
Bookkeeping

🤖 Generated with Claude Code

…nows when judging a refinement

Since zarr-developers#4640 an array model's refines compared each side's codecs as simplified for that side's own chunks. A richer scope can know more of a chunk than a poorer one: it claims the data type, a codec that comes before, or the codec itself. It then dropped a bytes codec's endian that the poorer reading kept, and refined_in(...) no longer refined the model it came from.

codec_refines compares the codec of a stage with another reading of it, simplifying both for the chunks the stage knows. A gain is judged by what the unclaimed codec reads as with the stage's claims, simplified the same way. ZarrV3ArrayMetadata.refines uses it for each codec.

Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant