Repository navigation
fix(zarr-metadata): compare codecs for the chunk the richer reading knows when judging a refinement - #463
Open
d-v-b wants to merge 2 commits into
Conversation
…nows when judging a refinement Since zarr-developers#4640 an array model's refines compared each side's codecs as simplified for that side's own chunks. A richer scope can know more of a chunk than a poorer one: it claims the data type, a codec that comes before, or the codec itself. It then dropped a bytes codec's endian that the poorer reading kept, and refined_in(...) no longer refined the model it came from. codec_refines compares the codec of a stage with another reading of it, simplifying both for the chunks the stage knows. A gain is judged by what the unclaimed codec reads as with the stage's claims, simplified the same way. ZarrV3ArrayMetadata.refines uses it for each codec. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
ZarrV3ArrayMetadata.refinescompares each codec for the chunk it receives in the model that refines, so a document read in a richer scope refines the same document read in a poorer one, whatever the richer scope learns of the chunks. Before, it did not when the richer scope simplified a codec for its chunk:refined_inof auint8array whosebytescodec writes anendian, read in a scope that claims nobytescodec, did not refine the model it came from.codec_refinesinzarr_metadata.v3.definitionmakes the comparison for one codec.Why
Two subagent reviews, of zarr-developers#4640 and zarr-developers#4641, found this independently. It is a regression from zarr-developers#4640, which is not in a release yet.
Since zarr-developers#4640 an array model's
refinescompared each side's codecs as simplified for that side's own chunks. A richer scope can know more of a chunk than a poorer one, and then drops abytescodec'sendianthat the poorer reading keeps. There are three routes:bytesreceived a chunk with nothing known of it;In each,
model.refined_in(richer).refines(model)wasFalse, whichtests/model/test_pair.pystates is always true. Its fixtures wrote noendianon a one-byte data type, so they missed it.ZarrV3ConsolidatedMetadata.refinesinherited the failure.Design notes
codec_refines(stage, other)is the orderrefinesdefines for declarations, walked along the stages of the model that refines. Both readings are simplified for the chunk that stage knows, which is at least what the other reading knew.read_as_claimedindefinition/_scope.pyis that read, taken out of_gained.codec_for_chunkis now_for_stage(stage.codec, stage), so one function simplifies a codec for a stage's chunks, whichever scope read the codec.refinesin both directions exactly where==holds.Tests and checks
tests/model/test_pair.py: six new cases oftest_refined_in_moves_a_model_up_the_order, one for each route, with and without a shard. All six fail without the fix.tests/model/test_invariants.py(new), over the package's Hypothesis strategies:tests/v3/test_pipelines.py: one parametrized test ofcodec_refinesover twelve pairs of readings.tests/v3/test_definitions.py: a shard in a shard with its members reordered at both depths, and a refused shard with its members reordered, which the review of fix(zarr-metadata): compare the declarations a configuration holds whatever order it writes them in zarr-developers/zarr-python#4641 listed as untested.just test-versions),ruff0.17.3,pyright,mkdocs build --strictand the pre-commit hooks.Bookkeeping
v3/_pipeline.py. Merge fix(zarr-metadata): simplify a codec for its chunk in its canonical spelling, and check the result #462 first, then mergemaininto this branch.🤖 Generated with Claude Code