Repository navigation
Conversation
Zarr treats a chunk as a logical array, but numcodecs codecs work on the memory they are given: most flatten with order="A" and many return a flat array. Each call site translated between the two by hand, and each guard was added after a bug report (zarr-developersgh-3558, zarr-developersgh-4553). The decode path of the array-to-array wrappers had no guard, so `Delta` and `FixedScaleOffset` followed by a `TransposeCodec` read back scrambled values. `zarr.codecs._numcodec_arrays` now holds the translation: give numcodecs a C-contiguous array, and read its output back in C order with the declared shape. The numcodecs wrappers and the vlen codecs call it instead of calling `encode` and `decode` on the numcodecs codec. `V2Codec` keeps its own handling, because v2 arrays with order="F" rely on memory order. `tests/test_codecs/test_memory_layout.py` checks that every registered array-to-array and array-to-bytes codec encodes and decodes a chunk the same way in C-contiguous, F-contiguous, strided and reversed layouts, and that a `TransposeCodec` next to a filter does not change the values read. A registered codec class without a case fails the suite. Remove `zarr._compat._reshape_view`, which had no other caller. Assisted-by: ClaudeCode:claude-fable-5-2
…layout Resolve the conflict in src/zarr/codecs/numcodecs/_codecs.py by keeping the adapter call. The decode fix from zarr-developers#4558 is the same guard, which the adapter already applies. Assisted-by: ClaudeCode:claude-fable-5-2
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Every codec that hands array chunks to numcodecs now does so through one adapter, which gives numcodecs a C-contiguous array and reshapes its output to the declared shape. A new test checks that each registered array-to-array and array-to-bytes codec gives the same result for any memory layout of a chunk.
Summary
Zarr treats a chunk as a logical array, while numcodecs codecs work on the memory they are given: most flatten with
order="A", and many return a flat array. Each call site translated between the two by hand, and each guard was added after a bug report (zarr-developersgh-3558, zarr-developersgh-4553, and the decode path fixed in zarr-developers#4558 upstream).src/zarr/codecs/_numcodec_arrays.py(new) holds the translation in four functions, one per kind of call: encode and decode, for array-to-array and array-to-bytes codecs. Each gives numcodecs a C-contiguous array and reshapes the output to the declared shape.src/zarr/codecs/numcodecs/_codecs.pyand both vlen codecs insrc/zarr/codecs/vlen_utf8.pycall the adapter instead of callingencodeanddecodeon the numcodecs codec.V2Codecis unchanged. It passes the new test as is, and v2 arrays withorder="F"rely on numcodecs reading memory order.zarr._compat._reshape_viewis removed; the vlen decoders were its only callers.Tests
tests/test_codecs/test_memory_layout.py(new) covers every array-to-array and array-to-bytes codec class that zarr registers:TransposeCodecbefore or after each filter as without one;What the agent ran:
mainbefore the upstream decode fix (fix: preserve Numcodecs filter shape and decode order zarr-developers/zarr-python#4558) and without the adapter, the new test failed in 10 cases, allDeltaorFixedScaleOffsetdecoding an F-contiguous chunk.np.ascontiguousarraycalls removed in turn, the test failed each time.mainwas merged in: 13529 passed, 1343 skipped, 13 xfailed. After the merge, the codec, pipeline-parity and v2 tests passed (1491 passed, 40 skipped, 6 xfailed).prekpassed on the changed files.cast_valuecases passed withcast-value-rsinstalled. The ZFPY and PCodec cases were skipped because those libraries were not installed. Nothing was run on a GPU.Notes
DeltaandFixedScaleOffsetlanded upstream in fix: preserve Numcodecs filter shape and decode order zarr-developers/zarr-python#4558 and is onmain. This branch hasmainmerged in, so it changes no behavior: it is a refactor plus tests.PackBitsfollowed by a 1-dimensionalTransposeCodecis rejected when the array is created, becauseArrayV3Metadatavalidates each codec against the shape of the array instead of the shape the previous codec produces.tests/, not inzarr.testing, and the hypothesis strategies still do not draw filter chains. Both would change the public testing API.🤖 Generated with Claude Code