Skip to content

refactor(codecs): pass array chunks to numcodecs through one adapter - #454

Draft
d-v-b wants to merge 2 commits into
mainfrom
fix/numcodecs-memory-layout
Draft

d-v-b wants to merge 2 commits into
mainfrom
fix/numcodecs-memory-layout

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Every codec that hands array chunks to numcodecs now does so through one adapter, which gives numcodecs a C-contiguous array and reshapes its output to the declared shape. A new test checks that each registered array-to-array and array-to-bytes codec gives the same result for any memory layout of a chunk.

Summary

Zarr treats a chunk as a logical array, while numcodecs codecs work on the memory they are given: most flatten with order="A", and many return a flat array. Each call site translated between the two by hand, and each guard was added after a bug report (zarr-developersgh-3558, zarr-developersgh-4553, and the decode path fixed in zarr-developers#4558 upstream).

  • src/zarr/codecs/_numcodec_arrays.py (new) holds the translation in four functions, one per kind of call: encode and decode, for array-to-array and array-to-bytes codecs. Each gives numcodecs a C-contiguous array and reshapes the output to the declared shape.
  • The numcodecs wrappers in src/zarr/codecs/numcodecs/_codecs.py and both vlen codecs in src/zarr/codecs/vlen_utf8.py call the adapter instead of calling encode and decode on the numcodecs codec.
  • V2Codec is unchanged. It passes the new test as is, and v2 arrays with order="F" rely on numcodecs reading memory order.
  • zarr._compat._reshape_view is removed; the vlen decoders were its only callers.
Tests

tests/test_codecs/test_memory_layout.py (new) covers every array-to-array and array-to-bytes codec class that zarr registers:

  • encoding a chunk gives the same result in C-contiguous, F-contiguous, strided and reversed layouts;
  • decoding an encoded chunk with an array-to-array codec gives the same result in each layout;
  • an array reads back the same values with a TransposeCodec before or after each filter as without one;
  • a registered codec class without a case fails the suite.

What the agent ran:

  • On main before the upstream decode fix (fix: preserve Numcodecs filter shape and decode order zarr-developers/zarr-python#4558) and without the adapter, the new test failed in 10 cases, all Delta or FixedScaleOffset decoding an F-contiguous chunk.
  • With each of the adapter's three np.ascontiguousarray calls removed in turn, the test failed each time.
  • The full suite passed locally before main was merged in: 13529 passed, 1343 skipped, 13 xfailed. After the merge, the codec, pipeline-parity and v2 tests passed (1491 passed, 40 skipped, 6 xfailed). prek passed on the changed files.
  • The cast_value cases passed with cast-value-rs installed. The ZFPY and PCodec cases were skipped because those libraries were not installed. Nothing was run on a GPU.
Notes
  • The decode fix for Delta and FixedScaleOffset landed upstream in fix: preserve Numcodecs filter shape and decode order zarr-developers/zarr-python#4558 and is on main. This branch has main merged in, so it changes no behavior: it is a refactor plus tests.
  • There is no changelog fragment yet. It needs the upstream PR number.
  • One case is a strict expected failure: PackBits followed by a 1-dimensional TransposeCodec is rejected when the array is created, because ArrayV3Metadata validates each codec against the shape of the array instead of the shape the previous codec produces.
  • The test lives in tests/, not in zarr.testing, and the hypothesis strategies still do not draw filter chains. Both would change the public testing API.

🤖 Generated with Claude Code

Zarr treats a chunk as a logical array, but numcodecs codecs work on the
memory they are given: most flatten with order="A" and many return a flat
array. Each call site translated between the two by hand, and each guard
was added after a bug report (zarr-developersgh-3558, zarr-developersgh-4553). The decode path of the
array-to-array wrappers had no guard, so `Delta` and `FixedScaleOffset`
followed by a `TransposeCodec` read back scrambled values.

`zarr.codecs._numcodec_arrays` now holds the translation: give numcodecs
a C-contiguous array, and read its output back in C order with the
declared shape. The numcodecs wrappers and the vlen codecs call it instead
of calling `encode` and `decode` on the numcodecs codec. `V2Codec` keeps
its own handling, because v2 arrays with order="F" rely on memory order.

`tests/test_codecs/test_memory_layout.py` checks that every registered
array-to-array and array-to-bytes codec encodes and decodes a chunk the
same way in C-contiguous, F-contiguous, strided and reversed layouts, and
that a `TransposeCodec` next to a filter does not change the values read.
A registered codec class without a case fails the suite.

Remove `zarr._compat._reshape_view`, which had no other caller.

Assisted-by: ClaudeCode:claude-fable-5-2
…layout

Resolve the conflict in src/zarr/codecs/numcodecs/_codecs.py by keeping the
adapter call. The decode fix from zarr-developers#4558 is the same guard, which the adapter
already applies.

Assisted-by: ClaudeCode:claude-fable-5-2
@d-v-b d-v-b changed the title fix(codecs): pass array chunks to numcodecs through one adapter refactor(codecs): pass array chunks to numcodecs through one adapter Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant