Skip to content

fix(codecs): store the crc32c checksum little-endian on every host - #455

Draft
d-v-b wants to merge 2 commits into
mainfrom
fix/crc32c-le
Draft

d-v-b wants to merge 2 commits into
mainfrom
fix/crc32c-le

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

The crc32c codec now stores its checksum as a little-endian unsigned 32-bit integer on every host, as the crc32c codec specification requires. Previously it used the host byte order, so on big-endian hosts zarr-python wrote and expected checksums that other implementations reject.

Agent notes
  • src/zarr/codecs/crc32c_.py: encode used np.array([...], dtype=np.uint32).view("B") and decode used np.uint32(...).tobytes(). Both give native byte order. A new _checksum_bytes helper returns int.to_bytes(4, "little"), and both paths use it.
  • On little-endian hosts the stored bytes do not change. On big-endian hosts zarr-python could not read crc32c chunks written by other implementations (or by itself on another host) until now, and the sharding index uses this codec.
  • The agent found this while writing zarr-extensions specs for the numcodecs.* checksum codecs. numcodecs' own checksum codecs already use explicit <u4.
Tests
  • test_crc32c_encoded_bytes pins the stored bytes for three inputs, including the RFC 3720 check value ("123456789" → 83 92 06 e3), and checks the round trip.
  • test_crc32c_decode_rejects_mismatched_checksum checks that decoding raises when the stored checksum is the right value in big-endian order.
  • CI cannot exercise a big-endian host; the tests pin the format the spec requires.
  • Full suite: 13372 passed, 1319 skipped, 9 xfailed. pre-commit (ruff, mypy, numpydoc) passes.

🤖 Generated with Claude Code

The crc32c codec spec stores the checksum as a little-endian uint32. The codec used the host byte order, so big-endian hosts wrote and expected checksums other implementations reject.

Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant