Skip to content

fix(codecs): compress numcodecs Blosc with the item size as typesize - #457

Draft
d-v-b wants to merge 2 commits into
mainfrom
fix/blosc-typesize
Draft

d-v-b wants to merge 2 commits into
mainfrom
fix/blosc-typesize

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

zarr.codecs.numcodecs.Blosc now compresses with the item size of the chunk's data type as its typesize when typesize is not configured, as numcodecs did for Zarr V2 arrays. Previously it always used a typesize of 1, so the default byte shuffle had no effect. Existing data reads unchanged, because each chunk records its typesize.

Agent notes
  • The bytes-to-bytes wrapper passes chunk_data.as_array_like(), a uint8 array, to numcodecs. numcodecs Blosc takes its typesize from the item size of the array it gets, so without a configured typesize it always used 1.
  • With the default shuffle=1, byte shuffling did nothing. shuffle=-1 (AUTOSHUFFLE) always meant bit-shuffle.
  • Example: a 512-byte uint16 chunk compressed to 528 bytes, and to 295 bytes with a typesize of 2.
  • When typesize is not configured, the wrapper now views the chunk as V<item_size>, which gives numcodecs the item size of the chunk's data type, as for Zarr V2 arrays.
    • It falls back to raw bytes when the data type has no fixed item size, or when the length is not a whole number of items (e.g. after a preceding checksum codec).
    • A configured typesize is passed through unchanged.
  • Why the metadata is unchanged: writing typesize into the metadata (as the native BloscCodec does) would break installs with numcodecs < 0.16, which zarr-python still supports, because numcodecs.Blosc only accepts typesize from 0.16. The typed view works with every supported version.
  • Decoding is unaffected: blosc chunks record their typesize and shuffle in the header, so data written before this change reads as before.
  • This lines numcodecs.blosc up with zarr migrate v3, which already resolves AUTOSHUFFLE from the item size (zarr-developers/zarr-python#4630).
  • The companion zarr-extensions spec draft for numcodecs.blosc (feat(codecs): add numcodecs.* compression and shuffle codecs zarr-extensions#5) is updated to describe this default.
Tests
  • test_blosc_typesize checks the typesize byte in the stored chunk header and the round trip for:
    • uint8 → 1, uint16 → 2, float64 → 8;
    • float64 after a numcodecs.crc32 → 1;
    • a configured typesize=4 (skipped on numcodecs < 0.16).
  • The uint16 and float64 cases fail on main and pass with this change.
  • Full suite: 13373 passed, 1319 skipped, 9 xfailed. pre-commit passes.

🤖 Generated with Claude Code

Without a configured typesize, numcodecs Blosc received raw bytes and always used a typesize of 1, so byte shuffling had no effect. View the chunk with the item size of its data type, as Zarr V2 did.

Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant