Repository navigation
Conversation
Without a configured typesize, numcodecs Blosc received raw bytes and always used a typesize of 1, so byte shuffling had no effect. View the chunk with the item size of its data type, as Zarr V2 did. Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
zarr.codecs.numcodecs.Bloscnow compresses with the item size of the chunk's data type as its typesize whentypesizeis not configured, as numcodecs did for Zarr V2 arrays. Previously it always used a typesize of 1, so the default byte shuffle had no effect. Existing data reads unchanged, because each chunk records its typesize.Agent notes
chunk_data.as_array_like(), auint8array, to numcodecs. numcodecsBlosctakes its typesize from the item size of the array it gets, so without a configuredtypesizeit always used 1.shuffle=1, byte shuffling did nothing.shuffle=-1(AUTOSHUFFLE) always meant bit-shuffle.uint16chunk compressed to 528 bytes, and to 295 bytes with a typesize of 2.typesizeis not configured, the wrapper now views the chunk asV<item_size>, which gives numcodecs the item size of the chunk's data type, as for Zarr V2 arrays.typesizeis passed through unchanged.typesizeinto the metadata (as the nativeBloscCodecdoes) would break installs with numcodecs < 0.16, which zarr-python still supports, becausenumcodecs.Blosconly acceptstypesizefrom 0.16. The typed view works with every supported version.numcodecs.bloscup withzarr migrate v3, which already resolves AUTOSHUFFLE from the item size (zarr-developers/zarr-python#4630).numcodecs.blosc(feat(codecs): add numcodecs.* compression and shuffle codecs zarr-extensions#5) is updated to describe this default.Tests
test_blosc_typesizechecks the typesize byte in the stored chunk header and the round trip for:uint8→ 1,uint16→ 2,float64→ 8;float64after anumcodecs.crc32→ 1;typesize=4(skipped on numcodecs < 0.16).uint16andfloat64cases fail onmainand pass with this change.🤖 Generated with Claude Code