Skip to content

feat: expose encoded chunk byte ranges - #4634

Draft
ss1738 wants to merge 3 commits into
zarr-developers:mainfrom
ss1738:feat/chunk-byte-range
Draft

ss1738 wants to merge 3 commits into
zarr-developers:mainfrom
ss1738:feat/chunk-byte-range

Conversation

@ss1738

@ss1738 ss1738 commented Nov 14, 2026 •

Copy link
Copy Markdown

Summary

Add Array.chunk_byte_range(coords) and AsyncArray.chunk_byte_range(coords) for locating a logical chunk's encoded bytes as (store_key, offset, length), returning None when it is absent. Related to #1113.

MEASURED (implementation): regular unsharded v2/v3 arrays use Store.getsize. One-level sharded v3 arrays read the shard index to locate the inner chunk. The method does not decode chunk payloads. A generic store's unsharded size lookup may fetch the entire object; the docstring documents that cost.

For reviewers

Please review the public API placement and coordinate semantics. Coordinates identify logical chunks, not elements or shards. Nested sharding, rectilinear grids, and transforms outside sharding explicitly raise NotImplementedError. Locations are not snapshots or payload-integrity checks.

Related work: #4028 discovers stored objects and reads decoded regions; this proposal returns an individual logical chunk's encoded location. #4040 explored a broader layout-inspection API and was closed. This PR does not add a general layout abstraction or change array reads.

Validation — MEASURED locally

macOS arm64 CPU, Python 3.12.13, upstream base 011d68af1be07de83a2f524f17f0fc2c782a197a:

New API + existing sharding tests: 406 passed, 1 skipped
Existing initialized-chunk/storage-size tests: 8 passed
Arrays guide examples + documentation guards: 5 passed
Mypy: Success: no issues found in 3 source files
Ruff 0.17.3: All checks passed
New helper statement coverage: 49 statements, 0 missed (100%)

Tests decode the returned physical ranges and compare them with expected data, and check missing inner chunks and index-only sharded IO. The existing skip is for a store without delete support. This is not a full-suite, cross-platform, or branch-coverage result.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

AI assistance and draft status

Code, description, and local validation were prepared with Codex at the account owner's request. Human review and the author attestation remain pending. This PR is a draft and does not claim that human review occurred.

TODO

  • Add unit tests
  • Document the API and add an executable user-guide example
  • Add a changelog fragment
  • GitHub Actions have all passed
  • Codecov passes
  • Complete human author review and replace the description with the author's own explanation before requesting review

@codecov

codecov Bot commented Nov 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.92%. Comparing base (011d68a) to head (6ffc9c7).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4634      +/-   ##
==========================================
+ Coverage   94.90%   94.92%   +0.01%     
==========================================
  Files          96       97       +1     
  Lines       14051    14105      +54     
==========================================
+ Hits        13335    13389      +54     
  Misses        716      716              
Files with missing lines Coverage Δ
src/zarr/core/_chunk_location.py 100.00% <100.00%> (ø)
src/zarr/core/array.py 98.60% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@d-v-b

d-v-b commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

this is a cool API. what's your definition of a chunk here? This API or a similar one should work for lots array selections, e.g. an entire sharded chunk, or an entire subchunk in a sharded chunk. And it would be interesting to extend this API to arbitrary array selections. I think the return type needs to get wider for that.

@ss1738

ss1738 commented Nov 14, 2026 •

Copy link
Copy Markdown
Author

Thanks!@d-v-b Here, "chunk" means a logical chunk at the resolution of Array.chunks. For a sharded array, that is an inner chunk within the shard; for an unsharded array, it is the chunk stored in its own object. The coordinates are chunk-grid coordinates, rather than element or shard coordinates.

The current return value describes one encoded byte range: (store_key, offset, length), or None for an absent chunk, including an absent inner chunk within a stored shard.

Supporting whole shards and arbitrary selections makes sense. A selection could involve multiple stored objects or byte ranges, so the single-range return type would need to change. It would also help to distinguish locating the encoded chunks that cover a selection from locating bytes that represent exactly that selection, since a partial selection may still require decoding a whole chunk.

Would you prefer this PR to keep the single-logical-chunk scope with clearer naming and documentation, or work toward a selection-based interface here?

@d-v-b

d-v-b commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Here, "chunk" means a logical chunk at the resolution of Array.chunks.

so it's worth keeping in mind that Array.chunks is a fiction that only exists for backwards compatibility with zarr v2, which had a chunks attribute on arrays. the zarr v3 spec doesn't define a single chunk shape, and whether an array selection is byte-range-contiguous depends on the chunk grid configuration and also how the codecs are configured. For example, any range of adjacent array elements in a 1D uncompressed array corresponds to a set of single byte ranges in storage. It would be nice to be able to express this, but I think it requires changes to our codec pipeline API

@ss1738

ss1738 commented Nov 14, 2026

Copy link
Copy Markdown
Author

Thanks, that corrects my framing. I agree that Array.chunks should not be the basis for a general selection-to-storage API.

MEASURED in a separate prototype on a Mac Mini, against upstream 011d68af: I mapped unit-step 1D selections through the stored grid and composed inner-codec ranges with shard-index offsets. For a little-endian int32 selection 2:6, the raw layout needs bytes [8, 24), while the Zstd layout needs the complete encoded chunk followed by decoding and slicing. For sharded layouts, the prototype reads the index once per touched shard and maps each fragment to its output selection. Reconstruction matched normal Zarr reads in 48 checks, including both index placements, cross-shard selections and missing inner chunks/shards.

This is limited to 1D int32 with bytes/Zstd and one level of regular sharding; it is not an implementation of the general pipeline API. The unsharded path also uses getsize, which may fetch payloads on some stores.

Would an optional planning capability on CodecPipeline, with codec-specific handling behind it, be a sensible direction? I would expect the result to retain the selection mapping and decoding context as well as storage ranges, and distinguish unsupported planning from an empty selection or missing data. Sharding could resolve its index and compose the inner plan into object-relative ranges.

I have left the PR implementation unchanged while exploring this. If that direction fits, I can sketch the interface before extending the public array API.

@ss1738

ss1738 commented Nov 14, 2026

Copy link
Copy Markdown
Author

I connected the result contract to the storage prototype and put the code, reproduction instructions and test output here: https://gist.github.com/ss1738/51a6b3d31deb816ae62b95b56361901a. This is an external feasibility adapter, not a change to the upstream pipeline or this PR.

MEASURED against 011d68af on arm64 macOS/Python 3.12.13: 14 test methods passed, including 292 reconstruction comparisons with normal Zarr reads. The fixtures cover 1D/2D/3D unit-step slice selections, little/big stored byte order, bytes/Zstd, regular one-level sharding with both index placements, sparse data, and MemoryStore/LocalStore. The count includes repeated unsharded fixtures for the irrelevant index-location parameter; it is not 292 distinct layouts. Additional tests cover checksum-valid malformed index entries, truncated data and planning budgets.

One useful I/O finding: planning [2:4, 2:4] in a raw 8x8 int32 chunk performs no store reads or size lookups and describes 16 payload bytes rather than the whole 256-byte chunk. Unsharded compressed plans use offset-to-end requests and resolve presence at execution. This is a byte-count fixture, not a latency benchmark.

There is also a tradeoff worth resolving: to validate shard-index ranges against object bounds, this version calls getsize. In the instrumented MemoryStore fixture that falls back to fetching the full object: four shards cost four index reads plus four full-object reads during planning. So I would not call this version index-only planning. Efficient size/version metadata or an explicit validation policy seems necessary here.

The experiment supports separating physical reads, reconstruction recipes, missing regions and planning dependencies. Would you prefer an interface sketch for an optional per-chunk planning capability next, or a separate design issue before further implementation? I have kept the public Array API unchanged. Other codecs/dtypes, rectilinear grids, integer/fancy indexing and snapshot consistency remain outside the validated scope.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants