Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4634 +/- ##
==========================================
+ Coverage 94.90% 94.92% +0.01%
==========================================
Files 96 97 +1
Lines 14051 14105 +54
==========================================
+ Hits 13335 13389 +54
Misses 716 716
🚀 New features to boost your workflow:
|
|
this is a cool API. what's your definition of a chunk here? This API or a similar one should work for lots array selections, e.g. an entire sharded chunk, or an entire subchunk in a sharded chunk. And it would be interesting to extend this API to arbitrary array selections. I think the return type needs to get wider for that. |
|
Thanks!@d-v-b Here, "chunk" means a logical chunk at the resolution of The current return value describes one encoded byte range: Supporting whole shards and arbitrary selections makes sense. A selection could involve multiple stored objects or byte ranges, so the single-range return type would need to change. It would also help to distinguish locating the encoded chunks that cover a selection from locating bytes that represent exactly that selection, since a partial selection may still require decoding a whole chunk. Would you prefer this PR to keep the single-logical-chunk scope with clearer naming and documentation, or work toward a selection-based interface here? |
so it's worth keeping in mind that |
|
Thanks, that corrects my framing. I agree that MEASURED in a separate prototype on a Mac Mini, against upstream This is limited to 1D int32 with bytes/Zstd and one level of regular sharding; it is not an implementation of the general pipeline API. The unsharded path also uses Would an optional planning capability on I have left the PR implementation unchanged while exploring this. If that direction fits, I can sketch the interface before extending the public array API. |
|
I connected the result contract to the storage prototype and put the code, reproduction instructions and test output here: https://gist.github.com/ss1738/51a6b3d31deb816ae62b95b56361901a. This is an external feasibility adapter, not a change to the upstream pipeline or this PR. MEASURED against One useful I/O finding: planning There is also a tradeoff worth resolving: to validate shard-index ranges against object bounds, this version calls The experiment supports separating physical reads, reconstruction recipes, missing regions and planning dependencies. Would you prefer an interface sketch for an optional per-chunk planning capability next, or a separate design issue before further implementation? I have kept the public Array API unchanged. Other codecs/dtypes, rectilinear grids, integer/fancy indexing and snapshot consistency remain outside the validated scope. |
Summary
Add
Array.chunk_byte_range(coords)andAsyncArray.chunk_byte_range(coords)for locating a logical chunk's encoded bytes as(store_key, offset, length), returningNonewhen it is absent. Related to #1113.MEASURED (implementation): regular unsharded v2/v3 arrays use
Store.getsize. One-level sharded v3 arrays read the shard index to locate the inner chunk. The method does not decode chunk payloads. A generic store's unsharded size lookup may fetch the entire object; the docstring documents that cost.For reviewers
Please review the public API placement and coordinate semantics. Coordinates identify logical chunks, not elements or shards. Nested sharding, rectilinear grids, and transforms outside sharding explicitly raise
NotImplementedError. Locations are not snapshots or payload-integrity checks.Related work: #4028 discovers stored objects and reads decoded regions; this proposal returns an individual logical chunk's encoded location. #4040 explored a broader layout-inspection API and was closed. This PR does not add a general layout abstraction or change array reads.
Validation — MEASURED locally
macOS arm64 CPU, Python 3.12.13, upstream base
011d68af1be07de83a2f524f17f0fc2c782a197a:Tests decode the returned physical ranges and compare them with expected data, and check missing inner chunks and index-only sharded IO. The existing skip is for a store without delete support. This is not a full-suite, cross-platform, or branch-coverage result.
Author attestation
AI assistance and draft status
Code, description, and local validation were prepared with Codex at the account owner's request. Human review and the author attestation remain pending. This PR is a draft and does not claim that human review occurred.
TODO