fix(indexing): define an explicit source token contract - #4348
Draft
d-v-b wants to merge 15 commits into
Draft
Conversation
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts. Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing. Assisted-by: Codex:GPT-6
…oundaries Assisted-by: Codex:GPT-6
Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates. Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
github-actions
Bot
added
the
needs release notes
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Documentation build overview
24 files changed ·
|
Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values. Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Depends on #4345. This draft targets upstream
main, so its diff includes the prerequisite audit commits until that PR merges. Follow-up-only diff.Tokenizing a lazy source previously depended on whether Dask was installed: the fallback could convert and read small remote arrays, hash object references instead of nested values, or return a fresh UUID on every call for large arrays.
Define a conservative source contract: hash plain NumPy arrays without object fields in full; delegate explicit source hooks and propagate their errors; reject other sources without conversion or serialization. Remove generic Dask and UUID fallbacks. Zarr and opaque sources can still be passed to
dask.array.from_arraywithname=False; update and execute the Dask example accordingly.Hashing has linear time and temporary-memory costs. Hooks own their versioning and I/O semantics. Tokens describe the source at tokenization time, not a snapshot or persistent cache identity, and cannot invalidate keys in graphs already constructed before a mutation. Object arrays, subclasses, and memory-mapped arrays need a source hook or Dask's naming opt-out.
Validation: an initial run passed 1,533 package tests/doctests and Dask example checks with Dask 2026.8 and TensorStore 0.1.84; 23 focused token/Dask checks passed after adding scalar, structured, string, noncontiguous, and mmap-buffer cases. Initial red run reproduced five unsafe/nondeterministic fallback failures. Applicable prek hooks pass; package pyright reports 0 errors (90 warnings).
Contract references: https://docs.dask.org/en/stable/custom-collections.html#implementing-deterministic-hashing and https://docs.dask.org/en/stable/generated/dask.array.from_array.html.
Corresponding fork review: d-v-b#336.
All four final follow-up heads were also combined and verified together: 1,633 package tests/doctests passed, no skips, and the strict documentation build passed.