diff --git a/packages/zarr-indexing/CONTRIBUTING.md b/packages/zarr-indexing/CONTRIBUTING.md index 632e24929f..bcbe51c079 100644 --- a/packages/zarr-indexing/CONTRIBUTING.md +++ b/packages/zarr-indexing/CONTRIBUTING.md @@ -8,16 +8,16 @@ just test # run the test suite (extra args go to pytest) just lint # ruff, same invocation as CI just typecheck # pyright, same invocation as CI just docs-check # strict build of the docs site -just check # all of the above +just check # the checks above plus TensorStore parity just docs-serve # serve the docs site locally ``` -Run them from this directory, or from anywhere in the repository as +Run them from this directory, or from the repository root as `just packages/zarr-indexing/`. -The test recipe runs against the workspace-root environment, because the -chunk-resolution tests exercise this package against `zarr`'s chunk grids and -`zarr` is deliberately not a dependency of this package. +The test recipe layers this package into the repository-root environment. +Chunk-resolution tests use this package’s own grids; the Dask example also +uses `zarr`, which is not a dependency of the base indexing package. ## License diff --git a/packages/zarr-indexing/README.md b/packages/zarr-indexing/README.md index 4328de1cc1..2992dcf280 100644 --- a/packages/zarr-indexing/README.md +++ b/packages/zarr-indexing/README.md @@ -5,9 +5,9 @@ Composable, lazy coordinate transforms for Zarr array indexing. Documentation: This package implements TensorStore-inspired index transforms. The core idea: -every indexing operation (slicing, fancy indexing, etc.) produces a coordinate -mapping from user space to storage space. These mappings compose lazily — no -I/O until you explicitly read or write. +each supported indexing operation (slicing, fancy indexing, etc.) produces a coordinate +mapping from user space to storage space. These mappings compose without reading selected source values. `LazyArray` +materializes them on request; the transform algebra itself performs no source I/O. Key types: @@ -15,9 +15,9 @@ Key types: accessor: `LazyArray.from_numpy(numpy_array).lazy[10:50, ::2].lazy.oindex[[3, 1, 1], :]` composes a transform and returns a new view without reading data, and `result()` materializes it into owned system memory. `LazyArray(source)` uses - the conservative basic reader; `from_numpy` explicitly selects NumPy's - optimized reader. Device arrays require an explicit custom reader responsible - for transferring values into the supplied system-memory output buffer. + the basic reader; `from_numpy` selects `numpy_reader`, which currently uses + the same slab-and-gather implementation. Device sources that refuse NumPy + conversion need a custom reader to transfer values into the output buffer. - `Reader` — the explicit backend execution boundary: transforms say which values belong in the result, while readers say how a backend obtains them - `IndexDomain` — a rectangular region of integer coordinates @@ -32,9 +32,11 @@ Key types: dimension can depend on the input - `compose` — chain two transforms into one -The package depends only on NumPy and the standard library; it does not import -`zarr`. It is developed in the [zarr-python](https://github.com/zarr-developers/zarr-python) -repository and consumed by `zarr` to resolve array indexing operations. +The base package depends on NumPy and the standard library; its optional testing +module also requires Hypothesis. The package does not import `zarr`. It is developed +in the [zarr-python](https://github.com/zarr-developers/zarr-python) repository, +and its examples include reading Zarr arrays through Dask. Installing it does not +replace Zarr's indexing implementation. ## Installation diff --git a/packages/zarr-indexing/benchmarks/README.md b/packages/zarr-indexing/benchmarks/README.md index 28ef3e6a16..e5d39b1a89 100644 --- a/packages/zarr-indexing/benchmarks/README.md +++ b/packages/zarr-indexing/benchmarks/README.md @@ -22,6 +22,19 @@ whether input arrays and transform construction are included. Repeated local column access measures cache reuse, which trades allocation against retained memory. Bounded coordinate batches avoid constructing the full coordinate array. +The current script reports `peak_mib` as the incremental peak tracked by +`tracemalloc` during a separate invocation after the timed calls. It is not +process RSS or total retained memory. Inputs and most transforms are constructed +before measurement. The case walks construct plans and consume projections +without retaining them; `all_coordinates` materializes the complete coordinate +array. There is no bounded-batch coordinate workload in this script. + +`local_rows` reuses one prepared table: its first timed invocation populates the +local-coordinate cache, while later timed invocations and the allocation probe +reuse that cache. Its median mixes a cold first call with warm calls, and its +reported peak excludes the already-retained cache. A separate fresh-table +measurement would be needed to quantify cold cache construction. + These scripts measure planning rather than codec or storage throughput. Repeat measurements with alternating operation order before interpreting small timing differences. Preserve raw benchmark output as an experiment artifact rather than diff --git a/packages/zarr-indexing/benchmarks/chunk_planning.py b/packages/zarr-indexing/benchmarks/chunk_planning.py index 934d68dcbc..d7eadf248d 100644 --- a/packages/zarr-indexing/benchmarks/chunk_planning.py +++ b/packages/zarr-indexing/benchmarks/chunk_planning.py @@ -25,6 +25,11 @@ def measure(operation: Callable[[], Any], repeats: int) -> dict[str, float]: + """Time repeated calls, then trace one additional call's incremental peak. + + State retained by earlier calls can affect the extra call; this is not a + measurement of process RSS or all memory owned by the operation's inputs. + """ samples = [] for _ in range(repeats): start = time.perf_counter() @@ -84,6 +89,8 @@ def read_local_rows() -> None: for row in range(len(table)): table.local[table.run(row)] + # The first timed call fills table.local; later calls, including the traced + # allocation probe, reuse it. The table and retained cache are not re-created. results["local_rows"] = measure(read_local_rows, args.repeats) part = plan_chunks( IndexTransform.from_shape((100, 100, 100)), diff --git a/packages/zarr-indexing/changes/4345.bugfix.md b/packages/zarr-indexing/changes/4345.bugfix.md new file mode 100644 index 0000000000..928114728f --- /dev/null +++ b/packages/zarr-indexing/changes/4345.bugfix.md @@ -0,0 +1 @@ +Reject invalid wire index-array values and unrepresentable normalized bounds/ranks, and raise an explicit error for unsupported intersections sharing an affine and lookup input axis instead of returning incorrect coordinates. Group negative chunk-coordinate tuples without merging distinct chunks. diff --git a/packages/zarr-indexing/changes/4345.doc.md b/packages/zarr-indexing/changes/4345.doc.md new file mode 100644 index 0000000000..562e114e4e --- /dev/null +++ b/packages/zarr-indexing/changes/4345.doc.md @@ -0,0 +1,3 @@ +Correct indexing, reader, serialization, cache, and integration descriptions to match supported behavior; qualify NumPy/TensorStore compatibility and performance claims. + +Describe current contracts in source and test docstrings instead of narrating prior implementations. Clarify that immutable index coordinates do not snapshot source values. diff --git a/packages/zarr-indexing/changes/4349.feature.md b/packages/zarr-indexing/changes/4349.feature.md new file mode 100644 index 0000000000..5043c83363 --- /dev/null +++ b/packages/zarr-indexing/changes/4349.feature.md @@ -0,0 +1 @@ +Add `Partition.result()` to execute a planned partition independently with the same global transform and chunk projection supplied during parent assembly. diff --git a/packages/zarr-indexing/docs/api/grid.md b/packages/zarr-indexing/docs/api/grid.md index b7c376eb85..012c9df6dc 100644 --- a/packages/zarr-indexing/docs/api/grid.md +++ b/packages/zarr-indexing/docs/api/grid.md @@ -11,8 +11,9 @@ objects whose `shape` is the valid data size and whose `codec_shape` preserves the full codec-buffer size at a regular-grid boundary. `dimension_grids_from_chunks` returns these compact dimensions: integer chunk -shapes become `FixedDimension` instances and explicit per-axis edge sequences -become `VaryingDimension` instances. `DimensionGridLike` remains the narrow +shapes become `FixedDimension` instances and positive per-axis edge sequences +become `VaryingDimension` instances. An empty-axis sequence of zeros (including +an empty sequence) becomes `FixedDimension(size=0, extent=0)`. `DimensionGridLike` remains the narrow protocol used by the chunk planner, while `EdgeDimensionGrid` is kept for explicit edge-based and coordinate-origin examples. diff --git a/packages/zarr-indexing/docs/api/index.md b/packages/zarr-indexing/docs/api/index.md index a5b7163bbd..8b7a5d6515 100644 --- a/packages/zarr-indexing/docs/api/index.md +++ b/packages/zarr-indexing/docs/api/index.md @@ -25,12 +25,11 @@ and the wire format built on top of it. - [`zarr_indexing.domain`](domain.md) — `IndexDomain`, a rectangular region of integer coordinates with an explicit (possibly non-zero) origin - [`zarr_indexing.output_map`](output_map.md) — `ConstantMap`, `DimensionMap`, - and `ArrayMap`: three representations of a set of integer coordinates, one - per storage dimension + and `ArrayMap`: three coordinate mappings that preserve order and duplicates, + one per storage dimension - [`zarr_indexing.transform`](transform.md) — `IndexTransform`, which pairs a domain with output maps, plus the indexing (`[...]`, `.oindex`, `.vindex`), - `intersect`, and `translate` operations, and `selection_to_transform` - transforms into one + `intersect`, `translate`, and `compose` operations, and `selection_to_transform` **Chunk resolution** diff --git a/packages/zarr-indexing/docs/api/lazy_array.md b/packages/zarr-indexing/docs/api/lazy_array.md index f855511d46..b6c8c0782f 100644 --- a/packages/zarr-indexing/docs/api/lazy_array.md +++ b/packages/zarr-indexing/docs/api/lazy_array.md @@ -7,21 +7,22 @@ reader and composes its transform without reading data. `result()` allocates owned system memory, then calls that reader once for each projected part. Rectangular parts write directly into their final slices; advanced placement may first use an owned dense temporary. `LazyArray(source)` assumes only basic -indexing, while `LazyArray.from_numpy(array)` explicitly selects NumPy's -optimized reader. +indexing, while `LazyArray.from_numpy(array)` selects `numpy_reader`. Both +currently use the same slab-and-gather implementation. The built-in readers lower through NumPy system memory and support sources -whose basic reads can be converted there. They do not implicitly transfer -device arrays; a device source needs an explicit custom reader that transfers -into the supplied system-memory output. Derived views and parts share their +whose basic reads can be converted there. A device source that refuses NumPy +conversion needs a custom reader that transfers into the output buffer. Derived views and parts share their reader and part views may be materialized concurrently, so stateful readers must synchronize their own mutable state. Every public `Partition.view.transform` directly maps that view's zero-origin coordinates into its raw `Partition.view.array`, including for non-first partitions. `Partition.projection.chunk_transform` intentionally stays local to -the selected chunk. During materialization the reader receives both frames in -one `ReadContext`: the public global transform in `context.transform` and the -same local plan in `context.projection`. +the selected chunk. During parent materialization (`view.result(parts=parts)`) the reader receives +both frames in one `ReadContext`: the public global transform in `context.transform` and the +same local plan in `context.projection`. Use `part.result()` to execute a +partition independently with both frames. Direct `part.view.result()` calls +resolve the general view with no projection. ::: zarr_indexing.lazy_array diff --git a/packages/zarr-indexing/docs/api/reader.md b/packages/zarr-indexing/docs/api/reader.md index 39129c4d47..80cd7b7ce7 100644 --- a/packages/zarr-indexing/docs/api/reader.md +++ b/packages/zarr-indexing/docs/api/reader.md @@ -12,7 +12,8 @@ ownership. `Reader.read_into(source, context, out)` receives a `ReadContext` whose `transform` maps zero-origin output-buffer coordinates to global coordinates in `source`, with `context.transform.domain.shape == out.shape`. Its optional -`projection` is the existing plan for a partitioned read. The projection's +`projection` is the existing plan for a partitioned read. Both parent +assembly and independent `Partition.result()` calls supply that projection. The projection's `chunk_transform` remains chunk-local, its `cell_transform` describes result placement, and its `chunk_domain` describes the grid cell. The global read transform and the projection's chunk transform deliberately use different @@ -34,7 +35,7 @@ class RecordingReader: self.calls = [] def read_into(self, source, context, out, /): - self.calls.append((source, context, out)) + self.calls.append((source, context, out.shape, out.dtype)) self.inner.read_into(source, context, out) @@ -44,7 +45,8 @@ view = LazyArray.from_numpy(array).with_reader(outer) values = view.result() ``` -Both wrappers observe the same three objects, in outer-to-inner order. This +Both wrappers observe the same arguments, in outer-to-inner order, and log +output metadata without retaining the output buffer. This delegation pattern supports policies such as logging and caching without library-defined wrapper primitives. diff --git a/packages/zarr-indexing/docs/design-notes.md b/packages/zarr-indexing/docs/design-notes.md index 59eef4dacd..cb80e5ba19 100644 --- a/packages/zarr-indexing/docs/design-notes.md +++ b/packages/zarr-indexing/docs/design-notes.md @@ -16,7 +16,7 @@ visual guide owns the mechanics of ## Relationship to TensorStore The core is [TensorStore's](https://google.github.io/tensorstore/index_space.html) -index-transform model, reimplemented in Python against NumPy. The visual guide +index-transform model, implemented here in Python against NumPy. The visual guide introduces the shared model in [Coordinates are addresses](guide/index.md#coordinates-are-addresses) and [Lazy views compose](guide/index.md#lazy-views-compose); the comparison here is @@ -28,13 +28,14 @@ about the deliberately matching semantics: - **Slice semantics.** Slice bounds are literal domain coordinates: no clamping, no negative wrapping, non-empty intervals must be contained in the domain, and a strided slice's domain origin is `trunc(start/step)` rounded - toward zero. Every one of those rules was executed against tensorstore 0.1.84 - and is pinned in `tests/test_tensorstore_parity.py`. -- **The wire format.** A canonical [ndsel](ndsel.md) transform body is, - field-for-field, a TensorStore `IndexTransform` minus the `kind` - discriminator, and `tests/test_ndsel_tensorstore.py` loads our bodies into - `tensorstore.IndexTransform(json=...)` and round-trips them back through our - engine layer. + toward zero. `tests/test_tensorstore_parity.py` compares the enumerated cases with + TensorStore when that optional dependency is installed. +- **The wire format.** [ndsel](ndsel.md) uses TensorStore's domain and + output-map field names for transform bodies, with an additional `kind` + discriminator. `tests/test_ndsel_tensorstore.py` checks interoperability for + the tested cases. Their validation rules differ, and loading and + re-emitting a message through the engine can normalize or discard metadata; + see [lowering to a transform](ndsel.md#lowering-to-a-transform). - **Chunk partitioning.** Both factor a transform over a grid before visiting any cell, rather than intersecting the whole transform with each chunk. TensorStore's `IndexTransformGridPartition` holds strided sets and index @@ -47,17 +48,19 @@ about the deliberately matching semantics: components within a mixed request. Both derive the per-chunk transforms from the partition ([the guide](guide/index.md#a-plan-is-a-product-of-per-axis-tables) shows the tables). TensorStore keeps strided sets implicit, while this - library materializes their per-axis rows for vectorized consumers. Diagonals are rejected here; supporting them needs a strided set per - *input* dimension spanning every storage axis that reads it, TensorStore's - representation. + library materializes their per-axis rows for vectorized consumers. Pure + affine diagonals need grouping by input dimension; mixed affine/index-array + dependencies need joint partitioning. TensorStore classifies a connected + component containing index-array edges as an index-array set + ([source](https://github.com/google/tensorstore/blob/66b2ce5290fa2ec5c8019682391421062ce767a2/tensorstore/internal/grid_partition.h#L58-L67)). Four deliberate differences: | | TensorStore | `zarr-indexing` | | --- | --- | --- | -| Dialect | One strict dialect everywhere: literal coordinates, no negative wrapping | The algebra keeps that dialect; each public boundary picks its own. [`LazyArray`](api/lazy_array.md) speaks positional NumPy, `zarr.Array.lazy` speaks literal. [`zarr_indexing.boundary`](api/boundary.md) is the translation | +| Dialect | Coordinate indices are literal; negative coordinates do not wrap | The algebra keeps that dialect; each public boundary picks its own. [`LazyArray`](api/lazy_array.md) speaks positional NumPy, `IndexTransform` speaks literal. [`zarr_indexing.boundary`](api/boundary.md) is the translation | | Scheduling | An internal C++ scheduler owns concurrency and chunk ordering | [`parts()`](api/lazy_array.md) exposes the partition structure so the caller's own scheduler — dask, a thread pool, a task queue — drives it | -| Wire format | Implementation-defined JSON, specified by what the implementation accepts | [ndsel](ndsel.md) is spec-first, with a vendored language-agnostic conformance corpus every implementation runs | +| Wire format | [Documented JSON schema](https://google.github.io/tensorstore/index_space.html#index-transform) | [ndsel](ndsel.md) is spec-first, with a vendored conformance corpus exercised by this implementation | | Backends | A driver ecosystem (zarr, N5, neuroglancer, GCS, …) built into the library | No drivers. The default reader needs `shape`, `dtype`, basic integer/slice indexing, and selected slabs convertible to NumPy system memory; other backends use explicit custom readers. A device reader owns transfer into the supplied system-memory output | The mechanics of a @@ -74,19 +77,18 @@ and caller-supplied grid; it does not own reads, writes, buffers, locks, or scheduling. Zarr can therefore plan reads against an inner codec-chunk grid and writes against an atomic shard grid; napari or dask can turn the same projections into tasks without putting a dask dependency in this package. -`coverage` is relative to that selected grid: `full` proves a blind replacement -safe, `partial` proves it is not, and `unknown` conservatively covers fancy -selections whose duplicates would require additional work to classify. - -The comparison also runs the other way. TensorStore is a mature, heavily -optimized C++ system whose performance this library cannot approach. Independent strided planning -here is per axis, but each materialized `ChunkProjection` is -Python-level bookkeeping over NumPy — two domains, two transforms and the -projection itself — so the per-part overhead of the object view is -significant; a consumer that reads the partition's tables directly pays no -per-chunk object construction. This library is small and depends on nothing beyond -NumPy, so the algebra can be adopted by a Python project that wants the model -without the C++ runtime. +`coverage` describes selection coverage relative to that grid. A `full` +classification can help a writer avoid reading old values, but does not by +itself prove that a write is safe: encoding requirements, conflicts, duplicate +semantics, and concurrency remain consumer responsibilities. `unknown` means +the planner has not established complete or partial coverage. + +This implementation performs Python-level bookkeeping over NumPy. This page +provides no benchmark establishing a general performance ordering against +TensorStore; costs depend on the selection and execution backend. + +The partition tables can be consumed without constructing a `ChunkProjection` +for each chunk. Materializing projections adds Python object construction. ## Bounding-box selections vs query selections @@ -102,11 +104,11 @@ and the coordinates it touches form a regular lattice. Basic indexing produces one, and composing basic indexing with basic indexing keeps one. **A query** is a transform with at least one `ArrayMap` — an explicit lookup -table of coordinates. It costs `O(n)` to store, it has no locality (the -coordinates may repeat, reverse, or scatter arbitrarily), and intersecting it -with a region means scanning it. `oindex`, `vindex`, and boolean masks all -produce one, and once an axis is a query, subsequent basic indexing cannot make -it a box again. A second query composes onto any axis of an existing one — +table of coordinates. Its stored coordinate arrays cost space proportional to their stored size. +Coordinates may repeat or scatter, but can also be contiguous and local. +Current query-resolution paths inspect these arrays. Fancy indexing can produce +a query, but singleton or constant selections can collapse to `ConstantMap`; +subsequent indexing can therefore make a query affine again. A second query composes onto any axis of an existing one — including the axes it merely broadcasts along — by evaluating the existing lookup tables at the new coordinates. @@ -116,9 +118,10 @@ planning and materialization. [ndsel](ndsel.md) encodes the same split in its message kinds: `point`, `box`, and `slice` desugar to constant and affine output maps and are always boxes; -`points` desugars to `index_array` maps, and a `transform` body is a box -exactly when none of its output maps carries an `index_array`. A consumer can -therefore classify a selection off the wire without materializing anything: +`points` desugars to `index_array` maps. A transform without index-array maps +is a box in the engine’s structural classification. Loading can further +simplify degenerate index arrays to constants, so an arbitrary incoming body +with `index_array` fields need not remain a query. For example: ```python from zarr_indexing import IndexTransform @@ -134,16 +137,13 @@ gather.to_json()["output"][0] # 'index_array_bounds': ['-inf', '+inf']} ``` -The distinction matters to consumers of a selection. A box can be tiled into -rectangular dask chunks or passed to a viewer or tile server that only accepts -rectangles; a query cannot, and has to be resolved into a gather. A box can also -be served as a single strided slab read, but the read has to be strided: reading -its bounding box and discarding the rest transfers proportionally more data as -soon as any stride exceeds 1. The two also behave differently under -partitioning: a box touches a regularly-spaced run of parts, in increasing -order, each at most once — a stride larger than a part's extent skips parts -outright, so the run is not contiguous — while a query can touch any subset of -them, in any order, more than once. +The representation helps a consumer choose a lowering strategy. Independent +affine axes can often be read with slices plus reversal, permutation, or +broadcasting. Arbitrary affine maps can also express diagonals, so the absence +of `ArrayMap` alone is not proof of a rectangular slab. Queries may be lowered +through gathers or covers, and can sometimes simplify to slices. Chunk plans +group selected coordinates by chunk while preserving result placement; repeated +coordinates do not imply repeated visits to the same chunk. [`LazyArray`](api/lazy_array.md) exposes the category directly: @@ -173,16 +173,20 @@ gather.shape # (3, 80) `bounding_box()` is defined for both: it is the hull, the smallest interval per storage dimension containing every coordinate the selection reaches. `strides()` is defined only for a box and gives the step per dimension. -Together the two describe a box selection completely. - -Both are needed, because a box is dense in its hull only when every stride is -1. The slab above spans a 40x77 hull over the 40x20 cells it selects, so a +These summaries omit traversal direction, input-axis correspondence, and +result layout. For example, forward and reversed views have identical bounds +and stride magnitudes but different ordered results. Use the transform for the +complete selection. + +For independent axes with multiple selected coordinates, a stride magnitude +greater than one leaves gaps in the hull. Singleton axes are an exception, +and a query can also cover every cell of its hull. The slab above spans a 40x77 hull over the 40x20 cells it selects, so a consumer that issued one rectangular read of the hull and discarded the rest would transfer 3.85x the data. A query's hull is looser still and carries no stride at all: 88 rows of hull over three selected rows. An empty *box* touches no coordinate to report an interval around, so `bounding_box()` is `None` while `strides()` still answers — the step is a property of the selection's shape, not -of the region it reaches. Only a query returns `None` from both. +of the region it reaches. An empty query returns `None` from both. There is deliberately no separate `BoxView` type today. A statically-typed rectangular-only view is a plausible next step, but it should be introduced by @@ -248,8 +252,8 @@ could accept and finishing the rest elsewhere. A reader lowers the complete transform and can compose through delegation instead. This resembles [zarrita.js store extensions](https://zarrita.dev/packages/zarrita.html), where storage-specific behavior is an explicit extension point rather than an -inferred array capability. The implementation remains independently authored: -no code is shared with TensorStore, xarray, or zarrita.js. +inferred array capability. This is an architectural analogy, not a claim of API or implementation +compatibility. ## Current scope @@ -263,7 +267,7 @@ re-bases every view to origin 0, so the positional dialect never exposes it; a caller working with `IndexTransform` directly will see it, and re-bases explicitly with `translate_domain_to` for NumPy-shaped coordinates. -Fancy selections compose without restriction: a second `oindex`/`vindex`/mask +Supported fancy selections compose across already-fancy views: a second `oindex`/`vindex`/mask step may land on any axis of an already-fancy view, including axes an existing index array merely broadcasts along, so `lazy.oindex[[2, 0], :].lazy.oindex[:, [1, 3]]` selects the outer product it @@ -273,29 +277,26 @@ which evaluates the existing lookup tables at the new coordinates — rather tha rewritten in place. Resolution classifies the result by structure (`index_array_structure`): pure per-axis outer products keep the orthogonal resolvers, and everything else — correlated maps, mixtures, index arrays -sharing an input axis (a diagonal gather, reachable only by hand-building a -transform) — takes the general reader/intersection path. Chunk planning +sharing an input axis (as in paired vectorized coordinates) — takes the general reader/intersection path. Chunk planning factors index arrays into connected dependency components before flattening, so independent groups do not expand one another. Vectorized selection preserves broadcast singletons to retain those dependencies. -Three limits remain, all intentional and all expected to be lifted: +Some current limits are: -- **Affine diagonals.** A hand-built transform in which two output maps read - one input dimension — two slice maps, or a slice map and an orthogonal index - array — is rejected at planning with `ValueError`; a correlated index array - varying over a dimension a slice map also reads is rejected with - `NotImplementedError`. No selection dialect produces either. Supporting them - needs a strided set per *input* dimension spanning all dependent storage - axes, TensorStore's connected-component representation. *Planned.* +- **Shared affine dependencies.** Planning rejects two affine output maps + sharing an input axis with `ValueError`. An index array sharing a varying + input axis with an affine map takes the general classification and raises + `NotImplementedError`. Pure affine diagonals would need grouping dependent + storage axes by input dimension; mixed components need joint partitioning. - **Finite explicit bounds only.** `IndexDomain` has no implicit or unbounded dimensions; the message layer will normalize a body with `"-inf"`/`"+inf"` bounds, but the engine layer refuses to lower one into a transform. - TensorStore supports both. *Planned.* + TensorStore supports both. - **Labels are carried, not propagated.** `IndexDomain` holds optional dimension labels and the wire format round-trips them, but indexing operations build new domains without them, so a label does not survive a - slice. *Planned.* + slice. ## Selection to chunk operations diff --git a/packages/zarr-indexing/docs/guide/index.md b/packages/zarr-indexing/docs/guide/index.md index 68192f58cb..e77a812816 100644 --- a/packages/zarr-indexing/docs/guide/index.md +++ b/packages/zarr-indexing/docs/guide/index.md @@ -216,12 +216,17 @@ description; the assertion's call to `result()` is the first operation in the example that materializes the selected data. !!! warning "Stop here: the materialization boundary" - Indexing through `.lazy[...]` never reads. These do: + Indexing through `.lazy[...]` composes a selection without reading source values. + These operations request values: - `result()` - eager indexing of the wrapper: `view[...]` - - `numpy.asarray(view)`, or passing the view to any NumPy function - (`numpy.add(view, 1)` converts, and therefore materializes, the view) + - `numpy.asarray(view)` and NumPy operations that convert the view + (`numpy.add(view, 1)` does so; `numpy.shape(view)` and `numpy.ndim(view)` + can use metadata without reading values) + + Dask tokenization may also inspect values, depending on the wrapped source + and tokenization path. Python arithmetic such as `view + 1` raises `TypeError` instead: this wrapper defers indexing, not a general compute graph. @@ -384,8 +389,10 @@ each bundles a sub-view of the request (`.view`), that chunk's projection Within one `Partition`, the frames divide: `Partition.view.transform` is a different, global transform — it maps the part view directly into the raw wrapped source — while only `Partition.projection.chunk_transform` uses -zero-origin chunk-local coordinates. Readers receive both so the global -source address and the local planning frame cannot be confused. +zero-origin chunk-local coordinates. Parent assembly passes both frames to +the reader. Independently scheduled `part.result()` calls supply the same +context. Calling `part.view.result()` instead resolves the view without the +partition record, so its context has `projection=None`. | Projection field | What its output coordinates mean | | --- | --- | diff --git a/packages/zarr-indexing/docs/guide/integrations.md b/packages/zarr-indexing/docs/guide/integrations.md index f787d11f6c..a0bc721175 100644 --- a/packages/zarr-indexing/docs/guide/integrations.md +++ b/packages/zarr-indexing/docs/guide/integrations.md @@ -3,8 +3,9 @@ For the complete path from indexing syntax to chunk coordinates, local selectors, and result positions, start with [From a selection to chunk operations](selection-flow.md). -This package supplies indexing plans. It does **not** supply scheduling, -caching, codecs, or async orchestration. A consumer decides when projections +The core package supplies indexing plans and synchronous readers. It does not +provide a general scheduler, codec pipeline, or async execution engine. A +synchronous cache is included as an example. A consumer decides when projections run, how decoded chunks are obtained, and where completed values are retained. An `IndexTransform` says which source values belong in a result; a `Reader` lowers that complete transform for one backend. The reader does not choose @@ -61,21 +62,21 @@ afterward, so accessing a row does not recalculate every point. ## One slab read or many part reads -A backend with its own native subset read — a Rust or C zarr implementation, -a database, an HTTP range endpoint — resolves a **dense box** (`is_box` with -every stride 1) best as a single read: hand it the whole selection and let it -dispatch to chunks, decode in parallel, and partial-decode shards on its own -side of the boundary. Splitting that read along this library's partitioning -only adds round-trips. Every **other** selection — a strided box, an `oindex` -or `vindex` gather — is where the partitioning earns its keep. The **cover** -of a read is the smallest step-1 slab enclosing every coordinate it needs; -partitioned, each part's cover is bounded by that part's box, so a sparse -selection can never force one read of its whole bounding hull (the smallest -rectangle containing every selected coordinate — a thousand rows for the two -of `oindex[[0, 999]]`). +A backend with an efficient native subset operation may benefit from receiving +one complete dense selection so it can choose its own chunk dispatch. Other +backends may benefit from partitioning, including for strided or fancy +selections. The tradeoff depends on the backend, chunk layout, latency, memory, +and selection; a single read is not universally fastest. + +A read's **cover** is the smallest unit-step slab enclosing its coordinates. +With this package's basic readers, partitioning limits each source read to the +part's selected cover. This may reduce over-reading, but a partition that spans +the source can still require the entire hull. Custom readers choose their own +source operations under the reader contract. The composed view carries enough to make that call at materialization time, -and re-partitioning is a pure setter, so the policy is three lines: +and `with_parts()` returns a view with a new partitioning. This example uses +unit strides as a sufficient condition for its independently mapped selections: ```python --8<-- "snippets/integrations.py:dense-box-repartition" @@ -87,20 +88,19 @@ the dense box becomes exactly one backend call. Both regimes go through ### Sources that accept only unit-step slices -The default `basic_reader` pushes strided and descending selections down as -positive-step slices, which reads the minimum but assumes the source accepts -any step. Many backends do not: FFI bindings and range requests often -support nothing but `slice(start, stop, 1)`. Select -[`unit_step_reader`][zarr_indexing.reader.UnitStepReader] for such a source -and every key it receives is an ascending unit-step slice per axis, with +For affine selections, `basic_reader` uses positive-step slices and applies +reversal or layout changes in memory. Fancy selections can require reading a +cover containing unselected values. The source must accept the emitted steps. +Use [`unit_step_reader`][zarr_indexing.reader.UnitStepReader] for a source that +accepts only unit steps. Every key it receives is an ascending unit-step slice per axis, with strides, reversals, and gathers applied to the in-memory block instead: ```python view = LazyArray(source).with_reader(unit_step_reader) ``` -A strided selection then over-reads its cover by the stride factor, which the -partitioning above bounds by one part. +For strided selections, the ratio of cover cells to selected cells depends on +stride, length, and endpoint alignment. Partitioning can reduce that cover. ## napari-like consumer @@ -205,13 +205,12 @@ decoded chunks. Every read delta follows directly from the viewport request: | 4 | `image[1:5, 2]` | `(0, 0)` | `(0, 0)`, `(1, 0)` | The evicted chunk is reloaded while the required ready chunk is retained. | | 5 | `image[3:5, 4:6]` | `(1, 1)` fails; no repeated read; `(1, 1)` succeeds after retry | `(0, 0)`, `(1, 1)` | Failure is retained until explicit retry; the repaired source then returns `[[28, 29], [36, 37]]`. | -Chunks required by an active request are pinned through assembly, so a request -may temporarily span more chunks than the steady-state capacity. Capacity is -counted in decoded chunks—not records or bytes—and eviction occurs only after -all requested values have been placed. Because pinning and materialization use -the same prepared tuple, those lifecycle decisions cannot drift from the parts -that are actually read, and the cache never has to infer or reconstruct a -projection. +The example defers eviction until a successful outermost request finishes, so +it can temporarily exceed capacity during assembly. Capacity counts decoded +chunks, not bytes, event records, or temporary arrays. Failed requests skip +that eviction step. The example assumes an unchanged source and one source/grid +per reader; it does not implement invalidation or synchronization for concurrent +requests. The prepared tuple supplies the projections used for each read. The event log makes the failure boundary equally explicit: diff --git a/packages/zarr-indexing/docs/guide/patterns.md b/packages/zarr-indexing/docs/guide/patterns.md index 9d22c4eb4e..f2f2ddbe2a 100644 --- a/packages/zarr-indexing/docs/guide/patterns.md +++ b/packages/zarr-indexing/docs/guide/patterns.md @@ -302,10 +302,11 @@ so they equal the zero-origin models after `translate_domain_to`: --8<-- "snippets/indexing_patterns.py:indexing-patterns" ``` -`LazyArray` adds nothing to these semantics: it is a regular array-like API -whose `.lazy`, `.lazy.oindex`, and `.lazy.vindex` accessors compile the same -dialects to the same transforms — the only difference is the return type, a -view instead of an array. The test suite holds the wrapper to this matrix. +`LazyArray` exposes the transform machinery through a positional array-like +API. Its `.lazy`, `.lazy.oindex`, and `.lazy.vindex` accessors return views and +normalize positions before composition. This boundary differs from the literal +coordinate semantics of `IndexTransform`, as the following table shows. The +executable matrix checks the documented cases, not every possible NumPy expression. ## Positions vs literal coordinates diff --git a/packages/zarr-indexing/docs/guide/selection-flow.md b/packages/zarr-indexing/docs/guide/selection-flow.md index 571baa3869..749a9c36ae 100644 --- a/packages/zarr-indexing/docs/guide/selection-flow.md +++ b/packages/zarr-indexing/docs/guide/selection-flow.md @@ -171,7 +171,7 @@ pairs: | `(0,)` | `[1]` | `[1]` | | `(1,)` | `[2, 2, 0]` | `[0, 2, 3]` | -Chunk `(1,)` is fetched once, but its local value `2` contributes to two result +Chunk `(1,)` is fetched once, but the value at local coordinate `2` contributes to two result positions. Chunk visitation order need not be result order; `out_selection` restores the requested arrangement. @@ -188,6 +188,9 @@ than one. Repeated coverage is not full; fancy coverage remains conservative. Zarr's merge operation can skip a read for a full data-extent write and allocate a fill-valued codec buffer when the selected data is smaller than that buffer. +That shortcut also requires the consumer's expected value layout and order; +complete coverage by a reversal or reordered gather alone does not establish +that the supplied buffer can be copied directly into codec order. Touching every chunk alone does not prove full coverage of each chunk. Planning does not fetch bytes, decode buffers, choose concurrency, or define diff --git a/packages/zarr-indexing/docs/index.md b/packages/zarr-indexing/docs/index.md index 87e5a4b7a1..598282b698 100644 --- a/packages/zarr-indexing/docs/index.md +++ b/packages/zarr-indexing/docs/index.md @@ -5,8 +5,7 @@ the *declaration* of an array indexing expression from the result of that expres Developed for use in [`zarr`](https://zarr.readthedocs.io). -Inspired by [TensorStore](https://google.github.io/tensorstore/), which pioneered -the approach used here. +Inspired by [TensorStore's index-transform model](https://google.github.io/tensorstore/index_space.html). ## Install @@ -28,8 +27,9 @@ you want its values: --8<-- "snippets/canonical_slice.py:landing-quickstart" ``` -Nothing is read until the `result()` call, however many selections are -composed. [Lazy views compose](guide/index.md#lazy-views-compose) shows how +Composing these selections does not read source values; the example reads them +at `result()`. Construction inspects source metadata, and Dask tokenization can +inspect source values. [Lazy views compose](guide/index.md#lazy-views-compose) shows how the chain stays one description, and where the materialization boundary is. ## Learn more diff --git a/packages/zarr-indexing/docs/ndsel.md b/packages/zarr-indexing/docs/ndsel.md index 94971d49bf..090dce64c4 100644 --- a/packages/zarr-indexing/docs/ndsel.md +++ b/packages/zarr-indexing/docs/ndsel.md @@ -14,11 +14,13 @@ before a transform is serialized. `zarr-indexing` implements ndsel in two layers | Layer | Module | Depends on | Job | | --- | --- | --- | --- | -| Message | [`zarr_indexing.messages`](api/messages.md) | stdlib only | JSON in, canonical JSON out. Validates and desugars. Never rounds, clamps, or drops information. | +| Message | [`zarr_indexing.messages`](api/messages.md) | stdlib only | Validates and desugars JSON, removing redundant constant-map fields. | | Engine | [`zarr_indexing.json`](api/json.md) | NumPy | Lowers a *canonical* body into an in-memory [`IndexTransform`](api/transform.md), and back. | Constraints that only make sense for a real array — finite bounds, index -arrays as `ndarray`s — live in the engine layer and nowhere else. As a result, +arrays as `ndarray`s — are checked during engine lowering. The message layer +also limits input rank to 32 and checks affine input-dimension references. +These checks are stricter than the draft's required message validation. As a result, `messages` normalizes a message with `"-inf"` bounds that `IndexTransform.from_json` refuses to lower. @@ -47,9 +49,13 @@ normalize_ndsel({"kind": "box", "inclusive_min": [10, 5], "shape": [40, 1]}) ``` Normalization is idempotent: re-tag the output with `kind: "transform"` and -normalizing it again returns the same body. Because the canonical body is -field-for-field a TensorStore `IndexTransform` minus `kind`, a normalized -message loads directly into `tensorstore.IndexTransform(json=...)`. +normalizing it again returns the same body. The canonical body uses +TensorStore's `IndexTransform` field vocabulary, but normalization does not +guarantee acceptance by TensorStore. For example, the message layer leaves +index-array content unchecked, whereas TensorStore validates it; TensorStore +also requires unique nonempty labels and restricts finite index values to +`[-(2**62 - 2), 2**62 - 2]`. See +[TensorStore's index-space constraints](https://google.github.io/tensorstore/index_space.html). Both entry points raise [`NdselError`](api/messages.md#zarr_indexing.messages.NdselError), which @@ -64,12 +70,13 @@ Four are shorthands; the fifth is the canonical form itself. | `kind` | Fields | Selects | | --- | --- | --- | | `point` | `coords` | A single element. Normalizes to rank 0 with one `constant` output map per dimension. | -| `box` | `inclusive_min`, one of `exclusive_max` / `inclusive_max` / `shape`, `labels` | A rectangular region. Exactly one upper-bound spelling may appear. | +| `box` | `inclusive_min`, one of `exclusive_max` / `inclusive_max` / `shape`, `labels` | A rectangular region. At most one upper-bound spelling may appear; omission gives implicit positive infinity. | | `slice` | `start`, `stop`, `step`, `labels` | A strided region, one Python-style slice per dimension. | | `points` | `coords` (a list of coordinate rows) | An explicit list of points — the `vindex` case. Normalizes to one `index_array` output map per dimension over a shared rank-1 input domain. | | `transform` | `input_rank`, `input_inclusive_min`, one of the three `input_*` upper bounds, `input_labels`, `output` | The full canonical form. | -Value rules the message layer enforces throughout: every integer is a 64-bit +Value rules for validated fields (excluding verbatim `index_array` payloads +and discarded constant-map fields): every integer is a 64-bit signed value; JSON booleans are **not** integers (Python's `isinstance(True, int)` is guarded against explicitly); the `"-inf"` / `"+inf"` sentinels are legal only in bound positions; and an implicit bound is the @@ -81,17 +88,21 @@ normalization intact. The engine layer converts between canonical bodies and `IndexTransform`s: ```python -from zarr_indexing import IndexTransform +from zarr_indexing import IndexTransform, normalize_ndsel +canonical = normalize_ndsel({"kind": "box", "shape": [2, 3]}) t = IndexTransform.from_json(canonical) -t.to_json() == canonical +assert t.to_json() == canonical ``` `IndexDomain` carries the same pair for a bare domain body, and each output map kind has a `to_json`; `output_index_map_from_json` dispatches the wire's -tagged union back to the right kind. +structurally discriminated union back to the right kind. Exact JSON equality +in this example is not a general round-trip guarantee: implicit flags are +removed, finite `index_array_bounds` are not retained or enforced by the +engine, and degenerate array maps are collapsed. -Two engine constraints apply here and only here. A canonical body carrying a +A canonical body carrying a `"-inf"` or `"+inf"` bound cannot be lowered — an `IndexDomain` addresses a finite array — so `IndexTransform.from_json` raises. And implicit bounds lower *by value*: the `[n]`-bracket flag is a message-layer concern, and the engine @@ -102,21 +113,17 @@ keeps only the integer. ndsel and TensorStore both **reject** an output map that carries both `input_dimension` and `index_array`. The in-memory [`ArrayMap`](api/output_map.md#zarr_indexing.output_map.ArrayMap), though, -records an `input_dimension` to pin the axis an orthogonal (`oindex`) array -varies over. The serializer bridges that gap in both directions: +records its dependency axes in its full-rank array shape, with no +`input_dimension` field: - **On serialize**, a non-degenerate `index_array` map is emitted *without* `input_dimension`. -- **On load**, the in-memory `input_dimension` is reconstructed from the - full-rank array's dependency axes — its non-singleton axes. An array that - solely owns a single non-singleton axis is orthogonal; arrays that share - non-singleton axes, or vary over several, are correlated (`vindex`), and get - `input_dimension = None`. A single 1-D array over a rank-1 domain is - inherently ambiguous between the two flavors and reconstructs as - orthogonal, which is behaviorally identical in that case. - -There is one deliberate exception, and it is the only place a round trip changes -representation rather than preserving it. An all-singleton `index_array` — size +- **On load**, dependency axes are the full-rank array's non-singleton axes. + Maps sharing these axes describe correlated coordinates. The engine also + accepts lower-rank nonempty arrays by prepending singleton axes; that + convenience is not a guarantee of compatibility with other ndsel consumers. + +An all-singleton `index_array` — size 1 — selects the same coordinate regardless of the input, so it is collapsed to a `constant` map on serialize: @@ -137,6 +144,11 @@ The transform is still valid and the output shape is unchanged. A length-1 `oindex` selection therefore round-trips behaviorally (an `ArrayMap` comes back as a `ConstantMap`) rather than by object identity. +Empty index arrays also serialize as constant maps with offset zero: the +empty input domain carries the fact that no coordinates are selected. This +avoids losing trailing shape information in JSON when an array has a leading +zero-length axis. + ## Conformance The package is checked against the language-agnostic ndsel conformance corpus, @@ -145,8 +157,9 @@ vendored unmodified under — one JSON file per message kind plus `errors.json`, with the source commit recorded in `PROVENANCE.md`. Each fixture is either a *success* case (`input` + expected `normalized` body) or an *error* case (`input` + expected -reason code), and an implementation is conformant iff `normalize` reproduces -every one. `tests/test_conformance.py` runs the whole corpus as one +reason code). Matching every fixture establishes corpus conformance; the +fixtures do not prove correctness for every possible input or universal +TensorStore compatibility. `tests/test_conformance.py` runs the whole corpus as one parametrized test per fixture, so a corpus update reports failures fixture by fixture rather than as a single opaque assertion. diff --git a/packages/zarr-indexing/docs/release-notes.md b/packages/zarr-indexing/docs/release-notes.md index 767cd56a84..d0687d8758 100644 --- a/packages/zarr-indexing/docs/release-notes.md +++ b/packages/zarr-indexing/docs/release-notes.md @@ -1,5 +1,6 @@