diff --git a/packages/zarr-indexing/CONTRIBUTING.md b/packages/zarr-indexing/CONTRIBUTING.md index 632e24929f..bcbe51c079 100644 --- a/packages/zarr-indexing/CONTRIBUTING.md +++ b/packages/zarr-indexing/CONTRIBUTING.md @@ -8,16 +8,16 @@ just test # run the test suite (extra args go to pytest) just lint # ruff, same invocation as CI just typecheck # pyright, same invocation as CI just docs-check # strict build of the docs site -just check # all of the above +just check # the checks above plus TensorStore parity just docs-serve # serve the docs site locally ``` -Run them from this directory, or from anywhere in the repository as +Run them from this directory, or from the repository root as `just packages/zarr-indexing/`. -The test recipe runs against the workspace-root environment, because the -chunk-resolution tests exercise this package against `zarr`'s chunk grids and -`zarr` is deliberately not a dependency of this package. +The test recipe layers this package into the repository-root environment. +Chunk-resolution tests use this package’s own grids; the Dask example also +uses `zarr`, which is not a dependency of the base indexing package. ## License diff --git a/packages/zarr-indexing/README.md b/packages/zarr-indexing/README.md index 4328de1cc1..c1b71058a1 100644 --- a/packages/zarr-indexing/README.md +++ b/packages/zarr-indexing/README.md @@ -5,19 +5,19 @@ Composable, lazy coordinate transforms for Zarr array indexing. Documentation: This package implements TensorStore-inspired index transforms. The core idea: -every indexing operation (slicing, fancy indexing, etc.) produces a coordinate -mapping from user space to storage space. These mappings compose lazily — no -I/O until you explicitly read or write. +each supported indexing operation (slicing, fancy indexing, etc.) produces a coordinate +mapping from user space to storage space. These mappings compose without reading selected source values. `LazyArray` +materializes them on request; the transform algebra itself performs no source I/O. Key types: -- `LazyArray` — wraps a system-memory/basic-indexing source and adds a `.lazy` - accessor: `LazyArray.from_numpy(numpy_array).lazy[10:50, ::2].lazy.oindex[[3, 1, 1], :]` +- `LazyArray` — wraps a system-memory/basic-indexing source with lazy indexing: + `LazyArray.from_numpy(numpy_array)[10:50, ::2].oindex[[3, 1, 1], :]` composes a transform and returns a new view without reading data, and `result()` materializes it into owned system memory. `LazyArray(source)` uses - the conservative basic reader; `from_numpy` explicitly selects NumPy's - optimized reader. Device arrays require an explicit custom reader responsible - for transferring values into the supplied system-memory output buffer. + the basic reader; `from_numpy` selects `numpy_reader`, which currently uses + the same slab-and-gather implementation. Device sources that refuse NumPy + conversion need a custom reader to transfer values into the output buffer. - `Reader` — the explicit backend execution boundary: transforms say which values belong in the result, while readers say how a backend obtains them - `IndexDomain` — a rectangular region of integer coordinates @@ -32,9 +32,11 @@ Key types: dimension can depend on the input - `compose` — chain two transforms into one -The package depends only on NumPy and the standard library; it does not import -`zarr`. It is developed in the [zarr-python](https://github.com/zarr-developers/zarr-python) -repository and consumed by `zarr` to resolve array indexing operations. +The base package depends on NumPy and the standard library; its optional testing +module also requires Hypothesis. The package does not import `zarr`. It is developed +in the [zarr-python](https://github.com/zarr-developers/zarr-python) repository, +and its examples include reading Zarr arrays through Dask. Installing it does not +replace Zarr's indexing implementation. ## Installation diff --git a/packages/zarr-indexing/benchmarks/README.md b/packages/zarr-indexing/benchmarks/README.md index 28ef3e6a16..e5d39b1a89 100644 --- a/packages/zarr-indexing/benchmarks/README.md +++ b/packages/zarr-indexing/benchmarks/README.md @@ -22,6 +22,19 @@ whether input arrays and transform construction are included. Repeated local column access measures cache reuse, which trades allocation against retained memory. Bounded coordinate batches avoid constructing the full coordinate array. +The current script reports `peak_mib` as the incremental peak tracked by +`tracemalloc` during a separate invocation after the timed calls. It is not +process RSS or total retained memory. Inputs and most transforms are constructed +before measurement. The case walks construct plans and consume projections +without retaining them; `all_coordinates` materializes the complete coordinate +array. There is no bounded-batch coordinate workload in this script. + +`local_rows` reuses one prepared table: its first timed invocation populates the +local-coordinate cache, while later timed invocations and the allocation probe +reuse that cache. Its median mixes a cold first call with warm calls, and its +reported peak excludes the already-retained cache. A separate fresh-table +measurement would be needed to quantify cold cache construction. + These scripts measure planning rather than codec or storage throughput. Repeat measurements with alternating operation order before interpreting small timing differences. Preserve raw benchmark output as an experiment artifact rather than diff --git a/packages/zarr-indexing/benchmarks/chunk_planning.py b/packages/zarr-indexing/benchmarks/chunk_planning.py index 934d68dcbc..d7eadf248d 100644 --- a/packages/zarr-indexing/benchmarks/chunk_planning.py +++ b/packages/zarr-indexing/benchmarks/chunk_planning.py @@ -25,6 +25,11 @@ def measure(operation: Callable[[], Any], repeats: int) -> dict[str, float]: + """Time repeated calls, then trace one additional call's incremental peak. + + State retained by earlier calls can affect the extra call; this is not a + measurement of process RSS or all memory owned by the operation's inputs. + """ samples = [] for _ in range(repeats): start = time.perf_counter() @@ -84,6 +89,8 @@ def read_local_rows() -> None: for row in range(len(table)): table.local[table.run(row)] + # The first timed call fills table.local; later calls, including the traced + # allocation probe, reuse it. The table and retained cache are not re-created. results["local_rows"] = measure(read_local_rows, args.repeats) part = plan_chunks( IndexTransform.from_shape((100, 100, 100)), diff --git a/packages/zarr-indexing/changes/4345.bugfix.md b/packages/zarr-indexing/changes/4345.bugfix.md new file mode 100644 index 0000000000..928114728f --- /dev/null +++ b/packages/zarr-indexing/changes/4345.bugfix.md @@ -0,0 +1 @@ +Reject invalid wire index-array values and unrepresentable normalized bounds/ranks, and raise an explicit error for unsupported intersections sharing an affine and lookup input axis instead of returning incorrect coordinates. Group negative chunk-coordinate tuples without merging distinct chunks. diff --git a/packages/zarr-indexing/changes/4345.doc.md b/packages/zarr-indexing/changes/4345.doc.md new file mode 100644 index 0000000000..562e114e4e --- /dev/null +++ b/packages/zarr-indexing/changes/4345.doc.md @@ -0,0 +1,3 @@ +Correct indexing, reader, serialization, cache, and integration descriptions to match supported behavior; qualify NumPy/TensorStore compatibility and performance claims. + +Describe current contracts in source and test docstrings instead of narrating prior implementations. Clarify that immutable index coordinates do not snapshot source values. diff --git a/packages/zarr-indexing/changes/4346.misc.md b/packages/zarr-indexing/changes/4346.misc.md new file mode 100644 index 0000000000..05886fa863 --- /dev/null +++ b/packages/zarr-indexing/changes/4346.misc.md @@ -0,0 +1 @@ +Expand planner property tests across signed origins and chunk IDs, custom grids, mixed affine and lookup dependencies, duplicate coordinates, and empty domains. Verify exact request coverage and storage mapping with an independent pointwise oracle. diff --git a/packages/zarr-indexing/changes/4347.bugfix.md b/packages/zarr-indexing/changes/4347.bugfix.md new file mode 100644 index 0000000000..bea60a7501 --- /dev/null +++ b/packages/zarr-indexing/changes/4347.bugfix.md @@ -0,0 +1 @@ +Reject constrained `index_array_bounds` when lowering JSON to an engine transform or output map, instead of silently dropping the constraint. Message normalization continues to preserve bounds for consumers that support them. diff --git a/packages/zarr-indexing/changes/4348.bugfix.md b/packages/zarr-indexing/changes/4348.bugfix.md new file mode 100644 index 0000000000..d148e8b8ad --- /dev/null +++ b/packages/zarr-indexing/changes/4348.bugfix.md @@ -0,0 +1 @@ +Define an explicit source token contract: hash plain NumPy arrays without object fields, delegate source hooks and propagate their errors, and reject unsupported sources without converting or serializing them. Remove optional-Dask and per-call UUID fallbacks. Document hashing costs, mutation limits, and `name=False` for Dask wrapping opaque sources. diff --git a/packages/zarr-indexing/changes/4349.feature.md b/packages/zarr-indexing/changes/4349.feature.md new file mode 100644 index 0000000000..5043c83363 --- /dev/null +++ b/packages/zarr-indexing/changes/4349.feature.md @@ -0,0 +1 @@ +Add `Partition.result()` to execute a planned partition independently with the same global transform and chunk projection supplied during parent assembly. diff --git a/packages/zarr-indexing/changes/4350.feature.md b/packages/zarr-indexing/changes/4350.feature.md new file mode 100644 index 0000000000..417b548bfd --- /dev/null +++ b/packages/zarr-indexing/changes/4350.feature.md @@ -0,0 +1,5 @@ +Make `LazyArray` indexing lazy by default: use `view[...]`, `view.oindex[...]`, +and `view.vindex[...]` directly, without a `.lazy` accessor. Iteration yields +lazy views; `result()` and NumPy conversion materialize values. Add synchronous +`write(values)` and assignment through composed views, and an explicit +`EagerArrayAdapter` for consumers such as Dask that require eager indexing. diff --git a/packages/zarr-indexing/docs/api/grid.md b/packages/zarr-indexing/docs/api/grid.md index b7c376eb85..012c9df6dc 100644 --- a/packages/zarr-indexing/docs/api/grid.md +++ b/packages/zarr-indexing/docs/api/grid.md @@ -11,8 +11,9 @@ objects whose `shape` is the valid data size and whose `codec_shape` preserves the full codec-buffer size at a regular-grid boundary. `dimension_grids_from_chunks` returns these compact dimensions: integer chunk -shapes become `FixedDimension` instances and explicit per-axis edge sequences -become `VaryingDimension` instances. `DimensionGridLike` remains the narrow +shapes become `FixedDimension` instances and positive per-axis edge sequences +become `VaryingDimension` instances. An empty-axis sequence of zeros (including +an empty sequence) becomes `FixedDimension(size=0, extent=0)`. `DimensionGridLike` remains the narrow protocol used by the chunk planner, while `EdgeDimensionGrid` is kept for explicit edge-based and coordinate-origin examples. diff --git a/packages/zarr-indexing/docs/api/index.md b/packages/zarr-indexing/docs/api/index.md index a5b7163bbd..a5c5498e8d 100644 --- a/packages/zarr-indexing/docs/api/index.md +++ b/packages/zarr-indexing/docs/api/index.md @@ -25,12 +25,11 @@ and the wire format built on top of it. - [`zarr_indexing.domain`](domain.md) — `IndexDomain`, a rectangular region of integer coordinates with an explicit (possibly non-zero) origin - [`zarr_indexing.output_map`](output_map.md) — `ConstantMap`, `DimensionMap`, - and `ArrayMap`: three representations of a set of integer coordinates, one - per storage dimension + and `ArrayMap`: three coordinate mappings that preserve order and duplicates, + one per storage dimension - [`zarr_indexing.transform`](transform.md) — `IndexTransform`, which pairs a domain with output maps, plus the indexing (`[...]`, `.oindex`, `.vindex`), - `intersect`, and `translate` operations, and `selection_to_transform` - transforms into one + `intersect`, `translate`, and `compose` operations, and `selection_to_transform` **Chunk resolution** @@ -49,7 +48,7 @@ and the wire format built on top of it. **Lazy arrays** - [`zarr_indexing.lazy_array`](lazy_array.md) — `LazyArray`, a wrapper for - system-memory/basic-indexing sources that adds a `.lazy` accessor for + system-memory/basic-indexing sources with lazy indexing for TensorStore-style deferred indexing, plus `Partition` and `parts()` / `with_parts()`, which determine the boxes a read is broken into. Device sources require an explicit custom reader that transfers into the supplied diff --git a/packages/zarr-indexing/docs/api/lazy_array.md b/packages/zarr-indexing/docs/api/lazy_array.md index f855511d46..429de1d441 100644 --- a/packages/zarr-indexing/docs/api/lazy_array.md +++ b/packages/zarr-indexing/docs/api/lazy_array.md @@ -2,26 +2,39 @@ title: lazy_array --- -`LazyArray.lazy[...]` is metadata-only: every derived view keeps the same +`LazyArray[...]` is metadata-only: every derived view keeps the same reader and composes its transform without reading data. `result()` allocates owned system memory, then calls that reader once for each projected part. Rectangular parts write directly into their final slices; advanced placement may first use an owned dense temporary. `LazyArray(source)` assumes only basic -indexing, while `LazyArray.from_numpy(array)` explicitly selects NumPy's -optimized reader. +indexing, while `LazyArray.from_numpy(array)` selects `numpy_reader`. Both +currently use the same slab-and-gather implementation. The built-in readers lower through NumPy system memory and support sources -whose basic reads can be converted there. They do not implicitly transfer -device arrays; a device source needs an explicit custom reader that transfers -into the supplied system-memory output. Derived views and parts share their +whose basic reads can be converted there. A device source that refuses NumPy +conversion needs a custom reader that transfers into the output buffer. Derived views and parts share their reader and part views may be materialized concurrently, so stateful readers must synchronize their own mutable state. Every public `Partition.view.transform` directly maps that view's zero-origin coordinates into its raw `Partition.view.array`, including for non-first partitions. `Partition.projection.chunk_transform` intentionally stays local to -the selected chunk. During materialization the reader receives both frames in -one `ReadContext`: the public global transform in `context.transform` and the -same local plan in `context.projection`. +the selected chunk. During parent materialization (`view.result(parts=parts)`) the reader receives +both frames in one `ReadContext`: the public global transform in `context.transform` and the +same local plan in `context.projection`. Use `part.result()` to execute a +partition independently with both frames. Direct `part.view.result()` calls +resolve the general view with no projection. + +`view[key]`, `view.oindex[key]`, and `view.vindex[key]` return lazy views, +and iteration yields lazy first-axis views. `result()` and `numpy.asarray(view)` +materialize values. `view.write(values)` synchronously writes to the original +source through the composed transform and returns `None`; `view[key] = values` +writes a selected sub-view. Writes require a writable source. + +For consumers requiring eager indexing, import `EagerArrayAdapter` from +`zarr_indexing` and wrap the view. The adapter delegates shape, rank, dtype, +NumPy conversion, and tokenization to the view, but materializes each +`adapter[key]`. Use it with `dask.array.from_array`; direct lazy indexing +is not a reliable Dask block-read interface. ::: zarr_indexing.lazy_array diff --git a/packages/zarr-indexing/docs/api/reader.md b/packages/zarr-indexing/docs/api/reader.md index 39129c4d47..80cd7b7ce7 100644 --- a/packages/zarr-indexing/docs/api/reader.md +++ b/packages/zarr-indexing/docs/api/reader.md @@ -12,7 +12,8 @@ ownership. `Reader.read_into(source, context, out)` receives a `ReadContext` whose `transform` maps zero-origin output-buffer coordinates to global coordinates in `source`, with `context.transform.domain.shape == out.shape`. Its optional -`projection` is the existing plan for a partitioned read. The projection's +`projection` is the existing plan for a partitioned read. Both parent +assembly and independent `Partition.result()` calls supply that projection. The projection's `chunk_transform` remains chunk-local, its `cell_transform` describes result placement, and its `chunk_domain` describes the grid cell. The global read transform and the projection's chunk transform deliberately use different @@ -34,7 +35,7 @@ class RecordingReader: self.calls = [] def read_into(self, source, context, out, /): - self.calls.append((source, context, out)) + self.calls.append((source, context, out.shape, out.dtype)) self.inner.read_into(source, context, out) @@ -44,7 +45,8 @@ view = LazyArray.from_numpy(array).with_reader(outer) values = view.result() ``` -Both wrappers observe the same three objects, in outer-to-inner order. This +Both wrappers observe the same arguments, in outer-to-inner order, and log +output metadata without retaining the output buffer. This delegation pattern supports policies such as logging and caching without library-defined wrapper primitives. diff --git a/packages/zarr-indexing/docs/design-notes.md b/packages/zarr-indexing/docs/design-notes.md index 59eef4dacd..8ce50c340a 100644 --- a/packages/zarr-indexing/docs/design-notes.md +++ b/packages/zarr-indexing/docs/design-notes.md @@ -16,7 +16,7 @@ visual guide owns the mechanics of ## Relationship to TensorStore The core is [TensorStore's](https://google.github.io/tensorstore/index_space.html) -index-transform model, reimplemented in Python against NumPy. The visual guide +index-transform model, implemented here in Python against NumPy. The visual guide introduces the shared model in [Coordinates are addresses](guide/index.md#coordinates-are-addresses) and [Lazy views compose](guide/index.md#lazy-views-compose); the comparison here is @@ -28,13 +28,14 @@ about the deliberately matching semantics: - **Slice semantics.** Slice bounds are literal domain coordinates: no clamping, no negative wrapping, non-empty intervals must be contained in the domain, and a strided slice's domain origin is `trunc(start/step)` rounded - toward zero. Every one of those rules was executed against tensorstore 0.1.84 - and is pinned in `tests/test_tensorstore_parity.py`. -- **The wire format.** A canonical [ndsel](ndsel.md) transform body is, - field-for-field, a TensorStore `IndexTransform` minus the `kind` - discriminator, and `tests/test_ndsel_tensorstore.py` loads our bodies into - `tensorstore.IndexTransform(json=...)` and round-trips them back through our - engine layer. + toward zero. `tests/test_tensorstore_parity.py` compares the enumerated cases with + TensorStore when that optional dependency is installed. +- **The wire format.** [ndsel](ndsel.md) uses TensorStore's domain and + output-map field names for transform bodies, with an additional `kind` + discriminator. `tests/test_ndsel_tensorstore.py` checks interoperability for + the tested cases. Their validation rules differ, and loading and + re-emitting a message through the engine can normalize or discard metadata; + see [lowering to a transform](ndsel.md#lowering-to-a-transform). - **Chunk partitioning.** Both factor a transform over a grid before visiting any cell, rather than intersecting the whole transform with each chunk. TensorStore's `IndexTransformGridPartition` holds strided sets and index @@ -47,17 +48,19 @@ about the deliberately matching semantics: components within a mixed request. Both derive the per-chunk transforms from the partition ([the guide](guide/index.md#a-plan-is-a-product-of-per-axis-tables) shows the tables). TensorStore keeps strided sets implicit, while this - library materializes their per-axis rows for vectorized consumers. Diagonals are rejected here; supporting them needs a strided set per - *input* dimension spanning every storage axis that reads it, TensorStore's - representation. + library materializes their per-axis rows for vectorized consumers. Pure + affine diagonals need grouping by input dimension; mixed affine/index-array + dependencies need joint partitioning. TensorStore classifies a connected + component containing index-array edges as an index-array set + ([source](https://github.com/google/tensorstore/blob/66b2ce5290fa2ec5c8019682391421062ce767a2/tensorstore/internal/grid_partition.h#L58-L67)). Four deliberate differences: | | TensorStore | `zarr-indexing` | | --- | --- | --- | -| Dialect | One strict dialect everywhere: literal coordinates, no negative wrapping | The algebra keeps that dialect; each public boundary picks its own. [`LazyArray`](api/lazy_array.md) speaks positional NumPy, `zarr.Array.lazy` speaks literal. [`zarr_indexing.boundary`](api/boundary.md) is the translation | +| Dialect | Coordinate indices are literal; negative coordinates do not wrap | The algebra keeps that dialect; each public boundary picks its own. [`LazyArray`](api/lazy_array.md) speaks positional NumPy, `IndexTransform` speaks literal. [`zarr_indexing.boundary`](api/boundary.md) is the translation | | Scheduling | An internal C++ scheduler owns concurrency and chunk ordering | [`parts()`](api/lazy_array.md) exposes the partition structure so the caller's own scheduler — dask, a thread pool, a task queue — drives it | -| Wire format | Implementation-defined JSON, specified by what the implementation accepts | [ndsel](ndsel.md) is spec-first, with a vendored language-agnostic conformance corpus every implementation runs | +| Wire format | [Documented JSON schema](https://google.github.io/tensorstore/index_space.html#index-transform) | [ndsel](ndsel.md) is spec-first, with a vendored conformance corpus exercised by this implementation | | Backends | A driver ecosystem (zarr, N5, neuroglancer, GCS, …) built into the library | No drivers. The default reader needs `shape`, `dtype`, basic integer/slice indexing, and selected slabs convertible to NumPy system memory; other backends use explicit custom readers. A device reader owns transfer into the supplied system-memory output | The mechanics of a @@ -74,19 +77,18 @@ and caller-supplied grid; it does not own reads, writes, buffers, locks, or scheduling. Zarr can therefore plan reads against an inner codec-chunk grid and writes against an atomic shard grid; napari or dask can turn the same projections into tasks without putting a dask dependency in this package. -`coverage` is relative to that selected grid: `full` proves a blind replacement -safe, `partial` proves it is not, and `unknown` conservatively covers fancy -selections whose duplicates would require additional work to classify. - -The comparison also runs the other way. TensorStore is a mature, heavily -optimized C++ system whose performance this library cannot approach. Independent strided planning -here is per axis, but each materialized `ChunkProjection` is -Python-level bookkeeping over NumPy — two domains, two transforms and the -projection itself — so the per-part overhead of the object view is -significant; a consumer that reads the partition's tables directly pays no -per-chunk object construction. This library is small and depends on nothing beyond -NumPy, so the algebra can be adopted by a Python project that wants the model -without the C++ runtime. +`coverage` describes selection coverage relative to that grid. A `full` +classification can help a writer avoid reading old values, but does not by +itself prove that a write is safe: encoding requirements, conflicts, duplicate +semantics, and concurrency remain consumer responsibilities. `unknown` means +the planner has not established complete or partial coverage. + +This implementation performs Python-level bookkeeping over NumPy. This page +provides no benchmark establishing a general performance ordering against +TensorStore; costs depend on the selection and execution backend. + +The partition tables can be consumed without constructing a `ChunkProjection` +for each chunk. Materializing projections adds Python object construction. ## Bounding-box selections vs query selections @@ -102,11 +104,11 @@ and the coordinates it touches form a regular lattice. Basic indexing produces one, and composing basic indexing with basic indexing keeps one. **A query** is a transform with at least one `ArrayMap` — an explicit lookup -table of coordinates. It costs `O(n)` to store, it has no locality (the -coordinates may repeat, reverse, or scatter arbitrarily), and intersecting it -with a region means scanning it. `oindex`, `vindex`, and boolean masks all -produce one, and once an axis is a query, subsequent basic indexing cannot make -it a box again. A second query composes onto any axis of an existing one — +table of coordinates. Its stored coordinate arrays cost space proportional to their stored size. +Coordinates may repeat or scatter, but can also be contiguous and local. +Current query-resolution paths inspect these arrays. Fancy indexing can produce +a query, but singleton or constant selections can collapse to `ConstantMap`; +subsequent indexing can therefore make a query affine again. A second query composes onto any axis of an existing one — including the axes it merely broadcasts along — by evaluating the existing lookup tables at the new coordinates. @@ -116,9 +118,10 @@ planning and materialization. [ndsel](ndsel.md) encodes the same split in its message kinds: `point`, `box`, and `slice` desugar to constant and affine output maps and are always boxes; -`points` desugars to `index_array` maps, and a `transform` body is a box -exactly when none of its output maps carries an `index_array`. A consumer can -therefore classify a selection off the wire without materializing anything: +`points` desugars to `index_array` maps. A transform without index-array maps +is a box in the engine’s structural classification. Loading can further +simplify degenerate index arrays to constants, so an arbitrary incoming body +with `index_array` fields need not remain a query. For example: ```python from zarr_indexing import IndexTransform @@ -134,16 +137,13 @@ gather.to_json()["output"][0] # 'index_array_bounds': ['-inf', '+inf']} ``` -The distinction matters to consumers of a selection. A box can be tiled into -rectangular dask chunks or passed to a viewer or tile server that only accepts -rectangles; a query cannot, and has to be resolved into a gather. A box can also -be served as a single strided slab read, but the read has to be strided: reading -its bounding box and discarding the rest transfers proportionally more data as -soon as any stride exceeds 1. The two also behave differently under -partitioning: a box touches a regularly-spaced run of parts, in increasing -order, each at most once — a stride larger than a part's extent skips parts -outright, so the run is not contiguous — while a query can touch any subset of -them, in any order, more than once. +The representation helps a consumer choose a lowering strategy. Independent +affine axes can often be read with slices plus reversal, permutation, or +broadcasting. Arbitrary affine maps can also express diagonals, so the absence +of `ArrayMap` alone is not proof of a rectangular slab. Queries may be lowered +through gathers or covers, and can sometimes simplify to slices. Chunk plans +group selected coordinates by chunk while preserving result placement; repeated +coordinates do not imply repeated visits to the same chunk. [`LazyArray`](api/lazy_array.md) exposes the category directly: @@ -157,13 +157,13 @@ arr = zarr.create_array({}, shape=(100, 80), chunks=(30, 40), dtype="int32") arr[:] = np.arange(8000).reshape(100, 80) lazy = LazyArray(arr) -slab = lazy.lazy[10:50, ::4] +slab = lazy[10:50, ::4] slab.is_box # True slab.bounding_box() # ((10, 50), (0, 77)) slab.strides() # (1, 4) slab.shape # (40, 20) -gather = lazy.lazy.oindex[[90, 3, 3], :] +gather = lazy.oindex[[90, 3, 3], :] gather.is_box # False gather.bounding_box() # ((3, 91), (0, 80)) gather.strides() # None @@ -173,16 +173,20 @@ gather.shape # (3, 80) `bounding_box()` is defined for both: it is the hull, the smallest interval per storage dimension containing every coordinate the selection reaches. `strides()` is defined only for a box and gives the step per dimension. -Together the two describe a box selection completely. - -Both are needed, because a box is dense in its hull only when every stride is -1. The slab above spans a 40x77 hull over the 40x20 cells it selects, so a +These summaries omit traversal direction, input-axis correspondence, and +result layout. For example, forward and reversed views have identical bounds +and stride magnitudes but different ordered results. Use the transform for the +complete selection. + +For independent axes with multiple selected coordinates, a stride magnitude +greater than one leaves gaps in the hull. Singleton axes are an exception, +and a query can also cover every cell of its hull. The slab above spans a 40x77 hull over the 40x20 cells it selects, so a consumer that issued one rectangular read of the hull and discarded the rest would transfer 3.85x the data. A query's hull is looser still and carries no stride at all: 88 rows of hull over three selected rows. An empty *box* touches no coordinate to report an interval around, so `bounding_box()` is `None` while `strides()` still answers — the step is a property of the selection's shape, not -of the region it reaches. Only a query returns `None` from both. +of the region it reaches. An empty query returns `None` from both. There is deliberately no separate `BoxView` type today. A statically-typed rectangular-only view is a plausible next step, but it should be introduced by @@ -248,8 +252,8 @@ could accept and finishing the rest elsewhere. A reader lowers the complete transform and can compose through delegation instead. This resembles [zarrita.js store extensions](https://zarrita.dev/packages/zarrita.html), where storage-specific behavior is an explicit extension point rather than an -inferred array capability. The implementation remains independently authored: -no code is shared with TensorStore, xarray, or zarrita.js. +inferred array capability. This is an architectural analogy, not a claim of API or implementation +compatibility. ## Current scope @@ -263,39 +267,36 @@ re-bases every view to origin 0, so the positional dialect never exposes it; a caller working with `IndexTransform` directly will see it, and re-bases explicitly with `translate_domain_to` for NumPy-shaped coordinates. -Fancy selections compose without restriction: a second `oindex`/`vindex`/mask +Supported fancy selections compose across already-fancy views: a second `oindex`/`vindex`/mask step may land on any axis of an already-fancy view, including axes an existing index array merely broadcasts along, so -`lazy.oindex[[2, 0], :].lazy.oindex[:, [1, 3]]` selects the outer product it +`lazy.oindex[[2, 0], :].oindex[:, [1, 3]]` selects the outer product it spells. An array-carrying transform is composed — the new selection is applied to an identity transform over the current domain and chained on with `compose`, which evaluates the existing lookup tables at the new coordinates — rather than rewritten in place. Resolution classifies the result by structure (`index_array_structure`): pure per-axis outer products keep the orthogonal resolvers, and everything else — correlated maps, mixtures, index arrays -sharing an input axis (a diagonal gather, reachable only by hand-building a -transform) — takes the general reader/intersection path. Chunk planning +sharing an input axis (as in paired vectorized coordinates) — takes the general reader/intersection path. Chunk planning factors index arrays into connected dependency components before flattening, so independent groups do not expand one another. Vectorized selection preserves broadcast singletons to retain those dependencies. -Three limits remain, all intentional and all expected to be lifted: +Some current limits are: -- **Affine diagonals.** A hand-built transform in which two output maps read - one input dimension — two slice maps, or a slice map and an orthogonal index - array — is rejected at planning with `ValueError`; a correlated index array - varying over a dimension a slice map also reads is rejected with - `NotImplementedError`. No selection dialect produces either. Supporting them - needs a strided set per *input* dimension spanning all dependent storage - axes, TensorStore's connected-component representation. *Planned.* +- **Shared affine dependencies.** Planning rejects two affine output maps + sharing an input axis with `ValueError`. An index array sharing a varying + input axis with an affine map takes the general classification and raises + `NotImplementedError`. Pure affine diagonals would need grouping dependent + storage axes by input dimension; mixed components need joint partitioning. - **Finite explicit bounds only.** `IndexDomain` has no implicit or unbounded dimensions; the message layer will normalize a body with `"-inf"`/`"+inf"` bounds, but the engine layer refuses to lower one into a transform. - TensorStore supports both. *Planned.* + TensorStore supports both. - **Labels are carried, not propagated.** `IndexDomain` holds optional dimension labels and the wire format round-trips them, but indexing operations build new domains without them, so a label does not survive a - slice. *Planned.* + slice. ## Selection to chunk operations diff --git a/packages/zarr-indexing/docs/guide/index.md b/packages/zarr-indexing/docs/guide/index.md index 68192f58cb..c09eed1b23 100644 --- a/packages/zarr-indexing/docs/guide/index.md +++ b/packages/zarr-indexing/docs/guide/index.md @@ -1,6 +1,6 @@ # Visual guide -The whole model in one sentence: indexing through `LazyArray.lazy` builds a +The whole model in one sentence: indexing through `LazyArray` builds a view, chunk planning partitions its coordinates, and `result()` materializes the view. This page follows one familiar NumPy selection, `source[2:5]`, through those stages. @@ -38,7 +38,7 @@ make the correspondence explicit: those result coordinates receive values `12`, `13`, and `14` from source coordinates `2`, `3`, and `4`. The wrapper below gives the same familiar selection a lazy spelling. Indexing -through `.lazy` creates `view`; the last line asks for its values and checks the +with `[...]` creates `view`; the last line asks for its values and checks the observable NumPy result. ```python @@ -105,7 +105,7 @@ different questions: | Surface | Meaning of an integer index | Meaning of `-1` | | --- | --- | --- | | `IndexDomain` and `IndexTransform` | A literal coordinate in the current domain | The actual address `-1`, if the domain contains it | -| `LazyArray.lazy` | A NumPy-style position in the current view | The last position, normalized before it reaches the transform algebra | +| `LazyArray` | A NumPy-style position in the current view | The last position, normalized before it reaches the transform algebra | `LazyArray` uses positions because it is an array-like wrapper: each derived view starts at position zero and negative indices wrap exactly as they do in @@ -216,21 +216,28 @@ description; the assertion's call to `result()` is the first operation in the example that materializes the selected data. !!! warning "Stop here: the materialization boundary" - Indexing through `.lazy[...]` never reads. These do: + Indexing through `[...]` composes a selection without reading source values. + These operations request values: - `result()` - - eager indexing of the wrapper: `view[...]` - - `numpy.asarray(view)`, or passing the view to any NumPy function - (`numpy.add(view, 1)` converts, and therefore materializes, the view) + - `numpy.asarray(view)` and NumPy operations that convert the view + (`numpy.add(view, 1)` does so; `numpy.shape(view)` and `numpy.ndim(view)` + can use metadata without reading values) + + Dask tokenization hashes plain NumPy data or delegates to a source hook. + Other sources require [an explicit hook or Dask naming opt-out](integrations.md#dask-tokenization-and-source-mutation). Python arithmetic such as `view + 1` raises `TypeError` instead: this wrapper defers indexing, not a general compute graph. - Nor does it write. There is no `__setitem__`, so `view[...] = values` - raises `TypeError` too, and a wrapped source needs no `__setitem__` of - its own. A consumer that writes plans the selection with `plan_chunks` - and performs its own read-modify-write, keeping chunk atomicity and - concurrent-writer policy on the backend's side of the boundary. + Iteration yields lazy first-axis views; call `result()` on each to read it. + + `view.write(values)` writes through the composed transform to the original + writable source, synchronously, and returns `None`. `view[key] = values` + writes the selected sub-view in the same way. These calls do not create + futures or transactions. Read-only sources remain usable for reads; + writes require source assignment support. Storage atomicity and concurrent + writer coordination remain the backend's responsibility. ## An index defines a result array {#an-index-defines-a-result-array} @@ -384,8 +391,10 @@ each bundles a sub-view of the request (`.view`), that chunk's projection Within one `Partition`, the frames divide: `Partition.view.transform` is a different, global transform — it maps the part view directly into the raw wrapped source — while only `Partition.projection.chunk_transform` uses -zero-origin chunk-local coordinates. Readers receive both so the global -source address and the local planning frame cannot be confused. +zero-origin chunk-local coordinates. Parent assembly passes both frames to +the reader. Independently scheduled `part.result()` calls supply the same +context. Calling `part.view.result()` instead resolves the view without the +partition record, so its context has `projection=None`. | Projection field | What its output coordinates mean | | --- | --- | @@ -399,7 +408,7 @@ rank one and source rank two. ### Order and duplicates need the request-side projection -Orthogonal indexing (`.lazy.oindex`) applies each axis's indexer +Orthogonal indexing (`.oindex`) applies each axis's indexer independently, like `numpy.ix_` — an outer product; the [pattern reference](patterns.md) develops the dialects. It can visit source cells in an order that does not match chunk order, and it can visit one diff --git a/packages/zarr-indexing/docs/guide/integrations.md b/packages/zarr-indexing/docs/guide/integrations.md index f787d11f6c..5eece1f3de 100644 --- a/packages/zarr-indexing/docs/guide/integrations.md +++ b/packages/zarr-indexing/docs/guide/integrations.md @@ -3,8 +3,9 @@ For the complete path from indexing syntax to chunk coordinates, local selectors, and result positions, start with [From a selection to chunk operations](selection-flow.md). -This package supplies indexing plans. It does **not** supply scheduling, -caching, codecs, or async orchestration. A consumer decides when projections +The core package supplies indexing plans and synchronous readers. It does not +provide a general scheduler, codec pipeline, or async execution engine. A +synchronous cache is included as an example. A consumer decides when projections run, how decoded chunks are obtained, and where completed values are retained. An `IndexTransform` says which source values belong in a result; a `Reader` lowers that complete transform for one backend. The reader does not choose @@ -61,21 +62,21 @@ afterward, so accessing a row does not recalculate every point. ## One slab read or many part reads -A backend with its own native subset read — a Rust or C zarr implementation, -a database, an HTTP range endpoint — resolves a **dense box** (`is_box` with -every stride 1) best as a single read: hand it the whole selection and let it -dispatch to chunks, decode in parallel, and partial-decode shards on its own -side of the boundary. Splitting that read along this library's partitioning -only adds round-trips. Every **other** selection — a strided box, an `oindex` -or `vindex` gather — is where the partitioning earns its keep. The **cover** -of a read is the smallest step-1 slab enclosing every coordinate it needs; -partitioned, each part's cover is bounded by that part's box, so a sparse -selection can never force one read of its whole bounding hull (the smallest -rectangle containing every selected coordinate — a thousand rows for the two -of `oindex[[0, 999]]`). +A backend with an efficient native subset operation may benefit from receiving +one complete dense selection so it can choose its own chunk dispatch. Other +backends may benefit from partitioning, including for strided or fancy +selections. The tradeoff depends on the backend, chunk layout, latency, memory, +and selection; a single read is not universally fastest. + +A read's **cover** is the smallest unit-step slab enclosing its coordinates. +With this package's basic readers, partitioning limits each source read to the +part's selected cover. This may reduce over-reading, but a partition that spans +the source can still require the entire hull. Custom readers choose their own +source operations under the reader contract. The composed view carries enough to make that call at materialization time, -and re-partitioning is a pure setter, so the policy is three lines: +and `with_parts()` returns a view with a new partitioning. This example uses +unit strides as a sufficient condition for its independently mapped selections: ```python --8<-- "snippets/integrations.py:dense-box-repartition" @@ -87,20 +88,19 @@ the dense box becomes exactly one backend call. Both regimes go through ### Sources that accept only unit-step slices -The default `basic_reader` pushes strided and descending selections down as -positive-step slices, which reads the minimum but assumes the source accepts -any step. Many backends do not: FFI bindings and range requests often -support nothing but `slice(start, stop, 1)`. Select -[`unit_step_reader`][zarr_indexing.reader.UnitStepReader] for such a source -and every key it receives is an ascending unit-step slice per axis, with +For affine selections, `basic_reader` uses positive-step slices and applies +reversal or layout changes in memory. Fancy selections can require reading a +cover containing unselected values. The source must accept the emitted steps. +Use [`unit_step_reader`][zarr_indexing.reader.UnitStepReader] for a source that +accepts only unit steps. Every key it receives is an ascending unit-step slice per axis, with strides, reversals, and gathers applied to the in-memory block instead: ```python view = LazyArray(source).with_reader(unit_step_reader) ``` -A strided selection then over-reads its cover by the stride factor, which the -partitioning above bounds by one part. +For strided selections, the ratio of cover cells to selected cells depends on +stride, length, and endpoint alignment. Partitioning can reduce that cover. ## napari-like consumer @@ -205,13 +205,12 @@ decoded chunks. Every read delta follows directly from the viewport request: | 4 | `image[1:5, 2]` | `(0, 0)` | `(0, 0)`, `(1, 0)` | The evicted chunk is reloaded while the required ready chunk is retained. | | 5 | `image[3:5, 4:6]` | `(1, 1)` fails; no repeated read; `(1, 1)` succeeds after retry | `(0, 0)`, `(1, 1)` | Failure is retained until explicit retry; the repaired source then returns `[[28, 29], [36, 37]]`. | -Chunks required by an active request are pinned through assembly, so a request -may temporarily span more chunks than the steady-state capacity. Capacity is -counted in decoded chunks—not records or bytes—and eviction occurs only after -all requested values have been placed. Because pinning and materialization use -the same prepared tuple, those lifecycle decisions cannot drift from the parts -that are actually read, and the cache never has to infer or reconstruct a -projection. +The example defers eviction until a successful outermost request finishes, so +it can temporarily exceed capacity during assembly. Capacity counts decoded +chunks, not bytes, event records, or temporary arrays. Failed requests skip +that eviction step. The example assumes an unchanged source and one source/grid +per reader; it does not implement invalidation or synchronization for concurrent +requests. The prepared tuple supplies the projections used for each read. The event log makes the failure boundary equally explicit: @@ -244,3 +243,34 @@ that implementation or reproduce its full worker/GPU lifecycle. · **API:** [API reference](../api/index.md) + +## Dask tokenization and source mutation + +`LazyArray.__dask_tokenize__()` combines the serialized transform with a source +value token. For plain NumPy arrays without object fields, it hashes all bytes +on every call, including large arrays. Time and temporary memory are linear in +source byte size. Equal supported contents and serialized transforms have equal +tokens; changes to numeric contents are visible on the next tokenization call. + +Other sources must define an explicit `__dask_tokenize__` hook. This includes +object arrays (also structured object fields), array subclasses, memory-mapped +arrays, and remote arrays. Known `numpy.memmap` and `mmap.mmap` backing is +rejected through ndarray base and memoryview object chains. Arbitrary buffer +provenance cannot be inferred from a plain ndarray. The wrapper does not +convert or serialize unsupported sources to discover their values. Unsupported sources raise `TypeError`, and +exceptions from explicit hooks propagate. Installing Dask does not change this +policy. A hook must describe the source's values or immutable version, and owns +its determinism and any I/O it performs. + +For an opaque source such as a Zarr array, use +`dask.array.from_array(EagerArrayAdapter(view), chunks=..., name=False)` to request a fresh graph +name without content tokenization. This opts out of content-based task sharing. +Import `EagerArrayAdapter` from `zarr_indexing`: it materializes each indexed block while the wrapped view keeps lazy indexing. See Dask's [from_array documentation](https://docs.dask.org/en/stable/generated/dask.array.from_array.html) +and [tokenization contract](https://docs.dask.org/en/stable/custom-collections.html#implementing-deterministic-hashing). + +Tokens describe values at tokenization time, not a snapshot. Mutating a source +after building a graph does not update existing Dask keys or invalidate cached +results. Concurrent mutation during hashing is unsupported. Applications must +manage source lifetimes and versions; these tokens are not persistent cache +identities. Readers and partitioning are omitted because they must preserve +values. diff --git a/packages/zarr-indexing/docs/guide/patterns.md b/packages/zarr-indexing/docs/guide/patterns.md index 9d22c4eb4e..df9a482c3e 100644 --- a/packages/zarr-indexing/docs/guide/patterns.md +++ b/packages/zarr-indexing/docs/guide/patterns.md @@ -302,17 +302,18 @@ so they equal the zero-origin models after `translate_domain_to`: --8<-- "snippets/indexing_patterns.py:indexing-patterns" ``` -`LazyArray` adds nothing to these semantics: it is a regular array-like API -whose `.lazy`, `.lazy.oindex`, and `.lazy.vindex` accessors compile the same -dialects to the same transforms — the only difference is the return type, a -view instead of an array. The test suite holds the wrapper to this matrix. +`LazyArray` exposes the transform machinery through a positional array-like +API. Its `[...]`, `.oindex[...]`, and `.vindex[...]` operations return views and +normalize positions before composition. This boundary differs from the literal +coordinate semantics of `IndexTransform`, as the following table shows. The +executable matrix checks the documented cases, not every possible NumPy expression. ## Positions vs literal coordinates | Surface | Meaning of an integer index | Meaning of `-1` | | --- | --- | --- | | `IndexDomain` and `IndexTransform` | A literal coordinate in the current domain | The address `-1`, when the domain contains it | -| `LazyArray.lazy` | A NumPy-style position in the current view | The last position, normalized before transform composition | +| `LazyArray` | A NumPy-style position in the current view | The last position, normalized before transform composition | The wrapper's three indexing modes all use positions in the current view. Each derived view begins at position zero, while the transform algebra underneath diff --git a/packages/zarr-indexing/docs/guide/selection-flow.md b/packages/zarr-indexing/docs/guide/selection-flow.md index 571baa3869..749a9c36ae 100644 --- a/packages/zarr-indexing/docs/guide/selection-flow.md +++ b/packages/zarr-indexing/docs/guide/selection-flow.md @@ -171,7 +171,7 @@ pairs: | `(0,)` | `[1]` | `[1]` | | `(1,)` | `[2, 2, 0]` | `[0, 2, 3]` | -Chunk `(1,)` is fetched once, but its local value `2` contributes to two result +Chunk `(1,)` is fetched once, but the value at local coordinate `2` contributes to two result positions. Chunk visitation order need not be result order; `out_selection` restores the requested arrangement. @@ -188,6 +188,9 @@ than one. Repeated coverage is not full; fancy coverage remains conservative. Zarr's merge operation can skip a read for a full data-extent write and allocate a fill-valued codec buffer when the selected data is smaller than that buffer. +That shortcut also requires the consumer's expected value layout and order; +complete coverage by a reversal or reordered gather alone does not establish +that the supplied buffer can be copied directly into codec order. Touching every chunk alone does not prove full coverage of each chunk. Planning does not fetch bytes, decode buffers, choose concurrency, or define diff --git a/packages/zarr-indexing/docs/index.md b/packages/zarr-indexing/docs/index.md index 87e5a4b7a1..045ba2103f 100644 --- a/packages/zarr-indexing/docs/index.md +++ b/packages/zarr-indexing/docs/index.md @@ -5,8 +5,7 @@ the *declaration* of an array indexing expression from the result of that expres Developed for use in [`zarr`](https://zarr.readthedocs.io). -Inspired by [TensorStore](https://google.github.io/tensorstore/), which pioneered -the approach used here. +Inspired by [TensorStore's index-transform model](https://google.github.io/tensorstore/index_space.html). ## Install @@ -21,15 +20,17 @@ pip install zarr-indexing ## Quickstart -Wrap an array, compose a lazy view through `.lazy`, and call `result()` when +Wrap an array, compose a lazy view with `view[...]`, and call `result()` when you want its values: ```python --8<-- "snippets/canonical_slice.py:landing-quickstart" ``` -Nothing is read until the `result()` call, however many selections are -composed. [Lazy views compose](guide/index.md#lazy-views-compose) shows how +Composing these selections does not read source values; the example reads them +at `result()`. Construction inspects source metadata. Dask tokenization hashes +plain NumPy data or delegates to an explicit source hook; other sources need +[a naming opt-out or source hook](guide/integrations.md#dask-tokenization-and-source-mutation). [Lazy views compose](guide/index.md#lazy-views-compose) shows how the chain stays one description, and where the materialization boundary is. ## Learn more diff --git a/packages/zarr-indexing/docs/ndsel.md b/packages/zarr-indexing/docs/ndsel.md index 94971d49bf..e73ac3245a 100644 --- a/packages/zarr-indexing/docs/ndsel.md +++ b/packages/zarr-indexing/docs/ndsel.md @@ -14,11 +14,13 @@ before a transform is serialized. `zarr-indexing` implements ndsel in two layers | Layer | Module | Depends on | Job | | --- | --- | --- | --- | -| Message | [`zarr_indexing.messages`](api/messages.md) | stdlib only | JSON in, canonical JSON out. Validates and desugars. Never rounds, clamps, or drops information. | +| Message | [`zarr_indexing.messages`](api/messages.md) | stdlib only | Validates and desugars JSON, removing redundant constant-map fields. | | Engine | [`zarr_indexing.json`](api/json.md) | NumPy | Lowers a *canonical* body into an in-memory [`IndexTransform`](api/transform.md), and back. | Constraints that only make sense for a real array — finite bounds, index -arrays as `ndarray`s — live in the engine layer and nowhere else. As a result, +arrays as `ndarray`s — are checked during engine lowering. The message layer +also limits input rank to 32 and checks affine input-dimension references. +These checks are stricter than the draft's required message validation. As a result, `messages` normalizes a message with `"-inf"` bounds that `IndexTransform.from_json` refuses to lower. @@ -47,9 +49,13 @@ normalize_ndsel({"kind": "box", "inclusive_min": [10, 5], "shape": [40, 1]}) ``` Normalization is idempotent: re-tag the output with `kind: "transform"` and -normalizing it again returns the same body. Because the canonical body is -field-for-field a TensorStore `IndexTransform` minus `kind`, a normalized -message loads directly into `tensorstore.IndexTransform(json=...)`. +normalizing it again returns the same body. The canonical body uses +TensorStore's `IndexTransform` field vocabulary, but normalization does not +guarantee acceptance by TensorStore. For example, the message layer leaves +index-array content unchecked, whereas TensorStore validates it; TensorStore +also requires unique nonempty labels and restricts finite index values to +`[-(2**62 - 2), 2**62 - 2]`. See +[TensorStore's index-space constraints](https://google.github.io/tensorstore/index_space.html). Both entry points raise [`NdselError`](api/messages.md#zarr_indexing.messages.NdselError), which @@ -64,12 +70,13 @@ Four are shorthands; the fifth is the canonical form itself. | `kind` | Fields | Selects | | --- | --- | --- | | `point` | `coords` | A single element. Normalizes to rank 0 with one `constant` output map per dimension. | -| `box` | `inclusive_min`, one of `exclusive_max` / `inclusive_max` / `shape`, `labels` | A rectangular region. Exactly one upper-bound spelling may appear. | +| `box` | `inclusive_min`, one of `exclusive_max` / `inclusive_max` / `shape`, `labels` | A rectangular region. At most one upper-bound spelling may appear; omission gives implicit positive infinity. | | `slice` | `start`, `stop`, `step`, `labels` | A strided region, one Python-style slice per dimension. | | `points` | `coords` (a list of coordinate rows) | An explicit list of points — the `vindex` case. Normalizes to one `index_array` output map per dimension over a shared rank-1 input domain. | | `transform` | `input_rank`, `input_inclusive_min`, one of the three `input_*` upper bounds, `input_labels`, `output` | The full canonical form. | -Value rules the message layer enforces throughout: every integer is a 64-bit +Value rules for validated fields (excluding verbatim `index_array` payloads +and discarded constant-map fields): every integer is a 64-bit signed value; JSON booleans are **not** integers (Python's `isinstance(True, int)` is guarded against explicitly); the `"-inf"` / `"+inf"` sentinels are legal only in bound positions; and an implicit bound is the @@ -81,17 +88,30 @@ normalization intact. The engine layer converts between canonical bodies and `IndexTransform`s: ```python -from zarr_indexing import IndexTransform +from zarr_indexing import IndexTransform, normalize_ndsel +canonical = normalize_ndsel({"kind": "box", "shape": [2, 3]}) t = IndexTransform.from_json(canonical) -t.to_json() == canonical +assert t.to_json() == canonical ``` `IndexDomain` carries the same pair for a bare domain body, and each output map kind has a `to_json`; `output_index_map_from_json` dispatches the wire's -tagged union back to the right kind. - -Two engine constraints apply here and only here. A canonical body carrying a +structurally discriminated union back to the right kind. Exact JSON equality +in this example is not a general round-trip guarantee: implicit flags are +removed and degenerate array maps are collapsed. + +`index_array_bounds` constrains raw index-array values before the map's offset +and stride are applied. The message layer preserves these bounds, but the +engine cannot retain them through map operations. Both `IndexTransform.from_json` +and `output_index_map_from_json` therefore raise `NdselError("invalid_json", ...)` +when bounds differ from `["-inf", "+inf"]`. This includes one-sided constraints, +empty and singleton arrays, and zero-stride maps. Omitted or explicitly unbounded +bounds remain supported. Use the message layer to preserve constrained documents +for consumers that support them; lowering never silently discards an index-array +constraint. + +A canonical body carrying a `"-inf"` or `"+inf"` bound cannot be lowered — an `IndexDomain` addresses a finite array — so `IndexTransform.from_json` raises. And implicit bounds lower *by value*: the `[n]`-bracket flag is a message-layer concern, and the engine @@ -102,21 +122,17 @@ keeps only the integer. ndsel and TensorStore both **reject** an output map that carries both `input_dimension` and `index_array`. The in-memory [`ArrayMap`](api/output_map.md#zarr_indexing.output_map.ArrayMap), though, -records an `input_dimension` to pin the axis an orthogonal (`oindex`) array -varies over. The serializer bridges that gap in both directions: +records its dependency axes in its full-rank array shape, with no +`input_dimension` field: - **On serialize**, a non-degenerate `index_array` map is emitted *without* `input_dimension`. -- **On load**, the in-memory `input_dimension` is reconstructed from the - full-rank array's dependency axes — its non-singleton axes. An array that - solely owns a single non-singleton axis is orthogonal; arrays that share - non-singleton axes, or vary over several, are correlated (`vindex`), and get - `input_dimension = None`. A single 1-D array over a rank-1 domain is - inherently ambiguous between the two flavors and reconstructs as - orthogonal, which is behaviorally identical in that case. - -There is one deliberate exception, and it is the only place a round trip changes -representation rather than preserving it. An all-singleton `index_array` — size +- **On load**, dependency axes are the full-rank array's non-singleton axes. + Maps sharing these axes describe correlated coordinates. The engine also + accepts lower-rank nonempty arrays by prepending singleton axes; that + convenience is not a guarantee of compatibility with other ndsel consumers. + +An all-singleton `index_array` — size 1 — selects the same coordinate regardless of the input, so it is collapsed to a `constant` map on serialize: @@ -137,6 +153,11 @@ The transform is still valid and the output shape is unchanged. A length-1 `oindex` selection therefore round-trips behaviorally (an `ArrayMap` comes back as a `ConstantMap`) rather than by object identity. +Empty index arrays also serialize as constant maps with offset zero: the +empty input domain carries the fact that no coordinates are selected. This +avoids losing trailing shape information in JSON when an array has a leading +zero-length axis. + ## Conformance The package is checked against the language-agnostic ndsel conformance corpus, @@ -145,8 +166,9 @@ vendored unmodified under — one JSON file per message kind plus `errors.json`, with the source commit recorded in `PROVENANCE.md`. Each fixture is either a *success* case (`input` + expected `normalized` body) or an *error* case (`input` + expected -reason code), and an implementation is conformant iff `normalize` reproduces -every one. `tests/test_conformance.py` runs the whole corpus as one +reason code). Matching every fixture establishes corpus conformance; the +fixtures do not prove correctness for every possible input or universal +TensorStore compatibility. `tests/test_conformance.py` runs the whole corpus as one parametrized test per fixture, so a corpus update reports failures fixture by fixture rather than as a single opaque assertion. diff --git a/packages/zarr-indexing/docs/release-notes.md b/packages/zarr-indexing/docs/release-notes.md index 767cd56a84..d0687d8758 100644 --- a/packages/zarr-indexing/docs/release-notes.md +++ b/packages/zarr-indexing/docs/release-notes.md @@ -1,5 +1,6 @@