perf(data): direct chunk reads with zarr-python 3 by janickm · Pull Request #199 · NVIDIA/ncore

janickm · 2026-10-14T07:28:04Z

zarr-python 3's codec pipeline decodes each chunk into a temporary buffer and copies it into the output array (zarr-python#2904), and its per-read overhead dominates reads of small frame arrays (~550 us per frame read vs ~200 us with zarr-python 2).

Basic selections (integers, unit-step slices, Ellipsis) of arrays of indexed tar archives opened for reading are now read directly: chunks are determined by zarr-python's indexer, read from the archive and blosc-decompressed into the output array (in place for complete chunks). Uncompressed single-chunk arrays (e.g., encoded images) are read-only views of their chunk record (no decoding, no copies). Large chunks of a read (>= 2 MiB) are decoded concurrently on a thread pool, bounded to two concurrent reads per process. Supported are regular chunk grids, C order, numeric / boolean data of native byte order and fixed-length bytes, without filters, uncompressed or blosc compressed (all arrays written by ncore), all other arrays and selections are read via zarr-python.

Read plans are derived per read (sub-microsecond, nothing is cached), results are read-only (as all read results), missing chunks read as the fill value, and the parallel reads state is re-created in forked processes.

NCORE_ZARR3_DIRECT_READS=0 reads all arrays via zarr-python.

Frame reads of the test sequence take ~240 us instead of ~550 us.

copy-pr-bot · 2026-10-14T07:28:08Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

janickm · 2026-10-14T07:29:17Z

/ok to test 34e1743

janickm · 2026-10-14T07:46:39Z

/ok to test 023cc64

janickm · 2026-10-14T10:02:34Z

/ok to test b38bb88

janickm · 2026-10-14T10:10:05Z

/ok to test f052a83

janickm · 2026-10-14T11:58:52Z

/ok to test 2d3c0ed

janickm · 2026-10-14T14:16:23Z

/ok to test ee42290

janickm · 2026-10-14T14:43:15Z

/ok to test 1f00b76

janickm · 2026-10-14T15:33:00Z

/ok to test 2008352

janickm · 2026-10-15T07:55:24Z

/ok to test 4a70f2e

janickm · 2026-10-19T08:57:07Z

/ok to test c17b77c

janickm · 2026-10-19T11:52:23Z

/ok to test b10bd34

janickm · 2026-10-19T11:58:54Z

/ok to test e63f3c8

janickm · 2026-10-19T13:56:19Z

/ok to test d817878

janickm · 2026-10-20T09:27:27Z

/ok to test 56bc675

janickm · 2026-10-20T10:19:40Z

/ok to test 3574994

janickm · 2026-10-21T08:37:34Z

/ok to test 1277ca0

janickm · 2026-10-21T08:53:59Z

/ok to test 7d0965e

zarr-python 3's codec pipeline decodes each chunk into a temporary buffer and copies it into the output array (zarr-python#2904), and its per-read overhead dominates reads of small frame arrays (~550 us per frame read vs ~200 us with zarr-python 2). Basic selections (integers, unit-step slices, Ellipsis) of arrays of stores opened for reading are now read directly: zarr-python's indexer determines the chunks of a read, which are read from the store and blosc-decompressed into the output array (in place for complete chunks that are contiguous blocks of the output array). Records of indexed tar archives are read directly from the archive (positional reads), records of other zarr-python stores (e.g., directories) via the store. - Uncompressed single-chunk arrays (e.g., encoded images) are read-only views of their chunk record (no decoding, no copies) - Large chunks of a read (>= 2 MiB) are decoded concurrently on a thread pool (at most 8 threads), bounded to two concurrent reads per process - Supported are all arrays written by ncore: regular chunk grids, C order, numeric / boolean data of native byte order and fixed-length bytes, uncompressed or blosc compressed, without filters. All other arrays (e.g., other codecs, written by other zarr writers) and selections are read via zarr-python Read plans are derived per read (sub-microsecond, nothing is cached), results are read-only (as all read results), missing chunks read as the fill value, and the parallel reads state is re-created in forked processes. NCORE_ZARR3_DIRECT_READS=0 reads all arrays via zarr-python. Frame reads of the test sequence take ~240 us instead of ~550 us. Multi-chunk arrays (e.g., point clouds chunked along points) read 1.4x to 3.7x faster, and up to ~17x for multiple large chunks (concurrent, in-place decompression).

janickm added this pull request to stack #201 October 14, 2026 07:28

janickm mentioned this pull request Oct 14, 2026

feat(data)!: support zarr-python 3 alongside zarr-python 2 #68

Closed

janickm force-pushed the stack/zarr3/06-direct-reads branch from 34e1743 to 023cc64 Compare October 14, 2026 07:45

janickm force-pushed the stack/zarr3/06-direct-reads branch from 023cc64 to b38bb88 Compare October 14, 2026 10:02

janickm force-pushed the stack/zarr3/06-direct-reads branch from b38bb88 to f052a83 Compare October 14, 2026 10:09

janickm force-pushed the stack/zarr3/06-direct-reads branch from f052a83 to 2d3c0ed Compare October 14, 2026 11:58

janickm force-pushed the stack/zarr3/06-direct-reads branch from 2d3c0ed to ee42290 Compare October 14, 2026 14:16

janickm force-pushed the stack/zarr3/06-direct-reads branch from ee42290 to 1f00b76 Compare October 14, 2026 14:43

janickm force-pushed the stack/zarr3/06-direct-reads branch from 1f00b76 to 2008352 Compare October 14, 2026 15:32

janickm force-pushed the stack/zarr3/06-direct-reads branch from 2008352 to 4a70f2e Compare October 15, 2026 07:55

janickm mentioned this pull request Oct 15, 2026

feat(data)!: typed node API for component groups #195

Open

janickm force-pushed the stack/zarr3/06-direct-reads branch from 4a70f2e to daab804 Compare October 15, 2026 10:20

janickm force-pushed the stack/zarr3/06-direct-reads branch from c17b77c to b10bd34 Compare October 19, 2026 11:52

janickm force-pushed the stack/zarr3/06-direct-reads branch from b10bd34 to e63f3c8 Compare October 19, 2026 11:58

janickm force-pushed the stack/zarr3/06-direct-reads branch from e63f3c8 to d817878 Compare October 19, 2026 13:56