Repository navigation
feat(data): asynchronous component reader API - #200
Merged
Merged
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
janickm
added this pull request to stack #201
October 14, 2026 07:28
Collaborator
Author
|
/ok to test 772164c |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 14, 2026 07:45
772164c to
4fc0ce3
Compare
Collaborator
Author
|
/ok to test 4fc0ce3 |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 14, 2026 10:02
4fc0ce3 to
2da6f4c
Compare
Collaborator
Author
|
/ok to test 2da6f4c |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 14, 2026 10:09
2da6f4c to
8d397a8
Compare
Collaborator
Author
|
/ok to test 8d397a8 |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 14, 2026 11:58
8d397a8 to
64f5779
Compare
Collaborator
Author
|
/ok to test 64f5779 |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 14, 2026 14:16
64f5779 to
83a25d1
Compare
Collaborator
Author
|
/ok to test 83a25d1 |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 14, 2026 14:43
83a25d1 to
5cbee05
Compare
Collaborator
Author
|
/ok to test 5cbee05 |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 14, 2026 15:32
5cbee05 to
592a27e
Compare
Collaborator
Author
|
/ok to test 592a27e |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 15, 2026 07:55
592a27e to
d8d030f
Compare
Collaborator
Author
|
/ok to test d8d030f |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 15, 2026 10:20
d8d030f to
794ce16
Compare
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 19, 2026 13:56
d0b9434 to
d13b6ac
Compare
Collaborator
Author
|
/ok to test d13b6ac |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 20, 2026 09:27
d13b6ac to
25b8674
Compare
Collaborator
Author
|
/ok to test 25b8674 |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 20, 2026 10:19
25b8674 to
8a969f1
Compare
Collaborator
Author
|
/ok to test 8a969f1 |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 20, 2026 11:58
8a969f1 to
3ae1545
Compare
Collaborator
Author
|
/ok to test 3ae1545 |
janickm
commented
Oct 20, 2026
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 20, 2026 12:48
3ae1545 to
f73982a
Compare
Collaborator
Author
|
/ok to test f73982a |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 21, 2026 08:37
f73982a to
3de89e6
Compare
Collaborator
Author
|
/ok to test 3de89e6 |
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 21, 2026 08:53
3de89e6 to
0135343
Compare
Collaborator
Author
|
/ok to test 0135343 |
janickm
marked this pull request as ready for review
October 21, 2026 08:59
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 21, 2026 09:11
0135343 to
8c2bf4e
Compare
janickm
force-pushed
the
stack/zarr3/07-async
branch
from
October 21, 2026 14:15
8c2bf4e to
5670f3f
Compare
Collaborator
Author
|
/ok to test 5670f3f |
Add asynchronous variants of all component reader methods reading array data (get_frame_data_async, get_frame_ray_bundle_data_async, get_frame_ray_bundle_return_data_async, get_generic_data_async, get_camera_mask_image_async, get_pc_*_async, labels get_data_async, ...), returning the same data as the synchronous methods, based on nodes.Array.read_async / read_bytes_async and IndexedTarArchive.read_async. Awaiting multiple reads concurrently overlaps their I/O and decompression, without blocking the running event loop: - zarr-python 3: chunks of remote indexed tar archives (asynchronous fsspec filesystems, e.g. s3fs, gcsfs, http) are read via concurrent exact range requests on the event loop of their filesystem, chunks of other zarr-python stores via their asynchronous API. On event loops of applications, blocking work (local reads, decompression, zarr-python's codec pipeline) runs on worker threads, on the event loops of the calling threads it runs inline - zarr-python 2: reads run on worker threads - IndexedTarArchive.read_async reads records of local archives on worker threads (common.async_util.run_in_thread) On remote storage, the asynchronous APIs fetch exact byte ranges concurrently, while the synchronous APIs read through the archive's buffered file object (documented in storage_and_access.rst). Metadata (attributes, names, timestamps) is held in memory and only provided synchronously.
Collaborator
Author
|
/ok to test 27912d5 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add asynchronous variants of all component reader methods reading array data (get_frame_data_async, get_frame_ray_bundle_data_async, get_frame_ray_bundle_return_data_async, get_generic_data_async, get_camera_mask_image_async, get_pc_*_async, labels get_data_async, ...), returning the same data as the synchronous methods, based on nodes.Array.read_async / read_bytes_async.
Awaiting multiple reads concurrently overlaps their I/O and decompression, without blocking the running event loop:
Metadata (attributes, names, timestamps) is held in memory and only provided synchronously.