Skip to content

feat(data)!: typed node API for component groups - #195

Open
janickm wants to merge 1 commit into
stack/zarr3/01-itarfrom
stack/zarr3/02-nodes
Open

janickm wants to merge 1 commit into
stack/zarr3/01-itarfrom
stack/zarr3/02-nodes

Conversation

@janickm

@janickm janickm commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Access the zarr hierarchies of component stores via typed group / array wrappers instead of zarr-python APIs, as a prerequisite of supporting multiple zarr-python versions:

  • nodes.Group / nodes.Array: create and read (nested) groups, arrays, binary blobs and (typed) attributes; the wrapped zarr-python objects are accessible via .zarr
  • stores.Store: the stores of zarr hierarchies (context managers), creating / opening their root groups and storing their compressed consolidated metadata; stores.IndexedTarZarrStore (indexed tar archives) and stores.ZarrStoreAdapter (any other zarr-python store, e.g., in-memory stores, or local directories via .directory())
  • Version specific operations are implemented by a zarr-python 2 backend (backend_zarr2.py) of a backend interface (backend.py), generic over the zarr-python store type of its version
  • Reads return read-only arrays: complete reads of arrays stored as a single uncompressed chunk (e.g., encoded images) are views of the chunk record (no copies), Array.read_bytes returns the chunk record of blobs, all other reads are decoded by zarr-python (missing chunks read as the fill value)
  • Port all components to the node API (single chunk arrays, blosc lz4 / bit-shuffle compression), sharing frame / point cloud path lookups
  • JSON documents (e.g., generic meta data) are typed as json_types.JsonDocument, the JsonLike type moved to json_types.py

Written stores are byte-identical to stores written by ncore 19.8.

BREAKING CHANGE: The groups of component writers / readers (self._group) are nodes.Group instances (the wrapped zarr-python group is accessible via self._group.zarr), arrays read via nodes.Array are read-only. The stores functions consolidate_compressed_metadata / open_compressed_consolidated and the ConsolidatedCompressedMetadataStore class are replaced by the stores.Store classes. JsonLike moved from ncore.impl.data.types to ncore.impl.data.json_types.

@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@janickm
janickm added this pull request to stack #201 October 14, 2026 07:28
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 7f4849a

@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 5d2f73c

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 5d2f73c to 0588900 Compare October 14, 2026 10:02
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 0588900

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 0588900 to 476622d Compare October 14, 2026 10:09
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 476622d

Comment thread ncore/impl/data/backend.py
Comment thread ncore/impl/data/backend.py Outdated
Comment thread ncore/impl/data/_backend.py Outdated
Comment thread ncore/impl/data/json_types.py
Comment thread ncore/impl/data/json_types.py
Comment thread ncore/impl/data/backend_zarr2.py
Comment thread ncore/impl/data/_zarr2.py Outdated
Comment thread ncore/impl/data/nodes.py Outdated
Comment thread ncore/impl/data/nodes.py Outdated
Comment thread ncore/impl/data/stores.py Outdated
@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 476622d to 3b73d69 Compare October 14, 2026 11:58
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 3b73d69

Comment thread ncore/impl/data/backend.py Outdated
Comment thread ncore/impl/data/backend_zarr2.py Outdated
Comment thread ncore/impl/data/backend.py Outdated
Comment thread ncore/impl/data/backend_zarr2.py Outdated
Comment thread ncore/impl/data/backend_zarr2.py Outdated
Comment thread ncore/impl/data/v4/components.py Outdated
Comment thread ncore/impl/data/v4/components.py
Comment thread ncore/impl/data/v4/components.py Outdated
Comment thread ncore/impl/data/v4/components.py Outdated
Comment thread ncore/impl/data/stores_test.py Outdated
@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 3b73d69 to 59bf2dd Compare October 14, 2026 14:12
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 59bf2dd

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 59bf2dd to 06d93b6 Compare October 14, 2026 14:43
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 06d93b6

Comment thread ncore/impl/data/backend.py Outdated
@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 06d93b6 to 621c8e5 Compare October 14, 2026 15:32
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 621c8e5

Comment thread ncore/impl/data/nodes.py Outdated
@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 621c8e5 to ba9c73b Compare October 15, 2026 07:55
@janickm

janickm commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test ba9c73b

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from ba9c73b to 10e58bb Compare October 15, 2026 10:20
@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from d291bc5 to 9152af1 Compare October 15, 2026 12:54
@janickm

janickm commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 9152af1

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 9152af1 to e2bc65b Compare October 15, 2026 13:45
@janickm

janickm commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test e2bc65b

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from e2bc65b to 6682f53 Compare October 15, 2026 13:58
@janickm

janickm commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 6682f53

@janickm
janickm marked this pull request as ready for review October 15, 2026 14:32
@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 6682f53 to 35c5aab Compare October 15, 2026 14:33
@janickm

janickm commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 35c5aab

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 35c5aab to 37725e8 Compare October 15, 2026 18:04
@janickm

janickm commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 37725e8

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 37725e8 to 2ad8192 Compare October 16, 2026 13:23
@janickm

janickm commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 2ad8192

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 2ad8192 to 09a05cf Compare October 16, 2026 13:37
@janickm

janickm commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 09a05cf

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 09a05cf to 75e3c1e Compare October 19, 2026 08:56
@janickm

janickm commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 75e3c1e

@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 75e3c1e to 1fb39e2 Compare October 19, 2026 11:52
@janickm

janickm commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 1fb39e2

Access the zarr hierarchies of component stores via typed group /
array wrappers instead of zarr-python APIs, as a prerequisite of
supporting multiple zarr-python versions:

- nodes.Group / nodes.Array: create and read (nested) groups, arrays,
  binary blobs and (typed) attributes; the wrapped zarr-python objects
  are accessible via .zarr
- stores.Store: the stores of zarr hierarchies (context managers),
  creating / opening their root groups and storing their compressed
  consolidated metadata; stores.IndexedTarZarrStore (indexed tar
  archives) and stores.ZarrStoreAdapter (any other zarr-python store,
  e.g., in-memory stores, or local directories via .directory())
- Version specific operations are implemented by a zarr-python 2
  backend (backend_zarr2.py) of a backend interface (backend.py),
  generic over the zarr-python store type of its version
- Reads return read-only arrays: complete reads of arrays stored as a
  single uncompressed chunk (e.g., encoded images) are views of the
  chunk record (no copies), Array.read_bytes returns the chunk record of
  blobs, all other reads are decoded by zarr-python (missing chunks read
  as the fill value)
- Port all components to the node API (single chunk arrays, blosc lz4 /
  bit-shuffle compression), sharing frame / point cloud path lookups
- JSON documents (e.g., generic meta data) are typed as
  json_types.JsonDocument, the JsonLike type moved to json_types.py

Written stores are byte-identical to stores written by ncore 19.8.

BREAKING CHANGE: The groups of component writers / readers (self._group)
are nodes.Group instances (the wrapped zarr-python group is accessible
via self._group.zarr), arrays read via nodes.Array are read-only. The
stores functions consolidate_compressed_metadata /
open_compressed_consolidated and the ConsolidatedCompressedMetadataStore
class are replaced by the stores.Store classes. JsonLike moved from
ncore.impl.data.types to ncore.impl.data.json_types.
@janickm
janickm force-pushed the stack/zarr3/02-nodes branch from 1fb39e2 to c62d1f8 Compare October 19, 2026 11:58
@janickm

janickm commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test c62d1f8

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant