Skip to content

feat(data)!: zarr format v3 stores - #198

Open
janickm wants to merge 1 commit into
stack/zarr3/04-zarr-python-3from
stack/zarr3/05-zarr-format-v3
Open

janickm wants to merge 1 commit into
stack/zarr3/04-zarr-python-3from
stack/zarr3/05-zarr-format-v3

Conversation

@janickm

@janickm janickm commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Support writing and reading component stores in zarr format v3 with zarr-python 3 (zarr format v2 remains the default, readable with all zarr-python and ncore versions):

  • SequenceComponentGroupsWriter(zarr_format=...) / from_reader(), and stores.Store.create_root_group(zarr_format=...), validated against the formats supported by the installed zarr-python version (nodes.check_zarr_format, nodes.SUPPORTED_ZARR_FORMATS)
  • data converters: --zarr-format CLI option (BaseDataConverterConfig. zarr_format)
  • the consolidated metadata of zarr format v3 hierarchies is stored as CBOR + zlib encoded root zarr.json with the consolidated metadata of all members (zarr.json.cbor.zlib, decodes several times faster than CBOR + LZMA), next to the node metadata records written by zarr-python (which opens such hierarchies as unconsolidated)
  • binary blobs are uint8 arrays (fixed-length bytes are not part of the zarr format v3 specification), compression uses the blosc codec of zarr format v3

Component, store and converter tests run for both zarr formats. The time-trimmed zarr format v2 / v3 serializations of the test sequence (test-data-v4-trimmed 0.2) are compared by formats_test.

BREAKING CHANGE: BaseDataConverterConfig requires zarr_format.

@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@janickm
janickm added this pull request to stack #201 October 14, 2026 07:28
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test dcc7242

@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 34a417e

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from 34a417e to ce84fda Compare October 14, 2026 10:02
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test ce84fda

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from ce84fda to 8813b8a Compare October 14, 2026 10:09
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 8813b8a

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from 8813b8a to 453876c Compare October 14, 2026 11:58
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 453876c

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from 453876c to 11d8e51 Compare October 14, 2026 14:16
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 11d8e51

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from 11d8e51 to 2729fd1 Compare October 14, 2026 14:43
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 2729fd1

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from 2729fd1 to dc33afe Compare October 14, 2026 15:32
@janickm

janickm commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test dc33afe

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from dc33afe to edddd15 Compare October 15, 2026 07:55
@janickm

janickm commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test edddd15

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from edddd15 to eae36c9 Compare October 15, 2026 10:20
@janickm

janickm commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test eae36c9

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from eae36c9 to 115d009 Compare October 15, 2026 10:38
@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from 43cfb7f to 7ee8e64 Compare October 19, 2026 11:52
@janickm

janickm commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 7ee8e64

@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from 7ee8e64 to 78e9033 Compare October 19, 2026 11:58
@janickm

janickm commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 78e9033

Comment thread ncore/impl/data/backend.py Outdated
Comment thread ncore/impl/data/backend.py Outdated
Comment thread ncore/impl/data/backend.py Outdated
Comment thread ncore/impl/data/backend.py Outdated
Comment thread ncore/impl/data/backend_zarr3.py Outdated
Comment thread ncore/impl/data/backend_zarr3.py Outdated
Comment thread ncore/impl/data/backend_zarr3.py
Comment thread ncore/impl/data/backend_zarr3.py Outdated
Comment thread ncore/impl/data/backend_zarr3.py Outdated
Comment thread ncore/impl/data/backend_zarr3.py Outdated
Comment thread ncore/impl/data/stores_test.py Outdated
Comment thread ncore/impl/data/stores_test.py Outdated
Comment thread ncore/impl/data/stores_test.py Outdated
Comment thread ncore/impl/data/stores_test.py Outdated
Comment thread ncore/impl/data/stores_test.py Outdated
@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from 78e9033 to ad8827f Compare October 19, 2026 13:56
@janickm

janickm commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test ad8827f

@janickm
janickm marked this pull request as ready for review October 19, 2026 14:24
@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from ad8827f to d254e62 Compare October 21, 2026 08:37
@janickm

janickm commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test d254e62

Support writing and reading component stores in zarr format v3 with
zarr-python 3 (zarr format v2 remains the default, readable with all
zarr-python and ncore versions):

- SequenceComponentGroupsWriter(zarr_format=...) / from_reader(), and
  stores.Store.create_root_group(zarr_format=...), validated against the
  formats supported by the installed zarr-python version
  (nodes.check_zarr_format, nodes.SUPPORTED_ZARR_FORMATS)
- data converters: --zarr-format CLI option (BaseDataConverterConfig.
  zarr_format)
- the consolidated metadata of zarr format v3 hierarchies is stored as
  CBOR + zlib encoded root zarr.json with the consolidated metadata of
  all members (zarr.json.cbor.zlib, decodes several times faster than
  CBOR + LZMA), next to the node metadata records written by
  zarr-python (which opens such hierarchies as unconsolidated)
- binary blobs are uint8 arrays (fixed-length bytes are not part of
  the zarr format v3 specification), compression uses the blosc codec
  of zarr format v3

Component, store and converter tests run for both zarr formats. The
time-trimmed zarr format v2 / v3 serializations of the test sequence
(test-data-v4-trimmed 0.2) are compared by formats_test.

BREAKING CHANGE: BaseDataConverterConfig requires zarr_format.
@janickm
janickm force-pushed the stack/zarr3/05-zarr-format-v3 branch from d254e62 to 2d7ae55 Compare October 21, 2026 08:53
@janickm

janickm commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 2d7ae55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant