Repository navigation
Conversation
Separate the record storage of indexed tar archives (_itar.py) from zarr-python, in preparation of supporting multiple zarr-python versions: - IndexedTarStore provides raw access to the records of an archive (mapping interface), ZarrIndexedTarStore adapts it to zarr-python's store interface (stores.ZarrIndexedTarStore / stores.zarr_store) - Read local archives via positional reads (os.pread), which don't serialize reads of multiple threads, and close their file descriptors also when stores are dropped without closing them - Serve records starting within the index tail read from memory - Share the CBOR + LZMA document encoding of the record index and the compressed consolidated metadata Archives are byte-identical to archives written by ncore 19.8 (tested via the sha256 of a fixed hierarchy written by ncore 19.8). BREAKING CHANGE: IndexedTarStore is no longer a zarr-python store, wrap it via stores.ZarrIndexedTarStore (or stores.zarr_store) to access zarr hierarchies of archives directly with zarr-python.
Collaborator
Author
|
Opened by mistake, this commit stays in #68. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the record storage of indexed tar archives (
ncore/impl/data/_itar.py) independently of zarr-python, in preparation of supporting multiple zarr-python versions. Factored out of #68.IndexedTarStoreprovides raw access to the records of an archive (mapping interface),stores.ZarrIndexedTarStoreadapts it to zarr-python's store interfaceos.pread), which don't serialize reads of multiple threads, and close their file descriptors also when stores are dropped without closing themArchives are byte-identical to archives written by ncore 19.8 (tested via the sha256 of a fixed hierarchy written by ncore 19.8).
Breaking change
IndexedTarStoreis no longer a zarr-python store. Wrap it viastores.ZarrIndexedTarStore(orstores.zarr_store) to access the zarr hierarchies of archives with zarr-python directly.