Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions src/datajoint/builtin_codecs/hash.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""
Hash-addressed storage codec with SHA256 deduplication.
Hash-addressed storage codec with MD5-based deduplication.
"""

from __future__ import annotations
Expand All @@ -12,11 +12,13 @@

class HashCodec(Codec):
"""
Hash-addressed storage with SHA256 deduplication.
Hash-addressed storage with content-addressed deduplication.

The ``<hash@>`` codec stores raw bytes using hash-addressed storage.
Data is identified by its SHA256 hash and stored in a hierarchical directory:
``_hash/{hash[:2]}/{hash[2:4]}/{hash}``
Data is identified by a 26-character Base32-encoded MD5 digest of the
content, and stored at ``{hash_prefix}/{schema}/{hash}`` (``hash_prefix``
defaults to ``_hash``). Stores with subfolding configured insert
additional path segments: ``{hash_prefix}/{schema}/{fold1}/{fold2}/{hash}``.

The database column stores JSON metadata: ``{hash, store, size}``.
Duplicate content is automatically deduplicated across all tables.
Expand Down
6 changes: 4 additions & 2 deletions src/datajoint/builtin_codecs/npy.py
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,9 @@ class NpyCodec(SchemaCodec):
Schema-addressed storage for numpy arrays as .npy files.

The ``<npy@>`` codec stores numpy arrays as standard ``.npy`` files
using schema-addressed paths: ``{schema}/{table}/{pk}/{attribute}.npy``.
using schema-addressed paths: ``{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}.npy``
(``schema_prefix`` defaults to ``_schema``; ``{token}`` is a random
per-write suffix).
Arrays are fetched lazily via ``NpyRef``, which provides metadata access
without I/O and transparent numpy integration via ``__array__``.

Expand Down Expand Up @@ -269,7 +271,7 @@ class Recording(dj.Manual):

Storage Details:
- File format: NumPy .npy (version 1.0 or 2.0)
- Path: ``{schema}/{table}/{pk}/{attribute}.npy``
- Path: ``{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}.npy``
- Database column: JSON with ``{path, store, dtype, shape}``

Deletion: Requires garbage collection via ``dj.gc.GarbageCollector``.
Expand Down
6 changes: 4 additions & 2 deletions src/datajoint/builtin_codecs/object.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ class ObjectCodec(SchemaCodec):
Schema-addressed storage for files and folders.

The ``<object@>`` codec provides managed file/folder storage using
schema-addressed paths: ``{schema}/{table}/{pk}/{field}/``. This creates
schema-addressed paths: ``{schema_prefix}/{schema}/{table}/{pk}/{field}_{token}/``
(``schema_prefix`` defaults to ``_schema``; ``{token}`` is a random
per-write suffix). This creates
a browsable organization in object storage that mirrors the database schema.

Unlike hash-addressed storage (``<hash@>``), each row has its own unique path
Expand Down Expand Up @@ -51,7 +53,7 @@ def make(self, key):
Storage Structure:
Objects are stored at::

{store_root}/{schema}/{table}/{pk}/{field}/
{store_root}/{schema_prefix}/{schema}/{table}/{pk}/{field}_{token}/

Deletion: Requires garbage collection via ``dj.gc.GarbageCollector``.

Expand Down
Loading