Skip to content

What @ocfl/ocfl-fs 1.0 beta offers for change detection and reading #67

Closed

Description

@johnf

Question

What does @ocfl/ocfl-fs 1.0 beta (1.0.0-beta.59 installed) offer that changes how and when ldacapi indexes? Specifically:

  • Detecting change: reading an object's head version or inventory digest cheaply, so the indexer can tell whether a crate changed since it was last indexed.
  • Listing objects in a storage root efficiently (streaming, pagination) at production scale.
  • Reading ro-crate-metadata.json and file metadata (size, digest, path) without reading content, including for metadata-only crates that list files they don't contain.
  • Writing: what Import and Deposit need (creating objects and versions) versus what indexing in place needs (read-only), and any locking or concurrency guarantees between a writer and an indexer.
  • Anything new in the 1.0 API (vs what src/ocfl.ts and src/ocfl-repository.ts use today) worth adopting.

Sources: node_modules/@ocfl/ocfl-fs, the ocfl-js repository, the OCFL 1.1 spec.

Activity

  1. self-assigned this
    on Nov 17, 2026
  2. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Answer

    Findings, with sources for each claim: docs/research/ocfl-fs-1-0-beta.md on research/ocfl-fs-1-0-beta. Sources: the installed 1.0.0-beta.59 source (node_modules/@ocfl/ocfl-fs/src/), ocfl-js main 6dc9df2 (guides, ADRs 0005 to 0007, commits since beta.59), and the OCFL 1.1 spec.

    Detecting change

    • No persistent change token is public. An opened object gives head.number and head.createdAt. The inventory digest is computed on every open but kept internal (_digest, src/object-on-disk.ts).
    • object.isCurrent() costs one stat (device, inode, size, mtime of the root inventory). It only works against an object held in memory, since the identity it compares is private, and it is "a cheap check, not a guarantee" (src/ocfl-object.ts).
    • Reading anything about an object, even its head number, means root.open(id), which reads, parses, hashes and checks the whole root inventory against its sidecar. root.has(id) is the only cheaper read, and it gives existence only.
    • On disk, OCFL 1.1 requires an inventory.json.<alg> sidecar beside every inventory, written last (§3.6, E058 to E062). The library doesn't give out object root paths (ADR 0006), but StorageRoot.layout is public and the id-to-path mapping is deterministic (§4.3, E083). During an ocfl-fs commit the sidecar briefly lags the root inventory.
    • There is no watch, event or "changed since" API.

    Listing

    • root.objects() is an async iterator over an opendir walk. It handles one object at a time, opening each one fully, in no order, and has no pagination, cursor, filter or resume. Damaged objects are yielded as { error, objectRoot, id? } and the walk carries on (src/list.ts).
    • Today src/ocfl.ts reads each inventory at least twice per run: once in idsMatching, once in indexObject.

    Reading

    • ro-crate-metadata.json can only be read through open() plus file.text().
    • file.path, file.digest, createdAt and modifiedAt come from the inventory with no I/O, and size() is one stat (src/ocfl-file.ts).
    • A file a crate lists but the object doesn't hold has no OcflFile at all. Today that is logged as an error in src/indexer/database.ts, which then falls back to contentSize.

    Writing and concurrency

    • Indexing in place uses only read calls, and none of them write.
    • Import and Deposit use create, update and a Draft: write, import, clear, and so on.
    • draft.import keeps files it doesn't replace, so scripts/crates-import.ts (no clear()) carries removed files into the next version.
    • Readers take no lock and never wait. Each write has one atomic commit point: the rename of the root inventory. An opened object is a snapshot.
    • Writers lock each object and refuse a second writer (UPDATE_IN_PROGRESS); they never queue or expire locks (ADR 0005).
    • During a remove, an object can open and then have files missing.

    Unreleased on ocfl-js main, breaking for ldacapi

    • 42550f8: a missing or empty storage root throws STORAGE_ROOT_NOT_FOUND instead of INVALID_STORAGE_ROOT/E069, which isMissing() in src/ocfl-repository.ts tests for.
    • 3b84570: draft.import(source, { target, filter }).
    • bb48d50: a missing stored file rejects with INVALID_OBJECT, not ENOENT.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions