Skip to content

What arocapi 5 beta offers an indexer #66

Closed

Description

@johnf

Question

What does arocapi 5 beta (5.0.0-beta.10 installed; check the latest beta and the main branch too) offer, or plan, that an indexer should build on? Specifically:

  • Its storage backends (storage/ocfl, storage/entity-meta, storage/s3, storage/directory): what each reads and whether any of them derive entities or file meta that ldacapi currently builds itself.
  • What arocapi reads from Entity and File rows (columns, meta keys) and from search documents, so we know the minimum an indexer must write.
  • Whether arocapi has or plans any indexer, ingest or deposit code, a deposit extension, or roCrateIds (an entity belonging to several crates).
  • The SearchBackend / OpenSearchBackend surface: bulk writes, per-call refresh, index settings and mappings.
  • Anything on its roadmap or open issues that would change the arocapi boundary (Indexer: redesign storage, indexing and index state #47's candidates: deriveUniqueEntityIds, pickSingleMemberOf, entityAsCrate, file storage meta, shared-entity upserts, a meta.crateId index).

Sources: node_modules/arocapi, github.com/crate-works/arocapi (code, issues, PRs), arocapi.crate-works.org.

Activity

  1. self-assigned this
    on Nov 17, 2026
  2. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Resolution

    Checked arocapi 5.0.0-beta.10 (installed). The latest beta and main have the same code: main is beta.10 plus one docs commit. Also checked arocapi's issues, PRs and ADRs, the docs site, and the RO-Crate API spec's deposit page.

    Findings: docs/research/arocapi-5-beta-indexer.md on branch research/arocapi-5-beta-indexer.

    Storage backends

    • None of them derives entities or file meta. OcflStorage, DirectoryStorage and S3Storage read a location the indexer wrote into meta (ocflObject+ocflPath, storagePath, storageKey), or a consumer locate function. EntityMetaMetadata serves whatever object is in Entity.meta.rocrate. None lists objects or parses crates (src/storage/*, arocapi ADR 0004).
    • OcflStorage opens one object per request and serves its head version. Size, ETag and last-modified come from OCFL, and Content-Type from File.mediaType. ocflPath is the decoded logical path (OCFL backend: object mapping, versions and delivery crate-works/arocapi#114).
    • The storage contract has no writes. A future deposit API would add a write key and locate files through locate (ADR 0004).

    What arocapi reads

    • Entity: the eight Record columns (id, name, description, entityType, memberOf, rootCollection, two licence ids). memberOf and rootCollection are resolved to {id,name}, and a missing parent comes back as null. /entities sorts on createdAt/updatedAt too. The model has indexes on memberOf, rootCollection and entityType only, none on meta.
    • File: filename, mediaType, size and meta (storage keys). /files?memberOf= filters on the file entity's memberOf.
    • arocapi itself reads only the storage keys from meta. The whole row goes to the consumer's access function and transformers.
    • Search: arocapi uses only hit ids, then loads Entity rows, dropping hits without a visible row. arocapi never derives a search document from an entity (ADR 0003). Fields used by searchVisibilityFilter must be in every document.

    Indexer, deposit, roCrateIds

    • None of these exists or is planned in arocapi. Its Prisma surface is read-only (findMany/findFirst/count), and /capabilities hard-codes deposit: { supported: false }. The docs say deposit is "coming soon", but there is no issue, branch or design. The old "crate exploder" (Write a "crate exploder" that can split a bundled crate into nice small parts crate-works/arocapi#1) was closed in March 2026, deferring to ldacapi.
    • roCrateIds comes from the RO-Crate API spec's deposit section, not arocapi. Materialisation is implementation-defined and may be many-to-one (one entity, several crates). The field is optional. arocapi's models have no such column.

    SearchBackend / OpenSearchBackend

    • The contract is index(docs, {tx?}) (upsert) and remove(ids) (idempotent). It has no count or clear: a rebuild is the consumer looping index.
    • OpenSearchBackend.index validates the whole batch, then sends one client.bulk (no chunking). It throws on the first failed item, and the bulk isn't atomic. It doesn't check that entity rows exist.
    • refresh (boolean | 'wait_for') is a factory option applied to every write. There is no per-call refresh.
    • createIndex() sets strict mappings from the field declarations and throws on a differing mapping. It sets no index settings.

    Roadmap against #47's candidates

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions