You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Prior art: how other RO-Crate/OCFL indexers work #68
How do existing RO-Crate / OCFL indexers solve the problems this map faces, and what can ldacapi borrow? Look at Oni's earlier indexers and LDaCA's previous stack (e.g. oni-ocfl, ocfl-tools, ro-crate-js / ro-crate-tools, arocapi's own examples), and report for each:
Which crate metadata they index: what counts as an entity, profile or type filtering, memberOf / hasMember / rootCollection handling, ldac: properties.
How they handle entities shared between crates (same @id in several crates).
How they compute aggregates for Oni (subtree counts, language, communicationMode, mediaType lists) and how they refresh ancestors when a crate changes.
How they decide licence / access control from crate metadata.
Where indexing ran (process, worker, CLI) and how state or progress was tracked.
Resolved: the findings come from reading the code of Oni (current main), the oni-ui CLI search indexer LDaCA ran, arocapi-ldaca, arocapi 5.0.0-beta.10 and its exploder branch, oni-ocfl, PARADISEC's Nabu (which serves the Oni entity API) and the LDaC profile. They're facts and options only; the decisions are for the grilling tickets. Full write-up with pinned source links: docs/research/indexer-prior-art.md on research/indexer-prior-art.
The Solr-era oni-indexer (UTS-eResearch / Arkisto-Platform) is gone (404), so it couldn't be read.
Which metadata they index
Oni structural index: one row per crate root only (crateId, licence, name, description), plus side tables for @type, memberOf (a NULL row means top level) and conformsTo. Files come from the OCFL inventory, not hasPart.
Oni and oni-ui search: a document only for entities whose conformsTo is the LDaC Collection/Object profile, or whose @type includes File. oni-ui also needs RepositoryCollection/RepositoryObject for members inside a bundled crate. People, places and languages are only embedded (getTree depth 1).
oni-ui walks across crates top-down: it starts at top-level collections from Oni's structural API and recurses on memberOf=<id>. Each document gets _root (the top-level collection), _memberOf (the immediate parent only; adding ancestors was dropped as "overwhelming"), _collectionStack (every ancestor) and _isTopLevel.
ldac: properties: oni-ui renames unprefixed terms whose context IRI is ldac: to ldac:<name>, so crates using either spelling index the same way. Text indexing follows (ldac:)indexableText/mainText, needs a text/* format, and is skipped if the licence sets allowTextIndex: false.
rootCollection: nobody reads it from the crate. It's always derived: by traversal (oni-ui), from the directory layout (arocapi's seed), or from Nabu's own tables.
arocapi-ldaca / arocapi exploder: turn every Collection/Object/File entity into its own small crate (like our ADR 0004, but stored). Their language/mediaType/communicationMode values were random dummy data.
Shared entities
No prior art solves it. The working precedent is one copy per crate: Oni search uses <crateId>/<@id> unless the @id starts with the crate id, oni-ui uses <top>/<crateId>/<@id>, and arocapi-ldaca's Entity has an autoincrement key. Oni's structural index avoids the problem by indexing roots only.
Oni deletes and reindexes one collection's subtree by crate-id prefix ("as a hack"), and can only delete the search index whole.
No precedent for a link table or reference counting.
Aggregates
oni-ui, at query time: ancestors are stored on every document. The portal computes subcollections (_memberOf), objects and facets (language, communicationMode, @type, which gives the file count) by aggregating over _collectionStack.@id = id. Ancestors never go stale. Descendants' copies of ancestor ids and names do, and the only fix was a full delete-and-rebuild.
Nabu, precomputed: an entities table holds items_count, essences_count and media_types. It was a SQL view until December 2025 and is now a table. after_commit callbacks recompute the parent and grandparent rows and queue async search reindexes. This stays simple because Nabu's depth is fixed at two levels.
arocapi: the dev Oni transformer hard-codes counts to zero. arocapi has no aggregate support.
Licence and access control
LDaC profile terms: a DataReuseLicense can carry ldac:access (OpenAccess / AuthorizedAccess), ldac:authorizationWorkflow (AuthorizationByApplication, AuthorizationByInvitation, AgreeToTerms, AccessControlList, SelfAuthorization) and ldac:openAccessIndex. No prior indexer reads any of them.
Oni / oni-ui, at index time: the content licence is the entity's own, else the parent's or root's, else a configured default. It must be an entity in the crate; a bare URI is rejected, and with nothing found the entity is skipped. The metadata licence and metadataIsPublic come from the ro-crate-metadata.json descriptor.
Oni, at query time: a config table maps licence URIs to groups. Licences not in the table are open to everyone. Results the user can't access are redacted rather than removed: only _text goes if the metadata is public, otherwise the hit is cut to a stub with error: not_authorized.
Nabu: maps its own access conditions to the LDaC workflow names for accessControl (Public / AgreeToTerms / AuthorizationByInvitation / AccessControlList), per request.
Where indexing ran
Oni: in the API process. Admin POSTs start it fire-and-forget; state lives in memory per index type ("indexed" means count > 0) and the API returns 409 while busy.
oni-ui: run-to-completion CLIs. The search CLI deletes and rebuilds the whole index, with refresh: true on each write. Failed documents are dumped as JSON files. Nothing is persisted.
arocapi-ldaca: a delete-all CLI, plus deposit routes that write to OCFL and index synchronously in the request.
oni-ocfl: only writes crates into OCFL (one transaction, after validation). Indexing is a separate, later step.
Nabu: the only one with persisted state. The entity table is updated synchronously in callbacks, and search reindexing runs as background jobs on Solid Queue, a database-backed job queue.
What ldacapi could borrow (options)
The profile-plus-File rule for what gets its own row.
ldac: normalisation using a bundled copy of the context.
Storing the full ancestor list on each document and aggregating in OpenSearch (oni-ui), or precomputed columns refreshed up the chain (Nabu).
Licence fallback chain: entity, then parent, then root, then a configured default.
Activity
johnf commentedon Nov 17, 2026
Resolved: the findings come from reading the code of Oni (current
main), the oni-ui CLI search indexer LDaCA ran, arocapi-ldaca, arocapi 5.0.0-beta.10 and itsexploderbranch, oni-ocfl, PARADISEC's Nabu (which serves the Oni entity API) and the LDaC profile. They're facts and options only; the decisions are for the grilling tickets. Full write-up with pinned source links:docs/research/indexer-prior-art.mdonresearch/indexer-prior-art.The Solr-era
oni-indexer(UTS-eResearch / Arkisto-Platform) is gone (404), so it couldn't be read.Which metadata they index
crateId, licence, name, description), plus side tables for@type,memberOf(a NULL row means top level) andconformsTo. Files come from the OCFL inventory, nothasPart.conformsTois the LDaC Collection/Object profile, or whose@typeincludesFile. oni-ui also needsRepositoryCollection/RepositoryObjectfor members inside a bundled crate. People, places and languages are only embedded (getTreedepth 1).memberOf=<id>. Each document gets_root(the top-level collection),_memberOf(the immediate parent only; adding ancestors was dropped as "overwhelming"),_collectionStack(every ancestor) and_isTopLevel.ldac:properties: oni-ui renames unprefixed terms whose context IRI isldac:toldac:<name>, so crates using either spelling index the same way. Text indexing follows(ldac:)indexableText/mainText, needs atext/*format, and is skipped if the licence setsallowTextIndex: false.rootCollection: nobody reads it from the crate. It's always derived: by traversal (oni-ui), from the directory layout (arocapi's seed), or from Nabu's own tables.exploder: turn every Collection/Object/File entity into its own small crate (like our ADR 0004, but stored). Their language/mediaType/communicationMode values were random dummy data.Shared entities
<crateId>/<@id>unless the@idstarts with the crate id, oni-ui uses<top>/<crateId>/<@id>, and arocapi-ldaca'sEntityhas an autoincrement key. Oni's structural index avoids the problem by indexing roots only.Aggregates
_memberOf), objects and facets (language, communicationMode,@type, which gives the file count) by aggregating over_collectionStack.@id = id. Ancestors never go stale. Descendants' copies of ancestor ids and names do, and the only fix was a full delete-and-rebuild.entitiestable holdsitems_count,essences_countandmedia_types. It was a SQL view until December 2025 and is now a table.after_commitcallbacks recompute the parent and grandparent rows and queue async search reindexes. This stays simple because Nabu's depth is fixed at two levels.countsto zero. arocapi has no aggregate support.Licence and access control
DataReuseLicensecan carryldac:access(OpenAccess / AuthorizedAccess),ldac:authorizationWorkflow(AuthorizationByApplication, AuthorizationByInvitation, AgreeToTerms, AccessControlList, SelfAuthorization) andldac:openAccessIndex. No prior indexer reads any of them.metadataIsPubliccome from thero-crate-metadata.jsondescriptor._textgoes if the metadata is public, otherwise the hit is cut to a stub witherror: not_authorized.accessControl(Public / AgreeToTerms / AuthorizationByInvitation / AccessControlList), per request.Where indexing ran
refresh: trueon each write. Failed documents are dumped as JSON files. Nothing is persisted.What ldacapi could borrow (options)
Filerule for what gets its own row.ldac:normalisation using a bundled copy of the context.allowTextIndex/ldac:openAccessIndexgating_text.accessControl.