Repository navigation
Description
Problem Statement
ldacapi's indexer was written to get crates searchable quickly. It doesn't hold up as the catalogue grows or as crates start arriving in more than one way.
- Indexing runs inside the API. A large crate stalls requests, and an API restart abandons the run.
- Admins can't trust the state they see. Index state lives in memory, so it's lost on restart and differs between replicas. Admins can't tell which crates are indexed, which failed, or why.
- Shared entities break on delete. Deleting one crate's indexes removes people and organisations that other crates also describe (Deleting a crate's database index removes entities other crates share #35).
- Every Oni page walks the whole tree. Collection counts and language lists are worked out per request with a recursive query, across up to ~30,000 entities.
- Errors are logged and skipped, so a crate can look indexed with pieces missing.
- The code isn't ready for what's next. Deposit, Import and index in place all need "index one crate in the repository", but the code is tangled together with module-level singletons and bundles listing crates, index state, indexing and file storage behind one interface.
- It assumes old versions. The code predates arocapi 5 and
@ocfl/ocfl-fs1.0 and doesn't use what they offer.
Solution
The indexer is rebuilt as a separate Worker that drains a durable queue of Index jobs kept in Postgres. Each job indexes one crate from the repository into both the Database index and the Search index, and precomputes that crate's share of Oni's Aggregates.
- The API, the admin pages, the CLI, Import and Deposit only ask for jobs.
- The Admin pages show one Index state per crate (not indexed, queued, indexing with progress, deleting, indexed, failed with the error) and warn when no worker is running.
- A crate is either wholly indexed, with an Indexed version, or it isn't.
- Shared entities survive the deletion of any one crate that contains them.
- Oni's counts and lists come from one flat query per page.
- Production gets this on a fresh VM by indexing every crate before traffic switches.
User Stories
- As an admin, I want indexing to run outside the API process, so that a large crate doesn't slow down search or browsing.
- As an admin, I want indexing to carry on when the API restarts, so that deploying the API doesn't abandon work.
- As an admin, I want to ask for one crate to be indexed from the admin pages, so that I can fix a single crate without touching the rest.
- As an admin, I want to ask for a crate and every crate under it to be indexed, so that I can refresh a whole collection spread across crates.
- As an admin, I want to ask for every crate in the repository to be indexed, so that I can bring the indexes into line with the repository.
- As an admin, I want indexing to skip crates whose head version is already indexed, so that "index everything" is cheap when little has changed.
- As an admin, I want to Reindex a crate regardless of its indexed version, so that I can rebuild it after a change in what the indexer derives.
- As an admin, I want to delete one crate's indexes, so that I can withdraw it from search without deleting it from the repository.
- As an admin, I want "delete everything" to discard waiting jobs and clear both indexes, so that I can start again from nothing.
- As an admin, I want "index everything" to delete the indexes of crates that are no longer in the repository, so that removed crates stop appearing in search.
- As an admin, I want asking again for a crate that is already waiting to do nothing, so that impatient clicks don't pile up work.
- As an admin, I want asking for a crate that is already being indexed to leave one follow-up job, so that a version deposited mid-run still gets indexed.
- As an admin, I want to see each crate's index state on the admin pages, so that I know what is searchable.
- As an admin, I want an indexed crate to show its version, when it was indexed, and how many entities and files it holds, so that I can check it against the repository.
- As an admin, I want a failed crate to show the error, the stage it failed in, the entity or file it was on, when it failed and the attempt count, so that I can fix the data without reading logs.
- As an admin, I want a crate being indexed to show its stage and a done-of-total counter, so that I can tell a slow job from a stuck one.
- As an admin, I want the top of the crates page to count crates per index state, so that I can see a whole run's progress at a glance.
- As an admin, I want a warning whenever no worker is running, so that I don't wait on jobs nobody will pick up.
- As an admin, I want any error inside a crate to fail its job, so that "indexed" always means the whole crate is in the indexes.
- As an admin, I want a failed crate fixed by simply asking again, so that I need no repair procedure after a half-finished job.
- As an admin, I want a dead worker's job to be retried automatically, up to three times, so that a crashed VM doesn't leave crates stuck.
- As an admin, I want the worker to pause and reconnect during a database or OpenSearch outage, so that an outage doesn't fail every waiting job.
- As an admin, I want a crate describing a collection, object or file that another crate already holds to fail its job, with an error naming both crates, so that I find and fix the data error.
- As an admin, I want admin pages to read only Postgres, so that they load quickly whatever the size of the repository.
- As an operator, I want to run more than one worker, so that a large backlog clears faster.
- As an operator, I want any number of API replicas, workers and CLI runs to share one queue with each job claimed once, so that scaling never indexes a crate twice at once.
- As an operator, I want
pnpm indexto queue jobs, drain the queue, print a summary of the crates it queued (indexed, skipped, deleted, failed, with each error) and exit non-zero on failure, so that I can script indexing. - As an operator, I want
pnpm crates:importto queue each crate it writes and drain the queue, so that imported crates are searchable when it finishes. - As an operator, I want
pnpm devto start the worker beside the API, so that local development indexes like production. - As an operator, I want the worker and CLI to log one line per job start, finish and failure, with fixed fields on failure, so that log monitoring can alert on failures.
- As an operator, I want the indexer to index an existing OCFL repository in place with "index everything", so that I can serve a repository without importing it.
- As an operator, I want the open-licence list to be required deployment configuration, so that each deployment states which licences are open and the app refuses to start without it.
- As an operator, I want the largest collection (~30,000 entities) indexed in under 10 minutes with the worker under 1 GB, so that a full rebuild fits a working day on modest hardware.
- As an operator, I want a fresh VM brought up by migrating, loading crates, starting the API and worker, indexing everything, then switching traffic once every crate shows indexed, so that going live needs no data migration.
- As a depositor, I want a deposit to succeed once the crate is stored, so that an indexing problem doesn't lose or reject my deposit.
- As a depositor, I want my deposit indexed from what was stored in the repository, so that what is searchable matches what was preserved.
- As a researcher using Oni, I want every profiled collection, object, person, organisation and software entity, and every file, to be findable, so that nothing described in a crate is hidden (ADR 0003).
- As a researcher, I want each entity's Virtual crate served with property names as deposited, so that what I see is what the depositor wrote (ADR 0004).
- As a researcher, I want files a crate describes but the repository doesn't hold to appear with no preview or download, so that a Metadata-only crate shows its full contents without broken links.
- As a researcher, I want collections and objects linked to their parent by
pcdm:memberOforpcdm:hasMember, and files byisPartOforhasPart, so that breadcrumbs and "Member of" are right. - As a researcher, I want each collection, object and file to know its root collection, even when its parent is in another crate, so that I can filter search by top-level collection.
- As a researcher, I want a person, organisation or software entity described by several crates to appear once, so that search results aren't full of duplicates.
- As a researcher, I want a shared entity to show one crate's whole description, preferring one with an open metadata licence, so that I never see a mix of descriptions or restricted metadata shown through an open copy.
- As a researcher, I want a shared entity to stay when one of the crates describing it is deleted, so that other collections keep their people and organisations (Deleting a crate's database index removes entities other crates share #35).
- As a researcher, I want Oni to show how many collections, objects and files are below an entity, and which languages, communication modes and media types it and they have, so that I can judge a collection before opening it.
- As a researcher, I want a collection's aggregates to include members held in other crates, so that a collection split across crates reads as one.
- As a researcher, I want aggregate lists to use the same labels as the search facets, so that clicking a value lands on a matching search.
- As a researcher, I want Oni pages with aggregates to load in about the same time as without them, so that browsing large collections stays quick.
- As a researcher, I want an entity's content licence to fall back through its ancestors in the same crate, so that an object without its own licence inherits its collection's.
- As a researcher without access to restricted content, I want entities with an open licence marked public and everything else marked as needing authorisation, so that I know what I can open.
- As a researcher, I want the text of text files searchable, so that I can find items by what they say.
- As a developer, I want the rules about one crate in a pure derivation step with no I/O, so that I can test them quickly and Deposit could later reuse them on a crate in hand.
- As a developer, I want the API to depend only on a small index queue module, never on the indexer, with a lint rule enforcing it, so that the boundary can't erode.
- As a developer, I want the indexer to expose a single "work the queue" operation used by both the worker and the CLI, so that the two can't drift apart on retries and heartbeats.
- As a developer, I want entry points to open the repository and clients and pass them in, with no module-level mutable state, so that tests can supply their own.
- As a developer, I want a crate builder for tests, so that each test states the crate it needs inline instead of relying on shared fixture files.
- As a maintainer, I want arocapi left read-only with every write in ldacapi, so that the redesign waits on no upstream release.
- As a maintainer, I want the id rule and file storage meta builder kept as plain functions with no LDaC imports, so that they could move upstream if another deployment needs them.
Implementation Decisions
Modules
-
Index queue (new, deep): the only indexing module the API, admin pages, CLI and (later) Deposit import.
- Interface: ask for index, reindex or delete jobs (one crate, a crate and those under it, everything, delete everything), and read index state (per crate, and counts per state).
- Enforces one running and one waiting job per crate.
-
Indexer (new, deep): one operation, "work the queue", run until the queue is empty (CLI) or forever (worker). Internally:
- claiming jobs, heartbeats and retries
- reading a crate from the repository
- derivation
- three writers: rows, search documents, aggregates
None of these is exposed.
-
Derivation (inside the indexer): pure. Crate metadata plus a few facts looked up beforehand (the parent crate chain, which files are held and their sizes, configuration) go in. Entity and file rows, crate links, search documents and aggregate rows come out. It imports neither configuration nor the database client.
-
Repository: opens or creates the OCFL repository and builds the content storage backend, the file storage meta builder and the id rule. It replaces the old
Repositoryinterface and module-level singletons, which are deleted. -
Access: the single "is this licence open" function, used both by derivation for the stored label and by the request-time licence check, plus the access-control label type.
-
Oni: the entity transformer reads aggregates with one flat query per page, replacing the recursive
memberOfquery. -
Worker entry point (new): starts "work the queue" forever. The API entry point builds the index queue and passes it to the app. The admin pages receive only the index queue.
-
A lint rule bans importing the indexer from anywhere except the indexer itself, the worker entry point, the CLI scripts and tests.
-
ldacapi stays one package. API, worker and CLI ship in one image.
Derivation rules
Each is named so that tests and implementation issues can point at it.
-
Entities indexed by profile: Collection, Object, Person, Organisation and SoftwareApplication get a row and a document when they declare the matching LDaC profile in
conformsTo, and every File does with no profile needed. Other entities appear only inside the virtual crate of what links to them. -
Parent of a collection or object:
pcdm:memberOf, else reversepcdm:hasMember. -
Parent of a file:
isPartOfor reversehasPart, preferring aRepositoryObject. -
Several candidate parents: the first wins, and a warning is logged.
-
Root collection: found by reading parent crates from the repository, never from the indexes. It is always null for people, organisations and software.
-
Entity ids: a relative or blank-node
@idbecomes<crate root id>/<@id>. Ids under the crate root, and absolute IRIs, are kept as they are. -
Content licence: the entity's own
license, else the nearest ancestor in the crate, else the crate root, else the configured default. -
Metadata licence: the metadata descriptor's
license, else the configured default, for every entity in the crate. -
Access control label:
Publicwhen the content licence is in the open-licence list, otherwiseAuthorizationByApplication, including when there is no licence. Crate licence metadata (ldac:access,ldac:authorizationWorkflow) is not read. -
Spellings: LDaC and PCDM properties are read only in their prefixed form (
ldac:,pcdm:). The older bare spelling is not read and nothing warns about it. The virtual crate serves property names as deposited, and the deposited crate is never changed. -
Files described but not held:
- The file gets its row and document, plus a File row with no storage location.
- Size is the stated
contentSize, else 0. Media type falls back toapplication/octet-stream. - Its virtual crate gets
display: false. - One warning per crate gives the count of such files.
-
File text: the text of every
text/*file is indexed whatever its licence, but only its first 1 MB. A file cut short is logged. -
Aggregates contribution: for each entity with something of this crate's at or below it, including ancestors in parent crates, the crate's row holds:
- the
collections,objects,subCollectionsandfilescounts below the entity - the
language,communicationModeandmediaTypevalues at or below it, using the search facet labels
Counts follow the indexer's classification. Files are counted whether held or not, and the entity itself isn't counted. People, organisations and software get zero counts and empty lists.
- the
-
Row and document contents: as today, except
meta.crateIdis dropped. The LDaC profile map becomes a constant inside derivation.
Schema (our own models; arocapi's Entity and File models are unchanged)
- Index job:
- Crate id (none for "delete everything"), kind (index, reindex or delete), status, attempt count, claiming worker.
- While running: stage and a done-of-total counter.
- When failed: message, stage, entity or file id, time.
- Partial unique indexes enforce one running and one waiting job per crate. Jobs are claimed with
FOR UPDATE SKIP LOCKED. - A successful job is deleted. A failed job is kept until the next job for that crate replaces it.
- Crate record: one per crate known to be in the repository, holding the name, indexed version, when it was indexed, and the entity and file counts its job wrote. Deleting a crate's indexes keeps the record. It is removed only when a job finds the crate gone from the repository.
- Worker: one row per worker or draining CLI run, with its last heartbeat.
- Crate link: one row per (crate, entity) pair, indexed both ways, holding that crate's derived copy of a shared entity. It is the only record of which crates contain an entity.
- Aggregates: one row per (crate, entity) pair with the counts and lists from derivation rule 13.
- The migration may empty the old Entity and File rows. There is no upgrade path for existing data.
Job lifecycle
- Index and reindex: clear the indexed version, write rows, then documents, then aggregates, then set the version, time and counts.
- Delete: clear the version, then remove documents, aggregates and rows. The indexed version is always written last and cleared first.
- Incremental: a crate is skipped only when its indexed version equals its head version.
- Missing crates:
- Indexing a named crate that is missing deletes its indexes and succeeds.
- "Index everything" queues deletes for crates that have indexes but are gone from the repository.
- Read once: a job carries only the crate id and opens the crate's OCFL object once.
- Errors:
- Any error inside a crate fails the job at the first error.
- A crate claiming a collection, object or file another crate holds is such an error. The holder keeps the row.
- Recovery is asking again; index and delete are both safe to repeat.
- Workers:
- A worker writes a heartbeat every 30 seconds, and that write also saves the running job's progress.
- A worker silent for 5 minutes is dead: its job returns to waiting with the attempt count raised, and is marked failed on the third return.
- A clean shutdown releases the job at once.
- A job checks it still owns its claim before writing its result.
- In an outage, the worker pauses and reconnects.
Shared entities (ADR 0006)
- Copies: each link keeps its crate's copy. The row and document are wholly one crate's copy: an open metadata licence wins first, then the lowest crate id, so a rebuild gives the same rows in any order.
- Delete: deleting a crate removes its links, then the entities and documents with no link left.
- Rebuild from links: a shared entity whose winning crate goes is rebuilt from the next crate's stored copy, with no repository read. Reindexing a crate that no longer contains an entity drops its link the same way.
Admin pages
- Data source: Postgres only; never the repository.
- One state per crate, in this order of precedence:
- a running or waiting job: indexing (stage, counter, start time), queued or deleting
- a failed job: failed (error, stage, entity or file id, attempt)
- otherwise the crate record: indexed (version, time, counts) or not indexed
- Summary: crates per state from one grouped query. "No worker running" shows whenever no heartbeat is fresh.
- Not shown: an "out of date" state, or a grand total of entities.
Performance
These figures are starting points, tuned by the benchmark.
- Target: the largest collection is indexed in under 10 minutes with the worker under 1 GB. This holds both as one crate and split across thousands of crates.
- Aggregates read: a page of 100 entities with aggregates is at most 50 ms slower than without. If it isn't, add an index on the aggregates table.
- Memory: one crate's metadata is held whole in memory. File text is read one file at a time.
- Database writes: batches of about 1,000 rows, each batch its own transaction. Aggregates go last. There isn't one transaction per crate.
- Search writes: bulk requests capped at about 5 MB or 1,000 documents. The worker never forces a refresh.
- Throughput: one job at a time per worker, with no worker threads.
- Parent cache: an in-memory cache of crate → parent collection (ids only), checked against the repository before use.
Configuration and arocapi
- Open licences:
OPEN_LICENSESbecomes a required environment setting with no default in code. - Default licences:
DEFAULT_LICENSEandDEFAULT_METADATA_LICENSEstay optional. - arocapi: stays read-only at 5.0.0-beta.10. The index-template workaround for search index settings stays in one place until OpenSearchBackend: a
settingsoption for the index it creates crate-works/arocapi#143 ships.
Testing Decisions
- What a good test is: a test checks what the indexer produces (rows, documents, aggregates, index state, HTTP responses) through a public seam, never how it gets there.
- Hard cases are staged with data, not mocks or fake clocks. A dead worker is a worker row with an old heartbeat.
- Each implementation issue names the derivation rules and hard cases it owes tests for, because coverage only shows that a line ran.
- Seams:
- Derivation: every derivation rule above, tested by name, with no database.
- "Work the queue": against real Postgres, OpenSearch and a temporary OCFL repository, for everything involving stored state or more than one crate:
- incremental skip, reindex and delete
- shared entities: the winner, rebuild when the winner is deleted, both indexing orders giving the same rows
- a second crate claiming a collection: its job fails naming both crates, and the holder keeps the row
- a child crate indexed before its parent, then reindexed
- a damaged crate fails while the others index
- a dead worker's job is retried, and fails on its third return
- two workers running at once index each crate once
- a failure part-way leaves no indexed version, and asking again succeeds
- Index queue: its own small tests (one waiting and one running job per crate, how index state is derived).
- Crate-to-HTTP: a thin layer, one or two tests per thing arocapi or Oni serves (search, root collection, access by licence, aggregates, file content).
- One test reaches inside the indexer: a worker wrongly judged dead has its result write refused.
- Not unit-tested: the three writers get no tests of their own.
- Not automated: an outage.
- Throw away and move: the old indexer tests are discarded and written fresh. The existing crate-to-HTTP tests move into the top-level test folder and keep their assertions.
- Test data: a crate builder replaces most fixture crates. The few realistic crates kept for Import and bootstrap move to the
ldac:spellings. Thesub_col_2.txtfixture becomes the named case "a file described but not held". - Coverage: the indexer and index queue are held at 100% from their first commit. The rest follows the auto-rising threshold already on
main. Logic in the import script moves out of the entry point so it is measured. - Prior art:
- The crate-to-HTTP tests (root collection, damaged crate, collection cycle, licence, Oni, search total) for the HTTP layer.
- The bootstrap test, as the one test that runs the real CLI end to end.
- The existing set-up of one database and search index per run, with test files run one at a time, plus a new "empty everything" helper before each test.
- Benchmark: built early in implementation over a generated 30,000-entity collection in both shapes, with text files of 50 KB, 1 MB and 5 MB. It checks the indexing target and the aggregates read.
Out of Scope
- The Deposit API's own design, including how a depositor polls index state. Deposit's only dependency here is one "add a job" call.
- Making upstream arocapi changes. The only upstream ask, OpenSearchBackend: a
settingsoption for the index it creates crate-works/arocapi#143, is already filed. - A no-gap rebuild after go-live (a second search index and a switch). Until then, "reindex all" runs with the API up, and a search mapping change takes a planned maintenance window.
- Migrating existing rows or documents. Production starts on a fresh VM.
- Scheduled indexing or watching OCFL for changes.
- Automatically reindexing child crates when a parent crate changes.
- Notifications and a health endpoint.
- Reading crate licence metadata, other LDaC authorisation workflows, and warnings about inconsistent licences.
- Support for the older bare spelling of
ldac:/pcdm:properties. - Listing a person's objects on their page.
Further Notes
- Decisions: each section above comes from a resolved ticket on Map: indexer redesign, where the reasoning and rejected options live:
- What the indexer derives from a crate
- Where indexing runs and what triggers it
- Where index state lives and what the admin pages read
- How shared entities are stored and deleted
- Oni aggregates: precompute or compute per request
- The access control rule
- The arocapi boundary
- Packaging the indexer, with Deposit in mind
- Layout of src/ after the redesign
- Cutting production over to the new indexer
- Progress and failure reporting for index runs
- Indexing performance at scale
- Test strategy for the indexer
- ADRs:
- 0005: indexing runs in a worker
- 0006: a shared entity's row is one crate's copy. Updated in docs: a crate that clashes with another fails its index job #93 so that a clash fails the job.
- 0007: index state lives in Postgres
- ADR 0003 (every entity is visible) and ADR 0004 (entity metadata is a virtual crate) constrain derivation.
- Where decisions conflicted, this spec follows the later one:
- The search index is never force-refreshed at the end of a job.
- The virtual crate isn't rewritten to the
ldac:/pcdm:spellings. - A crate with no resolvable licence and no configured default fails its job instead of being skipped.
- The admin pages show one state per crate, not one per index.
- A clash fails the job instead of being logged.
- Known limits:
- Search misses text past 1 MB in a file.
- One corrupt file keeps a crate failed, and errors surface one per attempt.
- A changed parent crate leaves its children's root collection and aggregates stale until they are reindexed.
- Labels lag a change to the open-licence list until a reindex.
- A crate placed in the repository by another tool doesn't appear on the admin pages until "index everything" runs.
- Outages aren't tested automatically.
- Open questions that don't block this spec:
- Should the text of restricted files be searchable? A question for Peter Sefton. Today's behaviour stands until it's answered.
- Revisit: access control goes off the licence id alone and ignores crate licence metadata
- For the implementation issues:
- Build the benchmark first.
- Check in a running Oni that
display: falsehides both the preview and the download. So far this has only been read from Oni's code. - Check production crates for member crates that carry a profiled stub of their parent collection, which would now fail as a clash.
- A follow-up issue deletes the index-template workaround once OpenSearchBackend: a
settingsoption for the index it creates crate-works/arocapi#143 ships. - Fold the small leftovers in Indexer: redesign storage, indexing and index state into the issue touching the same code.
Original request
The first version of this spec, as written from the map
The decided design for ldacapi's indexer, drawn up from the map Map: indexer redesign. It is meant to be clear enough to cut into implementation issues. Each section gives the gist and links the decision ticket where the detail lives. Where a later decision changed an earlier one, this spec follows the later one, and Decisions that conflict lists each case.
This is a breaking rewrite of src/indexer/, src/ocfl.ts and src/repository.ts. It assumes arocapi 5.0.0-beta.10 and @ocfl/ocfl-fs 1.0.0-beta.59, and waits on no upstream release.
Records
- ADRs, all merged on
main:- 0005 Indexing runs in a worker (docs: record that indexing runs in a worker draining a durable queue #80)
- 0006 A shared entity's row is one crate's copy (docs: record how shared entities are stored and deleted #83)
- 0007 Index state lives in Postgres (docs: record that index state lives in Postgres and the admin pages read only that #84)
- ADR 0004 was updated for crate spellings and files that aren't held (docs: the virtual crate serves property names as deposited and hides files not held #79).
- Glossary (merged): Index job, Worker, Reindex, Index state, Indexed version, Shared entity, Aggregates (docs: add Aggregates to the glossary #85).
- Background and today's problems: Indexer: redesign storage, indexing and index state. The bug this design fixes: Deleting a crate's database index removes entities other crates share.
1. What the indexer derives from a crate
From What the indexer derives from a crate, as amended by How shared entities are stored and deleted and The access control rule.
One pure derivation step (src/indexer/derive.ts) takes a crate's metadata, plus a few facts looked up beforehand (the parent crate chain for rootCollection, which files the repository holds and their sizes, configuration). It returns everything the three writers need: entity and file rows, crate links, search documents and aggregate rows. It does no I/O.
The derivation rules. Each one is tested by name at the derivation seam (see section 10):
- Entities indexed by profile. Collection, Object, Person, Organisation and SoftwareApplication get a row and a document when they declare the matching LDaC profile in
conformsTo. EveryFiledoes too, with no profile needed. Everything else appears only inside the virtual crate of what links to it. - Parent of a collection or object.
pcdm:memberOf, else reversepcdm:hasMember. The barememberOf/hasMemberspellings aren't read. - Parent of a file.
isPartOfor reversehasPart, preferring aRepositoryObject. Non-files don't use theisPartOf/hasPartfallbacks. - Several candidate parents. The first wins, and a warning is logged.
- Root collection. Unchanged for collections, objects and files (found by reading parent crates from the repository, never the indexes). Always null for Person, Organisation and SoftwareApplication.
- Entity ids. A relative
@idand a blank-node@idbecome<crate root id>/<@id>. An@idthat starts with the crate's root id, or is an absolute IRI, is kept as is, so an absolute IRI in two crates is one entity. - Content licence. The entity's own
license, else the nearest ancestor in the same crate (following rules 2 and 3), else the crate root, elseDEFAULT_LICENSE. - Metadata licence. The metadata descriptor's
license, elseDEFAULT_METADATA_LICENSE, for every entity in the crate. - Access control label.
Publicwhen the content licence id is in the open-licence list,AuthorizationByApplicationotherwise, including when there is no licence. Stored inEntity.meta.accessControl. - Spellings. LDaC and PCDM properties are read only in their prefixed form (
ldac:communicationMode,pcdm:memberOf). A crate on the older bare spelling is indexed without those properties, with no warning. The virtual crate serves property names exactly as deposited, and the deposited crate is never changed. - Files described but not held. The file gets its row and document, plus a
Filerow with no storage location inmeta.sizeis the statedcontentSize, or0.mediaTypefalls back toapplication/octet-stream. Its virtual crate getsdisplay: false. One warning per crate gives the count of such files. - File text. The text of every
text/*file goes into the search document whatever its licence, but only the first 1 MB. A file cut short is logged. - What the crate adds to aggregates. See section 3.
- Row and document contents. Row columns,
meta.rocrateand the search field list stay as today.meta.crateIdis dropped.indexTypebecomes a constant inderive.ts.
The crate is always read from the repository. Deposit writes to OCFL first and then queues a job, and a job carries only the crate id.
2. Shared entities and clashes
From How shared entities are stored and deleted (ADR 0006).
- A crate↔entity link table in our own models, indexed both ways, is the only record of which crates contain an entity. Counts, deletes and lookups by crate use it.
- A collection, object or file belongs to one crate. If a second crate has the same id, that is a data error. The crate that already holds the row keeps it, and the second crate's copy is refused. The same id described as different kinds in two crates is the same clash. See Open points for how a clash is reported.
- People, organisations and software are shared entities. Each link keeps that crate's own derived copy. The row and the document are wholly one crate's copy: the winner is a crate whose metadata licence is open, then the lowest crate id. A full rebuild gives the same rows whatever order crates are indexed in.
- Delete and reindex. Deleting a crate removes its links, then the entities (and documents) left with no link. A shared entity whose winning crate goes is rebuilt from the next crate's stored copy, with no repository read. Reindexing a crate that no longer contains an entity drops that link the same way.
- A crate that mentions a person without the LDaC profile gets no link.
3. Aggregates for Oni
From Oni aggregates: precompute or compute per request.
- A new table of our own (
prisma/models/aggregates.prisma), one row per (crate, entity) pair, for every entity that has something of this crate's at or below it. That covers the crate's own collections, objects and files, and their ancestors in parent crates. - Each row holds what this crate adds: the counts
collections,objects,subCollectionsandfilesbelow the entity, and thelanguage,communicationModeandmediaTypevalues at or below it. - A job writes and deletes only its own crate's rows. Deleting a crate deletes its rows without reading the repository.
- The API reads them with one flat query per page from the Oni entity transformer (
src/oni/), summing counts and combining lists. This replaces the recursivememberOfquery. - Lists use the same labels as the search facets ("English", "Spoken"). Counts use the indexer's own classification. Files held or not are counted, the entity itself isn't, and people, organisations and software have zero counts and empty lists.
- Child crates aren't reindexed automatically when a parent changes. Their rows (and
rootCollection) stay as they were until they are indexed again, and "reindex all" puts everything right.
4. Access control
From The access control rule.
- The rule is derivation rule 9. Crate licence metadata (
ldac:access,ldac:authorizationWorkflow,ldac:openAccessIndex) isn't read, and nothing warns about a missing or contradictory licence. OPEN_LICENSESbecomes a required environment setting, parsed inconfiguration.tswith no default in code. The four Creative Commons URLs move to.env.exampleand the deployment config.- One "is this licence open" function in
src/access/licences.tsserves both the indexer (for the stored label) and the check on each request (hasLicence). If the list changes, the request-time check picks it up straight away, but stored labels only catch up after a reindex.
5. Where indexing runs and what triggers it
From Where indexing runs and what triggers it (ADR 0005).
- Indexing never runs in the API. A worker process (
node src/worker.ts, same image) drains one durable queue, and the API only adds jobs.pnpm devstarts the worker beside the API. - The CLI uses the same queue.
pnpm indexandpnpm crates:importqueue jobs, then drain the queue until it is empty, and exit non-zero if any job they queued failed. - Triggers are explicit only: the admin pages, the CLI, Import and Deposit. There's no schedule and no watching of OCFL. Index in place is "index everything" run against an existing repository.
- The unit is one crate. "A crate and those under it" and "everything" expand into one job per crate. A job builds rows, then documents, then aggregates.
- Indexing is incremental: a crate whose indexed version equals its head version is skipped. Reindex deletes and rebuilds regardless.
- A crate has at most one running job and one waiting job. Asking again while one waits does nothing. Asking while one runs leaves one follow-up, which picks up a version deposited mid-run.
- Deletes are jobs in the same queue. "Delete everything" is one special job with no crate: it discards waiting jobs and clears all indexes. "Index everything" also queues deletes for crates no longer in the repository. Indexing a named crate that is missing deletes its indexes and succeeds.
- A Deposit succeeds once the crate is stored, and indexing follows as a job.
- A job that fails because of its crate waits to be asked again. Only a dead worker's job is retried automatically (section 6). In a database or OpenSearch outage the worker pauses and reconnects.
6. Index state and the admin pages
From Where index state lives and what the admin pages read (ADR 0007), with the additions from Progress and failure reporting for index runs.
Tables in prisma/models/index.prisma:
| Table | Holds |
|---|---|
| Index job | Crate id (none for "delete everything"), kind (index / reindex / delete), status, attempt count, claiming worker. While running: stage and a done-of-total counter. When failed: message, stage, entity or file id, time. One running and one waiting job per crate, enforced by partial unique indexes. Claimed with FOR UPDATE SKIP LOCKED. |
| Crate record | One per crate known to be in the repository: name, indexed version, when it was indexed, entity and file counts written by its job. |
| Worker | One per worker or draining CLI run, with its last heartbeat. |
| Crate link | Crate↔entity pairs, with that crate's derived copy for shared entities (section 2). |
- A job that succeeds is deleted, and the crate record holds its result. A failed job is kept until the next job for that crate replaces it. Who asked for what goes in the audit log.
- The indexed version is written last and cleared first. Index: clear the version, write rows, documents and aggregates, then set version, time and counts. Delete: clear the version, then remove documents, aggregates and rows. A failure part-way leaves no indexed version and a failed job. Recovery is asking again; there's no repair code.
- Deleting a crate's indexes keeps its record as not indexed. A record is removed only when a job finds its crate is gone from the repository.
- Workers. A worker writes a heartbeat every 30 seconds, and that write also saves its running job's progress. A worker silent for 5 minutes is treated as dead: its job goes back to waiting with the attempt count raised, and is marked failed on the third return. A clean shutdown releases the job at once. A job checks it still owns its claim before writing its result.
- Admin pages read only Postgres. Each crate shows one index state: not indexed, queued, indexing (with stage, counter and start time), deleting, indexed (version, time, counts) or failed (error, stage, entity or file id, attempt). A running or waiting job takes precedence, then a failed job, then the record. The top of the page counts crates per state, and "No worker running" shows whenever no heartbeat is fresh. There's no "out of date" state and no grand total of entities.
- A crate appears in the list once something has queued a job for it.
7. Failures, progress and logs
From Progress and failure reporting for index runs.
- Any error inside a crate fails the job at the first error, and nothing is skipped, so an indexed version always means the whole crate is indexed. Examples: an entity that can't be written, a held file that can't be read, a document that can't be indexed. Files described but not held are a normal case and don't fail the job.
- The stack trace goes to the log, tied to the job id.
- The worker and the CLI write the same lines:
- Info: one line when a job starts (job id, crate id, kind, attempt) and one when it finishes (duration, version, counts).
- Error: one line when a job fails, with fixed fields (crate id, job id, stage, stack trace).
- Debug: per-entity and per-file lines, and crates skipped as already indexed.
- When the queue is empty, the CLI prints a summary of the crates it queued (indexed, skipped, deleted, failed, then each failure with its error), read from Postgres.
- There are no notifications and no health endpoint.
8. Performance
From Indexing performance at scale.
- Target: the largest collection (~30,000 entities) is fully indexed in under 10 minutes on the production VM, with the worker under 1 GB. This has to hold both as one crate and split across thousands of crates.
- Aggregates read target: a page of 100 entities with aggregates takes no more than 50 ms longer than without. If it doesn't, add an index on the aggregates table.
- Both targets are checked by a benchmark built early in implementation. It runs over a generated 30,000-entity collection in both shapes, with text files of 50 KB, 1 MB and 5 MB. The figures below are starting points for it to tune.
- One crate's metadata is held whole in memory. File text is read one file at a time.
- Database writes go in batches of about 1,000 rows, each batch its own transaction. Aggregate rows are the last batch. There isn't one transaction per crate, so readers can see a partly indexed crate; the cleared indexed version marks it.
- Search writes go in bulk requests capped at about 5 MB or 1,000 documents. Refresh is never forced, and tests refresh for themselves.
- Each worker runs one job at a time, with no worker threads. More throughput means more workers.
- A job opens its OCFL object once. The worker caches crate → parent collection in memory (ids only), rechecked with a cheap
stat.
9. The arocapi boundary
From The arocapi boundary.
- arocapi stays read-only. ldacapi owns every write and proposes no crate-to-rows code upstream.
- The only upstream ask is the index
settingsoption (OpenSearchBackend: asettingsoption for the index it creates crate-works/arocapi#143, already filed). Until it ships, the index-template workaround stays, kept in one place. - The id rule and the file storage meta builder are plain functions with no LDaC imports, so they could move upstream one day.
- Upstream changes that would help but aren't needed: arocapi#142 (batch transformers) would replace the microtask batching in the Oni transformer.
10. Packaging, layout and tests
From Packaging the indexer, with Deposit in mind, Layout of src/ after the redesign and Test strategy for the indexer.
src/
index.ts API entry point: opens repository, storage, index queue; buildApp({ search, storage, indexQueue })
worker.ts worker entry point
app.ts configuration.ts logger.ts prisma.ts
admin/ oidc/
access/ auth.ts, licences.ts, access-control.ts (label type)
oni/ Oni entity transformer, aggregates read query, COLLECTION/OBJECT IRIs
repository/ open/create the OCFL repository, content storage backend, file storage meta builder, id rule
index-queue/
index-queue.ts the whole interface: add index/reindex/delete jobs, read index state
indexer/
indexer.ts the whole interface: "work the queue" (until empty, or forever)
jobs.ts crate.ts derive.ts rows.ts documents.ts aggregates.ts
scripts/ CLI entry points: ocfl-init, crates-import, index, generate-keys
test/ crate builder, helpers, crate-to-HTTP tests
prisma/models/ index.prisma, aggregates.prisma
- ldacapi stays one package. The API never imports the indexer: a Biome
noRestrictedImportsrule allows imports ofsrc/indexer/**only fromsrc/indexer/**,src/worker.ts,scripts/**andtest/**. Deposit's only dependency on indexing is one "add a job" call on the index queue. - These are fixed: the two interface files, and the rule that
derive.tsdoes no I/O and imports neitherconfiguration.tsnorprisma.ts. Other file names insideindexer/are a starting point. src/repository.ts,src/ocfl.ts, theRepositoryinterface andsrc/utils.tsare deleted. There's no module-level mutable state inrepository/,index-queue/orindexer/. Entry points open what they need and pass it in.- Tests:
- Rules about one crate are tested at
derive.ts. - Anything needing stored state or more than one crate goes through "work the queue", against real Postgres, OpenSearch and a temporary OCFL repository. That covers incremental skip, reindex, delete, shared entities, clashes, failed and dead-worker jobs, damaged crates and two workers at once.
- Crate-to-HTTP tests are a thin layer.
- One test reaches inside, at
jobs.ts: a worker wrongly judged dead. - Hard cases are staged with data, not fake clocks.
- A crate builder in
test/replaces most oftest-data/. The old indexer tests are thrown away. src/indexer/andsrc/index-queue/are held at 100% coverage from their first commit. The rest ofsrc/follows the auto-rising threshold that is already onmain.
- Rules about one crate are tested at
11. Going live
From Cutting production over to the new indexer.
-
No data is cut over. The release goes onto a brand new VM, and the code may assume empty index tables (its migration can empty the old Entity and File rows).
-
Order:
pnpm db:deploy- Put the crates in the repository.
- Start the API and worker.
- Index every crate.
- Switch traffic once the admin pages show every crate indexed and none failed.
This becomes the README's install section.
-
The release notes say "rebuild both indexes after upgrading".
Known limits
- No no-gap rebuild after go-live. "Reindex all" runs through the queue with the API up, and a search mapping change needs a planned maintenance window. Ruled out of scope on the map.
- Outages aren't tested automatically. "The worker pauses and reconnects" is checked by hand once.
- Search misses text past 1 MB in a file.
- One corrupt file keeps a crate failed until it is fixed, and errors surface one per attempt.
- A changed parent crate leaves children stale (
rootCollection, aggregates) until they are reindexed. - Labels lag a change to
OPEN_LICENSESuntil a reindex. - Crates on the older bare spelling lose their parent links and LDaC facets.
- Unlisted crates. A crate another tool put in the repository stays off the admin pages until "index everything" runs.
Decisions that conflict
In each case this spec follows the later decision:
- Refresh at the end of a job. "The arocapi boundary" said a job calls
indices.refreshonce when it finishes. "Indexing performance at scale" (later) says refresh is never forced. entityAsCraterewriting toldac:/pcdm:. "The arocapi boundary" described the rewrite. The amended "What the indexer derives from a crate" serves the virtual crate as deposited.- A crate with no resolvable licence. "What the indexer derives from a crate" kept today's "skipped with a warning". "Progress and failure reporting for index runs" (later) says nothing is skipped and any error fails the job. So a crate whose content or metadata licence can't be resolved, and has no configured default, fails its job.
- Showing the two indexes separately. "Where indexing runs and what triggers it" said the admin pages could do this. "Where index state lives and what the admin pages read" shows one state per crate.
Open points
- How a clash between crates is reported. "How shared entities are stored and deleted" and "Test strategy for the indexer" say the second crate's copy is refused, the rest of that crate is indexed, and the clash is reported as an error naming both crates on every run. "Progress and failure reporting for index runs" says any error fails the job. As written, a clash is the one data error that doesn't fail its job, and the only place it's reported is the error log line. It doesn't show on the admin pages. Decide before cutting the shared-entities issue: leave it log-only, record it on the crate record, or fail the job.
- Should the text of restricted files be searchable?: a question for Peter Sefton. Today's behaviour (all text indexed) stands until it's answered.
- Revisit: access control goes off the licence id alone and ignores crate licence metadata: a deliberate later revisit, not a blocker.
For the implementation issues
Notes from the decision tickets that belong in specific issues:
- Build the benchmark early, before the writers are tuned (section 8).
- Move
test-data/crates to theldac:spellings (they use barecommunicationModeandindexableText). Replacesub_col_2.txtwith a named inline case, "a file described but not held". - Check in a running Oni that
display: falsehides both the preview and the download. So far this was only read from Oni's code (src/composables/fileVisibility.ts), not run. - Check production crates for member crates that carry a profiled stub of their parent collection. Under section 2, such a stub would be refused as a clash.
- Move the logic in
scripts/crates-import.tsintosrc/. Entry points only parse arguments and callsrc/, so coverage can leave them out. Write that rule inCLAUDE.mdand beside the exclusion invitest.config.ts. - Each implementation issue names the derivation rules and hard-case tests it owes (sections 1 and 10).
- Follow-up issue: delete the index-template workaround once OpenSearchBackend: a
settingsoption for the index it creates crate-works/arocapi#143 ships. - Fold the small leftovers from Indexer: redesign storage, indexing and index state #47 into the issue that touches the same code:
_indexEntity, thecrate.rootcheck,objects()arguments, LDaCA constants inconfiguration.ts, log noise, and the "entities" count.
Activity