Repository navigation
Closed
17 / 1717 of 17 issues completedLabels
wayfinder:mapWayfinder mapWayfinder map
Description
Destination
A decided design for ldacapi's indexer, clear enough to cut into implementation issues: what the indexer derives from a crate, where indexing runs and what triggers it, where index state lives, how entities shared between crates are stored and deleted, what moves into arocapi, packaging, and how src/ is laid out afterwards. The map ends with a spec issue summing up the design and linking each decision, plus ADRs for the hard-to-reverse calls. Planning only.
Notes
- Breaking change. Everything in the indexer is open to rewriting or re-architecting. Question whether we index the right way and use crate metadata well, especially in light of arocapi 5 beta and
@ocfl/ocfl-fs1.0 beta. - OpenSearch is the search backend. Settled for this project; not a question on this map.
- Three ways crates arrive (see
GLOSSARY.md), all ending in "index one crate in the repository". Don't make decisions that block any of them; don't build for them either:- Deposit: crate sent through the deposit API, written to OCFL and indexed.
- Import: RO-Crates from a directory written to OCFL, then indexed.
- Index in place: an existing, valid OCFL repository indexed without importing files or creating versions.
- Deposit's own design is out of scope; its sharp edges for the indexer (sync vs async, failure semantics, indexing from the crate in hand,
roCrateIds) are covered by the tickets. - Skills:
grilling+domain-modelingfor grilling tickets;researchfor research tickets;codebase-designfor packaging and layout. - Domain:
GLOSSARY.md,docs/adr/. ADR 0003 (every entity is visible) and 0004 (entity metadata is a virtual crate) constrain the design. - arocapi owns the
EntityandFilemodels (prisma/arocapi); our own models live inprisma/models. Check with the maintainer before filing upstream issues or PRs (crate-works/arocapi, ocfl-js). - Production: the largest collection has ~30,000 entities. Licence metadata is inconsistent; only Sydney Speaks uses
ldac:authorizationWorkflowcorrectly. - Background, current state, known problems and what's already settled: Indexer: redesign storage, indexing and index state #47.
Decisions so far
- What @ocfl/ocfl-fs 1.0 beta offers for change detection and reading: no storable change token or watch API (detect change via our own record of head version, or the inventory sidecar); every read opens the full inventory;
objects()is an unordered walk; readers never wait on writers; unreleased breaking changes on ocfl-js main. - What arocapi 5 beta offers an indexer: arocapi only reads; no indexer, ingest or deposit code upstream, so ldacapi writes every row and document. Storage backends read keys we put in
meta.roCrateIdsis spec-only. No per-call refresh or index settings (arocapi#143 open). arocapi#142 (counts are the consumer's job) and arocapi#89 (no joins to our tables) bear on aggregates and shared entities. - Prior art: how other RO-Crate/OCFL indexers work: only LDaC Collection/Object/File entities get documents;
rootCollectionis always derived; no prior art solves shared entities (one copy per crate); aggregates either ancestors-on-document with query-time facets (oni-ui) or precomputed tables (Nabu); licence resolved at index time with parent fallback; only Nabu persists indexing state (DB-backed job queue). - What the indexer derives from a crate: profiled Collection/Object/Person/Organisation/SoftwareApplication plus every File get a row and document; parent chosen by profile (
pcdm:memberOf/pcdm:hasMember, files byisPartOf/hasPart); absolute IRIs shared across crates, relative and blank-node ids prefixed; content licence falls back through ancestors in the crate; one derivation step feeds both indexes; crates assumed to use the prefixedldac:/pcdm:spelling, the older bare spelling neither read nor rewritten, deposited crate untouched; files not held get a File row with no storage location anddisplay: false; always read from the repository. - Where indexing runs and what triggers it: never in the API; a separate worker drains one durable queue that the API and CLI add to (the CLI also drains it). Explicit triggers only (admin pages, CLI, Import, Deposit); one job per crate building both indexes, incremental by head version, reindex to force. Deposit succeeds once stored and is indexed asynchronously. At most one running and one waiting job per crate; deletes are jobs too; only a dead worker's job is retried automatically. ADR 0005 (docs: record that indexing runs in a worker draining a durable queue #80).
- The access control rule: from the content licence id alone (
Publicif in the open-licence list, elseAuthorizationByApplication, also when missing); crate licence metadata not read and no warnings (revisit: Revisit: access control goes off the licence id alone and ignores crate licence metadata #82); the list becomes deployment configuration; one "is this licence open" function for indexer and request-time check; label stays two values inEntity.meta.accessControl; all file text indexed whatever the licence (question for Peter Sefton: Should the text of restricted files be searchable? #81). - How shared entities are stored and deleted: a crate↔entity link table in our own models is the only record of which crates contain an entity (
meta.crateIddropped). A collection, object or file belongs to one crate; a second crate claiming it is a data error that fails that crate's index job (amended while writing the spec). People, organisations and software are shared entities: one row, wholly one crate's copy (open metadata licence first, then lowest crate id), each link keeping its crate's copy so a delete needs no repository read; theirrootCollectionis always null. Amends two lines of "What the indexer derives from a crate". ADR 0006 (docs: record how shared entities are stored and deleted #83). - Where index state lives and what the admin pages read: our own Postgres tables: an index job queue (claimed with
SKIP LOCKED; successful jobs deleted, one failed job kept per crate), one record per known crate (name, indexed version written last and cleared first, counts), and worker heartbeats (dead after 5 minutes, job retried up to three times). A half-finished job shows as failed and is fixed by asking again. The admin pages read only Postgres, show one index state per crate and crates per state, and warn when no worker is running. ADR 0007 (docs: record that index state lives in Postgres and the admin pages read only that #84). - The arocapi boundary: arocapi stays read-only and ldacapi owns every write; no crate-to-rows code goes upstream. The only upstream ask is the index
settingsoption (OpenSearchBackend: asettingsoption for the index it creates crate-works/arocapi#143, already filed); no per-call refresh or bulk chunking is proposed. The redesign assumes arocapi 5.0.0-beta.10 and waits on no release; the index-template workaround stays until #143 ships. The id rule and file storage meta builder stay plain functions with no LDaC imports. No ADR. - Oni aggregates: precompute or compute per request: precomputed. A new table of our own where each crate records what it adds (counts, languages, communication modes, media types) for each of its entities and their ancestors; a request adds the rows up in one flat query. A job writes only its own crate's rows. Lists are own values plus everything below, using the search index's labels; counts use the indexer's classification. No automatic re-index of child crates when a parent changes. Glossary gains Aggregates (docs: add Aggregates to the glossary #85).
- Packaging the indexer, with Deposit in mind: one private package, no workspace; the indexer is a folder in
src/. Two modules: the index queue (ask for jobs, read state; all the API and Deposit may import) and the indexer (one operation, "work the queue", used only by the worker and CLI). Derivation is a pure function so Deposit could later run it on a crate in hand. A Biome lint rule bans importing the indexer from anywhere else; code both sides need lives outside it. No ADR. - Layout of src/ after the redesign: folders
admin/,oidc/,access/,oni/,repository/,index-queue/,indexer/, withsrc/index.ts(API) andsrc/worker.tsas entry points and noapi/orshared/folder.repository.ts,ocfl.tsand theRepositoryinterface are deleted; entry points open what they need and pass it in, with no module-level mutable state. Each new folder's interface is one file named after it;derive.tsdoes no I/O.openLicensesbecomes a required environment setting andindexTypea constant in the indexer. Crate-to-HTTP tests move totest/, which the lint rule lets import the indexer. No ADR. - Cutting production over to the new indexer: no cutover of existing data. The new indexer goes onto a brand new VM (ldacapi isn't in production yet): migrate, put the crates in the repository, start the API and worker, index every crate, and switch traffic once the admin pages show every crate indexed and none failed. The code carries no upgrade path and may assume empty index tables; the way back is not switching. No ADR.
- Progress and failure reporting for index runs: any error inside a crate fails the job at the first error, nothing is skipped. A failed job keeps its message, stage, entity or file id, time and attempt count; the stack trace goes to the log. A running job shows its stage and a done-of-total counter, saved with the heartbeat. Worker and CLI log one line per job start, finish and failure; the CLI ends with a summary of the crates it queued. No notifications and no health endpoint. No ADR.
- Indexing performance at scale: covers both one 30,000-entity crate and thousands of small ones. Target: largest collection indexed in under 10 minutes with the worker under 1 GB, checked by a benchmark built early in implementation (which also checks the aggregates read). Metadata held whole in memory; file text one file at a time, first 1 MB only. Rows in batches of about 1,000, each its own transaction, not one transaction per crate. Search documents in bulk requests capped at about 5 MB, no forced refresh. One job at a time per worker. A job opens its OCFL object once; the worker caches crate → parent collection in memory. No ADR.
- Test strategy for the indexer: rules about one crate are tested at the pure derivation step; anything needing stored state or more than one crate goes through "work the queue" against real Postgres, OpenSearch and a temporary OCFL repository; crate-to-HTTP tests stay a thin layer. Hard cases are staged with data, not fake clocks (a dead worker is an old heartbeat row); an outage has no automated test. A crate builder in
test/replaces most oftest-data/. The old indexer tests are thrown away. A coverage threshold oversrc/starts at today's 89% of lines and rises automatically, with the new indexer folders at 100% and entry points left out as argument wrappers. No ADR. - Write the indexer redesign spec issue: the spec is Spec: indexer redesign. Every ADR and glossary PR is merged. It records four conflicts between decisions, each resolved in favour of the later one. The open point it raised is now decided: a crate that clashes with another fails its index job (ADR 0006 updated in docs: a crate that clashes with another fails its index job #93).
Not yet specified
Nothing: every remaining question is a ticket.
Out of scope
- The known-problem leftovers in Indexer: redesign storage, indexing and index state #47 (
_indexEntity, thecrate.rootcheck,objects()arguments, LDaCA constants inconfiguration.ts,utils.ts, thesub_col_2.txttest data, log noise, the "entities" count): small fixes, folded into implementation issues. - Cutting implementation issues: execution, after the spec.
- The deposit API's own design.
- Making the upstream arocapi changes themselves.
- A no-gap rebuild after go-live (second search index and a switch): until then "reindex all" runs through the queue with the API up, and a search mapping change takes a planned maintenance window. Ruled out in Cutting production over to the new indexer; the spec notes it as a known limit.
Metadata
Metadata
Assignees
Labels
wayfinder:mapWayfinder mapWayfinder map
Activity
docs: record that indexing runs in a worker draining a durable queue
johnf commentedon Nov 17, 2026
Destination reached:
Implementation is tracked as sub-issues of the spec. Two open questions continue outside the map, and neither blocks it: "Should the text of restricted files be searchable?" (#81) and "Revisit: access control goes off the licence id alone and ignores crate licence metadata" (#82).