Skip to content

Map: indexer redesign #65

Closed
17 of 17 issues completed
Closed

Description

@johnf

Destination

A decided design for ldacapi's indexer, clear enough to cut into implementation issues: what the indexer derives from a crate, where indexing runs and what triggers it, where index state lives, how entities shared between crates are stored and deleted, what moves into arocapi, packaging, and how src/ is laid out afterwards. The map ends with a spec issue summing up the design and linking each decision, plus ADRs for the hard-to-reverse calls. Planning only.

Notes

  • Breaking change. Everything in the indexer is open to rewriting or re-architecting. Question whether we index the right way and use crate metadata well, especially in light of arocapi 5 beta and @ocfl/ocfl-fs 1.0 beta.
  • OpenSearch is the search backend. Settled for this project; not a question on this map.
  • Three ways crates arrive (see GLOSSARY.md), all ending in "index one crate in the repository". Don't make decisions that block any of them; don't build for them either:
    1. Deposit: crate sent through the deposit API, written to OCFL and indexed.
    2. Import: RO-Crates from a directory written to OCFL, then indexed.
    3. Index in place: an existing, valid OCFL repository indexed without importing files or creating versions.
  • Deposit's own design is out of scope; its sharp edges for the indexer (sync vs async, failure semantics, indexing from the crate in hand, roCrateIds) are covered by the tickets.
  • Skills: grilling + domain-modeling for grilling tickets; research for research tickets; codebase-design for packaging and layout.
  • Domain: GLOSSARY.md, docs/adr/. ADR 0003 (every entity is visible) and 0004 (entity metadata is a virtual crate) constrain the design.
  • arocapi owns the Entity and File models (prisma/arocapi); our own models live in prisma/models. Check with the maintainer before filing upstream issues or PRs (crate-works/arocapi, ocfl-js).
  • Production: the largest collection has ~30,000 entities. Licence metadata is inconsistent; only Sydney Speaks uses ldac:authorizationWorkflow correctly.
  • Background, current state, known problems and what's already settled: Indexer: redesign storage, indexing and index state #47.

Decisions so far

  • What @ocfl/ocfl-fs 1.0 beta offers for change detection and reading: no storable change token or watch API (detect change via our own record of head version, or the inventory sidecar); every read opens the full inventory; objects() is an unordered walk; readers never wait on writers; unreleased breaking changes on ocfl-js main.
  • What arocapi 5 beta offers an indexer: arocapi only reads; no indexer, ingest or deposit code upstream, so ldacapi writes every row and document. Storage backends read keys we put in meta. roCrateIds is spec-only. No per-call refresh or index settings (arocapi#143 open). arocapi#142 (counts are the consumer's job) and arocapi#89 (no joins to our tables) bear on aggregates and shared entities.
  • Prior art: how other RO-Crate/OCFL indexers work: only LDaC Collection/Object/File entities get documents; rootCollection is always derived; no prior art solves shared entities (one copy per crate); aggregates either ancestors-on-document with query-time facets (oni-ui) or precomputed tables (Nabu); licence resolved at index time with parent fallback; only Nabu persists indexing state (DB-backed job queue).
  • What the indexer derives from a crate: profiled Collection/Object/Person/Organisation/SoftwareApplication plus every File get a row and document; parent chosen by profile (pcdm:memberOf/pcdm:hasMember, files by isPartOf/hasPart); absolute IRIs shared across crates, relative and blank-node ids prefixed; content licence falls back through ancestors in the crate; one derivation step feeds both indexes; crates assumed to use the prefixed ldac:/pcdm: spelling, the older bare spelling neither read nor rewritten, deposited crate untouched; files not held get a File row with no storage location and display: false; always read from the repository.
  • Where indexing runs and what triggers it: never in the API; a separate worker drains one durable queue that the API and CLI add to (the CLI also drains it). Explicit triggers only (admin pages, CLI, Import, Deposit); one job per crate building both indexes, incremental by head version, reindex to force. Deposit succeeds once stored and is indexed asynchronously. At most one running and one waiting job per crate; deletes are jobs too; only a dead worker's job is retried automatically. ADR 0005 (docs: record that indexing runs in a worker draining a durable queue #80).
  • The access control rule: from the content licence id alone (Public if in the open-licence list, else AuthorizationByApplication, also when missing); crate licence metadata not read and no warnings (revisit: Revisit: access control goes off the licence id alone and ignores crate licence metadata #82); the list becomes deployment configuration; one "is this licence open" function for indexer and request-time check; label stays two values in Entity.meta.accessControl; all file text indexed whatever the licence (question for Peter Sefton: Should the text of restricted files be searchable? #81).
  • How shared entities are stored and deleted: a crate↔entity link table in our own models is the only record of which crates contain an entity (meta.crateId dropped). A collection, object or file belongs to one crate; a second crate claiming it is a data error that fails that crate's index job (amended while writing the spec). People, organisations and software are shared entities: one row, wholly one crate's copy (open metadata licence first, then lowest crate id), each link keeping its crate's copy so a delete needs no repository read; their rootCollection is always null. Amends two lines of "What the indexer derives from a crate". ADR 0006 (docs: record how shared entities are stored and deleted #83).
  • Where index state lives and what the admin pages read: our own Postgres tables: an index job queue (claimed with SKIP LOCKED; successful jobs deleted, one failed job kept per crate), one record per known crate (name, indexed version written last and cleared first, counts), and worker heartbeats (dead after 5 minutes, job retried up to three times). A half-finished job shows as failed and is fixed by asking again. The admin pages read only Postgres, show one index state per crate and crates per state, and warn when no worker is running. ADR 0007 (docs: record that index state lives in Postgres and the admin pages read only that #84).
  • The arocapi boundary: arocapi stays read-only and ldacapi owns every write; no crate-to-rows code goes upstream. The only upstream ask is the index settings option (OpenSearchBackend: a settings option for the index it creates crate-works/arocapi#143, already filed); no per-call refresh or bulk chunking is proposed. The redesign assumes arocapi 5.0.0-beta.10 and waits on no release; the index-template workaround stays until #143 ships. The id rule and file storage meta builder stay plain functions with no LDaC imports. No ADR.
  • Oni aggregates: precompute or compute per request: precomputed. A new table of our own where each crate records what it adds (counts, languages, communication modes, media types) for each of its entities and their ancestors; a request adds the rows up in one flat query. A job writes only its own crate's rows. Lists are own values plus everything below, using the search index's labels; counts use the indexer's classification. No automatic re-index of child crates when a parent changes. Glossary gains Aggregates (docs: add Aggregates to the glossary #85).
  • Packaging the indexer, with Deposit in mind: one private package, no workspace; the indexer is a folder in src/. Two modules: the index queue (ask for jobs, read state; all the API and Deposit may import) and the indexer (one operation, "work the queue", used only by the worker and CLI). Derivation is a pure function so Deposit could later run it on a crate in hand. A Biome lint rule bans importing the indexer from anywhere else; code both sides need lives outside it. No ADR.
  • Layout of src/ after the redesign: folders admin/, oidc/, access/, oni/, repository/, index-queue/, indexer/, with src/index.ts (API) and src/worker.ts as entry points and no api/ or shared/ folder. repository.ts, ocfl.ts and the Repository interface are deleted; entry points open what they need and pass it in, with no module-level mutable state. Each new folder's interface is one file named after it; derive.ts does no I/O. openLicenses becomes a required environment setting and indexType a constant in the indexer. Crate-to-HTTP tests move to test/, which the lint rule lets import the indexer. No ADR.
  • Cutting production over to the new indexer: no cutover of existing data. The new indexer goes onto a brand new VM (ldacapi isn't in production yet): migrate, put the crates in the repository, start the API and worker, index every crate, and switch traffic once the admin pages show every crate indexed and none failed. The code carries no upgrade path and may assume empty index tables; the way back is not switching. No ADR.
  • Progress and failure reporting for index runs: any error inside a crate fails the job at the first error, nothing is skipped. A failed job keeps its message, stage, entity or file id, time and attempt count; the stack trace goes to the log. A running job shows its stage and a done-of-total counter, saved with the heartbeat. Worker and CLI log one line per job start, finish and failure; the CLI ends with a summary of the crates it queued. No notifications and no health endpoint. No ADR.
  • Indexing performance at scale: covers both one 30,000-entity crate and thousands of small ones. Target: largest collection indexed in under 10 minutes with the worker under 1 GB, checked by a benchmark built early in implementation (which also checks the aggregates read). Metadata held whole in memory; file text one file at a time, first 1 MB only. Rows in batches of about 1,000, each its own transaction, not one transaction per crate. Search documents in bulk requests capped at about 5 MB, no forced refresh. One job at a time per worker. A job opens its OCFL object once; the worker caches crate → parent collection in memory. No ADR.
  • Test strategy for the indexer: rules about one crate are tested at the pure derivation step; anything needing stored state or more than one crate goes through "work the queue" against real Postgres, OpenSearch and a temporary OCFL repository; crate-to-HTTP tests stay a thin layer. Hard cases are staged with data, not fake clocks (a dead worker is an old heartbeat row); an outage has no automated test. A crate builder in test/ replaces most of test-data/. The old indexer tests are thrown away. A coverage threshold over src/ starts at today's 89% of lines and rises automatically, with the new indexer folders at 100% and entry points left out as argument wrappers. No ADR.
  • Write the indexer redesign spec issue: the spec is Spec: indexer redesign. Every ADR and glossary PR is merged. It records four conflicts between decisions, each resolved in favour of the later one. The open point it raised is now decided: a crate that clashes with another fails its index job (ADR 0006 updated in docs: a crate that clashes with another fails its index job #93).

Not yet specified

Nothing: every remaining question is a ticket.

Out of scope

  • The known-problem leftovers in Indexer: redesign storage, indexing and index state #47 (_indexEntity, the crate.root check, objects() arguments, LDaCA constants in configuration.ts, utils.ts, the sub_col_2.txt test data, log noise, the "entities" count): small fixes, folded into implementation issues.
  • Cutting implementation issues: execution, after the spec.
  • The deposit API's own design.
  • Making the upstream arocapi changes themselves.
  • A no-gap rebuild after go-live (second search index and a switch): until then "reindex all" runs through the queue with the API up, and a search mapping change takes a planned maintenance window. Ruled out in Cutting production over to the new indexer; the spec notes it as a known limit.

Activity

  1. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Destination reached:

    Implementation is tracked as sub-issues of the spec. Two open questions continue outside the map, and neither blocks it: "Should the text of restricted files be searchable?" (#81) and "Revisit: access control goes off the licence id alone and ignores crate licence metadata" (#82).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions