From c981c143b12334a26ee38426a9e5b6a0833c1c56 Mon Sep 17 00:00:00 2001 From: John Ferlito Date: Tue, 17 Nov 2026 12:38:20 +1100 Subject: [PATCH] docs: record that indexing runs in a worker draining a durable queue Adds ADR 0005 and the glossary terms Reindex, Index job and Worker. Decided on the indexer redesign map (#65), ticket #70. --- GLOSSARY.md | 14 +++++++++++- docs/adr/0005-indexing-runs-in-a-worker.md | 25 ++++++++++++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) create mode 100644 docs/adr/0005-indexing-runs-in-a-worker.md diff --git a/GLOSSARY.md b/GLOSSARY.md index 820384a..d7672f5 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -21,9 +21,21 @@ A crate sent through the deposit API, which writes it to the OCFL repository and _Avoid_: Import, upload **Index**: -Building the database index and the search index from the repository, for one crate or all of them. Both can be deleted and rebuilt at any time. +Building the database index and the search index from the repository, for one crate or all of them. A crate whose current version is already indexed is left alone. Both indexes can be deleted and rebuilt at any time. _Avoid_: Import +**Reindex**: +Deleting a crate's indexes and building them again, whether or not its current version is already indexed. +_Avoid_: Force index, refresh + +**Index job**: +A request, waiting or running, to index or reindex one crate or to delete its indexes. Indexing several crates is one index job for each. +_Avoid_: Task, index run + +**Worker**: +The process that runs index jobs, separate from the API (ADR 0005). +_Avoid_: Indexer (the code that builds the indexes) + **Database index**: The Entity and File rows that arocapi reads, written by the indexer (arocapi's Rows). _Avoid_: Structural index diff --git a/docs/adr/0005-indexing-runs-in-a-worker.md b/docs/adr/0005-indexing-runs-in-a-worker.md new file mode 100644 index 0000000..63ec38f --- /dev/null +++ b/docs/adr/0005-indexing-runs-in-a-worker.md @@ -0,0 +1,25 @@ +# Indexing runs in a worker that drains a durable queue + +## Context + +Crates reach the repository three ways: Deposit, Import, and an existing OCFL repository indexed in place. All three end in the same work, indexing one crate. That work is slow and CPU-bound: the largest collection has about 30,000 entities, and parsing a crate blocks Node's one thread. The repository is the source of truth, and both indexes can be rebuilt from it at any time. `@ocfl/ocfl-fs` has no way to watch a repository or ask what changed. + +Running the same queue from a loop inside the API process was the alternative. It needs one process instead of two, but indexing a large crate would stall requests, could run the API out of memory, and would be interrupted by every API restart, and indexing capacity would follow the number of API replicas. + +## Decision + +Indexing never runs in the API process. Every request to index or delete a crate's indexes is an index job on one durable queue, and a separate worker process, the same image with its own command, runs the jobs. + +- The API only adds jobs: from the admin pages, and from Deposit once the crate is written to the repository. +- The CLI adds jobs and then runs the queue itself until it is empty, so it works with or without a worker running. +- Indexing starts only when something asks for it. Nothing runs on a schedule or watches the repository. +- A job is for one crate, names it by id only, and builds both indexes from the head version in the repository. Wider requests become one job per crate. +- A crate has at most one running job and one waiting job. + +## Consequences + +- A deployment runs two processes, and the worker needs the repository, the database and the search backend. If no worker is running, jobs wait, so the admin pages have to show that. +- A deposit succeeds once the crate is stored. Its entities appear in the API after its job has run, and an indexing failure is a failed job to ask for again, not a failed deposit. +- Any number of API replicas, workers and CLI runs can share the queue, because each job is claimed once. +- A crate changed or removed in the repository by another tool is stale in the indexes until someone asks to index it or to index everything. +- A job that fails because of its crate waits to be asked for again. A job whose worker died is run again.