Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,21 @@ A crate sent through the deposit API, which writes it to the OCFL repository and
_Avoid_: Import, upload

**Index**:
Building the database index and the search index from the repository, for one crate or all of them. Both can be deleted and rebuilt at any time.
Building the database index and the search index from the repository, for one crate or all of them. A crate whose current version is already indexed is left alone. Both indexes can be deleted and rebuilt at any time.
_Avoid_: Import

**Reindex**:
Deleting a crate's indexes and building them again, whether or not its current version is already indexed.
_Avoid_: Force index, refresh

**Index job**:
A request, waiting or running, to index or reindex one crate or to delete its indexes. Indexing several crates is one index job for each.
_Avoid_: Task, index run

**Worker**:
The process that runs index jobs, separate from the API (ADR 0005).
_Avoid_: Indexer (the code that builds the indexes)

**Database index**:
The Entity and File rows that arocapi reads, written by the indexer (arocapi's Rows).
_Avoid_: Structural index
Expand Down
25 changes: 25 additions & 0 deletions docs/adr/0005-indexing-runs-in-a-worker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Indexing runs in a worker that drains a durable queue

## Context

Crates reach the repository three ways: Deposit, Import, and an existing OCFL repository indexed in place. All three end in the same work, indexing one crate. That work is slow and CPU-bound: the largest collection has about 30,000 entities, and parsing a crate blocks Node's one thread. The repository is the source of truth, and both indexes can be rebuilt from it at any time. `@ocfl/ocfl-fs` has no way to watch a repository or ask what changed.

Running the same queue from a loop inside the API process was the alternative. It needs one process instead of two, but indexing a large crate would stall requests, could run the API out of memory, and would be interrupted by every API restart, and indexing capacity would follow the number of API replicas.

## Decision

Indexing never runs in the API process. Every request to index or delete a crate's indexes is an index job on one durable queue, and a separate worker process, the same image with its own command, runs the jobs.

- The API only adds jobs: from the admin pages, and from Deposit once the crate is written to the repository.
- The CLI adds jobs and then runs the queue itself until it is empty, so it works with or without a worker running.
- Indexing starts only when something asks for it. Nothing runs on a schedule or watches the repository.
- A job is for one crate, names it by id only, and builds both indexes from the head version in the repository. Wider requests become one job per crate.
- A crate has at most one running job and one waiting job.

## Consequences

- A deployment runs two processes, and the worker needs the repository, the database and the search backend. If no worker is running, jobs wait, so the admin pages have to show that.
- A deposit succeeds once the crate is stored. Its entities appear in the API after its job has run, and an indexing failure is a failed job to ask for again, not a failed deposit.
- Any number of API replicas, workers and CLI runs can share the queue, because each job is claimed once.
- A crate changed or removed in the repository by another tool is stale in the indexes until someone asks to index it or to index everything.
- A job that fails because of its crate waits to be asked for again. A job whose worker died is run again.
Loading