You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Decided by grilling. Recorded as ADR 0005 and glossary terms (Index job, Worker, Reindex) in #80.
Where it runs
Indexing never runs in the API process. A separate worker process (same image, its own command, e.g. node src/worker.ts) drains one durable queue. The API only adds jobs.
The CLI uses the same queue: pnpm index adds jobs, then drains the queue itself until it is empty, and exits non-zero if any job failed. pnpm crates:import queues each crate it writes and drains too. pnpm dev starts the worker beside the API.
Rejected: a loop inside the API draining the same queue (one process fewer, but large crates stall requests, API restarts interrupt jobs, capacity follows API replicas); CLI only.
Triggers
Explicit only: the admin pages, the CLI, Import and Deposit. No schedule, no watching OCFL. A sweep could be added later as one more thing that queues jobs.
Index in place is "index everything" run against an existing repository.
Granularity
The unit is always one crate. "A crate and those under it" and "everything" expand into one job per crate.
A job always builds both indexes, rows then documents. Indexing or deleting one index type on its own is dropped. The admin pages can still show the two states separately.
Incremental by default: a crate whose head version is already indexed is skipped. Reindex (force) deletes and rebuilds regardless. A full delete-and-rebuild is an explicit option, no longer what pnpm index does by default.
Jobs can run in any order: rootCollection is found by reading parent crates from the repository, not the indexes.
Deposit
A deposit succeeds once the crate is written to OCFL. Indexing follows as a job; an indexing failure is a failed job, not a failed deposit.
The job carries only the crate's id and reads the head version from the repository, never the crate the API had in hand.
Replicas and overlap
Any number of API replicas, workers and CLI runs. Each job is claimed by exactly one.
A crate has at most one running job and one waiting job. Asking again while one waits does nothing; asking while one runs leaves one follow-up.
A deposit landing mid-index: the running job indexes the version it opened and records it; the follow-up sees a newer head and indexes that.
Deleting and removed crates
Deleting a crate's indexes is a job in the same queue, serialised with indexing for that crate rather than refused.
"Delete everything" is one special job that discards waiting jobs and clears both indexes wholesale.
"Index everything" also queues a delete for each crate that has indexes but is no longer in the repository. Indexing a named crate that is missing deletes its indexes and succeeds.
Failures
A job that fails because of its crate (unreadable metadata, a database or search error) is marked failed with its error and waits to be asked again; no automatic retry.
A job whose worker died goes back to the queue, up to three times, then is marked failed.
In a database or OpenSearch outage the worker pauses and reconnects rather than failing every job.
Activity
docs: record that indexing runs in a worker draining a durable queue
johnf commentedon Nov 17, 2026
Resolution
Decided by grilling. Recorded as ADR 0005 and glossary terms (Index job, Worker, Reindex) in #80.
Where it runs
node src/worker.ts) drains one durable queue. The API only adds jobs.pnpm indexadds jobs, then drains the queue itself until it is empty, and exits non-zero if any job failed.pnpm crates:importqueues each crate it writes and drains too.pnpm devstarts the worker beside the API.Triggers
Granularity
pnpm indexdoes by default.rootCollectionis found by reading parent crates from the repository, not the indexes.Deposit
Replicas and overlap
Deleting and removed crates
Failures
Handed on
rootCollectionand aggregates stale, and nothing here reindexes the children.