Skip to content

Where indexing runs and what triggers it #70

Closed

Description

@johnf

Question

Where does indexing run and what triggers it? Decide:

  • Where: in the API process, a separate worker, or a CLI only (or a mix).
  • Triggers: the admin pages, Import, Deposit, indexing an existing repository in place, a schedule, or detected change in OCFL.
  • Granularity: one crate, a collection, everything; full rebuild versus incremental.
  • For Deposit: does indexing run synchronously (the depositor can read back what they deposited), and does a deposit fail if indexing fails?
  • Replicas: what happens with more than one API replica.

Must not block any of the three ways crates arrive (map Notes).

Activity

  1. self-assigned this
    on Nov 17, 2026
  2. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Resolution

    Decided by grilling. Recorded as ADR 0005 and glossary terms (Index job, Worker, Reindex) in #80.

    Where it runs

    • Indexing never runs in the API process. A separate worker process (same image, its own command, e.g. node src/worker.ts) drains one durable queue. The API only adds jobs.
    • The CLI uses the same queue: pnpm index adds jobs, then drains the queue itself until it is empty, and exits non-zero if any job failed. pnpm crates:import queues each crate it writes and drains too. pnpm dev starts the worker beside the API.
    • Rejected: a loop inside the API draining the same queue (one process fewer, but large crates stall requests, API restarts interrupt jobs, capacity follows API replicas); CLI only.

    Triggers

    • Explicit only: the admin pages, the CLI, Import and Deposit. No schedule, no watching OCFL. A sweep could be added later as one more thing that queues jobs.
    • Index in place is "index everything" run against an existing repository.

    Granularity

    • The unit is always one crate. "A crate and those under it" and "everything" expand into one job per crate.
    • A job always builds both indexes, rows then documents. Indexing or deleting one index type on its own is dropped. The admin pages can still show the two states separately.
    • Incremental by default: a crate whose head version is already indexed is skipped. Reindex (force) deletes and rebuilds regardless. A full delete-and-rebuild is an explicit option, no longer what pnpm index does by default.
    • Jobs can run in any order: rootCollection is found by reading parent crates from the repository, not the indexes.

    Deposit

    • A deposit succeeds once the crate is written to OCFL. Indexing follows as a job; an indexing failure is a failed job, not a failed deposit.
    • The job carries only the crate's id and reads the head version from the repository, never the crate the API had in hand.

    Replicas and overlap

    • Any number of API replicas, workers and CLI runs. Each job is claimed by exactly one.
    • A crate has at most one running job and one waiting job. Asking again while one waits does nothing; asking while one runs leaves one follow-up.
    • A deposit landing mid-index: the running job indexes the version it opened and records it; the follow-up sees a newer head and indexes that.

    Deleting and removed crates

    • Deleting a crate's indexes is a job in the same queue, serialised with indexing for that crate rather than refused.
    • "Delete everything" is one special job that discards waiting jobs and clears both indexes wholesale.
    • "Index everything" also queues a delete for each crate that has indexes but is no longer in the repository. Indexing a named crate that is missing deletes its indexes and succeeds.

    Failures

    • A job that fails because of its crate (unreadable metadata, a database or search error) is marked failed with its error and waits to be asked again; no automatic retry.
    • A job whose worker died goes back to the queue, up to three times, then is marked failed.
    • In a database or OpenSearch outage the worker pauses and reconnects rather than failing every job.

    Handed on

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions