Skip to content

Where index state lives and what the admin pages read #71

Closed

Description

@johnf

Question

Where does index state live, and what do the admin pages read? Decide:

  • What state exists per crate (queued / indexing / deleting / indexed / failed, last indexed version, errors) and where it's stored (e.g. Postgres in prisma/models).
  • How a queue survives a restart and works across replicas (if indexing is queued at all).
  • How the admin pages get state and counts without scanning the entity table by meta.crateId.
  • How a partial failure (search documents removed, database delete failed) is recorded and recovered.

Today: in-memory stateCache in src/ocfl.ts.

Activity

  1. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Input from Where indexing runs and what triggers it (#70, resolved): indexing runs in a separate worker draining one durable queue that the API and CLI add to. This ticket now also owns: where that queue is stored; recording the OCFL version each crate was indexed at (incremental indexing depends on it); keeping failed jobs and their errors; a crate having at most one running and one waiting job; returning a dead worker's job to the queue (up to three times); and the admin pages showing when no worker is running.

  2. self-assigned this
    on Nov 17, 2026
  3. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Passed on from "How shared entities are stored and deleted" (#72): that ticket adds a crate↔entity link table in our own models, keyed by crate id, as the only record of which crates contain an entity (meta.crateId is dropped, and per-crate entity counts come from the links). Open for this ticket: whether crates get their own table that the links point to. A shared entity (person, organisation, software) counts in each crate that contains it.

  4. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Resolution

    Decided by grilling. Recorded as ADR 0007 and glossary terms (Index state, Indexed version) in #84.

    Where it lives

    • Everything is in our own Postgres tables (prisma/models): index jobs (the queue), one record per crate, and workers.
    • Rejected: pg-boss (its own schema outside our migrations, its own job states to map onto ours); Redis with BullMQ (a new service); keeping every finished job as the history; counting rows by meta.crateId.

    The queue

    • A table of index jobs, claimed with FOR UPDATE SKIP LOCKED. One running and one waiting job per crate is enforced by partial unique indexes.
    • A job that succeeds is deleted; the record holds its result. A failed job is kept with its error until the next job for that crate is asked for, which replaces it. So a crate has at most one failed, one running and one waiting job, and the table doesn't grow.
    • Who asked for what stays in the audit log, not in job history.

    The crate record

    • One per crate known to be in the repository: name, indexed version, when it was indexed, and the number of entities and files the job wrote.
    • The indexed version is the last thing written and the first thing cleared. Index and reindex: clear the version, write rows, then documents, then set version, time and counts. Delete: clear the version, remove documents, then rows.
    • Deleting a crate's indexes keeps its record, as not indexed. A record is removed only when a job finds its crate is gone from the repository. "Delete everything" clears every version and keeps the records.
    • Incremental indexing skips a crate only when its indexed version equals its head version; it never skips a crate with no indexed version.

    Partial failure

    • A failure part-way (for example documents removed, database delete failed) leaves a record with no indexed version and a failed job holding the error. The crate shows as failed.
    • Recovery is asking again: index and delete are both safe to repeat. There is no separate partial state and no repair code.

    Workers

    • Each worker, and each CLI run while it drains the queue, writes a heartbeat every 30 seconds. A running job names its worker.
    • A worker silent for 5 minutes is treated as dead: its job goes back to waiting with its attempt count raised, and is marked failed on the third return. Any worker or CLI run makes this check when it looks for work. (60 seconds was rejected: parsing a large crate blocks Node's one thread, so a healthy worker can miss heartbeats.)
    • A worker shutting down cleanly removes itself and releases its job at once.
    • A job checks it still owns its claim before writing its result, so a worker wrongly judged dead drops its work.

    Admin pages

    • They read only Postgres, never the repository. A crate is listed once something has queued a job for it (Deposit, Import, "Index all"). A crate another tool put in the repository is unlisted until someone runs "Index all"; a repository indexed in place shows an empty list until then.
    • One index state per crate, not one per index: not indexed, queued, indexing, deleting, indexed (with version, time and counts), failed (with the error). Derived: a running or waiting job wins, then a failed job, then the record. This replaces the note in Where indexing runs and what triggers it #70 that the pages could show the two indexes separately.
    • No "out of date" state, as that would mean reading the repository on each page load.
    • The top of the page counts crates per index state, from one grouped query. No grand total of entities, since shared entities would be counted twice.
    • "No worker running" is shown whenever no heartbeat is fresh, whether or not jobs are waiting.

    Handed on

  5. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Input from "Oni aggregates: precompute or compute per request" (#73, resolved): aggregates live in a new table of our own, one row per (crate, entity) pair, written and deleted only by that crate's index job. It is one more thing an index job writes and a crate delete removes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions