Skip to content

Progress and failure reporting for index runs #87

Closed

Description

@johnf

Question

Beyond the one index state per crate that the admin pages already show (see #71), how does a long index run report progress and failures? What does the worker log per job and per crate, what does a kept failed job record (error, stage, attempt), what does the CLI print while it drains the queue, and does anything need to alert a person when jobs fail or no worker is running?

Activity

  1. self-assigned this
    on Nov 17, 2026
  2. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Resolution

    Decided by grilling. No ADR and no glossary change: nothing here is hard to reverse.

    Errors inside a crate

    • Any error fails the job: an entity that can't be written, a file held in the repository that can't be read, a document that can't be indexed. Nothing is logged and skipped, so an indexed version always means the whole crate is indexed.
    • Files the crate describes but the repository doesn't hold stay a normal case (a File row with no storage location, see What the indexer derives from a crate #69) and fail nothing.
    • The job stops at the first error. It does not carry on to collect every problem in the crate.
    • Rejected: "indexed with warnings" (a new state plus somewhere to keep warnings); log and carry on, as today (a crate looks indexed with pieces missing); collecting a capped list of errors.
    • Known cost: one corrupt file keeps a large collection failed until it is fixed, and a crate with many bad files is fixed one error per attempt. If that proves painful, the answer is a separate "check this crate" command, not a longer error list. Not planned.

    What a kept failed job records

    • The error message; the stage it failed in (reading the crate, deriving, database index, search index, aggregates); the entity or file id it was on, when there is one; when it failed; the attempt count.
    • The stack trace goes to the log only, tied to the job by its id.

    Progress

    • Progress of a whole run is the count of crates per index state the admin pages already show (Where index state lives and what the admin pages read #71).
    • A running job also records its stage and a done-of-total counter, saved in the worker's existing 30-second heartbeat write, so it adds no writes. The crate's page shows, for example, "Indexing: search index, 12,000 of 30,000, since 10:42".
    • The counter can lag by up to 30 seconds and updates only on page reload. It lets an admin tell a slow job from one hanging on a request, which the dead-worker check does not catch.
    • Rejected: state and start time only.

    Logs

    • The worker and the CLI write the same lines.
    • Info: one line when a job starts (job id, crate id, index or delete, attempt) and one when it finishes (duration, version, entity and file counts).
    • Error: one line when a job fails, with the crate id, job id, stage and stack trace. These fields are fixed so log monitoring can alert on them.
    • Debug only: per-entity and per-file lines, and crates skipped because they are already indexed.

    CLI

    • When the queue is empty the CLI prints a summary: how many crates were indexed, skipped, deleted and failed, then each failed crate with its error. No progress bar.
    • The summary covers the crates this run queued, read from Postgres, so it is right when a worker handled some of them. Exit code as decided in Where indexing runs and what triggers it #70: non-zero if any failed.

    Telling a person

    • ldacapi sends no notifications (no email, no webhook). The signals are the admin pages (failed counts, "no worker running"), the CLI's exit code and the error log line.
    • No health endpoint: the new VM has no monitoring to poll one. It can be added when there is.

    Handed on

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions