You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Beyond the one index state per crate that the admin pages already show (see #71), how does a long index run report progress and failures? What does the worker log per job and per crate, what does a kept failed job record (error, stage, attempt), what does the CLI print while it drains the queue, and does anything need to alert a person when jobs fail or no worker is running?
Decided by grilling. No ADR and no glossary change: nothing here is hard to reverse.
Errors inside a crate
Any error fails the job: an entity that can't be written, a file held in the repository that can't be read, a document that can't be indexed. Nothing is logged and skipped, so an indexed version always means the whole crate is indexed.
Files the crate describes but the repository doesn't hold stay a normal case (a File row with no storage location, see What the indexer derives from a crate #69) and fail nothing.
The job stops at the first error. It does not carry on to collect every problem in the crate.
Rejected: "indexed with warnings" (a new state plus somewhere to keep warnings); log and carry on, as today (a crate looks indexed with pieces missing); collecting a capped list of errors.
Known cost: one corrupt file keeps a large collection failed until it is fixed, and a crate with many bad files is fixed one error per attempt. If that proves painful, the answer is a separate "check this crate" command, not a longer error list. Not planned.
What a kept failed job records
The error message; the stage it failed in (reading the crate, deriving, database index, search index, aggregates); the entity or file id it was on, when there is one; when it failed; the attempt count.
The stack trace goes to the log only, tied to the job by its id.
A running job also records its stage and a done-of-total counter, saved in the worker's existing 30-second heartbeat write, so it adds no writes. The crate's page shows, for example, "Indexing: search index, 12,000 of 30,000, since 10:42".
The counter can lag by up to 30 seconds and updates only on page reload. It lets an admin tell a slow job from one hanging on a request, which the dead-worker check does not catch.
Rejected: state and start time only.
Logs
The worker and the CLI write the same lines.
Info: one line when a job starts (job id, crate id, index or delete, attempt) and one when it finishes (duration, version, entity and file counts).
Error: one line when a job fails, with the crate id, job id, stage and stack trace. These fields are fixed so log monitoring can alert on them.
Debug only: per-entity and per-file lines, and crates skipped because they are already indexed.
CLI
When the queue is empty the CLI prints a summary: how many crates were indexed, skipped, deleted and failed, then each failed crate with its error. No progress bar.
The summary covers the crates this run queued, read from Postgres, so it is right when a worker handled some of them. Exit code as decided in Where indexing runs and what triggers it #70: non-zero if any failed.
Telling a person
ldacapi sends no notifications (no email, no webhook). The signals are the admin pages (failed counts, "no worker running"), the CLI's exit code and the error log line.
No health endpoint: the new VM has no monitoring to poll one. It can be added when there is.
Test strategy for the indexer #89 (tests): "one bad entity fails the job and leaves no indexed version" replaces today's log-and-skip behaviour as something to test.
Activity
johnf commentedon Nov 17, 2026
Resolution
Decided by grilling. No ADR and no glossary change: nothing here is hard to reverse.
Errors inside a crate
What a kept failed job records
Progress
Logs
CLI
Telling a person
Handed on