Skip to content

Indexer: failure detail, progress and logs #102

Open

Description

@johnf

Parent

Spec: indexer redesign

What to build

Failures and progress an admin or operator can act on.

  • Fail at the first error: any error inside a crate fails the job, and nothing is logged and skipped.
  • A failed job records: its message, its stage (reading the crate, deriving, database index, search index, aggregates), the entity or file id it was on when there is one, when it failed, and its attempt count. The stack trace goes only to the log, tied to the job id.
  • Progress: a running job records its stage and a done-of-total counter, saved in the worker's 30-second heartbeat write so it adds no writes of its own. The crate's admin page shows it, for example "Indexing: search index, 12,000 of 30,000, since 10:42".
  • Logs: the worker and the CLI write the same lines.
    • Info: one line when a job starts (job id, crate id, kind, attempt) and one when it finishes (duration, version, entity and file counts).
    • Error: one line when a job fails, with fixed fields (crate id, job id, stage, stack trace).
    • Debug: per-entity and per-file lines, and crates skipped as already indexed.
  • CLI summary: when its queue is empty, the CLI prints how many of the crates it queued were indexed, skipped, deleted and failed, then each failed crate with its error. The summary is read from Postgres, so it's right when a worker handled some of them.

Acceptance criteria

  • A search client that throws on write fails the job with stage "search index". The crate has no indexed version, and asking again succeeds.
  • One entity that can't be written fails the job, with its id recorded, and leaves no indexed version.
  • The admin pages show a failed crate's message, stage, entity or file id, time and attempt, and a running crate's stage, counter and start time.
  • The progress counter is written only by the heartbeat.
  • The CLI summary lists indexed, skipped, deleted and failed counts and each failure, including jobs a worker ran.
  • Info and error log lines carry the fields above.
  • Indexer coverage stays at 100%. pnpm test and pnpm lint pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions