Skip to content

Indexer: dead workers, retries and outages #101

Open

Description

@johnf

Parent

Spec: indexer redesign

What to build

Workers that die, are wrongly judged dead, or lose their database or OpenSearch.

  • Dead workers: a worker whose heartbeat is older than 5 minutes is dead. Any worker or draining CLI run checks for dead workers when it looks for work. A dead worker's job returns to waiting with its attempt count raised, and is marked failed on its third return.
  • Clean shutdown: a worker shutting down on SIGTERM or SIGINT removes its worker row and releases its job at once.
  • Claim ownership: a job checks it still owns its claim before writing its result, so a worker wrongly judged dead drops its work.
  • Outages: during a database or OpenSearch outage, the worker pauses and reconnects rather than failing every waiting job.
  • Many workers: any number of workers and CLI runs can drain one queue, and each job is claimed once.

Acceptance criteria

  • A test stages a worker row with an old heartbeat and a running job naming it; working the queue returns the job to waiting and runs it. With the attempt count preset, the third return marks it failed. No fake timers.
  • A test at the indexer's internal job seam takes a claim away mid-job, and the job's result write is refused.
  • Two "work the queue" runs at once over several crates index each crate once.
  • A worker stopped with SIGTERM leaves no worker row and no running job.
  • The outage behaviour is checked by hand once, with the steps recorded in a comment here. There's no automated test.
  • Indexer coverage stays at 100%. pnpm test and pnpm lint pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions