Repository navigation
Open
Labels
ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agent
Description
Parent
What to build
The tracer bullet: an admin asks for a crate to be indexed, an Index job is queued, a Worker runs it, and the admin pages show the crate's Index state read from Postgres. Rows and search documents still come from today's indexing code, wrapped to index one crate. Later tickets replace it.
Follows ADR 0005 (indexing runs in a worker) and ADR 0007 (index state lives in Postgres).
- Schema (our own models):
- Index job: crate id, kind (index, reindex, delete), status, attempt count, claiming worker. Partial unique indexes allow one running and one waiting job per crate. Jobs are claimed with
FOR UPDATE SKIP LOCKED. - Crate record: name, Indexed version, when indexed, entity and file counts.
- Worker: id and last heartbeat.
- Index job: crate id, kind (index, reindex, delete), status, attempt count, claiming worker. Partial unique indexes allow one running and one waiting job per crate. Jobs are claimed with
- Index queue module (the only indexing module the API may import):
- Ask for index, reindex or delete of one crate, or "index everything" (one job per crate in the repository).
- Read each crate's index state, and the counts of crates per state.
- Asking again while a job waits does nothing; asking while one runs leaves one follow-up.
- Indexer module: one operation, "work the queue", run until empty or forever.
- Index or reindex: clear the indexed version, write rows, then documents, then set version, time and counts. A plain index skips the crate when its indexed version equals its head version.
- Delete: clear the version, then remove documents and rows. The crate record stays, as not indexed.
- A job carries only the crate id and reads the head version from the repository.
- A successful job is deleted. A failed job keeps its error until the next job for that crate replaces it, and isn't retried.
- A worker writes a heartbeat every 30 seconds.
- Worker entry point: "work the queue" forever.
pnpm devstarts it beside the API, and the Dockerfile and compose files can run it. - CLI:
pnpm indexqueues every crate (--forceto reindex), drains the queue, and exits non-zero if any job it queued failed. - Admin pages:
- They read only Postgres. Each crate listed has one state: not indexed, queued, indexing, deleting, indexed (version, time, counts) or failed (error).
- Precedence: a running or waiting job, then a failed job, then the record.
- Counts per state at the top of the page. "No worker running" whenever no heartbeat is fresh.
- Their actions call the index queue only.
- Wiring: the API entry point opens the OCFL repository, builds the content storage backend and the index queue, and passes them in. There's no module-level mutable state in the repository, index queue or indexer modules.
- Removed: the in-memory state cache, the
Repositoryinterface and its dynamic import, and the two-index state type. - Lint rule: Biome
noRestrictedImportslets only the indexer itself, the worker entry point, the CLI scripts and the top-level tests import the indexer. - Coverage: the index queue and indexer folders are held at 100% from this first commit. Entry points stay excluded, and only parse arguments and call into the source tree.
Fold in the matching leftovers from Indexer: redesign storage, indexing and index state: objects() arguments, the "entities" count and log noise where touched.
Acceptance criteria
- Asking for a crate on the admin pages queues a job; with the worker running, the crate shows as indexed with its version and counts, and is searchable.
- Indexing again without changes skips the crate. Reindex rebuilds it. Delete leaves it listed as not indexed and unsearchable.
- A second ask while one job waits adds nothing; an ask while one runs leaves exactly one follow-up.
- Stopping the API doesn't stop a running job in the worker.
-
pnpm indexindexes every crate in a repository and exits non-zero when one fails. - A failed job shows its error on the admin pages and is replaced by the next ask.
- "No worker running" shows when no worker heartbeat is fresh.
- Index queue tests cover one waiting and one running job per crate, and how index state is derived.
- The lint rule fails on an import of the indexer from the admin module.
- Index queue and indexer coverage is 100%.
pnpm testandpnpm lintpass.
Metadata
Metadata
Assignees
Labels
ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agent
Activity