Skip to content

Packaging the indexer, with Deposit in mind #76

Closed

Description

@johnf

Question

Should the indexer be its own package (making the repo a pnpm workspace), or stay in src/ with a clean module boundary? Weigh: Deposit reusing much of the indexer, a worker process (if one is decided), sharing with other arocapi deployments, and not going overboard for needs that may never come. Current lean: not a separate package yet, but written so Deposit can reuse it.

Activity

  1. self-assigned this
    on Nov 17, 2026
  2. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Resolution

    Decided by grilling. No ADR (a folder is easy to turn into a package later) and no new glossary terms.

    One package

    • ldacapi stays one private package. The indexer is a folder in src/, not a workspace package. The API, worker and CLI ship in one image.
    • Nothing is provided for other arocapi deployments to reuse it.
    • Rejected: a pnpm workspace with an indexer package (a second package.json, cross-package Prisma client wiring and Dockerfile changes, for no consumer outside this repo). If the interface below is kept narrow, splitting later is a mechanical move.

    Two modules

    • Index queue: ask for an index, reindex or delete job, and read index state. Used by the API (admin pages, and Deposit later), the CLI and the worker.
    • Indexer: works the queue. Used only by the worker entry point and the CLI.
    • The API never imports the indexer. Deposit's whole dependency on indexing is one "add a job" call on the index queue.

    The indexer's interface

    • One operation: "work the queue", either until it is empty (CLI) or forever (worker). The entry points are a few lines each.
    • Claiming jobs, heartbeats, retries, reading the repository, derivation and the three writers (rows, documents, aggregates) sit inside, as internal seams used only by the indexer's own tests.
    • Rejected: exposing "index one crate" and letting each entry point build its own loop (the worker and CLI could drift apart on retry and heartbeat behaviour).

    Derivation is pure

    • The step that turns crate metadata into rows, documents and aggregates does no I/O: crate metadata and a few looked-up facts in, rows/documents/aggregates out.
    • This keeps Deposit free to run it later on a crate in hand (for example to refuse a crate claiming a collection another crate owns). Nothing is built for that now.

    Enforcement

    • A Biome noRestrictedImports override makes pnpm lint fail when anything outside the indexer folder, the worker entry point and the CLI scripts imports from the indexer. Checked on a throwaway sample with the repo's Biome: it flags the import from an admin file and allows it from src/worker.ts.

    Code both sides need

    • The "is this licence open" function, the id rule, the file storage meta builder and the access-control label type live outside the indexer folder, imported by both the indexer and the API.
    • The Oni aggregate read query lives on the API side and reads the table directly.

    Handed on

    • Layout of src/ after the redesign #77: which folders the index queue, the indexer, the shared functions, the worker entry point and the CLI scripts live in, and the exact paths the lint rule names.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions