Skip to content

Indexer: meet the performance target #106

Open

Description

@johnf

Parent

Spec: indexer redesign

What to build

The new indexer meets the performance target on the benchmark: the largest collection (~30,000 entities) is fully indexed in under 10 minutes with the worker under 1 GB, both as one crate and split across thousands of crates.

  • A job opens its crate's OCFL object once and takes everything from it.
  • The worker keeps an in-memory cache of crate → parent collection (ids only, never persisted), checked against the repository with a cheap stat before use. A parent crate's metadata is then parsed once per worker, not once per child crate.
  • One job at a time per worker, with no worker threads.
  • Tune batch sizes (about 1,000 rows; bulk requests about 5 MB or 1,000 documents) from the benchmark's numbers.
  • If the target can't be met without changing a decided rule, stop and report rather than changing it.

Acceptance criteria

  • The benchmark's numbers for both shapes, on hardware like the production VM, are recorded in a comment here, and meet the target.
  • Each crate's OCFL object is opened once per job.
  • Indexing thousands of child crates parses the parent crate's metadata once per worker.
  • Indexer coverage stays at 100%. pnpm test and pnpm lint pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions