Skip to content

Indexing performance at scale #88

Closed

Description

@johnf

Question

How does the indexer stay within acceptable time and memory for the largest collection (~30,000 entities)? How are row writes, search documents and aggregate rows batched, is a crate's metadata held whole in memory or streamed, how many jobs does one worker run at once, and what indexing time for that collection is acceptable? Does anything need measuring before the spec can commit to numbers?

Activity

  1. self-assigned this
    on Nov 17, 2026
  2. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Resolution

    Decided by grilling. The spec states rules and one measured target; the numbers below are starting figures for a benchmark to tune, not promises.

    What the design must cover

    • Both shapes of the largest collection (~30,000 entities): one crate holding all of it, and the same collection split across thousands of crates.

    Target, and how it is checked

    • The largest collection is fully indexed in under 10 minutes on the production VM, with the worker under 1 GB of memory.
    • Nothing is measured before the spec: today's code is about to be rewritten. An early implementation issue builds a benchmark over a generated 30,000-entity collection, in both shapes, including text files of 50 KB, 1 MB and 5 MB.
    • The same benchmark checks the aggregates read left open by Oni aggregates: precompute or compute per request: a page of 100 entities with aggregates adds no more than 50 ms over the same page without them. If it does, add a database index on the aggregates table; the design doesn't change.

    Memory

    • One crate's metadata is held whole in memory and derived in one pass (the pure derivation step from What the indexer derives from a crate). No streaming parse.
    • File text is read one file at a time and never held together.
    • At most the first 1 MB of a file's text is read and put in the search index; a file cut short is logged. The file itself is untouched. A constant, not a setting. Known limit for the spec: a search can miss words past the cut.

    Database writes

    Search writes

    • Bulk requests capped at about 5 MB or 1,000 documents, whichever comes first. With text capped at 1 MB, a request always holds several documents.
    • The worker never forces an OpenSearch refresh. A crate can show as indexed about a second before its entities appear in search. Tests that need it refresh themselves.

    Worker

    • One job at a time per worker; more throughput comes from more workers. No worker threads. This is what bounds memory to one crate.

    Repository reads

    • A job opens its own OCFL object once and takes everything from that (every open reads and hashes the whole inventory; a file's size is then one stat, per What @ocfl/ocfl-fs 1.0 beta offers for change detection and reading).
    • The worker keeps an in-memory cache of crate → parent collection for crates it has read, rechecked with the cheap is-it-still-current stat, so a parent crate's metadata is parsed once per worker, not once per child crate. It holds only ids and is never persisted.

    Considered and rejected

    • One transaction per crate: holds locks and memory for minutes over 30,000 rows, and the indexed version already covers a half-written crate.
    • Streaming the metadata: parents, licence fallback and reverse links need the whole graph, and 30,000 entities is tens of megabytes.
    • Several jobs at once in one worker: parsing blocks Node's one thread, so jobs would take turns while memory grows.
    • No cap on text, a 10 MB cap, or skipping the text of large files: Nabu's reindex task (lib/tasks/search.rake) drops to one document per request above 1 MB and is slow there; cutting at 1 MB costs the same as a 1 MB file and keeps the file findable by its opening.
    • A measuring task on the map before the spec.

    Not checked

    • Nabu analyses file text twice per document and again per segment; ldacapi's text field is one plain text field, so large text may cost less here. Unmeasured, so the 1 MB line stands until the benchmark says otherwise.

    No ADR (every figure is easy to change) and no glossary change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions