You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
How does the indexer stay within acceptable time and memory for the largest collection (~30,000 entities)? How are row writes, search documents and aggregate rows batched, is a crate's metadata held whole in memory or streamed, how many jobs does one worker run at once, and what indexing time for that collection is acceptable? Does anything need measuring before the spec can commit to numbers?
Decided by grilling. The spec states rules and one measured target; the numbers below are starting figures for a benchmark to tune, not promises.
What the design must cover
Both shapes of the largest collection (~30,000 entities): one crate holding all of it, and the same collection split across thousands of crates.
Target, and how it is checked
The largest collection is fully indexed in under 10 minutes on the production VM, with the worker under 1 GB of memory.
Nothing is measured before the spec: today's code is about to be rewritten. An early implementation issue builds a benchmark over a generated 30,000-entity collection, in both shapes, including text files of 50 KB, 1 MB and 5 MB.
The same benchmark checks the aggregates read left open by Oni aggregates: precompute or compute per request: a page of 100 entities with aggregates adds no more than 50 ms over the same page without them. If it does, add a database index on the aggregates table; the design doesn't change.
Memory
One crate's metadata is held whole in memory and derived in one pass (the pure derivation step from What the indexer derives from a crate). No streaming parse.
File text is read one file at a time and never held together.
At most the first 1 MB of a file's text is read and put in the search index; a file cut short is logged. The file itself is untouched. A constant, not a setting. Known limit for the spec: a search can miss words past the cut.
Database writes
Multi-row writes of about 1,000 rows, each batch (entities, files, crate links) its own short transaction. A crate's aggregate rows are one last batch.
Bulk requests capped at about 5 MB or 1,000 documents, whichever comes first. With text capped at 1 MB, a request always holds several documents.
The worker never forces an OpenSearch refresh. A crate can show as indexed about a second before its entities appear in search. Tests that need it refresh themselves.
Worker
One job at a time per worker; more throughput comes from more workers. No worker threads. This is what bounds memory to one crate.
The worker keeps an in-memory cache of crate → parent collection for crates it has read, rechecked with the cheap is-it-still-current stat, so a parent crate's metadata is parsed once per worker, not once per child crate. It holds only ids and is never persisted.
Considered and rejected
One transaction per crate: holds locks and memory for minutes over 30,000 rows, and the indexed version already covers a half-written crate.
Streaming the metadata: parents, licence fallback and reverse links need the whole graph, and 30,000 entities is tens of megabytes.
Several jobs at once in one worker: parsing blocks Node's one thread, so jobs would take turns while memory grows.
No cap on text, a 10 MB cap, or skipping the text of large files: Nabu's reindex task (lib/tasks/search.rake) drops to one document per request above 1 MB and is slow there; cutting at 1 MB costs the same as a 1 MB file and keeps the file findable by its opening.
A measuring task on the map before the spec.
Not checked
Nabu analyses file text twice per document and again per segment; ldacapi's text field is one plain text field, so large text may cost less here. Unmeasured, so the 1 MB line stands until the benchmark says otherwise.
No ADR (every figure is easy to change) and no glossary change.
Activity
johnf commentedon Nov 17, 2026
Resolution
Decided by grilling. The spec states rules and one measured target; the numbers below are starting figures for a benchmark to tune, not promises.
What the design must cover
Target, and how it is checked
Memory
Database writes
Search writes
Worker
Repository reads
stat, per What @ocfl/ocfl-fs 1.0 beta offers for change detection and reading).stat, so a parent crate's metadata is parsed once per worker, not once per child crate. It holds only ids and is never persisted.Considered and rejected
lib/tasks/search.rake) drops to one document per request above 1 MB and is slow there; cutting at 1 MB costs the same as a 1 MB file and keeps the file findable by its opening.Not checked
textfield is one plain text field, so large text may cost less here. Unmeasured, so the 1 MB line stands until the benchmark says otherwise.No ADR (every figure is easy to change) and no glossary change.