You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Oni aggregates: precompute or compute per request #73
Input from "Where indexing runs and what triggers it" (#70, resolved): the unit of work is one index job per crate, incremental by default, run by a worker outside the API. Open for this ticket: if aggregates are precomputed, a job for one crate changes the aggregates of ancestors in other crates, and nothing in #70 reindexes ancestors or children when a related crate changes.
Aggregates are precomputed by the indexer into a new table of our own, where each crate records what it adds below each ancestor. The API adds those rows up per request. Decided with John on 2026-11-17.
1. Where the aggregates come from
A new table in our own models (prisma/models), used only for aggregates. arocapi's Entity and File tables don't change.
When the indexer indexes a crate, it writes one row per (crate, entity) pair for every entity that has something of this crate's at or below it: the crate's own collections, objects and files, and their ancestors in parent crates.
Each row holds what this crate adds for that entity: the counts collections, objects, subCollections and files below it, and the language, communicationMode and mediaType values at or below it.
A request adds up the counts and unions the lists of the rows for the entities on the page, in one flat query with no walk. This replaces the recursive memberOf query in src/oni.ts.
An index job writes and deletes only its own crate's rows. Deleting a crate deletes its rows and needs no repository read. Two jobs never write the same row.
Example: crates 2 and 3 each hold a collection that is a member of "Sydney Speaks" in crate 1.
Crate
Entity
Collections
Objects
Files
Languages
2
Sydney Speaks
1
200
600
English
2
Interviews 1970s
0
200
600
English
3
Sydney Speaks
1
300
900
English, Greek
3
Interviews 1980s
0
300
900
English, Greek
"Sydney Speaks" reads as 2 collections, 500 objects, 1,500 files, English and Greek.
Rejected:
Per request, recursive SQL (today): walks the whole subtree on every read; the largest collection has ~30,000 entities.
Ancestors on each search document, aggregated in OpenSearch per request (oni-ui's old API): the entity and list endpoints would depend on the search backend being up and refreshed.
Totals stored on the ancestor's own row: a job for one crate would rewrite another crate's rows, and two child crates indexing at once would race on the same parent.
2. What the three lists mean
Each list is the entity's own values plus the values of everything below it.
language: its own languages plus those of every collection, object and file below it.
communicationMode: the same. Sent as a list, which is how Oni reads it, although Oni's doc types it as a single value.
mediaType: the media types of the files below it; for a file, its own.
3. When a parent crate changes or is missing
No automatic re-index of child crates. A child crate's rows stay as they were until that crate is indexed again.
A parent crate moved below a different collection: the child crates still add to the old ancestors until they are reindexed. rootCollection behaves the same way.
A parent crate that can't be read: the child crate writes rows only for the ancestors it could find, and logs the warning it logs today.
A parent crate deleted: the child crates' rows for it stay, and nothing reads them because the collection has no row.
"Reindex all" puts everything right.
4. Which values the lists carry
The same labels the search index uses for its inLanguage, communicationMode and encodingFormat facets (names such as "English" and "Spoken", not ids or codes). Oni's link from a communicationMode value then lands on a matching search. The one derivation step feeds the row, the search document and the aggregates.
5. What is counted
The counts use the indexer's own classification, not an exact match on the pcdm type URIs.
collections and objects: every entity below that the indexer indexed as a collection or an object. subCollections: collections directly below. files: every file below, held or not (as "What the indexer derives from a crate", What the indexer derives from a crate #69, decided).
The entity itself is not counted.
People, organisations and software are never counted and have nothing below them: zero counts and empty lists.
No ADR: both indexes can be deleted and rebuilt, so the choice is cheap to reverse.
Handed on
"The arocapi boundary" (The arocapi boundary #75): the read is one query per page from a transformer, so arocapi#142 (batch transformers) would replace the microtask batching in src/oni.ts. The table is ours and arocapi never joins to it.
Activity
johnf commentedon Nov 17, 2026
Input from "Where indexing runs and what triggers it" (#70, resolved): the unit of work is one index job per crate, incremental by default, run by a worker outside the API. Open for this ticket: if aggregates are precomputed, a job for one crate changes the aggregates of ancestors in other crates, and nothing in #70 reindexes ancestors or children when a related crate changes.
johnf commentedon Nov 17, 2026
Resolution
Aggregates are precomputed by the indexer into a new table of our own, where each crate records what it adds below each ancestor. The API adds those rows up per request. Decided with John on 2026-11-17.
1. Where the aggregates come from
A new table in our own models (
prisma/models), used only for aggregates. arocapi'sEntityandFiletables don't change.collections,objects,subCollectionsandfilesbelow it, and thelanguage,communicationModeandmediaTypevalues at or below it.memberOfquery insrc/oni.ts.Example: crates 2 and 3 each hold a collection that is a member of "Sydney Speaks" in crate 1.
"Sydney Speaks" reads as 2 collections, 500 objects, 1,500 files, English and Greek.
Rejected:
2. What the three lists mean
Each list is the entity's own values plus the values of everything below it.
language: its own languages plus those of every collection, object and file below it.communicationMode: the same. Sent as a list, which is how Oni reads it, although Oni's doc types it as a single value.mediaType: the media types of the files below it; for a file, its own.3. When a parent crate changes or is missing
No automatic re-index of child crates. A child crate's rows stay as they were until that crate is indexed again.
rootCollectionbehaves the same way.4. Which values the lists carry
The same labels the search index uses for its
inLanguage,communicationModeandencodingFormatfacets (names such as "English" and "Spoken", not ids or codes). Oni's link from acommunicationModevalue then lands on a matching search. The one derivation step feeds the row, the search document and the aggregates.5. What is counted
The counts use the indexer's own classification, not an exact match on the pcdm type URIs.
collectionsandobjects: every entity below that the indexer indexed as a collection or an object.subCollections: collections directly below.files: every file below, held or not (as "What the indexer derives from a crate", What the indexer derives from a crate #69, decided).Records
GLOSSARY.mdgains Aggregates: docs: add Aggregates to the glossary #85Handed on
src/oni.ts. The table is ours and arocapi never joins to it.docs: add Aggregates to the glossary