Skip to content

Oni aggregates: precompute or compute per request #73

Closed

Description

@johnf

Question

Should the indexer precompute what Oni needs per entity, instead of computing it per request? Decide:

  • counts (today one recursive memberOf query per collection or object, across crates), and the subtree lists language, communicationMode and mediaType.
  • Where they live: database rows, search documents (e.g. ancestors stored on each document and aggregated in OpenSearch), or computed per request.
  • How ancestors in other crates are refreshed when a crate is indexed, re-indexed or deleted.

Scale: the largest collection has ~30,000 entities. Background: #49.

Activity

  1. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Input from "Where indexing runs and what triggers it" (#70, resolved): the unit of work is one index job per crate, incremental by default, run by a worker outside the API. Open for this ticket: if aggregates are precomputed, a job for one crate changes the aggregates of ancestors in other crates, and nothing in #70 reindexes ancestors or children when a related crate changes.

  2. self-assigned this
    on Nov 17, 2026
  3. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    Resolution

    Aggregates are precomputed by the indexer into a new table of our own, where each crate records what it adds below each ancestor. The API adds those rows up per request. Decided with John on 2026-11-17.

    1. Where the aggregates come from

    A new table in our own models (prisma/models), used only for aggregates. arocapi's Entity and File tables don't change.

    • When the indexer indexes a crate, it writes one row per (crate, entity) pair for every entity that has something of this crate's at or below it: the crate's own collections, objects and files, and their ancestors in parent crates.
    • Each row holds what this crate adds for that entity: the counts collections, objects, subCollections and files below it, and the language, communicationMode and mediaType values at or below it.
    • A request adds up the counts and unions the lists of the rows for the entities on the page, in one flat query with no walk. This replaces the recursive memberOf query in src/oni.ts.
    • An index job writes and deletes only its own crate's rows. Deleting a crate deletes its rows and needs no repository read. Two jobs never write the same row.
    • Nothing is counted twice, because a collection, object or file belongs to one crate ("How shared entities are stored and deleted", How shared entities are stored and deleted #72).

    Example: crates 2 and 3 each hold a collection that is a member of "Sydney Speaks" in crate 1.

    Crate Entity Collections Objects Files Languages
    2 Sydney Speaks 1 200 600 English
    2 Interviews 1970s 0 200 600 English
    3 Sydney Speaks 1 300 900 English, Greek
    3 Interviews 1980s 0 300 900 English, Greek

    "Sydney Speaks" reads as 2 collections, 500 objects, 1,500 files, English and Greek.

    Rejected:

    • Per request, recursive SQL (today): walks the whole subtree on every read; the largest collection has ~30,000 entities.
    • Ancestors on each search document, aggregated in OpenSearch per request (oni-ui's old API): the entity and list endpoints would depend on the search backend being up and refreshed.
    • Totals stored on the ancestor's own row: a job for one crate would rewrite another crate's rows, and two child crates indexing at once would race on the same parent.

    2. What the three lists mean

    Each list is the entity's own values plus the values of everything below it.

    • language: its own languages plus those of every collection, object and file below it.
    • communicationMode: the same. Sent as a list, which is how Oni reads it, although Oni's doc types it as a single value.
    • mediaType: the media types of the files below it; for a file, its own.

    3. When a parent crate changes or is missing

    No automatic re-index of child crates. A child crate's rows stay as they were until that crate is indexed again.

    • A parent crate moved below a different collection: the child crates still add to the old ancestors until they are reindexed. rootCollection behaves the same way.
    • A parent crate that can't be read: the child crate writes rows only for the ancestors it could find, and logs the warning it logs today.
    • A parent crate deleted: the child crates' rows for it stay, and nothing reads them because the collection has no row.
    • "Reindex all" puts everything right.

    4. Which values the lists carry

    The same labels the search index uses for its inLanguage, communicationMode and encodingFormat facets (names such as "English" and "Spoken", not ids or codes). Oni's link from a communicationMode value then lands on a matching search. The one derivation step feeds the row, the search document and the aggregates.

    5. What is counted

    The counts use the indexer's own classification, not an exact match on the pcdm type URIs.

    • collections and objects: every entity below that the indexer indexed as a collection or an object. subCollections: collections directly below. files: every file below, held or not (as "What the indexer derives from a crate", What the indexer derives from a crate #69, decided).
    • The entity itself is not counted.
    • People, organisations and software are never counted and have nothing below them: zero counts and empty lists.

    Records

    Handed on

    • "The arocapi boundary" (The arocapi boundary #75): the read is one query per page from a transformer, so arocapi#142 (batch transformers) would replace the microtask batching in src/oni.ts. The table is ours and arocapi never joins to it.
    • "Where index state lives and what the admin pages read" (Where index state lives and what the admin pages read #71): the aggregates table is one more thing an index job writes and a delete removes, per crate.
    • Not checked: read time of the flat query and row counts at production scale. Belongs with "Performance at scale" in the map's fog.
  4. added a commit that references this issue on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions