Skip to content

Deleting a crate's database index removes entities other crates share #35

Open

Description

@johnf

Problem

An entity's id is its primary key, so when several crates contain the same entity (for example https://github.com/Language-Research-Technology/crate-o), only the first crate to be indexed stores it. StructuralIndexer.index sets rootCollection to that crate, and every later crate's prisma.entity.create fails with a duplicate-key error, which is logged as "Error indexing entity" and skipped (src/indexer/structural.ts).

StructuralIndexer.delete(crateId) deletes by rootCollection, so deleting the first crate's index removes the shared entity (and its file) for every crate. The other crates have no copy until one of them is reindexed.

This predates the admin pages, but the admin pages' Delete action (#32) makes it easy to hit.

Reproduce

  1. Index two crates that both contain the same entity.
  2. Delete the index of whichever was indexed first.
  3. The entity is gone from the structural index, though the second crate still references it.

Expected

Deleting one crate's index leaves entities that other indexed crates still contain.

Notes

  • Entities counted twice show up as duplicate-key errors in the logs on every full index run (seen in CI on PR feat!: admin pages served by the API, with sign-in #33).
  • Possible directions: track which crates contain each entity (a join table) and delete an entity only when no crate is left; or reindex the remaining owners after a delete. Needs a design decision.

Activity

  1. changed the title [-]Deleting a crate's structural index removes entities other crates share[/-] [+]Deleting a crate's database index removes entities other crates share[/+] on Nov 4, 2026
  2. johnf commented on Nov 4, 2026

    @johnf
    CollaboratorAuthor

    Renamed from "Deleting a crate's structural index removes entities other crates share": #63 renamed the structural index to the database index.

    How it behaves since #63:

    • Ownership: a crate's rows are found by meta.crateId, not rootCollection. The first crate indexed still owns a shared entity's row, and later crates still fail its insert with a duplicate-key error. For example, https://github.com/Language-Research-Technology/crate-o appears in all three distributed_* test crates.
    • Search documents: the search indexer writes documents only for entities that have a row owned by the crate being indexed. The other crates skip the shared entity with "Skipped: no database row", so search total stays consistent.
    • The bug stands: deleting the owning crate's indexes removes the shared entity's row and its search document for every crate, until another crate that contains it is reindexed.
    • Logging: the duplicate-key failure is still logged at ERROR with a full Prisma stack on every full index run.
  3. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    The decision behind this bug is now the ticket #72 (How shared entities are stored and deleted) on the indexer redesign map, #65.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions