Skip to content

Indexer: pure derivation of entities, parents, ids, root collection, licences and spellings #99

Open

Description

@johnf

Parent

Spec: indexer redesign

What to build

The pure derivation step, with the spec's metadata rules. One pass over a crate's metadata produces every entity row and search document. It does no I/O and imports neither configuration nor the database client; the facts it needs (the parent crate chain, defaults, the open-licence function) are passed in. The rows and documents writers write its result, replacing today's two indexers' metadata handling.

Derivation rules delivered (numbered as in the spec, each with a test named after it at the derivation seam):

  1. Entities indexed by profile: profiled Collection, Object, Person, Organisation, SoftwareApplication, and every File.
  2. Parent of a collection or object: pcdm:memberOf, else reverse pcdm:hasMember.
  3. Parent of a file: isPartOf or reverse hasPart, preferring a RepositoryObject.
  4. Several candidate parents: the first wins, with a warning.
  5. Root collection: from parent crates read from the repository. Null for people, organisations and software.
  6. Entity ids: relative and blank-node ids are prefixed with the crate root id. Absolute IRIs are kept.
  7. Content licence: own → nearest ancestor in the crate → crate root → DEFAULT_LICENSE.
  8. Metadata licence: descriptor → DEFAULT_METADATA_LICENSE.
  9. Access control label: from the open-licence function.
  10. Spellings: only prefixed ldac:/pcdm: properties are read. The Virtual crate serves property names as deposited.
  11. Row and document contents: as today, with the LDaC profile map as a constant inside derivation.

Also:

  • A crate whose content or metadata licence can't be resolved and has no default fails its job.
  • Rows are written in multi-row batches of about 1,000, each its own transaction. Documents go in bulk requests capped at about 5 MB or 1,000 documents. The worker never forces a search refresh.
  • The id rule is a plain function with no LDaC imports, in the repository module.
  • File handling stays as today until the next ticket.

Fold in from Indexer: redesign storage, indexing and index state: _indexEntity, the crate.root check, the LDaCA constants in configuration, and the one-function utils module (moves into derivation).

Acceptance criteria

  • Each rule above has at least one test named after it at the derivation seam, with no database.
  • Derivation imports neither configuration nor the database client; the indexes are written only by the writers.
  • A crate on the bare spelling indexes without parents or LDaC facets, and with no warning.
  • A blank-node id in two crates gives two entities.
  • An object without a licence inherits its collection's licence from within the crate.
  • Today's metadata handling in the two indexers is gone. The crate-to-HTTP tests pass.
  • Indexer coverage stays at 100%. pnpm test and pnpm lint pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions