Skip to content

Indexer: files described but not held, and file text capped at 1 MB #100

Open

Description

@johnf

Parent

Spec: indexer redesign

What to build

Files in the derivation step, finishing the move off today's indexing code.

  • Rule 11, files described but not held (a Metadata-only crate, for example):
    • The file gets its row and document, plus a File row with no storage location, so arocapi answers 404 for its content.
    • Size is the stated contentSize, or 0 when unstated. Media type falls back to application/octet-stream.
    • Its virtual crate gets display: false.
    • One warning per crate gives the count of such files.
  • Rule 12, file text: the text of every text/* file goes into its search document whatever its licence, but only the first 1 MB. A file cut short is logged. File text is read one file at a time, never all held together.
  • File storage meta: built by a plain function with no LDaC imports, in the repository module.
  • A held file that can't be read fails the job.
  • Today's indexing code (the two indexer classes and their wrappers) is deleted.

Acceptance criteria

  • "A file described but not held" (the named case) has a File row with no storage location, size 0 or its stated size, and display: false, with one warning per crate.
  • Its content request answers 404, and its entity is searchable.
  • A 5 MB text file is searchable by words in its first 1 MB, not by words after, and the cut is logged.
  • A held file whose bytes can't be read fails the job.
  • No code from today's indexer remains. Indexer coverage stays at 100%. pnpm test and pnpm lint pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions