Skip to content

V1-14: Documents, connectors and the processing pipeline #652

Open

Description

@kelvinkipruto

Type: AFK. Can be built and merged without a decision.

Owner: koechkevin. Order: 5 of 12 for this owner, after #668 and before #653. Track: pipeline. Build order: #672

Part of PromiseTracker v3, version 1. Specification: docs/spec/promisetracker-v3-system-specification.md. Section numbers below refer to it.

What to build

Bringing in source documents by upload or link, reading plain text and HTML, with one processing
status that is also the work queue.

Start here

  • Read first: sections 7.8, 9.2 to 9.4, 9.7, 9.8, 13.4 and 13.5 of the specification.
  • Code goes in: documents, connectors and pipeline jobs in plugins/core/src. The uploadConnector and plaintextExtractor in plugins/core/src/adapters.ts exist, and the first is a stub.
  • Already there: the retry helper and circuit breaker from V1-25: Errors, logging, Sentry and the circuit breaker #666, and the Tika container in docker/compose.yaml for the next issue.
  • From v2: src/utils/ssrf.ts, src/utils/files.ts, src/utils/fileSignature.ts, and their tests tests/int/ssrf.int.spec.ts, tests/int/files.int.spec.ts, tests/int/fileSignature.int.spec.ts and tests/int/uploadValidation.int.spec.ts. Do not carry over src/lib/workflowLocks.ts. Read a v2 file with git show v2-final:<path>.
  • Migration: Yes.
  • How to work: AGENTS.md, section "Working on an issue".

Acceptance criteria

  • Documents collection as in section 7.8, with the processing states in section 9.4
  • The upload connector, and the url connector built on v2's SSRF safeguards: HTTPS only,
    private and metadata addresses refused, each redirect checked, time and size limits
    (section 13.4). v2's SSRF tests carry over and pass
  • File type checked from the first bytes, then the declared type, then the extension, using
    v2's file signature check. SVG refused
  • Uploads use Payload 4's upload versions, keep the original file, and carry a credit field.
    Identical files are stored once by checksum
  • The plaintext extractor reads text, Markdown and HTML, normalises Unicode and keeps every
    character, with a test using accented French text
  • Jobs use Payload 4's leases and concurrency keys so two workers never process one document
    (section 9.7). The v2 workflow lock is not carried over
  • Retries, backoff and time limits as in section 9.7, using the retry helper and circuit
    breaker from V1-25: Errors, logging, Sentry and the circuit breaker #666
  • Every stage writes its status and any error message, shown in the admin list
  • The uploader gets an email when their document fails after all retries, sent through
    Payload's email adapter. Until a provider is chosen (D40), emails go to the log, and tests
    check the logged message
  • The stages of one document run as one workflow, so each stage starts when the last one
    ends (section 9.8)
  • The hourly cleanupJobs job removes finished jobs, and failed ones after 7 days (sections
    9.8 and 13.9)

Blocked by

Unblocks

Note

It does not wait for the publishing rules, so the document pipeline can be built alongside the data model.

Metadata

Metadata

Assignees

Labels

featureFeaturesecurityPull requests that address a security vulnerabilitystatus: blockedWaits for the issues in its Blocked by sectiontrack: pipelineDocuments, extraction, AI and failure handlingv3PromiseTracker v3

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions