Repository navigation
Open
Labels
featureFeatureFeaturesecurityPull requests that address a security vulnerabilityPull requests that address a security vulnerabilitystatus: blockedWaits for the issues in its Blocked by sectionWaits for the issues in its Blocked by sectiontrack: pipelineDocuments, extraction, AI and failure handlingDocuments, extraction, AI and failure handlingv3PromiseTracker v3PromiseTracker v3
Milestone
Description
Type: AFK. Can be built and merged without a decision.
Owner: koechkevin. Order: 5 of 12 for this owner, after #668 and before #653. Track: pipeline. Build order: #672
Part of PromiseTracker v3, version 1. Specification: docs/spec/promisetracker-v3-system-specification.md. Section numbers below refer to it.
What to build
Bringing in source documents by upload or link, reading plain text and HTML, with one processing
status that is also the work queue.
Start here
- Read first: sections 7.8, 9.2 to 9.4, 9.7, 9.8, 13.4 and 13.5 of the specification.
- Code goes in: documents, connectors and pipeline jobs in
plugins/core/src. TheuploadConnectorandplaintextExtractorinplugins/core/src/adapters.tsexist, and the first is a stub. - Already there: the retry helper and circuit breaker from V1-25: Errors, logging, Sentry and the circuit breaker #666, and the Tika container in
docker/compose.yamlfor the next issue. - From v2:
src/utils/ssrf.ts,src/utils/files.ts,src/utils/fileSignature.ts, and their teststests/int/ssrf.int.spec.ts,tests/int/files.int.spec.ts,tests/int/fileSignature.int.spec.tsandtests/int/uploadValidation.int.spec.ts. Do not carry oversrc/lib/workflowLocks.ts. Read a v2 file withgit show v2-final:<path>. - Migration: Yes.
- How to work:
AGENTS.md, section "Working on an issue".
Acceptance criteria
- Documents collection as in section 7.8, with the processing states in section 9.4
- The
uploadconnector, and theurlconnector built on v2's SSRF safeguards: HTTPS only,
private and metadata addresses refused, each redirect checked, time and size limits
(section 13.4). v2's SSRF tests carry over and pass - File type checked from the first bytes, then the declared type, then the extension, using
v2's file signature check. SVG refused - Uploads use Payload 4's upload versions, keep the original file, and carry a credit field.
Identical files are stored once by checksum - The
plaintextextractor reads text, Markdown and HTML, normalises Unicode and keeps every
character, with a test using accented French text - Jobs use Payload 4's leases and concurrency keys so two workers never process one document
(section 9.7). The v2 workflow lock is not carried over - Retries, backoff and time limits as in section 9.7, using the retry helper and circuit
breaker from V1-25: Errors, logging, Sentry and the circuit breaker #666 - Every stage writes its status and any error message, shown in the admin list
- The uploader gets an email when their document fails after all retries, sent through
Payload's email adapter. Until a provider is chosen (D40), emails go to the log, and tests
check the logged message - The stages of one document run as one workflow, so each stage starts when the last one
ends (section 9.8) - The hourly
cleanupJobsjob removes finished jobs, and failed ones after 7 days (sections
9.8 and 13.9)
Blocked by
- V1-06: Languages, countries and the site shell #644, owned by kelvinkipruto
- V1-25: Errors, logging, Sentry and the circuit breaker #666, owned by koechkevin
Unblocks
- V1-11: Publishing rules and the cascade #649, owned by kelvinkipruto
- V1-15: plugin-tika: reading PDF, Word and Excel #653, owned by koechkevin
Note
It does not wait for the publishing rules, so the document pipeline can be built alongside the data model.
Metadata
Metadata
Assignees
Labels
featureFeatureFeaturesecurityPull requests that address a security vulnerabilityPull requests that address a security vulnerabilitystatus: blockedWaits for the issues in its Blocked by sectionWaits for the issues in its Blocked by sectiontrack: pipelineDocuments, extraction, AI and failure handlingDocuments, extraction, AI and failure handlingv3PromiseTracker v3PromiseTracker v3
Activity