diff --git a/docs/README.md b/docs/README.md index 83769ba..9511553 100644 --- a/docs/README.md +++ b/docs/README.md @@ -45,5 +45,5 @@ These pages document the completed PageIndex evaluation, future evaluation workf - [Calculate provider costs for PageIndex and vector RAG](token-cost-analysis.md) - [Compare vector RAG with the PageIndex system](vector-rag-vs-pageindex-analysis.md) - [Harden PageIndex before comparing retrieval architectures](pageindex-improvement-analysis.md) -- [Read the vendored PageIndex runtime patch](pageindex-runtime-patch.md) +- [Understand why Vectorless RAG patches PageIndex](pageindex-runtime-patch.md) - [Review deferred hardening](DEFERRED.md) diff --git a/docs/pageindex-runtime-patch.md b/docs/pageindex-runtime-patch.md index 796cf0a..40f77e9 100644 --- a/docs/pageindex-runtime-patch.md +++ b/docs/pageindex-runtime-patch.md @@ -1,147 +1,294 @@ - - + + -# PageIndex runtime patch +# Why Vectorless RAG patches PageIndex -Reference for `apps/api/patches/pageindex-runtime.patch`: every change it makes to the vendored PageIndex source, and whether each one fixes an upstream bug or adds something the ingestion pipeline needs. The patch applies to [VectifyAI/PageIndex](https://github.com/VectifyAI/PageIndex) at pinned revision `190f8b378be58199ca993566a9214dba72089c54` and touches `pageindex/client.py`, `pageindex/page_index.py`, `pageindex/retrieve.py`, `pageindex/utils.py`, and `requirements.txt`. The current patch revision is `+vr7`. +Vectorless RAG uses [PageIndex](https://github.com/VectifyAI/PageIndex)’s document-tree algorithm, but the pinned upstream runtime did not meet the ingestion pipeline’s requirements for accounting, deterministic failure handling, structured output, document coverage, and artifact semantics. The repository therefore applies a reviewable patch to one pinned PageIndex commit during the API image build. This page explains why the patch exists, what it changes, and how the project proves which code produced each artifact. -## How the patch is applied and versioned +## Why the upstream runtime was not sufficient -`apps/api/scripts/install-pageindex.sh` clones the pinned revision into `/opt/pageindex` during the image build, runs `git apply --check`, then applies the patch. The worker launches the patched CLI through `apps/api/src/vectorless_rag/run_pageindex.py`, which reaches it via `PYTHONPATH`. +The ingestion pipeline needs more than a plausible document tree. It must produce a reproducible artifact, charge every provider call to the correct operation, stop within configured limits, and report failures without exposing document content. The pinned upstream revision did not satisfy those requirements in five areas: -Three identity surfaces track the patch: +- **Provider control**: internal model calls bypassed the application’s usage ledger, budget reservations, tracing, timeout, retry, and failure-classification systems +- **Structured output**: calls that expected JSON did not validate JSON at the model boundary, and the fallback parser could corrupt valid text +- **Document robustness**: grouping, table of contents (ToC) handling, page repair, and recursive subdivision rejected or damaged valid academic papers +- **Artifact semantics**: upstream emitted one summary per node, while the v2 artifact requires separate direct and subtree summaries +- **Dependency maintenance**: the runtime depended on vulnerable, unmaintained `PyPDF2` and crashed when a PDF page had no extractable text -- **`pageindex_version`**: pinned as the literal `190f8b378be58199ca993566a9214dba72089c54+vr7` in `config.py`. Bump the `+vrN` suffix whenever patch behavior changes. -- **`patch_sha256`**: computed from the patch file at ingestion time in `pageindex_adapter.py`, so any byte change to the patch shifts artifact identity even without a version bump. -- **`configuration_hash` and `recipe_hash`**: both digest the full version string and the patch digest, so a rebuilt document under a new patch never reuses an old artifact. +These defects occur inside PageIndex’s batching and tree-construction logic. An adapter around the final result cannot recover a section that PageIndex discarded, reverse a corrupt page range, or account for a model call that already happened. Post-processing would also hide the cause by rewriting a damaged tree after the provider spend. The fixes therefore live at the PageIndex call sites and algorithms that own the behavior. -One scoping note: the prompt-injection hardening in the vendored source (`_SYSTEM_HARDENING`, `_secure_doc_text`, `_validate_physical_indices`, and the `` framing) is not part of this patch. The pinned upstream revision already contains it. +Replacing PageIndex with another indexing engine would answer a different research question. The project instead keeps the upstream algorithm, pins its source, and changes only the runtime behavior required for safe, measurable ingestion. -## Change summary +## How the patched runtime enters the application -Each change is either a fix for a defect in upstream PageIndex or a feature the pipeline needs that upstream lacks: +The build and ingestion paths keep upstream source, local modifications, and application policy separate: -| Area | File | Kind | +```mermaid +flowchart LR + Commit[Pinned PageIndex commit] --> Apply[Apply runtime patch] + Patch[pageindex-runtime.patch] --> Apply + Apply --> Test[Provider-free patch tests] + Test --> Image[API and worker image] + Worker[Ingestion worker] --> Adapter[PageIndex adapter] + Adapter --> Launcher[Accounted child launcher] + Image --> Launcher + Launcher --> Tree[Patched PageIndex tree] + Tree --> Artifact[Versioned v2 artifact] +``` + +`apps/api/scripts/install-pageindex.sh` clones PageIndex commit `190f8b378be58199ca993566a9214dba72089c54`, checks that the patch applies, and installs the patched tree at `/opt/pageindex`. The image build then runs `apps/api/scripts/test_pageindex_patch.py` against that exact checkout. + +At ingestion time, `PageIndexAdapter` starts `vectorless_rag.run_pageindex` as an isolated child process. The launcher establishes application usage ownership before it runs the upstream CLI. The patched PageIndex utilities import the application’s model-call functions through that boundary. + +## What the patch changes + +The current `+vr7` patch changes five upstream files: + +| Requirement | Upstream behavior | Patched behavior | | --- | --- | --- | -| Model-call plumbing routed through the app runtime | `utils.py` | Feature, plus a silent-failure bug fix | -| Strict JSON (`expect_json=True`) on structured calls | `page_index.py` | Feature, plus removal of a corrupting parser | -| Page grouping rewrite with oversized-page splitting | `page_index.py` | Bug fixes, plus a missing feature | -| Continuation merge (dedupe and page-order sort) | `page_index.py` | Bug fix | -| Coverage no longer gates verification | `page_index.py` | Bug fix | -| Subdivision failure degrades instead of propagating | `page_index.py` | Bug fix | -| Pageless-ToC routing in `tree_parser` | `page_index.py` | Bug fix | -| Offset fallback in `process_toc_with_page_numbers` | `page_index.py` | Crash fix | -| `process_none_page_numbers` robustness | `page_index.py` | Crash fixes | -| Large-node recursion repair | `page_index.py` | Bug fixes | -| Bottom-up direct and subtree summaries | `utils.py` | Missing feature | -| `pypdf` migration and empty-page guards | `client.py`, `retrieve.py`, `utils.py`, `requirements.txt` | Security and crash fix | -| litellm cost-map environment default | `utils.py` | Integration nit | +| Accounted model calls | Private retry helpers called the provider and returned an empty string after terminal failure | Application runtime records usage, reserves cost, traces calls, bounds retries, and raises classified failures | +| Valid structured output | JSON consumers accepted unvalidated text and used a corrupting cleanup parser | Eleven structured call sites require JSON and classify malformed responses | +| Bounded document batches | Groups overlapped and could exceed the model’s token limit | Groups are disjoint, enforce the cap, and split oversized pages without losing page markers | +| Stable continuation trees | Later batches blindly appended repeated or out-of-order ToC entries | Continuations deduplicate exact repeats and sort entries into safe page order | +| Academic-paper coverage | A valid ToC ending in the first half of a paper scored zero | Every placed title is checked regardless of document coverage | +| Recoverable subdivision | A failed optional subdivision discarded the whole document | The verified parent node remains when subdivision fails | +| Correct ToC routing | A detected ToC without printed page numbers was discarded | The runtime preserves its structure and locates pages through the pageless path | +| Safe page repair | Missing pages produced empty windows, `KeyError`, or malformed-response crashes | Search bounds use real page limits and validate every repair response | +| Bounded recursion | Large-node subdivision could recurse without a cap and delete valid children | Recursion stops at depth 16 and retains children within the original node range | +| Artifact summaries | Each node received one independent summary | Children summarize first, then each node receives direct and subtree summaries | +| Maintained PDF parsing | `PyPDF2` remained vulnerable and blank pages could yield `None` | `pypdf` replaces `PyPDF2`, and blank pages produce empty text | +| Deterministic startup | LiteLLM could fetch its model-cost map during import | The child uses LiteLLM’s bundled cost map | + +## Route model calls through application policy + +The patch removes upstream `llm_completion`, `llm_acompletion`, and `extract_json` implementations from `pageindex/utils.py`. It imports their replacements from `vectorless_rag.pageindex_runtime`. + +The upstream helpers retried every exception up to 10 times and returned `""` after terminal failure. That behavior converted provider failures into malformed trees. It also bypassed durable cost and attempt accounting. + +The application runtime adds these controls: + +- A usage-ledger row and cost reservation for every call +- Langfuse generation traces linked to the ingestion attempt +- Configured output-token and timeout limits +- Bounded retries with retryable and terminal failure classes +- Sanitized diagnostics such as `pageindex_llm_retry_exhausted`, `pageindex_content_filter`, `pageindex_empty_response`, and `pageindex_invalid_json` + +The replacement JSON parser also avoids two corrupting transformations in upstream code. Upstream replaced every `None` substring with `null` and removed every newline before parsing. Those substitutions could change titles or summaries that contained the same characters. + +Eleven PageIndex functions now pass `expect_json=True` when their result must be an object: + +- `check_title_appearance` +- `check_title_appearance_in_start` +- `toc_detector_single_page` +- `check_if_toc_extraction_is_complete` +- `check_if_toc_transformation_is_complete` +- `detect_page_index` +- `toc_index_extractor` +- `add_page_number_to_toc` +- `generate_toc_continue` +- `generate_toc_init` +- `single_toc_item_index_fixer` + +Malformed structured output now fails at the provider boundary instead of reaching later tree code as a partial dictionary. + +## Keep document batches within model limits + +`page_list_to_group_text` prepares token-bounded text groups for ToC generation. The patch rewrites this function because upstream grouping violated the boundary it was meant to enforce. + +The patched grouping algorithm makes four guarantees: + +- **No overlap**: a page appears in one group, so continuation batches do not regenerate the previous group’s final sections +- **Hard token cap**: every emitted group remains at or below `max_tokens` +- **Marker preservation**: oversized pages split by binary search, and each chunk retains its `` markers +- **No empty prompts**: an empty result raises before a provider call + +A final-result adapter cannot implement these guarantees because grouping happens before PageIndex sends prompts to the model. + +## Merge continuation results without corrupting ranges + +Long documents use `generate_toc_init` for the first group and `generate_toc_continue` for later groups. A continuation may repeat a section from its context with the original page, a shifted page, or a page that validation changed to `None`. Upstream appended every entry. + +The patch adds `merge_continuation_entries`. It removes exact repeats using normalized `(structure, title, physical_index)` keys, retains nonidentical entries, and applies a stable page-order sort with `None` entries last. + +Ordering matters because downstream `post_processing` derives each section’s end from the next section’s start. An out-of-order duplicate can create a reversed page range and invalidate the document. + +Patch `+vr4` initially rejected conflicting repeats. Document `2103.05633` showed why that rule was too strict: a continuation can legitimately repeat a section whose page marker is absent from the later chunk. Patch `+vr5` adopted the current deduplicate-and-sort behavior. + +## Preserve valid academic-paper structures -## Model-call plumbing (`utils.py`) +Several upstream assumptions treated recoverable academic-paper layouts as terminal failures. The patch changes those cases without relaxing title-level validation. -The patch deletes upstream's `llm_completion`, `llm_acompletion`, and `extract_json` and imports all three from `vectorless_rag.pageindex_runtime` instead. This is the integration seam that makes the vendored code a citizen of the pipeline. +### Verify front-loaded tables of contents -Upstream's helpers retried every exception up to 10 times and then returned an empty string, so a persistently failing provider produced silent garbage trees instead of a failed ingestion. The app runtime replaces that with: per-call usage-ledger rows and cost reservations, Langfuse generation traces, bounded retries with classified failure codes (`pageindex_llm_retry_exhausted`, `pageindex_content_filter`, `pageindex_empty_response`, and the rest of `pageindex_failures.py`), a per-call timeout, and a configured output-token cap. Kind: missing feature (accounting, observability, budget enforcement) plus a bug fix (failures now raise instead of returning `""`). +Upstream `verify_toc` returned zero without checking any title when the last placed entry occurred in the document’s first half. References and appendices make that layout common in academic papers. -Upstream's `extract_json` also mangled payloads before parsing: it replaced the substring `None` with `null` anywhere in the text and stripped all newlines, corrupting any title or summary containing them. The runtime's parser handles fenced JSON strictly and tolerates the raw control characters real PDF text puts into model output. Kind: bug fix. +Six of the tracked corpus’s 104 documents failed this gate. Document `2408.00724` had 22 pages and 11 correctly placed entries ending on page 10, followed by acknowledgments and a bibliography. The patch rejects only a ToC where no entry has a page. It still verifies every placed title and preserves the existing accuracy thresholds. -## Strict JSON on structured calls (`page_index.py`) +### Keep verified nodes when optional subdivision fails -Eleven call sites that expect a JSON object now pass `expect_json=True`: `check_title_appearance`, `check_title_appearance_in_start`, `toc_detector_single_page`, `check_if_toc_extraction_is_complete`, `check_if_toc_transformation_is_complete`, `detect_page_index`, `toc_index_extractor`, `add_page_number_to_toc`, `generate_toc_continue`, `generate_toc_init`, and `single_toc_item_index_fixer`. The runtime validates the completion as JSON at the call boundary and classifies malformed output as `pageindex_invalid_json` instead of letting a half-parsed dict propagate. Kind: missing feature (failure classification at the source). +`process_large_node_recursively` subdivides a verified node so later retrieval receives finer sections. Upstream propagated a nested subdivision failure and discarded the entire document. -## Page grouping rewrite (`page_index.py`) +Document `2305.16264` demonstrated this failure on a verified 50-page tree. The patch keeps the original node when nested processing fails. The result still has a valid range and summary, even though it has less structural detail. -`page_list_to_group_text` splits the document into token-bounded groups for ToC generation. The rewrite fixes two upstream defects and adds one missing capability: +### Route each ToC shape to the matching algorithm -- **Disjoint groups**: upstream overlapped each group with the previous group's last page. Overlap fed the same page to two generation batches, which made continuation batches re-list the overlap page's sections and inflate the ToC with duplicates. The rewrite drops `overlap_page` entirely. -- **Hard token cap**: upstream targeted an averaged group size rather than the cap, and a single page larger than `max_tokens` shipped a group that overflowed the model context. The rewrite enforces `max_tokens` per group. -- **Oversized-page splitting** (missing feature): a page whose text alone exceeds `max_tokens` is now split by binary search into chunks that each stay under the cap while preserving the `` markers the model needs for page attribution. Upstream had no path for this. -- **Empty-group guard**: grouping that produces an empty group raises instead of sending a blank prompt. +Upstream used the ToC path only when the detected ToC included printed page numbers. It discarded a real pageless ToC and regenerated structure from document text. -## Continuation merge (`page_index.py`) +The patch sends that case to `process_toc_no_page_numbers`, which preserves the detected headings and searches for their pages. If a numbered ToC yields no reliable printed-to-physical offset, `process_toc_with_page_numbers` falls back to the same path instead of passing `None` into offset arithmetic. -ToC generation for long documents runs in batches: `generate_toc_init` for the first group, `generate_toc_continue` for each following group. Continuation batches habitually re-list sections they were shown as context, sometimes with a shifted page number and sometimes with a page the chunk validator nullifies to `None`. Upstream concatenated batches blindly (`extend`), duplicating sections. +### Repair missing page numbers safely -The patch adds `merge_continuation_entries` and calls it from `process_no_toc`. It drops exact re-listings, keyed on case-folded and whitespace-normalized `(structure, title, physical_index)`, keeps everything else in upstream's keep-both spirit, and stable-sorts the result by page with `None` entries last. The sort matters because downstream `post_processing` derives each entry's `end_index` from the next entry's page; an out-of-order re-listing would manufacture a reversed range and fail the whole document at artifact validation. `tree_parser` filters the `None` entries before `post_processing`, so their tail position is safe. Kind: bug fix. +`process_none_page_numbers` asks the model to place headings that lack a physical page. The patch: -Revision history: `+vr4` shipped a stricter merge that raised `ValueError` on any conflicting re-listing. In production that failed document `2103.05633` deterministically, because disjoint grouping means a re-listed section's marker is absent from the later chunk, the validator nullifies its page to `None`, and `None` never equals the original page. `+vr5` replaced the raise with the dedupe-and-sort behavior above. +- Treats an explicit `physical_index=None` as missing +- Uses the document’s real first and last page as default search bounds +- Skips provider calls for empty search windows +- Removes optional `page` fields without raising `KeyError` +- Validates response shape before reading the result -## Coverage no longer gates verification (`page_index.py`) +These checks turn malformed repair data into controlled behavior instead of secondary exceptions. -`verify_toc` returned `0, []` without checking a single title whenever the last placed entry -fell in the document's first half, on the assumption that a table of contents spans its -document. Academic papers break that assumption as a matter of course: references and -appendices routinely fill more than half the pages, so a complete and correct table of -contents for the body scores zero. `meta_processor` reads that zero, finds no further -fallback mode, and raises `Processing failed`, discarding the document. +## Retain large-node structure -Six of the tracked corpus's 104 documents failed this way. `2408.00724` is representative: -22 pages, 11 correctly placed entries ending at page 10, with acknowledgments on page 11 and -the bibliography from page 12 on. The patch keeps the early return only for a table of -contents where nothing was placed at all. Coverage is not correctness: every title is still -checked individually and the accuracy thresholds are untouched. Kind: bug fix. +The patch repairs four defects in recursive subdivision: -## Subdivision failure degrades instead of propagating (`page_index.py`) +- **Recursion cap**: subdivision stops at depth 16 +- **Correct trigger**: the true inclusive page count or token count can trigger subdivision +- **Range filtering**: sections outside the parent’s original range are discarded +- **Child retention**: filtering compares children with the parent’s original end, not the rewound end used to insert the first subsection -`process_large_node_recursively` splits an oversized node into finer ones. When the nested -`meta_processor` raised, the exception propagated out of the whole build, so a document whose -own tree verified at 100% was discarded because one section refused to split - which is what -happened to `2305.16264`, a 50-page document that now yields 67 nodes. Subdividing is an -enhancement over a node that already carries a verified range and summary, so the patch -returns the node unsubdivided instead, exactly as the existing `depth >= 16` guard does. -Kind: bug fix. +The last defect silently removed every child after a large node was subdivided. A later edge case also removed a first subsection that began on the parent’s first page. The provider-free regression suite covers both range semantics. -## Pageless-ToC routing in `tree_parser` (`page_index.py`) +After the final repair, rebuilding `2103.05633` increased its tree from 37 nodes at depth 3 to 45 nodes at depth 4. The additional nodes are sections the old filter deleted. -Upstream routed a document to the ToC-driven path only when the detected ToC also printed page numbers. A document with a real ToC but no page numbers fell into the no-ToC path, which throws the detected ToC away and regenerates structure from raw text. The patch adds the middle branch: detected ToC without page numbers now routes to `process_toc_no_page_numbers`, which keeps the ToC's structure and locates pages by search. Kind: bug fix (upstream discarded signal it had already paid to detect). +## Build direct and subtree summaries -## Offset fallback in `process_toc_with_page_numbers` (`page_index.py`) +The v2 artifact distinguishes a summary generated from node text alone from one explicitly composed with descendant summaries. Upstream summarized every node independently and emitted one `summary`. Its tree builder temporarily added node text for summarization, then removed it when `if_add_node_text=no`. -`calculate_page_offset` returns `None` when no printed page number can be matched to a physical page. Upstream passed that `None` straight into `add_page_offset_to_toc_json` and crashed. The patch falls back to `process_toc_no_page_numbers` for the whole document instead. Kind: crash fix. +The patched summarizer walks the tree from children to parents. It writes: -## `process_none_page_numbers` robustness (`page_index.py`) +- **`direct_summary`**: a summary prompted with the node’s page-range text and no child-summary context +- **`subtree_summary`**: a summary prompted with the same text plus child summaries +- **`summary`**: the subtree summary retained for PageIndex compatibility -This function repairs ToC entries that arrived without a physical page by asking the model to place them within a neighbor-bounded page window. The patch fixes five defects in it: +The patched helper also tolerates a missing `text` field, which makes it safe outside the tree builder’s temporary-text lifecycle. Bottom-up order is part of artifact semantics because parent summaries cannot incorporate descendant summaries until child summaries exist. -- Detects entries whose `physical_index` is explicitly `None`, not only entries missing the key. -- Defaults the search window to the document's real bounds (`start_index` to `start_index + len(page_list) - 1`); upstream defaulted to `0` and `-1`, which produced empty or nonsensical windows for first and last entries. -- Skips the model call when the computed window contains no pages. -- Uses `pop('page', None)` where upstream's `del` raised `KeyError` on entries without a `page` field. -- Validates the model response's shape before indexing into it; upstream crashed on any malformed reply. +## Use a maintained PDF parser -Kind: crash fixes. +Patch `+vr7` replaces `PyPDF2` with [`pypdf 6.14.2`](https://pypdf.readthedocs.io/en/latest/meta/history.html) in `pageindex/client.py`, `pageindex/retrieve.py`, `pageindex/utils.py`, and `requirements.txt`. This removes the vulnerable direct dependency from both the application and vendored runtime. -## Large-node recursion repair (`page_index.py`) +`pypdf` can return `None` for a page without extractable text. The patch normalizes that result to `""` in every PageIndex extraction path, so blank pages remain valid pages. -`process_large_node_recursively` subdivides nodes too large to summarize in one pass. The patch fixes four defects: +Artifact metadata records the parser that produced page evidence: -- **Recursion cap**: `depth >= 16` returns the node as-is. Upstream had no bound, and a model that kept emitting subdividable children recursed without limit. -- **Subdivision trigger**: upstream required the node to exceed both the page limit and the token limit, and computed pages as `end - start` (off by one). Token-dense short nodes and long sparse nodes both escaped subdivision. The patch uses the true page count and triggers on either limit. -- **In-range filtering**: items the model returns with a page outside the node's own span are discarded before tree building. -- **Child retention**: after subdivision, upstream code rewound `node['end_index']` to the first subsection's start and then filtered children against that rewound end. Because every child was built against the original end, the filter deleted every child of every large node, silently flattening exactly the documents that needed subdivision most. The patch captures `original_end` before the rewind and keeps forward-running children (the artifact layer widens ancestors over them), while still dropping children that start before the node (they would annex a preceding sibling) and children spanning the node's original range (the real recursion hazard). During `+vr5` review, an intermediate version of this filter still lost a first subsection that starts on the node's own first page; comparing against `original_end` fixed that too, and `scripts/test_pageindex_patch.py` pins it. +- Multi-page PageIndex artifacts record `pypdf` and its installed version +- The application’s single-page shortcut records `pymupdf` and its installed version -Kind: bug fixes. Rebuilding document `2103.05633` after the final fix raised its tree from 37 nodes at depth 3 to 45 nodes at depth 4; the difference is structure the old filter was deleting. +Parser identity contributes to the artifact configuration hash. -## Bottom-up summaries (`utils.py`) +## Keep child startup deterministic -The v2 artifact distinguishes a node's own content from its subtree. Upstream summarized every node independently and in parallel from `node['text']`, produced a single `summary` field, and crashed with `KeyError` when node text was disabled (`--if-add-node-text no`, which is how the pipeline runs it). +The patch sets `LITELLM_LOCAL_MODEL_COST_MAP=True` before importing LiteLLM. The child process therefore reads the bundled model-cost table instead of fetching a remote copy at import time. -The patched `generate_summaries_for_structure` walks each tree bottom-up: it summarizes children first, then produces both a `direct_summary` (the node alone) and a `subtree_summary` (the node with its children's summaries folded into the prompt), storing the subtree summary as `summary` for compatibility. `generate_node_summary` accepts the child summaries and tolerates missing text. Kind: missing feature; the artifact schema's `direct_summary` and `subtree_summary` fields depend on it. +This setting does not replace application cost configuration. It removes an unrelated network dependency from child startup. -## `pypdf` migration and empty-page guards (`utils.py`) +## Why the project carries a patch instead of a fork -The patch replaces unmaintained `PyPDF2` with maintained `pypdf` in `client.py`, `retrieve.py`, `utils.py`, and `requirements.txt`. Multi-page artifacts generated through PageIndex record `pypdf` and its installed version in artifact identity; the single-page shortcut records the PyMuPDF parser it actually uses. The parser’s `extract_text()` can return `None` for pages without extractable text, and upstream concatenated that `None` into a string in three places (`extract_text_from_pdf`, `get_text_of_pages`, `get_page_tokens`). Each site now appends `page.extract_text() or ""`. Kind: security and crash fix. +A source patch keeps the local contract narrow and auditable. It also preserves a direct comparison with the pinned upstream implementation. -## litellm cost-map default (`utils.py`) +The current approach has four properties: -`LITELLM_LOCAL_MODEL_COST_MAP=True` is set before the `litellm` import so the child process uses the library's bundled model-cost table instead of fetching the remote one at import time. Kind: integration nit; keeps the isolated child deterministic and network-quiet at startup. +- **Exact baseline**: every image starts from one immutable upstream commit +- **Visible divergence**: `apps/api/patches/pageindex-runtime.patch` contains the complete source difference +- **Early drift detection**: `git apply --check` fails before the image can include a partial patch +- **Independent retirement**: a future upstream revision can replace individual hunks after their behavior and artifact impact are tested -## Verifying the patch +A long-lived fork would duplicate release, dependency, and merge maintenance without removing the need to pin and verify the exact code used in an experiment. -`apps/api/scripts/test_pageindex_patch.py` runs provider-free regression checks against a patched checkout: marker-preserving page grouping, continuation merge semantics (dedupe, keep-both, page ordering, `None` placement), the large-node first-subsection case, pageless-ToC routing, and bottom-up summary ordering. Run it against any checkout: +## How patch identity prevents stale artifact reuse + +Three values bind a built artifact to the runtime that produced it: + +- **`pageindex_version`**: `190f8b378be58199ca993566a9214dba72089c54+vr7` combines the upstream commit with the local patch revision +- **`patch_sha256`**: `PageIndexAdapter` hashes the complete patch file at ingestion time +- **`configuration_hash` and `recipe_hash`**: both include the version, patch digest, model configuration, parser identity, and other result-affecting options + +Any behavior change requires a `+vrN` bump. The patch digest changes even if a maintainer forgets the bump, but the explicit suffix remains required because operators and reports use it as the readable runtime version. + +An artifact built under a different patch cannot satisfy the current recipe key. Re-indexing creates a new immutable artifact and activation record instead of overwriting the old result. + +## File inventory + +The patch touches these upstream surfaces: + +| File | Patched responsibility | +| --- | --- | +| `pageindex/client.py` | Use `pypdf` when PageIndex stores per-page content | +| `pageindex/page_index.py` | Validate structured calls, group pages, merge continuations, route ToCs, repair page indices, verify coverage, and subdivide nodes | +| `pageindex/retrieve.py` | Use `pypdf` when retrieval reads source pages | +| `pageindex/utils.py` | Route model calls through the application, parse JSON, use `pypdf`, normalize blank pages, and build bottom-up summaries | +| `requirements.txt` | Replace `PyPDF2` with the pinned `pypdf` version | + +The application-side integration lives in: + +| File | Responsibility | +| --- | --- | +| `apps/api/scripts/install-pageindex.sh` | Clone the pinned commit and apply the patch | +| `apps/api/scripts/test_pageindex_patch.py` | Run provider-free tests against the patched checkout | +| `apps/api/src/vectorless_rag/pageindex_runtime.py` | Enforce provider accounting, tracing, limits, retries, and structured output | +| `apps/api/src/vectorless_rag/run_pageindex.py` | Establish usage ownership and emit sanitized failure reports | +| `apps/api/src/vectorless_rag/pageindex_adapter.py` | Launch the child, hash the patch, and build the versioned artifact | + +## How to verify the patch + +The API image build verifies both patch application and behavior. `install-pageindex.sh` fails if any hunk no longer matches the pinned commit. `test_pageindex_patch.py` then checks: + +- Complete `PyPDF2` removal and blank-page parsing through `pypdf` +- Token-capped, marker-preserving page grouping +- Continuation deduplication, ordering, and `None` placement +- Retention of a first subsection on the parent’s first page +- Preservation of a verified node after subdivision failure +- Routing for a ToC without printed page numbers +- Title verification for a front-loaded ToC +- Rejection of a ToC with no placed entries +- Child-before-parent summary generation + +Run the provider-free regression suite against a patched checkout: ```bash -uv --project apps/api run apps/api/scripts/test_pageindex_patch.py /path/to/patched/pageindex +uv --project apps/api run apps/api/scripts/test_pageindex_patch.py path_to_pageindex_checkout ``` -Inside the api container the installed copy lives at `/opt/pageindex`, and `install-pageindex.sh` fails the image build if the patch stops applying cleanly to the pinned revision. +Build the production API image to exercise the clone, patch, dependency installation, and regression checks together: + +```bash +make docker-build-api +``` + +Passing unit tests without the image build is not sufficient. Only the image build proves that the patch still applies to a clean checkout of the pinned upstream commit. + +## What the patch does not change + +The patch does not own every PageIndex-related behavior: + +- The pinned upstream revision already contains `_SYSTEM_HARDENING`, `_secure_doc_text`, `_validate_physical_indices`, and `` prompt framing +- Retrieval routing and evidence-budget policy remain in the Vectorless RAG application +- The patch does not add optical character recognition (OCR) or support scanned PDFs +- Provider-backed ingestion still incurs model calls and cost +- The single-page shortcut remains application code and does not run the PageIndex tree builder + +These boundaries matter during an upstream upgrade. A maintainer should not attribute upstream prompt hardening or application retrieval policy to the runtime patch. + +## How to update or retire patch hunks + +Treat an upstream PageIndex revision change as an artifact-producing behavior change: + +1. Update the pinned commit in `apps/api/scripts/install-pageindex.sh`. +2. Compare every local hunk with the new upstream behavior. +3. Remove only the hunks whose required behavior upstream now implements and tests. +4. Regenerate `apps/api/patches/pageindex-runtime.patch` against a clean checkout. +5. Bump the `+vrN` suffix in `Settings.pageindex_version`. +6. Run the provider-free patch suite and the production API image build. +7. Rebuild affected documents into new immutable artifacts. + +Do not edit the installed `/opt/pageindex` tree or patch a running container. Those changes bypass source review, patch hashing, and artifact identity.