feat: complete PageIndex v2 evaluation and hardening - #5
Merged
Conversation
Four of six strata shared one hardcoded prompt per stratum, repeated 20 times, carrying no information identifying the target passage: "what does the cited source passage make" cited nothing, "which paper supports the described finding" described nothing. Eighty of 120 cases were undetermined as posed, so any retrieval system scored at chance and four strata measured nothing. Excerpt selection compounded it. The sentence regex ran over each page flattened to a single line and kept only the first match, so whichever figure caption or running header sat at the top became that page's sole candidate, and reference answers quoted axis ticks, venue lines, author blocks, and pseudocode. Select every prose sentence per page instead, rejecting page furniture: captions, venue and copyright lines, reference entries, pseudocode, math and digit density, appendix prompt exemplars, sampled model output and quoted narrative, footnote and section-heading run-ins, inline symbol definitions, contact details, and sentences truncated mid-reference. All 537 documents keep at least 25 clean candidates. Sentences broken across a line by hyphenation are now discarded rather than repaired. The previous rule joined "dif- ferent" correctly but corrupted every compound that fell at a line break, shipping "finetuning" and "LLMbased" into reference answers that exact-phrase prompts then told retrieval to find. metadata_constrained_discovery constrained on a submission-date window derived from the arXiv ID. Document.submitted_date is model-extracted from each PDF's first page and the corpus stores latest versions, so the stamp is the revision date: 11 of 20 targets fell outside their own window and were filtered out of the candidate set, making those cases unscoreable while still labelled answerable. Constrain on a deterministic 40-document identifier subset, the only catalog constraint verifiable offline. Genuine paraphrase cannot be produced by string manipulation, so the two paraphrase strata carry authored text in pageindex-v2-authored.json, bound to its source excerpts by SHA-256. A missing, malformed, or stale entry routes every affected case to an authoring queue and aborts the build rather than pairing a paraphrase with a passage it no longer describes. Rebuilds now carry forward completed reviews whose case content is unchanged, so authoring and reviewing can interleave. Add scripts/review_evaluation.py to walk cases one at a time. It reseals the review source hash, and the stable item ID when an edit changes the prompt, then refuses any edit the suite validator would reject: empty fields, colliding item IDs, or a discovery prompt naming its target. It declines to write when the dataset changed underneath the session, so a second reviewer or a rebuild cannot be silently overwritten. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
make evaluation-run only wrote a plan and an empty scores.jsonl, so the suite had no path from a reviewed dataset to a report. Add scripts/execute_evaluation.py to run the plan and append scores. Execution drives GraphRunner in process rather than the HTTP surface. Each experiment kind needs per-trial control of the forced route, the oracle document set, or the evidence budget, and none of those are fields on ChatRequest; they are GraphRunOptions, which only an in-process caller can supply. route leaves the router free so its choice is measured; corpus_routing and forced_route_end_to_end pin the route so retrieval is measured without routing noise in front of it; oracle_document_retrieval also supplies the targets so node selection within a known document is isolated. Citations carry the document UUID while the suite is keyed by arXiv ID, so scoring resolves one to the other from the ready, indexed document set and refuses to start when any target is absent. A missing index would otherwise score as a retrieval miss and quietly understate the system. Route, document, support, and citation measurements are computed here. Claim recall is not: deciding whether an answer asserts a claim means reading both, so end-to-end trials are appended to judge-queue.jsonl for separate scoring. scores.jsonl is append-only and keyed by trial ID, so an interrupted run resumes without repeating work, --limit caps a single invocation, and --dry-run resolves documents and prints the plan without a provider call. Provider failures record included: false with a failure code rather than aborting, so one bad trial does not discard the rest of the run. Repetitions become a recorded parameter between one and three rather than a fixed three, defaulting to one. Repetitions estimate within-configuration variance; the across-item bootstrap works without them, and three passes tripled provider cost before a single pass had been read. The manifest carries the count so a one-pass run is never mistaken for a repeated one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The evaluation runs 1,440 trials through the query path, where flash costs $0.14/1M input against pro's $0.435 and $0.28/1M output against $0.87, at the same 1M context and with JSON output and tool calls still supported. pageindexModel stays on pro deliberately. It builds the index rather than serving queries, so changing it would invalidate any index already built and the frozen index hash the evaluation pins. Fill in the flash list prices, which run_evaluation's preflight requires in full and which were previously blank. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A local .env drifts as the example gains keys, and the gap only surfaces when something silently reads a default nobody chose. This one was missing 17 of 59 keys, including EVIDENCE_TOKEN_LIMIT and REQUEST_DEADLINE_SECONDS, which the evaluation depends on. sync_env.py treats the example as the schema and rewrites .env in its order and comments, substituting values already configured locally. Because .env holds live credentials, an existing value is never replaced unless --adopt names its key, never printed, and never dropped: a key the example does not list is preserved in a trailing section, since the script cannot distinguish a stale key from a deliberate local one. It verifies the rendered file round-trips every prior value before replacing anything, backs up at the original permissions, and reports secret-looking values as <set>/<empty>. --check writes nothing and exits non-zero on structural drift, so it can gate a setup step. Also ignore .env.bak. The backup carries the same credentials as .env and was untracked but unignored, so any git add -A would have committed it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Serving and indexing run on different models, DEEPSEEK_MODEL and PAGEINDEX_MODEL, but a single price triple covered both. While both were deepseek-v4-pro that was consistent; moving serving to flash made every pilot cost report understate ingestion by the ratio between the two, and silently, since a cost report offers no signal that its rate was wrong. Every ModelCall already records the model that produced it, so pricing resolves per call. Pricing gains an optional per-model override map and an environment_prefix so a missing rate is reported against the variable that actually supplies it, and _price_usage groups calls by model and prices each group at its own rate. With no overrides this is one group and reduces exactly to the previous behaviour. Add the PAGEINDEX_* price triple, and have each report pass the rates for the path it describes: pilot-report covers ingestion, query-cost-report covers serving. Each carries the other model as an override so a report that ever sees a mixed set still prices it correctly rather than applying one rate to everything. The pilot's experiment price snapshot records the indexing rates for the same reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
make evaluation-run requires INDEX_HASH as a 64-character digest, and the documentation called it the frozen index sha256, but nothing produced one. The pilot passes its configuration hash in that slot, which describes the settings rather than the index they built. An index is not one artifact: it is a PageIndex tree per document, each with its own artifact_sha256. freeze_index.py reduces that set to a single value ordered by arXiv ID with NUL-separated pairs, the construction the corpus manifest already uses for PDFs. It digests every ready document with an active index version, not only the documents cases target, because corpus routing searches the whole catalog: ingesting or reindexing an unrelated document changes what the router chooses between, and so changes the measurement. Results carrying different digests are not comparable. It also fails when a document the corpus manifest targets is absent from the index, since an unindexed target scores as a retrieval miss and would understate the system rather than report the setup error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PageIndex is vendored at /opt/pageindex and never installed into the environment, so it resolves only through PYTHONPATH, which the image sets. The child process environment is rebuilt from scratch rather than inherited and passed through only locale, TLS, and proxy variables, so the child ran without PYTHONPATH and died on `from pageindex import *` before reaching the handler that writes a failure report. With no report to read, the adapter fell back to pageindex_child_failed, which says only that the runtime failed. Child stdout and stderr both go to DEVNULL, deliberately, since a traceback can carry document text, so the actual ModuleNotFoundError was discarded and every ingestion attempt failed identically with no way to see why. Set PYTHONPATH explicitly from pageindex_source_dir rather than passing the ambient value through, so the child's import path stays deterministic and does not depend on how the parent happened to be started. Verified against a real document: attempts now reach PageIndex generation and complete 20 provider calls where they previously made none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… JSON PDF text layers carry raw control bytes: a lambda in one corpus paper extracts as 0x15. PageIndex passes page text to the model, the model quotes that text back inside its JSON strings, and strict parsing rejects the raw byte. The response is structurally valid JSON; only the rule about literal control characters in strings rejected it. Because the byte comes from the page, the failure is deterministic. Every retry regenerated the same content and failed identically, so an affected document consumed its whole attempt budget and then failed. Raising PAGEINDEX_LLM_MAX_ATTEMPTS from 3 to 8 changed nothing except spending 5 more provider calls per document, which is how this was confirmed rather than assumed. Parse the repair path with strict=False. Strict parsing is still tried first and genuinely malformed JSON is still rejected; only this rule is relaxed, and only after the existing repair has already run. Measured on 2010.11148: before the fix, 8 generation calls failed on JSON and the attempt died after 29 calls. After, 67 consecutive calls completed with no JSON failure.
Importing litellm in its default DEV mode runs load_dotenv() during pytest collection, injecting the repo .env into os.environ for the whole process; Settings separately reads env_file=".env" relative to the working directory. Together they let local overrides masquerade as code defaults in any test constructing Settings(). Pin LITELLM_MODE=PRODUCTION unconditionally and chdir every test to tmp_path via a suite-wide autouse fixture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real PageIndex output declares a section's range as its heading's pages while subsections run past it (21 forward violations, 0 backward, across sampled documents), re-lists sections across continuation batches with shifted or nullified pages, and emits subdivision children built against a node end that the code then rewinds. The v2 hardening rejected all three, so no document could index while the flat v1 pipeline had tolerated them. Artifact layer: widen subtree_page_range forward over overflowing children instead of rejecting; keep rejecting children that start before their parent. Record widened_nodes/widened_pages in ArtifactQuality (defaulted, extra="ignore" so older readers survive newer counters) and return a NormalizedTree dataclass from normalize_tree_v2. Runtime patch (+vr5): merge continuation batches by dropping exact re-listings and sorting by page so post_processing's next-entry rule stays monotonic; keep forward-overflowing subdivision children and test the recursion guard against the pre-rewind node end (the old filter deleted every child of every large node, and later a first subsection starting on the node's first page). Pass the full pageindex_version as generator_commit so a +vrN bump alone shifts recipe identity. All four previously failing documents now index end to end; rebuilding 2103.05633 recovered 8 nodes and one tree level the old filter deleted. docs/pageindex-runtime-patch.md documents every patch change and why. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The executor had never run: it passed LangGraph's checkpointer a URL built by replacing "+asyncpg", a substring DATABASE_URL does not contain, so it died before the first trial. --dry-run returns earlier, so a dry run could not surface it. api.py already had the correct transform and usage.py a third copy; all three now share driver_free_url in db.py. The run directory also existed only on the host while the target executed in the api container, where evaluation/run/ is excluded from the image. The api service now mounts apps/api/evaluation, and evaluation-execute runs as the host user so appends land as that user rather than as uid 10001. The mount also stops evaluation-freeze reading a corpus manifest baked at image build time. Unanswerable cases wrote one refusal bit into document_recall, support_recall and citation_precision, and the report averages each metric over a group that mixes strata, so every published retrieval number blended 100 ratios with 20 refusals. Those three are now unset for unanswerable cases, which the report already filters, and refusal correctness gets its own metric. route_correct now measures the route for every case, as its own comment said it should. Also: enforce the same request deadline the HTTP surface does, so a run that production returns as 504 is not scored as a clean success; terminalize the QueryRun, which otherwise stayed 'running' forever and made query-cost-report treat the window as incomplete; flush Observability so a paid run leaves its traces; write the judge entry before the resume ledger, because the reverse order loses claim-recall input for a trial already marked done; and stop catching ValueError, which turned dataset defects into excluded trials. report_evaluation counted failure codes among included scores, but a failure is exactly what makes a score excluded, so the counter could only ever report "none". Root scripts/ was outside every lint target and had drifted to ruff's default line length; it gets a ruff.toml of its own rather than the API config, which would reorder the API package's first-party imports as third-party. docs/pageindex-runtime-patch.md lacked its content declarations and its index entry, which made make api-test red.
…s nearest Pricing.for_model returned the base rates for any model it did not recognise, so a call recorded under a model the configuration has since moved off was priced confidently and wrongly, with missing_price_configuration reporting nothing. This is live: model_calls holds 53 calls under a bare deepseek-v4-pro from when that was the serving model, and DEEPSEEK_MODEL is now deepseek-v4-flash. Those calls matched neither the indexing override nor the serving model, so query-cost-report billed them at flash rates, roughly a third of their real cost, and build_pilot_report priced the same rows differently again. Pricing now names the model its base rates describe; an unresolved model is reported as PRICE_FOR_MODEL:<name> and suppresses the cost rather than taking a neighbouring rate. Callers that leave the model unset keep the single-rate behaviour, which is what the pilot's own snapshot wants. freeze_index collapsed rows into a dict keyed by arxiv_id, which is nullable and not unique. Two ready documents sharing an identifier dropped one digest and still produced a well-formed hash, so a corpus that had lost a document would freeze indistinguishably from a clean smaller one. It now refuses instead. The ArtifactQuality comment claimed the opposite of what the change does: a reader from the previous release still forbids extras, so the compatibility runs forwards, not backwards, and rolling back strands every manifest written since.
The docstring promises a configured value is never dropped, and two shapes broke it. A repeated key collapsed to whichever assignment came last. A value spanning several lines matched only on its first, so rendering emitted that line and discarded the rest - a PEM credential would come back as its BEGIN marker alone. Neither has a safe guess, and both destroy live credentials, so parsing refuses them with the offending line rather than resolving them. Masking keyed on the variable name alone, so a credential carried inside a value under a name like DATABASE_URL was printed in cleartext, which make env-check would put in a CI log. Values with URL userinfo are now masked on shape.
A 1,440-trial provider run had no ceiling of any kind. The machinery exists - model call reservations resolve their owning run to an experiment and refuse the next call once maximum_cost_usd is reached - but nothing armed it for the evaluation: run_evaluation minted an experiment UUID without creating the row, and no experiment_memberships row ever carried a query_run_id, so the reservation's owner lookup found nothing and the cap could not fire. The pilot gets this right; the evaluation path had simply never been wired to it. evaluation-run now requires MAX_COST and records it in the frozen manifest. execute_evaluation materialises the experiment before the first trial, because the plan is built on the host, which cannot reach the database, and enrolls each trial's run so the reservation can find its ceiling. Resuming leaves an existing experiment alone rather than rewriting its limit, so a resumed run cannot raise the number the plan was frozen with. The manifest field carries no default: an unbounded plan is exactly the failure this prevents, so there is nothing to fall through to.
…ating its rules A run killed mid-append leaves a torn final line in scores.jsonl, and reading it raised JSONDecodeError, so a 1,440-trial run that had already paid for hundreds of trials could not resume without hand repair. Only the last line can be partial - every earlier one was followed by a completed write - so that one is discarded and its trial re-runs, while a torn line anywhere else still refuses, because that is corruption rather than truncation. The torn line is removed from the file too: leaving it would let the next append run onto it and turn a recoverable tail into permanent mid-file damage. The interactive reviewer carried its own copies of the discovery strata and the completed-review statuses, which is exactly the divergence its docstring says it exists to prevent: tightening the leak rule in validate_suite would have left the reviewer accepting edits the validator later refuses. Both now come from evaluation_suite, along with the usable-status set the validator applies. Also: freeze_index reported missing_from_index on the path where it is always empty; _indexing_model_names returned the same name twice whenever the configured model has no litellm/ prefix, which is the shipped default; --limit documented 0 as a maximum when it means unlimited; and the conftest comment gave a reason for subprocess safety that only one of its seven call sites satisfies - they are safe because they use absolute paths, which is a different guarantee.
…der call The executor had no tests at all, and it is about to drive 1,440 paid trials. These pin the parts that need no database: that resume discards a torn final line and only a torn final line, that the discarded line leaves the ledger rewritten rather than intact, and that each experiment kind produces the overrides its measurement depends on - a route trial leaving the router free is the whole point of that arm.
Two of the eight documents that would not index were failing on this side of the PageIndex boundary, after the generator had already produced a tree. A page whose text contains the literal "<|endoftext|>" made tiktoken.encode raise, so a PDF could deny its own indexing by quoting a special-token sequence. These call sites measure and slice untrusted document text rather than assembling a prompt, so they now pass disallowed_special=() and treat those sequences as the characters they are. All seven sites are fixed, not the one that surfaced: the loader re-verifies token counts, so fixing only the builder would have produced documents that build and then fail every load, and retrieval's evidence assembly runs at query time, where the same page would have crashed a user's request instead of just its own ingestion. PageIndex derives a node's end from where the next entry starts, so an out-of-order or re-listed sibling makes that subtraction run backwards and yields a range like 4-3. Only the end is manufactured that way - the start is the physical index the model reported - so such a node collapses to that single page rather than costing a whole verified document. Pages its content really covers stay reachable through the unmapped-pages node. A range running outside the document is still fatal: that is a claim about pages that do not exist. The repair is counted in ArtifactQuality.reversed_ranges so a generator whose derivation starts drifting shows up rather than being quietly patched over, and the error message now names the range and the document length, because which way a range is invalid decides whether the tree is repairable. Neither change touches artifact identity, which digests generator_commit and patch_sha256, so the 96 documents already built stayed valid and the two affected documents recovered on a plain requeue.
Six of the 104 tracked documents could not be indexed. Neither cause was model quality, which is what the failure code pageindex_child_failed suggested. verify_toc refused to check a table of contents whose last placed entry fell in the document's first half, on the assumption that a table of contents spans its document. Academic papers break that assumption as a matter of course. 2408.00724 is 22 pages with 11 correctly placed entries ending at page 10: acknowledgments are on page 11 and the bibliography runs from page 12. Its table of contents was complete and correct for the body, scored 0 without a single title being checked, and the caller then found no fallback mode left and discarded the document. The early return now fires only when nothing was placed at all. Coverage is not correctness: every title is still verified individually and the accuracy thresholds are untouched. process_large_node_recursively let a failed subdivision propagate out of the whole build, so 2305.16264 - whose own tree verified at 100% across 50 pages - was thrown away because one 20-page section would not split. Subdividing is an enhancement over a node that already carries a verified range and summary, so it now returns the node whole, exactly as the existing depth guard does. That document yields 67 nodes at depth 3 once it is allowed to finish. All six index end to end. Both fixes are pinned by checks that fail against +vr5, and each check now loads its own module instances: every one of them monkeypatches module attributes without restoring them, so the shared module made ordering load-bearing, which a comment was holding together. The 98 documents already built are untouched. patch_sha256 and configuration_hash are recorded at build time and never re-derived on load, so they stay valid; only newly built documents carry the +vr6 hash.
…luation The investigation behind this branch does not survive in the commits: the failure taxonomy, the cost figures that were measured rather than guessed, the reasoning behind leaving two configuration hashes in the index, and the operating recipes that each cost an hour to rediscover. Written so the work can continue from the repository and this page alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
The validation path is provider-free; it does not submit corpus content or incur provider spend.