feat: complete PageIndex v2 evaluation and hardening

.env.example

@@ -23,7 +23,7 @@ POSTGRES_PORT=5432
 
 # REQUIRED to ingest or answer questions: hosted DeepSeek credentials and model names.
 DEEPSEEK_API_KEY=
-DEEPSEEK_MODEL=deepseek-v4-pro
+DEEPSEEK_MODEL=deepseek-v4-flash
 PAGEINDEX_MODEL=deepseek/deepseek-v4-pro
 # REQUIRED operational bounds for provider retries, PageIndex runtime, and abandoned worker jobs.
 PAGEINDEX_LLM_MAX_ATTEMPTS=3
@@ -37,13 +37,19 @@ INGESTION_LEASE_SECONDS=120
 INGESTION_HEARTBEAT_SECONDS=30
 INGESTION_STALE_GRACE_SECONDS=300
 CATALOG_BATCH_TOKEN_LIMIT=24000
+CATALOG_ROUTING_CONCURRENCY=4
 DOCUMENT_LIMIT=8
 EVIDENCE_TOKEN_LIMIT=40000
 REQUEST_DEADLINE_SECONDS=180
 # OPTIONAL: needed only for pilot and query spend reports. The input rate is the cache-miss rate.
-DEEPSEEK_INPUT_COST_PER_MILLION_USD=
-DEEPSEEK_CACHE_HIT_INPUT_COST_PER_MILLION_USD=
-DEEPSEEK_OUTPUT_COST_PER_MILLION_USD=
+# deepseek-v4-flash list prices; update from https://api-docs.deepseek.com/quick_start/pricing/
+DEEPSEEK_INPUT_COST_PER_MILLION_USD=0.14
+DEEPSEEK_CACHE_HIT_INPUT_COST_PER_MILLION_USD=0.0028
+DEEPSEEK_OUTPUT_COST_PER_MILLION_USD=0.28
+# PAGEINDEX_MODEL runs indexing and is priced separately; deepseek-v4-pro list prices.
+PAGEINDEX_INPUT_COST_PER_MILLION_USD=0.435
+PAGEINDEX_CACHE_HIT_INPUT_COST_PER_MILLION_USD=0.003625
+PAGEINDEX_OUTPUT_COST_PER_MILLION_USD=0.87
 
 # REQUIRED (local and production): host path mounted read-only at /corpus.
 ARXIV_PDF_HOST_DIR=./arxiv-pdfs

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -12,8 +12,8 @@ body:
     id: version
     attributes:
       label: Version or commit
-      description: Provide the v0.2.x version or full commit SHA.
-      placeholder: v0.2.0
+      description: Provide a supported release version or full commit SHA.
+      placeholder: v0.3.0
     validations:
       required: true
   - type: dropdown

.gitignore

@@ -1,4 +1,6 @@
 .env
+.env.bak
+.env.*.bak
 .venv/
 __pycache__/
 .pytest_cache/
@@ -17,6 +19,7 @@ apps/web/dist/
 apps/web/test-results/
 apps/web/playwright-report/
 apps/api/evaluation/run/
+apps/api/evaluation/pageindex-v2-authoring-queue.json
 apps/api/pilots/run/
 
 arxiv-pdfs/

CHANGELOG.md

@@ -7,7 +7,40 @@ This changelog records user-visible changes. Vectorless RAG follows semantic ver
 
 ## Unreleased
 
-No user-visible changes have been recorded after v0.2.0.
+No changes have been recorded after v0.3.0.
+
+## 0.3.0, 2026-07-31
+
+Version 0.3.0 completes the first bounded PageIndex v2 evaluation and hardens the retrieval, accounting, and operator workflows it exercised.
+
+### Added
+
+- Reviewed 120-case PageIndex v2 suite with source-bound authored prompts, interactive review, deterministic validation, and a generated review pack
+- Frozen-index experiment preparation, provider-free preflight, resumable concurrent execution, claim-level judging, and deterministic aggregate reporting
+- Database-enforced experiment ownership, immutable call reservations, cost ceilings, atomic call admission, and interrupted-owner recovery
+- `make env-check` and `make env-sync` for safe reconciliation of local configuration with `.env.example`
+- Operator commands for index freezing, evaluation restart and execution, claim-review status, query-cost reporting, and provider-free PageIndex patch checks
+
+### Changed
+
+- Completed the reviewed 120-case PageIndex v2 evaluation as a 1,080-trial experiment and recorded its metrics, accounting evidence, exclusions, and immutable artifact identities
+- Kept the evidence budget unchanged because the completed experiment found no consistent quality gain from larger windows
+- Reduced evaluation plans from 1,440 to 1,080 trials by removing a duplicate execution path
+- Changed the default serving model to `deepseek-v4-flash` while retaining `deepseek/deepseek-v4-pro` for PageIndex construction, with separate price configuration
+- Upgraded the vendored PageIndex patch to `+vr6`, including complete-corpus indexing support and stricter artifact validation
+- Expanded provider-free verification across evaluation execution, accounting, PageIndex parsing, retrieval limits, and PostgreSQL role boundaries
+
+### Fixed
+
+- Preserved valid PageIndex trees containing control characters, pageless tables of contents, continuation entries, large nodes, and sparse terminal sections
+- Prevented environment leakage into the PageIndex child process and test suite
+- Started the web development container directly through Vite, removing its startup dependency on `pnpm`
+- Made evaluation ledgers resumable after interrupted writes and prevented concurrent writers from erasing each other’s results
+- Routed content-based discovery, quoted passage lookup, and corpus claim checks through document evidence while reserving clarification for incomplete intent
+- Applied structured metadata constraints consistently to routing, guarded catalog SQL, and document retrieval
+- Preserved an empty hybrid catalog selection as an empty document restriction and required successful hybrid answers to cite document evidence
+- Contained malformed generated SQL and PostgreSQL expression errors without hiding infrastructure failures
+- Made model-call admission, experiment ownership, cost reservation, interruption recovery, and retry classification consistent across concurrent workers
 
 ## 0.2.0, 2026-07-27
 

Makefile

@@ -30,14 +30,14 @@ PILOT_OUTPUT ?= apps/api/pilots/run/result.json
 
 export UV_CACHE_DIR
 
-.PHONY: help env init install api-install web-install format api-format web-format lint api-lint web-lint \
+.PHONY: help env env-sync env-check init install api-install web-install format api-format web-format lint api-lint web-lint \
 	typecheck api-typecheck web-typecheck test api-test web-test test-unit test-integration \
 	test-integration-stack docs-check \
 	api-build web-build web-production-smoke playwright-install playwright playwright-live react-doctor react-doctor-diff astryx-doctor workflow-check migration-check \
 	compose-check helm-lint helm-template helm-check docker-build docker-build-api docker-build-web \
 	check api-check web-check deploy-check ci up rebuild dev dev-build dev-health down stop restart restart-app restart-web \
 	restart-langfuse restart-observability ps logs health web-dev migrate bootstrap-key \
-	download-corpus sync-corpus upload-document pilot-check pilot-run pilot-report query-cost-report artifact-verify artifact-gc pageindex-patch-test evaluation-build evaluation-check evaluation-review-pack evaluation-seed evaluation-run evaluation-report \
+	download-corpus sync-corpus upload-document pilot-check pilot-run pilot-report query-cost-report artifact-verify artifact-gc pageindex-patch-test evaluation-build evaluation-check evaluation-review-pack evaluation-review evaluation-review-status evaluation-seed evaluation-restart evaluation-run evaluation-freeze evaluation-execute evaluation-judge evaluation-judge-status evaluation-report \
 	secret-api-pepper secret-session secrets-langfuse
 
 help: ## Show all supported project commands.
@@ -50,6 +50,11 @@ env: ## Create .env from .env.example if absent and restrict its permissions.
 init: ## Create a mode-600 .env with generated local secrets without overwriting.
 	$(PYTHON) scripts/init_env.py
 
+env-sync: ## Add keys .env is missing from .env.example (ADOPT=KEY,KEY to take example values).
+	$(PYTHON) scripts/sync_env.py $(if $(ADOPT),--adopt "$(ADOPT)",)
+env-check: ## Report .env drift against .env.example without writing.
+	$(PYTHON) scripts/sync_env.py --check
+
 install: api-install web-install ## Install all frozen development dependencies.
 api-install: ## Install the frozen Python environment under apps/api.
 	$(UV) sync --project $(API_DIR) --frozen
@@ -58,15 +63,15 @@ web-install: ## Install the frozen root pnpm workspace.
 
 format: api-format web-format ## Format API and web sources.
 api-format: ## Format and autofix Python sources.
-	$(UV) run --project $(API_DIR) ruff format $(API_DIR)
-	$(UV) run --project $(API_DIR) ruff check --fix $(API_DIR)
+	$(UV) run --project $(API_DIR) ruff format $(API_DIR) scripts
+	$(UV) run --project $(API_DIR) ruff check --fix $(API_DIR) scripts
 web-format: ## Autofix lint findings in web sources.
 	$(PNPM) --filter @vectorless-rag/web exec eslint . --fix
 
 lint: api-lint web-lint ## Lint API and web sources.
 api-lint: ## Lint and check formatting for Python sources.
-	$(UV) run --project $(API_DIR) ruff check $(API_DIR)
-	$(UV) run --project $(API_DIR) ruff format --check $(API_DIR)
+	$(UV) run --project $(API_DIR) ruff check $(API_DIR) scripts
+	$(UV) run --project $(API_DIR) ruff format --check $(API_DIR) scripts
 web-lint: ## Lint web sources.
 	$(PNPM) --filter @vectorless-rag/web lint
 
@@ -85,6 +90,7 @@ test-unit: ## Run Python tests that require no external infrastructure.
 	cd $(API_DIR) && $(UV) run pytest tests -m 'not integration'
 test-integration: ## Run PostgreSQL integration tests with TEST_*_DATABASE_URL.
 	@test -n "$${TEST_OWNER_DATABASE_URL:-}" || { echo "TEST_OWNER_DATABASE_URL is required" >&2; exit 2; }
+	@test -n "$${TEST_APP_DATABASE_URL:-}" || { echo "TEST_APP_DATABASE_URL is required" >&2; exit 2; }
 	@test -n "$${TEST_DATABASE_URL:-}" || { echo "TEST_DATABASE_URL is required" >&2; exit 2; }
 	cd $(API_DIR) && $(UV) run pytest tests -m integration
 test-integration-stack: ## Run integration tests against an isolated disposable PostgreSQL project.
@@ -237,14 +243,39 @@ evaluation-check: ## Validate the tracked 120-case PageIndex suite without PDFs
 	cd $(API_DIR) && $(UV) run python scripts/check_evaluation_suite.py
 evaluation-review-pack: ## Rebuild the local PageIndex evaluation review pack.
 	cd $(API_DIR) && $(UV) run python scripts/build_evaluation_review_pack.py
+evaluation-review: ## Review PageIndex cases interactively (REVIEWER=name, optional STRATUM=).
+	@test -n "$(REVIEWER)" || { echo "set REVIEWER=<your identity>" >&2; exit 2; }
+	cd $(API_DIR) && $(UV) run python scripts/review_evaluation.py \
+		--reviewer "$(REVIEWER)" $(if $(STRATUM),--stratum "$(STRATUM)",)
+evaluation-review-status: ## Show PageIndex review progress.
+	cd $(API_DIR) && $(UV) run python scripts/review_evaluation.py --status
 evaluation-seed: ## Seed the reviewed PageIndex evaluation suite into Langfuse.
 	@test -f "$(DATASET)" || { echo "DATASET does not exist: $(DATASET)" >&2; exit 2; }
 	$(COMPOSE) exec -T api python scripts/seed_evaluation.py /dev/stdin \
 		--suite pageindex-v2 --corpus evaluation/pageindex-v2-corpus.json \
 		--name "$(DATASET_NAME)" < "$(DATASET)"
-evaluation-run: ## Run a frozen PageIndex evaluation experiment from a reviewed suite.
+evaluation-restart: ## Rebuild API and worker with the clean HEAD as the runtime release.
+	@test -z "$$(git status --porcelain)" || { echo "evaluation rebuild requires a clean worktree" >&2; exit 2; }
+	@release_sha="$$(git rev-parse HEAD)"; \
+		RELEASE="$$release_sha" $(COMPOSE) up -d --build --force-recreate api worker
+evaluation-run: ## Prepare a frozen PageIndex experiment (INDEX_HASH=…, MAX_COST=…).
 	@test -n "$(INDEX_HASH)" || { echo "INDEX_HASH=<sha256> is required" >&2; exit 2; }
-	cd $(API_DIR) && $(UV) run python scripts/run_evaluation.py --index-hash "$(INDEX_HASH)"
+	@test -n "$(MAX_COST)" || { echo "MAX_COST=<usd> is required; it becomes the run's ceiling" >&2; exit 2; }
+	@release_sha="$$(git rev-parse HEAD)"; \
+		RELEASE="$$release_sha" $(UV) run --project $(API_DIR) \
+		python $(API_DIR)/scripts/run_evaluation.py \
+			--index-hash "$(INDEX_HASH)" --max-cost-usd "$(MAX_COST)" $(if $(CONCURRENCY),--concurrency $(CONCURRENCY),)
+evaluation-freeze: ## Print the index digest for evaluation-run INDEX_HASH.
+	$(COMPOSE) exec -T api python scripts/freeze_index.py
+evaluation-execute: ## Execute prepared trials against DeepSeek (LIMIT=n, DRY_RUN=1).
+	$(COMPOSE) exec -T --user "$$(id -u):$$(id -g)" api python scripts/execute_evaluation.py \
+		$(if $(LIMIT),--limit $(LIMIT),) $(if $(DRY_RUN),--dry-run,) $(if $(CONCURRENCY),--concurrency $(CONCURRENCY),)
+evaluation-judge: ## Judge queued answer claims (REVIEWER=name, optional METHOD=human|automated).
+	@test -n "$(REVIEWER)" || { echo "set REVIEWER=<your identity>" >&2; exit 2; }
+	cd $(API_DIR) && $(UV) run python scripts/judge_evaluation.py \
+		--reviewer "$(REVIEWER)" $(if $(METHOD),--method "$(METHOD)",)
+evaluation-judge-status: ## Show answer-claim judgment progress.
+	cd $(API_DIR) && $(UV) run python scripts/judge_evaluation.py --status
 evaluation-report: ## Build the local report for the latest frozen evaluation experiment.
 	cd $(API_DIR) && $(UV) run python scripts/report_evaluation.py
 

README.md

@@ -1,42 +1,50 @@
 <!-- Content type: Landing -->
-<!-- Content plan: Goal: help researchers decide whether to run Vectorless RAG and reach the first cited answer. Audience: researchers, operators, and contributors. Content: status, use cases, architecture, costs, startup, testing, and project links. Open questions: measured vector-versus-PageIndex answer quality at full-corpus scale. -->
+<!-- Content plan: Goal: help researchers decide whether to run Vectorless RAG and reach the first cited answer. Audience: researchers, operators, and contributors. Content: release and research status, fit, architecture, costs, startup, verification, and project links. Open questions: measured vector-versus-PageIndex answer quality at full-corpus scale. -->
 
 # Vectorless RAG
 
 [![CI](https://github.com/ProofOfTechOrg/vectorless-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/ProofOfTechOrg/vectorless-rag/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/github/license/ProofOfTechOrg/vectorless-rag)](LICENSE)
 [![Release](https://img.shields.io/github/v/release/ProofOfTechOrg/vectorless-rag?include_prereleases)](https://github.com/ProofOfTechOrg/vectorless-rag/releases)
 
-Vectorless RAG is a self-hosted research preview for grounded question answering over local Portable Document Format (PDF) files. It uses PageIndex hierarchies instead of embeddings or a vector database, then returns inline citations that resolve to document pages.
+Vectorless RAG is a self-hosted research preview for grounded question answering across local PDFs. It builds PageIndex hierarchies without embeddings or a vector database and returns inline citations tied to document pages.
 
-Version 0.2.0 is a prerelease. It supports evaluation and bounded pilots, not production stability or unattended full-corpus ingestion.
+Version 0.3.0 is a prerelease. It supports evaluation and bounded pilots, not production stability or unattended full-corpus ingestion.
 
-## Choose this project for bounded research
+## Current research status
+
+The PageIndex v2 evaluation is complete: 1,080 trials produced 1,018 included scores and 62 excluded terminal failures. Free routing selected the required document path in only 18.8% to 31.7% of trials, while larger evidence budgets did not produce a consistent end-to-end quality gain. The experiment therefore does not justify increasing the evidence budget.
+
+The routing, structured-constraint, generated-SQL, hybrid-evidence, retry, and accounting defects exposed during the run are fixed and covered by provider-free tests. Those post-experiment corrections have not been remeasured against DeepSeek, and no replacement experiment is planned. A final vector-versus-PageIndex quality verdict still requires a separately designed comparison.
+
+Read the [completed evaluation results and handoff](docs/pageindex-evaluation-handoff.md) for metrics, accounting, defect history, artifact identities, and remaining limits.
+
+## Use cases
 
 Vectorless RAG fits these use cases:
 
-- Compare hierarchical, vectorless retrieval with a conventional vector retrieval-augmented generation (RAG) system
+- Use hierarchical, vectorless retrieval as one arm of a controlled comparison with conventional vector retrieval-augmented generation (RAG)
 - Ask cross-document questions over a small, controlled PDF corpus
 - Inspect retrieval routes, durable ingestion jobs, citations, token use, and provider costs
 - Run a self-hosted browser console with scoped API keys and server-side credential storage
 
 Do not use this preview for regulated workloads, scanned PDFs that require optical character recognition (OCR), or an unreviewed full-corpus migration.
 
-## Understand costs and limits before starting
+## Costs and operating limits
 
-Ingestion and live chat call DeepSeek and can incur provider charges. The default safety controls cap registration at 25 documents, keep full-corpus indexing disabled, and enforce a $10 frozen-pilot ceiling. Provider prices can change, so check the [official DeepSeek model and pricing reference](https://api-docs.deepseek.com/quick_start/pricing/) before a paid run.
+Ingestion and live chat call DeepSeek and can incur provider charges. The default configuration uses `deepseek-v4-flash` for serving and `deepseek/deepseek-v4-pro` for PageIndex construction. Confirm both underlying models and their current rates in the [official DeepSeek model and pricing reference](https://api-docs.deepseek.com/quick_start/pricing/) before a paid run.
 
-The configured `deepseek-v4-pro` model remains available in that reference. Deterministic unit, integration, deployment, and browser tests do not require provider credentials.
+Safety controls cap registration at 25 documents, keep full-corpus indexing disabled, and enforce a $10 frozen-pilot ceiling. The completed evaluation used `$5.389689` of conservative accounting exposure under its separate `$25` authorization; that ledger exposure is not a provider-invoice claim. Deterministic unit, integration, deployment, browser, and evaluation checks do not require provider credentials.
 
 Current limitations include:
 
 - Born-digital PDFs only, with no OCR
 - No API-key lifecycle interface or embedded PDF viewer
-- Source builds only, with no package-registry or container-registry release
+- Releases provide source archives but no prebuilt artifacts
 - A pilot gate instead of a production-scale claim
 - No final vector-versus-PageIndex quality verdict
 
-## Follow the request path
+## Architecture
 
 The browser-facing frontend (BFF) keeps API keys out of browser-readable storage. FastAPI authorizes each API request, LangGraph coordinates retrieval, and a durable worker builds PageIndex artifacts.
 
@@ -80,24 +88,18 @@ Read [get started with one cited answer](docs/getting-started.md) for prerequisi
 
 ## Run provider-free verification
 
-Install the frozen development dependencies before running the complete local gates:
+Install the frozen development dependencies, then run the complete continuous-integration equivalent:
 
 ```bash
 make install
-make docs-check
-make api-check
-make test-integration-stack
-make web-check
-make playwright
-make deploy-check ENV_FILE=.env.example
-make docker-build
+make ci
 ```
 
-`make playwright` uses the existing mock upstream only for deterministic browser coverage. It is not a product demo and does not replace the paid release gate.
+`make ci` runs documentation, API, web, isolated PostgreSQL, deployment, browser, and image gates without reading `DEEPSEEK_API_KEY`. Deterministic Playwright uses a mock upstream; it does not replace an explicitly authorized paid release check.
 
 Read [choose a test tier](docs/testing.md) for test prerequisites and provider boundaries.
 
-## Find the next task
+## Documentation and project policy
 
 The [documentation index](docs/README.md) routes setup, operation, deployment, security, evaluation, and historical design work. Start with:
 

SECURITY.md

@@ -12,6 +12,7 @@ Security fixes target these versions:
 | Version | Supported |
 | --- | --- |
 | Current `master` | Yes |
+| 0.3.x research previews | Yes |
 | 0.2.x research previews | Yes |
 | Earlier versions | No |