Retire the legacy SQLite targets table and everything that reads or writes it

[MaxGhenis]

Summary

Retires Chronicle's legacy SQLite target path: the targets / strata / stratum_constraints tables, everything that wrote or read them, and the Supabase helpers for a targets schema that doesn't exist. Follow-on to #348 (merged as 130b95b), which deleted 12 of the 14 loaders that fed the table.

No live consumer read the table. After #348, its only writers were db/etl_soi.py and db/etl_snap.py. Both copy values that the soi-table-1-1, soi-table-1-4 and usda-snap-fy69-to-current source packages already publish as facts through the bundle and the consumer-fact export. Those outputs don't change: a build-bundle --year 2023 of the three packages is byte-identical before and after (below).

Removed

• Tables: db/schema.py loses Target, Stratum, StratumConstraint, TargetType and GeographicLevel. chronicle init now creates only the four source-artifact tables.
• CLI: chronicle load, chronicle stats and chronicle query (db/cli.py). These now exit 2 with argparse's "invalid choice".
• Loaders: db/etl_soi.py and db/etl_snap.py, with tests/test_etl_soi.py, tests/test_etl_snap.py and tests/test_congressional_districts.py. The last of these held one test, pinning the GeographicLevel enum.
• Supabase helpers: query_targets, query_strata, insert_targets_batch and TARGETS_SCHEMA (db/supabase_client.py).
• Re-exports: chronicle/targets/loaders.py, chronicle/targets/client.py, and the legacy names re-exported from chronicle.targets and policyengine_chronicle.targets.
• Normalization helpers: as_target, target_kwargs and TargetBlueprint. chronicle/normalization/targets.py is renamed to transforms.py, since nothing about targets is left in it.
• Export script: scripts/export_to_json.py, which wrote ../policyengine.org/public/data/targets_summary.json.

Kept

• Source-artifact store: the SourceArtifact / SourceTable / SourceColumn / SourceRow tables with init, load-source-files and source-stats. They have no foreign key to the legacy tables.
• Enums: Jurisdiction (used by SourceArtifact and pe_source_inventory) and DataSource (a type on SourceFact and SourceReference).
• Supabase client: SupabaseConfig, get_supabase_client, query_sources and LEDGER_SCHEMA, used by chronicle/mirror.py, chronicle.client and chronicle.sources.
• Target source coverage: chronicle/targets/us_poverty.py. chronicle.targets and policyengine_chronicle.targets now export exactly its seven names.
• Lineage helpers: convert_units, scale_value, apply_share and format_derivation.
• Packaged data: db/data/ and the db package in the wheel. 387 source packages read their files through resource_package: db.

Moved

AGI_BRACKETS and TABLE_1_1_AGI_LABEL_TO_BRACKET were imported by the live SOI harness module (chronicle/jurisdictions/us/soi.py:27) from db/etl_soi.py. They now live in soi.py itself, byte for byte. That import was the one live dependency on a loader that #348's Consumers section did not list.

CI

• "Build source input database" (init / load all / stats) becomes "Smoke-test the source-artifact database CLI" (init / source-stats).
• The wheel smoke test imports policyengine_chronicle.targets as before. It now also checks that the coverage API works from the wheel and that query_targets is gone.

Consumers (audit, 2026-10-12)

Full evidence is in ~/PolicyEngine/_reviews/chronicle-legacy-targets-audit/. Five investigators ran first, then two adversarial passes tried to find a reader they had missed. Every check was read-only.

• Inside Chronicle: no module under chronicle/ reads the three tables. This was checked by grep for ORM selects and raw SQL, and by the import closure. chronicle/database.py and chronicle/mirror.py work on their own ledger tables, and the bundle and consumer-fact export never touched the legacy ones.

• Other repositories: none read or write the path. The search covered:

  • git grep of the default branch of every local clone under PolicyEngine, TheAxiomFoundation and CosilicoAI (about 400 repos)
  • shallow clones of the 197 org repos with no local clone
  • GitHub code search across those orgs plus ThesisInstitute and MaxGhenis

  Three past readers are dead: microplex-us's ArchSQLiteTargetProvider (deleted 2026-06-08, repo archived), microplex's run_supabase_calibration.py (archived) and cosilico-microdata's validation/targets_db.py (archived 2025-12-22). Microcosm and chronicle.institute use only chronicle build-bundle / build-consumer-artifact (chronicle.institute through the ledger alias), which this PR leaves working.

• Supabase: the repo's configured project has no targets schema, so query_targets and query_strata returned PGRST106 Invalid schema: targets. An older copy of the tables sits in that project's microplex schema:

  • 14,349 target rows, loaded December 2025 and unmodified since 2026-01-02
  • last observed index read 2026-05-30 (pg_stat_user_tables; pg_stat_statements complete since 2025-12-28)
  • no request in the 90 days of retained edge logs apart from the audit's own probes

  This PR leaves that hosted data alone. Whether to drop it is d1438.

• Websites: targets_summary.json fed a CosilicoAI/cosilico.ai calibration page, deleted 2026-02-23 in 1e8157df. The ../policyengine.org path came from the 2026-05-26 branding rename (Update PolicyEngine branding #6). PolicyEngine/policyengine.org doesn't exist, https://www.policyengine.org/data/targets_summary.json returns 404, and policyengine-app-v2's history never contains the name. No workflow ever ran the script.

• Packaging: Chronicle has never been on PyPI. policyengine-chronicle, policyengine-ledger, policyengine-arch-data and three older names all return 404, and no repo declares a git dependency on it. So removing the legacy names from policyengine_chronicle.targets breaks no installed user. The changelog fragment carries a migration note.

Not covered by this PR:

• codex/thesis-ledger-facts branch: it carries its own frozen copy of the legacy loaders, and its required "Arch checks" job still runs arch load all. A PR on main doesn't touch that branch; a separate task covers it.
• Supabase mirror: the hosted mirror and query_sources default to a ledger schema that the configured project lacks. That is a separate task too.

Invariants

The PR removes code and adds no computational logic, so there are no property-based tests. Instead, the following properties are checked directly:

• No consumer output changes. This is a differential check. chronicle build-bundle --year 2023 --source soi-table-1-1 --source soi-table-1-4 --source usda-snap-fy69-to-current was run on main a5280e18 and on this branch, and all 16 JSONL outputs are byte-identical. The merged consumer_facts.jsonl sha256 is 96a2e62e… on both sides. Output is in the audit folder's diff-bundle/.
• The SOI constant move is exact. The existing compiler-parity test (test_chronicle_soi_targets.py) compares _legacy_soi_table_1_1_source_record_specs, which reads the moved tables, against the record-set specs, and it passes.
• chronicle init creates exactly the four source-artifact tables and none of the legacy ones (test_init_creates_only_source_artifact_tables).
• None of the 14 legacy loader modules ships or imports, and no db/etl_*.py remains.
• The CLI rejects load (with every former source and all), stats and query.
• The legacy names are gone from chronicle.targets, policyengine_chronicle.targets, db.schema and db.supabase_client, and scripts/export_to_json.py doesn't exist (test_legacy_target_api_is_retired).

Tests

• ruff check chronicle policyengine_chronicle db scripts tests (locked ruff) passes, and ruff format --check passes on every touched file.
• 4,937 tests collect on the rebased head.
• The affected test files pass on the rebased head: 77 passed and 1 skipped (the live-Supabase credential test). They are test_chronicle_boundaries, test_chronicle_facts, test_chronicle_namespace, test_chronicle_normalization, test_policyengine_chronicle_imports, test_supabase_client, test_source_files, test_chronicle_soi_targets and test_us_poverty_target_coverage.
• The harness CLI tests that import chronicle.cli (test_chronicle_artifacts, test_chronicle_verify_raw) pass: 76 passed.
• CLI smoke:
  • chronicle init and source-stats work.
  • chronicle --help lists the remaining commands.
  • chronicle load all exits 2.
  • ledger --help works.

Coverage dropped with the deleted tests:

• test_etl_soi.py checked the loader's own Table 1.1/1.4 parser, which is gone. The same 2023 workbook values are pinned through the packages in test_chronicle_soi_targets.py.
• test_etl_snap.py's source-fact checks are covered by test_usda_snap_source_package_builds_fy24_national_and_state_facts, which now also pins the fiscal-2024 period and the source checksum.

Chronicle governance

This PR removes the legacy target path. It adds no source package, fact, schema contract or consumer contract.

• Approved Chronicle agent role: ledger-source-ingestor (as for Retire US legacy ETLs whose values don't trace to publisher bytes #348 and UK enablement tooling: source-inventory de-hardcode, UK consumer-artifact suite, legacy ETL cleanup #135). Most edits (db/, docs/, scripts/, .github/workflows/ci.yml) fall outside its listed paths. They are the same kinds of files UK enablement tooling: source-inventory de-hardcode, UK consumer-artifact suite, legacy ETL cleanup #135 and Retire US legacy ETLs whose values don't trace to publisher bytes #348 changed to retire legacy ETLs.
• Deterministic checks: listed above.
• Independent review: GPT-6.1 Sol (subfleet job 20261012-144145-chronicle-legacy-targets-review) approved efc80749 with no P0–P2 findings. It confirmed by mutation that the boundary, provenance, derivation-order, namespace-identity and public-removal tests fail when their guarded conditions break, and found no dangling references (dynamic imports, resource paths, package YAML, packaging). Its five P3s were fixed before the rebase onto main (55b2f30f): a guard on the canonical chronicle.normalization, the SNAP fiscal-2024 period and checksum moved into the package test, "no live consumer" wording, "last observed index read" for the Supabase date, and the historical CI description.

Decisions queued: d1438 (keep or drop the stale hosted microplex.* tables). Separately, d1437 asks whether to rotate a Supabase secret that the audit's tooling exposed to local session logs. It is unrelated to this code.

axiom: n/a: data infrastructure; no policy rule changes

🤖 Generated with Claude Code