[anth-volk]

Fixes #4022

Important

This PR is the base of a stack (#4042, #4046, #4049) on master. It no longer sits on the region reference files PR (#4020) or the PRs below it: the format 2 work is ported onto master's region code.

Its first commit bumps PolicyEngine.py to 6.12.0, the release containing PolicyEngine/policyengine.py#701, which publishes catalog format 2's fields. On 6.11.1 every catalog extraction stops with "the installed PolicyEngine.py does not publish catalog format 2's fields". uv.lock locks 6.12.0, whose country packages are unchanged from 6.11.1 (policyengine-core 3.33.3, policyengine-us 2.2.1, policyengine-uk 2.151.0). Merging also waits for the simulation API to support that bundle (the "Ensure PolicyEngine bundle is supported by simulation API" check).

Summary

Catalog format 2 stores the metadata the app's v3 workspace reads, taken only from PolicyEngine.py's public records (PolicyEngine/policyengine.py#701), never from a country package's tax-benefit system or a private PolicyEngine.py method. This branch is based on master; its first commit pins PolicyEngine.py 6.12.0.

• Schema. Revision f5c1d9b4685b (on master's countries revision fb1a3e40d3d6) was autogenerated against the v2 configuration with no corrections. It adds nullable columns with no defaults, so catalog versions published earlier keep them empty, and it writes no data:

  • tax_benefit_model_versions: catalog_format, entities, basic_inputs;
  • variables: is_input_variable, hidden_input, module_name, adds_parameter_path, subtracts_parameter_path, enum_values, unit, definition_period;
  • parameters: economy, household.

  entities and enum_values are typed with CatalogEntity and CatalogEnumValue.

• Extraction.

  • normalize_model_metadata reads entities (key, plural, person flag), basic inputs, variable flags and modules, the parameter path that adds or subtracts names (beside the list PolicyEngine.py resolves, which stays), enum values with their declared labels (an integer label stays an integer), each variable's unit and definition period, and parameter flags.
  • The unit (for example currency-USD) and the definition period (year, month or eternity) are needed by the app's household builder and breakdown on v2: currency formatting, "per month" and "per year" labels, and household validation. PolicyEngine.py's public Variable record carries both (6.7.5 and 6.10.0 already do).
  • Bracket schedules arrive as public parameter nodes, with their stored label or none.
  • Extraction first checks that PolicyEngine.py publishes these fields and names each one missing. It then fails with a specific message on:
    • other than one person entity;
    • a basic input or parameter path that is not published;
    • enum labels that do not match the possible values or are not JSON scalars;
    • a flag that is not a boolean.
  • An empty basic-input list is stored as empty.

• Default datasets come from PolicyEngine.py. By the user's decision, each country's default dataset is the default_dataset the installed PolicyEngine.py release names in its manifest. This PR removes the Public API's own list (REVIEWED_DEFAULT_DATASETS). Publication still stops if the manifest names no default, if the release does not certify it, or if the release routes national regions to another dataset.

• Storage and comparison. Publication stages and inserts the new fields. When the same PolicyEngine.py version is published again, a version stored before format 2 (null catalog_format) is compared without the format 2 fields and keeps them empty; a format 2 version must match on them too. Publication never rewrites a stored version.

• Parity check. tests/integration/test_v2_catalog_metadata_parity.py (opt-in through RUN_V2_CATALOG_METADATA_PARITY=1) compares every new field with /{country}/metadata for the US and the UK. Enum labels are compared with their types. CI does not run it: it loads the complete installed models, which CI no longer does since Replace the full-catalog integration tests with selected folders of the installed models #4150.

• Master's catalog tests. The value-shape catalog (tests/fixtures/v2_catalog_value_shapes.py) gives its added variables and parameters the format 2 fields, and the installed-folders test (tests/integration/test_v2_catalog_installed_folders.py) passes the real models' entities and an empty basic-input list to extraction, since its reduced models publish no variables.

Parity with /{country}/metadata

Measured with PolicyEngine.py #701's branch installed locally (6.10.0 with the new fields). 0 mismatches in every field:

Field	US	UK
Entities (key, plural, person flag, in order)	6	3
Basic inputs	3	5
Variables compared	6,167	1,170
Enum variables (labels and their types)	290	37
Input variables	925	300
Hidden inputs	54	0
Units (variables with one; 15 distinct in each country)	3,955	708
Definition periods (year / month / eternity)	4,210 / 1,953 / 4	1,160 / 10 / 0
Parameter-named adds	237	4
Parameter-named subtracts	2	0
gov parameters (economy and household flags)	100,276	1,551

Every gov bracket schedule is a catalog node, and its stored label matches the label /{country}/metadata shows when one is stored.

With #701's branch installed, the full installed extraction succeeds:

• US: default populace_us_2024; 51 states and 436 congressional districts as row filters on populace_us_2024_acs_local. Region data verification against the real files passes.
• UK: default microcosm_uk_2024_25; the national region and four nations on it; 650 constituencies and 361 local authorities as row filters on microcosm_uk_2024_25_dense. Region data verification of the UK files could not run locally, for lack of read access to the UK data, so it first runs in the seed job.

The PolicyEngine.py bump

• The first commit, "Update the PolicyEngine bundle to 6.12.0", pins policyengine[models]==6.12.0 in pyproject.toml and docker/Dockerfile and adds the bundle update's changelog fragment, changelog.d/update-policyengine-bundle-6.12.0.changed.md.
• The same commit regenerates uv.lock for 6.12.0. The installed-catalog counts that tests/integration/test_v2_catalog_installed.py pinned no longer need updating: Replace the full-catalog integration tests with selected folders of the installed models #4150 removed that test.
• Default datasets come from the release, so nothing else depends on the version.

Production-scale publication with format 2 fields

Measured on the earlier stack, which included the region reference files PR (#4020). Without it, regions come from PolicyEngine.py's region registry, as on master. #4150 has since removed this test from master; the numbers below are kept as a record.

RUN_V2_CATALOG_PUBLICATION_QUALIFICATION=1 pytest tests/integration/test_v2_catalog_publication_qualification.py, with #701's branch installed and a disposable Postgres 17, passed:

• Peak additional publisher memory: 8,147,310 bytes (7.8 MiB), against the test's 256 MiB limit (8,146,245 bytes before the unit and definition period were added).
• Publication time: 53.9 s for the whole catalog on a shared machine (86.5 s in an earlier, more loaded run).
• Rows published: 7,337 variables, 30,214 parameter nodes, 104,172 parameters, 1,194,360 parameter values, 4 datasets and 1,504 regions. 333 US places are left out.

Migration notes

• Route groups: none change. This prepares stored metadata that a later PR serves through the v2 catalog routes.
• API v1 routes: /{country}/metadata and the other v1 routes are unchanged.
• User-visible contract changes: none.
• Migration: one generated Alembic revision on the v2 chain. It merges one at a time with any other open v2 revision.

Environment variables

None read by the application, and no workflow changes. The parity test reads RUN_V2_CATALOG_METADATA_PARITY, which no CI job sets; a developer sets it to 1 to run the test locally.

Testing

On master after #4150, with PolicyEngine.py 6.12.0 installed. The stack was rebased onto master at 0f2477293. Its heads are 75f4a31ea (this PR), 718795501 (#4042), 33f77cba2 (#4046) and 9905cfe07 (#4049). Each head passed:

• ruff, ruff format, mypy and scripts/run_quality_guards.py, with scripts/export_migration_contracts.py producing no changes;
• its PR's changed unit tests and changed integration tests, on a disposable local Postgres 17 at that head's revision;
• Replace the full-catalog integration tests with selected folders of the installed models #4150's 16 tests, including test_v2_catalog_installed_folders.py and the value-shape publication test.

The parity test also passed with RUN_V2_CATALOG_METADATA_PARITY=1: 8 of 8. One review of the conflict resolutions found no remaining mistakes; its two wording and authorship findings are fixed.

The table below records the original commits' runs, before this PR moved onto master; the revision then followed the dropped saved-policy rename fa7fa1035444. On master it follows fb1a3e40d3d6, and each commit's changed tests pass there; the integration tests ran against a local Postgres, apart from the parity and installed-catalog tests, which need 6.12.0.

At the time, GitHub Actions runs on this branch were cancelled at the user's request, so that evidence is local. Commits are listed from the bottom of the branch.

Review. Two review rounds. The first found no substantiated bugs, and its quality findings are fixed. The second reviewed the default-dataset change and also found no substantiated bugs; its findings on this description and the documentation are fixed. A third review, of the unit and definition period, found no substantiated bugs; its quality findings (test names, and limits written twice) are fixed.

Commit	Failing run first	Passing run
d66267943 columns and revision	column test KeyError: 'catalog_format'; revision chain and new-revision tests (no child of fa7fa1035444); lifecycle test KeyError with the revision removed	focused unit tests; v2 lifecycle tests; alembic check clean
14673dc46 extraction and parity test	new modules failed at import (CATALOG_FORMAT, normalize_model_metadata); workflow test did not find the parity test	21 unit tests; parity 8 of 8 on the US and UK
07b7b0cfe storage and comparison	staging rows without the new columns; stored fields all null; changed format 2 field "DID NOT RAISE"	publication, route and lifecycle integration tests
b8a75d12c naming missing fields	8 cases failed; on a release without the fields the error was "variable age_group enum values do not match its possible values"	79 focused tests
a94b2e2d4 label types in the parity test	with integer labels turned into floats on purpose, the old comparison passed and the new one found 1 mismatch	parity 8 of 8
cca58e31a round-trip and change detection	with the fields left out of the comparison and labels stored as strings on purpose, all 4 new tests failed	integration and catalog unit tests
e46611fca typed JSON columns	ImportError: CatalogEntity	models, unit and integration tests
494f43465 default datasets from the release	5 failed against the previous code: a release default other than the fixed list was refused ("does not match the reviewed selection"), and the missing, empty, uncertified and national-route cases gave the old message	62 extraction tests; 188 catalog, Alembic and model unit tests
1f69e9144 documentation: 6.10.0 also lacks the fields	documentation only	67 extraction and metadata tests; ruff, mypy and quality guards
5c565e61f unit and definition period columns; revision f5c1d9b4685b regenerated with autogenerate (--rev-id f5c1d9b4685b, parent fa7fa1035444), now 13 columns	column test KeyError: 'unit'; revision test assert 11 == 13; lifecycle test on the previous revision KeyError: 'unit'	75 model, revision and lifecycle tests; alembic check clean; 1,129 v2 unit tests
376846411 extracting the unit and definition period	5 unit tests failed (record without unit, missing-field and type messages); parity test AttributeError: 'VariableRecord' object has no attribute 'unit'	31 unit tests; parity 8 of 8; 1,132 v2 unit tests
7c0eac46f storing and comparing them	staging rows without the two columns; comparison column counts [3, 6, …]; stored values missing; a changed unit or period "DID NOT RAISE"	11 staging and publication unit tests; 40 publication, route and lifecycle integration tests
7e0fb9518 test and fixture names that name the unit and period	tests only	the renamed tests pass
f803106ee text limits from the column lengths	ImportError: VARIABLE_TEXT_LIMITS; the over-long cases passed before too, as the limits already existed	100 extraction tests, including one over-long value per format 2 text field; 1,138 v2 unit tests

On the final branch, under the locked PolicyEngine.py 6.10.0:

• the full make test set: 3,521 passed;
• publication, route and lifecycle integration tests on a schema recreated at head on a disposable Postgres 17: 40 passed;
• alembic check: no new operations;
• ruff, ruff format, mypy, scripts/run_quality_guards.py, python -m scripts.rules validate and guard: clean;
• scripts/export_migration_contracts.py: no changes;
• the opt-in tests that call extract_installed_catalog fail with the missing-fields message: test_v2_catalog_installed.py (1 test), test_v2_catalog_publication_qualification.py (1 test) and every parity test (8). That is expected until the bump commit.

With #701's branch installed:

• the full make test set: 3,507 passed (before the final rebase);
• the parity test, including the unit and definition period: 8 passed;
• the publication qualification: passed (measurement above);
• test_v2_catalog_installed.py: differs only in the parameter_nodes count listed above.

There are no inline # noqa suppressions and no changes under rules/.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.18182% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.13%. Comparing base (39ccf27) to head (8196728).
⚠️ Report is 25 commits behind head on master.

Files with missing lines	Patch %	Lines
.../v2/catalog/extraction_variables_and_parameters.py	74.19%	4 Missing and 4 partials ⚠️
policyengine_api/data/v2/catalog/extraction.py	95.58%	2 Missing and 1 partial ⚠️
...yengine_api/data/v2/catalog/publication_staging.py	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #4023       +/-   ##
===========================================
+ Coverage   51.51%   88.13%   +36.62%     
===========================================
  Files         267      279       +12     
  Lines       15026    16380     +1354     
  Branches     2399     2668      +269     
===========================================
+ Hits         7740    14437     +6697     
+ Misses       6647     1282     -5365     
- Partials      639      661       +22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.