[anth-volk]

Fixes #1341

PolicyEngine/calibration-diagnostics#255, the dashboard's schema 9 reader, merged and deployed to production on 2026-10-26, so the dashboard reads the schema 9 files that builds publish once this merges. A Microcosm merge deploys nothing. One consumer is still not ready: microcosm-collector's planned content handler for calibration-diagnostics submissions (not written yet; the submission intake, in the still-open microcosm-collector#12, checks only file role, size and SHA-256) must accept schema 9, or that gap must be knowingly accepted before this merges.

Base

This was stacked on calibration-diagnostics-json-schema (#1298) because the JSON Schema has to be regenerated for schema 9: #1298 publishes a JSON Schema generated from CalibrationDiagnosticsV8 for consumers that check the file without importing Microcosm (the collector), and that schema refuses schema 9 (schema_version const 8, closed objects). This PR adds a schema 9 file beside it with the same generator. #1298 has merged and the base is now main, which includes #1336, which this PR needs (see "Tests"). main was merged in on 2026-10-26 (next section), and CI runs on this PR from that push on.

Merge of main, 2026-10-26

235e68e78 merges main at f6a790cf0 (251 commits, including #1298) with a merge commit; nothing was rebased or force-pushed.

• Conflicts: none. Ten files changed on both sides, and Git merged each automatically: README.md, docs/agent-guide.md, uk_runtime/diagnostics.py, uk_runtime/graph_terminal.py, microcosm/data/contract.py, tools/assemble_uk_dense_release_dir.py, and the UK diagnostics, dense assembler, US fiscal-refresh and contract tests. In each, main added code beside this PR's changes (withheld-row handling, new UK check names, the SPI Public Use Tape marker, the recertification release type) and changed none of the same lines.
• Schema 9 still carries everything schema 8 carries. main changes neither microcosm-diagnostics, the calibrate writer nor the transport diagnostics kernel, so both JSON Schema files are unchanged (tools/generate_calibration_diagnostics_json_schema.py --check: current for both; no regeneration commit). What main's UK writer adds (withheld-row metadata, uk_diagnostics.model_figures_withheld, consumption drift diagnostic_warnings) sits in fields both models share, and the UK writer still writes through the schema 9 writer. No code added on main compares a version with CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION (now 9). The US compatibility recertification (Recertify a published US release for a newer policyengine-us without a rebuild #1366) reads only target names, metadata and registry families from the diagnostics it carries, plus the top-level build.post_export_scoring block; schema 9 keeps all of them, and it copies the parent's file byte for byte. (Its tests use a schema 6 parent, so no test runs it on a schema 9 parent.)
• Labels that main adds:
  • NZ target references now carry hierarchies (773 rows, 98 of them active calibration rows: Stats NZ 2025 regional population by age band and Census 2023 tenure by region; plus 67 active territorial-authority pre-calibration rows). In Chronicle's source packages at the NZ feed commit 8f4cfd57, every geography, dimension and value those rows select has one label, so NZ needs no label choice. NZ transport builds are not activated yet either (the spec still has parked references).
  • UK: of the 22 new national references, the ONS single-year ages and DfE childcare counts select the same source concepts and periods as existing targets, the MHCLG single-adult-discount count selects the group-by value e92000001, so it carries a geography dimension value ("England"), which the UK choices cover, and the five OBR duties add accounting_basis values: "Accrued basis" is labelled only in the EFO receipts package at the UK feed commit 272c987; "Cash basis" also appears, with the same spelling, in three HMRC bulletin packages new in that re-pin, which no UK target reads.
  • US: the feed re-pin to cf9ece2 adds three facts (BEA series B4923C, 2023 to 2025, for the ESI stage) and keeps the packages behind the three US choices, which still apply.
  • Checked by reading the source packages (re-review, 2026-10-26): the NZ counts and labels above, the UK additions, and every package and Chronicle code change in abf0921b..272c987 (UK), 5c4912aa..8f4cfd57 (NZ) and 51aa40f..cf9ece2 (US); no identifier gains a second label.
  • Not checked: the re-pinned UK and US feeds and the NZ feed were not rebuilt here, so a new spelling difference in a package not listed above would show only in a build, which refuses it and names each spelling.
• Local checks at 235e68e78: the 12 test files this PR changes: 1,028 passed, 8 skipped, 0 failed; ruff check . clean; ruff format --check clean on the PR's changed Python files; tools/ci_test_plan.py verify ok; uv lock --check ok. The full suite was left to CI.
• CI, first run (235e68e78): every job passed except both engine-free legs, which stopped at test_graph_parity_pins.py. calibrate.adam@1 hashes the source of microcosm.calibrate.diagnostics, which this PR changes, so the calibrate parity pins were stale; this came from the PR's own change, not the merge, and CI had not run on the PR before.
• Re-pin (fb4655deb): tools/graph_parity_repin.py calibrate on the one pinned platform (arm64/darwin/py3.14); the direct call's output bytes are unchanged, only the implementation hash and node key move. Locally that test file went from 2 failed, 8 passed, 1 skipped to 10 passed, 1 skipped (Python 3.13) and 11 passed (3.14). Eleven test files that pin the hashes of the kernels this PR changes (graph parity, calibrate kernels, the transport and UK graphs, the spec-engine loader and coverage) pass with it: 288 passed, 1 skipped.
• CI, second run (fb4655deb): every job passed except engine-us (3.14), which reached the job's 90-minute timeout in the engine workflow tests with no failed test (GitHub reports it as cancelled). The same job passed on the first run in 65 minutes, and fb4655deb changes only a microcosm-graph test fixture, so it needs a re-run, not a fix.

Re-review, 2026-10-26: the diagnostics steps' node keys

The two graph kernels this PR changed now publish the schema 9 document that catalog_calibration_diagnostics builds with the country's label choices, but neither node's key covered that code:

• diagnostics.calibration@1 (transport) hashed microcosm.diagnostics.schema and the calibrate diagnostics module, but not microcosm.diagnostics.catalogs, microcosm.calibrate.hierarchy_labels or microcosm.calibrate.geography_constants (the UK choices read UK_GEOGRAPHY_ID_TO_LABEL).
• The UK full build's calibrated node (uk.full.gates.calibrated) bound none of those, nor the schema module whose CalibrationDiagnosticsV9 dump now decides its output bytes.

So an edit to a chosen label (for example the Belgian "Country total", or a US choice retired after a re-pin) left both node keys unchanged, and a graph run on an existing store with resume="auto" (the default) would reuse diagnostics carrying the old labels.

Fix (51d0967c8):

• diagnostics.calibration@1 also hashes the catalogs, label-choice and geography-constant modules. The node is downstream of the solve, so only it and what reads it get new keys.
• uk_terminal_only_code_hash, which only the calibrated node binds (its terminal_code parameter), also hashes those three modules and the schema module. As with the high-value homes code (UK local surface before the Budget build: high-value homes OBR leg, per-vehicle fuel flag, Northern Ireland energy share and housekeeping (#1278) #1306), the shared code hash of the two UK kernels is unchanged, so later label edits never re-key the preflight node or the dense solve.

82f85c708 corrects docs/transport-graph-kernels.md, which still listed the diagnostics output as microcosm.diagnostics.calibration v8, and notes the key coverage there and in the changelog fragment.

Problem

Schema 8 copies every provider, category, geography and dimension label onto every target row. Microcosm checks that the facts inside one target agree and, since #1336, that a US registry gives each geography ID one label; nothing checks dimension values across targets, and other countries only warn. Published files therefore give one identifier two labels:

• US national populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z: us:statutes/26/62#adjusted_gross_income under_1 (106 targets / 2), all (47 / 53), age 85_plus (51 / 1). The US feed pin of that release (Chronicle 51aa40f) and the current one (cf9ece2: 51aa40f plus a guard-value parsing fix and the BEA B4923C package, neither of which changes a label) still produce all three.
• US ACS local …-766b7c7-…: 51 states "Maine" / "ME" (fixed in the producer by Give each US geography one label across target sources #1336).
• UK (measured from the dashboard's published trees, whose target detail rows carry each row's schema 8 hierarchy): Chronicle's local facts carry the area as a dimension named geography whose value ID is the lower-case GSS code, labelled with each publisher's name. The October 2026 dense release microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe gives 88 areas two names (61 E06 unitary authorities "Hartlepool UA" / "Hartlepool", 22 Welsh W06, one each E08, S12, E12, plus constituencies E14001581 "Weston-Super-Mare" / "Weston-super-Mare" and W07000112 "Ynys Mon" / "Ynys Môn"); the October national releases give one ("Yorkshire And The Humber" / "Yorkshire and The Humber"). The geography nodes themselves have one label per ID in every current UK release.

Design: diagnostics schema 9

Schema 9 (CalibrationDiagnosticsV9) is schema 8 with each hierarchy label declared once in a top-level hierarchy_catalogs object and referred to by ID from the rows. Every other field is schema 8's.

{
  "schema_version": 9,
  "hierarchy_catalogs": {
    "providers":   [{"id": "obr", "label": "Office for Budget Responsibility"}],
    "categories":  [{"id": "obr.efo_receipts", "label": "Economic and fiscal outlook receipts", "provider_id": "obr"}],
    "geographies": [{"id": "K02000001", "label": "United Kingdom", "level": "country"}],
    "dimensions":  [{"id": "obr.efo_line", "label": "Economic and fiscal outlook line",
                     "values": [{"id": "income_tax", "label": "Income tax"}]}]
  },
  "targets": [{
    "name": "obr.income_tax@2025", "target_name": "obr.income_tax", "...": "every other schema 8 field",
    "hierarchy": {
      "category_id": "obr.efo_receipts",
      "geography_id": "K02000001",
      "dimensions": [{"id": "obr.efo_line", "value_id": "income_tax"}],
      "target": {"id": "obr.income_tax", "label": "Income tax receipts"}
    }
  }]
}

• Geographies are keyed by ID with one label and one level (each geography ID has one level in every published US file and UK tree).
• Dimensions list their values; a row's value_id is looked up under its own dimension. A row's dimensions keep producer order.
• Providers and categories move into catalogs too. Their labels have never varied, but once the row shape changes it costs nothing, the file then mirrors the country contracts' authoring catalogs (hierarchy.providers, hierarchy.categories), and a category names its provider once, so a category under two providers cannot be written. A row names only its category.
• The target's own label stays on the row; a target is one row.
• Catalogs are arrays, so a repeated ID stays visible (a JSON object keeps the last duplicate key silently).
• The model refuses an ID listed twice, a category whose provider is not listed, and a row that refers to an unlisted category, geography, dimension or value (looked up under its own dimension), or lists a dimension twice.

Change

• Models (microcosm.diagnostics.schema): CalibrationDiagnosticsV9, TargetDiagnosticV9, HierarchyCatalogs and its entries, CatalogHierarchyReference. CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION is now 9; producers keep assembling schema 8 (CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION); HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS is {8, 9}; parse_calibration_diagnostics reads 6 to 9. CalibrationDiagnosticsV8 is unchanged (its count checks moved into a shared function; the committed v8 JSON Schema regenerates byte for byte).
• Catalogs (microcosm.diagnostics.catalogs): catalog_calibration_diagnostics builds the catalogs from the targets, in first-use order, and raises HierarchyLabelConflictError listing every identifier the rows label two ways (or a geography placed at two levels), each spelling with its first target, unless a HierarchyLabelChoices names the label. A choice applies only where the rows disagree, and every disagreement it settles is logged as a warning naming the identifier, the spellings and the label written. expand_calibration_diagnostics gives back schema 8.
• Writer: publishes schema 9; a refusal returns DiagnosticsWriteFailure with the new code label_conflict, writes nothing and logs every conflict (the outcome message is shortened to 500 characters). As before, a diagnostics failure does not fail the dataset build; the release manifest records failed.
• Label choices (microcosm.calibrate.hierarchy_labels), applied by the calibrate writer and the transport diagnostics kernel from the registry's country (UK, US and Belgium; see the next three sections).
• Other producers: the UK writer (write_uk_calibration_diagnostics), the transport diagnostics kernel and the UK graph's final validation kernel publish the same schema 9 document; the two kernels raise on a conflict, as they already raise on an invalid schema 8 document.
• Readers: the release contract treats 8 and 9 alike wherever it handled the current version, and an available declaration may name 8 or 9 (failed: expected 8 or 9). The UK local scorer and the UK dense assembler accept both.
• JSON Schema: calibration_diagnostics.v9.schema.json is generated from CalibrationDiagnosticsV9, ships in the wheel beside the v8 file (checked with uv build --wheel), and lists the model rules it cannot express (CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS: schema 8's, with the row-hierarchy rule replaced by catalog-identity and reference rules). tools/generate_calibration_diagnostics_json_schema.py writes or checks both files, or one with --schema-version.
• Docs: docs/calibration-target-hierarchy.md (new schema 9 section), the agent guide, the transport kernel doc, the root, diagnostics, calibrate and data READMEs, and changelog.d/1341-diagnostics-schema-9-catalogs.changed.md.

What refusal means for the UK, and the handling

A catalog holds one label, so without handling the writer would refuse today's UK files: the dense release (88 values) and the national releases (1 value). UK builds would still finish, but their releases would carry calibration_diagnostics: failed, and UK release workflows stop when diagnostics are unavailable.

The least risky handling chooses the area's label from what Microcosm already treats as authoritative, only for the geography dimension and only where the rows disagree:

1. the payload's own geography label for that code (the label the dashboard already shows on the area's geography node);
2. else UK_GEOGRAPHY_ID_TO_LABEL (the UK, nations and English regions);
3. else the ONS April 2023 names in local_authority_names.json (added by the UK build's writers).

Checked on the published UK trees: all 88 dense-release values and the national one resolve; in all 89 the file's geography label equals one of the spellings, and for the 87 that the catalogs cover (86 dense, 1 national) it equals the catalog's label (for example "Hartlepool", "Kingston upon Hull, City of", "Yorkshire and The Humber"); the two constituencies, which no Microcosm catalog covers, take the file's own geography label ("Weston-super-Mare", "Ynys Môn"). The September 2026 local candidates (staging, before Microcosm gave each geography one label) still have two constituency geographies with two labels and would be refused; no current build produces them.

A UK geography (not a dimension value) with two labels is resolved only by the catalogs; a constituency there is refused, because Microcosm has no constituency name catalog.

The three US values

The current US feed pin still produces two spellings of three values, so a refusing writer would stop US national diagnostics. Each gets the label chosen in review on 2026-10-21; a US feed re-pin being prepared brings the same labels from Chronicle, after which these choices apply only to older feeds. As with Chronicle's REVIEWED_VALUE_LABEL_VARIANTS (chronicle#377), each choice covers only the two spellings reviewed, so a third spelling is refused, and each use is logged as a warning:

Value	Spellings in release 0fb05b6	Written as
us:statutes/26/62#adjusted_gross_income = under_1	"No adjusted gross income" (106), "No adjusted gross income and deficit" (2; chronicle#377 fixes the source)	"No adjusted gross income"
us:statutes/26/62#adjusted_gross_income = all	"All returns" (47), "All returns, total" (53)	"All returns, total"
age = 85_plus	"Age 85 plus" (51), "Age 85 and over" (1)	"Age 85 plus"

Real-file check: the public 0fb05b6 file is refused without choices (3 conflicts) and written as schema 9 with them, with one warning listing the three. The calibration dashboard (calibration-diagnostics#255) builds the same 23 tree files from that schema 9 file as from the schema 8 file with those three labels applied. The public 766b7c7 file is refused (its 51 state conflicts are what #1336 fixed in the producer).

The Belgian value

Each Belgian Chronicle package that the Belgian target contract (be/target_references.json) reads states its national figure as the group-by value measure = country_total and labels it with its own measure name. Microcosm copies that label into each target's hierarchy (ledger_targets._inherited_dimensions), so a Belgian calibration gives one value up to four labels. Without a choice, the writer refuses it, and because Belgium builds through the transport graph, whose diagnostics step (diagnostics.calibration@1) raises on a refusal, a Belgian build would have failed, not merely recorded failed diagnostics as an earlier version of this description said.

The value is now written as "Country total": the label chosen in review on 2026-10-22, and the label Chronicle's canonical register gives it (chronicle#377). Like the US choices, it applies only where the rows disagree, covers exactly the four package spellings (Chronicle registers the same four as reviewed variants, chronicle#265), so another spelling is still refused, and every use is logged as a warning:

Chronicle package (at 6d74e42)	Microcosm target	Role	Label for measure = country_total
spf_finances/pit_2023	spf_finances_pit_total	calibration	"Belgium PIT federal and local tax before withholding"
onss/contributions_2024	onss_employee_contribution_total	calibration	"Worker-borne personal social security contributions"
onem_rva/unemployment_2024	onem_unemployment_caseload	calibration	"Complete unemployment benefit recipients"
nbb/national_accounts_household_disposable_income_2024	nbb_household_disposable_income	validation	"Household disposable income gross"

All four are written as "Country total". The two published Belgian releases carry no schema 8 hierarchy, and no Belgian build was run here; the failure and the fix are shown by tests that write a Belgian-style file through the calibrate writer, which takes its choices the same way the transport kernel does (hierarchy_label_choices(registry country)).

Other countries and new variants

Country	Choice	Values covered
UK	a geography dimension value takes its area's label	88 dense-release areas, 1 national value
US	one reviewed label per value	AGI under_1, AGI all, age 85_plus
Belgium	"Country total"	measure = country_total

• New UK variants: Chronicle also registers UK spellings such as sex = female ("Female", "Females", "female") and property_type values. No current UK release has them on one value ID (checked in the published trees), so nothing is chosen for them; a build that brings two of them together is refused and its warning names them.
• That refusal is the point of schema 9: a new disagreement stops the diagnostics write with every conflicting spelling named, instead of reaching the dashboard.

Tests (red, then green)

Commit	What	Result
b08c64607	Tests only: test_hierarchy_catalogs.py (new), test_json_schema_v9.py (new), test_hierarchy_label_choices.py (new), and changes to the schema, JSON Schema, calibrate, UK diagnostics, transport kernel, US fiscal-refresh, UK scorer, dense assembler and release contract tests; the JSON Schema test helper picks the committed schema by the document's version.	In the five changed modules that import: 21 failed, 465 passed, 8 skipped. Failures: the scorer refuses and the dense assembler rewrites a schema 9 candidate, the US fiscal-refresh writer writes schema 8, the v9 JSON Schema functions and the tool's --schema-version do not exist. The other 7 changed modules fail to import the new names (ImportError).
eb758cc51	The implementation and docs; no test file changed	All the above pass (below)
a5945f9ec	Test only: test_hierarchy_label_choices.py gains the Belgian choice, a Belgian-style file written with "Country total" and one warning, and a fifth, unreviewed spelling refused; the tests of countries without choices stop using be	That file: 3 failed, 9 passed. Failures: hierarchy_label_choices("be") is empty; the Belgian file is refused (failed, not available); the unreviewed spelling is refused without the "(a spelling its reviewed list does not include)" marker, because no Belgian choice exists
5c6cb3116	The Belgian choice in microcosm.calibrate.hierarchy_labels, the hierarchy doc and the changelog fragment; no test file changed	That file 12 passed; with the writer's other test files (test_diagnostics.py, test_hierarchy_catalogs.py, test_transport_terminal_kernels.py) 96 passed, 0 failed
89041e00a	Test only: test_transport_terminal_kernels.py requires the diagnostics kernel's hash to include the catalogs, label-choice and geography-constant modules; test_uk_graph_terminal.py requires the UK terminal-only code hash to include those and the schema module, and the shared code hash of both UK kernels to exclude them (the existing high-value homes test now requires its two modules to be in that hash rather than to be all of it)	The two new tests fail; the changed existing test passes
51d0967c8	The two hashes (transport/terminal_kernels.py, uk_runtime/graph_terminal.py); no test file changed	Both pass; the 13 test files below and the kernel files pass
82f85c708	Documentation and comments only	The two touched test files: 52 passed

Existing test expectations changed in the red commit, on purpose: the written file is schema 9 while the in-memory payload stays schema 8; a non-finite value added to an already-validated model is refused as validation_error (the writer validates the schema 9 document it builds) instead of serialization_error; the contract's message says "publishes version 9".

At 51d0967c8 (Python 3.14, arm64/darwin): the 13 test files this PR changes, 1,062 passed, 8 skipped, 0 failed; 20 test files for the two changed kernels, the transport and UK graphs, the Orrery contract and reconstruction, the calibrate kernels and graph parity (including test_graph_parity_pins.py and test_acceptance_h_parity.py, which executes the calibrate graph and compares its bytes and key with the re-pinned values), 1,999 passed, 0 failed; ruff check . clean; ruff format --check clean on the 29 changed Python files; tools/ci_test_plan.py verify ok; uv lock --check ok; tools/generate_calibration_diagnostics_json_schema.py --check current for both files.

At 5c6cb3116: ruff check . clean; ruff format --check clean on the two changed Python files. The rest of this list was run at eb758cc51.

At eb758cc51, locally (before CI ran on this PR):

• every test file that touches the diagnostics code paths (41 files, found by searching the test tree for the writer, parser, version constant, JSON Schema helper and label choices): 2,263 passed, 13 skipped, 0 failed;
• ruff check . clean; ruff format --check clean on the changed files; tools/ci_test_plan.py verify ok; uv lock --check ok (uv.lock unchanged); tools/generate_calibration_diagnostics_json_schema.py --check current for both files.

On #1298's previous head (c3244bfb4, before it merged main with #1336), two ACS local tests failed here: the ACS local tool still labelled state population targets "CA" beside the IRS SOI label for the same state, which schema 9 refuses. With #1336 in the base they pass. So this PR must not land on a base without #1336.

Environment variables

No workflow, deployment or configuration change, and no new environment variable is read.

Notes

• Releases already published keep their schema 8 files; readers here and in the dashboard read both.
• Unpushed local commits were amended while implementing (test corrections and two follow-ups from the coordinator's notes: reviewed spellings, the chosen US labels and the warning log). The pushed history is three red-then-green pairs (schema 9, the Belgian choice, then the node keys), each red commit test-only, plus the merge of main, the calibrate re-pin and one documentation commit.
• Nothing was merged, marked ready or dispatched.

🤖 Generated with Claude Code