[anth-volk]

Closes #1509.

Summary

Chronicle main writes every consumer fact as chronicle.consumer_fact.v5 since PolicyEngine/chronicle#377 merged as 44eebe037 on 2026-11-03. v5 is the v4 row plus one optional top-level field, package_dimension_value_labels (dimension id, then value id, then text). That field holds a source package's own wording of a value, and only where it differs from the label the row exports.

Before this PR, a v5 feed would already have loaded. Microcosm's declared row-contract list (microcosm.build.chronicle_epoch) stopped at v4, but no code outside the tests read it. Review round 1 found this, and the maintainer decided that a feed whose pin or rows name an undeclared row contract is refused, so a new Chronicle format is caught at the re-pin. Review round 2 found that the UK and US builds checked only the pin's hand-typed list, not the rows they compile; they now check the rows too. Review round 3 found the same gap in the US target-parity generator, which is fixed here as well. This head also merges main, which re-pinned the US feed to Chronicle 41b2fb2b (#1506, still v3) and the NZ feed onto v5 (#1507); both pins pass the new checks.

This PR:

1. Declares v5 beside v1 to v4, matching Chronicle's own CONSUMER_FACT_ROW_CONTRACTS.
2. Refuses pins naming an undeclared row contract: the UK and US pin loaders, chronicle_feed_identity and validate_chronicle_feed_pin.
3. Refuses rows on an undeclared row contract, or rows the pin's list does not describe, everywhere Microcosm holds a feed to a reviewed pin: through validate_chronicle_feed_pin for full pins (transport kernels, NZ and Belgian generators), and through the new validate_pinned_row_contracts in the UK and US code that holds feeds to committed pins by digest.
4. Checks the shape of package_dimension_value_labels on the facts a hierarchy target selects. Microcosm never takes a label from that field, so each value keeps the one label Chronicle exports.

No feed pin moves in this PR. After the merge of main, the UK and Belgian pins name v4, the US pin v3 and the NZ pin v5. Every committed pin and every resource restating one names declared row contracts, and a test walks every package's JSON resources to check that.

Where Microcosm checks a feed's row contracts after this PR

Code	Feed it reads	Check on the pin	Check on the rows
ledger_artifact.validate_chronicle_feed_pin (through load_ledger_consumer_artifact(feed_pin=…)): the transport kernels (money_kernels, target_kernels directly; population_kernels, takeup_kernels through compile_target_surface), and the NZ and Belgian generators	an artifact with a manifest	its list must be declared	the rows' list must equal the pin's and the manifest's, then be declared
ledger_artifact.chronicle_feed_identity: reference factors, money normalization, the transport composer	—	its list must be declared	—
uk_runtime/chronicle_feed.load_uk_chronicle_feed, us_runtime/chronicle_feed.load_us_chronicle_feed	—	the committed list must be declared	—
UK full and national target loaders (uk_runtime.full_targets), tools/vendor_uk_ledger_facts.py: require_uk_chronicle_feed_row_contracts	the full artifact, held to the committed pin by digest	—	declared; on the committed feed exactly the pin's list; on a feed admitted by --allow-unpinned-feed, declared only
tools/certify_uk_release_cut.py: require_uk_chronicle_feed_row_contracts	the full artifact, held to the operator's --ledger-facts-sha256 and --ledger-manifest-sha256, not to the committed pin	—	declared; exactly the pin's list only when the operator gives the committed feed; on any other feed, declared only, with no override flag involved
tools/generate_uk_target_references.py, tools/generate_uk_local_target_references.py, tools/build_uk_ledger_compile_parity_signed_differences.py, tools/build_uk_desnz_constituency_energy_facts.py: require_uk_chronicle_feed_subset_row_contracts	a subset of the pinned artifact, or the rows the tool parses	—	declared, and each on the pin's list (a subset need not carry every listed contract)
tools/build_us_fiscal_refresh_release.py (_check_us_ledger_feed_row_contracts, after _check_committed_us_ledger_feed_pin) and tools/build_us_target_parity_manifest.py (after its digest check, before compiling or writing): require_us_chronicle_feed_row_contracts	the US bare feed, held to the committed pin by digest; the full pin check cannot run on a bare feed	—	declared; on the committed feed exactly the pin's list; on a feed admitted by the release tool's --allow-unpinned-feed, declared only (the parity generator has no such flag)
ledger_artifact._load_fact_rows without a pin	any	—	none: feeds loaded by path for diagnostics and experiments, and bare feeds no reviewed pin covers, carry their ids unchecked

The UK reference generators now read their facts file first, so the check runs before any contract or crosswalk processing. The DESNZ builder loads the UK pin module by path, as it already loads the energy module, so a first run still needs no UK country spec.

Not changed, with the reason. None of these holds its feed to a reviewed pin, which is the rule this PR enforces; their outputs are used beside the pinned feed, whose rows the builds above check.

• us_runtime/release_gate_preflight previews the registry from the operator's --ledger-facts-sha256, not the committed pin; the release tool itself refuses.
• The US ACS local release tool and county surface tool, and the scoring, audit and diagnostic tools, load feeds by path for diagnostics and experiments.
• The UK builders that read the feed's rows by path into committed resources:
  • tools/build_uk_high_value_home_stock.py records the manifest's identity; a fingerprint test, which runs only when the feed is present, compares its inputs with the pinned feed;
  • tools/build_uk_brma_rent_quantiles.py and tools/build_uk_scotland_brma_la_overlap.py check each source file's digest per fact.
• The UK childcare take-up fit tools (tools/fit_uk_childcare_takeup.py, tools/fit_uk_uc_childcare_takeup.py) take an operator-supplied digest.

What changed, module by module

• microcosm.build.chronicle_epoch:
  • CHRONICLE_CONSUMER_FACT_SCHEMA_VERSION_V5, declared in the chronicle epoch as v4 is.
  • require_accepted_consumer_fact_schema_versions(versions, *, subject). It refuses a value that is not a list or tuple, and raises UndeclaredConsumerFactSchemaVersionError (a ValueError) when an entry is undeclared.
    • A Chronicle-namespace id: the message says to declare it in microcosm.build.chronicle_epoch (DECLARED_IDENTITIES) first.
    • An id outside Chronicle's namespaces, such as arch.consumer_fact.v1: the message says it is not a Chronicle row contract and that chronicle_epoch cannot declare it.
    • Both messages list the declared row contracts.
• microcosm.build.ledger_artifact:
  • validate_chronicle_feed_pin runs the check after its exact comparisons, so a pin that differs from its artifact still reports the differing field.
  • chronicle_feed_identity runs it too.
  • New consumer_fact_schema_versions_of(rows), which LedgerConsumerArtifact.fact_schema_versions now uses.
  • New validate_pinned_row_contracts(fact_schema_versions, *, pinned, subject, subset=False).
  • The loader keeps rows exactly as published, so the new field is kept on the parsed rows. The module docstring, the loader comment and the fact_schema_versions docstring give the reasons that hold for loads without a pin.
• uk_runtime/chronicle_feed.py: the loader checks the committed list. New require_uk_chronicle_feed_row_contracts and require_uk_chronicle_feed_subset_row_contracts.
• uk_runtime/full_targets.py: both loaders check the rows after the digest checks, before either compiler runs.
• us_runtime/chronicle_feed.py: the loader checks the committed list. New require_us_chronicle_feed_row_contracts, the exact-or-declared check both US tools call.
• microcosm.build.ledger_targets: _validate_chronicle_hierarchy_labels calls _validate_package_dimension_value_labels on each selected fact.
  • The field, when present, must be an object of dimension id to an object of value id to non-empty text: the shape the v5 schema gives it and dimension_value_labels alike.
  • Any other shape raises LedgerHierarchyMetadataError, the error a missing Chronicle label raises.
  • This check walks every entry, where Microcosm's check of dimension_value_labels reads only the selected entries and takes any non-null value as text.
• us_runtime/congressional_district_vintage.py: comments in _translated_fact_shell and _state_total_proxy_source_district_fact record that both copy a source row whole. After a US re-pin onto v5, a derived district row would keep any package wording keyed by the source row's value ids; nothing reads the field.
• Tools: the UK vendoring tool, release certifier, both reference generators, compile-parity tool and DESNZ constituency builder, and the US fiscal-refresh release tool and target-parity generator, as in the table above.
• Docs:
  • docs/calibration-target-hierarchy.md: the v5 rule under "Chronicle inheritance".
  • docs/uk-chronicle-feed-repin.md: set the pin's list from the manifest, and which code refuses what.
  • docs/us-chronicle-feed-repin.md: a section on re-pinning onto a newer row contract, with the release tool's and parity generator's row checks and the district note. It sits beside Re-pin the US Chronicle feed to add Federal Student Aid's loan portfolio tables #1506's new re-pin section, which the merge keeps.
• Changelog: changelog.d/1509-chronicle-consumer-fact-v5.added.md.

Where the shape check runs

It runs when Microcosm compiles a target that has a calibration hierarchy, on the facts that target selects, as the dimension_value_labels check does. Callers treat its error as they treat a missing Chronicle label. target_reference_authoring re-raises it. The UK national and local compilers report the reference as unsupported, which stops the build year and is recorded for validation periods. A malformed field on a fact no target selects is not refused. The empty forms {} and {dimension: {}}, which the schema accepts, compile.

The fixtures

packages/microcosm-build/tests/fixtures/chronicle_consumer_fact/ holds Chronicle's own schemas and sample rows on either side of #377. SOURCE.md records the paths and digests.

• v5, from Chronicle 44eebe0378cd4aa708e283837e5a4ed1f7b18ef0, unchanged at main 28064d0:
  • consumer_fact.v5.schema.json, sha256 acd33a091cd8d192be5d264e3e94c13e00afb6fd1972e57d4576ac55a54f6970;
  • consumer_facts.v5.jsonl: IRS SOI Table 1.1 for tax year 2023. These rows label the AGI value all "All returns, total" and keep the package's "All returns" in the new field.
• v4, from Chronicle 1a3f2c41005ed0138fd82fe08a9d87201061e198 (main just before Stale TANF degenerate-input exclusion on main aborts every default US build before calibration (Build I fix never ported) #377):
  • consumer_fact.v4.schema.json, sha256 d953306d…, the digest the UK and Belgian pins carry;
  • consumer_facts.v4.jsonl: the same facts as real v4 rows, labelled "All returns".
• In both row files the raw-storage bucket name is replaced by the placeholder example-raw-bucket, and keys are sorted. Each row file validates against its own schema.

Tests: before and after each change

I ran only the tests this PR adds or modifies, by node ID, from a script that refuses an empty list, with HF_HUB_OFFLINE=1. CI runs the rest of the suite. Paths are under packages/microcosm-build/tests/engine_free/.

Round 1 (declaring v5, and the shape check): at 3ca0435a1, 11 of 20 cases failed; at 5dd751a40, all 20 passed. The failures:

• the accepted-set and declaration tests in shared/test_chronicle_epoch.py;
• the v5 load test in shared/test_ledger_artifact.py;
• the 8 malformed-shape cases in shared/test_ledger_targets.py, each DID NOT RAISE.

Round 2 (pins naming an undeclared row contract): at e70084b12, 12 of 36 cases failed, each DID NOT RAISE; at deadfb7f9, all 36 passed. The failures:

• the full-pin refusals, the chronicle_feed_identity refusals and the not-a-list case in shared/test_ledger_artifact.py;
• both cases of shared/test_transport_target_kernels.py::test_target_kernels_refuse_an_undeclared_row_contract;
• the UK loader cases in uk/test_uk_chronicle_feed.py and the US loader cases in us/test_us_chronicle_feed.py.

Round 3 (the rows UK and US builds read, and the review's points): at 30213d718, 20 of 55 cases failed; at 049b30d1a, all 55 passed. Every failure is on an assertion. These failed before the change:

• uk/test_uk_full_targets.py::test_full_inputs_hold_the_rows_row_contracts_to_the_pin, cases undeclared, not_in_the_pin and unstamped: DID NOT RAISE;
• uk/test_uk_full_targets.py::test_the_national_role_holds_the_rows_row_contracts, cases undeclared, not_in_the_pin and undeclared_unpinned_feed: the compile was reached;
• us/test_us_fiscal_refresh_builder.py::test_feed_row_contract_check_refuses_rows_the_pin_or_epoch_lacks, 4 cases, and test_main_checks_the_feed_row_contracts_before_compiling_targets;
• shared/test_target_reference_authoring.py::test_uk_reference_generators_refuse_rows_on_an_undeclared_row_contract, the national and local generators: the v6 refusal was absent from stderr;
• uk/test_uk_chronicle_feed.py::test_uk_feed_tools_check_the_row_contracts_of_the_rows_they_read, 6 tools;
• shared/test_ledger_artifact.py::test_feed_identity_refuses_a_foreign_row_contract_without_advising_a_declaration.

These passed before the change too, as guards or because the behaviour already held:

• shared/test_ledger_artifact.py::test_feed_pin_refuses_rows_carrying_a_row_contract_the_pin_does_not_list: the full pin check already refused this;
• the committed-resource walk, which now reads every package's JSON resources at any depth;
• the 13 uk/test_uk_full_targets.py tests that use the prepared fixture, which gained the row-contract lists;
• uk/test_uk_full_targets.py::test_the_national_line_drops_local_only_rows_before_reconciliation, whose fakes gained the lists;
• the DESNZ tool test, whose toy pin now lists v4;
• the round 2 tests rerun on the new message;
• the stale-docstring epoch test.

The green commit also gives row-contract lists to the fakes that stand in for a loaded artifact or a pin. That covers 6 tests in uk/test_uk_national_graph.py, 22 in uk/test_uk_rowwise_national_role.py and the corridor test in us/test_us_fiscal_refresh_builder.py. I ran them all locally:

• The national graph tests and the corridor test pass.
• Ten rowwise cases fail locally on staging handling ('skipped' == 'uploaded', UPLOAD_PENDING, staged_dataset_handover). All ten fail the same way without this change: review round 3 ran them at the round 2 head and at the merge base, and I ran three of them before this change. The cause is HF_HUB_OFFLINE=1, which the local runs set: with the hub offline, uk_runtime.rowwise_staging keeps a requested remote stage local and records the staged dataset as skipped, while these tests expect an upload to their stand-in hub. CI does not set the variable, and runs them.

Round 4 (the US target-parity generator, after merging main): at 9cc83b496, 3 of 7 cases failed; at 5fdbc1ef5, all 7 passed.

• us/test_us_chronicle_feed.py::test_parity_generator_holds_the_rows_row_contracts_to_the_pin, cases undeclared, not_in_the_pin and pin_lists_fewer, failed on the AssertionError the stubbed manifest build raises when the rows reach it.
• uk/test_uk_chronicle_feed.py::test_subset_check_requires_each_row_contract_on_the_pin (4 cases) covers the UK subset rule directly, and passes before and after: a declared row contract the pin does not list is refused; a strict subset of the pin's list, or rows with no stamp, pass.

On the merged head, the committed-resource and pin tests pass with the new NZ (v5) and US (41b2fb2b, v3) pins. So do the round 3 tests (55 cases) and the US corridor test.

ruff check and ruff format --check pass on every changed Python file. The committed UK and US pins load through the changed loaders.

Environment variables

None. The changed code reads no environment variable, and no workflow changes. Running the tests locally needs only HF_HUB_OFFLINE=1.

Follow-up (not in this PR)

• The US and Belgian label-choice tables in microcosm.calibrate.hierarchy_labels settle spellings that Chronicle now resolves to one label per value. They may become unnecessary once those feeds are re-pinned onto v5. This PR leaves them unchanged.
• Each country's re-pin onto Chronicle main moves its pin to ["chronicle.consumer_fact.v5"] and acd33a09…, set from the artifact, with the tests that restate the pin. The UK re-pin is the next pull request.

🤖 Generated with Claude Code