UK facts for chronicle#357, #356, #339, #326 and #358: stamp taxes, HBAI, Move to UC, OBR geography, UC by payment indicator, NI and LSOA energy, gas years, rural-urban lookups, UV404 by output area, residence type by age

[juaristi22]

Summary

Draft. Answers five Microcosm requests in one change. Source notes, decisions and parity checks are in docs/data-sources/uk-feed-326-339-356-357-358.md. The merged bundle is re-pinned from a local full build (see Bundle).

#357, facts for the microcosm#1095 ports

• HMRC Annual Stamp Tax Statistics 2024 to 2025, the corrected tables of 11 March 2026: hmrc-stamp-tax-statistics-2024-25, 352 facts.
  • Tables 8a and 8b: residential and non-residential SDLT transactions, property value and receipts by price band, for individuals, other buyers (non-natural persons) and all buyers, 2023-24 and 2024-25.
  • Table 6b: higher rates on additional dwellings, the transactions, value, receipts and surcharge element, all gross of refunds.
  • Every fact is stamped hmrc:england_and_northern_ireland, carries a £40,000 transaction-value constraint and names the corrected edition.
• The EFO publishes no split of SDLT by type of buyer. Its only SDLT breakdown is sheet 3.7, by property type and rate, which policyengine-uk#2322 combines with HMRC's held shares. Its five SDLT lines join obr-efo-receipts-march-2026 (+35 facts).
• Move to UC (item 3): dwp-mtuc-households-legacy-benefit-move-status-july-2022-june-2026, 80 facts from Stat-Xplore.
  • Households sent a migration notice, by each of the 19 combinations of legacy benefits and each move status (claimed, did not claim, in progress), with DWP's all-households row and totals. Great Britain.
  • Cumulative over notices from July 2022 to June 2026, the table policyengine-uk-data#534 used. For all households, claimed over claimed plus did not claim (1,822,423) is 86.74%, the rate policyengine-uk-data#534 draws; DWP's 87% divides by every notice (1,822,966), 86.71%. DWP's 18 August 2026 UC quarterly release reports the same release: 1.8 million households sent a notice.
  • Every fact is marked official statistics in development and carries the extraction date as its vintage.
• OBR table 4.9's DWP lines now carry DWP's geography, from DWP's benefit expenditure tables (Notes 1, 3 and 4). No value changes.
  • Great Britain, with residents overseas, for the core DWP lines.
  • England and Wales for DLA and PIP, Attendance Allowance, Carer's Allowance and Winter Fuel Payment.
  • Northern Ireland for the two NI rows, newly ported (+14).
  • Child benefit stays United Kingdom.
• HBAI, FYE 2025 release.
  • dwp-hbai-summary-fye2025: median equivalised income in 2024/25 prices and relative low income, before and after housing costs, FYE 1995 to FYE 2025 (124 facts).
  • dwp-hbai-uncertainty-fye2025: the FYE 2025 95% confidence bounds (8 facts), from the release's data-table ZIP.

#339: UC households by area with Payment Indicator, January 2025 to May 2026, re-extracted from Stat-Xplore on 15 October 2026.

• By constituency (21,488 facts), by local authority (11,900), and by constituency and number of children (150,416).
• Each area-month carries "Yes" (an award paid) and "No" (a nil award). Their sum is the all-claims count: against the retired extract it agrees to within 9 households per area-month (11 per child band), with a median absolute gap of 2.
• The three all-claims packages of 7 October are retired, as UK local-surface facts for the 2025 calibration: household counts by nation and area, communal-establishment residents by age, mid-2025 population estimates, UC households by area for calendar 2025, and BRMA geography and rent inputs (PolicyEngine/microcosm#1123, PolicyEngine/microcosm#1090) #313 retired the May 2025 snapshots. Their preserved responses stay under db/data.

#356

• DESNZ Northern Ireland: electricity 2009 to 2024 (4,616 facts) and gas 2015 to 2024 (991), every measure and district. All-domestic lines use the GB package's concepts.
• gas_year period type. DESNZ publishes no exact GB gas-year dates, only "mid-May to mid-May".
  • GB gas rows move from calendar_year to gas_year: their identity changes, their values do not.
  • NI gas rows carry the dates in DESNZ's methodology.
• Domestic meters and kWh by LSOA and data zone, 2023 and 2024: desnz-lsoa-domestic-{gas,electricity}-consumption-2024, 338,080 facts.
  • Their 50-56 MB workbooks are archived in R2 only and git-ignored. CI fetches them from gov.uk before any test, retrying and checking each sha256; without them, the tests that need them skip.
  • The MSOA/IZ workbooks are raw-only.
• Raw-only rural-urban lookups:
  • ONS 2021: OA, LSOA, MSOA, LAD21 and LAD24;
  • Scottish Government 2022: data zones and output areas;
  • NISRA: SA2011 settlements and DZ2021 urban/mixed/rural;
  • DEFRA 2011 districts, the classification BUS01 cites;
  • ONS 2011 counties.
• DfT publishes no country split of bus fares or support. dft-bus04i-operating-revenue-2025 carries BUS04ai's total operating revenue for Scotland, Wales and GB (6 facts).

#326: nrs-census2022-uv404-tenure-output-area (87,816 facts). It carries all occupied households and "Private rented: Total" for 46,363 output areas, from NRS's public UV404 output-area CSV. The ward tenure-by-bedrooms export remains unavailable.

#358: residence type by age, with 90 and over split out.

• ONS Census API: 8 packages, 6,666 facts. Single years to 100+, households and communal establishments, by sex and for all persons, for England, Wales and the English regions.
• NISRA builder: 6 packages, 18,180 facts. Single years by residence type for NI, LGD14 and PCON24.
• NRS UV101a at Scotland level: 189 facts. NRS publishes nothing finer than 85 and over by residence type.

Closes #326, closes #339, closes #356, closes #357, closes #358.

Bundle

• Re-pinned from a local build. I built the full bundle at year 2023 locally (75 minutes, peak resident memory about 15.6 GB) and replayed tests/test_chronicle_bundle.py against it.
  • Facts: 1,030,204 → 1,579,263 (+549,059, exactly the new packages' facts net of the retired UC ones).
  • Packages: 380 → 405 (28 added, three retired).
  • Geographies: 20,549 → 102,613. The new ones are 46,363 output areas, 35,673 LSOA-level keys, 26 pre-2015 NI districts and two scopes.
  • Periods: 511 → 523: gas_year 2015 to 2024, and HBAI's FYE 1995 and 1996.
  • Warnings: 120 → 122. DESNZ and NRS spell two Scottish data zones differently, and both spellings are kept.
  • Semantic duplicates stay at 2,324.
  • Every per-source, per-period, per-geography and per-entity change is attributed in the test's comments.
• Caught by the build: the UV404 output-area package labelled its tenure axis differently from the other NRS tenure packages. It now uses theirs, with the same fact keys.
• Memory. CI's bundle job passed on run 38760171563 (2 h 36 min), so the bundle fits a standard runner. The local peak (15.6 GB, on macOS) is close to the runner's 16 GB, so the job now logs its peak with GNU time.

Review notes

• Rulings taken during the work:
  • LSOA facts are meters and kWh only, with MSOA/IZ raw-only.
  • UV404 at output area, two columns.
  • A gas_year period type rather than undated calendar years.
  • LSOA workbooks R2-only, fetched in CI.
• CI time. The bundle grows from 1.03M to about 1.58M facts (+53%).
  • On Oct 14's fastest main run, the bundle test alone took 84 minutes.
  • Scaled up, a runner 1.6x slower would reach about 210 minutes, so the bundle job's timeout goes from 210 to 300 minutes.
  • Each LSOA build-suite took about 10 minutes locally.
• Microcosm's re-pin. Merging this breaks nothing in Microcosm: every reader checks its pinned feed (735820c). Re-pinning to this feed needs the changes below; a Microcosm tracking issue is drafted and will be filed separately.
  • UC area selectors.
    • The households-by-area references name no dimensions, so each meets a Yes and a No fact.
    • The four child-band families list dimensions: ["number_of_children"], which must equal a fact's dimension names, so they match no fact and drop out silently.
    • Add payment_indicator to those lists and pin Yes.
    • The May 2025 legacy families (uk.local_geography.uc_households.*) leave the feed; Microcosm's selectors on main don't use them.
  • OBR table 4.9. All 14 references pin K02000001, set by rule in tools/generate_uk_target_references.py, so 12 match no fact until the generator pins their new stamps. The hand-written obr.state_pension diagnostic pins it too. Values don't change.
  • DESNZ energy.
    • Two vendor selections raise at re-vendoring until they pin period type, domestic_scope and the GB areas:
      • the region selection, which expects 112 rows, would match 228 once Northern Ireland's 116 country-level rows join;
      • #1286's local-authority selection pins calendar_year for gas.
    • The energy stage's gas-connected share selects by year value alone. It would divide gas-year meters by calendar-year electricity meters unless Microcosm declares an alignment.
  • Schema hash. consumer_fact_schema_sha256 moves to d953306d…: following 5360644, the v4 description now lists gas_year. period.type stays a free string. policyengine_chronicle no longer loads v4 artifacts built under the old digest.
• Move to UC choices:
  • The table is cumulative because DWP advises taking a statistic from the tabulation that carries it, rather than summing finer cells that each carry random error.
  • The period is month 2026-06, the latest notice month, with the July 2022 to June 2026 window in period_coverage.
  • "Legacy non-customer" and "Unknown/Missing" are zero in every cell, so they are not ported, as in the Winter Fuel package. Other zeros are kept.
  • Stat-Xplore serves only the current release, so earlier releases can't be recovered. Each later extract lands beside this one.
• UV404 - cells emit no fact. No NRS document defines the symbol. The arithmetic and the file (168,890 - cells, no literal 0) both point to zero, but Chronicle doesn't assert it, so 4,910 output areas (10.6%) carry no private-rented fact.
• Output areas carry their code as their name, because NRS names them by code only. This bends Emit geography.name on consumer facts so Microcosm's hierarchy can label sub-national geographies (constituencies, local authorities and regions) #266's rule for want of any publisher name.
• xlsx_read_only_used_range is opt-in. Read-only parsing matched the full parse on 31 of 34 sheet-restricted packages and on the LSOA gas workbook (947,540 cells, 22 s against 65 min). It differs on three existing packages, so none of those changes. It re-measures each sheet's used range rather than trust its <dimension> element; a synthetic test pins that.
• BUS04 keeps DfT's end-year label, as dft-bus05i-revenue-support-2025 does: 2025 is the year ending March 2025.
• R2-only artifacts. Besides the LSOA workbooks, the three raw-only workbooks no build reads are R2-only and git-ignored: the two MSOA files and the ONS rural-urban workbook, 9 to 15 MB each. R2 is the archive and needs credentials; builds read these files from gov.uk by sha256.
• Northern Ireland energy. Every NI measure's notes carry DESNZ's caution that NI is not directly comparable with GB (methodology sections 4.3 and 5.3). The domestic electricity notes add that NI files three domestic_scope rows under the concept GB files once. NI gas notes say that DESNZ's year dates overlap or leave gaps.
• HBAI FYE 2021 and FYE 2022 (8 facts) carry Note 5's caution on telephone-only interviewing.
• SDLT stays United Kingdom. The new sheet 3.7 SDLT lines keep the stamp of obr.sdlt (sheet 3.8), which they sum to, as OBR prints them in one UK table. Stamping SDLT England and Northern Ireland would have to move obr.sdlt with them, a fact Microcosm binds, so that's a separate decision.

Chronicle Governance

• Approved Chronicle agent roles:
  • ledger-source-ingestor for the packages and the parsers in chronicle/sources/**;
  • ledger-contract-maintainer for the gas_year period type in chronicle/core.py and the v4 schema's description.
• In neither role's allowed paths (the roles file is stale, as in Add UK high-value housing sources: OBR surcharge costing, ONS house prices by constituency, and Price Paid, UK HPI and NSPL registrations #349 and Stats NZ territorial-authority resident population, 30 June 2025 (NZ, G8-d2) #355):
  • chronicle/source_package.py, chronicle/bundle.py and chronicle/artifacts.py;
  • db/data/**, scripts/*, docs/* (other than schemas), changelog.d/* and packages/uk_local_2025_sources.md;
  • the tests/test_issue_*.py files and tests/test_policyengine_chronicle_schema.py;
  • .github/workflows/ci.yml (artifact fetch, cache, memory log and the bundle job's timeout) and .gitignore.
• Deterministic checks run:
  • validate-package and build-suite for all 31 new and changed packages: valid, with 0 agent-acceptance errors. The only warnings are concept_alignment_validation_skipped and no_concept_alignments.
  • publish-raw, then verify-raw: 37 of 37 new R2 objects match their pins.
  • inventory-artifacts over db/data: valid, 446 entries, 40 R2-only, no checksum mismatches.
• LLM judge verdicts (independent agents, at 4187f66):
  • ledger-source-fidelity: PASS.
    • It traced facts from every new package to their publisher cells using its own parsers (ODS, raw XLSX XML, JSON, CSV), and all nine sampled files match their sha256 pins.
    • Every spec guards a row header and a cell type, and no fact sums, divides or rounds cells.
    • Minor: HBAI uncertainty's two bound columns have no publisher headers, so a swap would not trip a guard. test_hbai_confidence_intervals_bracket_the_estimates would catch it.
  • ledger-contract: PASS.
    • gas_year joins the vocabulary that validate_fact enforces, and the v4 change is description-only, as in b714b3c and 5360644.
    • The gas and table 4.9 restamps keep every value and its lineage.
    • Its nits are fixed in cbd3423: the changelog notes the digest move, and the description says pre-2016 gas years keep closing-year labels.
    • It also caught the merged bundle test's lost pruning fixture, fixed in 2480f4a.
  • ledger-boundary: PASS.
    • Every value is one publisher cell times a declared scale, periods are labels rather than alignments, and missing cells stay missing.
    • Microcosm's decisions stay in Microcosm.
    • Two disclosed representation choices remain: the month stamp on Move to UC's cumulative count, and the United Kingdom stamp on the sheet 3.7 SDLT lines.

Tests

• New: tests/test_issue_{357,356,326,358}_*.py, covering:
  • published values and counts;
  • geography, period and edition stamps;
  • markers that emit no fact;
  • R2-only storage;
  • parser behaviour, including read-only/full parse parity.
• Rewritten for UC households by area: re-extract the constituency, local-authority and by-children series with Payment Indicator as a dimension (PolicyEngine/microcosm#1198) #339 in tests/test_chronicle_uk_local_2025.py:
  • each area package has a Yes and No fact for every area-month, with published values pinned;
  • Yes + No reproduces the retired all-claims cube within 15 households;
  • the child package crosses every band with the indicator;
  • the old aliases are gone.
• Targeted existing files (artifacts, source packages, aliases, UK local, labels, geography names, UC composition and elements, governance, YAML, core): 450 passed, 1 skipped. The first round, with the energy and UK national files, gave 527 passed and 1 skipped after the DESNZ manifest-count fix.
• Bundle test: test_build_bundle_writes_merged_consumer_contract passes against the local bundle, and the file's other 24 tests pass. The full surface suite runs in CI.
• Review round 1:
  • The two existing tests the new facts changed are updated: the OBR expenditure package's count (164) and GB gas's period type.
  • New tests cover the read-only parse of a sheet whose <dimension> understates it (with formula cells), and ONS labels against option codes.
  • The LSOA tests and the merged-bundle test skip with the reason when the R2-only workbooks are neither present nor fetchable.
  • The merged-bundle test keeps its prune_suite_intermediates fixture (2480f4a).
  • All touched files (energy, ports, census, UV404, schema, consumer contract, cells, source packages, artifacts, core, bundle other than the merged test, UK local, geography names, alias drift): 453 passed.
• uv run ruff check chronicle policyengine_chronicle db scripts tests: passed.

🤖 Generated with Claude Code

[vahid-ahmadi]

Automated review pass (Claude Code, high effort), round 1 at daf0e5bb6

Thanks María, this is a big, careful piece of work. Verdict: the data is right, but CI is red on two stale tests, and the PR body understates what Microcosm's re-pin will need. Fix item 1 and correct item 2's notes before merge; the rest are shoulds and nits.

Every value we checked matches its source cell, and the fact counts reconcile exactly with the PR body (+549,059 net; details at the end).

Must

1. The surface job fails on two existing tests that this PR changes the ground under (run 38760171563, 2 failed, 5,048 passed):

   • tests/test_chronicle_obr_council_tax_decomposition.py::test_obr_table_4_1_council_tax_decomposition_rows: assert 164 == 150. The 14 new NI rows in obr-efo-expenditure-march-2026 change the package's count.
   • tests/test_issue_270_energy_trends_subnational.py::test_subnational_packages_keep_units_reference_periods_and_weather_basis: assert {'gas_year'} == {'calendar_year'}. That's the GB gas restamp.

   Both need updating to the new truth. The bundle job was still running when I posted.

Should

2. Microcosm's re-pin needs more than the PR body says. Merging this doesn't break Microcosm now: every reader checks the pinned feed (uk/chronicle_feed.json, 735820c). But when Microcosm re-pins to this feed:

   • UC children selectors stop matching, rather than double-count. List-form dimensions selectors must match the fact's dimension set exactly (ledger_targets.py:4105-4109). The four dwp.uc.households_by_area_children_* targets in uk_population_targets.json (lines 1438, 1473, 1508, 1543) and the 2,528 generated constituency selectors use dimensions: ["number_of_children"]. The new facts carry {number_of_children, payment_indicator}, so these targets drop out with no error. Only dwp.uc.households_by_area (line 1409, no dimensions key) double-matches as described. The fix is to add payment_indicator to dimensions as well as pinning Yes.
   • The region energy selector fails. subnational_domestic_consumption_and_meters_regions (uk/ledger_fact_vendor_selections.json:523) pins only the four desnz.subnational.domestic_* concepts and geography_level: [country, region], and expects 112 rows. The NI packages add 116 country-level rows at N92000002 under those concepts, so it matches 228 and ledger_fact_vendoring.py:262 raises.
   • Table 4.9: 13 of 14 selectors stop resolving, not 12. The generated target_references.json pins K02000001 for 12 single lines plus obr.universal_credit. The obr.state_pension diagnostic (uk_population_targets.json:31243) pins it too. Regenerating won't fix this, because tools/generate_uk_target_references.py:481 takes geography from hard-coded rules. The doc (line 140) says these selectors "pin no geography", which contradicts the PR body; only the hand-written file fits that.
   • Open microcosm#1286 breaks once it re-pins past this. Its subnational_domestic_meters_local_authorities_2024 selector pins calendar_year 2024 and expects 697 rows; the 347 GB gas rows become gas_year. Dropping the period pin readmits NI's three electricity rows per district, and domestic_energy_targets.py (around line 368) then raises "two electricity meter rows". The two PRs need sequencing, and #1286 needs to pin domestic_scope and the GB geographies.

   I'd put these in the "Microcosm's re-pin" section and open a tracking issue on microcosm. No microcosm issue mentions UK facts for chronicle#357, #356, #339, #326 and #358: stamp taxes, HBAI, Move to UC, OBR geography, UC by payment indicator, NI and LSOA energy, gas years, rural-urban lookups, UV404 by output area, residence type by age #359, gas_year or payment_indicator yet.

3. NI and GB energy share concepts, but DESNZ says they aren't comparable. The methodology booklet (July 2026) says NI electricity "is not directly comparable" with GB (§5.3, p.35; financial years against GB's February to January). It says the same of gas (§4.3, p.33; profile marker against the 73,200 kWh threshold). The NI gas header cites §4.3; the NI electricity header doesn't cite §5.3. NI also files three domestic_scope values under one concept, where GB has only all_domestic, so a selector that doesn't pin domestic_scope pools them. I'd add both caveats to the measures' concept_evidence_notes, and soften doc lines 179-180 ("so Microcosm can split the UK Energy Trends level").

4. The bundle job's memory is unproven. The 15.6 GB peak was measured on macOS; ubuntu-latest on a public repo has 16 GB. A memory kill fails with no bundle report, on every later PR. Logging peak memory in the step (/usr/bin/time -v) would settle it. The 300-minute timeout itself looks right.

5. CI fetches the LSOA workbooks from gov.uk, not R2. source_package.py:3435-3466 and :3481-3491 call httpx.get(spec["source_url"]); storage.r2 is never read. The PR body says "R2-only and fetched by sha256 in CI", which suggests otherwise (the ci.yml comment is accurate). The sha checks and the cache are sound. But if DESNZ replaces the attachment or gov.uk is down once the cache has expired (7 days unused), every PR's surface job fails, and the bundle job fails partway through a long build. An R2 fallback, or an early download step with retries, would fix it.

6. Without download enabled, the LSOA tests fail rather than skip. Without LEDGER_SOURCE_ARTIFACT_FETCH, the new and touched tests give 164 passed and 2 failed (FileNotFoundError on the LSOA workbooks); test_chronicle_bundle.py too. High-value housing facts for the UK local calibration: HM Land Registry Price Paid Data, UK HPI, ONS house price statistics by constituency, the National Statistics Postcode Lookup, and the OBR High Value Council Tax Surcharge costing #343's tests skip with a reason (test_issue_343_uk_high_value_housing.py:518, 543).

   • Relatedly, build_uk_local_energy_bus_packages.py:676-683 reads the LSOA files straight from db/data. So the builder fails on a clean checkout even with download enabled.

7. The read-only XLSX parser silently truncates when a sheet's <dimension> understates it. In cells.py _source_cells_from_xlsx_read_only, the fallback runs only when the element is missing. On a synthetic 5×3 sheet with <dimension ref="A1:B2"/>, the full parse gives 15 cells and read-only gives 4, with no error. Declared cells past the cut-off still fail loudly (specs.py:524-530), but the stored source cells are short. Removing the fallback branch broke no test. The CI parity test uses the small NI gas workbook; the 947,540-cell LSOA parity in the PR body isn't pinned anywhere. Resetting and recalculating the dimensions always would close it. (The opt-in itself checks out: off by default, only the two LSOA packages use it, and a missing sheets list raises.)

8. The gas_year schema claim holds, but the schema's description is now stale. v4 period.type is a free string, and the sha stays cc9efb90…, which Microcosm pins. But the description (consumer_fact.v4.schema.json:121) still lists the period types ending at week. Your 5360644 (9 Sep) updated it and re-pinned when quarter and week were added. The same applies here, or note why not. Microcosm's _normalize_period_value (ledger_targets.py:3758) also has no gas_year entry.

9. HBAI FYE 2021 and FYE 2022 facts lack the pandemic caveat. The package header comment (hbai_summary_fye2025/source_package.yaml:22) mentions it. But each fact's concept_evidence_notes is the same text for every year. Sheets 1_2b and 1_3a cite Note 5 (telephone-only interviewing). Adding it to the 2020/21 and 2021/22 record sets (16 facts) means a per-year validation would see the caution.

Nits

10. UC households by area: re-extract the constituency, local-authority and by-children series with Payment Indicator as a dimension (PolicyEngine/microcosm#1198) #339 doc (uk-feed-326-339-356-357-358.md:314-315): the maximum gaps are right (9 and 11), but the constituency-by-children median is 1, not 2.
11. Base fact count: the doc (line 334) says the bundle grows "from 1,030,137". The PR body's 1,030,204 is the one that reconciles.
12. SDLT geography: the 35 new OBR sheet 3.7 SDLT facts are stamped United Kingdom, while HMRC's are England and NI. Sheet 3.7 lists LBTT and LTT on their own rows. Keeping the existing obr.sdlt stamp is defensible and disclosed (efo_receipts_march_2026/source_package.yaml:17-19). It sits oddly beside the same PR relabelling table 4.9 to its true coverage, though.
13. Move to UC: "86.7% of those with an outcome" is 1,580,761 / 1,822,423; DWP's 87% divides by all notices (1,822,966). Both round to 86.7%, but naming the denominator helps once "in progress" grows.
14. UV404 - cells: the CSV has 168,890 - cells and no literal 0, which supports reading - as zero more than the header says. 4,910 output areas (10.6%) end up with no private-rented fact. Worth one line in the header (line 24).
15. NI gas years overlap by a day (e.g. 2023 ends 2024-05-15 and 2024 starts 2024-05-15), following DESNZ's wording. NI electricity uses inclusive ends. A period_coverage note would help. The year-labelling switch (NI 2015 by closing year, 2016 on by opening year) is documented.
16. Tests that wouldn't catch a regression: writing option_id into the ONS label column passed test_issue_358, and one label-against-code assertion would catch it. Dropping formula capture in read-only mode passed too, because the fixture has no formulas.
17. Governance: chronicle/sources/** is in the ingestor's allowed paths, so crediting the parsers to the contract-maintainer is off. source_package.py, bundle.py, artifacts.py and ci.yml are in neither role's list. That's a stale roles file, as Add UK high-value housing sources: OBR surcharge costing, ONS house prices by constituency, and Price Paid, UK HPI and NSPL registrations #349, Stats NZ territorial-authority resident population, 30 June 2025 (NZ, G8-d2) #355 and others show, not this PR. AGENTS.md:27-29 asks for the judge verdicts before merge; with core.py changed, at least ledger-contract should run.

What looks right

• Values.
  • HMRC stamp taxes: all 352 match the corrected ODS, with bands summing to totals; 6b is gross of refunds.
  • OBR sheet 3.7 (35) and the 14 NI 4.9 rows match. DWP's notes back every 4.9 relabel in every year the table covers: CA from Sept 2018, DLA/PIP/AA from April 2020, WFP from 2024/25.
  • HBAI: all 124 summary facts and 8 CI bounds match, pre-2002/03 years are stamped GB, and the price base is right.
  • Move to UC (80), and UC by payment indicator: 632 × 17 × 2, 350 × 17 × 2 and ×7 bands. Disclosure control is documented.
  • NI electricity and gas match: all 5,607 cells, with zero mismatches and none omitted.
  • Every LSOA and data-zone cell for 2023 and 2024 matches, with nothing missing or extra.
  • UV404: 87,816 = 2 × 46,363 − 4,910, and the totals match.
  • Census residence type by age, with 90 and over split out, for England and Wales regions, Scotland and Northern Ireland areas (PolicyEngine/microcosm#1278) #358 ONS, NISRA (incl. 18 PCON24) and UV101a all match.
  • BUS04ai: all 6 match.
• Pins and periods. All 9 raw-only SHA pins verify, including the 3 R2-only files re-downloaded, and all are OGL. The GB gas restamp changes only period_type and notes; the values are unchanged.
• Builders and counts. All six builders reproduce the committed YAMLs byte-for-byte. Per-package fact counts net to exactly +549,059. Package arithmetic: 380 + 28 − 3 = 405.
• Smaller decisions. Making xlsx_read_only_used_range opt-in, the 300-minute timeout, and keeping both Scottish data-zone spellings are all sensible.

What I checked

• Setup: a fresh worktree at daf0e5bb6 in a new uv venv. I read the diff against main (non-data files in full, packages by builder and spot or full checks).
• Tests:
  • new and touched tests run with and without artifact download;
  • ruff check is clean;
  • eight mutations of the parsers, gas_year and the layout checks;
  • synthetic edge-case workbooks for the read-only parser.
• Counts: every new and changed package built at 2023 on head and on main.
• Data: values checked independently against the raw workbooks, the CSVs, the ONS JSON and the DESNZ methodology booklet.
• Microcosm: main's selectors and vendor code, plus #1286's branch, read for the re-pin effects.
• CI: the logs of run 38760171563.
• Not checked: I didn't run the full bundle build. So the geographies (102,613), periods (523), warnings (122) and duplicates (2,324) rest on your local run. I also couldn't get HMRC's December edition to confirm the March correction touched only column I.

🤖 Generated with Claude Code

[juaristi22]

Response to round 1, at cbd3423

Thanks Vahid. All of it is addressed except two:

• item 12, the SDLT stamp: kept, with the reasoning below;
• item 10, the median: I can't reproduce 1.

CI run 38813536509 is on the new head.

Must

1. Fixed (51e5257):
   • The council-tax decomposition test now counts 164 facts, the 150 plus the 14 NI rows.
   • The subnational gas test expects gas_year and the new notes.

Should

2. Agreed on all four, and checked against Microcosm main. #1286 merged after your review, so its local-authority selection is on main now.
   • The OBR count is 12 of the 14 generated references, plus the hand-written obr.state_pension diagnostic: 13 selectors. Child Benefit and TV licence fee stay UK.
   • The PR body's "Microcosm's re-pin" section is rewritten. A Microcosm tracking issue covering all of it is drafted and will be filed separately.
   • The source notes' "pin no geography" line was wrong. It's corrected in 4187f66.
3. Done (0a2fdab):
   • Every NI measure's notes cite DESNZ's comparability statements: §4.2 and §4.3 for gas, §5.3 for electricity, and §5.2 for electricity before 2015.
   • The domestic electricity notes flag that NI files three domestic_scope rows under one concept.
   • The NI gas header now cites §4.3 for the comparison (it cited §4.2).
   • The doc's "Microcosm can split the UK Energy Trends level" is replaced with the caveats.
4. The bundle job passed on run 38760171563 (2 h 36 min end to end), so it fits a standard runner. The step now logs peak memory with GNU time (44d4470).
5. Fixed (44d4470):
   • scripts/fetch_r2_only_source_artifacts.py runs before any test in both jobs. It fetches each R2-only workbook a package reads from its source_url and checks the sha256, so a gov.uk outage or a replaced file fails that step by name.
   • Downloads retry three times on transport errors and 5xx answers.
   • No public R2 endpoint exists, so an R2 fallback would need credentials in CI. I left that out.
   • The PR body now says what CI actually does.
6. Fixed (44d4470):
   • The LSOA tests and the merged-bundle test skip with the reason, as High-value housing facts for the UK local calibration: HM Land Registry Price Paid Data, UK HPI, ONS house price statistics by constituency, the National Statistics Postcode Lookup, and the OBR High Value Council Tax Surcharge costing #343's do.
   • Checked by moving the workbooks aside, with an empty cache and fetching off: 2 and 1 skipped.
   • build_uk_local_energy_bus_packages.py reads the workbooks through the new read_registered_artifact: committed, cached or fetched, always sha-checked.
7. Fixed (44d4470):
   • The streamed parse always re-measures the used range.
   • A synthetic workbook test covers it: a 5×3 sheet with formula cells and <dimension ref="A1:B2"/>. It fails on the old code (4 cells against 15).
   • Both LSOA packages regenerate byte-identical.
   • I didn't pin the 947,540-cell parity in CI, because the full parse of that workbook takes about 65 minutes. The pinned LSOA fact counts and byte-identical regeneration guard it.
8. Done (0631599, cbd3423), as in 5360644:
   • The v4 description lists gas_year, and the hash is re-pinned: cc9efb90… → d953306d…. The description also says gas years to 2015 keep their closing-year labels, and the changelog notes the digest move.
   • The v3/v4 equality test leaves that one description out.
   • The Microcosm re-pin updates its pinned hash, which is in the drafted tracking issue. So is _normalize_period_value. It treats a gas_year value as a plain year, which is enough for value matching.
9. Done (0a2fdab): the FYE 2021 and FYE 2022 facts carry Note 5's caution. That's 8 facts, four a year, rather than 16.

Nits

10. I get a median of 2. For constituency × child band × month, the absolute gap between Yes + No and the retired cell is 2 over all 75,208 cells. It's 0 in 28%, 1 in 19% and 2 in 17%; the signed median is 0. Restricted to cells with households, it's still 2. I kept 2 and stated the definition in the doc. How did you compute 1?

11. Fixed: 1,030,204.

12. Kept United Kingdom. The sheet 3.7 lines sum to obr.sdlt (sheet 3.8), which Microcosm binds at UK. Restamping 3.7 alone would split one OBR table's lines from its own total. Moving both to England and Northern Ireland is a separate decision to make with Microcosm. It's noted in the PR body.

13. Fixed: both denominators are named, 86.74% against DWP's 86.71%.

14. Added to the header: 168,890 - cells and no literal 0, and 4,910 output areas (10.6%) with no private-rented fact.

15. Added to the NI gas notes:

    • from 2018, DESNZ's years share the 15 May boundary;
    • 2016 and 2017 overlap by a month;
    • 2015 ends before 2016 begins.

16. Added:

    • label-against-code assertions in test_issue_358, which fail if codes are written into the label column;
    • formula capture, covered by the synthetic read-only test.

17. Governance corrected in the PR body:

    • the parsers are ingestor paths;
    • source_package.py, bundle.py, artifacts.py and ci.yml are in neither role's list.

    Judge verdicts, from independent agents at 4187f66, are all PASS:

    • ledger-source-fidelity traced a fact from every new package to its publisher cell with its own parsers.
    • ledger-contract agrees with the period-type and schema change and the restamps. Its two nits are fixed in cbd3423. It also found that my skip helper had separated the merged bundle test from its prune_suite_intermediates fixture, a CI disk risk, fixed in 2480f4a.
    • ledger-boundary found nothing aligned, imputed or reconciled.

    Details are in the PR body.

🤖 Generated with Claude Code

[vahid-ahmadi]

Automated review pass (Claude Code, high effort), round 2 at cbd3423

Thanks María. Verdict: all round-1 points are resolved and CI is green on all three checks. Good to merge once it's out of draft. The one remaining should (item A) can be fixed in the PR body or the tracking issue; it doesn't need a code change here.

Round-1 points

1. Fixed (51e5257). The surface run gives 5,051 passed, 4 skipped. The one extra test is the new read-only test.
2. Fixed, apart from new item A below. The "Microcosm's re-pin" section now covers the UC dimensions list, the 228 region rows, the 12 + 1 table 4.9 selectors and #1286's calendar_year pin, and the doc's "pin no geography" line is corrected. The tracking issue isn't filed yet. Please link it here when it is.
3. Fixed (0a2fdab). All 272 NI electricity and 124 NI gas measure notes carry DESNZ's "not directly comparable" caveat, and the three-domestic_scope warning is on the domestic electricity notes.
4. Settled. The bundle job logs a peak of 14,857,244 kB (about 14.2 GiB) with 0 swaps, so it fits. That leaves about 1.8 GiB of headroom, so growth of around 10% in the bundle would reach the runner's limit. It's worth keeping streaming build_bundle in mind for the next large feed PR.
5. Fixed (44d4470).
   • The fetch script runs right after uv sync in both jobs, before lint and tests.
   • With httpx mocked: 5xx and connect errors retry at 5, 20 and 60 s and then raise. A 404 fails at once. A wrong sha raises "checksum mismatch for …" and isn't cached.
   • In this run the workbooks came from the cache, so the gov.uk download path hasn't yet run in CI.
6. Fixed.
   • With fetching off and an empty cache, the 12 touched test files give 232 passed and 3 skipped, with reasons. With fetching on, test_issue_356 gives 25 passed.
   • The three touched builders regenerate byte-identical output through read_registered_artifact.
7. Fixed. The new synthetic test fails with daf0e5bb6's cells.py swapped in, and it also fails if formula capture is dropped. Not pinning the 65-minute LSOA parity in CI is fine.
8. Fixed. Both v4 schema copies list gas_year and hash to d953306d…. The equality test leaves out only period.type.description. The closing-year wording matches the data: NI 2015 is the only gas_year value at or before 2015.
9. Fixed. 8 record sets is right; my "16" overcounted.
10. You're right, and I withdraw this nit.
    • Why my figure was wrong: I took the absolute gap over all 86,088 cells of the retired cube. That is 633 constituency rows including "Unknown" × 8 child bands including the "Not available prior to April 2019" marker × 17 months. The marker band is zero on both sides (10,744 cells with a gap of 0), and neither it nor the Unknown area emits a fact. That pulled my median down to 1.
    • Recomputed over the cells that do emit facts: 632 × 7 bands (keeping "Unknown or missing") × 17 = 75,208. From the preserved responses I get your figures exactly: median 2; 27.9% at 0, 18.8% at 1, 16.6% at 2; signed median 0; maximum 11.
    • Restricted to cells with households, the median is still 2. The doc's wording is correct.
11. Fixed: 1,030,204.
12. Kept, and I agree. Restamping sheet 3.7 alone would split its lines from their own total, obr.sdlt. Moving both to England and Northern Ireland is a decision for Microcosm.
13. Fixed: 86.74% against 86.71%, with both denominators named.
14. Fixed: 168,890 - cells, no literal 0, 4,910 output areas (10.6%).
15. Fixed. I checked the dates in the NI gas notes: 2016 and 2017 overlap by a month, 2015 ends before 2016 starts, and the years from 2018 share the 15 May boundary.
16. Fixed. The label-against-code assertions fail when codes are written into the label column (I mutated rows.py:1521: 1 failed, 19 passed).
17. Fixed in the PR body. The roles are corrected, and all three judge verdicts PASS at 4187f66.

New

A. should: the digest move touches more of Microcosm than the UK feed. The old cc9efb90… is also pinned in packages/microcosm-build/src/microcosm/build/nz/chronicle_feed.json, and it's hard-coded in packages/microcosm-build/tests/engine_free/uk/test_uk_chronicle_feed.py. Any NZ bundle built after this merges will carry d953306d. changelog.d/356-gas-year.fixed.md:1 and the PR body's "Schema hash" bullet name only the UK feed, so the tracking issue should list all three.

• Also, policyengine_chronicle/consumer.py:173-185 raises on any digest it doesn't know. So anyone who reads bundles through the wheel must re-pin before upgrading. Microcosm has its own pin check and doesn't depend on the wheel, so nothing breaks there. The changelog mentions this only in passing; one plain sentence would do.
  B. nit: chronicle/source_package.py:3453 does int(file_key), so an R2-only manifest entry with a non-year key would crash the fetch script. All current keys are 2025.
  C. nit: tests/test_chronicle_bundle.py:323-333: the skip helper reads and sha-checks about 106 MB just to test whether the files are there. Defensible, just slow.

2480f4a is right: prune_suite_intermediates is back on test_build_bundle_writes_merged_consumer_contract.

What I checked

• A fresh worktree at cbd3423 with a new uv venv. I read the full diff daf0e5bb6..cbd3423 apart from the bulk YAML, plus your response and the current PR body.
• The touched test files with fetching off and an empty cache, and test_issue_356 with fetching on.
• The retry and checksum behaviour, with httpx mocked.
• Three mutations (old cells.py, no formula capture, ONS label/code swap), each confirmed to fail.
• The three builders regenerated, with git status clean afterwards.
• The median recomputed from both Stat-Xplore cubes over five cell sets.
• NI caveat counts, Note 5 record sets, gas_year values and dates, and both v4 schema hashes.
• Microcosm main searched for cc9efb90, and the issue and PR search for the tracking issue (none yet).
• CI run 38813536509: the bundle job's memory and fetch step, and the surface job's counts.
• ruff check is clean. ruff format --check drift in source_package.py exists on main too, so it isn't from this PR.

🤖 Generated with Claude Code