UK facts for chronicle#357, #356, #339, #326 and #358: stamp taxes, HBAI, Move to UC, OBR geography, UC by payment indicator, NI and LSOA energy, gas years, rural-urban lookups, UV404 by output area, residence type by age

.github/workflows/ci.yml

@@ -29,12 +29,28 @@ jobs:
     # Kill a hung run instead of letting it hold a slot for GitHub's 360-minute
     # default. Successful runs on main-based branches took 76-147 min on Oct 7-8
     # (p50 137); the spread is runner speed, which varied 1.6x on one commit.
-    timeout-minutes: 210
+    # The bundle test alone took 84 min for 1.03M facts on Oct 14's fastest run.
+    # chronicle#356 and #339 grow the bundle by about 53%, which puts a slow
+    # runner near 210 min, so the limit is 300.
+    timeout-minutes: 300
+    # Source artifacts archived in R2 only (manifest storage.local: false), such
+    # as DESNZ's 50-56 MB LSOA energy workbooks (chronicle#356), are not in Git.
+    # A build that needs one fetches it from its manifest source_url, checks the
+    # pinned sha256 and caches it; the cache step below keeps it between runs.
+    env:
+      LEDGER_SOURCE_ARTIFACT_FETCH: "1"
 
     steps:
       - name: Check out repository
         uses: actions/checkout@v4
 
+      - name: Cache R2-only source artifacts
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/policyengine-chronicle/source-artifacts
+          key: source-artifacts-${{ hashFiles('db/data/**/manifest.yaml') }}
+          restore-keys: source-artifacts-
+
       - name: Install uv
         uses: astral-sh/setup-uv@v5
 
@@ -46,6 +62,13 @@ jobs:
       - name: Install dependencies
         run: uv sync --locked --all-extras
 
+      # Fetch the R2-only artifacts packages read (DESNZ's LSOA workbooks) before
+      # any test, from their source_url with retries and sha256 checks, so a
+      # publisher outage or a replaced download fails here by name rather than
+      # part-way through a long build.
+      - name: Fetch R2-only source artifacts
+        run: uv run python scripts/fetch_r2_only_source_artifacts.py
+
       - name: Lint Chronicle surface
         run: >
           uv run ruff check
@@ -58,10 +81,15 @@ jobs:
       - name: Test UK calibration source additions
         run: uv run pytest -q --durations=30 tests/test_chronicle_uk_national_2025.py
 
+      # GNU time reports the run's peak memory ("Maximum resident set size"): the
+      # merged bundle peaked near 15.6 GB in a local build, against the runner's
+      # 16 GB.
       - name: Test merged Chronicle bundles
-        run: >
-          uv run pytest -q --durations=30 tests/test_chronicle_bundle.py
-          --basetemp=/tmp/chronicle-bundle-ci
+        run: |
+          timer=""
+          if [ -x /usr/bin/time ]; then timer="/usr/bin/time -v"; fi
+          $timer uv run pytest -q --durations=30 tests/test_chronicle_bundle.py \
+            --basetemp=/tmp/chronicle-bundle-ci
 
       - name: Upload failed bundle reports
         if: failure()
@@ -79,11 +107,24 @@ jobs:
     name: Chronicle surface
     runs-on: ubuntu-latest
     timeout-minutes: 210
+    # Source artifacts archived in R2 only (manifest storage.local: false), such
+    # as DESNZ's 50-56 MB LSOA energy workbooks (chronicle#356), are not in Git.
+    # A build that needs one fetches it from its manifest source_url, checks the
+    # pinned sha256 and caches it; the cache step below keeps it between runs.
+    env:
+      LEDGER_SOURCE_ARTIFACT_FETCH: "1"
 
     steps:
       - name: Check out repository
         uses: actions/checkout@v4
 
+      - name: Cache R2-only source artifacts
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/policyengine-chronicle/source-artifacts
+          key: source-artifacts-${{ hashFiles('db/data/**/manifest.yaml') }}
+          restore-keys: source-artifacts-
+
       - name: Install uv
         uses: astral-sh/setup-uv@v5
 
@@ -95,6 +136,13 @@ jobs:
       - name: Install dependencies
         run: uv sync --locked --all-extras
 
+      # Fetch the R2-only artifacts packages read (DESNZ's LSOA workbooks) before
+      # any test, from their source_url with retries and sha256 checks, so a
+      # publisher outage or a replaced download fails here by name rather than
+      # part-way through a long build.
+      - name: Fetch R2-only source artifacts
+        run: uv run python scripts/fetch_r2_only_source_artifacts.py
+
       - name: Test Chronicle surface
         # --durations lists the slowest tests so a slowdown is visible in the
         # log instead of only in the job's total time.

.gitignore

@@ -11,3 +11,11 @@ venv/
 db/data/hmlr/price_paid_data_*/pp-*.csv
 db/data/hmlr/uk_hpi_full_file_*/UK-HPI-full-file-*.csv
 db/data/ons/nspl_*/NSPL_*.zip
+# DESNZ's LSOA domestic energy workbooks (chronicle#356) are 50-56 MB each; their bytes
+# are archived in R2 and fetched by sha256 from the publisher when a build needs them
+# (LEDGER_SOURCE_ARTIFACT_FETCH=1 in CI).
+db/data/desnz/lsoa_domestic_*_consumption_2024/LSOA_domestic_*_2010-2024.xlsx
+# Raw-only registrations of chronicle#356 that no build parses and that are 9-15 MB:
+# the DESNZ MSOA workbooks and the ONS 2021 rural-urban workbook. R2 holds them.
+db/data/desnz/msoa_domestic_*_consumption_2010_2024/MSOA_domestic_*_2010-2024.xlsx
+db/data/ons/ruc2021_lookups_march_2026/260331rucallsupplementarytables.xlsx

changelog.d/326.added.md

@@ -0,0 +1 @@
+Add Scotland's Census 2022 UV404 by 2022 output area for chronicle#326: all occupied households and the published 'Private rented: Total' for 46,363 output areas, from NRS's public output-area table, in place of the unrecoverable ward tenure-by-bedrooms table-builder export. Cells NRS prints as '-' emit no fact.

changelog.d/339.added.md

@@ -0,0 +1 @@
+Add Universal Credit households by Westminster constituency, by local authority and by constituency and number of children, January 2025 to May 2026, with Payment Indicator (Yes, No) as a dimension (chronicle#339). They replace the three all-claims extracts of 7 October 2026, whose packages are retired and whose preserved responses stay under `db/data`.

changelog.d/356-gas-year.fixed.md

@@ -0,0 +1 @@
+Add the `gas_year` period type for DESNZ's gas year (mid-May to mid-May since 2018, under its year label) and stamp the GB sub-national gas rows with it instead of calendar_year. DESNZ publishes no exact GB gas-year dates; Northern Ireland's gas rows carry the dated years DESNZ's methodology gives. No value changes (chronicle#356). The v4 consumer-fact schema's period-type description now lists `gas_year`, so the v4 schema digest moves from `cc9efb90…` to `d953306d…`: consumers that pin it (Microcosm's `chronicle_feed.json`) update it, and `policyengine_chronicle` no longer loads v4 artifacts built under the old digest.

changelog.d/356.added.md

@@ -0,0 +1 @@
+Add local energy, area classification and bus facts for chronicle#356: DESNZ sub-national electricity (2009 to 2024) and gas (2015 to 2024) consumption for Northern Ireland, every measure and district; domestic meter counts and consumption by LSOA and Scottish data zone for 2023 and 2024, whose 50-56 MB workbooks are archived in R2 only and fetched by sha256 in CI; DfT BUS04ai total operating revenue for Scotland, Wales and Great Britain; and raw-only registrations of the DESNZ MSOA/intermediate-zone workbooks and the ONS 2021, Scottish Government 2022, NISRA and DEFRA 2011 rural-urban classification lookups. An opt-in `xlsx_read_only_used_range` parser streams only a workbook's named sheets.

changelog.d/357-obr-table-4-9-geography.fixed.md

@@ -0,0 +1 @@
+Stamp OBR EFO table 4.9's DWP lines with the geography DWP's benefit expenditure tables give them: Great Britain (with people resident overseas) for housing benefit, incapacity benefits, Pension Credit, statutory maternity pay, Universal Credit, State Pension and jobseeker's allowance; England and Wales for DLA and PIP, Attendance Allowance, Carer's Allowance and Winter Fuel Payment. Port the table's two Northern Ireland social security rows, stamped Northern Ireland. Child benefit stays United Kingdom. No value changes (chronicle#357).

changelog.d/357.added.md

@@ -0,0 +1 @@
+Add facts for Microcosm's microcosm#1095 ports (chronicle#357): HMRC Annual Stamp Tax Statistics 2024 to 2025 (the 11 March 2026 corrected tables), Tables 8a and 8b by price band and buyer type and Table 6b's higher rates on additional dwellings, for 2023-24 and 2024-25; the EFO March 2026 sheet 3.7 SDLT lines by property type and rate (the EFO publishes no split by type of buyer); and DWP Households Below Average Income, FYE 2025 release, median equivalised income and relative low income before and after housing costs for FYE 1995 to FYE 2025, with the FYE 2025 95% confidence intervals from the release's data-table ZIP. Add DWP Move to Universal Credit households sent a migration notice, by combination of legacy benefits and move status, cumulative from July 2022 to June 2026, from Stat-Xplore (official statistics in development). A `zip_ods_used_range` parser reads ODS members of a ZIP.

changelog.d/358.added.md

@@ -0,0 +1 @@
+Add census usual residents by residence type and age with 90 and over split out, for chronicle#358: ONS Census 2021 residents in households (UR_HH) and communal establishments (UR_CE) by single year of age, by sex and for all persons, for England, Wales and the English regions, from the Census API (an `ons_census_observations_json_rows` parser); NISRA Census 2021 residence type by single year of age, by sex and for all persons, for Northern Ireland, its districts and its 2024 constituencies; and NRS UV101a at Scotland level, NRS's finest residence-type table, whose top band is 85 and over.

chronicle/artifacts.py

@@ -38,6 +38,7 @@
     "nz": frozenset({"ird", "mbie", "msd", "stats_nz", "treasury"}),
     "uk": frozenset(
         {
+            "defra",
             "desnz",
             "dft",
             "dwp",

chronicle/bundle.py

@@ -52,6 +52,8 @@
 UK_BUNDLE_SOURCES = (
     "desnz-energy-trends-domestic-electricity-2026",
     "desnz-energy-trends-domestic-gas-2026",
+    "desnz-lsoa-domestic-electricity-consumption-2024",
+    "desnz-lsoa-domestic-gas-consumption-2024",
     "desnz-monthly-annual-road-fuel-prices-august-2026",
     "desnz-need-england-wales-2023",
     "desnz-need-england-wales-2024",
@@ -63,7 +65,9 @@
     "desnz-qep-gas-unit-fixed-costs-2026",
     "desnz-road-transport-fuel-consumption-2024",
     "desnz-subnational-electricity-consumption-2024",
+    "desnz-subnational-electricity-consumption-northern-ireland-2024",
     "desnz-subnational-gas-consumption-2024",
+    "desnz-subnational-gas-consumption-northern-ireland-2024",
     "desnz-weekly-road-fuel-prices-september-2026",
     "dfc-ni-pension-credit-statistics-may-2026",
     "dfc-ni-state-pension-statistics-may-2026",
@@ -74,6 +78,7 @@
     "dfi-ni-public-transport-statistics-2025-26",
     "dft-bus01-passenger-journeys-2025",
     "dft-bus0415-fares-index-2026",
+    "dft-bus04i-operating-revenue-2025",
     "dft-bus05i-revenue-support-2025",
     "dft-nts-vehicle-ownership-2024",
     "dft-nts0303-mode-trips-2025",
@@ -89,6 +94,9 @@
     "dwp-esa-caseload-payment-type-phase-may-2018-march-2026",
     "dwp-hb-claimants-client-type-tenure-accommodation-type-september-2025-february-2026",
     "dwp-hb-claimants-client-type-tenure-january-2023-february-2026",
+    "dwp-hbai-summary-fye2025",
+    "dwp-hbai-uncertainty-fye2025",
+    "dwp-mtuc-households-legacy-benefit-move-status-july-2022-june-2026",
     "dwp-pension-credit-amount-band-type-february-2023-march-2026",
     "dwp-pension-credit-gender-type-february-2023-march-2026",
     "dwp-pension-credit-region-type-partner-age-february-2023-march-2026",
@@ -100,9 +108,9 @@
     "dwp-state-pension-region-type-gender-february-2023-march-2026",
     "dwp-uc-childcare-element-march-2021-may-2026",
     "dwp-uc-deductions-march-2025-february-2026",
-    "dwp-uc-households-by-constituency-children-january-2025-may-2026",
-    "dwp-uc-households-by-constituency-january-2025-may-2026",
-    "dwp-uc-households-by-local-authority-january-2025-may-2026",
+    "dwp-uc-households-by-constituency-children-payment-indicator-january-2025-may-2026",
+    "dwp-uc-households-by-constituency-payment-indicator-january-2025-may-2026",
+    "dwp-uc-households-by-local-authority-payment-indicator-january-2025-may-2026",
     "dwp-uc-households-carer-entitlement-payment-indicator-january-2023-may-2026",
     "dwp-uc-households-children-april-december-2025",
     "dwp-uc-households-children-child-entitlement-april-december-2025",
@@ -144,6 +152,7 @@
     "hmrc-spi-income-bands-2023-24",
     "hmrc-spi-income-by-area-2023-24",
     "hmrc-spi-income-by-region-2023-24",
+    "hmrc-stamp-tax-statistics-2024-25",
     "hmrc-tax-free-childcare-march-2026",
     "hmrc-vat-firm-sector-targets-2024-25",
     "hmrc-vat-firm-targets-2024-25",
@@ -165,15 +174,23 @@
     "nisra-census2021-households-lgd",
     "nisra-census2021-households-pcon24",
     "nisra-census2021-ms-f04-communal-residents-ward",
+    "nisra-census2021-residence-type-age-persons-lgd14",
+    "nisra-census2021-residence-type-age-persons-ni",
+    "nisra-census2021-residence-type-age-persons-parlcon24",
+    "nisra-census2021-residence-type-age-sex-lgd14",
+    "nisra-census2021-residence-type-age-sex-ni",
+    "nisra-census2021-residence-type-age-sex-parlcon24",
     "nisra-census2021-tenure-lgd",
     "nisra-mye-2025-lgd-single-year-age",
     "nisra-pcon24-population-by-age-2024",
     "nithc-annual-report-accounts-2024-25",
     "nrs-census2022-households-ukpc24",
     "nrs-census2022-uv101a-council-area",
+    "nrs-census2022-uv101a-scotland",
     "nrs-census2022-uv101a-ukpc24",
     "nrs-census2022-uv113-household-composition-country",
     "nrs-census2022-uv404-tenure-council-area",
+    "nrs-census2022-uv404-tenure-output-area",
     "nrs-census2022-uv407-central-heating-council-area",
     "nrs-dwellings-data-zone-2025",
     "nrs-households-data-zone-2025",
@@ -226,6 +243,14 @@
     "ons-census2021-ts041-households-pcon24",
     "ons-census2021-ts046-central-heating-ltla",
     "ons-census2021-ts054-tenure-lad",
+    "ons-census2021-ur-ce-age101-persons-ctry",
+    "ons-census2021-ur-ce-age101-persons-rgn",
+    "ons-census2021-ur-ce-age101-sex-ctry",
+    "ons-census2021-ur-ce-age101-sex-rgn",
+    "ons-census2021-ur-hh-age101-persons-ctry",
+    "ons-census2021-ur-hh-age101-persons-rgn",
+    "ons-census2021-ur-hh-age101-sex-ctry",
+    "ons-census2021-ur-hh-age101-sex-rgn",
     "ons-consumer-trends-current-price-2026",
     "ons-employee-workplace-pensions-summary-2024",
     "ons-families-households-2025",

chronicle/core.py

@@ -34,11 +34,18 @@
 # value COLUMNS
 # by the academic year's END year ("2024/25" -> 2025); that is source-layout
 # naming only and must not leak into fact periods.
+# gas_year is DESNZ's sub-national gas year (chronicle#356), neither a calendar nor
+# a fiscal year. The value is the publisher's year label: from 2016 the year the gas
+# year opens (since 2018 mid-May to mid-May, so 2024 is mid-May 2024 to mid-May
+# 2025), and for 2015 and earlier the closing year of an October-to-September year.
+# Great Britain's gas years publish no exact dates; Northern Ireland's are dated in
+# DESNZ's methodology (15 May to 15 May from 2018), and PeriodCoverage carries them.
 ALLOWED_PERIOD_TYPES = {
     "calendar_year",
     "tax_year",
     "fiscal_year",
     "academic_year",
+    "gas_year",
     "month",
     "quarter",
     "week",