[juaristi22]

Summary

Closes #419.

This emits the eight council tax band columns of NRS's dwelling estimates by 2022 data zone. They feed the Scottish band weights in PolicyEngine/microcosm#931's geography-first draw. In the meantime, PolicyEngine/microcosm#1519 reads them straight from the registered workbook bytes; it can vendor these facts once this merges.

• The measures. Both record sets, nrs.dwellings_data_zone_2025.2024 and nrs.dwellings_data_zone_2025.2025, gain eight measures beside nrs_dwellings_total, in its shape: unit count, aggregation sum, expected cell type number, and a header guard on row 5.

  Column	Header guard	Measure	Concept
  G	Council Tax band:\nA	nrs_dwellings_band_a	nrs.dwellings_band_a
  H	Council Tax band:\nB	nrs_dwellings_band_b	nrs.dwellings_band_b
  I	Council Tax band:\nC	nrs_dwellings_band_c	nrs.dwellings_band_c
  J	Council Tax band:\nD	nrs_dwellings_band_d	nrs.dwellings_band_d
  K	Council Tax band:\nE	nrs_dwellings_band_e	nrs.dwellings_band_e
  L	Council Tax band:\nF	nrs_dwellings_band_f	nrs.dwellings_band_f
  M	Council Tax band:\nG	nrs_dwellings_band_g	nrs.dwellings_band_g
  N	Council Tax band:\nH	nrs_dwellings_band_h	nrs.dwellings_band_h

  The guards match the header cells exactly, line break included. Each source_column_id is a slug (council_tax_band_a …). The SLC, Gambling Commission and CTAXBASE packages do the same for headers that contain a line break.

• How the file was edited. 265fb7e committed no generator for this package, and none exists elsewhere. The file round-trips byte for byte through yaml.safe_dump(sort_keys=False, allow_unicode=True, width=88). So the measures were added by loading it, setting both record sets' measures and dumping it again. The diff is the 16 measure blocks (208 lines); no row changed.

• The workbook is unchanged: sha256 5d61db5faf7e81451d838e5d394b8e0682bbf501341c1220aea4a78082ceb033, 1,223,284 bytes.

• The check. test_scottish_data_zone_council_tax_bands_sum_to_each_zone_total builds the package's facts and checks four things:

  • each of the 7,392 zones on both sheets has all nine measures;
  • the eight bands add up to nrs.dwellings_total in every zone;
  • Culter - 01's published bands (row 6) come out in column order;
  • the facts pass the consumer contract.

• Size. 118,272 new facts (7,392 zones × 8 bands × 2 sheets), kept on both sheets as the issue asks; the 2024 sheet alone would be 59,136. The bundle grows by 5.7%. The CI bundle step peaked at 3.4 GiB for 1.74M facts on 5 Nov.

Bundle test counts. I built single-source bundles of nrs-dwellings-data-zone-2025 at bundle year 2023 from main's package and from this branch's. They show:

• facts: 14,784 → 133,056 (+118,272);
• by_source.nrs and by_entity.dwelling: +118,272;
• calendar_year:2024 and calendar_year:2025: +59,136 each;
• no new geography, period or entity key, and no duplicate key or warning.

tests/test_chronicle_bundle.py now expects:

• fact_count and the row count: 2,062,484 → 2,180,756;
• nrs: 359,094 → 477,366;
• dwelling: 590,264 → 708,536;
• calendar_year:2024 and calendar_year:2025: +8 × 7,392 each.

Chronicle Governance

• Approved Chronicle agent role:
  • ledger-source-ingestor
• Deterministic checks run:
  • Source-package validation: chronicle validate-package nrs-dwellings-data-zone-2025, at --year 2025 and --year 2023. Both are valid: 18 measures, 14,784 rows and 133,056 source records, with no errors or warnings.
  • Source-cell preservation, selector guards and lineage: chronicle build-suite nrs-dwellings-data-zone-2025 --year 2025 is valid. It gives 133,056 facts, lineage coverage 1.0, and every agent-acceptance check passes. Its one warning, no_concept_alignments, is also raised by main's version of the package. chronicle validate-facts and validate-source-cells on its output are valid, with no missing labels, lineage or provenance.
  • Consumer-contract validation: the build-suite consumer facts, and validate_consumer_fact_contract in the new test.
  • Raw-facts boundary: each band is read from its publisher cell, with no scale, divisor, rounding or derived value. The test checks the bands' sum; Chronicle stores no sum.
• LLM judge verdicts:
  • ledger-source-fidelity: pending
  • ledger-contract: not applicable (no schema, identity or contract change)
  • ledger-boundary: pending

Tests

Run locally (Python 3.14.8):

• uv run pytest -q tests/test_chronicle_uk_local_2025.py -k council_tax_bands_sum: passed.
• Since 78d4c27, -k 'council_tax_bands_sum or band_header_guards': 17 passed. The new cases are test_scottish_data_zone_band_header_guards_reject_drift, one per sheet and band. Each puts the next band's header in the band's row-5 cell and expects that band's column-header guard to fail.
• uv run ruff check on the touched test files: clean.
• The chronicle validate-package, build-suite, validate-facts and validate-source-cells runs and the single-source bundles described above.

Left to CI: the full surface suite and tests/test_chronicle_bundle.py.

Disclosure note

Every new fact is a published NRS area aggregate. Nothing comes from survey records.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review of 1d72f0a

I checked the 16 new measures against the NRS workbook (downloaded again from the manifest's source_url), ran the new test, compared every emitted band fact with its cell, and broke two guards on purpose to see them fail.

Nit

1. tests/test_chronicle_uk_local_2025.py:110: the new test covers the happy path only. The guards do reject drift. When I moved nrs_dwellings_band_a from column G to H, build_facts raised expected column header 'Council Tax band:\nA', got 'Council Tax band:\nB'. When I dropped the line break from a guard, it raised too. But no test pins this for the 16 new guards. If a guard were later loosened or removed, nothing would fail until a shifted column happened to break the sum check. A small drift test in the style of test_wff_header_guards_reject_drift (tests/test_nz_targets.py:418) would cover it: shift one band's column and expect ValueError matching column header. This is optional.

Checked and fine

• Source pin: the live file at https://nrscotland.gov.uk/media/43tfx3zo/dwelling-estimates-by-2022-data-zones-24-25.xlsx has sha256 5d61db5f…b033 and is 1,223,284 bytes, the same as db/data/nrs/dwellings_data_zone_2025/manifest.yaml and the in-repo copy. The PR does not touch the manifest. Like most manifests (62 of 473 have one), it has no license field.
• Layout: on sheets 2024 and 2025, row 5 holds Council Tax band:\nA to \nH in columns G to N. Column F is dwellings per hectare (not emitted), and there is nothing after N. The guards match these cells exactly.
• Zones: each sheet has 7,392 rows, from row 6 to row 7397. Every code matches S01\d{6} (S01013482 to S01020873) and none repeats. All cells are integers, no zone has zero dwellings, and the bands sum to column E in every zone. The Scotland totals are 2,749,150 dwellings (2024) and 2,763,896 (2025).
• Emitted values: all 118,272 band facts equal their workbook cells, with no mismatch. Each one has unit count, aggregation sum, period calendar_year 2024 or 2025 (December stock, as the sheet titles say), geography vintage dz_2022 and a source_record_id of the form nrs.dwellings_data_zone_2025.<year>.<zone>.nrs_dwellings_band_<x>. validate_consumer_fact_contract passes.
• Conventions: band concepts as separate measures under the record set's existing record_set_spec_id, next to the total, match the shape of scotgov/council_tax_bands_2025 and voa/council_tax_stock_by_lad_2025. The slugged source_column_id matches theirs too.
• Bundle arithmetic: 2,062,484 + 118,272 = 2,180,756. For nrs, 359,094 + 118,272 = 477,366, and for dwelling, 590,264 + 118,272 = 708,536. The 8 × 7,392 = 59,136 increase per calendar year also checks out.
• pytest tests/test_chronicle_uk_local_2025.py -k council_tax_bands_sum passes locally (Python 3.13 venv). I did not run tests/test_chronicle_bundle.py locally, so the bundle counts are checked by arithmetic only.

CI at 1d72f0a: Chronicle bundle and Chronicle surface are still running.

Verdict: the emitter matches the source. One optional nit.

🤖 Generated with Claude Code

[juaristi22]

Thanks, Vahid. Nit 1 is in 78d4c27.

test_scottish_data_zone_band_header_guards_reject_drift now pins all 16 guards. For each sheet (2024, 2025) and band (A to H), it puts the next band's header in that band's row-5 cell, as if the columns had shifted; band H gets A's. It then expects a ValueError from that band's own selector on its column header.

The workbook is parsed once per session and shared with the band-sum test, so the 16 cases add about 30 s. All 17 pass locally.

🤖 Generated with Claude Code