[juaristi22]

Not for merge. This is the data work for one external request (#1326), kept on its own branch. It adds a project tool and its tests. Nothing in the build changes.

What it does

python -m tools.uk_scotland_property_income --h5 … --registry … --out … reads a released UK national file and the national target registry of the build that produced it. It writes private outputs outside the repository, and refuses any output directory inside a git working tree. It runs these steps from #1326:

• A2 guard. policyengine-uk returns zero incomes, zero tax and almost no weight when asked for a year before the dataset's own. refuse_earlier_period stops that before any engine call.
• A3 back-cast (backcast.py). It inverts the engine's own forward step: each input is divided by the growth in uprating_indices.yaml that apply_single_year_uprating would multiply it by, and household weights move by ONS population growth.
  • The indices are OBR national aggregates per person. Band-level differences are left to the reweight.
  • Inputs the forward step leaves flat stay at 2024-25 levels, and the manifest lists them. Among them are a few small income-tax inputs, such as Gift Aid and losses brought forward. Council tax, rent and student-loan plans aren't inverted.
• A4 reweight (registry.py, measures.py, rake.py).
  • Facts. The SPI 2023-24 facts are read back from the registry as the target value before any cross-grain control, divided by its uprating_factor. They are checked against HMRC's printed Tables 3.7 and 3.11 (spi_2023_24_published.csv), and the run stops on a mismatch.
  • Bindings. Each fact is measured on the build's own binding from uk_population_targets.json:
    • bands on total_income_after_loss_relief;
    • property amounts counted for taxpayers only;
    • regional total income with no taxpayer filter.
  • What is reweighted. A bounded minimum-divergence reweight, with ratios in [0.25, 4], fits UK property income counts and amounts by band, plus Scottish taxpayer counts by band. Each region's household total is a soft target.
  • Scotland's total income is left out. Meeting it moved weight onto a few very high-income records and pulled Scotland's other untargeted income away from Table 3.11.
  • Stop rule. The run stops unless every fact and region total is within 0.5%. Stopping at the iteration limit with the facts met is accepted and recorded.
  • Income tax isn't a target. policyengine-uk#2453 (the savings starting-rate fix) isn't in the release's engine.
• A6 releases (bands.py, release.py).
  • Release 1 is the ten requested NSND bands. Release 2 is four unions of them, each edge the requested edge nearest a 2023-24 Scottish rate start.
  • Headline: taxpayers with taxable property income above zero (after the £1,000 allowance). Landlords wholly under the allowance are counted beside them in the restricted and private tables only.
  • Records: a row with FRS values is keyed by its FRS person, in either channel, and a tape row by its stored income vector, since UK local file: support is counted on the FRS host, but SPI-drawn incomes carry no tape-record id #1287.
  • Per band: distinct records, effective records, and 90% intervals from a bootstrap that resamples FRS households and tape records across all Scottish persons.
  • Restricted version: every band, with exact record counts.
  • Public version:
    • primary suppression: fewer than 10 records, one record over 50%, or two over 85% of weight or amount;
    • secondary suppression over the hierarchy total → bins → bands;
    • withheld cells show no values and no record count.
  • Why the total counts as published: the 1p costing is about one per cent of the total property slice. So the withheld bins must pass together, and so must the withheld bands inside them.
  • Release note: each version carries one with the definitions, method and caveats.
• B costing (reform.py). The reform adds delta to every Scottish rate on taxable_property_income_after_allowances. The headline keeps the finance-cost reduction at 20%; the sensitivity raises it by delta as well. Each run first checks that the sensitivity at zero reproduces the engine on the file.

Validation

Tests:

• engine_free/uk/test_uk_scotland_property_income.py (23 tests) covers:
  • bands and their unions;
  • published-value arithmetic, and the check against the printed tables;
  • the bindings;
  • reweight fit, bounds and the iteration-limit stop;
  • record and cluster keys, the audit and the domain bootstrap;
  • one- and two-record dominance;
  • suppression with and without a published total;
  • the record-count subtraction from the review;
  • rendering.
• engine/uk/test_uk_scotland_property_income_engine.py (4 tests): the back-cast round-trips every column through the engine's own extend_single_year_dataset; the earlier-period guard; the reform adds exactly delta times the Scottish property slice and nothing elsewhere; the relief variant matches the engine at zero.

tools/ci_test_plan.py verify and ruff check . pass.

A run on the 19 Oct national release (6cba5d71, H5 810ba980…), at 1552cdc8b under policyengine-uk 2.134.4, took about 8 minutes:

Reweighted facts (34), largest miss	24.7% → under 0.001%
Region household totals, largest miss	under 0.001%
Reweighted facts that match HMRC's printed tables	34 of 34
Weight ratios	median 1.00; 1st–99th percentile 0.80–1.19; none at a bound
Kish effective sample size, after over before	0.998
Scottish total income by band (10, not reweighted to), mean absolute miss	9.8% → 3.8%
Scottish income tax by band (10, not reweighted to), mean absolute miss	10.4% → 7.3%
Other regions' SPI facts (330, not reweighted to)	unchanged, about 9–10% mean absolute miss from the back-cast

Scottish total income and income tax by band are no longer targets, and both improve. That suggests matching Scottish taxpayer counts moves Scotland's income distribution the right way. Scotland's untargeted investment, employment and pension income also stay at or better than the back-cast.

The quantity the outturn can test is the Scottish property split, which no fact binds. The reweight moves its Scottish total by under 2%, and the private audit shows each band before and after.

All outputs are licensed and stay local. Nothing from them is in this PR beyond the run-level diagnostics above.

Open

• Rerun on the national file from the build that carries UK national calibration reads stored input columns at the frame period: 76 £ rows compare 2024 values with 2025 targets #1324's fix (A1). Its SPI property rows are read at the right period there.
• Engine: the tool needs policyengine-uk ≥ 2.134.4. 2.130.1, which the release is bundled with, lacks the private uprating map, and its net_income doesn't deduct trade-loss relief, which moves the NSND bands. UK: 2023-24 property income of Scottish taxpayers by NSND band — the back-cast, reweight, disclosure and engine steps the analysis needs #1326 A5 named 2.130.1; this needs a decision there.
• Confirm the recipients' licence position before anything goes out from restricted/.
• File the policyengine-uk issue for the silent earlier-period result (A2).
• The bootstrap holds weights fixed, so the intervals leave out calibration error and are too narrow. Re-raking inside each replicate would add it.

Refs #1326. Relates #1145, #1276, #1287, #1324.

🤖 Generated with Claude Code

[vahid-ahmadi]

@juaristi22 thanks, this is my review at c4cddcdc9.

Close, with one disclosure fix before anything goes out. The definitions match the Scottish Government's request, the back-cast is an exact inverse of the engine's forward step, the reweight is gentle, and the 1p reform is the right shape for the April 2027 power. One thing has to change before the public tables are sent: the public record counts let a reader rebuild a withheld band's exact count by subtraction (item 1). The rest are "should" items, mostly about what we claim in the release note and which engine we run.

What I ran. A fresh clone at the head, uv sync --all-packages --locked --extra uk (policyengine-uk 2.134.4). No Hugging Face login, no H5 and no UKDS data.

• Engine-free tests: the 16 in test_uk_scotland_property_income.py pass.
• Engine tests: the 4 in test_uk_scotland_property_income_engine.py pass. They use synthetic households only.
• ruff check on the new files and tools/ci_test_plan.py verify pass.
• Not run: the tool itself (python -m tools.uk_scotland_property_income), since it needs the licensed national H5 and its registry. So I couldn't check the run figures in the PR body, the source keys on real rows (item 5) or the registry inversion on real specs (item 9). I also didn't run the whole engine-free suite. The PR adds only new files, so it can't affect other tests.

Findings

1. must: the public tables give away the exact record count of a withheld band. render blanks a withheld band's values but still prints its exact distinct_records when it has 10 or more (release.py:343-346). Secondary suppression always withholds a band with 10 or more records, so its count is printed. Bins and bands are unions of each other, and a source sits in one band only (its NSND income). So: published bin count, minus the published bands in it, minus the secondary band's printed count, gives the exact count of the band shown as "<10". I checked this with the module on a synthetic table (band 1 has 6 sources, the other bands 30 each):

   • Release 2, PA–£24,999: 66 records, published.
   • Release 1, PA–£14,999: "<10". £15,000–£19,999: 30, withheld to protect another band. £20,000–£24,999: 30, published.
   • 66 − 30 − 30 = 6.

   Fix: in public mode, show distinct_records only for published cells, and print "withheld" for the rest. Showing "10 or more" for secondary cells would also work. Please add a test that rebuilds the floor band from the two public tables and fails today.

2. should: count landlords on the same concept as the headline amount. cells keeps property_income > 0 (__main__.py:229-234). The headline amount is taxable_property_income, which is after the £1,000 property allowance (__main__.py:51, and A5 in UK: 2023-24 property income of Scottish taxpayers by NSND band — the back-cast, reweight, disclosure and engine steps the analysis needs #1326). So the count includes landlords whose whole profit sits under the allowance. Their taxable amount is zero, and most of them never report it, so HMRC's outturn mostly can't see them. The build binds on property_income > 0, so the reweight is right as it is. For the release, I'd count taxable_property_income > 0 as the headline and keep property_income > 0 beside it, so the SG compares like with like.

3. should: the Scottish income-tax check isn't independent of the reweight. The PR body gives "Scottish income tax by band (10 facts, not reweighted to), mean absolute miss 10.4% → 4.3%" as "the main check that the reweight moves the Scottish distribution the right way". But the reweight fits Scottish taxpayers and total income in the same bands (__main__.py:184-188). Tax in a band follows almost directly from how many taxpayers there are and how much income they have, so most of that improvement is mechanical. The genuine out-of-sample quantity is the Scottish property split, which no fact binds (I confirmed this against registry.py:28-35 and the bindings in uk_population_targets.json). It is still moved by the UK property facts, though: Scottish landlords get the same push as everyone else's to fix the UK miss. Please:

   • say in the release note that UK property by band, and Scottish taxpayers and total income by band, are calibrated, and only the Scottish property split is a test;
   • add the Scottish property count and amount before and after the reweight to private/audit_*.csv (today audit reads w1 only), so we can see how far the reweight moved the number we are sending.

4. should: the engine claim in the PR body is wrong, and the engine changes NSND income. The body says the tool "runs in any environment with policyengine-uk ≥ 2.123". It doesn't run on 2.130.1, the engine the release is bundled with:

   • backcast.py:20-23 imports the private _uprating_index_map. It isn't in 2.130.1, which reads the YAML inline in apply_single_year_uprating, so the import fails.
   • net_income differs too. In 2.134.4 it deducts trade-loss relief against general income (net_income_before_trade_loss_relief − trade_loss_relief_against_general_income). 2.130.1 doesn't. So the NSND bands themselves depend on the engine.
   • uprating_indices.yaml gains trading_loss (and two property purchase prices) after 2.130.1.

   I'd run the final numbers under the lock (2.134.4), say so in the release, and drop the 2.130.1 item from A5. That is your call in the open list. Either way, run() should record the engine (it does, in the manifest) and refuse an engine older than the one the tests passed on.

5. should: check the SPI source key on real rows. source_keys keys every non-FRS row by its rounded income vector (release.py:57-63). property_components.py:330-336 says the SPI channel also holds rows that keep FRS values: "the channel's dependants and the copies of twins with no taxable income". If one of those is a Scottish taxpaying landlord, the same FRS person counts twice: once as frs:<id>, once as spi:<vector>. That would push a band over the floor and split its weight across two keys, which also weakens the dominance check. It's probably rare, since those rows mostly have no taxable income. Please either key SPI-channel rows that aren't tape recipients (SPI_INCOME_RECIPIENT_COLUMN false) by person_source_id, or check on the run that no person_source_id among the cells sits under both kinds of key. Put the result in the private audit.

6. should: refuse a reweight that didn't converge. run() records raked.converged and the largest miss but carries on either way (__main__.py:205-208, 316-319). For a one-off release, I'd stop if it didn't converge, or if any raked fact misses by more than a set tolerance, say 0.5%. Related: the region household totals are soft targets in the same squared-miss objective (rake.py:85-87), not hard constraints. So "each region's household total is held fixed" should read "held to within X%", with X from the run.

7. should: resample FRS households, not persons, in the bootstrap. Keeping a source's clones together is right (release.py:134-142), and the fixed-weight caveat is already in your open list. Two more things make the intervals too narrow:

   • FRS couples often own let property jointly, and each partner reports half. Persons in one FRS household are resampled independently, so their correlation is lost. Resampling by FRS household (and by tape record for SPI rows) would fix it.
   • The draw is over selected landlords only (cells), so every replicate has the same number of landlord sources. A domain bootstrap usually resamples all Scottish sources and lets the number of landlords vary. That matters most for the 1p total.

   Neither change touches the estimates, only the intervals. Given CVs of 35–73% per band already, I'd note it in the release rather than hold the release for it.

8. should: say what the back-cast holds at 2024-25 levels. The back-cast is an exact inverse of the mapped step. The engine test round-trips (I re-ran it), and the growth read at str(year) matches apply_single_year_uprating. But two caveats belong in the docstring and the release note:

   • The forward step leaves many stored income-tax inputs flat, so the back-cast leaves them at 2024-25 levels too. Examples are gift_aid, loss_relief, capital_allowances, employment_expenses, deficiency_relief and pension_contributions_via_salary_sacrifice, none of them in uprating_indices.yaml. "Council tax, rent and student-loan plans aren't inverted; none of them enters income tax" (backcast.py:10-12) is true, but it isn't the whole list. The effect is small (a few per cent on minor items), but it is not zero.
   • The indices are OBR per-capita aggregates (property income ÷ 1.0372, finance costs ÷ 1.273, weights ÷ 1.0107 under 2.134.4). The band-level differences are left to the reweight to fix. You say this in UK: 2023-24 property income of Scottish taxpayers by NSND band — the back-cast, reweight, disclosure and engine steps the analysis needs #1326. One line in the tool docstring would carry it into the run.

   The engine test only compares the columns in divided (test_..._engine.py:73-81), so it can't catch a column the forward step moves and the back-cast misses. Comparing every column except NOT_INVERTED would close that.

9. should: cross-check a few recovered 2023-24 facts against the published tables. published_value trusts the registry metadata (registry.py:54-63): the value before the cross-grain control, divided by uprating_factor. The test covers the arithmetic on a synthetic spec. Comparing the Scotland rows and two UK property rows on the real registry with SPI 2023-24 Tables 3.7 and 3.11 directly (Chronicle spi_income_by_region_2023_24) would show that the reweight hits the published numbers and not a re-derived copy. A line in the private audit is enough.

10. should: check the relief variant against the engine at run time. property_finance_cost_relief in reform.py:59-74 copies the engine's formula. It matches 2.134.4 and 2.130.1 (I diffed both), and the engine test checks that it matches at delta=0. The run doesn't, and an engine bump that changes the engine formula would make the variant silently wrong. A delta=0 run of the variant on the real file, asserted equal to baseline within your 5p tolerance, would cost one more simulation.

11. should, for Vahid and the SG: which year the 1p costing is for. The reform is the right shape. With the same Scottish band limits, adding delta to every Scottish rate on the property slice adds exactly delta × taxable_property_income_after_allowances (reform.py:46-50), since that slice sits wholly above the allowances. That matches the power from April 2027, which requires Scottish property rates to use the same limits as the other NSND rates. rates_only is also the variant that matches pe-uk's reading that the finance-cost reduction stays at the property basic rate for Scottish taxpayers, so I'd make it the headline and show relief_follows as a sensitivity. But the costing is static on 2023-24 data and law, two years before the power can be used. If the SG wants a 2027-28 figure, that's a separate run on the forward file. Either way, the note should say "static, 2023-24, no behavioural response".

12. nit: the lowest band is a fixed £12,570. nsnd >= PERSONAL_ALLOWANCE_2023 (bands.py:16, __main__.py:233) leaves out marriage-allowance transferors (allowance £11,310) who pay tax on NSND income between £11,310 and £12,569. It also puts blind person's allowance holders in the bottom band below their own allowance. Both groups are tiny. A footnote that "PA" means £12,570 is enough.

13. nit: dominance is one record over 50%. That's the floor you set in UK: 2023-24 property income of Scottish taxpayers by NSND band — the back-cast, reweight, disclosure and engine steps the analysis needs #1326, and it's adequate here with the 10-record rule and rounding (counts to 100, amounts to £0.1m). A two-record rule (the top two over, say, 85%) is common in HMRC and ONS practice. It would be cheap to add in _primary and _passes.

14. nit: output hygiene.

• _refuse_working_tree passes if git is missing or fails (__main__.py:82-90), and it doesn't stop a cloud-synced folder. A note in the docstring is enough.
• manifest.json sits at the top of out/ (__main__.py:349), next to public/. It holds no counts, only paths and ratio quantiles, but moving it into private/ makes "send public/" a safe instruction.
• _text is defined twice (__main__.py:107, release.py:42).

Disclosure overall. With item 1 fixed, the rules are a reasonable minimum for aggregates from EUL microdata sent to a government body: 10 distinct records, the dominance check, secondary suppression over total → bins → bands, and the cautious assumption that the 1p costing reveals the total. I found no route to microdata. Licensed outputs are refused inside a git tree, private/ holds the audit, and nothing licensed is in the diff. The PR body gives only run-level diagnostics (reweight misses, weight-ratio quantiles, Kish ratio), with no counts. The restricted route still waits on the licence confirmation in your open list. That is a decision for us, not a code change.

What looks right

• NSND income: net_income − taxable_savings_interest_income − taxable_dividend_income (__main__.py:137-141) is income before the allowances. It is after the property allowance and trade-loss relief, with savings taken gross of the PSA and net of ISA interest. That is the HMRC concept.
• Bindings: the reweight's facts are measured on the build's own bindings. Bands are on total_income_after_loss_relief, the property filters require income_tax > 0, and regional total income has no taxpayer filter (measures.py:48-69 against uk_population_targets.json).
• Rate starts: SCOTTISH_RATE_STARTS_2023 matches the 2023-24 Scottish bands (starter £2,162, basic £13,118, intermediate £31,092, top at £125,140). Each Release 2 edge is the nearest requested edge.
• Earlier-period guard: refuse_earlier_period is right to exist. The pe-uk issue for A2 is still worth filing.
• Build: nothing under packages/*/src, pyproject.toml or ci_test_plan.py changes, and the tool imports nothing from microcosm.

🤖 Generated with Claude Code

[juaristi22]

@vahid-ahmadi thanks, this was a thorough review. Everything is addressed in 142acc507, a2d6939e5 and 1552cdc8b. I reran the tool on 6cba5d71 at 1552cdc8b, on a clean tree under policyengine-uk 2.134.4; it takes about 8 minutes.

1. Fixed.
   • Public tables print a band's record count only when the band is published. Withheld cells read "withheld".
   • test_public_record_counts_do_not_rebuild_a_withheld_band is your case: a thin band beside two bands of 30 in one bin. Before the fix the counts gave 66 − 30 − 30 = 6. Now the remainder is the two withheld bands together.
   • On the real file, the one published bin that holds withheld bands leaves a remainder of two bands together. By construction, that remainder passes the floor and the dominance rules.
2. Done.
   • The headline count is taxpayers with taxable_property_income > 0.
   • Landlords wholly under the allowance are counted beside it in the restricted tables (all_landlords_count) and in the private audit.
   • They aren't in the public tables, because the difference between the two counts would be a thin cell of its own.
3. Done, and it changed the reweight.
   • What the split showed. I split the reweight into its pieces. The Scottish total-income facts were doing harm: meeting Scotland's total income by band moved weight onto a few very high-income records. Scotland's untargeted property, interest, dividend and other income, employment income and pension income (Table 3.11) all moved away from the published figures.
   • The change. 1552cdc8b drops them and reweights to Scottish taxpayer counts by band instead.
   • Result. Every one of those Scottish components is now at or better than the back-cast. Scottish total income by band, now untargeted, improves from about 10% to about 4% off on average, and Scottish income tax by band from 10% to 7%.
   • Size of the move. The reweight now moves the Scottish property total by under 2%. Before, the move was in double digits.
   • Release note. It says the targets are UK property by band and Scottish taxpayer counts by band, why Scotland's total income is left out, and that only the Scottish property split is a test. The UK facts and the counts both still move it.
   • Audit and PR body. The private audit has every band before and after the reweight. I've dropped the income-tax check from the PR body as evidence.
4. Done.
   • The PR body is corrected.
   • run() refuses policyengine-uk older than 2.134.4 and records the engine in the manifest.
   • The final numbers will run under the lock, as you suggest. Dropping 2.130.1 from A5 in UK: 2023-24 property income of Scottish taxpayers by NSND band — the back-cast, reweight, disclosure and engine steps the analysis needs #1326 is María's call.
5. Done.
   • Rows that keep their FRS person's values are keyed by person_source_id, in either channel (person_is_spi_income_recipient false).
   • On the run, none of the headline cells is such an SPI-channel row. The manifest records that.
   • The manifest also records how many FRS persons host tape records among the cells. Those stay separate records, because the tape row carries another taxpayer's incomes.
6. Done.
   • The run stops unless every fact and region total is within 0.5% (--max-miss).
   • With the smaller set of facts, the misses reach zero before the divergence term stops shrinking, so L-BFGS-B can hit its iteration limit (now 20,000) with the facts met. The run accepts that stop and records it. It refuses any other stop without convergence.
   • On 6cba5d71 it stopped at the limit, with every miss under 0.001%.
   • The note says "held to within" with the run's own figure.
7. Done.
   • The bootstrap resamples FRS households, and tape records for tape rows, across all Scottish persons, so the number of landlords varies between replicates.
   • The intervals widened a little.
8. Done.
   • The docstring and the release note say what stays at 2024-25 levels, and that the indices are OBR national aggregates per person.
   • The manifest lists every stored numeric column the back-cast leaves alone.
   • The engine test now round-trips every column except the three custom pieces, and its fixture includes an unmapped gift_aid.
9. Done.
   • spi_2023_24_published.csv holds HMRC's printed Table 3.7 figures (UK property) and Table 3.11 figures (Scotland), taken from the workbook Chronicle vendors.
   • Every run compares the read-back facts with them, within half the printed unit, and stops on a mismatch.
   • All 34 reweighted facts agree on 6cba5d71.
10. Done. Before costing, each run checks the relief variant at zero against the engine on the real file, to within 5p.
11. Done.
    • costing.csv marks rates_only as the headline and relief_follows as a sensitivity.
    • The note says the costing is static, on 2023-24 data and law, with no behavioural response, and that a 2027-28 figure needs a separate run on the forward file.
12. Done, as a footnote in the release note.
13. Done. A cell is also withheld when its top two records hold over 85%. That applies to the bands, the bins and the costing.
14. Done.
    • The manifest has moved to private/.
    • The docstring says the git check needs git on the PATH and can't see cloud-synced folders.
    • as_text is shared between the modules.

Tests: the 23 engine-free and 4 engine tests pass locally, and so do ruff check . and tools/ci_test_plan.py verify.