[juaristi22]

Part of #931. This is the prep PR of the geography-first spine plan. It carries all four of the plan's prep commits, and none of them changes an output of today's build.

Commit 4 builds the conditioned area supports from the registrations in PolicyEngine/chronicle#413. It was built at that PR's head (df468b8) before the merge, then re-pinned to the merge commit (c51f40d2b), where every registration pins the same bytes. Two commits re-pin the UK feed onto the merge of PolicyEngine/chronicle#421 and read Scotland's band weights from its NRS facts. No compiled value moves. Four more commits cut what the graph executor holds in memory, so that the move's 15-clone spine can fit in 32 GiB; they move no output either.

Why

The move, a later PR, clones every FRS household K times at the start of the spine. Each clone then gets its own small area, and every later stage runs per clone. Four things have to hold before that, and all four hold today without moving an output:

1. Floors count survey households, not rows. Once clones come before enrichment, a floor that counts household_id or person_household_id reads K clones as K sources. household_source_id, the support channel's immediate parent, does too, because a copy made from a clone names the clone there.
2. The fits trained on the frame see each survey record once. Without that, their training sets grow K-fold, and the SPI reference transport refuses the duplicate twins outright.
3. The shared area draw chooses each household's weight column. The move weights a household's output area by the dwellings in its own council-tax band.
4. The draw's supports carry what it conditions on. Each area needs its IMD decile, so a clone can be held to the decile FRS records, and its households in each council-tax band, so the household's band can weight the draw.

The machinery is country-neutral

Per María's ask, nothing new names a UK column. The new microcosm.build.source_lineage takes a country's copy lineage as data:

• CopyLineage: the entity copies are made of and its true-source column (default: the shared source_household_id).
• CopyBranch: each copying step, in path order, with its marker and ordinal columns.

Its readers are:

• record_sources: every record's true source, for any entity, through household membership.
• copy_paths: every household's ordered copy path.
• replica_keys and collapse_replicas: one row per record and path without the named branches, with the weights summed exactly.

The UK declares its lineage once, as household_lineage.UK_COPY_LINEAGE. weight_by is added to the shared atomic_geography operator itself. The new microcosm.build.atomic_support_conditioning builds a support's conditioning columns from any country's point records, shares and supports (commit 4), and the shared spine schema gains an optional observed_constraints field.

Commit 1: every floor counts true source households (246b33a)

• The switch. receipt_counts.SOURCE_HOUSEHOLD_COLUMN is now the shared source_household_id, read through record_source_households, the UK's entry on the shared reader.
  • The reader refuses a frame without the column, and a null source.
  • Stages 10–12 and 15 run before the SPI support channel and opt in with before_copies. Their households are then their own sources, and a frame that already carries copy lineage is still refused.
• Spine stages switched:
  • the take-up signal receipts, the apprentice receipt and the annual-income receipts (stages 10–12, 15);
  • the SPI pay floor, and its pension coherence, which used to read household_source_id directly and skipped the null check;
  • company cars;
  • SPI benefit coherence;
  • UC take-up's pools;
  • the legacy closure;
  • salary sacrifice's receipt and its bonus donor floor;
  • Carer's Allowance;
  • the trading allowance.
• Dense, national and tools switched:
  • the support floor (dense and national);
  • the sampled-rung private-rent region floor;
  • the single-year age report and the dataset-size evaluation;
  • the council-tax, BRMA-distribution, weighted-integrity, benefit-cap and UC-relationship tools.
• Graph declarations. The five finite stages that declared household_source_id now declare source_household_id. UC take-up's pooling text and two notes name it too. Updating that wording was María's choice; it is the one evidence digest that moves (see "The proof").
• A static test scans the UK runtime and tools. It refuses any distinct count of household_id, person_household_id or household_source_id, outside two named structural checks that each benefit unit sits in one household.
• Stage hashes. receipt_counts, household_lineage and source_lineage are bound into the hash of every stage whose module or helpers import them. A test derives that set from the imports.

Commit 2: the FRS-trained fits fold geographic clones (f00739e)

• The six fits:

  • the SPI stage-2 and reference forests;
  • the housing shell;
  • the UC reporter redraw;
  • the UC capital donor cells;
  • the salary-sacrifice forest.

  A seventh, the SPI employment shell from main's UK: carry the FRS occupation (SOC 2020 major group), and redraw it on the SPI channel #1490, joins after the merge (668e13f51, below).

• What each fit does now. It folds a record's geographic clones into the record's first training row, summing their weights exactly with math.fsum. SPI, high-value and CGT copies stay rows of their own, as now. Today's spine has no geographic clones, so every fit's rows and weights are unchanged, value for value.

• Only clones that agree fold (77b1eb151, from review). Each fit passes what it reads, and clones fold only where they agree on it. Clones that differ stay rows of their own, each variant with its own clones' summed weight, so the fit sees every clone's value at its share. That will happen once the move gives clones their own ids: through a draw keyed on those ids, or a stage run per area.

• The clone path. uk_household_clone_paths now reads through copy_paths on UK_COPY_LINEAGE, as María asked. The paths are identical on the 3 November spine (24 distinct path shapes) and on a two-clone pool of it, so the dense identity keys and draws don't move. Three identity tests now match the shared reader's branch names in their error messages.

• Declarations. The housing shell and UC capital stages declare the lineage live where they run. A test refuses any collapsing stage whose surface would hide a lineage column; in the move that includes the geographic clone index.

• Replicated-fixture tests. One per fit: the stage runs on its fixture and on two production-expanded geographic clones of it (expand_uk_geographic_pool), and the fit must receive the same rows and weights, or clone 0 must draw what it draws alone. Each test fails with the fold switched off.

• The SPI income stage now binds spi_rank_transport into its hash.

Commit 3: the shared operator's weight_by (6e7a256)

• The key. A sampling stage can declare weight_by (input, map, default) in place of its single weight. Each household samples by the column map names for its code, or by default when the code is missing. A code the map does not name is refused.
• Nothing else moves. Declarations without weight_by keep their canonical bytes, definition digest and draws. That covers New Zealand's committed declaration and today's UK one, and the invented fixture's draws are now pinned. Only the geography kernels' implementation hashes move, in the UK and New Zealand alike.
• Cache keys downstream. Node keys chain on their producers' keys, not on output bytes, so every node downstream of the geography nodes gets a new key too. The first UK dense and New Zealand runs after merge recompute from the geography nodes onward, through calibration and export, and give the same outputs. A run with resume="require" refuses at preflight instead of recomputing.
• Review. Max: this commit touches the operator New Zealand's transport graph pins as geography.assign_atomic@1.

Commit 4: the conditioned area supports (cad323b)

• What they are. Each of the three UK supports gains a conditioned form, declared beside it in sources.yaml under the new role uk_atomic_area_conditioned_support. It keeps its support's bytes column for column: uk_conditioned_support_base recovers them exactly, and the tests compare them with the registered digest. It adds the columns the move conditions on. Only the move binds them; today's build reads the registered supports.

• imd_decile. The area's deprivation decile within its nation, an exact-match code for the shared operator.

  • Vintage. The 2019 vintage of each index: IoD 2019, WIMD 2019, SIMD 2020v2 and NIMDM 2017. The FRS 2024-25 documentation lists no change to IMDE, IMDW, IMDS or IMDN since 2023-24, which predates the 2025 indices. An internal comparison (no figures quoted) also fits England's FRS decile shares by region to IoD 2019 better than to IoD 2025, as it does for the 2023-24 FRS. The vintage is still to be confirmed with DWP. If it turns out to be 2025, only the England and Wales column is rebuilt.

  • Rule. The May 2026 NSPL is the last edition to carry the 2019-vintage ranks. It gives every live residential (small-user) postcode its small area and rank.

    • An area takes the decile most of its postcodes carry, using the publishers' own deciles.
    • A tie goes to the decile its parent area favours (LSOA, data zone or super data zone), and an area without such a postcode takes its parent's decile.
    • NISRA publishes no deciles, so Northern Ireland's are its 890 SOA ranks in tenths of 89.
    • The relation is inferred_modal.

  • How each area's decile was set:

    Nation	Own postcodes	Tie broken by parent	Tie broken by order	From parent
    England and Wales (188,880 areas)	188,841	4	0	35
    Scotland (46,363 areas)	45,780	485	5	93
    Northern Ireland (3,780 areas)	3,691	88	1	0

• Band weights. Each band column is the area's households in that council-tax band times 10^6, an exact integer, for the move's weight_by stage.

  • England and Wales: bands A to I. They come from the area's LSOA mix in VOA CTSOP1.1 (voa_council_tax_band_mix_lsoa.json, the mix UK: data fixes to model the High Value Council Tax Surcharge (mansion tax) correctly, and the related housing and council-tax gaps #1240's band node reads). The mix is stated to 6 dp and ends at exactly 1, so nothing is rounded. Only Wales has band I, and the assembler refuses any in England.
  • Scotland: bands A to H. They come from the dwellings by band of the area's 2022 data zone, in which its output areas nest, in NRS's December 2024 estimates (3778eb2c6), rounded half to even. Since c8066be53 they are read from the facts Chronicle emits for them. The largest per-area deviation is 3 millionths of a household. Every area in a zone has the zone's mix, and a test checks this. NRS counts dwellings on the valuation list, as VOA does in England and Wales.
  • Northern Ireland has none. It has domestic rates, not council-tax bands.

• Zero-mass cells. For each support and each weight column (households and every band), the provenance lists the FRS region × decile cells and region cells with no mass. So the move can choose its fallback (band weight → household weight, then IMD → region) before it draws. Today every region has households in every decile. The only empty cells are band H in seven of England's region × decile cells, and band I across England.

• Inputs. tools/build_uk_conditioned_area_supports.py reads three sets of inputs:

  • five chronicle#412 registrations, from exact local files whose sha256 must match the registrations: the May 2026 NSPL, IoD 2019 File 1, WIMD 2019, the SIMD 2020v2 lookup, and NIMDM 2017 as the authority for the NI ranks;
  • NRS's dwellings by band and 2022 data zone at December 2024, streamed from the pinned UK feed (c8066be53);
  • the committed VOA band mix.

  uk/uk_atomic_area_conditioned_supports.provenance.json records every input's identity, the Chronicle commit and feed read, a fingerprint of the NRS facts, each nation's decile rule counts, households by decile, the rounding and the zero-mass cells. It holds published area data only. The support bytes are local inputs, as today's are.

• Declarations.

  • sources.yaml gains three rows pinning the bytes. Each keeps its support's vintages and adds the decile vintage and, outside Northern Ireland, the band vintage. The five new geography vintages are reciprocally compatible with every other.
  • In spine.yaml, the FRS channel declares observed_constraints: [imd_decile]. The shared spine schema now admits this optional field.
  • validate_uk_country_source_projection now requires three things: each system's support and its conditioned form, the conditioned vintages extending the support's, and the spine declaration. uk_atomic_support_register(conditioned=True) reads the conditioned pins for the move's preflight.

• Country-neutral. microcosm.build.atomic_support_conditioning holds the pieces any country can use: modal_area_codes, apportion_weights, extend_atomic_support and zero_mass_cells. A test draws on added columns with the shared operator unchanged. The code column is an exact-match constraint, and weight_by chooses the weight column.

• Re-pinned to chronicle#413's merge (7f74d4c32). At the merge commit c51f40d2b, each of the five registrations pins the same filename, URL, sha256 and size as at the PR head. The tool's CHRONICLE block and the provenance now name the merge, and a test holds the two together. Rerunning the tool regenerates the three support files byte for byte.

• Review. Max: this commit adds a shared module and an optional field to the shared spine schema. No other country declares the field, and their specs resolve as before.

Merging main (cd15bca, f9421ea) and the occupation stages (668e13f)

• Conflicts. Only two files conflicted, both generated: the coverage manifest and the H2 uk_spine.json. Each is regenerated on the merged tree, with the environment synced to main's lock (policyengine-uk 2.174.1).
  • The manifest differs from main's only in the 25 source-manifest pins this branch moves.
  • The fixture differs from main's only in the slices of the stages this PR changes, and UC take-up's contract hash.
• Main's new stages count true sources. spi_employment_shell and aps_occupation (UK: carry the FRS occupation (SOC 2020 major group), and redraw it on the SPI channel #1490) count their receipts' support through receipt_counts, which here reads source_household_id. So in the merge:
  • both join _COUNTING_STAGES;
  • their household slices declare source_household_id in place of household_source_id, since the strict reader refuses a slice without it, as for commit 1's five stages;
  • their fixtures carry each copy's true source and clone index.
• A seventh FRS-trained fit (668e13f51). The employment shell trains its occupation classifier and industry donors on the frame's FRS adults, so it now folds geographic clones like commit 2's six fits.
  • Its slice declares the support clone index, and it joins the sweep.
  • A replicated-fixture test holds clone 0 to the frame's own fit, training shares and draws, and fails with the fold off.
  • aps_occupation trains on APS donors, not on the frame, so it needs no fold.
• The second merge (f9421ea47, main 9d080e793). Only the H2 fixture conflicted, and it is regenerated. Main's re-pin of the UK feed onto chronicle#411 rewrote the feed block of scotgov_chargeable_dwellings_by_band.json; its counts are unchanged.
  • The Scottish conditioned support named that file's digest in its band columns' source, so it was rebuilt with the same weights and new metadata (a5e994ae, 1,028,760 bytes). The data-zone mix in 3778eb2c6 has since replaced that file as Scotland's band source.
  • The England and Wales and Northern Ireland supports rebuild byte for byte.

Review fixes

From Vahid's two rounds and the planning review:

• The fold keeps each variant (77b1eb151). Commit 2 above has the details. A replicated fixture with differing clones uses the planning review's case: a landlord's property income, drawn apart per clone, reaches the UC reporter redraw as two rows at half the unit's weight each.
• Scotland's bands come from each data zone (3778eb2c6). Chronicle already registers NRS's dwelling estimates by 2022 data zone, band columns included, but emits only the total as facts. The tool now reads bands A to H from those bytes, pinned by sha256. It uses the December 2024 sheet, which is nearest the FRS 2024-25 year and VOA's stock at 31 March 2025.
  • Each area takes its zone's mix through the support's own lsoa_code. All 7,392 zones match NRS's.
  • The tool refuses a sheet whose header or zone count differs, or in which a zone's bands miss its total.
  • The Scottish support is rebuilt (0c7d7826, 1,607,702 bytes). England and Wales and Northern Ireland rebuild byte for byte, and the zero-mass cells are unchanged.
  • The band vintage uk_ctaxbase_2025 is renamed uk_nrs_dz_dwellings_2024.
  • Emit the council-tax band columns of NRS dwellings by 2022 data zone (nrs-dwellings-data-zone-2025) chronicle#419 asked Chronicle to emit the band columns as facts. Emit NRS's council tax bands by 2022 data zone chronicle#421 does, and the support now reads them from the feed with the same numbers (below).
• The static floor scan sees more forms (fd902f63a). It now catches drop_duplicates(key), groupby(key).ngroups, len(df.groupby(key)), .loc[rows, "column"] and columns named through a module constant. It found one more structural row check in tools/verify_uk_identity_stability.py, now named.
• Person sources are required (193870e6f). The person draws and annual income no longer fall back to each row's household.
• The legacy clone_index filter is gone (bb68b570e). The SPI income stage refuses a table carrying the artifact's bare clone_index; no spine stage writes it.
• Notes (e9d557c39):
  • the UC capital donor minimum counts rows, and its classification is the move's constants audit;
  • the limit of Scotland's council-level band mix, which 3778eb2c6 has since lifted;
  • the US passages in docs/geography-assignment.md describe Consolidate the US survey graph build and enrichment pipeline #893.

Picking up chronicle#421 (f086e01, c8066be)

• The feed re-pin (f086e0169). The UK feed moves from 6475380 to 80a3482, the merge of Emit NRS's council tax bands by 2022 data zone chronicle#421 (2,030,062 rows; the row contract and schema digest are unchanged).
  • It brings NRS's band facts (118,272 rows).
  • From the Chronicle merges in between, it also brings UK land value inputs for chronicle#408, on main (re-lands #410) chronicle#415's land-value inputs (326,495 rows), which nothing reads yet. Register UK area inputs for Microcosm's geography-first spine: IMD by small area with crosswalks, 2024-25 Band D, 2011 OAC chronicle#413 adds no rows, and Write and count bundle rows per package instead of holding them all chronicle#414 changes only how the bundle is written.
  • No compiled value moves on either surface. Both reference files and both compile-parity receipts are byte-identical.
  • The memberships. The national membership moves its feed label and the matched-fact counts of the nine MHCLG council-tax stock targets. Their selectors also match the council-tax rows Bound QRF leaf storage for near-discrete targets #415 adds, and every candidate resolves to the same fact and value. The local membership moves its feed label and the 1,263 DESNZ constituency keys, whose derived keys hash the feed identity.
  • The rest. The vendored resources and the DESNZ constituency resource carry the same rows under the new feed identity, and the census and the validation register restate the pin. The H2 fixture moves the contract hashes of the 19 stages that bind vendored resources. The high-value home stock's feed-gated check passes, so it isn't rebuilt.
• Scotland reads the facts (c8066be53). The tool streams NRS's December 2024 record set from the pinned feed and checks the whole feed against the pin. It still refuses a zone without all nine counts, or whose bands miss its total.
  • The facts equal the workbook's bands in all 7,392 zones, and they come from the same bytes (5d61db5f).
  • The Scottish support's arrays are unchanged. Only its eight band columns' source moves, to the record set and the counts' fingerprint (a8ca096b, 1,607,718 bytes). England and Wales and Northern Ireland rebuild byte for byte, and every receipt is unchanged.
  • The facts are streamed rather than vendored, because their 66,528 rows would make a resource of over 80 MB.
  • As for the high-value home stock, a feed-gated test checks on each re-pin that the feed still carries the facts the supports were built from. The re-pin runbook says to rebuild them if it doesn't.
  • The tool's Chronicle block now names Restore ASEC ED_VAL via a pinned sidecar and close the source-carry class #421's merge, where the five raw-only registrations pin the same bytes.

The executor keeps less in memory (a0e31d8, 5dd5e1d, 13ce7a3, 2fe3341)

• Why. At 15 clones, a spine run would hold more than this 32 GiB machine has:

  • every structural population version, until the run ends;
  • every stage's output Frame, kept in last_result;
  • a second copy of the frame at each structural write.

  The move needs these fixes before its f100 15-clone spine, not before its f010 arms.

• E1: keep only the versions the caller reads (a0e31d886, 2fe33410e).

  • run_graph(..., retain_populations=...) lives in the shared microcosm-graph. None, the default, keeps today's behaviour.
  • Given a collection, a version leaves the run once its last reader has applied its result. Its last reader is an ordinary member reading it as incumbent, or a structural node reading it as base. Only the named versions and their mass ledgers are attached, and an unknown name is refused before any work.
  • The UK opts in. The spine names its final version, each dense and national phase names what it reads from its own manifest, and the atomic-assignment measurement names uk.full.expand.
  • Charter amendment 29 in docs/graph-acceptance.md records it.

• E2: release each stage's result once its evidence is captured (5dd5e1d62). The shared stage_evidence.release_stage_results clears last_result after the snapshot, walking the same wrappers as the fit-weight probe. The UK kernels call it when the registry asks. The spine turns it on, and so does the full build's --spine-request route, which prepares its spine the same way. Inside the stages, only the snapshot reads last_result.

• E3: write structural versions one column at a time (13ce7a3a3). Each entity's ids are copied once and shared, and each column is copied only when written, in the same order. A structural cache hit still checks each stored column, but keeps none of them.

• Other releases (2fe33410e). The executor also lets go of each node's context, decoded artifacts, incumbent and result once nothing reads them.

• Review. Max: this changes the shared executor and adds amendment 29, which you may want to renumber or reword.

The proof: no output change

• The proof predates the merge and the review fixes, and carries over. It wasn't re-run, because everything after it is of a kind already shown to move nothing:

  • the two occupation stages count support by source_household_id instead of household_source_id, the switch commit 1 made, and the real spine shows the two columns agree on every household row;
  • the folds change nothing without geographic clones, which their tests check;
  • main's edits to files this PR also touches (spi_income.py, uc_capital_coherence.py, and the counting module's docstring) merged without conflicts;
  • the review fixes add refusals no spine reaches (clone_index), pass the same sources explicitly, or are notes and tests;
  • the data-zone band mix rebuilds the Scottish conditioned support, which no stage reads;
  • the feed re-pin moves no compiled value (both reference files and both compile-parity receipts are byte-identical), and the switch to the facts leaves that support's arrays unchanged.

• Real spine, before and after. Each spine is a K=1 full spine from the licensed sources, with a fresh checkpoint store and --staging-local-only. The base is fc5bb9014, the merge base before main was merged in. There are two heads: cad323b79 (all four commits) and, earlier, 6e7a25628 (commits 1–3). Both heads give every result below.

  • uk_frame_content_identity is equal: 3939a72ea5693441c3af4efea61ccc1ad8ae5d11f02ce91574cf61bae1cb003b on all three.
  • The H5 payloads are identical (tools/compare_uk_h5_payload.py exits 0).
  • All 52 stage-evidence payloads in the checkpoint stores are byte-equal, as are the sidecar's stage evidence and fit weight records. UC take-up's evidence doesn't carry the pooling text, so it doesn't move either; the new wording appears only in the sidecar's copy of the stage spec.
  • The spine gate report differs only in release_id, which names each build's start time.
  • Stage timings match: the collapse adds no measurable time to the five fits.

• The executor's memory fixes, before and after. Base c8066be53, head cf8432c9b, both K=1 full spines from the same launcher and inputs. The launcher now passes --aps-tab, which the spine requires since UK: carry the FRS occupation (SOC 2020 major group), and redraw it on the SPI channel #1490.

  • uk_frame_content_identity is equal: b19e3f90daf4b1a69c66d4d2e6c4c6c3d05198e08255cc6d03917e5b1f826d07.
  • The sidecar's stage evidence and fit-weight records are equal, and all 54 stage-evidence payload digests are equal.
  • The H5 payloads are identical, and the spine gate report differs only in release_id.
  • Peak memory was 23.8 GB on the base and 23.2 GB on the head. At K=1 the peak is the engine; the frames these fixes free only grow 15-fold at 15 clones.

• Precondition. On the 3 November spine and on both spines here, household_source_id and source_household_id agree on every household row, with every copy type present.

• Commit 4 binds nothing today. No stage reads the conditioned supports, and the spine above is unchanged with them declared. The commit after it, 7f74d4c32, re-pins the provenance to chronicle#413's merge and rewords one provenance note. The support files regenerate byte for byte.

• H2. uk_spine.json is regenerated. Only the inputs of the seven stages whose declarations changed move, plus UC take-up's stage contract hash. fixture.json moves only in the two texts. The coverage manifest moves only in its source-manifest pins. Commit 4 moves none of the three: they regenerate byte for byte.

• Tests run locally:

  • Engine-free test files that import a touched module, Orrery tests included: 112 for commits 1–3 and 121 for commit 4. All pass apart from one environment check, test_uk_cgt_projection's check that the UK engine extra is absent, which fails wherever the extra is installed.
  • Mounted checks. Commit 4's tests also ran with the real supports mounted: their bytes, base, zero-mass table and Scottish band shares.
  • Engine-UK files that import a touched module, H2 parity included: 21 for commits 1–3 and 9 for commit 4, all passing.
  • After merging main:
    • the 190 engine-free files that import a shared module this PR or main changed, Orrery included, pass apart from the same environment check;
    • the 10 engine-UK files, H2 parity included, pass.
  • After the second merge and the review fixes:
    • the conditioned-support tests with the rebuilt supports mounted, and main's changed feed and target tests, pass;
    • each fix's own tests pass: the seven fold sites, the SPI income engine tests, and H2 parity again.
  • After 3778eb2c6: the conditioned-support and tool tests, the UK country adapter, the atomic-support provenance and the shared spec tests pass with the rebuilt supports mounted.
  • After c8066be53: with the pinned feed available and the rebuilt supports mounted, 21 files pass, with 3 skips: the feed pin, full-targets, reference, local-target, census, validation-level, vendoring, ledger-target, local-parity, DESNZ, high-value home, council-tax, conditioned-support and country-adapter tests; the shared epoch, ledger-artifact and spec tests; and the engine coverage-manifest and H2 parity tests.
  • The executor's memory fixes: the graph tests (175 passed, 1 platform skip) and the UK and shared build tests (369 after merging c8066be53) pass. The new tests run the toy graphs cold, warm and under resume="require", with and without retain_populations.
  • ruff, and ci_test_plan.py verify, on each state.

  The full suites are left to CI.

What moves without being an output

• Cache keys: every stage that now binds the counting and lineage modules, the geography kernels, and every node downstream of them. The first runs after merge recompute from there and give the same outputs (commit 3).
• Every UK spine node key (5dd5e1d62). stage_evidence.py and graph_kernels.py are bound into every UK stage kernel's implementation hash.
  • The first spine after merge recomputes every stage.
  • A dense build re-keys from its spine with --spine-request, and from uk.full.expand.owned onward with --input-h5.
  • National keys don't move, and no committed pin binds these hashes.
• The feed pin (f086e0169): the vendored resources' feed headers, both memberships, the census, the validation register, the DESNZ constituency keys and 19 H2 contract hashes.
• The UK spec (commit 4): three source rows, five vintages, the spine channel's observed constraints, and one new package resource. None reaches a graph input. The support loader is a contract-only kernel, and only the country adapter reads the sources resource.
• Local reports:
  • the single-year age report and the dataset-size evaluation count a zero-weight row as carrying nothing and a household's copies once, so their support counts can only fall;
  • the weighted-integrity tool's carriers are distinct sources, not rows;
  • on sampled rungs, the private-rent region floor counts sources.

Found in passing

• Stale stage hashes. Eight stages counted through receipt_counts without binding it into their hash, so an edit to it alone left their cache keys in place. With the lineage modules, 24 stages were missing a binding; all are now bound, and the test above derives the set.
• Consolidate the US survey graph build and enrichment pipeline #893's doc text. docs/geography-assignment.md describes the US work of Consolidate the US survey graph build and enrichment pipeline #893, including a block adapter (us_runtime/atomic_block_support.py). Consolidate the US survey graph build and enrichment pipeline #893 closed unmerged on 9 October 2026, so that code never reached main. The doc now says so at the head of each US passage (e9d557c39).
• Northern Ireland's deciles are coarse. NIMDM 2017 ranks 2001 SOAs, and they don't nest in the 2021 data zones. The postcodes of 1,607 of NI's 3,780 data zones span more than one decile, against 357 of 188,880 areas in England and Wales and 2,363 of 46,363 in Scotland. The modal decile is recorded as inferred_modal; the move may choose to relax IMD in Northern Ireland.
• Reading the WIMD ODS needs odfpy, which isn't a dependency. The tool runs as uv run --no-sync --with odfpy, which leaves the lockfile alone; it is documented in the tool.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review of 7f74d4c. I read the PR body, #931 and all five commits, focusing on the shared lineage reader (source_lineage.py), the UK lineage and counting entry points, the six folds, the static count test, weight_by in the shared operator, the conditioning module and the conditioned-support provenance. I read the code only: I could not run the tests locally, so for the tests I rely on CI (see below).

The design holds up. record_sources refuses a frame that carries any declared copy column without source_household_id, so before_copies cannot be used to count a cloned frame by row ids. collapse_replicas keeps the first row in row order (clone 0 on the clone-major pool) and sums with math.fsum, and every caller reads the summed weight only on the kept rows. The provenance's rule counts match the table in the body.

Should

1. uc_capital_coherence.py:717: is the donor-cell floor a true-source floor, or a row threshold? int(pool.sum()) >= minimum_cell_donors (20) counts benefit-unit rows after the fold. This was already the case at the merge base, so it is not new here, but it sits outside commit 1's switch. uc_capital_coherence (stage 33) runs after high_value_home_support (stage 23). The graph declares the high-value lineage columns live at this stage, and the fold deliberately keeps high-value copies as rows of their own. So a high-value copy counts as a second donor. Failure case: an exact cell holds 12 survey benefit units, 8 of them in households that high-value support copied once. That is 20 rows, so the exact cell passes the floor, though it rests on 12 survey units. The static test cannot see this, because the count is a mask sum, not a distinct count of an id. If it is meant as a source floor, count the pool by replica_keys(..., merge=<every branch>) (true source plus member rank) and add the stage to commit 1's list. If it is a row threshold by design, a one-line note at the constant would settle it.

2. source_lineage.py:332-365 and household_lineage.py:157: the fold never checks that replicas agree on what the fit reads. This is not a defect at this head, but it is a design question to settle before the move. Today's clones are identical, and so are the replicated fixtures (with_geographic_clones copies the frame). In the move, the clones will already have been through per-area stages, and UK graph: move geography assignment ahead of enrichment (geography-first reordering) #931(b) still has to decide what each stage conditions on. Failure case: a fit reads a column that a per-area stage rewrites, for example uc_reporter_redraw's screen, which "is each row's own engine output". If clones 0 and 1 of one benefit unit are both screened in with different values of that column, the fit trains on clone 0's value at the summed weight, and clone 1's value is dropped without any record. Suggested fix: give collapse_geographic_clones the fit's predictor and target columns, and refuse (or receipt) any key whose replicas differ on them. Add one replicated fixture where the clones differ, so the choice is tested before the move rather than inherited.

3. tools/build_uk_conditioned_area_supports.py:368-386: in Scotland, the band weights use each council's mix, so (as the docstring says) a band moves mass only between councils. Within a council, a band-H household is placed in proportion to households, exactly as today, while England and Wales get the mix at LSOA level. NRS's "Small area statistics on households and dwellings, 2025" gives dwellings by council tax band at 2022 data-zone level, from the Assessors' Portal. Scotland's 2022 output areas nest in 2022 data zones, so that source would give Scotland the same within-council resolution as England and Wales, if it can be registered in Chronicle. If it can't, a provenance note saying why would let readers of the move's Scottish band results know the limit.

Nit

4. test_uk_receipt_counts.py:451: _counted_keys misses len(df.drop_duplicates("household_id")) (the key is an argument, not the receiver), groupby("household_id").ngroups, and a column named through a constant (df[HOUSEHOLD_ID_COLUMN].nunique(), where the slice is a Name). None of these appear in the UK runtime today. Adding the first two forms would close the obvious gaps.

5. frs_annual_income.py:481-485 and frs_person_draws.py:336-340: person_sources=None falls back to own_source_households(person), which counts each row's household id with no lineage check. Only tests and tools/verify_uk_identity_stability.py reach it today. On a cloned person table, it would count K clones as K sources. Making person_sources required, or refusing when the person table carries a clone column, would close it.

Checked and fine

• copy_paths: the marker and ordinal agreement, the partial-column refusal and the required branches match the old hand-rolled reader branch for branch.
• The person and benunit keys rank members by id. The pool remaps ids by adding a constant, so the rank is the same across clones.
• weight_by: declarations without it keep their bytes. An unmapped code or an empty code is refused. The sampling stays in exact integer arithmetic: band weights at 10^6 scale stay within 2^53, and the apportion_weights totals are checked against that range.
• modal_area_codes: the tie-break order, the parent fallback and the refusal when both area and parent are empty. The provenance counts (E&W 188,841/4/35, Scotland 45,780/485/5/93, NI 3,691/88/1) match the body. The Scottish rounding (largest deviation 3, total -1,092 at 10^6 scale) is recorded.
• The SPI reference forest's population is taken from the folded training_people, so its weights are not double-counted.
• No UKDS or other licensed data is committed. The provenance holds only published area aggregates.

CI. There is no run at 7f74d4c or at cad323b. The last Tests run (41954535023) was at 6e7a256 (commits 1 to 3), and all 11 jobs passed. Commit 4 and the re-pin have only the local runs described in the body, so a CI run at head is needed before this leaves draft.

Verdict: commits 1 to 3 do what they say for floors and geographic clones. Items 1 and 2 need an answer, and item 2 should be settled before the move relies on the fold.

🤖 Generated with Claude Code

[vahid-ahmadi]

Round 2 at 668e13f. Since 7f74d4c there are two commits: the merge of main (cd15bca) and the SPI employment fold (668e13f). I read both, along with the files behind each round-1 item. I read the code only and rely on CI for the tests. There are no replies on the PR yet, so all five items carry over.

Round 1 items

1. uc_capital_coherence.py:721 donor-cell floor, rows or true sources: open. The merge touched only the policyengine-uk version note in this file, and int(pool.sum()) >= minimum_cell_donors is unchanged.
2. The fold does not check that replicas agree on the fit's columns: open. source_lineage.py and household_lineage.py are unchanged, and the new SPI fold inherits the same question.
3. Scottish band weights at council level rather than data zone (NRS small area statistics, 2025): open. build_uk_conditioned_area_supports.py is unchanged.
4. Static count test misses drop_duplicates("household_id") and .ngroups: open.
5. person_sources=None fallback counts own household ids: open.

A reasoned won't-fix on any of these is fine. Items 1 and 2 still need an answer before the move.

New code

No new findings. Checked and fine:

• spi_employment_shell.py:556-572: the fold runs after the training mask, including weights > 0, and before the person_id sort. So each record keeps its first positive-weight row, which is clone 0 on the clone-major pool. fit_weights feeds the fit, the held-out folds and the industry donors. _person_table keeps person row order, so training, weights and person.iloc[train_rows] stay aligned.
• The fold also stops a record's clones from landing in different held-out folds. The folds group by household_id, and clones carry distinct ids, so without the fold the held-out log loss would leak.
• Drawn SPI rows are not folded. Each clone keeps its own person_id key and its own draw, and the new test checks that clone 0 draws what the frame draws alone.
• Merge: aps_occupation trains on APS donors rather than frame rows, and it uses frame weights only for shares, so it rightly has no fold. The two new stages declare source_household_id and join _COUNTING_STAGES. The clone-collapse sweep now lists spi_employment_shell, and the graph declares the clone index it reads.

CI. Tests run 42075566346 at 668e13f: all 11 jobs passed (lint, engine-free and engine-uk on 3.13 and 3.14, engine-us, integration-uk, wheels, installed-emitter, select-countries). This closes round 1's point that CI had not run at head.

Verdict: the new fold is consistent with the six earlier ones and CI is green. Items 1 and 2 remain the ones to settle before the move relies on the fold.

🤖 Generated with Claude Code

[juaristi22]

Notes at 668e13f, on top of Vahid's two rounds. The SPI employment fold (668e13f) already covers the seventh FRS-trained fit, so it isn't listed here. Items 1 to 4 change no output. Item 5 would, so it belongs in a follow-up.

1. A concrete case for Vahid's item 2 (fold agreement).

• property_components adds finance costs to FRS landlords' property_income. Both the incidence and the amount are drawn on uniforms keyed by person_id (property_components.py:554, :581).
• uc_reporter_redraw runs six stages later. It reads property_income through benunit_investment_income (uc_reporter_redraw.py:86, :603).
• So once the move gives clones distinct person ids, a landlord's clones reach the reporter with different predictors, even though no stage in between conditions on area. The fold then trains on the first screened-in clone's value, at the summed weight.

This column would make a natural replicated fixture with clones that differ. The same question applies to any id-keyed draw that sits between the clone step and a folded fit.

2. The legacy clone_index filter in spi_income.py:1451-1452.

• What it does. The stage still filters FRS training rows on a bare clone_index == 0.
• Why it does nothing today. No spine stage writes that column, which is the artifact name. The fold reads household_clone_index, and expand_uk_geographic_pool drops a bare clone_index. The new replicated test also has to drop the column to compare runs.
• The latent risk. A frame that did carry it would train clone 0 at clone 0's own weight, not at the fold's summed weight.
• Suggestion. Have the stage refuse a frame that carries clone_index. That means:
  • dropping the column from the _dead_support fixture after the support rebuild (test_support/microcosm_build/uk_spi_income.py:188);
  • updating the two engine tests that assert the clone-0 filter (packages/microcosm-build/tests/engine/uk/test_uk_spi_income.py:393-407, :926-936).

3. The US sections of docs/geography-assignment.md.

• The body says the US block adapter "no longer exists". In fact it never reached main.
• These files all lived on Consolidate the US survey graph build and enrichment pipeline #893, which closed unmerged on 9 October:
  • us_runtime/atomic_block_support.py, graph_geography.py and atomic_block_api_sources.py;
  • the two graph_atomic_survey_*.py graphs;
  • the experiments/ receipts the doc links.
• The doc text came across with af89d22.
• The stale text covers the intro (including the broken link at :22), :167-173 and :229-302.

This PR already edits the doc, so a one-line note at the head of those sections, saying they describe #893's unmerged work, would cover it. Alternatively, a follow-up could trim them. The body could also say "never merged" rather than "no longer exists".

4. Cache keys after commit 3.

• The body says only the geography kernels' cache keys move.
• In fact node keys chain on their producers' node keys, not on output bytes (packages/microcosm-graph/src/microcosm/graph/keys.py:196-204). The executor also computes every key before it runs anything (executor.py:2277-2299).
• So every node downstream of the geography nodes gets a new key too. The first UK dense and New Zealand runs after merge recompute from the geography nodes onward, through calibration and export, and give the same outputs.
• A run with resume="require" refuses at preflight instead of recomputing.

One sentence in the body would stop a full recompute being read as a regression.

5. Follow-up, not for this PR: group gate folds by true source household.

• salary_sacrifice and uc_reporter_redraw run after high_value_home_support, and by design they keep its copies as rows.
• Their cross-fitted gates group folds by the row's household id (salary_sacrifice.py:641, uc_reporter_redraw.py:367). Each copy has its own id.
• So one household's copies can land in different folds, and a copy's out-of-fold score can come from a gate trained on its twins.
• Passing the true source household as groups would keep all of a household's copies in one fold. salary_sacrifice already calls record_source_households for its bonus floor.
• This moves fold assignment, so it changes outputs and belongs in its own PR.

For the move (nothing to change here)

• The weight_by input. The spine's council_tax_band is never NA: unbanded households and Northern Ireland carry the placeholder "D" (council_tax_constants.py:18). The move should pass NA where council_tax_band_observed is false. Otherwise an unbanded household draws by band-D dwellings without error.
• The decile codes. The conditioned supports store imd_decile as strings, "1" to "10" (atomic_area_support.py:404-406). The operator refuses non-string constraint codes, so the FRS decile has to reach the area draw as the same strings.

🤖 Generated with Claude Code

[juaristi22]

Thanks both. The answers are in five commits, on top of a second merge of main (f9421ea47, main 9d080e793), now at 77b1eb151. Item by item:

Vahid

1. UC capital donor floor: rows or true sources? Rows, by design today: the floor counts the cell's donor benefit-unit rows after the fold, so a high-value copy counts as a donor of its own. Switching it to true sources can change which cells widen, so it can't land in this no-output PR. The plan already lists minimum_cell_donors among the K-sensitive constants the move classifies, as a source count, a weight or a hyperparameter. e9d557c39 says so at the constant.

2. The fold never checks that replicas agree. Settled with María: keep each variant. The changes are in 77b1eb151.

   • Each fit now passes what it reads, and clones fold only where they agree on it.
   • Clones that differ stay rows of their own. Each variant carries the summed weight of the clones that share it, so the fit sees each clone's value at its share and nothing is dropped.
   • collapse_replicas takes the fit's inputs, and collapse_geographic_clones requires them, so no fit folds blind.
   • The seven fits pass their predictors and targets. The SPI stage passes the raw columns its two forests derive from at fold time; its later steps are deterministic in them.
   • A replicated fixture where clones differ uses María's property-income case: the landlord's clone 1 gets 500 more of property income. It holds uc_reporter_redraw to training on both variants at half the unit's weight each, while the other units fold.
   • One consequence for the move: a record's variants are now separate training rows with different household ids. Cross-fitted folds should therefore group by true source, which is María's item 5 below.

3. Scottish band weights at council level. I've kept the council mix and recorded the limit in the provenance (e9d557c39). The note names a mix by 2022 data zone, such as NRS's small area statistics, as the way to lift it; no such source is registered in Chronicle yet. The support bytes are unchanged. Registering that source is a Chronicle follow-up.

4. Static count test gaps. fd902f63a closes them. The scan now also sees:

   • len(df.drop_duplicates(key)), with the key as an argument or subset=;
   • groupby(key).ngroups and len(df.groupby(key));
   • .loc[rows, "column"] tuple slices;
   • a column named through a module constant.

   Snippet tests pin each form. The wider scan found one more count, a structural row check in tools/verify_uk_identity_stability.py that every SPI band donor holds one carrier. It is now named beside the other two.

5. The person_sources=None fallback. 193870e6f makes person_sources required in all four functions. The tests, and the identity tool (which compares only the draws), pass own_source_households(person) explicitly.

María (planning review)

1. Property income as the case of clones that differ. That case is now the fixture in item 2 above.
2. The legacy clone_index filter. bb68b570e addresses it:
   • the SPI income stage refuses a table carrying clone_index, and a test checks it;
   • _dead_support drops the column after its rebuild;
   • the two engine tests count reference rows on all four households (four and three);
   • they count the reference fits from the transport's fold receipts, since a fold with no recipient isn't fitted.
3. The US doc passages. e9d557c39 adds a note at the head of each saying it describes Consolidate the US survey graph build and enrichment pipeline #893, which closed unmerged on 9 October. The body now says the code never reached main.
4. Cache keys after commit 3. Agreed. The body now says every node downstream of the geography nodes gets a new key, so the first UK dense and New Zealand runs after merge recompute from there and give the same outputs, and resume="require" refuses at preflight.
5. Gate folds by true source. Agreed that it changes outputs, so it goes in its own PR. It matters more now, since with variants a record's clones can be separate training rows.

For the move. I've passed both notes to the session implementing the stacked move PR:

• pass NA as the weight_by input where council_tax_band_observed is false;
• pass the FRS decile as the strings "1" to "10".

The second merge. Main's re-pin of the UK feed onto chronicle#411 rewrote the feed block of the Scottish band resource; its counts are unchanged. So the Scottish conditioned support is rebuilt with the same weights and new metadata (a5e994ae, 1,028,760 bytes). The other two supports rebuild byte for byte.

None of this moves an output: the fold changes nothing without clones, no spine stage writes clone_index, the sources passed are the same, and the rest are notes and tests. The PR body is updated.

🤖 Generated with Claude Code

[juaristi22]

Vahid's item 3, the Scottish band weights, now come by data zone (3778eb2c6).

First, a correction to my reply above. I said no data-zone source was registered in Chronicle. In fact Chronicle already registers NRS's dwelling estimates by 2022 data zone (nrs-dwellings-data-zone-2025), and the band columns are in those bytes. Chronicle emits only the workbook's total as facts. So:

• The source. The support tool now pins that workbook by sha256, beside chronicle#412's files, and reads bands A to H from its December 2024 sheet. That sheet is nearest the FRS 2024-25 year and VOA's stock at 31 March 2025.
• The mapping. Each Scottish output area takes its 2022 data zone's mix through the support's own lsoa_code. All 7,392 zones match NRS's, in both directions.
• The checks. The tool refuses a sheet whose header or zone count differs, or in which a zone's bands miss its total. Today the bands sum exactly to the total in every zone of both sheets.
• The basis. NRS counts dwellings on the valuation list, as VOA does in England and Wales. CTAXBASE counted chargeable dwellings.
• The rebuild. The Scottish support is rebuilt (0c7d7826, 1,607,702 bytes). England and Wales and Northern Ireland rebuild byte for byte, and the zero-mass cells don't change.
• The spec and tests. The band vintage uk_ctaxbase_2025 is renamed uk_nrs_dz_dwellings_2024. The mounted test now checks that every area in a zone has that zone's mix.

PolicyEngine/chronicle#419 asks Chronicle to emit the band columns as facts. Once it does, the support can read them from the feed instead, and the numbers will be the same.

No output moves, since only the move binds these supports. The PR body is updated.

[vahid-ahmadi]

Round 3 at 3778eb2. Since round 2 (668e13f) there are seven commits: the second merge of main (f9421ea) and six review fixes (fd902f6, 193870e, bb68b57, e9d557c, 77b1eb1, 3778eb2). I read the six fix commits and the merge's file list, with María's two replies. In a scratch clone I ran these engine-free test files, and all pass: the shared lineage reader, the clone-collapse sweep, the UC reporter redraw and the SPI income stage in full, plus the receipt-count scan, the conditioned supports and their tool, the apprentice draw and annual income. I also ran one extra SPI test (item 6) and checked the NRS workbook with the tool's own reader.

Earlier items

1. UC capital donor floor, rows or sources: fixed as a documented choice. The note at uc_capital_coherence.py:68-73 says it is a row threshold, and leaves it to the move's audit of K-sensitive constants. One addition for that audit: since 77b1eb1, clones of one unit that differ on capital or cell also count as separate donors (:739).
2. Fold never checks that replicas agree: fixed for six of the seven fits. collapse_replicas groups on key plus inputs (source_lineage.py:337, :381), and collapse_geographic_clones requires them. The UC reporter fixture with the 500 property-income clone pins the variant rule. The SPI stage is the exception; see item 6.
3. Scottish band weights at council level: fixed. Bands now come from each 2022 data zone in NRS's dwelling estimates. I downloaded the registered workbook: sha256 5d61db5f… and 1,223,284 bytes match the tool's pin. With _nrs_band_counts, both the 2024 and 2025 sheets give 7,392 zones, bands summing to each zone's total, and no zone with zero dwellings. So apportion_weights cannot hit a zero denominator. The provenance records Scotland's rounding as 19,887 areas, largest deviation 3 and total +20.
4. Static scan gaps: fixed (test_uk_receipt_counts.py:489-492 for .ngroups, with snippet tests from :665).
5. person_sources=None fallback: fixed. The argument is now required (frs_annual_income.py:462, frs_person_draws.py:204).

María's planning items 2 to 4 (the bare clone_index refusal at spi_income.py:1454, the #893 notes at docs/geography-assignment.md:10, :171 and :237, and the cache-key sentence in the body) are in as described.

New finding

Should

6. spi_income.py:1478 with spi_rank_transport.py:325-326: in the SPI stage, keeping each variant turns a differing clone into a hard refusal, not a training row. The reference forest's rows go to cross_fit_transport keyed on person_source_id (spi_income.py:1582). The transport refuses any duplicate key: "training identities must be unique (one row per twin)". Geographic clones share that key, so two variants of one reference person always stop the stage. Failure case, run as a scratch test: _dead_support(), then _geographic_clones_of (two clones), then add 500 to property_income for one clone-1 adult in the reference population (aged 16 or over, recipient role, employment income above zero). The stage raises that ValueError from spi_income.py:1576. Without the edit, the same cloned fixture runs through, as the PR's own replicated test does. So for this fit the body's "the fit sees every clone's value at its share" does not hold. Instead the move will get a generic transport error the first time a per-clone stage or an id-keyed draw ahead of the SPI stage moves one of _forest_inputs' columns. Today nothing reaches it, because there are no clones. Suggested fix: make the choice explicit for this one fit. Either fold SPI by record alone and record how many records had differing clones, since the transport needs one reference row per twin. Or refuse at the fold with a UK graph: move geography assignment ahead of enrichment (geography-first reordering) #931 message naming the columns that differ. Then add the differing-clone case to test_geographic_clones_train_the_frs_forests_as_their_survey_records, and state the SPI exception in the body and in changelog.d/931-uk-fits-fold-geographic-clones.added.md:3.

Checked and fine

• collapse_replicas with inputs: dropna=False makes a missing value agree only with a missing value. observed=True keeps unobserved categories out of the groups. ngroup() with sort=False keeps first-appearance order, so each variant keeps its first row, and weights stay math.fsum sums.
• Each fit's inputs are aligned to its entity rows. The housing shell reindexes by household_id, and the UC reporter passes its benefit-unit predictors plus reported_before. For SPI, the steps between the fold and stage 2 read only columns in _forest_inputs: the pension-age zeroing (state_pension_reported), the owner-manager flags (in FRS_ONLY_SPI_FILL_PERSON_COLUMNS) and the ISA addition. So folded clones still agree at stage 2.
• The reference forest and stage 2 still group folds by person_household_id (spi_income.py:1561), so variants can split across folds. That is María's item 5, already agreed as its own PR.
• 3778eb2: the Scottish numerators key on the support's lsoa_code (the 2022 data zone). They are refused if any zone is missing or lacks bands A to H. The tool's band vintage 2024_nrs_dz_dwellings is the value of the spec's uk_nrs_dz_dwellings_2024 (sources.yaml:215-217), and the tool tests pass.

CI. Tests run 42209756229 at 3778eb2 was still running at 12:05Z. lint, wheels, select-countries, installed-emitter, integration-uk and engine-uk (3.13 and 3.14) have passed. engine-free and engine-us (3.13 and 3.14) were still running; on round 2's run engine-free took about 1 hour 40 minutes. The run at 77b1eb1 was cancelled.

Verdict: items 1, 3, 4 and 5 are closed. Item 2 is closed apart from its SPI consequence in item 6, which is the one thing to settle before the move relies on the SPI fold. It moves no output today.

🤖 Generated with Claude Code