Repository navigation
Move the UK spine to geography first (#931) - #1538
Draft
juaristi22 wants to merge 21 commits into
Draft
juaristi22 wants to merge 21 commits into
juaristi22 wants to merge 21 commits into
Conversation
…ine-block machinery (#931) Per-clone engine blocks in the dense measure step, and the per-clone spine batches that follow, run policyengine-uk over part of the pool, so the formulas that read beyond their own households see the part. This adds the pieces the next commits wire in. - microcosm.frame.adapters.engine_population is country-agnostic: numpy and duck-typed simulations, no engine import. It holds the pool-share correction c_b = f_b * S_b / S with its zero rules, checks (a) and (b), and injection into a simulation, its baseline and every sibling, refused once a branch exists or a recorder is armed. A per-instance _run_formula recorder, which core's clone rebinds into later branches, and a known-periods walk feed a judge over declarations, parameter predicates and pass policies. A held value is judged unless the simulation holding it ran it after arming, and a run reaches the position window by the last year its own period covers. The verdict carries names, periods and booleans only. Household batches, the nesting check, the scatter back to one engine's order, a float32 share bound that follows numpy's pairwise-sum leaves and counts the adds forming each key, and a per-row bound for values that add share terms to others complete it. - microcosm.frame.adapters.policyengine_uk_population classifies pe-uk 2.174.1's 19 WHOLE_POPULATION_VARIABLES: three corrected shares, two stored inputs, eight zero at policy, attendance inert through private_school_vat_change, six refused ranks. It adds the window from REBALANCING_YEARS (2026 on), the constructor's structural reform factories, sha256 pins of the sources the classes rest on, the spine batch column, and uk_engine_population(), which imports pe-uk and validates only when called, failing closed. Nothing imports either module yet, so no kernel, stage or target hash moves and no output changes. Engine-free tests drive the machinery on fake core-style simulations, covering each guard condition a small change to the module could drop. Engine tests hold the declaration to the installed engine: list, adds, pins and what they cover, predicates at 2024 and 2025 and moved off current law, reform factories, window, the readers of attendance and private-school VAT, and drift lists of axis-free reductions and of whole-array branches in formulas and the engine functions they reach. They correct two synthetic clones that differ to one engine over the pool within the float32 bounds, household_tax per row, with household-local outputs bitwise equal. A build test holds the batch column and the share inputs to the build's own constants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The annual-income simulation carries each input's other spells from the dataset's year by the factor the engine projects that input by. The factor was the median of the engine's own year-on-year ratios over the rows the simulation holds. That is a statistic of whichever block ran, so per-clone engine blocks that differ would carry the same spell by different factors, and no block would match one engine over the pool. Following María's ruling of 6 Nov, the factor is now the product of one plus the engine's own rate, for each year after the dataset's, in the growth series policyengine-uk's dataset projection uses for the input: average earnings for pay, mixed income per head for profit. The rates are read from the simulation's parameters. The engine's rows must still agree with the factor within 1e-5, or the resolver refuses, as before. This moves data on today's national and dense lines. The median sat within float32 rounding of the parameter, so an annual input with other spells can move by one float32 step. The *_annual matrix entries, calibrated weights and H5 bytes can move, and other_spells_engine_factors changes in every resolver receipt that carries it. A K=1 parity check should expect *_annual inputs within one float32 step, not an identical H5. The full target and problem kernels and the national target kernels re-key with measure_simulation; no spine stage does. An engine-free test holds the factor to the parameters and to the rows' tolerance. An engine test pins the declared series to policyengine-uk's own uprating map and checks the factor on a synthetic frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he whole-population formulas it runs (#931) After the move the dense measure step's per-clone engine blocks are no longer identical copies, so policyengine-uk's share formulas (shareholding, corporate_land_value, consumption_shareholding) would read a block's denominator instead of the pool's. - The shared engine_population module gains EngineBlock, the life of one block on its own engine. It validates the declaration, refuses the position window and periods before the data, and runs checks (a) and (b). It sets each share formula to its own value times c_b = f_b * S_b / S in the simulation and its baseline, and arms the formula-run recorder. Every simulation built later gets the same arrays and recorder. At the end it checks that each still holds the block's keys and weights, then judges them all. A record pass records refusals and a raise pass raises; work after the verdict is refused. The country passes its declaration and a pass. validate_pool_shares and PopulationDeclaration.correction_reads come with it. - UKMeasureResolver takes engine_pool_shares (scratch mode only) and opens an EngineBlock with the UK declaration and the measure pass, which records and never raises. A sibling that would set a key, its inputs or the weights is refused. close_engine_block judges the block. - full_measure builds each block's pool shares from the totals the representation already records and passes them to the block's resolver. After the blocks run, exact holds only if every block validated the declaration, passed both checks, applied the correction in every simulation it built, kept its keys and weights to the end, agreed with the others on its growth and annual-income factors, and got a clean verdict. exact_basis names this basis. exact_scope says float32-level, not bitwise, and names the Universal Credit rebalancing draw the window refuses. The identical-copies checks stay as diagnostics. block_sensitivity matches the land rows' engine variables, so present_in_this_run is no longer always empty. - The two frame modules join UK_TARGET_COMPILE_MODULE_NAMES. The fake engine the shared tests use moves to test_support so the build tests can use it too. No data changes on today's lines. Identical copies take a correction of 1 and give the same matrix and local metrics, bitwise, as without it, and a single-block resolver records none of the new keys, so national receipts keep their bytes. Today's K=15 dense receipts change by design: - the representation gains exact_basis, share_correction, population_reach and annual_income_projection_factors; - exact_scope is rewritten, and the release manifest's engine_measurement limitation copies it; - block_sensitivity is rewritten; - each block's resolver receipt gains engine_pool_shares and engine_population_reach; - exact now also needs a clean guard verdict, a stricter release guard on that line. The full and national target kernels re-key; no spine stage does. On a synthetic pool of three clones that differ, three blocks match one block over the pool: household-local and annual rows and every local metric bitwise, and the land rows and each block's share formulas within the float32 bound. Uncorrected land rows miss by more than 1%, and a decile read, a 2026 period and a counterfactual on corporate wealth are each refused. Engine-free tests run EngineBlock on fake engines. They drive checks (a) and (b), a sibling that moves its keys or weights, the window and a raise pass, and hold its receipts to the representation's finalisation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…931) One simulation over the move's K=15 spine pool does not fit in memory, so the UK adapter runs a call over more than one clone clone by clone. It decides from the frame, never from a constructor flag: a household table carrying household_clone_index with more than one clone batches, and a frame without the column, or with one clone, takes the single simulation it always has. National never batches. - The shared engine_population module gains materialize_in_batches and frame_key_sums. Before any engine is built, the driver judges the call's requests: a period in the rebalancing window, a refused rank, a stored formula the frame does not supply, and attendance or its levels and carriers while it is unstored. It forms each batch's share from the frame, c_b = S_b / S at unscaled weights. Each batch, in ascending clone order, then runs an EngineBlock under the spine's raising pass: checks (a) and (b), the pool's shares set in the simulation and its baseline, the recorder armed, the requested values, the verdict. The engine is deleted and collected before the next is built. Every batch must grow the keys and the weights by one factor. People come back in the frame's order and groups by id, as one engine returns them. The driver refuses a recording pass, since its arrays carry no verdict. - PolicyEngineUKEngine passes the UK declaration and the spine pass. It gains simulation_batches(frame), parameters(period) for the stage-52 restatement, and batch_record(). Its optional clones, the spine's K once G1 adds it, refuses a frame carrying any other number of clones, so no call runs per clone or over a sub-pool. microcosm.frame.rules and the RulesEngine.materialize signature are unchanged. - The spine sidecar's rules_engine gains engine_batches only once a call batched. The engine identity is still hashed from the provenance alone, and no batch cap is added. - graph_kernels binds engine_population and policyengine_uk_population beside the adapter in all 16 engine-calling stages. A test derives those stages from their engine calls. No data changes on today's lines: no spine frame carries household_clone_index before G1, so every call takes the single simulation, unchanged. Identities move. Every spine stage kernel hashes graph_kernels.py, so every spine node re-keys, and the 16 engine stages also bind the two modules. engine_population.py is a target compile module, so the full and national target kernels re-key again. The spine graph declaration (the H2 fixture uk_spine.json), source_stages.json and the coverage manifest are unchanged. An engine test runs a synthetic three-clone frame whose clones differ, interleave by id and list people out of order, at the dataset year and projected a year on. Against one simulation over the same frame, the share formulas match within the float32 bound, household_tax within its per-row bound, and household-local values bitwise, with people and groups in one engine's order. A one-clone frame is bitwise the single simulation. A batched attendance request, a decile, a 2026 period, unstored months and a frame with fewer clones than K are each refused before any simulation is built. Engine-free tests run the driver on fake engines, one engine alive at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ine values (#931) policyengine-uk's attends_private_school ranks household net income over the people a simulation holds, so once a spine's engine calls run clone by clone, one clone's simulation would rank that clone alone. Stage 52 now restates the formula's last step itself, over every person in the frame: the members' income tax and National Insurance summed per household as policyengine-core sums them, net income from the household's market income and benefits, the household's weight once per member, microdf's MicroSeries.percentile_rank (the rank the formula calls) over the people of positive weight, the attendance rates and population factor from the engine's own parameters (parameters(period), at the two paths PRIVATE_SCHOOL_ATTENDANCE_PARAMETERS names), the one-million floor, and the stored draw for under-18s. Every input is the engine's own value at the calibration year, decided by a household's own records. - Where the engine computes the frame in one simulation (national, and every spine today), the stage asks for the engine's attendance beside those inputs, stores the engine's flags, and raises if the restatement differs from them bit for bit, so each such build re-checks it on its own data. Where the adapter computes the frame clone by clone (simulation_batches above one), it asks for the inputs only and stores the restatement. An engine without simulation_batches runs one. - The evidence gains rank_route (engine_with_parity or pooled_restatement) and engine_parity_checked, a boolean: whether the engine's flags were checked. A failed check raises, so neither records a mismatch. weighted_pupils stays. The parity error names the formula, never a count. - The stage's notes in sources.yaml describe both routes. The notes in source_stages.json are re-derived from it, and the release input coverage manifest is regenerated (its source_manifest_sha256 fields). - Left for G1, which makes the clone-by-clone route live and regenerates the spine fixtures: the stage's source line ("evaluated by the pinned engine"), the notes' sentence that the stage materializes the engine's attendance, and the H2 fixture's sources/fixture.json, which keeps the stage's earlier notes until tools/graph_uk_spine_fixture.py runs. All three hold while every call takes one simulation. The store_rules_engine_input operation describes both routes: each stores attendance as an engine input from the stored draw, at the calibration year and the spine's design weights. No data changes on today's lines: no spine frame carries household_clone_index before G1, so every call takes one simulation and stores the engine's flags, as before. Identities move. Stage 52's kernel hashes private_school_attendance.py, so stage 52 and every node after it re-key. The packaged resources enter the full and national target kernels' country-spec fingerprint, so those re-key again. National's stage-52 evidence gains the two fields. The spine graph declaration (uk_spine.json) is unchanged, since stage contracts do not cover notes. An engine test runs a synthetic pool of three clones of twelve households: some alike in every clone and some not, most with several children, weights spread about one to ten, ties, a household of no weight, and each clone under the floor while the pool is over it. Household ids interleave the clones and the people are listed out of order. Each child's draw sits 3% under or over the rate the formula gives it over the pool, from the formula's own percentiles and the engine's parameters, so a rank, rate, interpolation or factor a few percent off would store other flags. A spy on microdf's percentile_rank shows the restatement hands it the formula's net incomes and weights and gets its percentiles back, bit for bit, in one simulation and clone by clone. The stage run clone by clone stores what one simulation stores, where a clone's own simulation imputes nobody; a population factor of zero, or 6% lower, breaks the parity check; below the floor nobody attends on either route, while design weights just under it impute on both, since the formula reads the engine's weights at the calibration year; and a frame of one clone stores the engine's attendance as the stage asked for it before. Engine-free tests run both routes, the parity refusal and the floor on a fake engine with a stand-in rank. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raph doc (#931) The dense runbook and the local pre-flight still said a per-clone engine run was exact when its blocks were identical copies scaled to the pool. Exact now means every block corrected to the pool's share denominator with a clean guard (exact_basis pool_share_injection_with_population_guard): equal to one engine over the pool to float32 rounding, not bitwise, with nothing computed in policyengine-uk's Universal Credit rebalancing years (2026 on). - docs/uk-dense-release-assembly-runbook-762.md says so where it describes --engine-blocks, the pre-flight and the engine_measurement limitation. It adds that a calibration year or dated measure's year of 2026 or later leaves every per-clone run inexact and refuses every batched spine call, and that the pre-flight does not require exact_basis, so a receipt from before the correction passes too; that is open for Max. - docs/uk-full-build-graph.md gains a section on the corrected measure blocks and their guard, on how a spine engine call over more than one clone runs clone by clone and what refuses it, and on stage 52's rank over the whole frame. - tools/preflight_uk_local_release_candidate.py: the docstring and the comment over the engine check only. The gate code is unchanged. test_uk_local_release_preflight gains two cases: a per-clone resolution whose representation has another mode, or another block count, fails however exact it says it is. - Two changelog fragments cover group (0). One describes the engine blocks, the spine batches, stage 52 and the shared modules. The other describes the annual-income factor for other spells, the one data change, which can move an annual input by one float32 step on both lines. Only comments, docs, fragments and tests change. No kernel hashes these files, so no node key, H2 fixture, source_stages.json or coverage manifest changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…estatement (#931) Review follow-ups to batching spine engine calls by clone. - The spine sidecar's engine_batches is now read from the final frame, never from the calls this process ran. A resumed spine whose engine stages hit their checkpoints runs no kernel, so it under-reported batches; now it records what a fresh run does. PolicyEngineUKEngine.batch_record(frame) takes the same decision and refusals as simulation_batches. - PopulationDeclaration gains restated: stored formulas a caller may compute over the pool from household-local engine values, with the parameters each reads. Each pe-uk simulation builds and edits its own parameter tree, and parameters(period) is the adapter's fresh copy. So a new check_restated_parameters holds every batch's tree to the adapter's wherever a restatement reads it. The UK declares attends_private_school with its attendance rates and population factor, the parameters stage 52 reads. - PolicyEngineUKEngine(clones=...) accepts any whole number. A frame that drops the clone column still takes one simulation; G1's integration check rules that out after G1. - The test that finds the engine-calling stages matches a materialize call on any receiver, and any transform taking an engine. It keeps isa_subscriptions, which only checks the country, on an asserted allow-list. No data changes on today's lines: no spine frame carries household_clone_index before G1, so every call still takes one simulation and no sidecar gains engine_batches. Identities move again: every spine node re-keys through graph_kernels.py and the engine modules, and the full and national target kernels re-key through the two frame modules. The spine graph declaration (uk_spine.json), source_stages.json and the coverage manifest are unchanged. An engine test adds a frame carrying none of the keys' inputs, which gives the pool's values bitwise. A batch whose own tree moved a restated parameter is refused before it computes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The WAS donor's income, DVTotinc_bhcR8, is net regular household income before housing costs less council tax: HBAI's concept. Both WAS-trained stages matched it to the engine's household_net_income, which also takes off stamp duty, the TV licence and the business rates and corporate stamp duty that shareholding spreads over the households the engine simulates, and adds benefits in kind. At was_lisa (stage 24) those shares already read the corporate wealth was_wealth had just drawn, at the spine's weights before calibration. Both stages now read the engine's HBAI net income before the reset to zero, the concept LCFS, ETB services and NTS read (#1164), named hbai_household_net_income on donor and recipient (María's ruling, 6 Nov). - was_wealth materializes UK_HBAI_NET_INCOME_VARIABLE, and UK_WAS_ENGINE_PREDICTOR_NAMES gives it the predictor name UK_HBAI_NET_INCOME_PREDICTOR; the donor cleaning maps DVTotinc_bhcR8 to the same name. The chain's segments, their order and seeds are unchanged. - Its derive step declares the donor column in direct_maps, and the drift check refuses a declared direct map the cleaning does not read. - was_lisa takes the WAS household predictor, so the ownership logit, the balance forest and the realised profile's income tertiles read it. The profile key becomes by_hbai_household_net_income_tertile and the logit's feature log1p_hbai_household_net_income. Its declared engine reads are now what it materializes (UK_WAS_LISA_ENGINE_PREDICTORS): num_adults and num_children, from the FRS family roles since #1095, leave the list. - sources.yaml carries the declarations and both stages' notes. source_stages.json is re-derived from it, the release input coverage manifest is regenerated (its source_manifest_sha256 fields), and so is the H2 fixture: sources/fixture.json carries the two stages' declarations and notes, and stage 52's notes from 019f5d4, which waited for the tool to run; uk_spine.json carries the two stages' new contract hashes. No synthetic fact in the fixture moves. - docs/uk-lisa-1003.md and a changelog fragment describe the concept. What moves: every WAS draw can move on both lines, and with them the Lifetime ISA draws and the later stages that read wealth (means-tested capital, CGT selection, property components, high-value support). The size is unmeasured; a sampled f010 arm measures it. was_wealth, was_lisa and every later spine node re-key. Tests: the WAS recipient reads the engine's HBAI income, not a household_net_income decoy, and the donor maps DVTotinc_bhcR8 to the same name; the packaged declarations name both sides; drift to the old engine read or a wrong direct map is refused; the LISA recipient and donor carry the WAS predictor and its declared engine reads are what it materializes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of 640b2cf. Its text said the WAS and Lifetime ISA stages read one income concept on both sides. The code matches the ruling, but the two sides are not the same income: policyengine-uk's HBAI net income also counts HBAI's in-kind items (free school meals, school milk and fruit, the free TV licence's value, Healthy Start vouchers, the Tax-Free Childcare top-up) and the Warm Home Discount, and the WAS user guide defines the income behind DVTotinc_bhcR8 only as total net regular household income less the council tax paid, so how it treats pension contributions, maintenance paid, student loan repayments and negative incomes is not verified. - The was_wealth and was_lisa notes, docs/uk-lisa-1003.md, the changelog fragment and the code comments call the recipient's income the engine's closest concept to the donor's and name those residuals. The was_wealth notes keep the income among the predictors that are not fully like-for-like. - household_net_income is described as adding the imputed value of NHS, education and transport spending, not "benefits in kind", which HBAI's income also counts in part. - Only the LCFS and ETB services stages name the income as a predictor; NTS reads the same engine variable to build gross income. The comments and test docstrings say so, and the two tests are renamed to match. - The changelog's What moves adds the calibrated weights on both lines. - source_stages.json is re-derived from sources.yaml (the two notes), the release input coverage manifest is regenerated (its source_manifest_sha256 fields), and the H2 fixture's sources/fixture.json carries the two notes. uk_spine.json does not move: notes are outside the stage contract hash. What moves: nothing in the data or any contract hash. Text only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#931) María's ruling of 6 Nov on the move, in line with workstream 2 of microcosm#1521 and the first must item of Vahid's review there. Two of the three ONS land rows cannot usefully bind weights: - ons.land.corporate_land_value (E43B, £2,063,534m): policyengine-uk's corporate_land_value hands out its aggregate by each household's share of the weighted corporate-sector wealth, so a released file meets that aggregate at any weights. Bound to the column measured at the starting weights, the row pinned only the weighted mass of the WAS corporate and private pension wealth, which no other row targets, not land. - ons.land.land_value (E44R, £6,872,455m) counts the land of financial corporations, government and NPISH, which the engine's land_value does not hold. With the household row (E44N, £4,485,171m) and the corporate row it asked for £323,750m that no weights could supply. Both get entries in uk/calibration_measure_exclusions.json in the obr.council_tax_england shape (tracking microcosm#1521, approved 2026-11-06, expiring 2027-02-06) and are measured on every evaluation. ons.land.household_land_value stays bound. Its binding note in uk/uk_population_targets.json said households_and_npish; E44N is the households sector alone, as the Chronicle package labels it, and the note now says so. The contract is hand-written. tools/generate_uk_target_references.py reads no binding note, so target_references.json and the membership report do not move (their regeneration test needs the pinned Chronicle feed, which is not on this machine, and skips here). What moves: the solve drops two rows on both lines, so calibrated weights move slightly; a calibrate-only rerun on the move's spine attributes it. Released corporate land value still equals the engine's aggregate. Tests: the packaged register's count and census take the two rows, and a new test pins their fields, the published values the reasons quote against target_reference_membership.json, the household row staying bound and its corrected note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…931) Follow-up to 410e822. Its test pinned the register's two entries, their fields and the published values they quote, but not what the entries do to a run. The new engine-free test compiles the three committed ONS land references with stand-in values and applies the packaged register's land entries, as both target loaders do after reconciliation. The solve surface keeps ons.land.household_land_value alone, the exclusion receipt carries the two entries' review fields, and the compiled register, which every run keeps beside the solve surface as its band-edge register (a national run writes it as national_contract_registry.json), still holds all three. On a synthetic four-household frame the national problem encoder binds the household row alone, with the receipt and the compiled register's digest in its bindings, while the compiled register materializes and scores all three rows at their weighted values. What moves: nothing. Test only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of 410e822 and 161c777. The two entries hold the right rows, but some of their text contradicted itself, the record of what moves was short, and one claim in 410e822's message was wrong. - The total-economy reason, the changelog, the test docstring and the census comment now tie the £323,750m gap to the rows: E44R also counts the land of financial corporations, government and NPISH, which neither E44N nor E43B covers, and policyengine-uk's land_value is the sum of the two columns those rows read. The old text said the engine's land_value does not represent or hold financial corporations' land, while the corporate entry says policyengine-uk describes its corporate aggregate as the land of non-financial and financial corporations. That entry now says "describes as", not "labels". - The census comment no longer says the corporate row is met by construction: the column meets the engine's own aggregate at any weights, not E43B. The adjudications give Vahid's suggestion in his order (met by construction once the engine's parameter comes from the target's ONS series), and say the calibrate-only rerun is to attribute the change, since it has not run. - The two held rows' binding notes in uk/uk_population_targets.json record the hold, as the other held rows' notes do. The parity register's land concern names the two exclusions; uk_data_target_parity.json is regenerated with tools/census_uk_data_target_parity.py, and --check passes. The UK_BLOCK_SENSITIVE_MEASURE_COLUMNS comment and its test's comment no longer say the national targets bind the two variables. - 410e822's message said the target-reference regeneration test skipped because the pinned Chronicle feed is not on this machine. The feed is here. tools/generate_uk_target_references.py, run with the test's arguments on that feed (facts and manifest hashes and the row count match uk/chronicle_feed.json) and on this commit's contract, reproduces target_references.json and target_reference_membership.json byte for byte. It ran directly, through the engine slot, rather than inside pytest, whose own load of the 10 GB feed would double the memory. The local generator keeps only local-level contract rows, so the land rows' notes cannot reach the local surface. What moves, all from 410e822 (this commit is text, a regenerated parity register and tests): - Both lines split each family's equal share of the objective evenly over its bound rows (the dense line within each grain, where the land rows are amounts). ons.land.household_land_value, now the ons_land family's only bound row, carries the family's whole share: three times its loss weight before, on both lines. No other row's weight changes. The calibrate-only rerun measures this together with the two dropped rows. - The national frozen scoring register (national_target_registry.json, the approved register) loses the two rows, so a national run given a frozen register derived before 410e822 refuses, as with obr.ni_employer. The dense loader checks the reconciled register, which keeps them. - The incumbent and release-cut scores, uk_target_fit and calibration_diagnostics.json read only the rows the solve binds, so the two rows leave them. - Neither line resolves land_value or corporate_land_value any more: the dense measure receipt's national_inputs falls by two, and a per-block dense run's block_sensitivity.present_in_this_run is empty. Evaluations that score the un-excluded register (tools/evaluate_uk_incumbent_surface.py) still measure the two rows, as they do every held row; 161c777's test shows the compiled register can score them, doing the scoring itself. - The dense release lists the two exclusions among its limitations, each with the first 300 characters of its reason, which cite only published ONS figures and code facts. - The edited files are UK country-spec resources, so the national and dense target, measure and solve nodes re-key; full_measure.py feeds only those kernels. Spine nodes, source_stages.json, the release input coverage manifest and the H2 fixture do not move. The regional_home_values stage note still says the home value reaches three national rows, ons.land.land_value among them. It changes with group (c)'s rewrite of that stage, which regenerates source_stages.json and the H2 fixture anyway. Tests: test_the_corporate_and_total_land_rows_are_measured_diagnostics also pins the "4.7 percent" the total reason quotes and the two held notes. test_the_two_land_diagnostics_are_measured_but_not_fitted adds a bound row of another family (obr.tv_licence_fee) and pins the household row's family_equal weight at three times its share beside the other two land rows, with the other row's weight unchanged; its docstring says it applies the entries as the loaders do, without calling them. test_uk_measure_simulation, test_uk_full_measure, test_uk_data_target_parity, test_uk_population_targets, test_uk_incumbent_surface_evaluation, test_spec_source_citations, test_uk_target_references (its feed cases skip without CHRONICLE_UK_FACTS; the generator run above stands in for the national one) and test_uk_solve_scope pass, and ruff is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every UK copy layer took its id offset from ten to the digit count of the largest id it saw. Once (b) clones households at the spine's start, every offset would grow with K, and the K=1 national spine would stop being the clone-0 slice of the K=15 spine. Copy ids now come from fixed decimal fields declared in uk_runtime/copy_id_table.py, so every copy id is a function of its source id and its copy path, at today's stage order. The table is UK-local (María's rulings of 7 Nov); the shared table, a declaration digest and the int32 check come with the follow-up stage. - The table, least significant first: vintage (10 ordinals at 10**8, kept for pooled FRS vintages; nothing mints it), geographic_support (100 at 10**9), spi_support_channel (3 at 10**11: the SPI copy at 1, the band donor at 2), high_value_support (6 at 10**12), cgt_support_split (10,000 at 10**13), cgt_incidence_clone (2 at 10**17) and cgt_residential_split (8 at 10**18). The largest mintable id, about 7.2e18, is checked below int64 when the module loads. Helpers: radix, require_below, mint (refuses an ordinal outside its field, a row that already carries one and an int64 overflow), ordinal and parent. - Mint sites: frs_spine refuses raw ids at or above 10**8. Stages 16 (with its _remap_ids, which had no overflow check), 17 (sources read from the source-id columns), 26, 42, 43 and 46 mint through the table. Stage 42 refuses a household needing more than 10,000 copies with a fixed message. The anchor (45) pairs each twin by the twin field's radix. The executor takes each id-offset stage's declared radix, refuses an incumbent id at or above it and recovers parents on exact int64 ids; the row walk it replaces upcast all-numeric rows to float64, inexact above 2**53. Refusals print counts, never ids. The tests-only replace path mints through the table and refuses a base whose ids reach the SPI field. - Dense: the dense expand keeps its multiplier above the spine's largest id. A table-minted spine's twins and arms leave no int64 room above them, so every dense entry point (uk.full.expand, the ladder and crosswalk clone routes, the row-wise tool's plans) refuses n_clones > 1 in one message naming #931 and the clone-first spine. Dense K>1 builds wait for (c). K=1, and a spine without copy rows, expand as before; the expand kernel no longer multiplies an empty column by a multiplier above int64. - Declarations: the id_remapping texts in sources.yaml, source_stages.json and the code dicts name the table field. stack_zero_weight_donors and stack_income_band_donor_households gain the key, required by sources.schema.json and checked against the code. copy_id_table is bound into frs_spine and every minting stage, spi_support into stages 16 and 17, rowwise_geography into 43 and 45 and cgt_support into 26. What moves: base FRS rows keep their ids. Every copy row's five id columns change: SPI copies +10**11, band donors +2*10**11, high-value copies k*10**12, CGT copies k*10**13, twins +10**17, arms j*10**18. Relative id order is unchanged at K=1, so sorts, ties, ranks and folds are unchanged. Values move on copy rows through every draw keyed on the copy's own id: stage 18's stage-2 fill, stages 19-24, 27 and 28-41, every twin's Advani-Summers prior (so the CGT gains layer re-realises), stages 48-54 and policyengine-uk's warm_home_discount_random_draw on copy households. Base rows move second-order through pooled steps: stages 26-28's selections and walks, the residential solve, the anchor, the CGT walks and calibration. Draws keyed on person_source_id do not move. National ids pass 2**53 from stage 42 on. Dense K>1 builds are refused until (c). Every node re-keys. The coverage manifest's source digests and the H2 fixture are regenerated: the fixture's declarations, and its synthetic CGT asset-type, property purchase and rental facts, which are sized to the synthetic spine's copy-row values. Left for 1b: the identity tool's decoders still read the retired rule, so 11 tests in test_uk_identity_stability_receipts fail until 1b; also size_evaluation's decoders, the remaining id-printing messages and the stochastic_assignment binding sweep. Tests: the table's fields, bounds, order and ceiling, and the helpers' round trips and refusals; each copy stage's ids against its roots, found by lineage; a K=1 chain through all six layers sorts as the retired rule did and keeps base ids; clone-first K=1 and K=3 frames nest exactly and the retired rule does not (test_support/microcosm_build/ uk_clone_first_frames.py); the anchor pairs at a one-digit CGT ordinal; the executor's refusal and parents, exact above 2**53; the dense refusal and the K=1 path; the fixed CGT message; an AST guard against the retired rule; the binding tests. H2 parity passes on the regenerated fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three reviews of 22bcdbe found texts that overstate or misstate what it does, and refusals and dense paths that no test drives. This commit corrects the texts and adds the tests. The identity tool's decoders and the other items ruled into 1b stay in the next commit. - Stage 46's notes said the geography identity kernel and the identity tool read the arm rule. The kernel reads lineage columns, and the tool decodes the retired rule until 1b, so the notes now name the executor only, as the module docstring does. source_stages.json is re-derived, and the coverage manifest's source digests and the H2 fixture's notes are regenerated. - The row walk the executor replaced read a row of only integer and float columns as float64; an all-integer row stays int64. The _source_lineage docstring and the changelog now say so, and the exactness test carries a float column, on which the replaced walk fails. - The dense refusal fires wherever the dense offset would overflow int64, as a spine with residential arms at multiples of 10**18 does at K >= 2. Its docstring, its message and the changelog now say that. require_below's message no longer says that only a later stage may fill the field. - The changelog says each copy row now sits at its parent's ids plus its layer's offset, where it used to sit a power of ten above the largest id its layer saw. It also says that national ids pass 2**53 from the twin on (and at stage 42 from ordinal 901), that only the new refusals print counts, that create_uk_spi_support_tables no longer takes id_multiplier, and that copy rows join across versions by lineage, not by build ids. - The CGT split's bound check moves into a helper so its boundary can be tested. The lineage test helper reports counts, never ids or keys. What moves: no value or id. Every node re-keys again, through the text in graph_kernels.py, copy_id_table.py and rowwise_dataset.py and the helper in cgt_support.py. Tests: - Stage 16 on clone-first K=1 and K=3 frames nests exactly. - The executor takes the declared radix where the frame's maximum would give another, names each stage's own field, and refuses parentless rows by count. - The id_remapping texts quote the table's radices. - Stages 26 and 42 mint from the immediate parent on frames with SPI copies in FRS id spaces. - frs_spine (end to end), stages 42, 43 and 46 and the replace path refuse incoming ids by count. - The CGT bound holds 10,000 rows and refuses 10,001. - A drifted SPI id_remapping is refused. - The copy-id binding set is derived from imports. - The dense refusal and the K=1 path run through the pool expand, the crosswalk route, both dry-run plans and the graph's uk.full.expand, on ids above 2**53. - The twin's negative-prior count is pinned. - H2 parity passes on the regenerated fixture. The 11 tests in test_uk_identity_stability_receipts still fail until 1b, as at 22bcdbe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
22bcdbe and its follow-up 2f0f8ce moved every mint site, the executor and the anchor onto the copy-id table's fixed radices. Two decoders outside the stages still used the retired rule. The identity tool read the CGT support copies and the residential arms at ten to the digit count of the largest pre-split id. The dataset-size evaluation multiplied by a dense multiplier of 10**19, which a one-clone expansion of a table-minted spine records, and overflowed int64. This commit moves the identity tool onto the table and both decoders onto exact integers. It also makes the copy path's refusals count-only, adds static guards against the retired rule, and binds the shared keyed-draw helpers into every stage that draws through them. - Identity tool: _support_copy_lineage and _residential_arm_lineage take each copy's ordinal and parent at its field's declared radix (copy_id_table.ordinal and parent). They read the household table only, as exact int64; a non-integer id column is refused. Each receipt's id_multiplier now records the declared radix (10**13 for the support split, 10**18 for the arms). Before any receipt, the tool refuses a spine whose copy ids disagree with their lineage columns under the table, as every spine built before the table does. The refusal is one count-only message saying to run the tool at the commit that built the spine. The refusal and the decode are two small functions, so a retired-rule path could replace them and nothing else. - Size evaluation: dense ids decode back to spine ids exactly, at the multiplier the expansion recorded (the new id_multiplier argument of weight_tables), else the spine's, else the clone-0 ids'. 10**19 is included. A clone offset that int64 cannot hold, a non-integer id column and a multiplier that does not fit the dataset are refused by count. A dense file that keeps its true sources computes its spine stretch again; it used to fail on int(None). - Count-only refusals: - rowwise_geography: _remap_ids' overflow (it printed the multiplier and the largest id), clone_entity_frame's and assign_household_geography's duplicate checks (up to five ids), and the crosswalk route's check for duplicate household ids before cloning; - rowwise_dataset: its uniqueness and person-link checks; - spi_support: its uniqueness and person-link checks, the donor's duplicate and missing-source checks, and the selected-household check; - spi_spine: the stage-14 retained-leaf check; - cgt_structure: the CGT carrier check; - cgt_imputation: the redraw's missing-weight check. - Guards: only the dense line's own modules name id_multiplier_for_values until (c), each with its reason: rowwise_geography, rowwise_dataset, the package re-export, size_evaluation and the row-wise tool. Every module that mints, bounds or decodes copy ids reads the table, the identity tool among them, and names no retired rule. - Bindings: every spine stage whose module or bound helpers import microcosm.build.stochastic_assignment now binds it. A derived test holds the set, and a helper-only edit to it re-keys all 36 such stages and no control. No dense node calls it. - Texts: the stage-46 notes and the residential split's docstring say the identity tool reads the arm rule too. The changelog's binding sentence names only the helpers that mint or weigh copies (spi_support, cgt_support, rowwise_geography). What moves: no data value or id. Stage keys re-key on 31 stages that newly bind stochastic_assignment: frs_spine, frs_take_up, frs_person_draws, frs_household_draws, frs_brma, frs_annual_income_spells, spi_income_band_donors, hmrc_spi_income_spine, spi_housing_shell, was_wealth, was_lisa, property_components, property_purchases, nts_bus_travel, lcfs_consumption, etb_vat, etb_services, uc_reporter_redraw, spi_benefit_coherence, uc_capital_coherence, uc_take_up, uc_legacy_closure_take_up, pension_credit_take_up, child_benefit_take_up, uc_deduction_attributes, cgt_incidence_clone, cgt_incidence_anchor, salary_sacrifice, student_loans, isa_subscriptions and childcare_copy_take_up. Every spine node re-keys anyway through graph_kernels.py, as do the dense nodes that hash rowwise_dataset.py. Regenerated: source_stages.json (the stage-46 notes line), the coverage manifest's 25 source digests and the H2 fixture's notes line in fixture.json; uk_spine.json is unchanged. Tests: the 11 identity-tool tests that failed at 22bcdbe and 2f0f8ce pass again. The hand-built split frame is minted through copy_id_table; the other fixtures come from the real stages. New tests cover: - the old-spine refusal, through main and directly; - the arm decoder above 2**53; - a support copy at ordinal 9,999, which a float read misplaces; - the size evaluation's recorded, spine and clone-0 paths at 10**19, an older dense run at its own multiplier, the int64 refusal and the source-column stretch; - the count-only messages, with no id in them; - the executor's household_source_id branch; - the two static guards and the binding test. Synthetic mutations make each guard fail. The parent's weight_tables fails the four new size tests (OverflowError, TypeError). The 114 touched engine-free test files pass. The H2 fixture regenerates byte-identically, and H2 parity and the engine graph and ladder-clone tests pass in the engine slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three reviews of 50b7ea1 found no must. This commit takes their should findings and the cheap nits, and corrects four claims 50b7ea1 made. - Bindings: stage 14 (frs_hmrc_spine_leaves) draws its employer pension contributions on keyed uniforms through employer_pension_contributions, but did not bind stochastic_assignment, because the derived test read direct imports only. It now binds it, and the test follows one more step: a stage draws when its module or a bound helper imports the helpers, or imports another first-party module's function that draws through them (a fixed point over the import graph, re-exports and module attributes included). spi_support_channel, which shares spi_spine with stage 14's draw and stage 18's SPI imputation but calls neither, is the one exemption, pinned to those two names. employer_pension_contributions itself stays unbound until the wider binding sweep. - Size evaluation: the spine-stretch refusal printed no count and chained a pandas KeyError listing every missing household id; it now counts the dense rows before the lookup and raises unchained. The misfit refusal says an inferred multiplier reads household ids only and names weight_tables(id_multiplier=), the one route for a recorded multiplier. - Count-only refusals: the row-wise tool's dry-run input checks now call the build's own (rowwise_dataset._require_unique and _require_person_links) instead of listing up to five ids, and the spine sampler's three refusals in national_sampling count families. - Guards: an AST guard refuses ten raised to a len(str(...)) power outside rowwise_geography (the dense rule), copy_id_table (field spans) and national_sampling's legacy candidate decoder, so the retired rule cannot come back written out inline. - Texts, correcting 50b7ea1: - a retired-rule decode would not replace two functions and nothing else; it would also thread the person and benefit-unit tables back through the two lineage functions and their four callers, and pair the twins in the tool at the three pair_clone_households call sites (the tool's comment now says so); - only the identity tool used the retired rule; size evaluation kept the dense line's own rule and overflowed at 10**19, and its non-integer id refusal names the dtype, not a count; - no caller passes a recorded dense multiplier yet: tools/evaluate_uk_dataset_size.py infers it from household ids (the docstrings say so); - the arm tests decoded a synthetic copy chain, not stage output; the stage-built tests below cover that now. The changelog names what each SPI and CGT stage binds and why, limits the count-only sentence to the UK modules (the shared graph executor's EXPAND checks are unchanged), describes the inferred multiplier, and says the identity tool refuses a pre-table spine with copy rows. What moves: no data value or id. Stage 14's key moves with its new binding; every spine node re-keys anyway through graph_kernels.py, and the dense population nodes that hash national_sampling.py re-key, as they already do through rowwise_dataset.py in this push. No generated file changes. Tests: new size tests decode an older dense run at K=15 on odd ids above 2**53 at all three bases (a float64 subtraction fails it), and pin the uncovered-spine refusal (no id in the message, its cause or the traceback) and the negative clone_index refusal. New identity-tool tests hold the old-spine gate to each stored ordinal exactly and to an empty vintage field, refuse a support copy whose parent the split did not copy, and run the gate on frames that stages 16, 17, 26, 42, 43, 45 and 46 built (stage 44's redraw done by hand) and the E8 residential, clone-pair and support-split receipts on the last of them. The dry run, the sampler, the CGT carrier and redraw checks, the SPI uniqueness checks and the support-copy refusals now have exact count-only messages pinned. Synthetic mutations make each new guard and test fail. The 30 touched engine-free test files pass (1,331 tests). The coverage manifest's check passes, the H2 fixture regenerates byte-identically, and H2 parity and the engine graph tests pass in the engine slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CGT support split divided each selected household into ceil(w / 60)
copies of its own weight. Once (b) clones households at the spine's start,
each geographic clone carries 1/K of its source's weight, so a clone's count
would shrink with K and the K=1 spine would stop nesting in the K=15 one.
As María ruled on 7 Nov, the count now reads the geographic clone count
times the row's own weight, so every clone of a household gets the count its
source gets at K=1. Nothing moves at K=1.
- Count: cgt_support_copy_counts(weights, maximum_copy_weight, *,
clone_count) returns max(1, ceil(K x w / 60)). For K > 1 the quotient is
snapped to the nearest integer when it is within K ulps of it, so an exact
multiple of 60 keeps its count at any K (at K=7, 7 x (60m / 7) lands above
60m for 626 of m < 20,000 in a synthetic sweep). The snap gives up the
converse case: a quotient within K ulps above an integer counts one copy
fewer than at K=1. At K=1 nothing is snapped, and the count is
ceil(w / 60) bit for bit.
- Stage: split_cgt_support_households takes a required clone_count, and the
transform reads it from the frame (geographic_clone_count): one more than
the largest household_clone_index, or 1 without the column. A non-integer
column, a missing or negative index, and indices that are not exactly 0
to K - 1 are refused by count. Copies keep w / n, so each weighs at most
60 / K. The selection walk still accumulates each row's own weight, and
nothing sums geographic siblings.
- Receipt and gate: the receipt's parameters gain clone_count and nothing
else. stage_health compares K x the heaviest copy and K x the heaviest
selected household with 60, so a receipt at K > 1 no longer fails
falsely. At K=1 both checks and their messages are byte-identical, and a
receipt without a positive integer clone_count is refused. The identity
tool's recompute reads K from the pre-split table and passes it on; its
receipt is unchanged.
- Texts: copy_rule in the code dict, sources.yaml and source_stages.json
now reads 'copies = ceil(geographic clone count x household weight /
maximum_copy_weight); every copy carries weight / copies'. The stage notes
give the cap as 60 divided by the geographic clone count before the
incidence clone. The docstrings, the cap's comment and a changelog
fragment are updated to match. The mass-change reason is unchanged.
What moves: no value at K=1. On synthetic frames the pre-commit and new
code give the same results: the stage's ids, copy counts, weights (bit for
bit), mass records and frame content identity, the identity tool's receipts,
and the gate's verdicts and messages. The only difference is that stage
evidence gains clone_count. Implementation hashes move for
cgt_support_split, and for high_value_home_support, which binds cgt_support.
So node keys move from stage 26 on. The spine gates hash stage_health, and
every stage after the assembled gate waits on its verdict, so in a gated
build every node from stage 14 on re-keys. cgt_support_split's stage
contract moves with its texts. Regenerated:
- source_stages.json (copy_rule and notes, re-derived from the typed
projection);
- the coverage manifest's 25 source digests;
- the H2 fixture: the two declaration lines in fixture.json, and the
stage's contract hash in uk_spine.json.
Tests:
- K=1 counts equal ceil(w / 60) bitwise on random and edge weights: exact
multiples of 60, one ulp either side, tiny and large.
- K=3, 7 and 15 replicas count as their sources, including the K=7 exact
multiples the unsnapped rule overcounts. The snap's documented trade is
pinned.
- The clone-count refusals ({0, 2}, negative, missing, float, bool) give
counts.
- On K=3 and K=7 clone-first frames, the transform gives every clone its
source's copies, with weights nested within rtol 1e-12 and every copy at
most 60 / K.
- The packaged gate passes a K=3 stage receipt with w <= 60 < 3w, and fails
it when the count is ignored. Hand-built receipts pin the K=1 messages and
both scaled checks.
- The identity tool's K=1 receipt equals the one ceil(w / 60) gives, with no
new key. At K=3 it receipts green, and it reports a recompute that ignores
K.
Eight source mutations each fail a new test. The 51 touched engine-free
files pass (2,317 tests). The coverage manifest's check passes, the H2
fixture regenerates and passes its check, and H2 parity passes in the
engine slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three reviews of 914cae4 found no must. This commit takes their should findings and the cheap nits, and corrects three claims 914cae4 made. - Identity tool at K > 1: the E8 recompute counted each family exactly, on a weight rebuilt by re-summing the family's copies. At K > 1 that sum drifts by more ulps than the count's K-ulp snap absorbs. On the commit's own nesting scenario, the tool flagged the stage's correct count of the 7,620 family in every clone at K = 11 and K = 13. At K > 1 the stored count must now lie between the counts at the folded weight times 1 - 1e-9 and 1 + 1e-9, the tolerance the tool already gives family weights. At K = 1 the check stays exact, so K = 1 receipts and problems are unchanged. The tolerance cannot tell an exact multiple's n from n + 1. The count's unit tests pin the snap, and the flags and family weights still report a moved count. copies_recomputed stays the exact recount. - Gate tolerance: test_split_gives_every_geographic_clone_its_sources_copies now runs the packaged stage gate on the transform's receipt. At K = 7 it also asserts that 7 x the heaviest copy lands above 60, so the copy check's 1e-12 tolerance stays exercised. - Texts, correcting 914cae4: - Its changelog fragment and message said every clone of a household gets its source's count. That holds for a selected clone at exactly 1/K of its source's weight. The walk still accumulates each row's own weight, so the household at which a band's walk reaches its support mass, and households tied on wealth, can be selected in some clones and not others, lowest household id first. The fragment and the split's docstrings now say so, and a new test pins the boundary case. - Its Tests line omitted that the clone-first split scenario exhausts every pool, so its selection is the same at every K. - The spine gates re-key because they hash the country spec as well as stage_health; the fragment now names both. The fragment also says the snap adds a copy at K = 7 only for some exact multiples. The module docstring, the clone-split factor's comment and the split's refusal sentence name the incidence clone stage (cgt_incidence_clone). geographic_clone_count's docstring says a refusal gives a count or the column's dtype. Not changed here: the snap rule itself and a mention of it in copy_rule, which wait for a ruling on the snap; the stage notes' "the clone stage", which changes when the stage's declarations are next regenerated; and the tool's cross-check of K against the stage receipt, which lands in (b). What moves: no value, at any K. cgt_support.py changes only in docstrings and a comment, so its implementation hash moves again, and the stages that bind it re-key as in 914cae4. sources.yaml, source_stages.json, the coverage manifest and the H2 fixture are untouched; the fixture holds stage contract hashes, not code hashes. The tool's verdicts move only at K > 1, which no spine runs yet. Tests: the identity tool's nesting receipts are green at K = 7, 11, 13 and 15; a count moved off a non-multiple is reported at K = 1, 3 and 11; at K = 1 a moved count on an exact multiple is still reported by the count check. Six synthetic mutations each fail a new test: the gate without its tolerance, the tool exact at every K, the tolerance at K = 1, the tolerance ignoring K, a tolerance of 1e-1 and a strict walk. The touched and related engine-free files pass (243 and 275 tests). The coverage manifest's check and the source_stages projection check pass, and ruff is clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) The re-reviews of da5a350 and 9721448 found no must and no code should. This commit takes their four nits. - Bindings: the stochastic_assignment exemption of spi_support_channel now names the stage's transform, and the binding test checks that the transform names no drawing function, directly or through its module, as well as the drawing names the module imports. Before, a support-channel transform calling stage 14's employer pension draw, which spi_spine already imports, would have passed both binding tests. - The row-wise tool's --source-lineage-modulus help still described the spine's retired copy-id scheme. It now says spine inputs carry copy ids in the copy-id table's fixed fields. - Test comments on the CGT nesting scenario: the 9,000 household is the first candidate for the exact-total correction, not where it always lands. At 7 and 13 clones the shared fallback puts it on the last copy of the 7,620 family's last clone. At 13 clones an exact recount moves 12 of the 13 clones, not every one. - The identity tool's E8 docstring says copies_recomputed stays the exact recount, so at K > 1 it can exceed copies by one for each exact-multiple family while the count check passes. What moves: nothing. No runtime module changes, so no node key moves. Ruff is clean, the stochastic_assignment binding tests pass, and the re-review's mutant (the exempt transform naming the employer pension draw) now fails the binding test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
13 tasks
The land changelog said the two held rows "are measured on every evaluation", and 161c777's test was named for that. That phrase is the exclusions register's convention, and the register's entries keep it, but no build step reports a held row's estimate. As María ruled on 8 Nov, the changelog and the test now say what happens: the rows stay on the run's compiled register, and a tool that scores the un-excluded register, such as tools/evaluate_uk_incumbent_surface.py, measures them, as it does every held row. - The changelog fragment's sentence is reworded. - test_the_two_land_diagnostics_are_measured_but_not_fitted is renamed test_the_two_land_diagnostics_leave_the_solve_but_stay_on_the_compiled_register, and its docstring and the scoring comment say the same. What moves: nothing. Ruff is clean and the renamed test passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #931. This is the output-moving half of the geography-first UK spine, stacked on #1519, which changes no outputs.
The spine clones every FRS household K times at its start: K=15 on the dense line, K=1 on national. Each clone gets its own small area and its own later draws (wealth, consumption, SPI copies and so on). The K=1 national spine nests in the K=15 dense one wherever draws are keyed: it is the clone-0 slice.
This is a draft, opened early so the commits can be followed as they land. The base is
uk-931-geography-first-prep(#1519). CI runs only on PRs to main, so this PR gets none until #1519 merges.Commit groups
frs_brmafold, the SPI-copy area draw, G7, G8 inside the WAS chain, gates, dense re-basingGroup (0)
After the move, the dense build's per-clone engine blocks are no longer identical copies. policyengine-uk's
shareholding,corporate_land_valueandconsumption_shareholdingdivide by a weighted total over the simulated households, so a block would read its own total instead of the pool's (PolicyEngine/policyengine-uk#2896 asks pe-uk to fix the cause).microcosm.frame.adapters.engine_population, and policyengine-uk's declaration of its 19 whole-population formulas. The shared module does the pool-share correction, a recorder of formula runs, and a guard that judges them. Nothing imports it yet.exactnow certifies corrected blocks plus a clean guard (exact_basis), equal to one pooled engine to float32 rounding. Today's identical copies take a correction of 1 and give the same matrix and metrics, bitwise; national runs one block and records none of the new keys.household_clone_indexcarries more than one clone. National never batches, and today no spine frame carries the column, so every call still takes one simulation.engine_batchesonly when the final frame carries more than one clone. A follow-up commit reads that record from the frame, so a resumed spine reports what a fresh run does, and holds each batch's parameter tree to the one the stage-52 restatement reads.rank_route,engine_parity).Data changes in group (0): only the annual-income factor's float32 step, kept by ruling on 7 Nov. Everything else is inert until G1.
What moves without being an output:
measure_simulationand the two frame modules);The spine graph declaration (
uk_spine.json),source_stages.jsonand the coverage manifest are unchanged.For Max:
engine_populationmodule (pool shares, the recorder,judge,EngineBlock,materialize_in_batches,frame_key_sums);exactand whether the preflight, assembler or microcosm-data mirror should also readexact_basis;WAS and land
was_wealthandwas_lisa, match the donor's household income to the engine's HBAI net income before negative incomes are reset to zero, under one predictor name,hbai_household_net_income. The donor's is WASDVTotinc_bhcR8, net regular income before housing costs less council tax. The LCFS and ETB services stages already read the engine's concept (UK: move the LCFS stage to the Living Costs and Food Survey 2024-25 #1164). Before, both stages readhousehold_net_income, which also takes off stamp duty, the TV licence, and the business rates and corporate stamp duty thatshareholdingspreads over the simulated households.ons.land.corporate_land_value(E43B) andons.land.land_value(E44R) out of the solve as measured diagnostics, inuk/calibration_measure_exclusions.json, tracked to workstream 2 of Epic: UK land value data for land value tax results by constituency #1521.corporate_land_valuehands out the engine's own aggregate by each household's share of corporate-sector wealth, so a released file meets it at any weights.land_valuedoes not hold.ons.land.household_land_value(E44N) stays bound, and its binding note now says households alone.national_contract_registry.json).tools/evaluate_uk_incumbent_surface.py, measures them, as it does every held row.What moves:
ons.land.household_land_value, now theons_landfamily's only bound row, carries three times its loss weight before. No other row's weight changes. Weights move slightly through both effects, and a calibrate-only rerun attributes them.uk_target_fitandcalibration_diagnostics.json.Group (a)
Commits 1a, 1b and 2 are in, each with its review follow-up. Commits 3 to 7 are to do.
uk_runtime/copy_id_table.py, so a copy id depends only on its source id and its copy path, not on K.kin a field takesid + k × radixon all five id columns. The fields, least significant first:vintage(reserved, 10^8),geographic_support(10^9),spi_support_channel(10^11),high_value_support(10^12),cgt_support_split(10^13),cgt_incidence_clone(10^17) andcgt_residential_split(10^18).cgt_support, adds tests for the dense line, stage-16 nesting and the executor, and corrects texts.tools/verify_uk_identity_stability.py) still decodes the old rule, and 11 of its tests fail there. Don't run it from those two commits. They were pushed together with 1b.weight_tables(id_multiplier=)), though none does yet.tools/evaluate_uk_dataset_size.pyinfers the multiplier from the spine's household ids or the clone-0 household ids. It refuses, by count, a dataset the inferred multiplier does not fit, where it used to miscount silently.microcosm.build.stochastic_assignmentnow binds it, so an edit to the keyed draws re-keys the stages it moves. That is 32 more stages, 37 in all.ceil(K × w / 60), as ruled on 7 Nov. K is the frame's geographic clone count, 1 on every national spine today.ceil(w / 60)bit for bit.clone_count, and the stage-health gate and the identity tool scale their checks by K.What moves in (a) so far:
*_build_idcolumns carry the new copy ids, so copy rows join across versions by lineage (source id and copy path), not by build ids.clone_count.Rulings applied
exact.copy_rulesays so;Proof plan
🤖 Generated with Claude Code