Repository navigation
Reconcile regional Census family-type counts under d1932 - #1398
Merged
Merged
Conversation
An allocated-residual reference may now declare a discrepancy_policy, which admits a signed gap between a region's published total and its stated leaves. The policy is closed: kind signed_proportional_reconciliation, an integer max_abs, an exact decimal max_relative, combine "either", and a ruling and disclosure. With a policy declared, counts are read as exact integers and reconciled in Fractions: - the gap must pass either bound, and a nonzero total over empty leaves refuses; - a negative or zero gap, and a positive gap with no declared recipient, scale every leaf in proportion; a positive gap with one declared recipient goes wholly to it; - the reconciled leaves must be nonnegative and sum exactly to the total; - any growth ratio is exact and applied once, after reconciliation. Each target carries an exact receipt (version 2) with the leaves, total, gap, increments, reconciled counts, ratio and final targets as rationals. References without a policy take the legacy paths unchanged. NZ declares the d1932 policy in family_reconciliation.json (max_abs 3, max_relative 0.001) with a family_discrepancy profile dimension: central scales all three family types, and the positive-to-type-3 sensitivity sends a positive gap to one-parent families. Region 99, which has no ERP series, uses the national ERP pair. The 51 regional family-type targets (17 regions by 3 types) stay parked as requires_census_family_nucleus_standard until the G8-r3b multiplicity classifier exists. Calibration bindings add 17 household region indicators and three parked family-type columns carrying the 51 reference bindings. The generator, profiles, inventory, docs and the NZ golden (fingerprint and resource hashes) are regenerated to match. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_reference_documents read family_reconciliation.json before it validated the chronicle feed pin, so a package with a changed feed identity and no family declaration raised FileNotFoundError instead of refusing on the pin. The feed identity is now verified first, as the docstring says. Three tests assumed the family declaration's shape everywhere: the family reconciliation checks indexed record_set_id and aggregate_fact_key on every selector (precal's MBIE bond rents name no record set, and the tenure roll-forward rows name no fact key), the NZ resource list lacked family_reconciliation.json, and the ageing-scenario selections lacked the new family_discrepancy dimension, which every scenario resolves to central. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first review of this PR, cut off at the six-hour wall before its
verdict, left three of its mutants alive and noted one contract leak:
- relative_strict (`<` for `<=` on the relative bound): no test sat at
|R| / L == max_relative. test_signed_relative_bound_is_inclusive does,
at 4 / 4000 with max_abs=3 failing, for R = +4 and -4, and refuses R = 5.
- max_relative_via_float (the bound read through float): the new
test_signed_relative_bound_compares_exact_rationals puts a gap 1e-18 past
max_relative="0.1". Fraction("0.1") refuses it; Fraction(float("0.1")),
about 5.6e-18 larger, would not.
- exact_reader_accepts_fraction (a non-integer count read as a rational):
test_signed_count_reader_refuses_non_integer_counts reads 41.5, "41.5"
and "83/2" and expects the exact-count refusal.
- An explicit null positive_allocation_member_keys was read as an absent
key. Present now means exactly one recipient, so null refuses like any
other malformed list; the recipient-contract test gains a positive_null case.
Each new test fails under the mutant it targets (the reviewer's runtime
mutants, plus null_recipient_accepted for the null case).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1394 now carries #1382's head (c4ddcb4), whose fix moved the factor-currency check into prepare_transport_build in build/transport/cli.py. The merged cli.py is c4ddcb4's blob; no #1398 commit touches it. The NZ golden conflicted: both sides changed the fingerprint, the resource list and the country_package.json hash. It was regenerated from the merged tree with the test's own summary function. Keys only one side changed keep that side's values; the three both-changed keys are the merged tree's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The region-99 support note in the generator, calibration_bindings.json and the calibration-targets doc said G8-m2 (#1393) was not in this stack's base. The merge of #1394's head brought in #1382's fix4 head, which carries #1375's merge of main, so #1393 (4e31e42) is now an ancestor. Drop the dependency clause; the support statement itself is unchanged. calibration_bindings.json is regenerated with the generator, and the NZ golden follows its hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second review of this PR (Opus 5.5 at 2ffc63b, APPROVE_WITH_NITS) left two follow-ups. Both are fixed here. max_relative accepted values above 1, such as "5": a tolerance larger than the leaves it bounds. The policy now refuses a max_relative above 1. The tests accept 1, 1.0 and 0.999, and refuse 1.000000000000000001, 1.5 and 5 with the cap's own message. The disclosure said base-3-rounded type counts and totals "can differ by up to 3 families". The review showed that fails under random rounding to base 3, where a cell can move by 2: true counts (1, 1, 1) with total 3 can publish as (3, 3, 3) with total 3, a gap of 6. The repo doesn't say which rounding Stats NZ used. The disclosure now says the sums need not agree and that the observed regional gaps are at most 3 families. The change is in family_reconciliation.json, the test's copy and the doc. The generator regenerates the reference inventory (35 copies) and the profile transforms (204 copies) that repeat the text; with the text masked, both files are unchanged. The NZ golden follows their hashes, and the family test's pinned digest of the 34 receipts, which carry the text, is re-pinned. The relative bound's denominator stays the stated type leaves, which is the "0.1% of stated leaves" in the design that d1932 adopted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
22 of 25 tasks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…only check The spec-only package test flags any string shaped like `module.path:name` as a Python entrypoint. This PR's family-discrepancy data carries 159 fact keys of the form `ledger.aggregate_fact.v2:<hex>` in three places the checker did not know, so `test__given_country_specs__then_no_python_entrypoints_are_declared` failed at this branch's reviewed head too (stacked PRs got no CI): - signed partition operands, which add `discrepancy_policy` and may name one `positive_allocation_member_keys` recipient; - that positive recipient roster itself; - `raw_fact_keys` on each `regional_family_reconciliations` receipt. The partition admission now mirrors the compiler's own field rule in `ledger_targets._validate_allocation_reference`: `discrepancy_policy` opens the signed fields, and a positive roster needs it and holds exactly one partition member. Raw fact keys pass only as exact copies of the receipt's own `leaf_selectors` and `total_selector` pins; all 34 receipts satisfy that. Any other shape still falls through to the entrypoint scan. New example tests cover the admitted shapes and seven rejected ones; five mutants of the new rules each turn a test red. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main now holds #1375 as 6af97bf/846b48a9, so this branch had two merge bases with main (fddbc70 and 840d933). Git merges them cleanly, but GitHub marked the PR conflicting and ran no CI. This merge gives one merge base; its tree equals `git merge-tree --write-tree origin/main 8853469`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 29, 2026
#1395 (money normalization) and this branch each add one resource to nz/country_package.json, so the NZ golden spec summary conflicted. Git merged country_package.json and the two shared test files cleanly: the package lists both resources in sorted order, and each test file's diff against main equals this branch's own change. tests/golden/nz_country_spec.json is regenerated from the merged spec with the golden test's own _loaded_spec_summary and canonical_json_bytes. Against main it changes exactly the resource hashes this branch changes (calibration_bindings, family_reconciliation, reference_inventory, reference_profiles, target_references), the resource list, the fingerprint, and country_package.json's hash, the one merged file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merging main (#1395 and #1397) into this branch moved two pinned digests. Each test passes at main and at this branch's previous head 0a8568b, and fails only on the combination, because each pins output over NZ data both lines touched. test_prepare_matches_pinned_legacy_output_digest hashes prepare_calibration_columns over nz/calibration_bindings.json, which this branch extends; #1397 pinned it from the pre-refactor implementation at 4f333f2. On the merged bindings the pre-refactor implementation, main's and this head's all give 1b22f4e1, and on main's bindings the pre-refactor implementation still gives the old pin dca0e085, so the refactor stays byte-equivalent. test_nz_factor_receipts_are_byte_identical_after_additive_factors pins the v2 receipts before #1395's donor factors were added, over target_references.json and reference_profiles.json, which this branch changes. Its three receipts at 0a8568b (this branch's data, no donor factors) equal the merged tree's (b0f8648d, 8989fa75, e246f615), so the donor factors still leave them byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Merged by the NZ hub (session local_e407a527) at abc0243 with --merge. Gates: |
MaxGhenis
added a commit
that referenced
this pull request
Oct 29, 2026
Only tests/golden/nz_country_spec.json conflicted: #1398 and this PR both move the NZ spec fingerprint. The golden was regenerated from the merged spec with the golden test's own _loaded_spec_summary and canonical_json_bytes. Against main it moves only the fingerprint and money_normalization.json's hash (5e817f0b…). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 29, 2026
Only tests/golden/nz_country_spec.json conflicted: #1398 and this PR both move the NZ spec fingerprint. The NZ hub regenerated the golden from the merged spec with the golden test's own _loaded_spec_summary and canonical_json_bytes. Against main it moves only the fingerprint and money_normalization.json's hash (5e817f0b…). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 30, 2026
The #1398 review found that two tests pinned digests of outputs that NZ resources feed, so any change to those inputs forced a re-pin unrelated to what the tests guard. They now compare against oracles instead. - Calibration columns: a frozen copy of 4f333f2's prepare_calibration_columns serves as the oracle. Tests compare the live code with it on the NZ fixture, on generated NZ-domain tables (valid and defective, empty, boxed NumPy identities and relabelled indexes) and on generated interval inputs. Each call gets its own copy of every argument, and the comparison includes what each call left in its copies. On today's bindings the frozen code reproduces the retired pin, 1b22f4e1. - NZ factor receipts: a receipt must match the receipt resolved from only its selected factor, for each aging profile, and must not change when the factor purpose vocabulary drops money_normalization_input. A Hypothesis property drops and reorders the unread factors. - The 1.654 scan reads JSON values. It finds the exact rate as a JSON number, a numerator/denominator object, or a whole numeric token in text (signs and exponents included), and ignores other numbers such as 1.6541 or 1.654e1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
G8-r3a for Microcosm NZ v0: signed reconciliation of the regional Census family-type counts under Max's ruling d1932. Targets
mainand lands after #1394; the commit message has the full description.What changes
ledger_targets.py: an allocated-residual reference may declare a closeddiscrepancy_policy. With it, counts are exact integers reconciled in Fractions, within an absolute or relative bound, with an exact receipt. Without it, every path is the legacy one; each new branch is gated on"discrepancy_policy" in operands[2].family_reconciliation.json(d1932: max_abs 3, max_relative 0.001), afamily_discrepancyprofile dimension (central, and the positive-to-type-3 sensitivity), 17 household region indicators, and three parked family-type columns carrying the 51 reference bindings. All 51 regional family-type targets stay parked (requires_census_family_nucleus_standard) until G8-r3b's multiplicity classifier. Nothing enters the fit.Invariants (for every input of the stated kind)
test_signed_exact_closure_nonnegativity_and_preserved_shares, Hypothesis).test_signed_compiled_erp_bridge_scales_exact_closure_once, Hypothesis).test_legacy_policy_absent_receipt_is_byte_identical).Notes
ledger_targets.pyis in the target compilers' hash modules, and UK's native target kernels hash it too, so cache keys change for every country. Under the UK hub's Budget-window rule this does not merge before 2026-10-29 12:00Z (8am ET).holdout_references.jsonis unchanged) and no stage column is rewritten before calibration.Hub validation (10/25–26, serial
-n 0)test_nz_spec_package.py1,201,test_ledger_targets.py451,test_ledger_allocated_residual.py129,test_transport_composed_graph.py166 andtest_nz_family_reconciliation.py16. A plugin-inert control passes.clip_negative,double_erp_bridge,drop_region_99,regional_growth_for_99,bounds_require_both,sensitivity_on_negative,policy_default_accepts_negative,float_arithmetic,totals_as_fit_rowsandrows_activatedeach fail their test.household_total_as_family_totalis refused at fixture setup by the shared-universe check (ledger_targets.py, "total and partition must share one source count universe"), the guard it targets.test_ledger_allocated_residual.py136,test_nz_family_reconciliation.py16,test_ledger_targets.py451 andtest_nz_target_references.py50 pass. Ruff, format,git diff --checkand the generator's--checkpass.relative_strict,max_relative_via_float,exact_reader_accepts_fraction) and a newnull_recipient_acceptedeach fail the new tests.test_country_spec.py136,test_nz_spec_package.py1,201,test_ledger_allocated_residual.py136,test_nz_family_reconciliation.py16,test_nz_target_references.py50 andtest_transport_composed_graph.py172 pass.--checkpasses, andtest_ledger_allocated_residual.py142,test_nz_family_reconciliation.py16,test_nz_target_references.py50,test_reference_profiles.py122,test_country_spec.py136,test_nz_spec_package.py1,201 andtest_ledger_targets.py451 pass. A mutant that drops the cap fails the three above-one refusal cases.Review
max_relativeabove 1 now refuses.float();ledger_value_ratiois written as "109/100", and nothing reads it;max_relative, and NZ's 138 declarations are all 0.001.test_ledger_allocated_residual.py142 andtest_nz_family_reconciliation.py16 pass. Four mutants (no cap, a strict cap, a float cap, the old disclosure) each fail.Status
be27d7e merges Add exact income-band intervals and boundary diagnostics #1394's head, which carries Add feed-derived NZ reference ageing factors (G8-r2a) #1382's fix4 (Belgium OK'd it 10/26). The merged
transport/cli.pyis c4ddcb4's blob and the only transport file the merge changes. The NZ golden conflicted and was regenerated from the merged tree; per-key attribution shows no mismatches.b08107d updates the region-99 support text now that G8-m2 is an ancestor. The generator regenerates
calibration_bindings.json, and the golden follows its hash.9c7c4a2 fixes round 2's two follow-ups (see Review). It touches no transport file. The generator regenerates the reference inventory and profiles, which repeat the disclosure; with the text masked they are unchanged. The golden and the pinned receipt digest follow.
Round 3 approved 9c7c4a2.
Landing (2026-10-29). The NZ hub retargeted this PR to
mainon 2026-10-29, so it ran CI for the first time; stacked PRs never ran it.main(fddbc70) through update-branch; 0df4561 and 215c71c merge Add feed-derived NZ reference ageing factors (G8-r2a) #1382's CI import fix. Delta review of that fix (GPT-6.1 Sol, job 20261029-083505): APPROVE, no findings. Merging the fixed head intomaindiffers from merging the previously reviewed head only in the fixture import lines oftest_reference_factors.pyandtest_reference_profiles.py.test_spec_only_country_packages.py::test__given_country_specs__then_no_python_entrypoints_are_declaredfails at the reviewed head 9c7c4a2 too. That test flags anymodule.path:namestring as a Python entrypoint, and this PR's data carries 159ledger.aggregate_fact.v2:<hex>fact keys in three shapes it didn't admit: signed partition operands, which adddiscrepancy_policyand may name onepositive_allocation_member_keysrecipient; that positive roster; andraw_fact_keyson each regional reconciliation receipt. The hub's r3a validation plan didn't include that file, and the stack had no CI.ledger_targets._validate_allocation_reference:discrepancy_policyopens the signed fields, and a positive roster needs it and names exactly one partition member. It checks field sets and key shapes, not the compiler's full validation; the compiler still refuses a bad policy or recipient set, and the scan still reads every selector and policy. Raw fact keys pass only as exact copies of the receipt's ownleaf_selectorsandtotal_selectorpins, which all 34 receipts are. Any other shape still falls through to the entrypoint scan. Seven new rejection cases cover it, and five mutants of the new rules each fail a test. No source or data file changes.test_uk_target_compile_modules.py::test_every_direct_import_of_a_listed_module_is_listed_or_excluded: Add feed-derived NZ reference ageing factors (G8-r2a) #1382's 1d5fff4 madeledger_targets, a UK target compile module, importLexicalFloatfromledger_artifact. Add feed-derived NZ reference ageing factors (G8-r2a) #1382's b36c702 first addedledger_artifacttoEXCLUDED_DIRECT_IMPORTS; 8853469 merges it through Add exact income-band intervals and boundary diagnostics #1394.mainafter Allocate Census residuals through shared reference profiles #1375's landing. It changes no file, but Allocate Census residuals through shared reference profiles #1375 landed as 6af97bf, so this branch had two merge bases withmain; GitHub marked the PR conflicting and ran no CI until this merge gave it one.ledger_artifactinstead: both UK target kernels decode their facts through it, so an edit to its decoder could move a UK target value without moving either kernel's implementation hash. Add feed-derived NZ reference ageing factors (G8-r2a) #1382's 7cd1d26 lists it inUK_TARGET_COMPILE_MODULE_NAMES(see Add feed-derived NZ reference ageing factors (G8-r2a) #1382); 0a8568b merges it through Add exact income-band intervals and boundary diagnostics #1394.gh pr checksexit 0; run 40897375644): engine-free, engine-uk and engine-us on Python 3.13 and 3.14, integration-uk, lint, select-countries and wheels.dump_be.pyand its three BE suites (test_spec_engine_country_bundles.py,test_country_spec.py,test_spec_only_country_packages.py) at 0a8568b, alone and merged with Belgium's open Rewrite the Belgian country package for donor transport #1369 (dc157a7). The BE dump matchesmain's byte for byte; with Rewrite the Belgian country package for donor transport #1369 merged in, it matchesmainplus Rewrite the Belgian country package for donor transport #1369. The BE golden comparison holds in both, and all three suites pass with Rewrite the Belgian country package for donor transport #1369 merged in.Sibling merge (2026-10-29, after Normalize donor money with exact declared factors #1395 and Add receipt contract v2 and selection-aware reference preflight #1397 landed). This PR and Normalize donor money with exact declared factors #1395 both branched from Add exact income-band intervals and boundary diagnostics #1394, so neither line's CI ever saw the other.
main(41f9a8c). Both lines add a resource tonz/country_package.json, sotests/golden/nz_country_spec.jsonconflicted. Git merged the package (it now lists bothfamily_reconciliation.jsonandmoney_normalization.json),test_reference_profiles.pyandtest_spec_engine_country_bundles.pycleanly; each test file's diff againstmainequals this PR's own change. The hub regenerated the golden from the merged spec with the golden test's own_loaded_spec_summaryandcanonical_json_bytes. Againstmain's golden it changes this PR's five resource hashes, the package's own hash, the resource list and the fingerprint, and nothing else.mainand at 0a8568b alone, because each pins output over NZ data both lines touch. abc0243 re-pins both, and only both:test_prepare_matches_pinned_legacy_output_digesthashesprepare_calibration_columnsovernz/calibration_bindings.json, which this PR extends. On the merged bindings, Add receipt contract v2 and selection-aware reference preflight #1397's pre-refactor implementation (4f333f2),main's code and this head's code all give 1b22f4e1. Onmain's bindings the pre-refactor implementation still gives the old pin, dca0e085. So the refactor stays byte-equivalent.test_nz_factor_receipts_are_byte_identical_after_additive_factorspins the v2 receipts from before Normalize donor money with exact declared factors #1395's donor factors, overtarget_references.jsonandreference_profiles.json, which this PR changes. The three receipts at 0a8568b (this PR's data, no donor factors) equal the merged tree's (b0f8648d, 8989fa75, e246f615), so the donor factors still leave them byte-identical.test_spec_engine_reemission.py::test_normative_mutation_changes_hash_and_names_legacy_payload_field, came from the hub's own environment: its QRF worker venv declares Python 3.14.4 and the hub runs 3.14.7. CI passed that test at every earlier head.jobs/review-microcosm-1398-merge): APPROVE_WITH_NITS, no blocking findings; Needs Max: no. The merge. Everymain-only file equalsmain's blob and every file only this PR touches equals 0a8568b's, andgit merge-filereproduces the three overlap files byte for byte. The golden. Its own regeneration gives the same bytes. The pins. It found the two stale pins on its own (GitHub CI on 30edef9 also failed first on the receipts test), and it checked abc0243's re-pins against 4f333f2's own code and against 0a8568b's receipts. Tests. It ran 39 test files in CI mode, plus, with the pinned feed, the six NZ files that skip without an artifact, and it confirmed the reemission failure comes from the hub's environment. Nits for a follow-up. The two re-pinned tests still hash live NZ data, so the next NZ data change will move them again, and their inputs could be frozen asreference_selection_legacy.py.txtis; and a substring check intest_nz_money_normalizationis fragile.gh pr checksexit 0; run 40922086707): engine-free, engine-uk and engine-us on Python 3.13 and 3.14, integration-uk, lint, select-countries and wheels.mainat 41f9a8c, alone and merged with Rewrite the Belgian country package for donor transport #1369 (dc157a7): the BE dump matchesmain's byte for byte, andmainplus Rewrite the Belgian country package for donor transport #1369's with Rewrite the Belgian country package for donor transport #1369 merged in; the golden comparison holds in both; and all three BE suites pass with Rewrite the Belgian country package for donor transport #1369 merged in. abc0243 changes only two NZ test files, which neitherdump_be.pynor the BE suites read.🤖 Generated with Claude Code