diff --git a/docs/agent-guide.md b/docs/agent-guide.md index 0860d1e94a..65c0018adc 100644 --- a/docs/agent-guide.md +++ b/docs/agent-guide.md @@ -392,9 +392,13 @@ local `--spec-dir`. Extensions can wrap a factory in `TransportExtension` to declare additional sources and ordered checkpoints. Source names must be new; skeleton predecessor sets must stay identical except for explicit `package_inputs` artifact edges -from extension nodes to a terminal `transport.package@1` receipt. Other skeleton -nodes cannot gain predecessors. Checkpoints run after calibration and before export, -and must not depend on the exported dataset. `transport_rules_node` supplies +from extension nodes to a terminal `transport.package@1` receipt and the +battery evidence edges described below. Checkpoints run after calibration and before export, +and must not depend on the exported dataset. Explicit `gate_inputs` artifact +edges may attach required extension diagnostics to a skeleton +`gates.battery@1` node; its existing report edge governs export permission. +Only package receipts and battery nodes may gain these declared artifact +predecessors. `transport_rules_node` supplies the prepared engine reference and period from the binding, so extensions do not repeat period values. An optional `entitlement_graph.json` legacy resource installs the entitlement diff --git a/docs/concept-coverage/axiom-be.md b/docs/concept-coverage/axiom-be.md index ecfe3666fb..2b95f7e908 100644 --- a/docs/concept-coverage/axiom-be.md +++ b/docs/concept-coverage/axiom-be.md @@ -1,6 +1,6 @@ # Concept coverage: axiom:be -Engine version `b105e2b3a3086ddd2de447d58a9b951346870dd1`; concept schema `d824f5e97d58`. Generated by `tools/refresh_concept_coverage.py`; do not edit by hand. +Engine version `b105e2b3a3086ddd2de447d58a9b951346870dd1`; concept schema `5a55669d0447`. Generated by `tools/refresh_concept_coverage.py`; do not edit by hand. - Engine inputs: 1867 - Fed by a concept: 105 @@ -46,6 +46,7 @@ Engine version `b105e2b3a3086ddd2de447d58a9b951346870dd1`; concept schema `d824f | `fact:person.enrolled_full_time` | rulespec-be has no input that tests full-time enrollment. The full-time wording in non-test rule text concerns work regimes (parental leave, the work bonus, unemployment admission, the Flemish jobbonus). The one exception is a Flemish school-allowance parameter citing 'voltijds secundair onderwijs', and that module selects it by education level alone. The student statuses the tree does take are supplied legal statuses, and the rules that consume them contain no full-time test: belgium_self_employed_is_student (RD 38 art. 12bis) and the Child-entity family-benefit training and 'qualifying' flags. | | `fact:person.take_up_seed` | rulespec-be computes legal entitlements and has no take-up or participation input. Its only claim-type inputs are eligibility conditions or elections, not decisions to take up the program the rule computes: belgium_social_integration_social_benefit_rights_asserted (a RIS art. 3-4 condition), brussels_apa_required_other_benefit_rights_claimed (an APA eligibility conjunct), and belgium_pit_article_1458_pension_savings_reduction_claimed (a non-cumulation election against the art. 145/7 reduction that also needs pension-savings payments). | | `fact:household.mortgage_principal` | rulespec-be at b105e2b3 has no input for mortgage capital repayments. The only housing-loan inputs are the interest deductions belgium_social_integration_eligible_annual_mortgage_interest_on_property (RIS art. 25 §4) and brussels_apa_eligible_annual_mortgage_interest (APA art. 15), which this mapping feeds from mortgage_interest. No non-test rule text mentions capital repayment, and the PIT reductions module takes no loan input. | +| `fact:household.source_household_id` | Original donor household lineage is build provenance and weight diagnostic support, not a rulespec-be legal input. | ## Inputs no concept covers diff --git a/docs/concept-coverage/axiom-nz.md b/docs/concept-coverage/axiom-nz.md index 9e42d0447b..1b213f1607 100644 --- a/docs/concept-coverage/axiom-nz.md +++ b/docs/concept-coverage/axiom-nz.md @@ -1,6 +1,6 @@ # Concept coverage: axiom:nz -Engine version `6fe181fc4c65763150fe97501ef3443774e64869`; concept schema `d824f5e97d58`. Generated by `tools/refresh_concept_coverage.py`; do not edit by hand. +Engine version `6fe181fc4c65763150fe97501ef3443774e64869`; concept schema `5a55669d0447`. Generated by `tools/refresh_concept_coverage.py`; do not edit by hand. - Engine inputs: 668 - Fed by a concept: 89 @@ -46,6 +46,7 @@ Engine version `6fe181fc4c65763150fe97501ef3443774e64869`; concept schema `d824f | `fact:person.weeks_worked` | rulespec-nz at 6fe181f has no input the concept determines: the closest, in_work_tax_credit_weekly_periods (Family, income_tax/family_scheme/tax_credits.yaml, s MD 10(2)), counts weeks in which the person or their partner meets the earner requirement, which needs the union of two partners' work weeks that no group rule gives (reference_member drops a working partner; sum_over_members double-counts and adds children's weeks), and minimum_family_tax_credit_weekly_periods and minimum_family_full_time_earner_weeks count only weeks as a full-time earner. | | `fact:person.educational_attainment` | rulespec-nz at 6fe181f has no input for completed education: no rule or input name mentions a qualification, degree, diploma or attainment level; the education inputs are current enrollment or school attendance (this mapping binds some of them to education_enrollment and enrolled_full_time); and young_person_is_study_ready is what youth_young_parent_payment/core.yaml calls an agency-adjudicated composite input (s 50). | | `fact:household.property_tax` | The only rates input, rr_rates_payable_before_rebate (Person) in statutes/rates_rebates/rates_rebate.yaml, cannot be bound at 6fe181f, because the dense engine could not compile that module and nz.json lists it only under cli_walk_inputs, which the adapter's surface loader does not read; once the module compiles, allocate_to_reference_person of property_tax is the candidate (approximate, because the ratepayer need not be the reference person). | +| `fact:household.source_household_id` | Original donor household lineage is build provenance and weight diagnostic support, not a rulespec-nz legal input. | ## Inputs no concept covers diff --git a/docs/concept-coverage/policyengine-uk.md b/docs/concept-coverage/policyengine-uk.md index a5fb8216e5..d8444ef67a 100644 --- a/docs/concept-coverage/policyengine-uk.md +++ b/docs/concept-coverage/policyengine-uk.md @@ -1,6 +1,6 @@ # Concept coverage: policyengine-uk -Engine version `2.122.2`; concept schema `d824f5e97d58`. Generated by `tools/refresh_concept_coverage.py`; do not edit by hand. +Engine version `2.122.2`; concept schema `5a55669d0447`. Generated by `tools/refresh_concept_coverage.py`; do not edit by hand. - Engine inputs: 285 - Fed by a concept: 40 @@ -46,6 +46,7 @@ Engine version `2.122.2`; concept schema `d824f5e97d58`. Generated by `tools/ref | `fact:household.reference_person_id` | is_household_head is formula-owned (the oldest member). The UK build overrides it with the FRS household reference person (frs_spine.py), an override the loader accepts, but it is not an input, so the mapping does not target it. Every housing input is household-level, so nothing is allocated to the reference person. | | `fact:person.liquid_financial_assets` | policyengine-uk holds financial wealth on the household: savings ('Household liquid savings'), corporate_wealth ('Wealth held in corporations directly or through investment funds'), gross_financial_wealth and net_financial_wealth are household inputs, and no transform puts a person amount on a household input. Its person-level GBP stock inputs are student_loan_balance, a debt; car_list_price, the price vehicle excise duty's expensive-car supplement tests; and lifetime_isa_balance ('Current value of the person's Lifetime ISAs, cash and stocks and shares together'), one savings wrapper, which the engine documents as already inside gross_financial_wealth and net_financial_wealth. savings and corporate_wealth are two of Universal Credit's capital sources (gov.dwp.universal_credit.means_test.capital.sources), and lifetime_isa_countable_capital, the Lifetime ISA at its surrender value, is its one person-level source (capital.person_sources). uc_reported_capital (benefit unit; 'Claimant-level capital for Universal Credit when household-level asset data cannot be attributed across multiple benefit units', default -1) is Universal Credit capital: when it is zero or more, uc_assessable_capital takes it in place of both the household-capital proxy it otherwise sums from the household sources, net of capital other benefit units report, and apportions by claimant and partner count, and the person-level sources of the unit's claimant and partner, so feeding it would be a modelling choice for a UK build, not a mapping of this concept. | | `fact:person.public_pension_income` | The State Pension is computed from state_pension_reported, which is formula-owned: the loader accepts it as an override, but it is not an input. | +| `fact:household.source_household_id` | Original donor household lineage is build provenance and weight diagnostic support, not a policyengine-uk rules input. | ## Inputs no concept covers diff --git a/docs/concept-coverage/policyengine-us.md b/docs/concept-coverage/policyengine-us.md index 9c029adf6f..025a06e0d1 100644 --- a/docs/concept-coverage/policyengine-us.md +++ b/docs/concept-coverage/policyengine-us.md @@ -1,6 +1,6 @@ # Concept coverage: policyengine-us -Engine version `2.2.1`; concept schema `d824f5e97d58`. Generated by `tools/refresh_concept_coverage.py`; do not edit by hand. +Engine version `2.2.1`; concept schema `5a55669d0447`. Generated by `tools/refresh_concept_coverage.py`; do not edit by hand. - Engine inputs: 926 - Fed by a concept: 55 @@ -46,6 +46,7 @@ Engine version `2.2.1`; concept schema `d824f5e97d58`. Generated by `tools/refre | Concept | Reason | |---|---| | `fact:person.liquid_financial_assets` | policyengine-us splits this stock across three person inputs, which the populace-us donor H5 also carries as person columns. Their engine documentation: bank_account_assets ('Value of checking, savings, and money market accounts. Imputed from SIPP TVAL_BANK.'), stock_assets ('Value of stocks and mutual funds. Imputed from SIPP TVAL_STMF.') and bond_assets ('Value of bonds and government securities. Imputed from SIPP TVAL_BOND.'). All three uprate with CPI-U (gov.bls.cpi.cpi_u), the concept's consumer-price index family. The documentation names no term deposits. In the donor (microcosm.build.us_runtime.scf_wealth and sipp_financial_assets), each household takes its whole three-column vector from either the SCF or the SIPP, about half each, and the vector sits on the member with the lowest CPS line number (the householder is line 1), with every other member at zero. SCF-sourced households carry a draw of an SCF family-unit total, whose bank column comes from SCF liq (checking, savings, money-market and call accounts) and so leaves out term deposits. SIPP-sourced households carry a draw from a model trained on individual SIPP holdings (December person records, each with that person's own TVAL_BANK, TVAL_STMF and TVAL_BOND), whose bank leaf includes certificates of deposit: its quality screen reads the SIPP CD value allocation flags (AJSCDVAL, AJOCDVAL, AOCDVAL). Neither half holds each owner's share. spm_unit_assets (SPM unit; read by the federal CCDF asset test is_ccdf_asset_eligible, the Massachusetts, Indiana, Iowa and Texas child-care asset tests ma_ccfa_asset_eligible, in_ccdf_asset_eligible, ia_cca_asset_eligible and tx_ccs_asset_eligible, and Illinois's HBI resource test il_hbi_resource_eligible) is a unit stock the engine defines only by its label 'SPM unit assets', so feeding it would be a modelling choice. No transform here splits one amount across three inputs: a share pair splits it in two, and a fraction is never decoded. So decode cannot recover the concept from these inputs, and encoding it waits for a reviewed three-way split. | +| `fact:household.source_household_id` | Original donor household lineage is build provenance and weight diagnostic support, not a policyengine-us rules input. | ## Inputs no concept covers diff --git a/docs/concept-schema-transport-adr.md b/docs/concept-schema-transport-adr.md index c3480a6292..72bd5bc1a5 100644 --- a/docs/concept-schema-transport-adr.md +++ b/docs/concept-schema-transport-adr.md @@ -22,6 +22,11 @@ on built benefit units; the other three mappings list it as unmapped with a reason. A scale, sum, scaled sum, share or fraction now computes from annual flows only (invariant 11), so none of them can rescale the stock, add it to a year's income, or split it. +**Amended 2026-10-10:** schema version 3 adds original +`household.source_household_id` lineage for donor support diagnostics. The +reader scopes explicit raw household identifiers by source year and carries +the same key across support clones and their benefit units. Rules mappings +leave this provenance concept unmapped. ## Context @@ -42,7 +47,7 @@ working through the same layer. ### 1. Primitive facts are the neutral layer `microcosm.frame.concepts` declares the record-level facts a household survey -observes and that keep their meaning across countries. There are 29, on two +observes and that keep their meaning across countries. There are 30, on two entities. Engine group entities (tax units, benefit units, SPM units, families) are engine constructs built from relationships. Today the US unit operator builds them from raw CPS roster columns, and the UK adapter requires @@ -52,6 +57,7 @@ not yet built from them. | Topic | Concepts | |---|---| +| Source lineage | household `source_household_id` | | Demography | `age`, `sex`, `legal_marital_status` | | Relationships | `partner_person_id`, `parent_1_person_id`, `parent_2_person_id`, household `reference_person_id` | | Labour income | `employment_income`, `nonfarm_self_employment_income`, `farm_self_employment_income` | @@ -66,7 +72,7 @@ not yet built from them. Each concept declares: - **Unit.** Years, base currency, hours per week, weeks, category, boolean, - person id or unit interval. + person id, source id or unit interval. - **Period semantics.** The kernel's `year`, `month` or `point`, plus what the value measures over it: an annual flow, a state at the reference date, a usual rate over the weeks worked, or persistent state. diff --git a/packages/microcosm-build/src/microcosm/build/country_spec.py b/packages/microcosm-build/src/microcosm/build/country_spec.py index 1be91fdb52..d4a2d861b2 100644 --- a/packages/microcosm-build/src/microcosm/build/country_spec.py +++ b/packages/microcosm-build/src/microcosm/build/country_spec.py @@ -115,6 +115,8 @@ "aggregate_admin", "area_support", "calibration_reference_coverage", + "cell_support", + "cell_support_target", "cgt_projection_entrants", "column_implication", "degenerate_release_surface", diff --git a/packages/microcosm-build/src/microcosm/build/nz/benefit_unit_rule.json b/packages/microcosm-build/src/microcosm/build/nz/benefit_unit_rule.json index 0fe9753e62..bbd0c4affe 100644 --- a/packages/microcosm-build/src/microcosm/build/nz/benefit_unit_rule.json +++ b/packages/microcosm-build/src/microcosm/build/nz/benefit_unit_rule.json @@ -21,6 +21,11 @@ "household_reference_person": "reference_person_id", "citation": "packages/microcosm-frame/src/microcosm/frame/concepts.py:880-919" }, + "source_household_lineage": { + "concept": "fact:household.source_household_id", + "unit_column": "family.family_source_household_id", + "rule": "Each unit inherits its household's exact original-source key. The donor reader scopes raw source_household_id by source_year, merging support clones across channels; no row-order or support-row-id fallback is permitted." + }, "composition": [ { "member": "adult", diff --git a/packages/microcosm-build/src/microcosm/build/nz/export_contract.json b/packages/microcosm-build/src/microcosm/build/nz/export_contract.json index d241df1d02..86c164a39b 100644 --- a/packages/microcosm-build/src/microcosm/build/nz/export_contract.json +++ b/packages/microcosm-build/src/microcosm/build/nz/export_contract.json @@ -58,6 +58,8 @@ "family_weight" ], "optional": [ + "source_household_id", + "family_source_household_id", "age", "sex", "region_code", diff --git a/packages/microcosm-build/src/microcosm/build/nz/gates.json b/packages/microcosm-build/src/microcosm/build/nz/gates.json index f5388c5385..423410d0dd 100644 --- a/packages/microcosm-build/src/microcosm/build/nz/gates.json +++ b/packages/microcosm-build/src/microcosm/build/nz/gates.json @@ -1,7 +1,7 @@ { "version": 2, "country": "nz", - "policy": "New Zealand is greenfield and has no incumbent population, so parity, export_surface, and target_surface are absent. Public NZ facts and Treasury oracles govern the build. Gates whose thresholds are D1 method-card items (MC15 fit tolerances, effective sample size, weight ratio) stay not_applicable with the reason 'awaiting D1'; macro_realism waits on packaged evidence and reviewed bands. Donor support and release input coverage remain applicable and fail closed while their producers and checks are pending. Thresholds are never borrowed from Belgium, the US, the UK or Armenia and never set from a candidate. Active reference, profile, support, release-input-coverage, weights, nonzero, formula-ownership, and nonnegative gates fail closed.", + "policy": "New Zealand is greenfield and has no incumbent population, so parity, export_surface, and target_surface are absent. Public NZ facts and Treasury oracles govern the build. MC15 weight quality follows the 2026-10-10 D1 ruling d1281: original-source household ESS at least 4000 and max/positive-median at most 5 block release; scenario-complete cell diagnostics are mandatory and failing cells are suppressed. Global ESS 16000 and the tighter cell support screen are tracked, never blocking. Fit and aggregate tolerances still await their Chronicle inventories and reviewed evidence; macro_realism waits on packaged evidence and reviewed bands. Donor support and release input coverage remain applicable and fail closed while their producers and checks are pending. Thresholds are never borrowed from another country or set from a candidate.", "phases": [ "terminal" ], @@ -137,16 +137,185 @@ "gate": "weight_ess", "phase": "terminal", "criticality": "release_blocking", - "not_applicable": "awaiting D1: the ex-ante NZ effective-sample-size floor (method card MC15) is not yet ruled.", - "notes": "Activate before a candidate release; never set the threshold from the candidate to make it pass." + "notes": "MC15 budget (b): Kish ESS on calibrated household weights summed by original source household, clones merged. The floor is an absolute count, independent of row count.", + "parameters": { + "aggregation_column": "source_household_id", + "required_weight_entity": "household", + "required_weight_kind": "calibrated", + "minimum_effective_sample_size": 4000 + }, + "evidence_absent_blocks": true }, { "id": "weight_max_to_median_ratio", "gate": "weight_ratio", "phase": "terminal", "criticality": "release_blocking", - "not_applicable": "awaiting D1: the ex-ante NZ max-to-median weight-ratio threshold (method card MC15) is not yet ruled.", - "notes": "Activate before a candidate release; do not copy an empirical US, UK, Belgium, or Armenia fence." + "notes": "MC15 budget (b): max over positive median of original-source-aggregated calibrated household weights is at most 5, inclusive.", + "parameters": { + "aggregation_column": "source_household_id", + "required_weight_entity": "household", + "required_weight_kind": "calibrated", + "maximum_max_to_median_ratio": 5 + }, + "evidence_absent_blocks": true + }, + { + "id": "reported_cell_support", + "gate": "cell_support", + "phase": "terminal", + "criticality": "release_blocking", + "parameters": { + "aggregation_column": "source_household_id", + "required_weight_entity": "household", + "required_weight_kind": "calibrated", + "required_scenarios": [ + "S0", + "S1", + "S2", + "S3", + "S4", + "S5", + "V1", + "V2", + "V3" + ], + "required_cells": [ + "income_decile_by_family_type/1/single", + "income_decile_by_family_type/1/couple", + "income_decile_by_family_type/1/sole_parent", + "income_decile_by_family_type/1/couple_with_children", + "income_decile_by_family_type/2/single", + "income_decile_by_family_type/2/couple", + "income_decile_by_family_type/2/sole_parent", + "income_decile_by_family_type/2/couple_with_children", + "income_decile_by_family_type/3/single", + "income_decile_by_family_type/3/couple", + "income_decile_by_family_type/3/sole_parent", + "income_decile_by_family_type/3/couple_with_children", + "income_decile_by_family_type/4/single", + "income_decile_by_family_type/4/couple", + "income_decile_by_family_type/4/sole_parent", + "income_decile_by_family_type/4/couple_with_children", + "income_decile_by_family_type/5/single", + "income_decile_by_family_type/5/couple", + "income_decile_by_family_type/5/sole_parent", + "income_decile_by_family_type/5/couple_with_children", + "income_decile_by_family_type/6/single", + "income_decile_by_family_type/6/couple", + "income_decile_by_family_type/6/sole_parent", + "income_decile_by_family_type/6/couple_with_children", + "income_decile_by_family_type/7/single", + "income_decile_by_family_type/7/couple", + "income_decile_by_family_type/7/sole_parent", + "income_decile_by_family_type/7/couple_with_children", + "income_decile_by_family_type/8/single", + "income_decile_by_family_type/8/couple", + "income_decile_by_family_type/8/sole_parent", + "income_decile_by_family_type/8/couple_with_children", + "income_decile_by_family_type/9/single", + "income_decile_by_family_type/9/couple", + "income_decile_by_family_type/9/sole_parent", + "income_decile_by_family_type/9/couple_with_children", + "income_decile_by_family_type/10/single", + "income_decile_by_family_type/10/couple", + "income_decile_by_family_type/10/sole_parent", + "income_decile_by_family_type/10/couple_with_children", + "beneficiary_superannuitant_other/beneficiary", + "beneficiary_superannuitant_other/superannuitant", + "beneficiary_superannuitant_other/other" + ], + "minimum_cell_ess": 100, + "maximum_contribution_cv": 2, + "maximum_single_donor_dollar_share": 0.1, + "target_minimum_cell_ess": 400, + "target_maximum_single_donor_dollar_share": 0.05 + }, + "evidence_absent_blocks": true, + "notes": "MC15: missing, malformed or scenario-incomplete source-household diagnostics block. Cells below ESS 100, above contribution CV 2 or donor dollar share 10% are suppressed, never reported; properly suppressed cells do not block. Structural-zero cells are listed separately. Subgroup rows report entitlement, without invented subgroup payment comparators." + }, + { + "id": "weight_effective_sample_size_target_a", + "gate": "weight_ess", + "phase": "terminal", + "criticality": "diagnostic", + "parameters": { + "aggregation_column": "source_household_id", + "required_weight_entity": "household", + "required_weight_kind": "calibrated", + "minimum_effective_sample_size": 16000 + }, + "notes": "MC15 target (a): report whether global original-source household ESS reaches 16000 on every build; never blocks release." + }, + { + "id": "reported_cell_support_target_a", + "gate": "cell_support_target", + "phase": "terminal", + "criticality": "diagnostic", + "parameters": { + "aggregation_column": "source_household_id", + "required_weight_entity": "household", + "required_weight_kind": "calibrated", + "required_scenarios": [ + "S0", + "S1", + "S2", + "S3", + "S4", + "S5", + "V1", + "V2", + "V3" + ], + "required_cells": [ + "income_decile_by_family_type/1/single", + "income_decile_by_family_type/1/couple", + "income_decile_by_family_type/1/sole_parent", + "income_decile_by_family_type/1/couple_with_children", + "income_decile_by_family_type/2/single", + "income_decile_by_family_type/2/couple", + "income_decile_by_family_type/2/sole_parent", + "income_decile_by_family_type/2/couple_with_children", + "income_decile_by_family_type/3/single", + "income_decile_by_family_type/3/couple", + "income_decile_by_family_type/3/sole_parent", + "income_decile_by_family_type/3/couple_with_children", + "income_decile_by_family_type/4/single", + "income_decile_by_family_type/4/couple", + "income_decile_by_family_type/4/sole_parent", + "income_decile_by_family_type/4/couple_with_children", + "income_decile_by_family_type/5/single", + "income_decile_by_family_type/5/couple", + "income_decile_by_family_type/5/sole_parent", + "income_decile_by_family_type/5/couple_with_children", + "income_decile_by_family_type/6/single", + "income_decile_by_family_type/6/couple", + "income_decile_by_family_type/6/sole_parent", + "income_decile_by_family_type/6/couple_with_children", + "income_decile_by_family_type/7/single", + "income_decile_by_family_type/7/couple", + "income_decile_by_family_type/7/sole_parent", + "income_decile_by_family_type/7/couple_with_children", + "income_decile_by_family_type/8/single", + "income_decile_by_family_type/8/couple", + "income_decile_by_family_type/8/sole_parent", + "income_decile_by_family_type/8/couple_with_children", + "income_decile_by_family_type/9/single", + "income_decile_by_family_type/9/couple", + "income_decile_by_family_type/9/sole_parent", + "income_decile_by_family_type/9/couple_with_children", + "income_decile_by_family_type/10/single", + "income_decile_by_family_type/10/couple", + "income_decile_by_family_type/10/sole_parent", + "income_decile_by_family_type/10/couple_with_children", + "beneficiary_superannuitant_other/beneficiary", + "beneficiary_superannuitant_other/superannuitant", + "beneficiary_superannuitant_other/other" + ], + "minimum_cell_ess": 400, + "maximum_single_donor_dollar_share": 0.05 + }, + "notes": "MC15 target (a): flag every cell with ESS at least 400 and maximum donor dollar share at most 5%; tracked on every build and never release-blocking." }, { "id": "macro_realism_nz", diff --git a/packages/microcosm-build/src/microcosm/build/nz/scenarios.json b/packages/microcosm-build/src/microcosm/build/nz/scenarios.json index b947b1981a..6fba457c8f 100644 --- a/packages/microcosm-build/src/microcosm/build/nz/scenarios.json +++ b/packages/microcosm-build/src/microcosm/build/nz/scenarios.json @@ -159,6 +159,36 @@ "beneficiary_superannuitant_other" ] }, + "reporting_cells": { + "entity": "family", + "measure": "accommodation_supplement", + "period": "2026-27", + "annualisation_factor": 52, + "tables": [ + { + "name": "income_decile_by_family_type", + "axes": [ + { + "column": "income_decile", + "values": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] + }, + { + "column": "family_type", + "values": ["single", "couple", "sole_parent", "couple_with_children"] + } + ] + }, + { + "name": "beneficiary_superannuitant_other", + "axes": [ + { + "column": "receipt_group", + "values": ["beneficiary", "superannuitant", "other"] + } + ] + } + ] + }, "comparators": { "method_card_row": "MC2", "references": "holdout_references.json, holdout_use accommodation_supplement_dollar_comparator" diff --git a/packages/microcosm-build/src/microcosm/build/nz/source_stages.json b/packages/microcosm-build/src/microcosm/build/nz/source_stages.json index ca16e8c7e7..df21d43b57 100644 --- a/packages/microcosm-build/src/microcosm/build/nz/source_stages.json +++ b/packages/microcosm-build/src/microcosm/build/nz/source_stages.json @@ -42,6 +42,7 @@ "person_id", "household_id", "person_household_id", + "source_household_id", "household_weight", "support_stratum", "donor_country_code", diff --git a/packages/microcosm-build/src/microcosm/build/nz/spec/sources.yaml b/packages/microcosm-build/src/microcosm/build/nz/spec/sources.yaml index fb6070dd5f..ee65dc0bbb 100644 --- a/packages/microcosm-build/src/microcosm/build/nz/spec/sources.yaml +++ b/packages/microcosm-build/src/microcosm/build/nz/spec/sources.yaml @@ -1,9 +1,9 @@ sources: -- byte_size: 2897 +- byte_size: 2928 id: nz_populace_us_support_contract loader: kernel:load_populace_us_support_pool role: nz_populace_us_support_pool - sha256: cc1eef0769c79917f33c5ba0f99edf861070143168db48e0145b71efeee751d1 + sha256: cc82c68878ac0fc07ecbd796c1a66de6f34786ef8785125046425c37f26aff31 vintage_authorities: - id: nz_support_pool_2024_buildp kind: survey_period @@ -60,6 +60,7 @@ stages: - person_id - household_id - person_household_id + - source_household_id - household_weight - support_stratum - donor_country_code diff --git a/packages/microcosm-build/src/microcosm/build/transport/artifact_types.py b/packages/microcosm-build/src/microcosm/build/transport/artifact_types.py index 22086c3345..46e13c8d90 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/artifact_types.py +++ b/packages/microcosm-build/src/microcosm/build/transport/artifact_types.py @@ -20,6 +20,7 @@ __all__ = [ "KERNEL_OUTPUTS", "COMPARISON_TYPE", + "CELL_DIAGNOSTICS_TYPE", "DIAGNOSTICS_TYPE", "EXPORT_DESCRIPTOR_TYPE", "EXPORT_READBACK_TYPE", @@ -32,6 +33,8 @@ TARGET_SURFACE_TYPE = ArtifactType("microcosm.targets.surface", 1) #: Hold-out comparators measured on a population (canonical JSON). COMPARISON_TYPE = ArtifactType("microcosm.targets.comparison", 1) +#: Source-household support and donor dollar influence in each scenario cell. +CELL_DIAGNOSTICS_TYPE = ArtifactType("microcosm.takeup.cell-diagnostics", 1) #: One calibration's ``microcosm-diagnostics`` model, validated, as canonical #: JSON; the type version is the diagnostics schema version. DIAGNOSTICS_TYPE = ArtifactType( @@ -54,6 +57,9 @@ "targets.compile@1": MappingProxyType({"surface": TARGET_SURFACE_TYPE}), "targets.problem@1": MappingProxyType({"problem": PROBLEM_TYPE}), "takeup.compare@1": MappingProxyType({"comparison": COMPARISON_TYPE}), + "takeup.cell_diagnostics@1": MappingProxyType( + {"cell_diagnostics": CELL_DIAGNOSTICS_TYPE} + ), "diagnostics.calibration@1": MappingProxyType( {"diagnostics": DIAGNOSTICS_TYPE} ), diff --git a/packages/microcosm-build/src/microcosm/build/transport/compose.py b/packages/microcosm-build/src/microcosm/build/transport/compose.py index 30b0f2b000..dad9b9d5ee 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/compose.py +++ b/packages/microcosm-build/src/microcosm/build/transport/compose.py @@ -210,7 +210,9 @@ class TransportExtension: Checkpoints name nodes in the extended graph and run after calibration, before the skeleton H5 is written; they cannot read the exported dataset. ``package_inputs`` explicitly connect typed extension evidence to a - skeleton package receipt. No other skeleton node can gain predecessors. + skeleton package receipt. ``gate_inputs`` connect required extension + evidence to a skeleton battery, whose verdict already governs export. + No other skeleton node can gain predecessors. The factory receives a deep copy of the caller's spec. """ @@ -218,6 +220,7 @@ class TransportExtension: sources: tuple[SourceRef, ...] = () checkpoints: tuple[str, ...] = () package_inputs: tuple[tuple[str, ArtifactInput], ...] = () + gate_inputs: tuple[tuple[str, ArtifactInput], ...] = () def __post_init__(self) -> None: if not callable(self.factory): @@ -238,6 +241,12 @@ def __post_init__(self) -> None: raise TypeError( "Extension package inputs require typed artifact edges." ) + if not isinstance(self.gate_inputs, tuple): + raise TypeError("Extension gate inputs must be a tuple.") + for target, edge in self.gate_inputs: + _safe_node_id(target) + if not isinstance(edge, ArtifactInput): + raise TypeError("Extension gate inputs require typed artifact edges.") def __call__( self, @@ -1220,9 +1229,19 @@ def compose_transport_graph( additional: list[Node] = [] additional_sources: list[SourceRef] = [] package_inputs: dict[str, list[ArtifactInput]] = {} + gate_inputs: dict[str, list[ArtifactInput]] = {} source_names = {source.name for source in skeleton.sources} for factory in config.extensions: if isinstance(factory, TransportExtension): + for target, edge in factory.gate_inputs: + if ( + target not in compiled.order + or skeleton.node(target).kernel != "gates.battery@1" + ): + raise ValueError( + "Extension gate evidence may target only a skeleton battery node." + ) + gate_inputs.setdefault(target, []).append(edge) for target, edge in factory.package_inputs: if ( target not in compiled.order @@ -1254,12 +1273,20 @@ def compose_transport_graph( for node in nodes: _safe_node_id(node.id) additional.extend(nodes) - if not additional and not additional_sources and not package_inputs: + if ( + not additional + and not additional_sources + and not package_inputs + and not gate_inputs + ): _check_extension_checkpoints(spec, config, skeleton, compiled) return skeleton skeleton_nodes = [] for node in skeleton.nodes: - edges = tuple(package_inputs.get(node.id, ())) + edges = ( + *package_inputs.get(node.id, ()), + *gate_inputs.get(node.id, ()), + ) aliases = [edge.name for edge in (*node.artifact_inputs, *edges)] if len(aliases) != len(set(aliases)): raise ValueError( @@ -1280,9 +1307,11 @@ def compose_transport_graph( _check_extension_checkpoints(spec, config, graph, extended) validate_transport_calibration_ancestry(spec, graph=graph, config=config) for node in skeleton.nodes: - expected = set(compiled.predecessors[node.id]) | { - edge.producer for edge in package_inputs.get(node.id, ()) - } + expected = ( + set(compiled.predecessors[node.id]) + | {edge.producer for edge in package_inputs.get(node.id, ())} + | {edge.producer for edge in gate_inputs.get(node.id, ())} + ) if set(extended.predecessors[node.id]) != expected: raise ValueError( f"An extension would re-key skeleton node {node.id!r}; " diff --git a/packages/microcosm-build/src/microcosm/build/transport/entitlement.py b/packages/microcosm-build/src/microcosm/build/transport/entitlement.py index 1c644d05ce..d06fcec41d 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/entitlement.py +++ b/packages/microcosm-build/src/microcosm/build/transport/entitlement.py @@ -18,7 +18,8 @@ ``variant_gap`` name each tier's gap producer template. ``package`` names the skeleton receipt node and the bands artifact alias; its optional ``inputs`` adds other typed extension evidence, such as a held-out validation tripwire. -The only skeleton change is these explicit artifact edges on the final receipt. +The extension adds explicit artifact edges to the final receipt and, when a +cell-support gate is declared, to its matching skeleton gate battery. """ from __future__ import annotations @@ -45,7 +46,12 @@ _sequence, transport_resource, ) -from .takeup_kernels import BANDS_TYPE, GAP_TYPE +from .takeup_kernels import ( + BANDS_TYPE, + CELL_DIAGNOSTICS_TYPE, + GAP_TYPE, + reporting_cell_declarations, +) __all__ = ["ENTITLEMENT_RESOURCE", "entitlement_extension", "transport_extensions"] @@ -56,6 +62,7 @@ "simulate.solve_zero@1", "takeup.gap@1", "takeup.bands@1", + "takeup.cell_diagnostics@1", "takeup.compare@1", "transport.scenario_override@1", } @@ -188,7 +195,106 @@ def _rows(spec): return result, gaps -def _types(rows, skeleton=None): +def _require_matching_cell_targets(spec, contract): + """Refuse a tracked cell target that disagrees with the support contract. + + The support contract's target thresholds decide the per-cell flags in the + diagnostics; the ``cell_support_target`` gate reports them. Both are spec + data, so a spec that changes one copy without the other would flag cells + against one threshold and report against another. + + Every active target gate is checked, ``"not_applicable": null`` included, + as the gate parser treats it. Each shared value must be declared on both + sides. Lists compare as multisets, ordered by their canonical JSON, so a + malformed list is refused with this error rather than a ``TypeError``. + """ + pairs = ( + ("minimum_cell_ess", "target_minimum_cell_ess"), + ( + "maximum_single_donor_dollar_share", + "target_maximum_single_donor_dollar_share", + ), + ("required_cells", "required_cells"), + ("required_scenarios", "required_scenarios"), + ("aggregation_column", "aggregation_column"), + ) + for gate in transport_resource(spec, "gates")["gates"]: + if gate["gate"] != "cell_support_target" or not _applicable(gate): + continue + target = _mapping(gate.get("parameters"), "cell-support target parameters") + for own, declared in pairs: + if own not in target or declared not in contract: + raise ValueError( + f"The cell-support target gate {gate['id']!r} and the support " + f"contract must both declare {own!r} (contract: {declared!r})." + ) + if not _same_declaration(target[own], contract[declared]): + raise ValueError( + f"The cell-support target gate {gate['id']!r} declares " + f"{own}={target[own]!r}, but the support contract declares " + f"{declared}={contract[declared]!r}; the two must agree." + ) + + +def _applicable(gate): + """Return whether a gate entry runs: ``not_applicable`` absent or null.""" + return gate.get("not_applicable") is None + + +def _same_declaration(mine, theirs): + """Compare two declared values, lists as multisets.""" + if isinstance(mine, list | tuple) and isinstance(theirs, list | tuple): + + def key(item): + return json.dumps(item, sort_keys=True) + + return sorted(mine, key=key) == sorted(theirs, key=key) + return mine == theirs + + +def _cell_contract(spec): + resources = spec.get("resources", {}) + if not any(name in resources for name in ("gates", "gates.json")): + return None + matches = [ + row + for row in transport_resource(spec, "gates")["gates"] + if row["gate"] == "cell_support" and _applicable(row) + ] + if not matches: + return None + if len(matches) != 1: + raise ValueError( + "An entitlement graph needs exactly one cell-support contract." + ) + gate = matches[0] + parameters = _mapping(gate["parameters"], "cell-support parameters") + thresholds = { + name: parameters[name] + for name in ( + "minimum_cell_ess", + "maximum_contribution_cv", + "maximum_single_donor_dollar_share", + "target_minimum_cell_ess", + "target_maximum_single_donor_dollar_share", + ) + } + _require_matching_cell_targets(spec, parameters) + reporting = transport_resource(spec, "scenarios")["gap_conventions"][ + "reporting_cells" + ] + declarations = reporting_cell_declarations(reporting) + if set(parameters["required_cells"]) != {row["name"] for row in declarations}: + raise ValueError( + "The support gate must require exactly the declared reported cells." + ) + _, gaps = _rows(spec) + if set(parameters["required_scenarios"]) != set(gaps): + raise ValueError("The support gate must require every declared scenario.") + return gate, thresholds, reporting + + +def _types(rows, skeleton=None, *, cell_support=False): types = ( {} if skeleton is None @@ -199,9 +305,12 @@ def _types(rows, skeleton=None): } ) for row, _ in rows: - outputs = {"takeup.gap@1": (ArtifactOutput("gap", GAP_TYPE),)}.get( - row["kernel"], _artifact_outputs(row) - ) + if row["kernel"] == "takeup.gap@1": + outputs = (ArtifactOutput("gap", GAP_TYPE),) + if cell_support: + outputs += (ArtifactOutput("cell_diagnostics", CELL_DIAGNOSTICS_TYPE),) + else: + outputs = _artifact_outputs(row) types.update({(row["id"], output.name): output.type for output in outputs}) return types @@ -211,25 +320,86 @@ class _EntitlementFactory: def __call__(self, spec: Mapping, config: TransportGraphConfig, skeleton: Graph): document = _document(spec) rows, gaps = _rows(spec) - types = _types(rows, skeleton) + cell_contract = _cell_contract(spec) + types = _types(rows, skeleton, cell_support=cell_contract is not None) nodes: list[Node] = [] for row, scenario in rows: params = { key: _parameter(spec, config, value, scenario) for key, value in row.get("params", {}).items() } - if row["kernel"] == "takeup.gap@1" and "artifact_outputs" not in row: - row = { - **row, - "artifact_outputs": [ + if row["kernel"] == "takeup.gap@1" and cell_contract is not None: + gate, thresholds, reporting = cell_contract + contract_params = gate["parameters"] + params.update( + source_household_column=contract_params["aggregation_column"], + expected_weight_kind=contract_params["required_weight_kind"], + cell_support=canonical_json(thresholds).decode("utf-8"), + reporting_cells=canonical_json(reporting).decode("utf-8"), + ) + if ( + params.get("weight_entity") + != contract_params["required_weight_entity"] + ): + raise ValueError( + "Reported cells must use the declared household weight entity." + ) + extra_columns = { + contract_params["required_weight_entity"]: { + contract_params["aggregation_column"] + } + } + extra_columns.setdefault(reporting["entity"], set()).update( + { + reporting["measure"], + *( + axis["column"] + for table in reporting["tables"] + for axis in table["axes"] + ), + } + ) + inputs = [dict(item) for item in row.get("inputs", [])] + for entity, columns in extra_columns.items(): + matching = next( + (item for item in inputs if item["entity"] == entity), None + ) + if matching is None: + inputs.append({"entity": entity, "columns": sorted(columns)}) + else: + existing = list(matching["columns"]) + matching["columns"] = existing + sorted(columns - set(existing)) + row = {**row, "inputs": inputs} + if row["kernel"] == "takeup.gap@1": + outputs = list( + row.get( + "artifact_outputs", + [ + { + "name": "gap", + "type": { + "name": GAP_TYPE.name, + "schema_version": GAP_TYPE.schema_version, + }, + } + ], + ) + ) + if cell_contract is not None and not any( + output["name"] == "cell_diagnostics" for output in outputs + ): + outputs.append( { - "name": "gap", + "name": "cell_diagnostics", "type": { - "name": GAP_TYPE.name, - "schema_version": GAP_TYPE.schema_version, + "name": CELL_DIAGNOSTICS_TYPE.name, + "schema_version": CELL_DIAGNOSTICS_TYPE.schema_version, }, } - ], + ) + row = { + **row, + "artifact_outputs": outputs, } nodes.append( _node( @@ -243,6 +413,26 @@ def __call__(self, spec: Mapping, config: TransportGraphConfig, skeleton: Graph) ) bands = _mapping(document["bands"], "entitlement bands") aliases = {f"scenario_{index}": name for index, name in enumerate(sorted(gaps))} + if cell_contract is not None: + nodes.append( + Node( + id=bands["id"] + ".cell_diagnostics", + kernel="takeup.cell_diagnostics@1", + population=bands["population"], + params={ + "artifact_scenarios": canonical_json(aliases).decode("utf-8") + }, + artifact_inputs=tuple( + ArtifactInput( + alias, gaps[name], "cell_diagnostics", CELL_DIAGNOSTICS_TYPE + ) + for alias, name in aliases.items() + ), + artifact_outputs=( + ArtifactOutput("cell_diagnostics", CELL_DIAGNOSTICS_TYPE), + ), + ) + ) nodes.append( Node( id=bands["id"], @@ -268,7 +458,8 @@ def entitlement_extension(spec: Mapping) -> TransportExtension: rows, _ = _rows(spec) bands = _mapping(document["bands"], "entitlement bands") package = _mapping(document["package"], "entitlement package") - types = _types(rows) + cell_contract = _cell_contract(spec) + types = _types(rows, cell_support=cell_contract is not None) types[bands["id"], "bands"] = BANDS_TYPE edges = [ ArtifactInput( @@ -282,11 +473,43 @@ def entitlement_extension(spec: Mapping) -> TransportExtension: row["name"], row["producer"], artifact, types[row["producer"], artifact] ) ) + checkpoints = tuple(document.get("checkpoints", (bands["id"],))) + gate_inputs = [] + if cell_contract is not None: + collector = bands["id"] + ".cell_diagnostics" + edges.append( + ArtifactInput( + "cell_diagnostics", collector, "cell_diagnostics", CELL_DIAGNOSTICS_TYPE + ) + ) + checkpoints += (collector,) + screen_gate = cell_contract[0] + for row in transport_resource(spec, "transport_graph")["nodes"]: + if ( + row["kernel"] == "gates.battery@1" + and row.get("params", {}).get("phase") == screen_gate["phase"] + ): + gate_inputs.append( + ( + row["id"], + ArtifactInput( + "cell_diagnostics", + collector, + "cell_diagnostics", + CELL_DIAGNOSTICS_TYPE, + ), + ) + ) + if not gate_inputs: + raise ValueError( + "Cell diagnostics need a matching declared gate-battery node." + ) return TransportExtension( factory=_EntitlementFactory(), sources=tuple(SourceRef(**row) for row in document.get("sources", ())), - checkpoints=tuple(document.get("checkpoints", (bands["id"],))), + checkpoints=checkpoints, package_inputs=tuple((package["id"], edge) for edge in edges), + gate_inputs=tuple(gate_inputs), ) diff --git a/packages/microcosm-build/src/microcosm/build/transport/gate_bindings.py b/packages/microcosm-build/src/microcosm/build/transport/gate_bindings.py index 7d81dee0b5..3b396d5197 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/gate_bindings.py +++ b/packages/microcosm-build/src/microcosm/build/transport/gate_bindings.py @@ -98,6 +98,8 @@ from microcosm.calibrate.solve import effective_sample_size from microcosm.frame import Frame +from .weight_quality import aggregate_source_weights + __all__ = [ "DIAGNOSTICS", "PROBLEM", @@ -117,6 +119,7 @@ #: Alias of one calibration's schema-8 diagnostics #: (``diagnostics.calibration@1`` ``diagnostics``). DIAGNOSTICS = "diagnostics" +CELL_DIAGNOSTICS = "cell_diagnostics" # --------------------------------------------------------------------------- @@ -332,7 +335,11 @@ def _exact_median(positive: np.ndarray) -> float: return float(statistics.median(Fraction(float(value)) for value in central)) -def weight_summary(weights: Sequence[float] | np.ndarray) -> dict[str, Any]: +def weight_summary( + weights: Sequence[float] | np.ndarray, + *, + source_household_ids: Sequence[object] | np.ndarray | None = None, +) -> dict[str, Any]: """The concentration summary of one shipped weight vector. The ESS fraction divides the Kish effective sample size by every record, @@ -348,7 +355,11 @@ def weight_summary(weights: Sequence[float] | np.ndarray) -> dict[str, Any]: invalid concentration summaries are refused rather than compared. """ - values = np.asarray(weights, dtype=np.float64) + values = ( + np.asarray(weights, dtype=np.float64) + if source_household_ids is None + else aggregate_source_weights(weights, source_household_ids) + ) if values.ndim != 1 or values.size == 0: raise ValueError("Weights must be a non-empty vector.") if not np.isfinite(values).all() or (values < 0).any(): @@ -410,38 +421,70 @@ def weight_summary(weights: Sequence[float] | np.ndarray) -> dict[str, Any]: def weight_ess_gate( - weights: Sequence[float] | np.ndarray, *, minimum_ess_fraction: float + weights: Sequence[float] | np.ndarray, + *, + minimum_ess_fraction: float | None = None, + minimum_effective_sample_size: float | None = None, + source_household_ids: Sequence[object] | np.ndarray | None = None, ) -> GateResult: """Require the shipped weights to retain an effective-sample-size floor.""" - minimum = _number("minimum_ess_fraction", minimum_ess_fraction) - if not 0 < minimum <= 1: + if minimum_ess_fraction is None and minimum_effective_sample_size is None: + raise ValueError( + "Declare minimum_ess_fraction or minimum_effective_sample_size." + ) + minimum = ( + None + if minimum_ess_fraction is None + else _number("minimum_ess_fraction", minimum_ess_fraction) + ) + if minimum is not None and not 0 < minimum <= 1: raise ValueError("minimum_ess_fraction must lie in (0, 1].") + absolute = ( + None + if minimum_effective_sample_size is None + else _number("minimum_effective_sample_size", minimum_effective_sample_size) + ) + if absolute is not None and not absolute > 0: + raise ValueError("minimum_effective_sample_size must be strictly positive.") + thresholds = { + **({"minimum_ess_fraction": minimum} if minimum is not None else {}), + **({"minimum_effective_sample_size": absolute} if absolute is not None else {}), + } try: - summary = weight_summary(weights) + summary = weight_summary(weights, source_household_ids=source_household_ids) except _InvalidWeightSummaryError as exc: return GateResult( name="weight_ess", passed=False, failures=(str(exc),), - details={"minimum_ess_fraction": minimum}, + details=thresholds, ) fraction = float(summary["ess_fraction"]) failures = ( (f"ESS fraction {fraction:.6g} is below the reviewed minimum {minimum:.6g}.",) - if not fraction >= minimum + if minimum is not None and not fraction >= minimum else () ) + if absolute is not None and not summary["effective_sample_size"] >= absolute: + failures = ( + *failures, + f"ESS {summary['effective_sample_size']:.6g} is below the reviewed " + f"absolute minimum {absolute:.6g}.", + ) return GateResult( name="weight_ess", passed=not failures, failures=failures, - details={**summary, "minimum_ess_fraction": minimum}, + details={**summary, **thresholds}, ) def weight_ratio_gate( - weights: Sequence[float] | np.ndarray, *, maximum_max_to_median_ratio: float + weights: Sequence[float] | np.ndarray, + *, + maximum_max_to_median_ratio: float, + source_household_ids: Sequence[object] | np.ndarray | None = None, ) -> GateResult: """Backstop a shipped-weight max to positive-median concentration blowout.""" @@ -449,7 +492,7 @@ def weight_ratio_gate( if not maximum > 0: raise ValueError("maximum_max_to_median_ratio must be strictly positive.") try: - summary = weight_summary(weights) + summary = weight_summary(weights, source_household_ids=source_household_ids) except _InvalidWeightSummaryError as exc: return GateResult( name="weight_ratio", @@ -821,18 +864,328 @@ def _formula_owned_export( ) -def _weight_ess(*, frame: Frame, minimum_ess_fraction: float) -> GateResult: +def _weight_inputs( + frame: Frame, + aggregation_column: str | None, + required_weight_entity: str | None, + required_weight_kind: str | None, +) -> tuple[str, np.ndarray, np.ndarray | None, dict[str, object]]: entity, weights = _weighted_entity(frame) - result = weight_ess_gate(weights, minimum_ess_fraction=minimum_ess_fraction) - return _with_failures(result, failures=(), details={"weight_entity": entity}) + if required_weight_entity is not None and entity != required_weight_entity: + raise ValueError( + f"Weight gates require {required_weight_entity!r} weights, got {entity!r}." + ) + kind = frame.weights_for(entity).kind.value + if required_weight_kind is not None and kind != required_weight_kind: + raise ValueError( + f"Weight gates require {required_weight_kind!r} weights, got {kind!r}." + ) + lineage = None + details: dict[str, object] = {"weight_entity": entity} + if aggregation_column is not None: + if not isinstance(aggregation_column, str) or not aggregation_column: + raise ValueError("aggregation_column must name a lineage column.") + table = frame.table(entity) + if aggregation_column not in table: + raise ValueError( + f"Missing source-household lineage column {aggregation_column!r} " + f"on {entity!r}." + ) + lineage = table[aggregation_column].to_numpy() + details.update( + aggregation_column=aggregation_column, + weight_kind=kind, + n_support_rows=int(weights.size), + ) + return entity, weights, lineage, details -def _weight_ratio(*, frame: Frame, maximum_max_to_median_ratio: float) -> GateResult: - entity, weights = _weighted_entity(frame) +def _weight_ess( + *, + frame: Frame, + minimum_ess_fraction: float | None = None, + minimum_effective_sample_size: float | None = None, + aggregation_column: str | None = None, + required_weight_entity: str | None = None, + required_weight_kind: str | None = None, +) -> GateResult: + _, weights, lineage, details = _weight_inputs( + frame, aggregation_column, required_weight_entity, required_weight_kind + ) + result = weight_ess_gate( + weights, + minimum_ess_fraction=minimum_ess_fraction, + minimum_effective_sample_size=minimum_effective_sample_size, + source_household_ids=lineage, + ) + return _with_failures(result, failures=(), details=details) + + +def _weight_ratio( + *, + frame: Frame, + maximum_max_to_median_ratio: float, + aggregation_column: str | None = None, + required_weight_entity: str | None = None, + required_weight_kind: str | None = None, +) -> GateResult: + _, weights, lineage, details = _weight_inputs( + frame, aggregation_column, required_weight_entity, required_weight_kind + ) result = weight_ratio_gate( - weights, maximum_max_to_median_ratio=maximum_max_to_median_ratio + weights, + maximum_max_to_median_ratio=maximum_max_to_median_ratio, + source_household_ids=lineage, + ) + return _with_failures(result, failures=(), details=details) + + +def _cell_rows( + cell_diagnostics: object, + required_scenarios: Sequence[str], + required_cells: Sequence[str], + aggregation_column: str, + required_weight_entity: str, + required_weight_kind: str, +) -> list[tuple[str, Mapping[str, Any]]]: + """Refuse absent identities or incomplete declared cell/scenario coverage.""" + + scenarios = _strings("required_scenarios", required_scenarios) + cells = _strings("required_cells", required_cells) + if not isinstance(cell_diagnostics, Mapping): + raise ValueError("Cell diagnostics must be a decoded artifact mapping.") + if ( + cell_diagnostics.get("schema_version") != 1 + or cell_diagnostics.get("kind") != "transport_cell_support_diagnostics" + ): + raise ValueError("Unsupported cell diagnostics schema.") + for key, expected in ( + ("source_household_column", aggregation_column), + ("weight_entity", required_weight_entity), + ("weight_kind", required_weight_kind), + ): + if not expected or cell_diagnostics.get(key) != expected: + raise ValueError( + f"Cell diagnostics require {key}={expected!r}, got " + f"{cell_diagnostics.get(key)!r}." + ) + entries = cell_diagnostics.get("scenarios") + if not isinstance(entries, list | tuple) or not entries: + raise ValueError("Cell diagnostics are missing declared scenarios.") + found: list[str] = [] + all_cells: set[str] | None = None + result = [] + for entry in entries: + if not isinstance(entry, Mapping) or entry.get("scenario") not in scenarios: + raise ValueError("Cell diagnostics contain an undeclared scenario.") + scenario = entry["scenario"] + found.append(scenario) + rows = entry.get("cells") + if not isinstance(rows, list | tuple) or not rows: + raise ValueError(f"Cell diagnostics for {scenario} are missing cells.") + names = [row.get("name") for row in rows if isinstance(row, Mapping)] + if len(names) != len(rows) or len(set(names)) != len(names): + raise ValueError(f"Cell diagnostics for {scenario} repeat or omit cells.") + if not set(cells) <= set(names): + raise ValueError( + f"Cell diagnostics for {scenario} have incomplete declared cells: " + f"missing {[name for name in cells if name not in names]}." + ) + if all_cells is None: + all_cells = set(names) + elif set(names) != all_cells: + raise ValueError( + "Cell diagnostics must cover the same cells in every scenario." + ) + for row in rows: + period = row.get("period") + if ( + isinstance(period, bool) + or not isinstance(period, int | str) + or period == "" + ): + raise ValueError(f"Cell {row['name']} needs an explicit period.") + for flag in ("structural_zero", "suppressed", "meets_tracked_target"): + if not isinstance(row.get(flag), bool): + raise ValueError(f"Cell {row['name']} needs boolean {flag}.") + for metric in ("source_household_ess", "dollar_contribution_ess"): + if _number(metric, row.get(metric)) < 0: + raise ValueError(f"Cell {row['name']} has negative {metric}.") + contributors = row.get("positive_contributors") + if ( + isinstance(contributors, bool) + or not isinstance(contributors, int) + or contributors < 0 + ): + raise ValueError(f"Cell {row['name']} needs a contributor count.") + if row["structural_zero"]: + if ( + contributors + or row["dollar_contribution_ess"] + or row.get("contribution_cv") != 0 + or row.get("maximum_single_donor_dollar_share") != 0 + ): + raise ValueError( + "Structural-zero cells cannot have dollar support." + ) + elif contributors: + if _number("contribution_cv", row.get("contribution_cv")) < 0: + raise ValueError("Contribution CV must be non-negative.") + share = _number( + "maximum_single_donor_dollar_share", + row.get("maximum_single_donor_dollar_share"), + ) + if not 0 < share <= 1: + raise ValueError( + "Nonzero cells need positive donor dollar support." + ) + elif ( + row.get("contribution_cv") is not None + or row.get("maximum_single_donor_dollar_share") is not None + or row["dollar_contribution_ess"] != 0 + ): + raise ValueError( + "Cells without positive contributors have no dollar ratios." + ) + result.append((scenario, row)) + if len(set(found)) != len(found) or set(found) != set(scenarios): + raise ValueError( + "Cell diagnostics have missing or repeated scenarios: " + f"missing {[name for name in scenarios if name not in found]}." + ) + return result + + +def _cell_support( + *, + cell_diagnostics: object, + required_scenarios: Sequence[str], + required_cells: Sequence[str], + aggregation_column: str, + required_weight_entity: str, + required_weight_kind: str, + minimum_cell_ess: float, + maximum_contribution_cv: float, + maximum_single_donor_dollar_share: float, + target_minimum_cell_ess: float, + target_maximum_single_donor_dollar_share: float, +) -> GateResult: + """Require complete diagnostics and suppression, allowing sparse cells.""" + + rows = _cell_rows( + cell_diagnostics, + required_scenarios, + required_cells, + aggregation_column, + required_weight_entity, + required_weight_kind, + ) + minimum = _number("minimum_cell_ess", minimum_cell_ess) + cv_limit = _number("maximum_contribution_cv", maximum_contribution_cv) + share_limit = _number( + "maximum_single_donor_dollar_share", maximum_single_donor_dollar_share + ) + target_minimum = _number("target_minimum_cell_ess", target_minimum_cell_ess) + target_share = _number( + "target_maximum_single_donor_dollar_share", + target_maximum_single_donor_dollar_share, + ) + if ( + minimum <= 0 + or cv_limit < 0 + or not 0 < share_limit <= 1 + or target_minimum <= 0 + or not 0 < target_share <= 1 + ): + raise ValueError("Cell support thresholds are outside their valid ranges.") + failures = [] + suppressed = [] + structural = [] + for scenario, row in rows: + identity = f"{scenario}/{row['name']}" + zero = row["structural_zero"] + unsupported = row["positive_contributors"] == 0 + should_suppress = ( + zero + or unsupported + or ( + row["source_household_ess"] < minimum + or row["contribution_cv"] > cv_limit + or row["maximum_single_donor_dollar_share"] > share_limit + ) + ) + if should_suppress and not row["suppressed"]: + failures.append(f"{identity}: failing cell is not suppressed.") + tracked = ( + not zero + and not unsupported + and ( + row["source_household_ess"] >= target_minimum + and row["maximum_single_donor_dollar_share"] <= target_share + ) + ) + if row["meets_tracked_target"] != tracked: + failures.append(f"{identity}: tracked-target flag is inconsistent.") + if row["suppressed"]: + suppressed.append(identity) + if zero: + structural.append(identity) + return GateResult( + name="cell_support", + passed=not failures, + failures=tuple(failures), + details={ + "n_cell_scenarios": len(rows), + "suppressed_cells": suppressed, + "structural_zero_cells": structural, + }, + ) + + +def _cell_support_target( + *, + cell_diagnostics: object, + required_scenarios: Sequence[str], + required_cells: Sequence[str], + aggregation_column: str, + required_weight_entity: str, + required_weight_kind: str, + minimum_cell_ess: float, + maximum_single_donor_dollar_share: float, +) -> GateResult: + """Report the tighter cell screen without blocking release.""" + + rows = _cell_rows( + cell_diagnostics, + required_scenarios, + required_cells, + aggregation_column, + required_weight_entity, + required_weight_kind, + ) + minimum = _number("minimum_cell_ess", minimum_cell_ess) + maximum = _number( + "maximum_single_donor_dollar_share", maximum_single_donor_dollar_share + ) + if minimum <= 0 or not 0 < maximum <= 1: + raise ValueError("Tracked cell support thresholds are outside valid ranges.") + flags = { + f"{scenario}/{row['name']}": not row["structural_zero"] + and row["positive_contributors"] > 0 + and row["source_household_ess"] >= minimum + and row["maximum_single_donor_dollar_share"] <= maximum + for scenario, row in rows + } + return GateResult( + name="cell_support_target", + passed=all(flags.values()), + failures=tuple( + f"{identity}: tracked cell target not met." + for identity, passed in flags.items() + if not passed + ), + details={"meets_tracked_target": flags}, ) - return _with_failures(result, failures=(), details={"weight_entity": entity}) def _pending_support(*, donor_support_bounds: object) -> GateResult: @@ -897,6 +1250,12 @@ def _problem_evidence( return {"problem_sha256": context.artifacts[PROBLEM].sha256} +def _cell_diagnostics_evidence( + context: EvidenceContext, parameters: Mapping[str, Any] +) -> object: + return {CELL_DIAGNOSTICS: context.artifacts[CELL_DIAGNOSTICS]} + + # --------------------------------------------------------------------------- # The registry # --------------------------------------------------------------------------- @@ -905,7 +1264,7 @@ def _problem_evidence( def _parameters(gate: object) -> frozenset[str]: """Every keyword the evaluator takes that no artifact or frame supplies.""" - supplied = {"frame", SURFACE, PROBLEM, DIAGNOSTICS} + supplied = {"frame", SURFACE, PROBLEM, DIAGNOSTICS, CELL_DIAGNOSTICS} return frozenset( name for name in inspect.signature(gate).parameters if name not in supplied ) @@ -977,6 +1336,20 @@ def _parameters(gate: object) -> frozenset[str]: parameter_keys=_parameters(_weight_ratio), frame_argument="frame", ), + "cell_support": FunctionBinding( + name="cell_support", + gate=_cell_support, + parameter_keys=_parameters(_cell_support), + artifact_arguments=MappingProxyType({CELL_DIAGNOSTICS: CELL_DIAGNOSTICS}), + evidence=_cell_diagnostics_evidence, + ), + "cell_support_target": FunctionBinding( + name="cell_support_target", + gate=_cell_support_target, + parameter_keys=_parameters(_cell_support_target), + artifact_arguments=MappingProxyType({CELL_DIAGNOSTICS: CELL_DIAGNOSTICS}), + evidence=_cell_diagnostics_evidence, + ), "support": FunctionBinding( name="support", gate=_pending_support, @@ -1044,6 +1417,11 @@ def validate_required_gate_parameters( if binding is None: continue missing = sorted(required_gate_parameters(binding) - set(entry.parameters)) + if entry.gate == "weight_ess" and not ( + {"minimum_ess_fraction", "minimum_effective_sample_size"} + & set(entry.parameters) + ): + missing.append("minimum_ess_fraction or minimum_effective_sample_size") if missing: raise ValueError( f"gate entry {entry.id!r} omits parameters {missing} that the " diff --git a/packages/microcosm-build/src/microcosm/build/transport/gate_kernels.py b/packages/microcosm-build/src/microcosm/build/transport/gate_kernels.py index 3cc0498479..9ae6a4ec4e 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/gate_kernels.py +++ b/packages/microcosm-build/src/microcosm/build/transport/gate_kernels.py @@ -121,6 +121,7 @@ from . import artifact_types, binding_identity, graph_inputs, target_kernels from .artifact_types import ( + CELL_DIAGNOSTICS_TYPE, COMPARISON_TYPE, DIAGNOSTICS_TYPE, EXPORT_DESCRIPTOR_TYPE, @@ -140,6 +141,7 @@ sha256_text, string_param, ) +from .takeup_kernels import decode_cell_diagnostics from .target_kernels import decode_target_surface __all__ = [ @@ -603,6 +605,7 @@ def _json_artifact(payload: bytes) -> object: TARGET_SURFACE_TYPE: decode_target_surface, COMPARISON_TYPE: _json_artifact, DIAGNOSTICS_TYPE: _json_artifact, + CELL_DIAGNOSTICS_TYPE: decode_cell_diagnostics, GATE_REPORT_TYPE: decode_gate_report, EXPORT_DESCRIPTOR_TYPE: _json_artifact, EXPORT_READBACK_TYPE: _json_artifact, diff --git a/packages/microcosm-build/src/microcosm/build/transport/takeup_kernels.py b/packages/microcosm-build/src/microcosm/build/transport/takeup_kernels.py index e1a4658a75..8b5c7a6ae9 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/takeup_kernels.py +++ b/packages/microcosm-build/src/microcosm/build/transport/takeup_kernels.py @@ -6,12 +6,18 @@ The estimate is the sum of effective weight times annualised measure and the gap is exactly that estimate minus the selected comparator. Reference metadata can label any declared table cells, including receipt strata. +An optional declared reporting grid adds entitlement breakdowns without +subgroup payment comparators. Its support diagnostics merge clone weights and +dollar contributions by explicit original-source household identity. Screened +cells carry suppression reasons and no reportable numeric values. ``takeup.bands@1`` consumes the gap artifacts from declared scenarios. For each table cell and each of its estimate, actual and gap values it records the minimum, central scenario and maximum, naming the scenario at every bound. Equal-valued ties use the lexicographically first scenario label. No scenario list, rate or comparator is held in the kernel implementation. +``takeup.cell_diagnostics@1`` combines every declared scenario's diagnostics; +absent scenarios refuse the collection before the release gates read it. """ from __future__ import annotations @@ -19,6 +25,7 @@ import json import math from collections.abc import Mapping +from itertools import product import numpy as np @@ -36,7 +43,8 @@ ) from microcosm.graph.canonical import canonical_json -from . import graph_inputs, target_kernels +from . import graph_inputs, target_kernels, weight_quality +from .artifact_types import CELL_DIAGNOSTICS_TYPE from .graph_inputs import ( canonical_document_param, context_frame, @@ -55,13 +63,19 @@ __all__ = [ "BANDS_TYPE", + "CELL_DIAGNOSTICS_TYPE", "GAP_TYPE", "TAKEUP_BANDS", "TAKEUP_GAP", + "TAKEUP_CELL_DIAGNOSTICS", "TakeupBandsKernel", "TakeupGapKernel", "decode_bands", "decode_gap", + "decode_cell_diagnostics", + "source_cell_diagnostics", + "apply_cell_support", + "reporting_cell_declarations", "register_takeup_kernels", ] @@ -69,6 +83,26 @@ BANDS_TYPE = ArtifactType("microcosm.takeup.bands", 1) _GAP_KIND = "transport_entitlement_gap" _BANDS_KIND = "transport_entitlement_bands" +_CELL_DIAGNOSTICS_KIND = "transport_cell_support_diagnostics" +_CELL_FIELDS = frozenset( + { + "source_household_ess", + "positive_contributors", + "contribution_cv", + "maximum_single_donor_dollar_share", + "dollar_contribution_ess", + "structural_zero", + } +) +_SUPPORT_FIELDS = frozenset( + { + "minimum_cell_ess", + "maximum_contribution_cv", + "maximum_single_donor_dollar_share", + "target_minimum_cell_ess", + "target_maximum_single_donor_dollar_share", + } +) _METRICS = ("E", "A", "gap") _ROW_SEMANTICS = ("name", "period", "entity", "measure", "filter", "metadata") _SCENARIO_SEMANTICS = ( @@ -82,6 +116,202 @@ ) +def _kish(values: np.ndarray) -> float: + maximum = float(np.max(values, initial=0)) + if maximum == 0: + return float(0) + scaled = values / maximum + return float(np.sum(scaled) ** 2 / np.sum(scaled**2)) + + +def source_cell_diagnostics( + weights, annualised_amounts, source_household_ids, *, support=None +) -> dict: + """Summarise independent household support after merging clones. + + Inputs align to household rows, with benefit-unit amounts already summed + within each household. Support includes households in the cell even when + their amount is zero. The contribution CV includes those zeros. A zero + weighted total with nonzero amounts is unsupported, not a structural zero. + """ + from .weight_quality import aggregate_source_weights + + values = np.asarray(weights, dtype=np.float64) + amounts = np.asarray(annualised_amounts, dtype=np.float64) + identities = np.asarray(source_household_ids, dtype=object) + aggregate_source_weights(values, identities) + if amounts.shape != values.shape or not np.isfinite(amounts).all(): + raise ValueError( + "Cell amounts must be finite and align with household weights." + ) + if bool((amounts < 0).any()): + raise ValueError("Cell entitlement amounts must be nonnegative.") + selected = ( + np.ones(values.shape, dtype=bool) + if support is None + else np.asarray(support, dtype=bool) + ) + if selected.shape != values.shape: + raise ValueError("Cell support must align with household weights.") + values, amounts, identities = ( + vector[selected] for vector in (values, amounts, identities) + ) + if not len(values): + return { + "source_household_ess": float(0), + "positive_contributors": 0, + "contribution_cv": float(0), + "maximum_single_donor_dollar_share": float(0), + "dollar_contribution_ess": float(0), + "structural_zero": True, + } + # Scaling before multiplication avoids overflow and cancels out in every + # diagnostic. Clone dollar contributions, rather than clone amounts, add. + weight_max = float(np.max(values, initial=0)) + amount_max = float(np.max(amounts, initial=0)) + scaled_weights = values / weight_max if weight_max else values + scaled_amounts = amounts / amount_max if amount_max else amounts + household_weights = aggregate_source_weights(scaled_weights, identities) + contributions = aggregate_source_weights( + scaled_weights * scaled_amounts, identities + ) + total = float(np.sum(contributions)) + structural_zero = amount_max == 0 + if total: + mean = total / len(contributions) + cv = float(np.std(contributions) / mean) + maximum_share = float(np.max(contributions) / total) + else: + cv = None if not structural_zero else float(0) + maximum_share = None if not structural_zero else float(0) + return { + "source_household_ess": _kish(household_weights), + "positive_contributors": int(np.count_nonzero(contributions > 0)), + "contribution_cv": cv, + "maximum_single_donor_dollar_share": maximum_share, + "dollar_contribution_ess": _kish(contributions), + "structural_zero": structural_zero, + } + + +def _support_thresholds(value: Mapping) -> dict: + if set(value) != _SUPPORT_FIELDS: + raise ValueError( + "Cell support must declare exactly its screen and target thresholds." + ) + return {name: _number(value[name], name, positive=True) for name in _SUPPORT_FIELDS} + + +def apply_cell_support(row: Mapping, diagnostics: Mapping, thresholds: Mapping) -> dict: + """Remove every reportable numeric value from an unsupported cell.""" + thresholds = _support_thresholds(thresholds) + reasons = [] + if diagnostics["structural_zero"]: + reasons.append("structural_zero") + else: + for field, threshold, lower in ( + ("source_household_ess", "minimum_cell_ess", True), + ("contribution_cv", "maximum_contribution_cv", False), + ( + "maximum_single_donor_dollar_share", + "maximum_single_donor_dollar_share", + False, + ), + ): + value = diagnostics[field] + if value is None or ( + value < thresholds[threshold] + if lower + else value > thresholds[threshold] + ): + reasons.append(field) + result = {**row, "suppressed": bool(reasons), "suppression_reasons": reasons} + if reasons: + result.update({metric: None for metric in _METRICS}) + return result + + +def _diagnostic_cell(row: Mapping, diagnostics: Mapping, thresholds: Mapping) -> dict: + screened = apply_cell_support(row, diagnostics, thresholds) + share = diagnostics["maximum_single_donor_dollar_share"] + return { + "name": row["name"], + "period": row["period"], + **diagnostics, + "suppressed": screened["suppressed"], + "meets_tracked_target": bool( + not diagnostics["structural_zero"] + and diagnostics["source_household_ess"] + >= thresholds["target_minimum_cell_ess"] + and share is not None + and share <= thresholds["target_maximum_single_donor_dollar_share"] + ), + } + + +def decode_cell_diagnostics(payload: bytes) -> dict: + """Validate scenario-complete source-household diagnostics bytes.""" + document = _document( + payload, + _CELL_DIAGNOSTICS_KIND, + {"weight_entity", "weight_kind", "source_household_column", "scenarios"}, + ) + for name in ("weight_entity", "weight_kind", "source_household_column"): + _text(document[name], f"cell diagnostics.{name}") + scenarios = document["scenarios"] + if not isinstance(scenarios, list) or not scenarios: + raise ValueError("Cell diagnostics need a non-empty list of scenarios.") + labels = [] + expected_cells = None + for scenario in scenarios: + if not isinstance(scenario, dict) or set(scenario) != {"scenario", "cells"}: + raise ValueError( + "Cell diagnostics scenario rows name exactly scenario and cells." + ) + labels.append(_text(scenario["scenario"], "cell diagnostics scenario")) + rows = scenario["cells"] + if not isinstance(rows, list) or not rows: + raise ValueError("Every diagnostics scenario needs its declared cells.") + cells = set() + for row in rows: + if not isinstance(row, dict) or set(row) != { + "name", + "period", + *_CELL_FIELDS, + "suppressed", + "meets_tracked_target", + }: + raise ValueError("Unsupported cell diagnostics row shape.") + cell = _cell(row) + if cell in cells: + raise ValueError("Cell diagnostics repeat a cell.") + cells.add(cell) + for flag in ("structural_zero", "suppressed", "meets_tracked_target"): + if not isinstance(row[flag], bool): + raise ValueError(f"Cell diagnostics {flag} must be boolean.") + count = row["positive_contributors"] + if isinstance(count, bool) or not isinstance(count, int) or count < 0: + raise ValueError( + "Cell diagnostics contributors must be a nonnegative count." + ) + for field in _CELL_FIELDS - {"structural_zero", "positive_contributors"}: + if row[field] is None and field in { + "contribution_cv", + "maximum_single_donor_dollar_share", + }: + continue + if _number(row[field], field) < 0: + raise ValueError("Cell diagnostics values must be nonnegative.") + if expected_cells is not None and cells != expected_cells: + raise ValueError( + "Cell diagnostics scenarios must cover identical table cells." + ) + expected_cells = cells + if labels != sorted(set(labels)): + raise ValueError("Cell diagnostics scenarios must be distinct and ordered.") + return document + + def _number(value: object, label: str, *, positive: bool = False) -> float: if isinstance(value, bool) or not isinstance(value, int | float): raise TypeError(f"{label} must be a finite number.") @@ -147,22 +377,50 @@ def decode_gap(payload: bytes) -> dict: raise ValueError("A gap artifact needs a non-empty list of table cells.") seen = set() for row in rows: - if not isinstance(row, dict) or set(row) != _ROW_KEYS: + optional = {"suppressed", "suppression_reasons", "reporting_kind"} + if not isinstance(row, dict) or set(row) - optional != _ROW_KEYS: raise ValueError("Unsupported gap row shape.") cell = _cell(row) if cell in seen: raise ValueError("A gap artifact repeats a table cell.") seen.add(cell) - for name in ("entity", "measure", "reference"): + entitlement = row.get("reporting_kind") == "entitlement" + if row.get("reporting_kind") not in (None, "entitlement"): + raise ValueError("Unsupported gap reporting kind.") + for name in ("entity", "measure"): _text(row[name], f"gap row.{name}") - if row["reference"] != row["name"]: + if not entitlement and row["reference"] != row["name"]: raise ValueError("A gap row must trace to its named reference.") + if entitlement and ( + row["reference"] is not None or row["ledger_fact_key"] is not None + ): + raise ValueError( + "Entitlement breakdowns cannot claim a subgroup comparator." + ) if row["filter"] is not None: _text(row["filter"], "gap row.filter") if not isinstance(row["metadata"], dict): raise ValueError("gap row.metadata must be an object.") _number(row["annualisation_factor"], "annualisation_factor", positive=True) + if "suppressed" in row: + reasons = row.get("suppression_reasons") + if not isinstance(row["suppressed"], bool) or not isinstance(reasons, list): + raise ValueError("Gap suppression needs a boolean and a reason list.") + if bool(reasons) != row["suppressed"]: + raise ValueError("Gap suppression reasons disagree with its flag.") + if row["suppressed"]: + if any(row[metric] is not None for metric in _METRICS): + raise ValueError( + "A suppressed cell cannot expose reportable values." + ) + continue estimate = _number(row["E"], "E") + if entitlement: + if row["A"] is not None or row["gap"] is not None: + raise ValueError( + "Entitlement breakdowns have no subgroup actual or gap." + ) + continue actual = _number(row["A"], "A") gap = _number(row["gap"], "gap") if gap != estimate - actual: @@ -203,7 +461,7 @@ def decode_bands(payload: bytes) -> dict: "Bands scenarios must be distinct, ordered and include the central scenario." ) rows = document["bands"] - if not isinstance(rows, list) or not rows: + if not isinstance(rows, list): raise ValueError("Bands need a non-empty list of table cells.") seen = set() for row in rows: @@ -244,6 +502,88 @@ def _require_output( ) +def reporting_cell_declarations(document: Mapping) -> tuple[dict, ...]: + """Expand a spec-declared table grid without deriving subgroup comparators.""" + if set(document) != { + "entity", + "measure", + "period", + "annualisation_factor", + "tables", + }: + raise ValueError( + "Reporting cells must declare their measure, period, factor and tables." + ) + entity = _text(document["entity"], "reporting entity") + measure = _text(document["measure"], "reporting measure") + factor = _number( + document["annualisation_factor"], "reporting factor", positive=True + ) + tables = document["tables"] + if not isinstance(tables, list) or not tables: + raise ValueError("Reporting cells need a non-empty declared table list.") + rows = [] + for table in tables: + if not isinstance(table, dict) or set(table) != {"name", "axes"}: + raise ValueError("Reporting tables declare exactly name and axes.") + table_name = _text(table["name"], "reporting table name") + axes = table["axes"] + if not isinstance(axes, list) or not axes: + raise ValueError("Reporting tables need non-empty axes.") + columns, domains = [], [] + for axis in axes: + if not isinstance(axis, dict) or set(axis) != {"column", "values"}: + raise ValueError("Reporting axes declare exactly column and values.") + columns.append(_text(axis["column"], "reporting axis column")) + values = axis["values"] + if not isinstance(values, list) or not values: + raise ValueError("Reporting axes need non-empty domains.") + if any(isinstance(value, list | dict) or value is None for value in values): + raise ValueError("Reporting axis values must be non-null scalars.") + if len({canonical_json(value) for value in values}) != len(values): + raise ValueError("Reporting axes cannot repeat values.") + domains.append(values) + if len(set(columns)) != len(columns): + raise ValueError("Reporting axes cannot repeat columns.") + for values in product(*domains): + selectors = dict(zip(columns, values, strict=True)) + rows.append( + { + "name": table_name + "/" + "/".join(str(value) for value in values), + "period": document["period"], + "entity": entity, + "measure": measure, + "filter": None, + "metadata": {"table": table_name, "selectors": selectors}, + "annualisation_factor": factor, + "reference": None, + "ledger_fact_key": None, + "reporting_kind": "entitlement", + } + ) + if len({_cell(row) for row in rows}) != len(rows): + raise ValueError("Reporting grids cannot repeat a table cell.") + return tuple(rows) + + +def _collapse_cell( + frame, entity: str, values: np.ndarray, weight_entity: str +) -> np.ndarray: + if entity == weight_entity: + return values + positions = target_kernels.matrix_module._nested_positions( + entity, frame, weight_entity + ) + collapsed = np.zeros(frame.n(weight_entity), dtype=np.float64) + np.add.at(collapsed, positions, values) + return collapsed + + +def _cell_support_from_reference(frame, target, weight_entity: str) -> np.ndarray: + mask = target._filter_mask(frame, frame.n(target.entity)) + return _collapse_cell(frame, target.entity, mask, weight_entity) > 0 + + class TakeupGapKernel(KernelBase): """``takeup.gap@1``: reference-driven annualised weighted entitlement.""" @@ -265,15 +605,28 @@ class TakeupGapKernel(KernelBase): "scenario", } ) - _optional = frozenset({"facts_sha256", "annualisation_sha256"}) + _optional = frozenset( + { + "facts_sha256", + "annualisation_sha256", + "source_household_column", + "cell_support", + "reporting_cells", + "expected_weight_kind", + } + ) def implementation_hash(self) -> str: return source_hash( type(self), graph_inputs, target_kernels, + weight_quality, *target_kernels._COMPILE_MODULES, *target_kernels._MATRIX_MODULES, + source_cell_diagnostics, + apply_cell_support, + reporting_cell_declarations, dependencies=self.capabilities.dependencies, ) @@ -281,7 +634,23 @@ def run(self, context: KernelContext) -> KernelResult: require_params( context, self.ref, required=self._required, optional=self._optional ) - _require_output(context, self.ref, "gap", GAP_TYPE) + diagnostics_enabled = "cell_support" in context.params + declared_outputs = { + item.name: item.type for item in context.node.artifact_outputs + } + expected_outputs = {"gap": GAP_TYPE} + if diagnostics_enabled: + expected_outputs["cell_diagnostics"] = CELL_DIAGNOSTICS_TYPE + if declared_outputs != expected_outputs: + raise ValueError( + f"{self.ref} must declare exactly its typed artifact outputs." + ) + if bool("source_household_column" in context.params) != diagnostics_enabled: + raise ValueError( + "Cell support needs its declared source-household lineage column." + ) + if "reporting_cells" in context.params and not diagnostics_enabled: + raise ValueError("Reported entitlement cells require support diagnostics.") country = string_param(context, self.ref, "country") scenario = string_param(context, self.ref, "scenario") document = canonical_document_param(context, self.ref, "references") @@ -322,6 +691,22 @@ def run(self, context: KernelContext) -> KernelResult: frame, registry, weight_entity, self.ref ) weights = frame.weights_for(weight_entity) + if diagnostics_enabled: + lineage_column = string_param(context, self.ref, "source_household_column") + if lineage_column not in frame.table(weight_entity): + raise ValueError( + f"Missing source-household lineage column {lineage_column!r}." + ) + expected_kind = string_param(context, self.ref, "expected_weight_kind") + if weights.kind.value != expected_kind: + raise ValueError( + f"Cell diagnostics require {expected_kind!r} household weights." + ) + identities = frame.table(weight_entity)[lineage_column].to_numpy() + thresholds = _support_thresholds( + canonical_document_param(context, self.ref, "cell_support") + ) + diagnostics_rows = [] rows = [] for index, (spec, reference, trace) in enumerate( zip(registry.specs, references, surface.trace, strict=True) @@ -333,22 +718,85 @@ def run(self, context: KernelContext) -> KernelResult: ) estimate = float(np.sum(weights.values * annualised)) actual = float(spec.value) - rows.append( - { - "name": spec.name, - "period": spec.period, - "entity": spec.entity, - "measure": spec.measure, - "filter": spec.filter, - "metadata": dict(reference.metadata), - "annualisation_factor": factor, - "reference": trace["reference"], - "ledger_fact_key": trace["ledger_fact_key"], - "E": estimate, - "A": actual, - "gap": estimate - actual, + row = { + "name": spec.name, + "period": spec.period, + "entity": spec.entity, + "measure": spec.measure, + "filter": spec.filter, + "metadata": dict(reference.metadata), + "annualisation_factor": factor, + "reference": trace["reference"], + "ledger_fact_key": trace["ledger_fact_key"], + "E": estimate, + "A": actual, + "gap": estimate - actual, + } + if diagnostics_enabled: + diagnostics = source_cell_diagnostics( + weights.values, + annualised, + identities, + support=_cell_support_from_reference( + frame, problem.targets[index], weight_entity + ), + ) + diagnostics_rows.append(_diagnostic_cell(row, diagnostics, thresholds)) + row = apply_cell_support(row, diagnostics, thresholds) + rows.append(row) + if "reporting_cells" in context.params: + reporting = canonical_document_param(context, self.ref, "reporting_cells") + declarations = reporting_cell_declarations(reporting) + table = frame.table(reporting["entity"]) + for table_definition in reporting["tables"]: + for axis in table_definition["axes"]: + column = axis["column"] + if column not in table: + raise ValueError( + f"Reported cell columns are missing: {[column]}." + ) + if not table[column].isin(axis["values"]).all(): + raise ValueError( + f"Reported cell axis {column!r} has missing or " + "undeclared values." + ) + for declaration in declarations: + table = frame.table(declaration["entity"]) + selectors = declaration["metadata"]["selectors"] + required_columns = {*selectors, declaration["measure"]} + missing = sorted(required_columns - set(table.columns)) + if missing: + raise ValueError(f"Reported cell columns are missing: {missing}.") + selected = np.ones(len(table), dtype=bool) + for column, value in selectors.items(): + selected &= table[column].to_numpy() == value + values = table[declaration["measure"]].to_numpy(dtype=np.float64) + annualised = _collapse_cell( + frame, + declaration["entity"], + values * selected * declaration["annualisation_factor"], + weight_entity, + ) + support = ( + _collapse_cell( + frame, + declaration["entity"], + selected.astype(np.float64), + weight_entity, + ) + > 0 + ) + row = { + **declaration, + "E": float(np.sum(weights.values * annualised)), + "A": None, + "gap": None, } - ) + diagnostics = source_cell_diagnostics( + weights.values, annualised, identities, support=support + ) + diagnostics_rows.append(_diagnostic_cell(row, diagnostics, thresholds)) + rows.append(apply_cell_support(row, diagnostics, thresholds)) payload = canonical_json( { "schema_version": 1, @@ -363,8 +811,22 @@ def run(self, context: KernelContext) -> KernelResult: } ) decode_gap(payload) + artifacts = {"gap": payload} + if diagnostics_enabled: + diagnostics_payload = canonical_json( + { + "schema_version": 1, + "kind": _CELL_DIAGNOSTICS_KIND, + "weight_entity": weight_entity, + "weight_kind": weights.kind.value, + "source_household_column": lineage_column, + "scenarios": [{"scenario": scenario, "cells": diagnostics_rows}], + } + ) + decode_cell_diagnostics(diagnostics_payload) + artifacts["cell_diagnostics"] = diagnostics_payload return KernelResult( - artifacts={"gap": payload}, + artifacts=artifacts, receipt={ "scenario": scenario, "n_cells": len(rows), @@ -446,7 +908,16 @@ def run(self, context: KernelContext) -> KernelResult: ordered_labels = sorted(documents) for cell in sorted(cells[baseline], key=lambda value: canonical_json(value)): central_row = cells[baseline][cell] - for metric in _METRICS: + if any( + cells[label][cell].get("suppressed", False) for label in ordered_labels + ): + continue + metrics = ( + ("E",) + if central_row.get("reporting_kind") == "entitlement" + else _METRICS + ) + for metric in metrics: low = min(ordered_labels, key=lambda label: cells[label][cell][metric]) high = min( ordered_labels, key=lambda label: -cells[label][cell][metric] @@ -487,12 +958,80 @@ def run(self, context: KernelContext) -> KernelResult: ) +class TakeupCellDiagnosticsKernel(KernelBase): + """Collect all declared scenarios without losing an absent scenario.""" + + ref = "takeup.cell_diagnostics@1" + capabilities = Capabilities( + Determinism.DETERMINISTIC, + numeric=Numeric.BITWISE, + seed_source=SeedSource.NONE, + ) + + def implementation_hash(self) -> str: + return source_hash(type(self), decode_cell_diagnostics, graph_inputs) + + def run(self, context: KernelContext) -> KernelResult: + require_params(context, self.ref, required=frozenset({"artifact_scenarios"})) + _require_output(context, self.ref, "cell_diagnostics", CELL_DIAGNOSTICS_TYPE) + aliases = canonical_document_param(context, self.ref, "artifact_scenarios") + labels = [_text(label, "diagnostics scenario") for label in aliases.values()] + if not labels or len(set(labels)) != len(labels): + raise ValueError("Cell diagnostics need distinct declared scenarios.") + if set(aliases) != set(context.artifacts): + raise ValueError( + "Cell diagnostics are missing a declared scenario artifact." + ) + documents = {} + for alias, label in aliases.items(): + value = context.artifacts[alias] + if value.type != CELL_DIAGNOSTICS_TYPE: + raise ValueError("Every scenario needs typed cell diagnostics.") + document = decode_cell_diagnostics(value.payload) + if ( + len(document["scenarios"]) != 1 + or document["scenarios"][0]["scenario"] != label + ): + raise ValueError( + "Cell diagnostics disagree with their declared scenario." + ) + documents[label] = document + first = documents[sorted(documents)[0]] + identity = ("weight_entity", "weight_kind", "source_household_column") + if any( + any(document[name] != first[name] for name in identity) + for document in documents.values() + ): + raise ValueError( + "Scenario cell diagnostics disagree on household weight lineage." + ) + payload = canonical_json( + { + "schema_version": 1, + "kind": _CELL_DIAGNOSTICS_KIND, + **{name: first[name] for name in identity}, + "scenarios": [ + documents[label]["scenarios"][0] for label in sorted(documents) + ], + } + ) + decode_cell_diagnostics(payload) + return KernelResult( + artifacts={"cell_diagnostics": payload}, + receipt={ + "n_scenarios": len(documents), + "cell_diagnostics_sha256": sha256_text(payload), + }, + ) + + TAKEUP_GAP = TakeupGapKernel() TAKEUP_BANDS = TakeupBandsKernel() +TAKEUP_CELL_DIAGNOSTICS = TakeupCellDiagnosticsKernel() def register_takeup_kernels(registry: KernelRegistry) -> None: - """Register the two kernels without modifying a global registry.""" + """Register the reporting kernels without modifying a global registry.""" - for kernel in (TAKEUP_GAP, TAKEUP_BANDS): + for kernel in (TAKEUP_GAP, TAKEUP_BANDS, TAKEUP_CELL_DIAGNOSTICS): registry.register(kernel) diff --git a/packages/microcosm-build/src/microcosm/build/transport/weight_quality.py b/packages/microcosm-build/src/microcosm/build/transport/weight_quality.py new file mode 100644 index 0000000000..270636fa6e --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/transport/weight_quality.py @@ -0,0 +1,49 @@ +"""Original-source aggregation for weight concentration diagnostics.""" + +from __future__ import annotations + +from collections.abc import Sequence + +import numpy as np +import pandas as pd + + +def aggregate_source_weights( + weights: Sequence[float] | np.ndarray, + source_household_ids: Sequence[object] | np.ndarray, +) -> np.ndarray: + """Sum clone weights by explicit source identity, preserving first-seen order. + + A unique identity vector returns the original vector without arithmetic, + so concentration summaries are exactly equal when there are no clones. + Empty, null, unhashable or misaligned identities fail closed. + """ + + values = np.asarray(weights, dtype=np.float64) + identities = np.asarray(source_household_ids, dtype=object) + if values.ndim != 1 or not values.size: + raise ValueError("Weights must be a non-empty vector.") + if not np.isfinite(values).all() or (values < 0).any(): + raise ValueError("Weights must be finite and non-negative.") + if identities.ndim != 1 or identities.size != values.size: + raise ValueError("Source-household lineage must align with the weight vector.") + if pd.isna(identities).any() or any( + isinstance(value, str) and not value.strip() for value in identities + ): + raise ValueError( + "Source-household lineage must not contain missing identities." + ) + try: + codes, unique = pd.factorize(identities, sort=False) + except TypeError as exc: + raise ValueError( + "Source-household identities must be hashable scalars." + ) from exc + if unique.size == values.size: + return values + sums = np.zeros(unique.size, dtype=np.float64) + with np.errstate(over="raise", invalid="raise"): + np.add.at(sums, codes, values) + if not np.isfinite(sums).all(): + raise ValueError("Aggregated source-household weights must be finite.") + return sums diff --git a/packages/microcosm-build/tests/engine_free/shared/test_nz_spec_package.py b/packages/microcosm-build/tests/engine_free/shared/test_nz_spec_package.py index 2ca37223fb..d448fa05fa 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_nz_spec_package.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_nz_spec_package.py @@ -1528,14 +1528,12 @@ def test_every_refusal_branch_names_the_broken_scenario( class TestDeclaredResources: - def test_threshold_gates_await_d1_and_carry_no_numeric_threshold(self) -> None: + def test_reference_tolerances_remain_deferred(self) -> None: spec = load_country_spec("nz") by_function = {gate.gate: gate for gate in spec.gates.gates} for function in ( "per_family_fit", "aggregate_admin", - "weight_ess", - "weight_ratio", ): assert by_function[function].not_applicable.startswith("awaiting D1") @@ -1552,9 +1550,58 @@ def numeric_leaves(value: object) -> Iterator[object]: assert not [ leaf for gate in spec.gates.gates + if gate.gate in {"per_family_fit", "aggregate_admin"} for leaf in numeric_leaves(gate.parameters) ] + def test_mc15_ruled_thresholds_and_tracked_targets_are_declared(self) -> None: + spec = load_country_spec("nz") + gates = {gate.id: gate for gate in spec.gates.gates} + ess = gates["weight_effective_sample_size"] + ratio = gates["weight_max_to_median_ratio"] + support = gates["reported_cell_support"] + target = gates["weight_effective_sample_size_target_a"] + cell_target = gates["reported_cell_support_target_a"] + assert ess.parameters["minimum_effective_sample_size"] == 4000 + assert "minimum_ess_fraction" not in ess.parameters + assert ratio.parameters["maximum_max_to_median_ratio"] == 5 + assert support.parameters["minimum_cell_ess"] == 100 + assert support.parameters["maximum_contribution_cv"] == 2 + assert support.parameters["maximum_single_donor_dollar_share"] == 0.1 + assert target.parameters["minimum_effective_sample_size"] == 16000 + assert cell_target.parameters["minimum_cell_ess"] == 400 + assert cell_target.parameters["maximum_single_donor_dollar_share"] == 0.05 + # The support contract's copy of target (a) drives the per-cell flags; + # it must match the tracked target gate that reports them. + assert support.parameters["target_minimum_cell_ess"] == 400 + assert support.parameters["target_maximum_single_donor_dollar_share"] == 0.05 + assert set(cell_target.parameters["required_cells"]) == set( + support.parameters["required_cells"] + ) + assert set(cell_target.parameters["required_scenarios"]) == set( + support.parameters["required_scenarios"] + ) + for gate in (ess, ratio, support, target, cell_target): + assert gate.not_applicable is None + assert gate.parameters["aggregation_column"] == "source_household_id" + assert gate.parameters["required_weight_entity"] == "household" + assert gate.parameters["required_weight_kind"] == "calibrated" + assert all( + g.criticality == "release_blocking" and g.evidence_absent_blocks + for g in (ess, ratio, support) + ) + assert target.criticality == cell_target.criticality == "diagnostic" + scenarios = _load("scenarios.json")["scenarios"] + assert support.parameters["required_scenarios"] == tuple( + s["id"] for s in scenarios + ) + from microcosm.build.transport.takeup_kernels import reporting_cell_declarations + + grid = _load("scenarios.json")["gap_conventions"]["reporting_cells"] + assert set(support.parameters["required_cells"]) == { + row["name"] for row in reporting_cell_declarations(grid) + } + def test_currency_bridge_is_one_declared_multiplication(self) -> None: bridge = _load("currency_bridge.json") assert (bridge["from_currency"], bridge["to_currency"]) == ("USD", "NZD") diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_composed_graph.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_composed_graph.py index 4aae6811f4..aa04829be0 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_transport_composed_graph.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_composed_graph.py @@ -53,6 +53,7 @@ from microcosm.build.transport import cli as cli_module from microcosm.build.transport import compose as compose_module +from microcosm.build.transport.artifact_types import COMPARISON_TYPE from microcosm.build.transport.cli import ( EXPORT_FILENAME, execute_transport_graph, @@ -70,6 +71,7 @@ from microcosm.build.transport.terminal_kernels import materialize_export from microcosm.frame.kernels import SimulateRulesKernel from microcosm.graph import ( + ArtifactInput, Capabilities, ContentStore, Determinism, @@ -1777,6 +1779,28 @@ def test_extension_checkpoints_are_returned_in_declaration_order(composed): compose_module.transport_endpoints(composed.spec, extensions=(duplicate,)) +@pytest.mark.parametrize("target", ["nz.export.prepare", "absent.node"]) +def test_extension_gate_evidence_cannot_target_other_skeleton_nodes(composed, target): + extension = compose_module.TransportExtension( + toy_extension, + gate_inputs=( + ( + target, + ArtifactInput( + "cell_diagnostics", + "nz.test.validate.compare", + "comparison", + COMPARISON_TYPE, + ), + ), + ), + ) + with pytest.raises(ValueError, match="only a skeleton battery"): + compose_transport_graph( + composed.spec, replace(composed.config, extensions=(extension,)) + ) + + @pytest.mark.parametrize("failure_type", [ImportError, NotImplementedError]) def test_cli_main_refuses_unavailable_rules_engine( composed, tmp_path, monkeypatch, capsys, failure_type diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_entitlement_graph.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_entitlement_graph.py index 8043e17a36..f74565130d 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_transport_entitlement_graph.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_entitlement_graph.py @@ -21,12 +21,19 @@ compose_transport_graph, validate_transport_activation, ) -from microcosm.build.transport.takeup_kernels import decode_bands, decode_gap +from microcosm.build.transport.takeup_kernels import ( + decode_bands, + decode_cell_diagnostics, + decode_gap, + reporting_cell_declarations, +) +from microcosm.frame.concepts import SOURCE_HOUSEHOLD_ID_COLUMN from microcosm.graph import compile_graph, graph_to_json from microcosm.graph.keys import node_key, source_content_key from test_support.microcosm_build.transport_entitlement import ( AS_INPUTS, AS_OUTPUTS, + EXPORT_FILENAME, TOY_DECLARATIONS, make_entitlement_fixture, run_entitlement_graph, @@ -99,6 +106,311 @@ def _declared_keys(fixture, graph, exported): return keys +def _cell_support_fixture(tmp_path): + fixture = make_entitlement_fixture(tmp_path / "fixture", variants=False) + spec = copy.deepcopy(fixture.spec) + resources = spec["resources"] + grid = { + "entity": "family", + "measure": "toy_as_weekly", + "period": "2026-27", + "annualisation_factor": 52, + "tables": [ + { + "name": "toy_type", + "axes": [ + { + "column": "family_type", + "values": [ + "single", + "couple", + "sole_parent", + "couple_with_children", + ], + } + ], + } + ], + } + resources["scenarios"]["gap_conventions"] = {"reporting_cells": grid} + resources["gates"]["gates"].append( + { + "id": "toy_cell_support", + "gate": "cell_support", + "phase": "terminal", + "criticality": "release_blocking", + "evidence_absent_blocks": True, + "parameters": { + "required_scenarios": [ + row["id"] for row in resources["scenarios"]["scenarios"] + ], + "required_cells": [ + row["name"] for row in reporting_cell_declarations(grid) + ], + # CREATE owns and carries the declared source-household concept. + "aggregation_column": SOURCE_HOUSEHOLD_ID_COLUMN, + "required_weight_entity": "household", + "required_weight_kind": "calibrated", + "minimum_cell_ess": 100, + "maximum_contribution_cv": 2, + "maximum_single_donor_dollar_share": 0.1, + "target_minimum_cell_ess": 400, + "target_maximum_single_donor_dollar_share": 0.05, + }, + } + ) + return fixture.reprepare(spec) + + +def test_cell_support_graph_enforces_scenario_complete_diagnostics_and_suppresses_cells( + tmp_path, +): + fixture = _cell_support_fixture(tmp_path) + graph = fixture.graph() + collector = "nz.as.bands.cell_diagnostics" + gate = graph.node("nz.gates.terminal") + assert any( + edge.name == "cell_diagnostics" and edge.producer == collector + for edge in gate.artifact_inputs + ) + manifest, store, exported = run_entitlement_graph( + tmp_path / "supported", fixture, graph=graph + ) + assert exported.exists() + diagnostics = decode_cell_diagnostics( + _artifact(manifest, store, collector, "cell_diagnostics") + ) + assert {row["scenario"] for row in diagnostics["scenarios"]} == { + row["id"] for row in fixture.spec["resources"]["scenarios"]["scenarios"] + } + assert all( + cell["suppressed"] + for scenario in diagnostics["scenarios"] + for cell in scenario["cells"] + ) + assert ( + decode_bands(_artifact(manifest, store, "nz.as.bands", "bands"))["bands"] == [] + ) + report = json.loads(_artifact(manifest, store, "nz.gates.terminal", "gate_report")) + assert report["enforcement"]["artifact_permitted"] is True + + +def test_cell_support_graph_adds_diagnostics_to_explicit_gap_output_declarations( + tmp_path, +): + fixture = _cell_support_fixture(tmp_path) + spec = copy.deepcopy(fixture.spec) + entitlement = spec["resources"]["entitlement_graph"] + node_groups = [ + entitlement.get(template, []) + for template in ("nodes", "scenario_nodes", "variant_nodes") + ] + node_groups.extend( + scenario.get("nodes", []) + for scenario in spec["resources"]["scenarios"]["scenarios"] + ) + for declared_nodes in node_groups: + for row in declared_nodes: + if row["kernel"] == "takeup.gap@1": + row["artifact_outputs"] = [ + { + "name": "gap", + "type": {"name": "microcosm.takeup.gap", "schema_version": 1}, + } + ] + graph = fixture.reprepare(spec).graph() + compile_graph(graph) + assert all( + {output.name for output in node.artifact_outputs} == {"gap", "cell_diagnostics"} + for node in graph.nodes + if node.kernel == "takeup.gap@1" + ) + + +def _with_cell_target(fixture, **overrides): + """Return the fixture's spec with a tracked cell-target gate appended.""" + spec = copy.deepcopy(fixture.spec) + gates = spec["resources"]["gates"]["gates"] + contract = next(g for g in gates if g["gate"] == "cell_support")["parameters"] + parameters = { + "required_scenarios": list(contract["required_scenarios"]), + "required_cells": list(contract["required_cells"]), + "aggregation_column": contract["aggregation_column"], + "required_weight_entity": "household", + "required_weight_kind": "calibrated", + "minimum_cell_ess": contract["target_minimum_cell_ess"], + "maximum_single_donor_dollar_share": contract[ + "target_maximum_single_donor_dollar_share" + ], + } + parameters.update(overrides) + gates.append( + { + "id": "toy_cell_support_target", + "gate": "cell_support_target", + "phase": "terminal", + "criticality": "diagnostic", + "parameters": parameters, + } + ) + return spec + + +def test_cell_support_target_matching_the_contract_builds(tmp_path): + fixture = _cell_support_fixture(tmp_path) + fixture.reprepare(_with_cell_target(fixture)).graph() + + +@pytest.mark.parametrize( + "overrides", + [ + {"minimum_cell_ess": 1_000_000}, + {"maximum_single_donor_dollar_share": 1}, + {"required_cells": ["toy_type/single"]}, + {"required_scenarios": ["S0"]}, + {"aggregation_column": "household_id"}, + ], + ids=["ess", "share", "cells", "scenarios", "aggregation"], +) +def test_cell_support_target_drift_from_the_contract_is_refused(tmp_path, overrides): + fixture = _cell_support_fixture(tmp_path) + with pytest.raises(ValueError, match="the two must agree"): + fixture.reprepare(_with_cell_target(fixture, **overrides)).graph() + + +def test_cell_support_contract_target_drift_is_refused(tmp_path): + # The reviewer's mutant: change only the contract's copy of the target. + fixture = _cell_support_fixture(tmp_path) + spec = _with_cell_target(fixture) + contract = next( + g for g in spec["resources"]["gates"]["gates"] if g["gate"] == "cell_support" + ) + contract["parameters"]["target_minimum_cell_ess"] = 1_000_000 + with pytest.raises(ValueError, match="the two must agree"): + fixture.reprepare(spec).graph() + + +def test_cell_support_target_lists_in_another_order_build(tmp_path): + fixture = _cell_support_fixture(tmp_path) + contract = _cell_support_contract(fixture.spec) + fixture.reprepare( + _with_cell_target( + fixture, + required_cells=list(reversed(contract["required_cells"])), + required_scenarios=list(reversed(contract["required_scenarios"])), + ) + ).graph() + + +def test_cell_support_target_with_null_not_applicable_is_checked(tmp_path): + # The gate parser treats "not_applicable": null as an active gate. + fixture = _cell_support_fixture(tmp_path) + spec = _with_cell_target(fixture, minimum_cell_ess=1_000_000) + spec["resources"]["gates"]["gates"][-1]["not_applicable"] = None + with pytest.raises(ValueError, match="the two must agree"): + fixture.reprepare(spec).graph() + + +@pytest.mark.parametrize( + "key", + [ + "minimum_cell_ess", + "maximum_single_donor_dollar_share", + "required_cells", + "required_scenarios", + "aggregation_column", + ], +) +def test_cell_support_target_omitting_a_shared_value_is_refused(tmp_path, key): + fixture = _cell_support_fixture(tmp_path) + spec = _with_cell_target(fixture) + del spec["resources"]["gates"]["gates"][-1]["parameters"][key] + with pytest.raises(ValueError, match="must both declare"): + fixture.reprepare(spec).graph() + + +@pytest.mark.parametrize("malformed", ["mixed", "mappings", "string"]) +def test_cell_support_target_malformed_lists_are_refused_by_name(tmp_path, malformed): + fixture = _cell_support_fixture(tmp_path) + cells = list(_cell_support_contract(fixture.spec)["required_cells"]) + overrides = { + "mixed": {"required_cells": [*cells, 1]}, + "mappings": {"required_cells": [{"name": cell} for cell in cells]}, + # The same characters as one cell name, as a list: equal only if a + # string were sorted like a list. + "string": {"required_cells": sorted(cells[0])}, + }[malformed] + if malformed == "string": + spec = _with_cell_target(fixture, **overrides) + _cell_support_contract(spec)["required_cells"] = cells[0] + else: + spec = _with_cell_target(fixture, **overrides) + with pytest.raises(ValueError, match="the two must agree"): + fixture.reprepare(spec).graph() + + +def test_cell_support_contract_with_null_not_applicable_is_the_contract(tmp_path): + fixture = _cell_support_fixture(tmp_path) + spec = copy.deepcopy(fixture.spec) + next(g for g in spec["resources"]["gates"]["gates"] if g["gate"] == "cell_support")[ + "not_applicable" + ] = None + nodes = {node.id: node.kernel for node in fixture.reprepare(spec).graph().nodes} + assert nodes == {node.id: node.kernel for node in fixture.graph().nodes} + assert "nz.as.bands.cell_diagnostics" in nodes + + +def _cell_support_contract(spec): + return next( + g for g in spec["resources"]["gates"]["gates"] if g["gate"] == "cell_support" + )["parameters"] + + +@pytest.mark.parametrize("absence", ["diagnostics", "scenario", "lineage"]) +def test_cell_support_graph_absence_prevents_export(tmp_path, absence): + fixture = _cell_support_fixture(tmp_path) + graph = fixture.graph() + if absence == "diagnostics": + gate = graph.node("nz.gates.terminal") + changed = replace( + gate, + artifact_inputs=tuple( + edge for edge in gate.artifact_inputs if edge.name != "cell_diagnostics" + ), + ) + graph = replace( + graph, + nodes=tuple( + changed if node.id == gate.id else node for node in graph.nodes + ), + ) + elif absence == "scenario": + collector = graph.node("nz.as.bands.cell_diagnostics") + changed = replace(collector, artifact_inputs=collector.artifact_inputs[:-1]) + graph = replace( + graph, + nodes=tuple( + changed if node.id == collector.id else node for node in graph.nodes + ), + ) + else: + changed_nodes = [] + for node in graph.nodes: + if node.kernel == "takeup.gap@1": + node = replace( + node, + params={ + **node.params, + "source_household_column": "missing_source_identity", + }, + ) + changed_nodes.append(node) + graph = replace(graph, nodes=tuple(changed_nodes)) + with pytest.raises((ValueError, RuntimeError), match="gate|scenario|lineage"): + run_entitlement_graph(tmp_path / "blocked", fixture, graph=graph) + assert not (tmp_path / "blocked" / EXPORT_FILENAME).exists() + + @EXECUTED @example(income=0, costs=1000, assets=0, beneficiary=False) @given( diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_gate_bindings.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_gate_bindings.py index 5b24952c1d..1cb0ba2f15 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_transport_gate_bindings.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_gate_bindings.py @@ -97,6 +97,7 @@ from microcosm.calibrate.registry import TargetSpec from microcosm.frame import Frame, WeightKind, Weights from microcosm.graph import ContentStore, compile_graph, run_graph +from microcosm.graph.canonical import canonical_json from test_support.microcosm_build.transport_graph import ( PERIOD, TOY_POPULATION, @@ -501,7 +502,10 @@ def _evaluate(gate: str, parameters: dict, evidence: dict, *, drop=()): class TestRegistry: def test_extends_the_default_registry_with_the_bound_gates(self) -> None: assert set(TRANSPORT_GATE_REGISTRY) == ( - set(DEFAULT_REGISTRY) | set(BOUND) | set(PENDING) + set(DEFAULT_REGISTRY) + | set(BOUND) + | set(PENDING) + | {"cell_support", "cell_support_target"} ) for name in DEFAULT_REGISTRY: if name not in BOUND: @@ -1341,6 +1345,63 @@ def _is_nearest_float(value: float, exact: Fraction) -> bool: class TestWeightGates: + @pytest.mark.parametrize( + ("gate", "parameters", "failures"), + [ + ("weight_ess", {"minimum_ess_fraction": 0.5}, ()), + ( + "weight_ess", + {"minimum_ess_fraction": 1.0}, + ("ESS fraction 0.833333 is below the reviewed minimum 1.",), + ), + ("weight_ratio", {"maximum_max_to_median_ratio": 1.0}, ()), + ( + "weight_ratio", + {"maximum_max_to_median_ratio": 0.5}, + ( + "Max/positive-median weight ratio 1.0 exceeds the reviewed " + "maximum 0.5.", + ), + ), + ], + ) + def test_legacy_row_gate_results_remain_byte_identical( + self, gate, parameters, failures + ) -> None: + # The existing row gates accept design weights with no lineage. Pin + # every recorded field and failure line, including zero-row semantics. + frame = toy_frame(weights=[0, 1, 1, 1, 1, 1], kind=WeightKind.DESIGN) + result = _evaluate(gate, parameters, {"frame": frame}).result + actual = canonical_json( + { + "name": result.name, + "passed": result.passed, + "failures": result.failures, + "details": dict(result.details), + } + ) + expected = canonical_json( + { + "name": gate, + "passed": not failures, + "failures": failures, + "details": { + "n_records": 6, + "positive_weight_records": 5, + "zero_weight_records": 1, + "total_weight": 5.0, + "effective_sample_size": 5.0, + "ess_fraction": 5 / 6, + "median_positive_weight": 1.0, + "max_weight": 1.0, + "max_to_median_positive_weight": 1.0, + **parameters, + "weight_entity": "household", + }, + } + ) + assert actual == expected + @pytest.mark.parametrize( ("weights", "gate", "keyword", "threshold", "summary_field", "expected"), [ diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_source_household_lineage.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_source_household_lineage.py new file mode 100644 index 0000000000..880e45bd21 --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_source_household_lineage.py @@ -0,0 +1,164 @@ +"""Explicit source lineage survives clone transport and benefit-unit creation.""" + +from __future__ import annotations + +import json + +import numpy as np +import pandas as pd +import pytest +from hypothesis import HealthCheck, given, settings +from hypothesis import strategies as st + +from microcosm.frame.concepts import ( + SOURCE_HOUSEHOLD_ID_COLUMN, + TransportRule, + Unit, + concept, + split_for_transport, +) +from microcosm.frame.transport import read_populace_us_donor +from microcosm.frame.unit_construction import build_benefit_units +from test_support.microcosm_build.transport_population import ( + UNIT_RULE, + donor_pin, + run_population, + write_population_sources, +) + +PROPERTY = settings( + max_examples=12, + deadline=None, + database=None, + suppress_health_check=[HealthCheck.function_scoped_fixture], +) + + +def _cloned_tables(tables): + """One original household has support copies in two different channels.""" + person = tables["person"] + clone = person.loc[person["person_household_id"] == 101].copy() + clone["person_id"] = clone["person_id"] + 100 + clone["person_household_id"] = 303 + clone["person_support_channel"] = "puf" + tables["person"] = pd.concat([person, clone], ignore_index=True) + household = tables["household"] + clone_household = household.loc[household["household_id"] == 101].copy() + clone_household["household_id"] = 303 + clone_household["household_support_channel"] = "puf" + tables["household"] = pd.concat([household, clone_household], ignore_index=True) + # A pool-scoped PH_SEQ is deliberately different from both raw lineage + # and support-row ids. It is not the source key the reader must use. + tables["person"]["PH_SEQ"] = tables["person"]["person_household_id"].map( + {101: 7001, 202: 7002, 303: 7001} + ) + + +def _read(sources): + return read_populace_us_donor(sources.donor, **donor_pin(sources.donor)) + + +def test_source_household_identity_is_a_declared_carry_concept(): + item = concept(f"fact:household.{SOURCE_HOUSEHOLD_ID_COLUMN}") + assert item.dtype == "str" + assert item.unit is Unit.SOURCE_ID + assert item.transport is TransportRule.CARRY + assert not item.nullable + + +@PROPERTY +@given(order=st.permutations(tuple(range(9))), reverse_households=st.booleans()) +def test_source_household_key_merges_cross_channel_clones_without_row_order( + tmp_path, order, reverse_households +): + def edit(tables): + _cloned_tables(tables) + if reverse_households: + tables["household"] = tables["household"].iloc[::-1].reset_index(drop=True) + + sources = write_population_sources( + tmp_path / "sources", edit_tables=edit, permutation=order + ) + donor = _read(sources) + households = donor.tables["household"].set_index("household_id") + keys = households[SOURCE_HOUSEHOLD_ID_COLUMN] + assert keys.loc[101] == keys.loc[303] + assert keys.loc[101] != keys.loc[202] + assert keys.nunique() == 2 + assert json.loads(keys.loc[101]) == [["int", "2024"], ["int", "101"]] + assert json.loads(keys.loc[202]) == [["int", "2024"], ["int", "202"]] + tables, dropped = split_for_transport(donor.tables) + assert SOURCE_HOUSEHOLD_ID_COLUMN not in dropped["household"] + pd.testing.assert_series_equal( + tables["household"][SOURCE_HOUSEHOLD_ID_COLUMN], + donor.tables["household"][SOURCE_HOUSEHOLD_ID_COLUMN], + ) + units, _ = build_benefit_units(tables["person"], tables["household"], UNIT_RULE) + expected = units[UNIT_RULE.household_column].map(keys) + np.testing.assert_array_equal(units[UNIT_RULE.source_household_column], expected) + + +def test_source_household_key_distinguishes_reused_raw_ids_in_another_vintage(tmp_path): + def edit(tables): + _cloned_tables(tables) + other_year = tables["person"]["person_household_id"] == 303 + tables["person"].loc[other_year, "source_year"] = 2023 + + sources = write_population_sources(tmp_path / "sources", edit_tables=edit) + keys = ( + _read(sources) + .tables["household"] + .set_index("household_id")[SOURCE_HOUSEHOLD_ID_COLUMN] + ) + assert keys.loc[101] != keys.loc[303] + assert keys.nunique() == 3 + + +@pytest.mark.parametrize("missing", ["source_year", "source_household_id"]) +def test_partial_source_household_lineage_is_refused(tmp_path, missing): + def edit(tables): + tables["person"] = tables["person"].drop(columns=missing) + + sources = write_population_sources(tmp_path / "sources", edit_tables=edit) + with pytest.raises(ValueError, match="source lineage requires year and household"): + _read(sources) + + +def test_inconsistent_source_household_lineage_is_refused(tmp_path): + def edit(tables): + tables["person"].loc[1, "source_household_id"] = 102 + + sources = write_population_sources(tmp_path / "sources", edit_tables=edit) + with pytest.raises( + ValueError, match="source household lineage disagrees within household" + ): + _read(sources) + + +def test_absent_source_household_lineage_never_falls_back_to_support_row_ids(tmp_path): + def edit(tables): + tables["person"] = tables["person"].drop( + columns=["source_year", "source_household_id"] + ) + + sources = write_population_sources(tmp_path / "sources", edit_tables=edit) + tables = _read(sources).tables + assert SOURCE_HOUSEHOLD_ID_COLUMN not in tables["household"] + units, _ = build_benefit_units(tables["person"], tables["household"], UNIT_RULE) + assert UNIT_RULE.source_household_column not in units + + +def test_create_graph_carries_source_household_keys_to_benefit_units(tmp_path): + sources = write_population_sources(tmp_path / "sources", edit_tables=_cloned_tables) + result = run_population(tmp_path / "graph", sources) + # Every transported unit inherits the household key by membership, + # including the distinct support rows belonging to the same original. + frame = result.store.load_frame(result.manifest.nodes["nz.open"].frame_key) + households = frame.table("household").set_index("household_id") + units = frame.table("family") + keys = households[SOURCE_HOUSEHOLD_ID_COLUMN] + assert keys.loc[101] == keys.loc[303] + np.testing.assert_array_equal( + units[UNIT_RULE.source_household_column], + units[UNIT_RULE.household_column].map(keys), + ) diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_takeup_kernels.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_takeup_kernels.py index 4cef3bbdc0..a910444333 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_transport_takeup_kernels.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_takeup_kernels.py @@ -20,16 +20,20 @@ from microcosm.build.transport.takeup_kernels import ( BANDS_TYPE, + CELL_DIAGNOSTICS_TYPE, GAP_TYPE, TAKEUP_BANDS, + TAKEUP_CELL_DIAGNOSTICS, TAKEUP_GAP, decode_bands, + decode_cell_diagnostics, decode_gap, register_takeup_kernels, + reporting_cell_declarations, ) from microcosm.build.transport.target_kernels import compile_target_surface from microcosm.calibrate.matrix import build_constraint_matrix -from microcosm.frame import Frame +from microcosm.frame import Frame, WeightKind, Weights from microcosm.graph import ( ArtifactOutput, ArtifactType, @@ -137,12 +141,15 @@ def _gap_node(*, scenario="S0", factors=None, references=None, **params) -> Node def _context(node, *, frame=None, facts=None, artifacts=None) -> KernelContext: + weight_entity = node.params.get("weight_entity", "family") return KernelContext( node=node, tables={} if frame is None else {entity: frame.table(entity).copy() for entity in ENTITIES}, - weights={} if frame is None else {"family": frame.resolve_weights("family")}, + weights={} + if frame is None + else {weight_entity: frame.resolve_weights(weight_entity)}, strata=pd.Series([], dtype=object, name="stratum") if frame is None else frame.strata.copy(), @@ -591,3 +598,190 @@ def test_annualisation_resource_digest_must_be_sha256(tmp_path) -> None: facts=_facts(tmp_path), ) ) + + +_SUPPORT_THRESHOLDS = { + "minimum_cell_ess": 1, + "maximum_contribution_cv": 2, + "maximum_single_donor_dollar_share": 1, + "target_minimum_cell_ess": 4, + "target_maximum_single_donor_dollar_share": 0.5, +} +_REPORTING_GRID = { + "entity": "family", + "measure": "family_support", + "period": 2026, + "annualisation_factor": 52, + "tables": [ + { + "name": "toy_entitlement", + "axes": [{"column": "income_decile", "values": [1, 2]}], + } + ], +} + + +def _diagnostic_frame(*, lineage=True) -> Frame: + original = _frame(values=[1] * 8, weights=[1] * 6) + tables = {entity: original.table(entity).copy(deep=True) for entity in ENTITIES} + if lineage: + tables["household"]["source_household_id"] = pd.array( + ["A", "A", "B", "B", "C", "C"], dtype="string" + ) + tables["family"]["income_decile"] = [1, 1, 1, 1, 2, 2, 2, 2] + return Frame( + tables, + original.schema, + {"household": Weights(np.ones(6), WeightKind.CALIBRATED)}, + original.strata.copy(), + ) + + +def _diagnostic_node(*, scenario="S0", thresholds=None, reporting=True) -> Node: + node = _gap_node( + scenario=scenario, + weight_entity="household", + source_household_column="source_household_id", + expected_weight_kind="calibrated", + cell_support=canonical_text( + _SUPPORT_THRESHOLDS if thresholds is None else thresholds + ), + **({"reporting_cells": canonical_text(_REPORTING_GRID)} if reporting else {}), + ) + return replace( + node, + artifact_outputs=( + ArtifactOutput("gap", GAP_TYPE), + ArtifactOutput("cell_diagnostics", CELL_DIAGNOSTICS_TYPE), + ), + ) + + +def _diagnostic_result(tmp_path, *, scenario="S0", thresholds=None, reporting=True): + return TAKEUP_GAP.run( + _context( + _diagnostic_node( + scenario=scenario, thresholds=thresholds, reporting=reporting + ), + frame=_diagnostic_frame(), + facts=_facts(tmp_path), + ) + ) + + +def test_cell_diagnostics_merge_clones_and_do_not_duplicate_household_unit_support( + tmp_path, +) -> None: + result = _diagnostic_result(tmp_path, reporting=False) + document = decode_cell_diagnostics(result.artifacts["cell_diagnostics"]) + cells = document["scenarios"][0]["cells"] + assert document["weight_kind"] == "calibrated" + assert cells[1]["source_household_ess"] == 3 + assert cells[1]["positive_contributors"] == 3 + assert cells[1]["dollar_contribution_ess"] <= 3 + assert all(not cell["meets_tracked_target"] for cell in cells) + + +def test_failing_cells_are_suppressed_in_gap_and_never_enter_scenario_bands( + tmp_path, +) -> None: + thresholds = {**_SUPPORT_THRESHOLDS, "minimum_cell_ess": 100} + result = _diagnostic_result(tmp_path, thresholds=thresholds) + rows = decode_gap(result.artifacts["gap"])["gaps"] + assert all(row["suppressed"] for row in rows) + assert all(row[metric] is None for row in rows for metric in ("E", "A", "gap")) + cells = decode_cell_diagnostics(result.artifacts["cell_diagnostics"])["scenarios"][ + 0 + ]["cells"] + assert all(cell["suppressed"] for cell in cells) + assert decode_bands(_bands([result.artifacts["gap"]]))["bands"] == [] + + +def test_entitlement_grid_has_complete_cells_without_invented_subgroup_comparators( + tmp_path, +) -> None: + result = _diagnostic_result(tmp_path) + rows = [ + row + for row in decode_gap(result.artifacts["gap"])["gaps"] + if row.get("reporting_kind") == "entitlement" + ] + assert {row["name"] for row in rows} == { + row["name"] for row in reporting_cell_declarations(_REPORTING_GRID) + } + assert all( + row["A"] is None and row["gap"] is None and row["reference"] is None + for row in rows + ) + assert all(row["E"] is not None and not row["suppressed"] for row in rows) + bands = decode_bands(_bands([result.artifacts["gap"]]))["bands"] + assert { + row["metric"] for row in bands if row["name"].startswith("toy_entitlement/") + } == {"E"} + + +def test_cell_diagnostics_require_explicit_lineage_and_calibrated_weights( + tmp_path, +) -> None: + with pytest.raises(ValueError, match="Missing source-household lineage"): + TAKEUP_GAP.run( + _context( + _diagnostic_node(), + frame=_diagnostic_frame(lineage=False), + facts=_facts(tmp_path), + ) + ) + frame = _diagnostic_frame() + uncalibrated = Frame( + {entity: frame.table(entity) for entity in ENTITIES}, + frame.schema, + {"household": Weights(np.ones(6), WeightKind.DESIGN)}, + frame.strata, + ) + with pytest.raises(ValueError, match="require.*calibrated"): + TAKEUP_GAP.run( + _context(_diagnostic_node(), frame=uncalibrated, facts=_facts(tmp_path)) + ) + + +@pytest.mark.parametrize("value", [None, 3]) +def test_reporting_grid_refuses_unclassified_units_instead_of_omitting_them( + tmp_path, value +) -> None: + original = _diagnostic_frame() + tables = {entity: original.table(entity).copy(deep=True) for entity in ENTITIES} + tables["family"].loc[0, "income_decile"] = value + frame = Frame( + tables, + original.schema, + {"household": original.weights_for("household")}, + original.strata, + ) + with pytest.raises(ValueError, match="axis 'income_decile'.*undeclared values"): + TAKEUP_GAP.run( + _context(_diagnostic_node(), frame=frame, facts=_facts(tmp_path)) + ) + + +def test_cell_diagnostics_collector_requires_every_declared_scenario(tmp_path) -> None: + results = [_diagnostic_result(tmp_path, scenario=label) for label in ("S0", "S1")] + node = Node( + "diagnostics", + TAKEUP_CELL_DIAGNOSTICS.ref, + population="open", + params={ + "artifact_scenarios": canonical_text({"central": "S0", "alternate": "S1"}) + }, + artifact_outputs=(ArtifactOutput("cell_diagnostics", CELL_DIAGNOSTICS_TYPE),), + ) + artifacts = { + name: _value(result.artifacts["cell_diagnostics"], CELL_DIAGNOSTICS_TYPE) + for name, result in zip(("central", "alternate"), results, strict=True) + } + with pytest.raises(ValueError, match="missing a declared scenario"): + TAKEUP_CELL_DIAGNOSTICS.run( + _context(node, artifacts={"central": artifacts["central"]}) + ) + result = TAKEUP_CELL_DIAGNOSTICS.run(_context(node, artifacts=artifacts)) + document = decode_cell_diagnostics(result.artifacts["cell_diagnostics"]) + assert [row["scenario"] for row in document["scenarios"]] == ["S0", "S1"] diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_weight_quality.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_weight_quality.py new file mode 100644 index 0000000000..9af18e8e91 --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_weight_quality.py @@ -0,0 +1,519 @@ +"""Original-source weight quality and cell screens over synthetic donors.""" + +from __future__ import annotations + +import math +from dataclasses import replace + +import numpy as np +import pandas as pd +import pytest +from hypothesis import given, settings +from hypothesis import strategies as st + +from microcosm.build.country_spec import GatesManifest, load_country_spec +from microcosm.build.gate_battery import EvidenceContext, GateStatus, evaluate_phase +from microcosm.build.transport.gate_bindings import ( + TRANSPORT_GATE_REGISTRY, + weight_ess_gate, + weight_ratio_gate, + weight_summary, +) +from microcosm.build.transport.takeup_kernels import ( + apply_cell_support, + source_cell_diagnostics, +) +from microcosm.frame import EntitySchema, Frame, WeightKind, Weights + +PROPERTY = settings(max_examples=60, deadline=None, database=None) +_WEIGHTS = st.lists(st.integers(min_value=1, max_value=1000), min_size=2, max_size=24) +_SCALE = st.one_of( + st.floats(min_value=0.01, max_value=100, allow_nan=False, allow_infinity=False), + st.integers(min_value=-20, max_value=20).map( + lambda exponent: math.ldexp(1, exponent) + ), +) +_THRESHOLDS = { + "minimum_cell_ess": 100, + "maximum_contribution_cv": 2, + "maximum_single_donor_dollar_share": 0.1, + "target_minimum_cell_ess": 400, + "target_maximum_single_donor_dollar_share": 0.05, +} + + +def _frame(weights, ids=None, *, kind=WeightKind.CALIBRATED, entity="household"): + size = len(weights) + tables = { + "person": pd.DataFrame( + {"person_id": range(size), "person_household_id": range(size)} + ), + "household": pd.DataFrame({"household_id": range(size)}), + } + if ids is not None: + tables[entity]["source_household_id"] = ids + return Frame( + tables, + EntitySchema(person_entity="person", group_entities=("household",)), + {entity: Weights(np.asarray(weights, dtype=np.float64), kind)}, + ) + + +def _report(gate, parameters, *, frame=None, artifacts=None): + manifest = GatesManifest.from_mapping( + { + "country": "xx", + "version": 1, + "policy": "Synthetic weight-quality gate fixtures.", + "phases": ["terminal"], + "gates": [ + { + "id": "quality", + "gate": gate, + "phase": "terminal", + "criticality": "release_blocking", + "parameters": parameters, + "evidence_absent_blocks": True, + } + ], + } + ) + return evaluate_phase( + manifest, + "terminal", + EvidenceContext(frame=frame, artifacts=artifacts or {}), + registry=TRANSPORT_GATE_REGISTRY, + ) + + +def _source_parameters(**extra): + return { + "aggregation_column": "source_household_id", + "required_weight_entity": "household", + "required_weight_kind": "calibrated", + **extra, + } + + +def _cell_parameters(): + return _source_parameters( + required_scenarios=["S0", "S1"], + required_cells=["decile_1_single"], + **_THRESHOLDS, + ) + + +def _cell_artifact(*, scenarios=("S0", "S1"), suppressed=True): + diagnostics = source_cell_diagnostics([1, 1], [10, 1], [10, 20]) + return { + "schema_version": 1, + "kind": "transport_cell_support_diagnostics", + "weight_entity": "household", + "weight_kind": "calibrated", + "source_household_column": "source_household_id", + "scenarios": [ + { + "scenario": scenario, + "cells": [ + { + "name": "decile_1_single", + "period": 2026, + **diagnostics, + "suppressed": suppressed, + "meets_tracked_target": False, + } + ], + } + for scenario in scenarios + ], + } + + +@PROPERTY +@given(weights=_WEIGHTS) +def test_property_unique_sources_equal_row_ess_and_ratio_exactly(weights): + row = weight_summary(weights) + source = weight_summary(weights, source_household_ids=list(range(len(weights)))) + assert source["effective_sample_size"] == row["effective_sample_size"] + assert ( + source["max_to_median_positive_weight"] == row["max_to_median_positive_weight"] + ) + + +@PROPERTY +@given(weights=_WEIGHTS) +def test_property_merging_clones_never_increases_ess(weights): + row = weight_summary(weights) + source = weight_summary( + weights, source_household_ids=[index // 2 for index in range(len(weights))] + ) + assert source["effective_sample_size"] <= row["effective_sample_size"] + 1e-12 + # Every weight is positive and at least one pair merges, so independence + # drops strictly; using row weights cannot satisfy this stronger check. + assert source["effective_sample_size"] < row["effective_sample_size"] + + +@PROPERTY +@given(weights=_WEIGHTS, scale=_SCALE) +def test_property_source_weight_ess_and_ratio_are_scale_invariant(weights, scale): + ids = [index // 2 for index in range(len(weights))] + original = weight_summary(weights, source_household_ids=ids) + scaled = weight_summary( + [weight * scale for weight in weights], + source_household_ids=ids, + ) + assert scaled["effective_sample_size"] == pytest.approx( + original["effective_sample_size"], rel=1e-12 + ) + assert scaled["max_to_median_positive_weight"] == pytest.approx( + original["max_to_median_positive_weight"], rel=1e-12 + ) + + +def test_clones_cannot_pass_an_absolute_source_household_floor(): + weights = [1, 1, 1, 1, 1, 1] + ids = ["original-c", "original-a", "original-b"] * 2 + assert weight_ess_gate(weights, minimum_effective_sample_size=4).passed + report = _report( + "weight_ess", + _source_parameters(minimum_effective_sample_size=4), + frame=_frame(weights, ids), + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert outcome.result.details["effective_sample_size"] == 3 + assert report.blocking_outcomes(release_candidate=False) + + +def test_absolute_floor_does_not_change_when_row_count_changes(): + for count in (2, 3, 4, 7): + result = weight_ess_gate( + [1] * count, + minimum_ess_fraction=0.5, + minimum_effective_sample_size=4, + ) + assert result.passed is (count >= 4) + assert result.details["minimum_effective_sample_size"] == 4 + + +def test_ratio_uses_aggregated_source_weights(): + weights = [1, 1, 1, 1, 1, 4] + assert not weight_ratio_gate(weights, maximum_max_to_median_ratio=3).passed + report = _report( + "weight_ratio", + _source_parameters(maximum_max_to_median_ratio=3), + frame=_frame(weights, [10, 10, 20, 20, 30, 30]), + ) + assert report.outcomes[0].status is GateStatus.PASSED + assert report.outcomes[0].result.details["max_to_median_positive_weight"] == 2.5 + + +@pytest.mark.parametrize( + ("gate", "threshold"), + [ + ("weight_ess", {"minimum_effective_sample_size": 1}), + ("weight_ratio", {"maximum_max_to_median_ratio": 5}), + ], +) +def test_source_weight_gate_missing_lineage_blocks(gate, threshold): + report = _report(gate, _source_parameters(**threshold), frame=_frame([1, 1])) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert "source_household_id" in " ".join(outcome.result.failures) + assert report.blocking_outcomes(release_candidate=False) + + +@pytest.mark.parametrize( + ("gate", "threshold"), + [ + ("weight_ess", {"minimum_effective_sample_size": 1}), + ("weight_ratio", {"maximum_max_to_median_ratio": 5}), + ], +) +@pytest.mark.parametrize( + ("kind", "entity", "reason"), + [ + (WeightKind.DESIGN, "household", "calibrated"), + (WeightKind.CALIBRATED, "person", "household"), + ], +) +def test_source_weight_gates_refuse_wrong_weight_provenance( + gate, threshold, kind, entity, reason +): + report = _report( + gate, + _source_parameters(**threshold), + frame=_frame([1, 1], [10, 20], kind=kind, entity=entity), + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert reason in " ".join(outcome.result.failures) + assert report.blocking_outcomes(release_candidate=False) + + +@PROPERTY +@given( + weights=_WEIGHTS, + scale=_SCALE, + data=st.data(), +) +def test_property_every_cell_diagnostic_is_scale_invariant(weights, scale, data): + amounts = data.draw( + st.lists( + st.integers(min_value=0, max_value=1000), + min_size=len(weights), + max_size=len(weights), + ) + ) + ids = [index // 2 for index in range(len(weights))] + original = source_cell_diagnostics(weights, amounts, ids) + scaled = source_cell_diagnostics( + [weight * scale for weight in weights], amounts, ids + ) + assert scaled.keys() == original.keys() + for field in original: + if isinstance(original[field], float): + assert scaled[field] == pytest.approx(original[field], rel=1e-12, abs=1e-12) + else: + assert scaled[field] == original[field] + + +@PROPERTY +@given( + weights=st.lists(st.integers(min_value=1, max_value=100), min_size=4, max_size=20), + increase=st.integers(min_value=2, max_value=100), + data=st.data(), +) +def test_property_increasing_a_donor_weight_never_decreases_its_dollar_share( + weights, increase, data +): + ids = np.arange(len(weights)) // 2 + source_count = int(ids.max()) + 1 + donor = data.draw(st.integers(min_value=0, max_value=source_count - 1)) + amounts = np.asarray( + data.draw( + st.lists( + st.integers(min_value=1, max_value=1000), + min_size=len(weights), + max_size=len(weights), + ) + ) + ) + weights = np.asarray(weights) + before = source_cell_diagnostics(weights, amounts, ids) + enlarged = weights.copy() + enlarged[ids == donor] *= increase + after = source_cell_diagnostics(enlarged, amounts, ids) + contributions = np.zeros(source_count) + larger_contributions = np.zeros(source_count) + np.add.at(contributions, ids, weights * amounts) + np.add.at(larger_contributions, ids, enlarged * amounts) + shares = contributions / contributions.sum() + larger_shares = larger_contributions / larger_contributions.sum() + assert larger_shares[donor] >= shares[donor] + assert before["maximum_single_donor_dollar_share"] == pytest.approx( + float(shares.max()), rel=1e-12 + ) + assert after["maximum_single_donor_dollar_share"] == pytest.approx( + float(larger_shares.max()), rel=1e-12 + ) + + +def test_cell_diagnostics_merge_clone_dollars_and_distinct_contributors(): + diagnostics = source_cell_diagnostics([1, 1, 1, 1], [2, 3, 0, 5], [10, 10, 20, 30]) + assert diagnostics["source_household_ess"] == pytest.approx(8 / 3) + assert diagnostics["positive_contributors"] == 2 + assert diagnostics["maximum_single_donor_dollar_share"] == 0.5 + assert diagnostics["dollar_contribution_ess"] == 2 + assert not diagnostics["structural_zero"] + + +def test_structural_zero_cell_is_identified_without_positive_contributors(): + diagnostics = source_cell_diagnostics([1, 1], [0, 0], [10, 20]) + assert diagnostics["positive_contributors"] == 0 + assert diagnostics["structural_zero"] + assert diagnostics["dollar_contribution_ess"] == 0 + + +def test_empty_declared_cell_is_a_structural_zero(): + diagnostics = source_cell_diagnostics( + [1, 1], [10, 20], [10, 20], support=[False, False] + ) + assert diagnostics == { + "source_household_ess": 0, + "positive_contributors": 0, + "contribution_cv": 0, + "maximum_single_donor_dollar_share": 0, + "dollar_contribution_ess": 0, + "structural_zero": True, + } + + +def test_failing_cell_is_suppressed_before_reporting(): + diagnostics = source_cell_diagnostics([1, 1], [10, 1], [10, 20]) + original = {"E": 11, "A": 4, "gap": 7} + screened = apply_cell_support(original, diagnostics, _THRESHOLDS) + assert screened["suppressed"] + assert screened["suppression_reasons"] + assert screened["E"] is None + assert screened["A"] is None + assert screened["gap"] is None + assert original == {"E": 11, "A": 4, "gap": 7} + + +def test_supported_cell_retains_its_entitlement_and_tracks_tighter_target(): + diagnostics = source_cell_diagnostics([1] * 500, [10] * 500, list(range(500))) + original = {"E": 5000, "A": 4000, "gap": 1000} + screened = apply_cell_support(original, diagnostics, _THRESHOLDS) + assert not screened["suppressed"] + assert screened["E"] == 5000 + assert screened["gap"] == 1000 + + +def test_missing_cell_diagnostics_artifact_blocks_every_build_posture(): + report = _report("cell_support", _cell_parameters(), artifacts={}) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.EVIDENCE_ABSENT + assert "cell_diagnostics" in outcome.reason + assert report.blocking_outcomes(release_candidate=False) + assert report.blocking_outcomes(release_candidate=True) + + +def test_missing_declared_scenario_in_cell_diagnostics_blocks(): + report = _report( + "cell_support", + _cell_parameters(), + artifacts={"cell_diagnostics": _cell_artifact(scenarios=("S0",))}, + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert "S1" in " ".join(outcome.result.failures) + assert report.blocking_outcomes(release_candidate=False) + + +def test_suppressed_failing_cells_are_diagnostic_and_do_not_block(): + report = _report( + "cell_support", + _cell_parameters(), + artifacts={"cell_diagnostics": _cell_artifact()}, + ) + assert report.outcomes[0].status is GateStatus.PASSED + assert report.outcomes[0].result.details["suppressed_cells"] == [ + "S0/decile_1_single", + "S1/decile_1_single", + ] + assert not report.blocking_outcomes(release_candidate=True) + + +def test_unsuppressed_failing_cell_blocks_release(): + report = _report( + "cell_support", + _cell_parameters(), + artifacts={"cell_diagnostics": _cell_artifact(suppressed=False)}, + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert "not suppressed" in " ".join(outcome.result.failures) + assert report.blocking_outcomes(release_candidate=True) + + +def test_missing_declared_cell_in_diagnostics_blocks(): + parameters = _cell_parameters() + parameters["required_cells"] = ["decile_1_single", "decile_2_single"] + report = _report( + "cell_support", + parameters, + artifacts={"cell_diagnostics": _cell_artifact()}, + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert "decile_2_single" in " ".join(outcome.result.failures) + assert report.blocking_outcomes(release_candidate=False) + + +@pytest.mark.parametrize( + ("key", "incorrect", "reason"), + [ + ("weight_entity", "person", "weight_entity"), + ("weight_kind", "design", "weight_kind"), + ("source_household_column", "row_id", "source_household_column"), + ], +) +def test_cell_diagnostics_wrong_provenance_blocks(key, incorrect, reason): + artifact = _cell_artifact() + artifact[key] = incorrect + report = _report( + "cell_support", _cell_parameters(), artifacts={"cell_diagnostics": artifact} + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert reason in " ".join(outcome.result.failures) + assert report.blocking_outcomes(release_candidate=False) + + +def test_adopted_global_target_is_recorded_and_never_blocks(): + gates = load_country_spec("nz").gates + entry = next( + row for row in gates.gates if row.id == "weight_effective_sample_size_target_a" + ) + report = evaluate_phase( + replace(gates, gates=(entry,)), + "terminal", + EvidenceContext(frame=_frame([1] * 5, list(range(5)))), + registry=TRANSPORT_GATE_REGISTRY, + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert outcome.result.details["effective_sample_size"] == 5 + assert outcome.result.details["minimum_effective_sample_size"] == 16000 + assert not report.blocking_outcomes(release_candidate=True) + + +def test_adopted_cell_targets_are_flagged_and_never_block(): + gates = load_country_spec("nz").gates + entry = next( + row for row in gates.gates if row.id == "reported_cell_support_target_a" + ) + artifact = _cell_artifact(scenarios=entry.parameters["required_scenarios"]) + diagnostics = source_cell_diagnostics([1] * 5, [10] * 5, list(range(5))) + for scenario in artifact["scenarios"]: + scenario["cells"] = [ + { + "name": name, + "period": 2026, + **diagnostics, + "suppressed": True, + "meets_tracked_target": False, + } + for name in entry.parameters["required_cells"] + ] + report = evaluate_phase( + replace(gates, gates=(entry,)), + "terminal", + EvidenceContext(artifacts={"cell_diagnostics": artifact}), + registry=TRANSPORT_GATE_REGISTRY, + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + flags = outcome.result.details["meets_tracked_target"] + assert len(flags) == 43 * 9 + assert not any(flags.values()) + assert not report.blocking_outcomes(release_candidate=True) + + +def test_adopted_four_thousand_floor_rejects_cloned_row_support(): + gates = load_country_spec("nz").gates + entry = next(row for row in gates.gates if row.id == "weight_effective_sample_size") + weights = np.ones(6000) + assert weight_ess_gate(weights, minimum_effective_sample_size=4000).passed + report = evaluate_phase( + replace(gates, gates=(entry,)), + "terminal", + EvidenceContext(frame=_frame(weights, np.tile(np.arange(3000), 2))), + registry=TRANSPORT_GATE_REGISTRY, + ) + outcome = report.outcomes[0] + assert outcome.status is GateStatus.FAILED + assert outcome.result.details["effective_sample_size"] == 3000 + assert outcome.result.details["minimum_effective_sample_size"] == 4000 + assert report.blocking_outcomes(release_candidate=True) diff --git a/packages/microcosm-build/tests/golden/nz_country_spec.json b/packages/microcosm-build/tests/golden/nz_country_spec.json index 49b236568e..0a9c751b51 100644 --- a/packages/microcosm-build/tests/golden/nz_country_spec.json +++ b/packages/microcosm-build/tests/golden/nz_country_spec.json @@ -1,6 +1,6 @@ { "country": "nz", - "fingerprint": "09ed486fea7b25bcdf4ba3c9ee9ddc971b0b57c7fa6076308b61437273415677", + "fingerprint": "7c7064be95a241a4757246ff9de728ad75c2501c162ae2232214c39414cba61d", "gate_ids": [ "calibration_per_family_fit", "national_and_region_admin_aggregates", @@ -14,6 +14,9 @@ "nonnegative_columns", "weight_effective_sample_size", "weight_max_to_median_ratio", + "reported_cell_support", + "weight_effective_sample_size_target_a", + "reported_cell_support_target_a", "macro_realism_nz" ], "geography_spine_stage": "assign_nz_regions", @@ -38,21 +41,21 @@ "as_rate_bridge.json": "5eea897159508fb71820f22cd8ccf4993075a750462bd97312d9e89ab01cba15", "axiom_input_closure.json": "c3c164f437b05d598af7e604d2cb46365a8323a8043da49b08c5e0cdf32e2ec1", "axiom_rules_bindings.json": "72905af77132ff6a3c01ada4b6210be4c369d9c6dc49dc8c1b673e4bac0b5fb1", - "benefit_unit_rule.json": "9e507e78f180566a1eece74da2a35d88478dbe13487c3a64ee2b76c9ecc7a653", + "benefit_unit_rule.json": "b4ed95161177668964ffcc39556c14600e2e08bda350be9ea5f52e83cc3afd41", "country_package.json": "6c4c1d19a0666958a4dc1079e5363084a347e817fc0c9cca59eada45bdc18873", "currency_bridge.json": "aa6a4b286a6a1a015088f32ac013148553422c3d45eeef7d7a15af08388857ca", - "export_contract.json": "f0629a3990a47f3f94114c4d7b6680636ef7b1bd34bb8e950976d3065f029fda", - "gates.json": "c56f55deb8c1bdf0755a6223e871653e66a350a7ac78803f85bed5b47be88791", + "export_contract.json": "2c7414fefe395c4f5822643dcd3bb3544ebb2040259aa007b828a99bd01ae210", + "gates.json": "18a78dde49b6b979bf46039fa1d779bf30362b853514739e66bd669c2aaebeba", "geography_spine.json": "353821844d85d2622c328b4986dd9e3a8075c13498b9f56d349c4578edf7495a", "holdout_references.json": "52b5a2360502156da1392a68350e4e7f61b5fd420fa74428819b65886a9f8655", "precal_references.json": "43b13849e3d74ebca7d7a9d46817463e3ad1333c3aedd7a3eac39e870510fba0", "release_contract.json": "6cd5e3a4a2bf30e4025e04e964c393b24040937d13db7e6a07b3966a55f3b114", - "scenarios.json": "5dbf2b9603fa55fd9abd7319e188f882ffe471e99e7d691b81980522a9bef82f", - "source_stages.json": "cc1eef0769c79917f33c5ba0f99edf861070143168db48e0145b71efeee751d1", + "scenarios.json": "30f96f3063773d77e3c70119c8f5d99d3528eca87428903da00ca071ab34cfaf", + "source_stages.json": "cc82c68878ac0fc07ecbd796c1a66de6f34786ef8785125046425c37f26aff31", "spec/bundle.yaml": "25ef74ecbc00c3f2ecca2d30ddd5e46a582657863c217da33ee9a396569d2cf2", "spec/catalogs.yaml": "3ab2025af39cba93ead707b9463e428430a45b30696a43d0906cbcd97ed36631", "spec/geography.yaml": "0dbf9f850dc3cef983f166bf6783022094894ec2461ad71f4809607bcfd390a9", - "spec/sources.yaml": "6ce8861328eff3b70b787570d1150b0c636269ba4c4f8583052b9ed3ad9ed523", + "spec/sources.yaml": "3d801b0a4171edd6a9083048e1d3c8ee8c1a74d3c4cee25c64ae467e7535c7eb", "spec/spine.yaml": "b3e6a2832ff7671777cae113b7fc45f5e3492bb519526f229118d0851083533e", "spec/vintages.yaml": "1ecc1a3f952bd25b834c394314e8c933ddb5b095fd64ba492dddb03e3cdd803e", "target_references.json": "5cc916bc477e250f562469aaf2d144693fa3aa2a7824cc11096ad35b7c13b543" diff --git a/packages/microcosm-frame/src/microcosm/frame/adapters/axiom_concept_mappings/be.json b/packages/microcosm-frame/src/microcosm/frame/adapters/axiom_concept_mappings/be.json index 77d561bc2e..786b741497 100644 --- a/packages/microcosm-frame/src/microcosm/frame/adapters/axiom_concept_mappings/be.json +++ b/packages/microcosm-frame/src/microcosm/frame/adapters/axiom_concept_mappings/be.json @@ -1779,6 +1779,7 @@ "structural_inputs": [], "unmapped": { "fact:household.mortgage_principal": "rulespec-be at b105e2b3 has no input for mortgage capital repayments. The only housing-loan inputs are the interest deductions belgium_social_integration_eligible_annual_mortgage_interest_on_property (RIS art. 25 §4) and brussels_apa_eligible_annual_mortgage_interest (APA art. 15), which this mapping feeds from mortgage_interest. No non-test rule text mentions capital repayment, and the PIT reductions module takes no loan input.", + "fact:household.source_household_id": "Original donor household lineage is build provenance and weight diagnostic support, not a rulespec-be legal input.", "fact:person.educational_attainment": "rulespec-be has no input that reads completed education. No non-test rule text mentions an attainment level, certificate or ISCED level. Every education input describes current enrollment or training: flanders_school_allowance_education_level, study_allowance_is_higher_education, and the Child-entity family-benefit training flags, including wallonia_family_benefits_child_in_qualifying_diploma_training. Where this mapping binds these, it feeds them from education_enrollment.", "fact:person.enrolled_full_time": "rulespec-be has no input that tests full-time enrollment. The full-time wording in non-test rule text concerns work regimes (parental leave, the work bonus, unemployment admission, the Flemish jobbonus). The one exception is a Flemish school-allowance parameter citing 'voltijds secundair onderwijs', and that module selects it by education level alone. The student statuses the tree does take are supplied legal statuses, and the rules that consume them contain no full-time test: belgium_self_employed_is_student (RD 38 art. 12bis) and the Child-entity family-benefit training and 'qualifying' flags.", "fact:person.liquid_financial_assets": "rulespec-be at b105e2b3 takes movable capital only as means-test inputs. belgium_social_integration_movable_capital_amount (RIS art. 27, in regulations/social_integration/resource_calculation.yaml and statutes/social_integration/payable_amount.yaml) lives on Household, and no transform puts a person amount on a Household input; the module also counts it through an owner fraction it takes as two further inputs (belgium_social_integration_movable_capital_numerator_share over belgium_social_integration_movable_capital_account_holder_count). brussels_apa_movable_capital_amount (Person, be-bru/statutes/disability/elderly_care_allowance.yaml; APA order of 28 January 2021 art. 16: 'Pour les capitaux mobiliers, placés ou non, il est porté en compte une somme égale à 6 % des capitaux.') is the one Person-level candidate. It counts all movable capital, invested or not, which is broader than the concept. The APA module also reads it as the movable capital of the beneficiary and the person they form a household with, not the beneficiary's alone: brussels_apa_movable_capital_deemed_annual_income turns it into deemed income at 6 %, and brussels_apa_annual_income_before_disregard multiplies the summed income by brussels_apa_household_income_split_factor, which is 0.5 when the person forms a household and both members are entitled to the allowance (art. 5 § 2: 'il est tenu compte pour chacune des deux personnes de la moitié du revenu du ménage') and 1 otherwise. It is left for a reviewed Belgian binding.", diff --git a/packages/microcosm-frame/src/microcosm/frame/adapters/axiom_concept_mappings/nz.json b/packages/microcosm-frame/src/microcosm/frame/adapters/axiom_concept_mappings/nz.json index c7efec54fb..49f6e3cd45 100644 --- a/packages/microcosm-frame/src/microcosm/frame/adapters/axiom_concept_mappings/nz.json +++ b/packages/microcosm-frame/src/microcosm/frame/adapters/axiom_concept_mappings/nz.json @@ -1496,6 +1496,7 @@ "structural_inputs": [], "unmapped": { "fact:household.property_tax": "The only rates input, rr_rates_payable_before_rebate (Person) in statutes/rates_rebates/rates_rebate.yaml, cannot be bound at 6fe181f, because the dense engine could not compile that module and nz.json lists it only under cli_walk_inputs, which the adapter's surface loader does not read; once the module compiles, allocate_to_reference_person of property_tax is the candidate (approximate, because the ratepayer need not be the reference person).", + "fact:household.source_household_id": "Original donor household lineage is build provenance and weight diagnostic support, not a rulespec-nz legal input.", "fact:person.educational_attainment": "rulespec-nz at 6fe181f has no input for completed education: no rule or input name mentions a qualification, degree, diploma or attainment level; the education inputs are current enrollment or school attendance (this mapping binds some of them to education_enrollment and enrolled_full_time); and young_person_is_study_ready is what youth_young_parent_payment/core.yaml calls an agency-adjudicated composite input (s 50).", "fact:person.realized_capital_gains": "rulespec-nz at 6fe181f has no input for gains on disposals: no rule mentions gains, disposals or the bright-line test, and the only capital input, income_tax_expenditure_or_loss_is_capital_nature (income_tax/core/taxable_income.yaml, s DA 2), classifies expenditure for the general deduction limitation.", "fact:person.sex": "rulespec-nz at 6fe181f has no sex- or gender-dependent input: none of the 668 dense-surface input names or 40 module YAMLs mention sex, gender, male, female, man, woman, wife or husband, and the only related hit, sole_parent_is_mother_or_father_or_child_regarded_as_child (social_security/main_benefits/entitlement.yaml), tests parenthood, which this mapping binds to the parent_of_coresident_child role.", diff --git a/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_uk_concepts.py b/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_uk_concepts.py index b86e95299a..0ca5fdd6b0 100644 --- a/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_uk_concepts.py +++ b/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_uk_concepts.py @@ -491,6 +491,10 @@ def _uk_take_up(program: str, entity: str) -> InputBinding: ), ), unmapped={ + "fact:household.source_household_id": ( + "Original donor household lineage is build provenance and weight " + "diagnostic support, not a policyengine-uk rules input." + ), "fact:person.legal_marital_status": ( "policyengine-uk derives marital status from benefit-unit " "composition; marital_status and is_married are formula-owned." diff --git a/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us_concepts.py b/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us_concepts.py index d0b4f1bbf7..a9c8b1a981 100644 --- a/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us_concepts.py +++ b/packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us_concepts.py @@ -620,6 +620,10 @@ def _us_take_up(program: str, entity: str, engine_input: str) -> InputBinding: ), ), unmapped={ + "fact:household.source_household_id": ( + "Original donor household lineage is build provenance and weight " + "diagnostic support, not a policyengine-us rules input." + ), "fact:person.liquid_financial_assets": ( "policyengine-us splits this stock across three person inputs, " "which the populace-us donor H5 also carries as person columns. " diff --git a/packages/microcosm-frame/src/microcosm/frame/concepts.py b/packages/microcosm-frame/src/microcosm/frame/concepts.py index 34ae07f0e4..7e03035f6d 100644 --- a/packages/microcosm-frame/src/microcosm/frame/concepts.py +++ b/packages/microcosm-frame/src/microcosm/frame/concepts.py @@ -89,6 +89,7 @@ "PERSON_ID_COLUMN", "RENTED_TENURES", "SEX_DOMAIN", + "SOURCE_HOUSEHOLD_ID_COLUMN", "TENURE_DOMAIN", "AlignmentRelation", "CanonicalConceptKind", @@ -117,7 +118,7 @@ #: concept is added, removed or changes any declared field, so artifacts that #: pin :func:`concept_schema_sha256` can say which schema they were written #: against. -CONCEPT_SCHEMA_VERSION = 2 +CONCEPT_SCHEMA_VERSION = 3 #: The two entities concept content lives on. Engine group entities (tax #: units, benefit units, SPM units, families) are engine constructs built @@ -133,6 +134,8 @@ PERSON_ID_COLUMN = CONCEPT_FRAME_SCHEMA.person_id_column PERSON_HOUSEHOLD_ID_COLUMN = CONCEPT_FRAME_SCHEMA.membership_column("household") HOUSEHOLD_ID_COLUMN = CONCEPT_FRAME_SCHEMA.id_column("household") +#: Original donor household identity; support clones carry the same key. +SOURCE_HOUSEHOLD_ID_COLUMN = "source_household_id" _PERSON_ID = PERSON_ID_COLUMN _HOUSEHOLD_ID = HOUSEHOLD_ID_COLUMN _PERSON_HOUSEHOLD_ID = PERSON_HOUSEHOLD_ID_COLUMN @@ -164,6 +167,7 @@ class Unit(StrEnum): CATEGORY = "category" BOOLEAN = "boolean" PERSON_ID = "person_id" + SOURCE_ID = "source_id" UNIT_INTERVAL = "unit_interval" @@ -660,6 +664,8 @@ def _validate_unit_contract(self) -> None: raise ValueError( f"Pointer concept {self.id!r} must be an int named *_person_id." ) + if self.unit is Unit.SOURCE_ID and self.dtype != "str": + raise ValueError(f"Source identity concept {self.id!r} must be str.") def _validate_bounds(self) -> None: numeric = self.dtype in ("float", "int") @@ -1247,6 +1253,24 @@ def _pointer( index_family=IndexFamily.OWNER_HOUSING_COSTS, entity="household", ), + # --- Source lineage --------------------------------------------------- + Concept( + id=f"fact:household.{SOURCE_HOUSEHOLD_ID_COLUMN}", + label="Original source household identity", + definition=( + "Exact identity of the original sampled source household, scoped " + "by source vintage when source household identifiers are " + "vintage-local. All support clones retain the same identity. " + "Derived only from explicit source lineage, never from row order " + "or the support row's household id." + ), + dtype="str", + unit=Unit.SOURCE_ID, + period="point", + temporal_basis=TemporalBasis.PERSISTENT, + provenance=ProvenanceClass.DERIVED, + transport=TransportRule.CARRY, + ), ) #: Concepts by id, read-only. diff --git a/packages/microcosm-frame/src/microcosm/frame/transport.py b/packages/microcosm-frame/src/microcosm/frame/transport.py index 8455e20251..8353dfa07f 100644 --- a/packages/microcosm-frame/src/microcosm/frame/transport.py +++ b/packages/microcosm-frame/src/microcosm/frame/transport.py @@ -19,6 +19,7 @@ HOUSEHOLD_ID_COLUMN, PERSON_HOUSEHOLD_ID_COLUMN, PERSON_ID_COLUMN, + SOURCE_HOUSEHOLD_ID_COLUMN, ContentBasis, Unit, concept_for_column, @@ -468,6 +469,9 @@ class DonorBank: transport. No destination country or year is inferred by this reader. ``dropped_parent_cycle_edges`` counts directed parent pointers removed by the reader's age-based two-person-cycle rule, including support copies. + The household table carries ``source_household_id`` only when explicit + source year and raw household lineage are available. Its typed JSON key + merges clones across support channels and distinguishes source vintages. """ tables: Mapping[str, pd.DataFrame] @@ -524,6 +528,47 @@ def _pointed_rows( return present, rows +def _source_household_ids( + person: pd.DataFrame, household: pd.DataFrame +) -> pd.arrays.StringArray | None: + """Scope raw household lineage by vintage and retain clone identity. + + The pooled donor's raw ``source_household_id`` is vintage-local. Its + remapped ``PH_SEQ`` and row ``household_id`` serve different roles and + are never substituted. Support channel is excluded because support + copies of one original household may live in different channels. + """ + columns = ("source_year", SOURCE_HOUSEHOLD_ID_COLUMN) + if not any(column in person for column in columns): + return None + if not all(column in person for column in columns): + raise ValueError("Donor source lineage requires year and household IDs.") + try: + parts = [_source_id_payloads(person[column]) for column in columns] + except ValueError as error: + raise ValueError( + "Donor source household lineage requires exact IDs." + ) from error + identities = [ + json.dumps(items, ensure_ascii=False, separators=(",", ":")) + for items in zip(*parts, strict=True) + ] + by_household = {} + for household_id, identity in zip( + person[PERSON_HOUSEHOLD_ID_COLUMN], identities, strict=True + ): + if household_id in by_household and by_household[household_id] != identity: + raise ValueError( + "Donor source household lineage disagrees within household." + ) + by_household[household_id] = identity + try: + values = [by_household[value] for value in household[HOUSEHOLD_ID_COLUMN]] + except KeyError as error: + raise ValueError("Donor source household lineage lacks a household.") from error + return pd.array(values, dtype=pd.StringDtype(storage="python")) + + def _donor_extras(person: pd.DataFrame, household: pd.DataFrame) -> dict: """Translate donor-only columns before crossing the concept decode seam.""" required = ( @@ -676,6 +721,7 @@ def pointer(present: np.ndarray, rows: np.ndarray) -> pd.arrays.IntegerArray: "weights": weights, "support_strata": strata, "source_person_ids": source_ids, + "source_household_ids": _source_household_ids(person, household), } @@ -806,6 +852,8 @@ def read_populace_us_donor(path: str | Path, *, sha256: str, size: int) -> Donor tables["person"][name] = values tables["person"]["liquid_financial_assets"] = staged["liquid_financial_assets"] tables["household"]["reference_person_id"] = staged["reference_person_id"] + if staged["source_household_ids"] is not None: + tables["household"][SOURCE_HOUSEHOLD_ID_COLUMN] = staged["source_household_ids"] violations = validate_concept_tables(tables) dropped = 0 if violations and all(v.code == "parent_cycle" for v in violations): diff --git a/packages/microcosm-frame/src/microcosm/frame/unit_construction.py b/packages/microcosm-frame/src/microcosm/frame/unit_construction.py index 33057abd17..dca0c33b3f 100644 --- a/packages/microcosm-frame/src/microcosm/frame/unit_construction.py +++ b/packages/microcosm-frame/src/microcosm/frame/unit_construction.py @@ -81,6 +81,7 @@ HOUSEHOLD_ID_COLUMN, PERSON_HOUSEHOLD_ID_COLUMN, PERSON_ID_COLUMN, + SOURCE_HOUSEHOLD_ID_COLUMN, _exceeds_int64, _parsing, _record_fields, @@ -263,6 +264,11 @@ def household_column(self) -> str: """The unit table's column naming the household the unit nests in.""" return f"{self.entity}_household_id" + @property + def source_household_column(self) -> str: + """Original donor lineage, distinct from every other entity column.""" + return f"{self.entity}_{SOURCE_HOUSEHOLD_ID_COLUMN}" + @property def head_column(self) -> str: """The unit table's column naming the unit's head.""" @@ -367,6 +373,8 @@ def build_benefit_units( ``_household_id`` and ``_head_person_id`` (all ``int64``); and each person's unit id (``int64``), aligned to ``person``'s rows and index and named ``person__id``. + When the household table carries original source lineage, the unit + table also carries ``_source_household_id`` unchanged. Raises: ValueError: If a needed column is missing, the frame breaks a @@ -391,6 +399,11 @@ def build_benefit_units( rule.head_column: ids[head_rows], } ) + if SOURCE_HOUSEHOLD_ID_COLUMN in household: + lineage = household.set_index(_HOUSEHOLD_ID)[SOURCE_HOUSEHOLD_ID_COLUMN] + family[rule.source_household_column] = family[rule.household_column].map( + lineage + ) membership = pd.Series( ids[head], index=person.index, name=rule.membership_column, dtype=np.int64 ) diff --git a/packages/microcosm-frame/tests/engine_free/shared/test_concept_mapping.py b/packages/microcosm-frame/tests/engine_free/shared/test_concept_mapping.py index 2e91d08378..44161173e2 100644 --- a/packages/microcosm-frame/tests/engine_free/shared/test_concept_mapping.py +++ b/packages/microcosm-frame/tests/engine_free/shared/test_concept_mapping.py @@ -52,6 +52,7 @@ ) from microcosm.frame.concepts import ( CONCEPTS, + SOURCE_HOUSEHOLD_ID_COLUMN, AlignmentRelation, CanonicalConceptKind, ConceptAlignment, @@ -253,6 +254,36 @@ def test_declarations_match_the_engine_kind(self) -> None: class TestExecution: + @pytest.mark.parametrize("name", sorted(MAPPINGS)) + @settings(max_examples=20, deadline=None, database=None) + @given(tables=concept_frames(), data=st.data()) + def test_source_household_lineage_changes_no_engine_input( + self, name, tables, data + ) -> None: + mapping = MAPPINGS[name] + parameters = _parameters(mapping, data) + before = mapping.encode(tables, **parameters) + household = tables["household"].copy() + household[SOURCE_HOUSEHOLD_ID_COLUMN] = pd.array( + data.draw( + st.lists( + st.text(min_size=1, max_size=20), + min_size=len(household), + max_size=len(household), + ) + ), + dtype=pd.StringDtype(storage="python"), + ) + after = mapping.encode( + {"person": tables["person"], "household": household}, **parameters + ) + for entity in ("person", "household"): + pd.testing.assert_frame_equal(before.tables[entity], after.tables[entity]) + assert before.deferred == after.deferred + assert ( + SOURCE_HOUSEHOLD_ID_COLUMN not in mapping.decode(after.tables)["household"] + ) + @PROPERTY @given(name=mapping_names, tables=concept_frames(), data=st.data()) def test_encode_matches_a_row_by_row_reference(self, name, tables, data) -> None: diff --git a/packages/microcosm-frame/tests/engine_free/shared/test_concepts.py b/packages/microcosm-frame/tests/engine_free/shared/test_concepts.py index 723a1ed509..b13e73b30d 100644 --- a/packages/microcosm-frame/tests/engine_free/shared/test_concepts.py +++ b/packages/microcosm-frame/tests/engine_free/shared/test_concepts.py @@ -83,7 +83,7 @@ #: Pinned schema digest. A change to any concept's declared contract moves it: #: bump CONCEPT_SCHEMA_VERSION when the change is deliberate, then re-pin. -SCHEMA_SHA256 = "d824f5e97d58292d62f65ffae60bc8654408711344b638d4e3caa95d6c2e8d1e" +SCHEMA_SHA256 = "5a55669d0447a686692911a6327ae298f19741dde29559f2654a75807878a141" class TestDeclaredSchema: @@ -214,7 +214,7 @@ def test_lookups(self) -> None: concepts_for_entity("tax_unit") def test_schema_digest_is_pinned(self) -> None: - assert CONCEPT_SCHEMA_VERSION == 2 + assert CONCEPT_SCHEMA_VERSION == 3 assert concept_schema_sha256() == SCHEMA_SHA256 def test_schema_digest_ignores_alignments(self) -> None: diff --git a/packages/microcosm-frame/tests/engine_free/shared/test_transport_donor.py b/packages/microcosm-frame/tests/engine_free/shared/test_transport_donor.py index 475a4b692d..71257bbddb 100644 --- a/packages/microcosm-frame/tests/engine_free/shared/test_transport_donor.py +++ b/packages/microcosm-frame/tests/engine_free/shared/test_transport_donor.py @@ -660,8 +660,13 @@ def test_reader_scopes_repeated_person_id_to_exact_source_lineage( exact_leaf = person.loc[0, "source_person_id"] person.loc[3, "source_person_id"] = exact_leaf if scope == "year": - person.loc[3, "source_household_id"] = person.loc[0, "source_household_id"] - person.loc[3, "source_year"] = person.loc[0, "source_year"] - 1 + household_members = ( + person["person_household_id"] == person.loc[3, "person_household_id"] + ) + person.loc[household_members, "source_household_id"] = person.loc[ + 0, "source_household_id" + ] + person.loc[household_members, "source_year"] = person.loc[0, "source_year"] - 1 donor = _fake_donor(monkeypatch, tmp_path, tables) assert donor.source_person_ids[0] != donor.source_person_ids[3] for index in (0, 3): diff --git a/packages/microcosm-frame/tests/engine_free/us/test_concept_mapping.py b/packages/microcosm-frame/tests/engine_free/us/test_concept_mapping.py index 81c48a34bb..62c6553341 100644 --- a/packages/microcosm-frame/tests/engine_free/us/test_concept_mapping.py +++ b/packages/microcosm-frame/tests/engine_free/us/test_concept_mapping.py @@ -40,10 +40,14 @@ def test_formula_owned_aggregates_are_never_targets() -> None: } -def test_every_concept_but_liquid_assets_reaches_the_us_engine() -> None: +def test_every_concept_but_liquid_assets_and_lineage_reaches_the_us_engine() -> None: # The engine splits liquid financial assets across three person inputs, - # and no transform splits one amount three ways. - assert set(MAPPING.unmapped) == {"fact:person.liquid_financial_assets"} + # and no transform splits one amount three ways. Original donor lineage + # stays build provenance rather than becoming a policy input. + assert set(MAPPING.unmapped) == { + "fact:person.liquid_financial_assets", + "fact:household.source_household_id", + } reason = MAPPING.unmapped["fact:person.liquid_financial_assets"] for name in ("bank_account_assets", "stock_assets", "bond_assets"): assert name in reason diff --git a/packages/microcosm-frame/tests/golden/concept-coverage/axiom-be.json b/packages/microcosm-frame/tests/golden/concept-coverage/axiom-be.json index 260fd668e9..74291ee075 100644 --- a/packages/microcosm-frame/tests/golden/concept-coverage/axiom-be.json +++ b/packages/microcosm-frame/tests/golden/concept-coverage/axiom-be.json @@ -1172,7 +1172,7 @@ "engine": "axiom:be", "engine_version": "b105e2b3a3086ddd2de447d58a9b951346870dd1", "input_count": 1867, - "schema_sha256": "d824f5e97d58292d62f65ffae60bc8654408711344b638d4e3caa95d6c2e8d1e", + "schema_sha256": "5a55669d0447a686692911a6327ae298f19741dde29559f2654a75807878a141", "structural_inputs": [], "uncovered_inputs": [ { @@ -9989,6 +9989,7 @@ "unknown_inputs": [], "unmapped_concepts": { "fact:household.mortgage_principal": "rulespec-be at b105e2b3 has no input for mortgage capital repayments. The only housing-loan inputs are the interest deductions belgium_social_integration_eligible_annual_mortgage_interest_on_property (RIS art. 25 \u00a74) and brussels_apa_eligible_annual_mortgage_interest (APA art. 15), which this mapping feeds from mortgage_interest. No non-test rule text mentions capital repayment, and the PIT reductions module takes no loan input.", + "fact:household.source_household_id": "Original donor household lineage is build provenance and weight diagnostic support, not a rulespec-be legal input.", "fact:person.educational_attainment": "rulespec-be has no input that reads completed education. No non-test rule text mentions an attainment level, certificate or ISCED level. Every education input describes current enrollment or training: flanders_school_allowance_education_level, study_allowance_is_higher_education, and the Child-entity family-benefit training flags, including wallonia_family_benefits_child_in_qualifying_diploma_training. Where this mapping binds these, it feeds them from education_enrollment.", "fact:person.enrolled_full_time": "rulespec-be has no input that tests full-time enrollment. The full-time wording in non-test rule text concerns work regimes (parental leave, the work bonus, unemployment admission, the Flemish jobbonus). The one exception is a Flemish school-allowance parameter citing 'voltijds secundair onderwijs', and that module selects it by education level alone. The student statuses the tree does take are supplied legal statuses, and the rules that consume them contain no full-time test: belgium_self_employed_is_student (RD 38 art. 12bis) and the Child-entity family-benefit training and 'qualifying' flags.", "fact:person.liquid_financial_assets": "rulespec-be at b105e2b3 takes movable capital only as means-test inputs. belgium_social_integration_movable_capital_amount (RIS art. 27, in regulations/social_integration/resource_calculation.yaml and statutes/social_integration/payable_amount.yaml) lives on Household, and no transform puts a person amount on a Household input; the module also counts it through an owner fraction it takes as two further inputs (belgium_social_integration_movable_capital_numerator_share over belgium_social_integration_movable_capital_account_holder_count). brussels_apa_movable_capital_amount (Person, be-bru/statutes/disability/elderly_care_allowance.yaml; APA order of 28 January 2021 art. 16: 'Pour les capitaux mobiliers, plac\u00e9s ou non, il est port\u00e9 en compte une somme \u00e9gale \u00e0 6 % des capitaux.') is the one Person-level candidate. It counts all movable capital, invested or not, which is broader than the concept. The APA module also reads it as the movable capital of the beneficiary and the person they form a household with, not the beneficiary's alone: brussels_apa_movable_capital_deemed_annual_income turns it into deemed income at 6 %, and brussels_apa_annual_income_before_disregard multiplies the summed income by brussels_apa_household_income_split_factor, which is 0.5 when the person forms a household and both members are entitled to the allowance (art. 5 \u00a7 2: 'il est tenu compte pour chacune des deux personnes de la moiti\u00e9 du revenu du m\u00e9nage') and 1 otherwise. It is left for a reviewed Belgian binding.", diff --git a/packages/microcosm-frame/tests/golden/concept-coverage/axiom-nz.json b/packages/microcosm-frame/tests/golden/concept-coverage/axiom-nz.json index 850c369949..0660e15e34 100644 --- a/packages/microcosm-frame/tests/golden/concept-coverage/axiom-nz.json +++ b/packages/microcosm-frame/tests/golden/concept-coverage/axiom-nz.json @@ -1383,7 +1383,7 @@ "engine": "axiom:nz", "engine_version": "6fe181fc4c65763150fe97501ef3443774e64869", "input_count": 668, - "schema_sha256": "d824f5e97d58292d62f65ffae60bc8654408711344b638d4e3caa95d6c2e8d1e", + "schema_sha256": "5a55669d0447a686692911a6327ae298f19741dde29559f2654a75807878a141", "structural_inputs": [], "uncovered_inputs": [ { @@ -4285,6 +4285,7 @@ "unknown_inputs": [], "unmapped_concepts": { "fact:household.property_tax": "The only rates input, rr_rates_payable_before_rebate (Person) in statutes/rates_rebates/rates_rebate.yaml, cannot be bound at 6fe181f, because the dense engine could not compile that module and nz.json lists it only under cli_walk_inputs, which the adapter's surface loader does not read; once the module compiles, allocate_to_reference_person of property_tax is the candidate (approximate, because the ratepayer need not be the reference person).", + "fact:household.source_household_id": "Original donor household lineage is build provenance and weight diagnostic support, not a rulespec-nz legal input.", "fact:person.educational_attainment": "rulespec-nz at 6fe181f has no input for completed education: no rule or input name mentions a qualification, degree, diploma or attainment level; the education inputs are current enrollment or school attendance (this mapping binds some of them to education_enrollment and enrolled_full_time); and young_person_is_study_ready is what youth_young_parent_payment/core.yaml calls an agency-adjudicated composite input (s 50).", "fact:person.realized_capital_gains": "rulespec-nz at 6fe181f has no input for gains on disposals: no rule mentions gains, disposals or the bright-line test, and the only capital input, income_tax_expenditure_or_loss_is_capital_nature (income_tax/core/taxable_income.yaml, s DA 2), classifies expenditure for the general deduction limitation.", "fact:person.sex": "rulespec-nz at 6fe181f has no sex- or gender-dependent input: none of the 668 dense-surface input names or 40 module YAMLs mention sex, gender, male, female, man, woman, wife or husband, and the only related hit, sole_parent_is_mother_or_father_or_child_regarded_as_child (social_security/main_benefits/entitlement.yaml), tests parenthood, which this mapping binds to the parent_of_coresident_child role.", diff --git a/packages/microcosm-frame/tests/golden/concept-coverage/policyengine-uk.json b/packages/microcosm-frame/tests/golden/concept-coverage/policyengine-uk.json index ac9626aaf2..df366a6550 100644 --- a/packages/microcosm-frame/tests/golden/concept-coverage/policyengine-uk.json +++ b/packages/microcosm-frame/tests/golden/concept-coverage/policyengine-uk.json @@ -395,7 +395,7 @@ "engine": "policyengine-uk", "engine_version": "2.122.2", "input_count": 285, - "schema_sha256": "d824f5e97d58292d62f65ffae60bc8654408711344b638d4e3caa95d6c2e8d1e", + "schema_sha256": "5a55669d0447a686692911a6327ae298f19741dde29559f2654a75807878a141", "structural_inputs": [ { "entity": "benunit", @@ -1383,6 +1383,7 @@ "unknown_inputs": [], "unmapped_concepts": { "fact:household.reference_person_id": "is_household_head is formula-owned (the oldest member). The UK build overrides it with the FRS household reference person (frs_spine.py), an override the loader accepts, but it is not an input, so the mapping does not target it. Every housing input is household-level, so nothing is allocated to the reference person.", + "fact:household.source_household_id": "Original donor household lineage is build provenance and weight diagnostic support, not a policyengine-uk rules input.", "fact:person.legal_marital_status": "policyengine-uk derives marital status from benefit-unit composition; marital_status and is_married are formula-owned.", "fact:person.liquid_financial_assets": "policyengine-uk holds financial wealth on the household: savings ('Household liquid savings'), corporate_wealth ('Wealth held in corporations directly or through investment funds'), gross_financial_wealth and net_financial_wealth are household inputs, and no transform puts a person amount on a household input. Its person-level GBP stock inputs are student_loan_balance, a debt; car_list_price, the price vehicle excise duty's expensive-car supplement tests; and lifetime_isa_balance ('Current value of the person's Lifetime ISAs, cash and stocks and shares together'), one savings wrapper, which the engine documents as already inside gross_financial_wealth and net_financial_wealth. savings and corporate_wealth are two of Universal Credit's capital sources (gov.dwp.universal_credit.means_test.capital.sources), and lifetime_isa_countable_capital, the Lifetime ISA at its surrender value, is its one person-level source (capital.person_sources). uc_reported_capital (benefit unit; 'Claimant-level capital for Universal Credit when household-level asset data cannot be attributed across multiple benefit units', default -1) is Universal Credit capital: when it is zero or more, uc_assessable_capital takes it in place of both the household-capital proxy it otherwise sums from the household sources, net of capital other benefit units report, and apportions by claimant and partner count, and the person-level sources of the unit's claimant and partner, so feeding it would be a modelling choice for a UK build, not a mapping of this concept.", "fact:person.partner_person_id": "Partnerships reach policyengine-uk only through benefit-unit membership, a structure the bundle must already carry (the adapter builds no units); is_benunit_head and is_married are formula-owned.", diff --git a/packages/microcosm-frame/tests/golden/concept-coverage/policyengine-us.json b/packages/microcosm-frame/tests/golden/concept-coverage/policyengine-us.json index 01a6c79c5d..6c0440fc9c 100644 --- a/packages/microcosm-frame/tests/golden/concept-coverage/policyengine-us.json +++ b/packages/microcosm-frame/tests/golden/concept-coverage/policyengine-us.json @@ -547,7 +547,7 @@ "engine": "policyengine-us", "engine_version": "2.2.1", "input_count": 926, - "schema_sha256": "d824f5e97d58292d62f65ffae60bc8654408711344b638d4e3caa95d6c2e8d1e", + "schema_sha256": "5a55669d0447a686692911a6327ae298f19741dde29559f2654a75807878a141", "structural_inputs": [ { "entity": "family", @@ -4038,6 +4038,7 @@ ], "unknown_inputs": [], "unmapped_concepts": { + "fact:household.source_household_id": "Original donor household lineage is build provenance and weight diagnostic support, not a policyengine-us rules input.", "fact:person.liquid_financial_assets": "policyengine-us splits this stock across three person inputs, which the populace-us donor H5 also carries as person columns. Their engine documentation: bank_account_assets ('Value of checking, savings, and money market accounts. Imputed from SIPP TVAL_BANK.'), stock_assets ('Value of stocks and mutual funds. Imputed from SIPP TVAL_STMF.') and bond_assets ('Value of bonds and government securities. Imputed from SIPP TVAL_BOND.'). All three uprate with CPI-U (gov.bls.cpi.cpi_u), the concept's consumer-price index family. The documentation names no term deposits. In the donor (microcosm.build.us_runtime.scf_wealth and sipp_financial_assets), each household takes its whole three-column vector from either the SCF or the SIPP, about half each, and the vector sits on the member with the lowest CPS line number (the householder is line 1), with every other member at zero. SCF-sourced households carry a draw of an SCF family-unit total, whose bank column comes from SCF liq (checking, savings, money-market and call accounts) and so leaves out term deposits. SIPP-sourced households carry a draw from a model trained on individual SIPP holdings (December person records, each with that person's own TVAL_BANK, TVAL_STMF and TVAL_BOND), whose bank leaf includes certificates of deposit: its quality screen reads the SIPP CD value allocation flags (AJSCDVAL, AJOCDVAL, AOCDVAL). Neither half holds each owner's share. spm_unit_assets (SPM unit; read by the federal CCDF asset test is_ccdf_asset_eligible, the Massachusetts, Indiana, Iowa and Texas child-care asset tests ma_ccfa_asset_eligible, in_ccdf_asset_eligible, ia_cca_asset_eligible and tx_ccs_asset_eligible, and Illinois's HBI resource test il_hbi_resource_eligible) is a unit stock the engine defines only by its label 'SPM unit assets', so feeding it would be a modelling choice. No transform here splits one amount across three inputs: a share pair splits it in two, and a fraction is never decoded. So decode cannot recover the concept from these inputs, and encoding it waits for a reviewed three-way split." } }