Skip to content

G8-r2c E: map NZ wages onto scaled IRD bands with quantile_map@2 - #1475

Closed
MaxGhenis wants to merge 5 commits into
nzhub/g8r2c-intfrom
nzhub/g8r2c-e-wage-map
Closed

MaxGhenis wants to merge 5 commits into
nzhub/g8r2c-intfrom
nzhub/g8r2c-e-wage-map

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

G8-r2c hunk E (#1183): NZ maps P1 wages onto IRD's wage bands with transport.quantile_map@2. Draft. Base nzhub/g8r2c-int; F (#1476) and G (#1478) stack on it. N/ is packages/microcosm-build/src/microcosm/build/nz/, T/ is B/transport/, tests are in packages/microcosm-build/tests/engine_free/shared/, and file:line is at G's head (64ef20af) unless a line says otherwise.

Landing (R2C-21a, #1183 comment 6604753490). E lands with F and G in #1478's one merge commit, and the hub then closes this PR as included. E's head has two merge bases with main (D's head 91bf4c0f and C's fix-round head acfa6b94, which main took in through their own PRs), so GitHub marks this PR as conflicting against main, though git's own merge changes exactly E's 28 files. The fix branch's top, f43e8fbb (#1486), merges main 4a36edcf and has one merge base with main, so #1478 lands the whole stack and main's history keeps every commit of E, F and G.

Change

  • Wage interval table ird_wage_salary_national_ty2026 in N/reference_intervals.json, generated by tools/generate_nz_target_references.py:1181 (wage_interval_table) from the pinned feed's labels: person, employment_income_cents, right-closed, nil_band: absent, negative_income_policy: refuse, first_lower_cut_cents: 0, 235 bands (rows 7-241) ending in the open tail over $1,000,000. Declared and validated only, with no companions.
    • The design says no wage label shares an endpoint. Three do: rows 27, 57 and 87 read "$20,000 - $21,000", "$50,000 - $51,000" and "$80,000 - $81,000". The generator pins them (WAGE_SHARED_ENDPOINT_ROWS, :150) and refuses any change, and the right-closed cuts give each endpoint to the lower band, as the taxable table does. Row 240 spells its upper "$1 million".
  • P1 activated. The 235 precal rows are active (:736 adds quantile_producer and the table id), so _reference_selection drops their exemptions.
  • Scaled edges, N/earnings_bands.json, written by tools/build_nz_transport_graph.py:304 (earnings_bands_document). For each aging_index profile, every edge is float(Fraction(cut_cents) × P1 factor / 100) (scaled_edge, :299), with the factor id, its operational factor, its receipt SHA-256 and the wage table's SHA-256. Each profile carries the bare band list @2 reads (map_bands, keys {"bands"} only).
    • The factors match the design: 1.026690, 1.028580 and 1.030467 (N/reference_aging.json, p1_ty2026_to_ty2027_*).
    • The builder's --check runs validate_earnings_bands (:384), which recomputes every edge and refuses a stale edge, factor receipt, table digest or tail by name.
  • Nodes (earnings_nodes, :413): nz.earnings.map (transport.quantile_map@2) rewrites person.employment_income on nz.central beside nz.money.normalize, reading the precal surface; nz.earnings.cents (money.minor_units@1, half-even) writes employment_income_cents. Both precede nz.encode.person.
  • Tail (R2C-9). earnings_bands.json tails.central anchors grouped_pareto_mle@1 on rows 231-241 (row 231 is "$500,001 - $550,000"); tails.vpt on rows 227-241 (row 227 is "$300,001 - $350,000"). Bracket ["1.00000095367431640625", "64"], 64 iterations. No two-point closed form.
  • Export (R2C-6). The builder's roster adds the @2 rewrite output (:1139-1154), so person.employment_income is in optional and both entity_slices of N/export_contract.json.
  • Variants. tools/build_nz_entitlement_graph.py:626 passes each calibration variant's aging_index (V1 low, V2 high) to its own map node.
  • pending_outputs drops P1 earnings; N/money_normalization.json's employment_income entry names the producer.

Shared code read at int

  • _surface_bands (T/population_kernels.py:411-455) reads only lower, upper and reference from the params and never compares an edge with the reference's own bounds. @2's estimated_alpha (T/quantile_kernels.py:54-112) calls it, then requires the anchors to be the band list's top suffix. So the scaled TY2027 edges pair with the TY2026 rows without a shared-code change. grouped_pareto_mle reads edges as exact binary64 fractions (B/tail_estimators.py, _edge), and α is scale-invariant.

Invariants and tests

Invariant Test Red under its mutant
Every edge equals cut × factor / 100, rounded once, for every profile; the worked band is (51,429.00, 52,457.58] test_nz_transport_declarations.py::test_every_profiles_edges_equal_cut_times_factor_exactly unscaled edges
The builder refuses a hand-edited edge, a stale factor receipt, a stale table, a changed tail test_the_builder_check_refuses_each_stale_earnings_declaration (4 cases) each case is its own mutant
The map and cents precede the encoder on every calibration branch; V1 low, V2 high, VPT re-anchored test_the_wage_map_and_cents_precede_the_person_encoder_on_every_branch drop nz.earnings.map
R2C-6 export test_r2c6_exports_employment_income_on_the_export_and_both_slices drop it from the slices
P1 active, no exemption test_nz_reference_intervals.py::test_resolution_is_not_activation_and_no_active_row_requests_the_proxy, test_nz_target_references.py receipt test park the rows again
Wage cuts match the pinned feed's labels (feed audit) test_nz_target_references.py::test_wage_cuts_match_the_feed_labels_and_the_committed_table —
Edges stay contiguous and increasing for any positive factor test_nz_earnings_bands.py::test_property_edges_stay_contiguous_and_increasing_for_any_positive_factor (Hypothesis, 60 examples) —
Scaled-edge map = unscaled map × f within 2^-49 relative test_nz_earnings_bands.py::test_scaled_edge_map_equals_unscaled_map_times_the_factor —
Toy: the @2 support row is quantile_mapped, 0 outside, both bounds open; α = 2 on scaled toy edges test_nz_transport_toy_graph.py::test_the_wage_map_v2_support_row_is_quantile_mapped_with_none_outside, test_the_wage_map_v2_fits_the_toy_alpha_on_scaled_edges toy map back to @1

Differential tolerance. 2^-49 relative (about 1.8e-15). The scaled path rounds each edge once and then interpolates; the unscaled path interpolates and multiplies by f once more. The uniform branch takes three roundings and the Pareto branch two, so the paths differ by a few half-ulps. Measured on 4,000 lognormal values with α = 2.4: at most 4.30e-16 (low), 3.90e-16 (central), 2.95e-16 (high), matching design 80's 3.9e-16.

R2C-19 shared-test edits (commit 385d5967)

Line ranges at this head, for Belgium's hub:

  • test_spec_engine_country_bundles.py:246 (G moves this line to its sorted place under R2C-23)
  • test_transport_composed_graph.py:2755 (_GAPS drops the wage gap)
  • test_transport_evidence_kernels.py:2710 and :2715 (one quantile_mapped row; pending drops person.employment_income; all 307 derived rows miss exactly rent and liquid assets, so no other count moves)
  • test_transport_quantile_map_v2.py:673-696 (_spec_without_central_wage_producer, _spec_with_map) and :729-732 (the packaged spec has no central gap; without its producer it has one)

R2C-19's mutant: _money_producer_gaps accepting any kernel turns test_composition_accepts_v2_as_the_quantile_mapped_money_producer red at the transport.currency@1 refusal.

Hashes

The hub hashed 42 engine-free transport kernels (no simulate.*) at each head, each in its own tree: int 3e5f5edb, E 8ec98c1c, F 623afa09 and G a3fe1952, after which G's head 64ef20af changes only two test files. E, F and G hash identically to int, as does the int before D's merge (ba08d0f9), so this hunk moves no kernel hash.

Kernel int 909fe3cb int 3e5f5edb (= ba08d0f9, E, F, G)
calibrate.ordered_adam@1 0d4577effa147ebe 0d4577effa147ebe
calibration.prepare@1 180823901b9a0d99 180823901b9a0d99
concepts.encode@1 7815fae8ed573af6 7815fae8ed573af6
concepts.encode_groups@1 062f88411ad605ab 062f88411ad605ab
diagnostics.calibration@1 30654dfc7038f560 30654dfc7038f560
export.prepare@1 af31fa6e1690b90c af31fa6e1690b90c
export.readback@1 96be393c760ca768 96be393c760ca768
gates.battery@1 ac050bab09f48dc2 ac050bab09f48dc2
geography.assign_atomic@1 d60f7ec388903678 d60f7ec388903678
geography.derive@1 d60f7ec388903678 d60f7ec388903678
geography.gate@1 d60f7ec388903678 d60f7ec388903678
geography.person_location@1 3efcbe3741d04497 3efcbe3741d04497
geography.support_from_facts@1 ad0a8e73d20405be ad0a8e73d20405be
geography.support_import@1 d60f7ec388903678 d60f7ec388903678
measure.combine@1 d609832882ffc113 d609832882ffc113
money.annual_transfers@1 c8a067cd750ed2fe c8a067cd750ed2fe
money.minor_units@1 c8a067cd750ed2fe c8a067cd750ed2fe
money.normalize@1 f10fbeeaf87467b6 f10fbeeaf87467b6
reporting.carry_income_decile@1 80335f0183c27410 80335f0183c27410
reporting.income_decile@1 80335f0183c27410 80335f0183c27410
reporting.receipt_group@1 80335f0183c27410 80335f0183c27410
takeup.assign@1 b2a3a50cc376108b b2a3a50cc376108b
takeup.bands@1 1ceca3ba7c7c9b92 1ceca3ba7c7c9b92
takeup.cell_diagnostics@1 1ceca3ba7c7c9b92 1ceca3ba7c7c9b92
takeup.compare@1 1605d7e8519f4b0f 1605d7e8519f4b0f
takeup.gap@1 54f79560d896fab4 54f79560d896fab4
targets.compile@1 dc91d1479f8f84b1 dc91d1479f8f84b1
targets.problem@1 fe401c736551e351 fe401c736551e351
transport.boundary@1 fc9d10670f6a092f fc9d10670f6a092f
transport.create@1 865c21926cee0ac0 865c21926cee0ac0
transport.create@2 865c21926cee0ac0 865c21926cee0ac0
transport.currency@1 9c3457ddb908aa96 9c3457ddb908aa96
transport.donor_ranges@1 62def362139a52e4 62def362139a52e4
transport.donor_receipt@1 62def362139a52e4 62def362139a52e4
transport.input_closure@1 23134112061495b7 23134112061495b7
transport.package@1 e86be552287a6eb3 e86be552287a6eb3
transport.quantile_map@1 b25bb6538b19dd64 b25bb6538b19dd64
transport.quantile_map@2 b845aac2c57f667b 76c81605aa416dc9 (moved)
transport.scenario_override@1 062f88411ad605ab 062f88411ad605ab
transport.support_bounds@1 2303c60a7ec6135c 2303c60a7ec6135c
transport.unit_attributes@1 f50ec74a1bfe35bb f50ec74a1bfe35bb
transport.weights_audit@1 1f3d475013456127 1f3d475013456127

transport.quantile_map@2 moves only through int's merge of B's tail_estimators.py docstring fix, which it digests.

Tests

Closed list at int ba08d0f9, E 3e850e07 and F f5ed82ba (G at #1478), no --maxfail, -n 2. Counts read passed / skipped / failed. Merges of D's head (91bf4c0f) moved int to 3e5f5edb, E to 8ec98c1c and F to 623afa09, and against each earlier head they change only T/compose.py, test_transport_measurement_prefit.py and test_transport_measurement_proof.py. The hub reran those two files at each new head (-n 0, with the feed):

File int 3e5f5edb E 8ec98c1c F 623afa09
test_transport_measurement_prefit.py 47 47 47
test_transport_measurement_proof.py 33 33 33

The E-G review ran F's list at 623afa09 and found the same one red. F never lands alone, so its one red test never reaches main: G passes it, and E, F and G land together in #1478's merge (R2C-21a).

The closed list at the earlier heads:

File int E F
frame/test_transport_donor.py 94 / 2s 94 / 2s 94 / 2s
frame/test_transport_quantile.py 79 79 79
test_country_spec.py 143 143 143
test_nz_as_area_crosswalk.py 67 / 3s 67 / 3s 67 / 3s
test_nz_as_bridge.py 34 34 34
test_nz_axiom_input_closure.py 19 / 2s 19 / 2s 19 / 2s
test_nz_benefit_gross_up.py 16 16 16
test_nz_earnings_bands.py — 4 4
test_nz_family_reconciliation.py 16 16 16
test_nz_joint_geography.py 22 / 11s 22 / 11s 22 / 11s
test_nz_money_normalization.py 56 56 56
test_nz_msd_benefit_rates.py 6 6 6
test_nz_msd_pre_activation_reconciliation.py 5 5 5
test_nz_reference_aging.py 22 22 22
test_nz_reference_intervals.py 14 14 16
test_nz_spec_package.py 1203 1203 1203
test_nz_target_references.py 54 56 60
test_nz_transport_declarations.py 31 42 42
test_nz_transport_toy_graph.py 23 25 25
test_reference_interval_support.py 12 12 12
test_reference_intervals.py 107 107 107
test_reference_profiles.py 129 129 129
test_reference_summed_bands.py 23 23 23
test_spec_engine_country_bundles.py 22 22 22
test_spec_only_country_packages.py 37 37 37
test_transport_annual_transfers.py 75 75 75
test_transport_binding_identity.py 32 32 32
test_transport_bridge_differential.py 37 37 37
test_transport_codecs_geography.py 22 22 22
test_transport_column_kernels.py 31 31 31
test_transport_composed_graph.py 469 469 468 / 1 failed
test_transport_declared_units.py 18 18 18
test_transport_dependent_children.py 24 24 24
test_transport_entitlement_graph.py 67 67 67
test_transport_evidence_kernels.py 151 151 151
test_transport_gate_bindings.py 131 131 131
test_transport_gate_kernels.py 31 31 31
test_transport_graph.py 15 15 15
test_transport_grossup_selectors.py 26 26 26
test_transport_measure_kernels.py 37 37 37
test_transport_measurement_prefit.py 47 47 47
test_transport_measurement_proof.py 27 27 27
test_transport_money_kernels.py 40 40 40
test_transport_population_kernels.py 32 32 32
test_transport_quantile_map_v2.py 38 38 38
test_transport_receipt_v2.py 117 117 117
test_transport_receipt_v2_kernel.py 14 14 14
test_transport_reference_selection.py 60 60 60
test_transport_registry.py 59 59 59
test_transport_reporting_kernels.py 31 31 31
test_transport_run_entry_factors.py 75 75 75
test_transport_run_entry_forms.py 229 229 229
test_transport_source_household_lineage.py 8 8 8
test_transport_takeup_kernels.py 38 38 38
test_transport_target_kernels.py 68 68 68
test_transport_terminal_kernels.py 18 18 18
test_transport_weight_quality.py 30 30 30
uk/test_transport_gate_bindings.py 7 7 7
Total 4338 / 18s 4357 / 18s 4362 / 18s / 1 failed

Skips (18 at each head), each needing a source this container lacks: test_nz_joint_geography.py 11 (NZ_AS_AREA_SOURCES and NZ_JOINT_GEOGRAPHY_SOURCES unset); test_nz_as_area_crosswalk.py 3 (pinned ArcGIS bytes); test_nz_axiom_input_closure.py 2 (Axiom dense engine and a rulespec-nz checkout); frame/test_transport_donor.py 2 (pinned real donor not cached). Every NZ_CHRONICLE_ARTIFACT feed audit ran, against the artifact I built from Chronicle 8f4cfd57 (both digests match the pin).

Also at each head: ci_test_plan.py verify prints verification=ok; the three source-free NZ builders' --check and generate_nz_target_references.py --check (with the artifact) match; ruff is clean on the changed files.


Generated by Claude Code

The generator adds the TY2026 wage interval table
ird_wage_salary_national_ty2026 (rows 7-241, right-closed, no nil band,
negative wages refused, open tail over $1,000,000) from the pinned feed's
labels, and activates the 235 P1 precal rows, which drops their
exemptions. The transport builder writes earnings_bands.json: for each
aging_index profile, every edge is cut_cents x P1 factor / 100, computed
in Fraction and rounded once to binary64, with the factor id, its
receipt SHA-256 and the wage table's SHA-256. nz.earnings.map
(transport.quantile_map@2, grouped_pareto_mle@1 on rows 231-241) rewrites
person.employment_income beside nz.money.normalize, and nz.earnings.cents
writes employment_income_cents half-even, both before nz.encode.person.
Each calibration variant repeats them with its own profile (V1 low, V2
high). R2C-6 adds person.employment_income to the export contract and
both slices.

Three wage labels repeat the previous upper endpoint ($20,000, $50,000,
$80,000: rows 27, 57, 87); the generator pins them and the right-closed
cuts give each endpoint to the lower band.

Invariants: every edge equals cut x factor exactly for every profile
(example test, plus a builder --check that refuses a hand-edited edge,
a stale factor receipt, a stale table and a changed tail); the edges
stay contiguous and increasing for any positive factor (Hypothesis); the
scaled-edge map equals the unscaled map x f within 2^-49 relative
(measured at most 4.3e-16); the toy's @2 support row is quantile_mapped
with 0 records outside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LwQq6Fwj4yUEPsD3wfNstU
test_nz_spec_package pins the active row count per reference set. G8-r2c
E activates the 235 wage rows, so precal's active count moves from 80 to
315. The calibration and hold-out counts stay as they were.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LwQq6Fwj4yUEPsD3wfNstU
The hub's R2C-19 lifts the freeze for these tests on E. Each edit follows
from what E changes in NZ's own resources:

- test_spec_engine_country_bundles: NZ's legacy resources include
  earnings_bands.json.
- test_transport_composed_graph: _GAPS no longer names the
  person.employment_income quantile-map gap, which nz.earnings.map closes.
- test_transport_evidence_kernels: the roster gains one quantile_mapped
  row (R2C-6), and derived rows miss only rent and liquid assets; the
  per-row assertion over every derived row stays.
- test_transport_quantile_map_v2: _spec_with_map first removes the
  packaged nz.central producer of employment_income, so the kernel under
  test stands alone; that spec without a map still reports a central
  wage gap; the packaged spec reports none. Both accepted kernels and the
  transport.currency@1 refusal stay.

Mutant run: with _money_producer_gaps accepting any kernel,
test_composition_accepts_v2_as_the_quantile_mapped_money_producer goes
red (the currency@1 refusal finds no gap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LwQq6Fwj4yUEPsD3wfNstU
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Closing as included: #1478 landed this PR's commits on main in merge commit f8f543a4, at G's reviewed head f43e8fb. The NZ hub reviewed the stack as one change (review 213 v4, #1183 comment 6615417145).

🤖 Generated with Claude Code

@MaxGhenis MaxGhenis closed this Nov 4, 2026
@MaxGhenis
MaxGhenis deleted the nzhub/g8r2c-e-wage-map branch November 4, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants