Skip to content

NZ: re-declare the ageing factors from the September 2026 QES - #1507

Merged
MaxGhenis merged 2 commits into
mainfrom
nzhub/qes-sep2026-redeclare
Nov 4, 2026
Merged

MaxGhenis merged 2 commits into
mainfrom
nzhub/qes-sep2026-redeclare

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Re-declares New Zealand's earnings ageing factors from Stats NZ's September 2026 quarter Quarterly Employment Survey (QES) and re-pins NZ's Chronicle feed to 89178aaf, Chronicle main at the merge of PolicyEngine/chronicle#406 (tests of each QES record set's vintage and year guards), which follows PolicyEngine/chronicle#405 (the September package).

Why

NZ's declarations expired after 3 November 2026 (valid_until). The transport CLI's run-entry check refuses an NZ run with --resume auto or forbid when today's UTC date falls outside a selected factor's window (reference_factors.py:747-766, transport/cli.py:710-736); only a --resume require replay of stored work skips it (transport/cli.py:727-728). G8-r2 hub rulings 3 §2 set the method: once Stats NZ publishes a target quarter, the observed value replaces its projection, and the projection runs from the latest contiguous observed quarter. For TY2027 that gives N = (Jun26 + Sep26 + Sep26·q + Sep26·q²) / 4, with q = (1 + g)^(1/4) and g from PREFU 2026 (2.7%, ±0.5 points for the low and high profiles). A re-declaration is a new feed pin plus reviewed expected factors and receipts.

What changes

  • Feed pin (nz/chronicle_feed.json), all six moving fields: Chronicle 89178aaf, 12,370 facts, facts_sha256 8580143f…, manifest_sha256 ea0cc7ef…, schema chronicle.consumer_fact.v5 (acd33a09…). Seven operand quarters (2024-Q3 to 2026-Q2) take new fact keys and September's source hash, and 2026-Q3 joins at 1,714.86.
  • Schema v5. Give each dimension value one Chronicle-declared label, keep the package's own wording beside it (consumer_fact.v5) chronicle#377 gives each dimension value one label and moves consumer facts from v4 to v5: v4's fields plus an optional top-level package_dimension_value_labels, a source package's own wording of a value where it differs (consumer_fact.v5.schema.json:244 at 89178aaf). None of the feed's 12,370 rows carries it, and no Microcosm code at this head names it. Microcosm's loader carries each row's schema_version without gating on it (ledger_artifact.py:464-470), and is_accepted_consumer_fact_schema_version has no caller outside tests. The gate sits in the feed pin: validate_chronicle_feed_pin compares the schema ids the rows carry, the manifest's list and the schema digest with the pin exactly (ledger_artifact.py:257-291), so the pin moves to v5 with them. Between the old pin (8f4cfd57) and this one:
    • 12,352 facts appear in both feeds. 12,336 of them differ only in schema_version; the other 16, stats_nz_household_net_worth 2023 facts, also lose a trailing space in layout.groupby_value_label, a field no NZ resource, generator or test reads;
    • 16 June 2026 QES facts give way to 18 September 2026 facts.
  • Generator (tools/generate_nz_reference_aging.py): the one-line hash rule becomes _qes_hash(period) (TY2024 quarters from June 2025, 2024-Q2 from June 2026, later quarters from September 2026). _OPERANDS gains 2026-Q3 (line 12054) and the seven moved keys, whose line numbers stay, and _EXPECTED takes the new factors. Validity runs from 2026-11-04 to 2027-02-02, the day before Stats NZ releases the December quarter (3 February 2027, NZ time). A decision reference names this re-declaration.
  • Regenerated, each by its generator against the new artifact:
    • reference_aging.json, reference_inventory.json and msd_pre_activation_reconciliation.json;
    • earnings_bands.json, by tools/build_nz_transport_graph.py, which copies each P1 factor and its receipt hash and scales the 235 wage bands (only those values move);
    • the golden nz_country_spec.json, which pins every resource's hash.
  • Tests and doc. test_nz_reference_aging.py takes the new factors and window. Its re-anchor test now adds a synthetic December quarter, since September is real now, and its rounding test picks a P1 factor whose truncation differs. Four NZ test files and docs/nz-calibration-targets.md move pins derived from the factors: the central coarse cuts, the IRD weights, the P1 factors and design 80's worked band, the donor factors and their numerator formula, and the feed's schema version. Three clock pins in test_transport_composed_graph.py move from 10/24 to 11/05, inside the new window (Belgium hub's line review: OK (R2C-19). Belgium reconfirmed it at b44229cd4 at 19:52Z on 11/04 and at 4fcfc14ef at 21:02Z, since that head keeps the same trees. The file's patch-id stays cc60fdd73b4c7cfc, and with the pins back at 10/24 all three tests fail on the currency check).
  • Changelog: changelog.d/1507-nz-qes-september-2026-redeclaration.changed.md.

Factors (millionths)

Factor Before (June 2026 QES) After (September 2026 QES)
ird_ty2024_to_ty2027 low, central, high 1115056, 1117109, 1119158 1117460, 1118491, 1119519
p1_ty2026_to_ty2027 low, central, high 1026690, 1028580, 1030467 1028904, 1029853, 1030799
donor_cy2024_to_ty2027 low, central, high 1077090, 1079073, 1081052 1079413, 1080408, 1081401
observed_ty2026_over_ty2024 1086069 1086069

Invariants

  • Feed identity: chronicle_feed.json pins seven fields. reference_aging.json repeats them 21 times (at its top level and in each declaration and receipt), and reference_inventory.json and msd_pre_activation_reconciliation.json once each; every copy equals the pin. The golden pins the sha256 of 54 files, these four among them; the other 50 carry no copy, and earnings_bands.json, one of them, carries each P1 factor's receipt hash. validate_chronicle_feed_pin checks six of the seven fields against the loaded rows and manifest (ledger_artifact.py:257-291); source_commit stays the caller's declaration (ledger_artifact.py:94-98).
  • One fact per operand: each operand period resolves to exactly one fact, from the newest vintage that publishes it, with the declared lexical value (test_each_reviewed_operand_resolves_once_with_exact_lexical_evidence).
  • Exact factors: each of the nine TY2027 factors equals N over its source mean, and the observed diagnostic equals the TY2026 mean over the TY2024 mean (exactly 668029/615089, with no growth); each rounds half-even to six digits. The resolver raises when its result differs from the declared expected factor (reference_factors.py:611-614), so a one-millionth change refuses.
  • Inclusive window: require_current_factors accepts 2026-11-04 and 2027-02-02 and refuses 2026-11-03 and 2027-02-03 (test_factor_currency_uses_inclusive_declared_boundaries).
  • Re-anchoring: a newly observed quarter refuses the old expected factor and re-anchors the projection on it.
  • Earnings bands: each P1 edge equals float(cut × factor / 100) computed from Fraction, and each profile's factor_receipt_sha256 equals its receipt in reference_aging.json (test_every_profiles_edges_equal_cut_times_factor_exactly, validate_earnings_bands).

Checks

  • Independent arithmetic: the hub's expected_factors.py (Decimal at 60 digits, outside the repo) produced _EXPECTED. The generator resolves every declaration from the artifact and raises on any difference from its expected factor; it exited clean, so the repo's resolver agrees with the hub's arithmetic on all ten.

  • Gated files, one at a time with the feed staged (NZ_CHRONICLE_ARTIFACT), peak tree RSS and footprint in GiB:

    "Head" means tree 427910fc6, the change commit's tree: a623c14ef now, 4608f5457 before the message fix. It differs from this PR's head tree 74c61ef44 only by the changelog fragment.

    File Tree Result Peak RSS Peak footprint Wall s
    test_ledger_artifact.py 216c40578 55 passed 1.720 1.566 34.8
    test_nz_family_reconciliation.py 216c40578 16 passed 1.353 1.267 26.8
    test_nz_joint_geography.py 216c40578 22 passed, 11 skipped 1.360 1.274 22.6
    test_nz_msd_pre_activation_reconciliation.py 216c40578 5 passed 1.348 1.262 21.6
    test_nz_reference_aging.py 216c40578 22 passed 1.585 1.499 26.8
    test_nz_reference_intervals.py head 16 passed 1.359 1.273 27.9
    test_nz_money_normalization.py head 56 passed 1.359 1.274 32.2
    test_nz_earnings_bands.py head 4 passed 1.363 1.277 27.0
    test_nz_target_references.py head 61 passed 1.352 1.267 45.9
    test_nz_transport_declarations.py head 62 passed 1.553 1.403 247.2
    test_transport_composed_graph.py, the three clock-pin tests head 3 passed 1.375 1.289 230.2
    test_country_spec.py -k nz head 3 passed 1.352 1.267 29.2

    The first five ran at the hub's earlier commit 216c40578, which the squash replaced. Its tree differs from the head's in earnings_bands.json, the golden, four other NZ test files, the doc and the changelog fragment; none of the five files changed after it. Two of them (test_ledger_artifact, test_nz_joint_geography) load the whole NZ spec, which includes earnings_bands.json; CI reruns their feed-free tests at the head, and cloud review 220 reran both files whole with the feed (55 passed; 22 passed, 11 skipped).

    At 216c40578, test_nz_reference_intervals.py failed test_central_ageing_keeps_fractional_cent_cuts_exactly, and the hub's memory guard stopped test_nz_target_references.py at 2.063 GiB after 26 of its 61 tests, 6 of them failed. The test-pin moves fixed both files, and their reruns at the head passed.

    The 11 skips in test_nz_joint_geography.py come from its staged_joint fixture (:557-567), which needs the census source directories (NZ_AS_AREA_SOURCES, NZ_JOINT_GEOGRAPHY_SOURCES), not the feed.

  • Generators: each ran against the local artifact and exited 0. test_nz_transport_declarations.py runs the target-reference and MSD generators' --check with the feed. Cloud review 220 ran all four --checks against its own rebuild of the feed, and each exited 0.

  • CI: run 41909406184 on 4fcfc14ef, merged with main 3e7a7045 (Encode person_role bindings and their closure defaults on the person table (S2b PR 2) #1480 and Prove extension calibration nodes' measurement producers in compose #1488 landed today and touch no file here): pending. The feed-gated tests skip in CI.

  • Reviews, both at b44229cd4, whose trees 4fcfc14ef keeps:

    • Cloud review 220 (Activate the NZ graph against pinned Chronicle facts (NZ graph G8) #1183 comment 6640772483): APPROVE_WITH_NITS, needs Max: no. It rebuilt the feed from Chronicle 89178aaf (same digests), recomputed all ten factors at 60 digits, ran all twelve gated files with the feed (the counts above) and turned three mutants red. Nit 1, a P1 rounding test that can't tell one rounding from intermediate rounding, goes to NZ's follow-up PR. Nit 2, the epoch table's missing v5, belongs to open microcosm#1510.
    • GPT-6.1 Sol claim audit (Subfleet job 20261104-145428-nz1507-claims): APPROVE_WITH_NITS, needs Max: no. From git objects alone it recomputed the ten factors, all 1,407 finite band edges and the golden's 54 resource hashes, and checked every cited line range. Its five fixes all concerned wording, and this body and the change commit's message now carry them: the feed-identity and exact-factor invariants, the shared-fact counts, the operand line numbers, and the replay exception to the currency check.
    • Hub correction: the earlier body and commit message named v5's new field publisher_dimension_value_labels; Chronicle's schema at 89178aaf calls it package_dimension_value_labels. Chronicle's tests assert the old name's absence from the v5 schema, the harness documentation and the tested output rows. The body and the commit message now use the schema's name, and a scan found the field on none of the feed's 12,370 rows.
    • Delta review of these text changes (GPT-6.1 Sol, Subfleet job 20261104-160400-nz1507-delta): APPROVE_WITH_NITS, needs Max: no. It found all five fixes in place and no numerical or code defect. Its three nits concerned this body's wording (two in the hub correction, one passive), and this body now carries them.

🤖 Generated with Claude Code

@MaxGhenis
MaxGhenis force-pushed the nzhub/qes-sep2026-redeclare branch 3 times, most recently from a8ae6fc to 2703a89 Compare November 4, 2026 19:22
@MaxGhenis
MaxGhenis changed the base branch from nzhub/qes-sep2026-base to main November 4, 2026 19:23
@MaxGhenis
MaxGhenis force-pushed the nzhub/qes-sep2026-redeclare branch from 2703a89 to b44229c Compare November 4, 2026 19:23
MaxGhenis and others added 2 commits November 4, 2026 15:41
Stats NZ's September 2026 quarter QES moves nine of NZ's ten earnings
ageing factors; the observed TY2026-over-TY2024 ratio stays 1.086069.
The old declarations held through 2026-11-03. Since then the transport
CLI refuses every NZ run with --resume auto or forbid; only a --resume
require replay of stored work skips its currency check.

Re-pin NZ's Chronicle feed to the NZ-suite artifact of Chronicle
89178aaf (12,370 rows; facts sha256 8580143f..., manifest ea0cc7ef...),
the merge of chronicle#406. That commit carries chronicle#405 (the
September 2026 QES vintage), #406 (tests of each QES record set's
vintage and year guards) and chronicle#377. #377 gives each dimension
value one label and moves consumer facts to chronicle.consumer_fact.v5:
v4's fields plus an optional package_dimension_value_labels (a source
package's own wording where it differs), which no NZ row carries. So
the pin's schema list and schema digest move too.

tools/generate_nz_reference_aging.py takes the September 2026 operands,
adds the third QES vintage to its hash rule, and declares the ten
factors, valid from 2026-11-04 to 2027-02-02. The hub's own checker,
outside this repository, computed the ten factors from those operands
at 60 digits, and the generator's resolver reproduced all ten when it
ran.

Each NZ generator then ran against the artifact: reference_aging.json,
reference_inventory.json and msd_pre_activation_reconciliation.json
(the last two change only the feed identity).
tools/build_nz_transport_graph.py rewrote earnings_bands.json, which
scales each wage cut by its profile's P1 factor and copies that factor
and its receipt hash. The golden nz_country_spec.json pins every
resource's hash, so it moved last.

test_nz_reference_aging.py follows the generator. Four more NZ test
files and docs/nz-calibration-targets.md pinned values that depend on
the old factors or the v4 schema; they now pin the new ones. The three
clock pins in test_transport_composed_graph.py move from 2026-10-24 to
2026-11-05, inside the new window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the nzhub/qes-sep2026-redeclare branch from b44229c to 4fcfc14 Compare November 4, 2026 20:44
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 4, 2026 23:11
@MaxGhenis
MaxGhenis merged commit fc5bb90 into main Nov 4, 2026
11 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged by the NZ hub (session local_e407a527) at 4fcfc14 with --merge. Gates: gh pr checks exit 0; MERGEABLE; not draft; no CHANGES_REQUESTED review. Reviewed head: 4fcfc14. Reviews: cloud review 220 (#1183 comment 6640772483), the GPT-6.1 Sol claim audit (Subfleet 20261104-145428-nz1507-claims) and the GPT-6.1 Sol delta review (20261104-160400-nz1507-delta) all returned APPROVE_WITH_NITS with needs Max: no, and the body (r3) or the change commit's message carries every nit's fix. Belgium hub's line review: OK (R2C-19) at 4fcfc14, 21:02Z. CI run 41909406184 on 4fcfc14 merged with main 3e7a704: 11 of 11 passed. Main has since gained only #1513, one US test file this PR doesn't touch. A merge commit keeps the reviewed commit messages.

@MaxGhenis
MaxGhenis deleted the nzhub/qes-sep2026-redeclare branch November 4, 2026 23:12
MaxGhenis added a commit that referenced this pull request Nov 5, 2026
Tighten NZ's ageing and declarations tests after #1488 and #1507
anth-volk added a commit that referenced this pull request Nov 5, 2026
main re-pinned the US feed to Chronicle 41b2fb2b (#1506) and the NZ feed
onto chronicle.consumer_fact.v5 (#1507). The one conflict is in
docs/us-chronicle-feed-repin.md, where #1506's new re-pin section and
this branch's section on re-pinning onto a newer row contract sit in the
same place: both are kept, the general section first. #1506's sentence
saying the US loader carries a row's stamp unchecked now adds that the
release and target-parity tools check the stamps against the pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Nov 5, 2026
Review round 3 of #1510 found that tools/build_us_target_parity_manifest.py
holds the feed to the committed US pin by digest, compiles it and writes
committed files, but checked no row contracts, so a US re-pin that left
the hand-typed list stale passed its own rebuild steps.

- us_runtime.chronicle_feed: require_us_chronicle_feed_row_contracts, the
  exact-or-declared check: rows of the committed feed must carry exactly
  the pin's list; rows of any other feed, declared row contracts only.
- The release tool's _check_us_ledger_feed_row_contracts now calls it,
  and the target-parity generator calls it after its digest check,
  before compiling or writing.
- Wording: validate_pinned_row_contracts and the UK row check say the
  release certifier holds its feed to the operator's digests, so it
  applies the exact rule only when given the committed feed. The
  changelog says the same, names the US generator, and lists the
  readers left unchecked because no reviewed pin holds their feed (the
  US preflight and ACS tools, the scoring and diagnostic tools, the UK
  high-value home stock, BRMA rent-quantile and Scotland BRMA-overlap
  builders, and the childcare take-up fit tools).
- The UK and US re-pin guides name the certifier's rule and the
  generator's check; notes that said the NZ pin carries the v4 digest
  now say NZ is on v5 (#1507).

Refs #1509

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant