Bind state × AGI-band SOI targets and re-pin the US feed for TY2023 state bands (#940)

changelog.d/940-state-agi-band-targets.added.md

@@ -0,0 +1 @@
+State x AGI-band calibration targets from IRS SOI Historic Table 2 (#940): return counts and AGI for each state's bands from $100k up, with $500k-$1M and $1M+ split, bound as shares of the state's published partition rebased onto the state total that binds. The US Chronicle feed is re-pinned to add the TY2023 state AGI-band facts (40,178 rows), and a release coverage requirement per state and measure demands each state's $1M+ AGI and return-count row.

docs/us-acs-local-soi-target-surface.md

@@ -66,6 +66,21 @@ Chronicle `b571381` consumer artifact gives the same counts.
 | `irs_soi` | 3,819 | 607 | 30,913 |
 | **Admin specs** | **3,972** | **760** | **31,066** |
 
+**Re-measured 2026-10-04 on `consumer_facts_us_51aa40f.jsonl`** (sha256
+`965a29ac…`, the #940 re-pin: the c5e5bf8 feed plus the TY2023 Historic
+Table 2 state AGI-band facts). The 408 state AGI-band rows (51 states x the
+four bands from $100k up x returns and AGI; `docs/us-fact-to-target.md`,
+"State AGI bands bind as shares of the state total") join `state` and `full`;
+their role is `soi_fiscal_distribution`, so `totals` is unchanged:
+
+| Family | `state` | `totals` | `full` |
+|---|---:|---:|---:|
+| `irs_soi` | 4,227 | 607 | 31,321 |
+| **Admin specs** | **4,380** | **760** | **31,474** |
+
+With the 487 population marginals `state` now calibrates to 4,867 targets.
+The rest of this section describes the 2026-09-22 measurement.
+
 The 487 population marginals (51 states and 436 congressional districts) are
 added on top in every mode, so `state` calibrates to 4,459 targets: Build P's
 set, and Build O's 4,461 minus the Vermont under-$1 taxable-interest pair the
@@ -80,7 +95,7 @@ Historic Table 2 state tables:
 | Record set spec | Specs | Content |
 |---|---:|---|
 | `irs_soi.historic_table_2.state_broad_totals.v1` | 2,397 | 47 all-income-range measures x 51 states, including AGI, income tax, and ACA premium tax credit returns and amounts |
-| `irs_soi.historic_table_2.state_agi_counts_and_amounts.v1` | 912 | taxable interest by AGI band |
+| `irs_soi.historic_table_2.state_agi_counts_and_amounts.v1` | 912 | taxable interest by AGI band (1,320 on the #940 feed: plus 408 TY2023 return-count and AGI bands) |
 | `irs_soi.historic_table_2.state_eitc.v1` | 510 | EITC returns and amounts by number of qualifying children |
 
 It holds no congressional-district SOI row and no row from the TY2023
@@ -307,15 +322,17 @@ change forces the same review.
 |---|---:|
 | `usda_snap` | 102 |
 | `cms_medicaid` (enrollment) | 51 |
-| `irs_soi` state, Historic Table 2 | 3,819 |
+| `irs_soi` state, Historic Table 2 | 4,227 |
 | `irs_soi` state, district file (district-file-only measures) | 302 |
 | `irs_soi` district (427 districts x 51 measures, less 34 banded) | 21,743 |
-| **Admin specs** | **26,017** |
+| **Admin specs** | **26,425** |
 
 Of the 21,743 district rows, 19,181 are rebased to a Historic Table 2 parent
 and 2,562 keep a district-file parent. The 2,189 (state, concept) district
 blocks each sum to their parent within 1e-9. Adding the 487 population
-marginals gives 26,504 targets, before the holdout. The feed-gated test
+marginals gives 26,912 targets, before the holdout. The Historic Table 2
+state count includes the 408 state AGI-band rows of #940 (3,819 and 26,017
+admin specs before the 2026-10-04 re-pin). The feed-gated test
 `test_pinned_feed_state_cd_surface_matches_its_contract` pins these counts
 and the reconciliation.
 

docs/us-chronicle-feed-repin.md

@@ -13,7 +13,33 @@ any other feed, and `test_us_chronicle_feed.py` fails if the parity resources,
 the generator and the pin disagree, or if the scope file changes without a
 new pin.
 
-## Why the feed moved
+## The 2026-10-04 re-pin: TY2023 state AGI bands (microcosm#940)
+
+The state x AGI-band targets of #940 need the TY2023 Historic Table 2 state
+file, whose $500k-$1M and $1M+ classes are separate facts
+(PolicyEngine/chronicle#291). The pin moved from `c5e5bf8` to `51aa40f`, a
+commit that is `c5e5bf8` plus only that package (Chronicle branch
+`feed/us-c5e5bf8-state-agi-2023`, tag `microcosm-us-feed-940-v1`), not to
+Chronicle main:
+
+- At Chronicle main the artifact-year restamp guard (chronicle#292) stops
+  building the TY2022 and TY2020 files at `--year 2023`, so ten scoped pairs
+  (the congressional-district file, the W-2 tips and 401(k) items, the IRA
+  tables, the `state_2022` US rows) produce no row, and chronicle#304 renames
+  the Historic Table 2 and district-file capital-gains concepts. Adopting
+  those is microcosm#1030's re-pin and needs its own decisions.
+- Built at `51aa40f`, the feed keeps all 39,158 `c5e5bf8` cells with equal
+  values and no changed field, and adds exactly the 1,020 TY2023 state AGI
+  cells (`irs_soi.ty2023.historic_table_2.state_agi.<st>`, 51 new scope
+  pairs). Compiled at 2024 with the packaged CD crosswalk, aging and the
+  Medicaid substitutions, every target other than the state AGI bands is
+  identical on both feeds, name, value and metadata; the bands go from 306
+  TY2022 rows to 408 TY2023 rows. The comparison script and its report are in
+  `experiments/940-state-agi-bands/feed-repin/`.
+- A later re-pin to Chronicle main carries the 51 pairs over: chronicle#291
+  adds the same package to main.
+
+## Why the feed moved (2026-09-18)
 
 `_validate_chronicle_hierarchy_labels`
 (`packages/microcosm-build/src/microcosm/build/ledger_targets.py`) requires
@@ -28,15 +54,20 @@ the first labelled US export.
 
 | Field | Value |
 |---|---|
-| Chronicle commit | `c5e5bf8aa84960c1a200ee47303b19c953092d0f` |
-| Feed file | `consumer_facts_us_c5e5bf8.jsonl`, 39,158 rows, 164,603,204 bytes |
-| `facts_sha256` | `b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801` |
+| Chronicle commit | `51aa40fd28e2ca81937752515e773d123fecfb71` (`c5e5bf8` plus the TY2023 state AGI package; tag `microcosm-us-feed-940-v1`) |
+| Feed file | `consumer_facts_us_51aa40f.jsonl`, 40,178 rows, 169,038,757 bytes |
+| `facts_sha256` | `965a29ac9458edb58a05184917ee4a19772fd3dc7b4588db6eb6da5803641aac` |
 | Consumer fact schema | `chronicle.consumer_fact.v3`, schema file sha256 `bdb51e2a…` (unchanged from the UK pin) |
-| Scope | 586 (record set, period) pairs; 62 package runs over build years 2020 to 2029 |
+| Scope | 637 (record set, period) pairs; 63 package runs over build years 2020 to 2029 |
 | Consumer artifact | none: refused at this commit, see below |
 
+The previous pin was `c5e5bf8aa84960c1a200ee47303b19c953092d0f`
+(`consumer_facts_us_c5e5bf8.jsonl`, 39,158 rows, `facts_sha256`
+`b8543739…`, 586 pairs, 62 runs); the rest of this document describes its
+2026-09-18 rebuild unless it says otherwise.
+
 The feed is too large for the repository. Its home on the build machine is
-`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_c5e5bf8.jsonl`,
+`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_51aa40f.jsonl`,
 beside the previous pins; `tools/build_us_target_parity_manifest.py` reads it
 there by default. A holder of the Chronicle commit regenerates it byte for
 byte with the commands below.

docs/us-fact-to-target.md

@@ -111,6 +111,81 @@ exclusion register instead.
   wiring is safe and normal (the keogh ALD facts rode the feed unmapped for
   weeks).
 
+### State AGI bands bind as shares of the state total (microcosm#940)
+
+A state's total return count and AGI leave its top tail free, and the top
+tail is what progressive state rate schedules tax: #940 measured Build P's
+ACS-local release within 0.12% of Colorado's SOI returns and AGI while
+holding 24% of Colorado's SOI AGI above $1M. IRS SOI Historic Table 2
+publishes each state's returns and AGI in ten AGI classes, including
+$500k–$1M and $1M+ separately. `_soi_reference_from_fact` rescues those
+bands from the cross-period refusal and `_rebase_soi_state_agi_bands`
+binds them:
+
+- **Which rows.** `return_count` and `adjusted_gross_income` from the
+  per-state HT2 AGI record sets (`irs_soi.<ty>.historic_table_2.state_agi.<st>`),
+  all filing statuses, bands from
+  `US_SOI_STATE_AGI_BAND_MINIMUM_LOWER_BOUND` ($100,000) up. The floor is the
+  national size-of-AGI floor for the same reason: the SOI slice materializer
+  counts every tax unit in a band, filer or not. The HT2 `us` rows never
+  qualify; the national AGI shape belongs to Table 1.1.
+- **The value.** `control x band / partition`. The partition is the sum of
+  the state's published bands of the same vintage, measure and record set
+  over the whole AGI line, negative AGI under $1 included; HT2 publishes
+  amounts that add exactly and counts rounded to tens. The control is the
+  state's HT2 all-returns total
+  (`irs_soi.<ty>.historic_table_2.state_broad.<st>.all.<measure>`) as the
+  registry binds it: the row latest-vintage selection kept and the
+  exclusions let through, read from the compiled registry, not from the
+  facts. A band's share of the total it calibrates beside is therefore the
+  share it records. A state whose total is excluded has no control and binds
+  no bands of that measure; two bound totals for one state and measure
+  raise. Congressional-district `<st>_total` rows never anchor a state.
+- **Periods.** The value lands at the control's period and ages with the
+  state total: AGI on the CBO AGI series, counts never. The bands therefore
+  stay the same share of the state total they bind beside, at every stage.
+  The control may be older than the bands (TY2023 bands on the TY2022 state
+  total): the newest published shares scale onto the level the state total
+  binds at, rather than binding a second, differently aged level.
+- **One vintage.** State bands are the one family selected by vintage, not
+  per row. Every eligible vintage (not after the build period) of a binding
+  band stays a candidate through latest-vintage selection, and
+  `_with_one_state_agi_band_vintage` then keeps, per state and measure, the
+  newest vintage whose published partition is complete. The TY2022 package's summed `500k_plus` row never binds beside
+  TY2023's split rows, and a gapped newest vintage falls back to the last
+  complete one. The choice is made inside selection, so the exclusion vintage
+  guard and the exclusion receipt see only the band ids that can bind.
+- **Malformed partitions.** In a vintage that is examined, overlapping
+  bands, a NaN bound, a value that is infinite, NaN or does not parse, or one
+  source record id with two different rows raise; an identical re-emission of a sub-floor band counts once. A
+  binding band's own fact is a target fact: like every other family, the
+  compile refuses two facts for one source record id, identical or not.
+  Vintages older than the one chosen are not examined.
+- **The floor holds at every period.** A state band below $100k, or for a
+  single filing status, is refused even in its own tax year, so a same-year
+  vintage can never bind raw sub-floor levels.
+- **Receipts.** Every rebased row carries `state_agi_band_share`,
+  `uprating_factor` and the control's record id.
+- **Release gate.** `US_FISCAL_TARGET_COVERAGE_REQUIREMENTS` carries one
+  requirement per state and measure,
+  `irs_state_agi_top_tail_<st>_<measure>` (102 in all), each matching that
+  state's `[$1M, inf)` row by `state_fips` and both AGI bounds. A feed
+  without the split bands, or a state whose partition or bound total is
+  missing for either measure, fails the release and names the state, rather
+  than shipping without the constraint; a second row for another state
+  cannot stand in. Together the two rows pin each state's AGI above $1M.
+- **Period contract.** Without aging, a band rebased onto an older state
+  total holds that total's period's dollars; `find_period_contract_violations`
+  reads `uprating_to_period` for rebased rows, the same period target aging
+  ages them from.
+- **Agreement with Table 1.1.** On the pinned feed the states' $500k–$1M and
+  $1M+ rows sum to 98–99% of Table 1.1's TY2023 classes aged the same way;
+  Table 1.1 also counts returns filed from other areas and Puerto Rico
+  (`test_pinned_feed_state_top_tail_agrees_with_table_1_1`).
+- **Support.** The rows can bind only where the pool has records in the
+  band. `experiments/940-state-agi-bands/` measures pre-calibration support
+  per state and band on the #982 fix's offline outputs.
+
 ## 4. Keep the exclusion register honest
 
 `US_FISCAL_TARGET_SUPPORT_EXCLUSIONS` is keyed by `source_record_id` and its