From 0f343249fd9a53816f336b9d34b811f38b650727 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 27 Sep 2026 04:55:28 -0400 Subject: [PATCH 1/8] Choose the capital-gains rebase control by concept and period, not feed order Historic Table 2 and congressional-district N01000 count Form 1040 line 7 returns (gain, loss-limited loss or distributions only); Table 1.4 col 37, the population capital_gains_gross measures, counts Schedule D gain returns only. The rebase control is now the latest Table 1.4 fact not after the build period, CD and HT2 rows never qualify, a tie between matched records refuses the compile, and a line-7 row without a usable control is dropped instead of shipping as a level. Rebased rows declare the bridge in soi_source_concept / soi_control_concept. On the pinned feed the 51 national_state HT2 state return counts go from 29,599,604 (2.39x the national 12,392,020 target) back to Build P's 12,289,836; amounts keep their values. national_state registry d315c75804ef -> 65e4dde11c83 (microcosm#1035). Co-Authored-By: Claude Opus 5.5 --- ...035-capital-gains-control-concept.fixed.md | 1 + docs/us-chronicle-feed-repin.md | 10 + docs/us-soi-capital-gains-concepts.md | 120 +++++ .../build/us_runtime/fiscal_targets.py | 139 ++++- .../tests/test_us_fiscal_targets.py | 484 +++++++++++++++++- 5 files changed, 728 insertions(+), 26 deletions(-) create mode 100644 changelog.d/1035-capital-gains-control-concept.fixed.md create mode 100644 docs/us-soi-capital-gains-concepts.md diff --git a/changelog.d/1035-capital-gains-control-concept.fixed.md b/changelog.d/1035-capital-gains-control-concept.fixed.md new file mode 100644 index 0000000000..0e190d99d2 --- /dev/null +++ b/changelog.d/1035-capital-gains-control-concept.fixed.md @@ -0,0 +1 @@ +Rebase Historic Table 2 capital-gains rows only onto the Table 1.4 Schedule D taxable-net-gain control, the concept `capital_gains_gross` measures. Feed order no longer picks the control: congressional-district and Historic Table 2 rows (Form 1040 line 7, gain or loss) never qualify, a tie between matched records refuses the compile, and a line-7 row with no usable control is dropped instead of shipping as a level. On the pinned feed the 51 `national_state` state return-count targets move from 29,599,604 (2.39x the national target) to 12,289,836 (microcosm#1035). diff --git a/docs/us-chronicle-feed-repin.md b/docs/us-chronicle-feed-repin.md index 14aee6d60c..8bf63b09cf 100644 --- a/docs/us-chronicle-feed-repin.md +++ b/docs/us-chronicle-feed-repin.md @@ -171,6 +171,16 @@ compiled register goes from 32,867 to 32,843 targets, and the count, the only change: 32,842 compiled targets and 5,694 in `national_state` (`d315c75804ef`). +**Erratum (27 September 2026, microcosm#1035).** Row order changed too. The +new feed is sorted by the re-derived `aggregate_fact_key`s, and the +capital-gains rebase kept the first of two equal-period controls. The +congressional-district US row (TY2022 data stamped ty2023) now sorted ahead of +Table 1.4 ty2023. It took over the returns control and raised the 51 Historic +Table 2 state return counts from 12,289,836 to 29,599,604. The control is now +chosen by concept and period alone +([us-soi-capital-gains-concepts.md](us-soi-capital-gains-concepts.md)). The +counts are unchanged and `national_state` is `65e4dde11c83`. + The labelled feed exposed one latent defect in microcosm, fixed in this change: `apply_us_medicaid_enrollment_substitutions` built Rhode Island's substituted spec by cloning a neighbouring state's spec and kept that state's diff --git a/docs/us-soi-capital-gains-concepts.md b/docs/us-soi-capital-gains-concepts.md new file mode 100644 index 0000000000..841d6fdf85 --- /dev/null +++ b/docs/us-soi-capital-gains-concepts.md @@ -0,0 +1,120 @@ +# SOI capital-gains concepts and the rebase control + +`capital_gains_gross` targets come from three IRS SOI products that count +different populations under the same measure ids (`net_capital_gains_returns`, +`net_capital_gains_amount`). This page records what each one counts, which +one the model measures, and how `fiscal_targets.py` combines them +(microcosm#1035). + +## What each table counts + +| Source | Columns | What it counts | +|---|---|---| +| Table 1.4 (`23in14ar.xls`) | col 37 / 38, "Sales of capital assets reported on Form 1040, Schedule D: Taxable net gain" | Schedule D returns that net to a gain, and that gain | +| Table 1.4 | col 39 / 40, "Taxable net loss" | Schedule D returns that net to a loss, and the loss after the $3,000 limit | +| Table 1.4 | col 35 / 36, "Capital gain distributions reported on Form 1040" | Returns that report only distributions, without Schedule D | +| Historic Table 2 (`22in55cmcsv.csv`), congressional-district file (`22incd.csv`) | N01000 / A01000 | "Number of returns with net capital gain (less loss)" and its amount, at Form 1040 line 7 (both documentation guides) | + +Line 7 carries a Schedule D gain, a loss-limited Schedule D loss, or +distributions reported without Schedule D (2022 Form 1040 instructions, line +7, Exception 1). Columns 35, 37 and 39 are disjoint: Table 1.3 row 18, "Sales +of capital assets net gain", equals col 35 + col 37 exactly in TY2022 and +TY2023. + +N01000 is the union of the three Table 1.4 populations: + +| Tax year | HT2 US N01000 | Table 1.4 col 35 + 37 + 39 | Difference | HT2 A01000 vs col 36 + 38 − 40 | +|---|---:|---:|---:|---:| +| 2020 | 29,008,620 | 29,003,885 | +0.016% | −0.63% | +| 2021 | 32,996,180 | 33,076,998 | −0.244% | −0.29% | +| 2022 | 30,465,850 | 30,461,045 | +0.016% | −0.17% | +| 2023 | 29,481,840 | 29,434,188 | +0.162% | −0.28% | + +In TY2022, col 37 (12,915,122) is 42.4% of N01000. The amounts are close +(A01000 is 1.4% below col 38), because distributions and capped losses are +small in dollars. The counts differ by 2.36x. + +The congressional-district file covers a subset of HT2: it leaves out +territories, APO/FPO and foreign addresses and returns without a matched ZIP +code. Its US N01000 is 98.0% of HT2's. + +## What the model measures + +`capital_gains_gross` maps to PE-US `capital_gains`, which is short-term plus +long-term gains, i.e. Schedule D before the loss limit. Distributions reported +without Schedule D are the separate `non_sch_d_capital_gains`, with its own +Table 1.4 col 35 target. The SOI materializer counts a tax unit when the +unit's summed `capital_gains` is positive and sums its positive part, so the +count is col 37 and the sum is col 38. The Build P release met col 37 at +12,391,446 against 12,392,020. + +No released record has negative capital gains, so the model has no Schedule D +loss returns. An N01000 count therefore has no model counterpart at any +level. + +## How the targets combine + +- **Controls.** Only a family registered with the model's concept + (`_SOI_CAPITAL_GAINS_FAMILY_CONCEPTS`, today Table 1.4 alone) can supply the + national level, and the latest national all-AGI fact not after the build + period wins. HT2 and congressional-district rows never do, whatever their + period stamp. Two different records at the winning period raise + `AmbiguousSoiCapitalGainsControlError`, so feed order never decides. +- **Historic Table 2 rows** enter only as shares. Each state row is its share + of the HT2 US total, scaled to the control, landed at the control's period, + and tagged `soi_source_concept = form_1040_line_7_net_gain_or_loss` and + `soi_control_concept = schedule_d_taxable_net_gain`. A row with no control + at or after its own period is dropped, because a line-7 level would be a + 2.36x mismatch. The HT2 national all-AGI rows retire because the control's + own row owns the national concept. +- **Congressional-district rows** are not controls. Their own targets, which + are on the `full` surface only, are not reconciled here. + +The share step assumes that each state's share of line-7 returns equals its +share of Schedule D gain returns. No IRS state product publishes a gain-only +count, so this cannot be checked directly. AGI composition is one measurable +source of difference: weighting each state's HT2 N01000 by AGI class with +Table 1.4's gain share per class would move TY2022 state targets by −2.9% (WV) +to +4.3% (DC), with a median of 1.6% and no state beyond 5%. + +## How the returns control drifted + +The rebase used to keep the first fact among equal-period candidates. The +July pin listed Table 1.4 ty2023 first, so Build P rebased the 51 state +return counts onto 12,392,020 (factor 0.406751165649407; states summing to +12,289,836). The September re-pin (`consumer_facts_us_c5e5bf8`) sorts rows by +`aggregate_fact_key`, a content hash. There, the congressional-district US row +sorts first; it carries TY2022 data stamped ty2023 (PolicyEngine/chronicle#117). +It became the control (factor 0.97964), and the states summed to 29,599,604, +2.39x the national target of the same model quantity on the same surface. The +amount control happened to stay on Table 1.4, but reversing the feed would +have moved it to the congressional-district row (factor 0.92455 instead of +0.77190). + +## Effect on the pinned feed + +Compiled at 2024 with target aging and the packaged congressional-district +crosswalk, then narrowed to `national_state`: + +| Targets | Before | After | +|---|---:|---:| +| 51 HT2 state return counts, sum | 29,599,604 | 12,289,836 | +| California | 3,774,052 | 1,566,997 | +| Texas | 2,108,705 | 875,540 | +| Florida | 2,093,099 | 869,060 | +| New York | 1,974,435 | 819,791 | +| National Table 1.4 return count (unchanged) | 12,392,020 | 12,392,020 | +| `national_state` registry | `d315c75804ef` | `65e4dde11c83` | + +- All 60 HT2 return-count specs move by the same factor, 0.415202720927. That + is the 51 state rows plus 9 congressional-district proxies, which are on + `full` only. +- The 60 matching amount specs keep their values and gain only the two + concept keys. +- No spec enters or leaves either surface, which keeps 32,842 compiled and + 5,694 on `national_state`. +- The state return counts are back at the Build P values; California's + 1,566,996.66 equals the July scorecard's. + +`test_pinned_feed_no_soi_state_family_sums_past_its_national_target` holds +the invariant across all 63 SOI state families on the surface. diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 1cf1d0ec54..4b5334f83c 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -45,6 +45,7 @@ from microcosm.calibrate.geography_constants import US_STATE_FIPS_TO_POSTAL __all__ = [ + "AmbiguousSoiCapitalGainsControlError", "US_FISCAL_MACRO_REALISM_BANDS", "US_FISCAL_TARGET_REGISTRY", "US_FISCAL_TARGET_SPECS", @@ -1471,13 +1472,62 @@ class _SoiTotalControl: period_key: tuple[int, int, str] +#: What an SOI record-set family's ``net_capital_gains_*`` columns count, read +#: from the published IRS workbooks and documentation guides, TY2020-TY2023 +#: (microcosm#1035): +#: +#: - Table 1.4 cols 37/38, "Sales of capital assets reported on Form 1040, +#: Schedule D: Taxable net gain", cover Schedule D returns that net to a +#: gain. Returns reporting only capital gain distributions on Form 1040 +#: (col 35) and Schedule D loss returns (col 39) are outside them: Table 1.3 +#: row 18 "Sales of capital assets net gain" equals col 35 + col 37 exactly. +#: - Historic Table 2 and the congressional-district file carry N01000/A01000, +#: which both documentation guides define as "Number of returns with net +#: capital gain (less loss)" / "Net capital gain (less loss) amount" at Form +#: 1040 line 7: a Schedule D gain, a loss-limited Schedule D loss, or +#: distributions only. HT2 US N01000 equals Table 1.4 col 35 + col 37 + +#: col 39 within 0.25% in each of TY2020-TY2023, and is 2.36x col 37 in +#: TY2022. Its amount, net of losses, is 1.4% below col 38 in TY2022. +#: +#: A family absent from this register has no reviewed concept and is never a +#: capital-gains control. +_SOI_SCHEDULE_D_TAXABLE_NET_GAIN = "schedule_d_taxable_net_gain" +_SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS = "form_1040_line_7_net_gain_or_loss" +_SOI_CAPITAL_GAINS_FAMILY_CONCEPTS: dict[str, str] = { + "table_1_4": _SOI_SCHEDULE_D_TAXABLE_NET_GAIN, + "historic_table_2": _SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS, + "congressional_district": _SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS, +} +#: The concept ``capital_gains_gross`` measures. It is the positive part of +#: PE-US ``capital_gains`` (short-term plus long-term, i.e. Schedule D, before +#: the loss limit); distributions reported without Schedule D are the separate +#: ``non_sch_d_capital_gains``. Its indicator therefore counts Table 1.4 col 37 +#: returns and its sum is col 38. The model carries no Schedule D loss returns, +#: so a line-7 count has no model counterpart at any level. +_SOI_CAPITAL_GAINS_MODEL_CONCEPT = _SOI_SCHEDULE_D_TAXABLE_NET_GAIN + + +class AmbiguousSoiCapitalGainsControlError(ValueError): + """Two different concept-matched records tie for one capital-gains control.""" + + def _rebase_stale_soi_capital_gains_distributions( registry: TargetRegistry, facts: tuple[object, ...], *, target_period: int | str, ) -> TargetRegistry: - """Use stale SOI capital-gains rows as shares, not hard old-year totals.""" + """Use Historic Table 2 capital-gains rows as shares of a Table 1.4 level. + + HT2 rows count Form 1040 line 7 (gain or loss), a population + ``capital_gains_gross`` cannot measure, so they never ship as levels. Each + one ships as its share of the HT2 national total, scaled to the + concept-matched control of ``_soi_capital_gains_active_totals`` at or after + its own period, and declares the bridge in ``soi_source_concept`` / + ``soi_control_concept``. Without such a control, the row is dropped. The + national all-AGI HT2 rows retire because the control's own row owns the + national concept. + """ controls = _soi_capital_gains_active_totals( facts, @@ -1494,15 +1544,11 @@ def _rebase_stale_soi_capital_gains_distributions( key = _soi_capital_gains_control_key_from_spec(spec) control = controls.get(key) source_total = stale_national_totals.get((*key, spec.metadata["source_period"])) - if control is None: - specs.append(spec) - continue - if source_total in (None, 0): - continue - if not _period_not_before( + if control is None or not _period_not_before( control.period_key, _period_key_from_value(spec.metadata["source_period"]) ): - specs.append(spec) + continue + if source_total in (None, 0): continue if _is_national_all_agi_spec(spec): @@ -1525,6 +1571,8 @@ def _rebase_stale_soi_capital_gains_distributions( "uprating_index_source_record_id": control.source_record_id, "uprating_factor": _format_float(factor), "stale_distribution_rebased_to_active_total": "true", + "soi_source_concept": _SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS, + "soi_control_concept": _SOI_CAPITAL_GAINS_MODEL_CONCEPT, }, ) ) @@ -1536,13 +1584,25 @@ def _soi_capital_gains_active_totals( *, target_period: int | str, ) -> dict[tuple[str, str, str], _SoiTotalControl]: + """The capital-gains control per (measure, filing status, universe). + + Only a national full-AGI fact whose family counts what + ``capital_gains_gross`` measures qualifies, so HT2 and + congressional-district rows never do, whatever their period stamp. The + latest qualifying period not after the build period wins, and the choice + depends on the fact set alone: two different records at the winning + period refuse the compile. Feed order used to break that tie, and the + September re-pin, which re-sorted the feed by content hash, handed the + returns control to the congressional-district US row (microcosm#1035). + """ controls: dict[tuple[str, str, str], _SoiTotalControl] = {} + tied_record_ids: dict[tuple[str, str, str], set[str]] = {} target_period_key = _period_key_from_value(target_period) for fact in facts: key = _soi_capital_gains_control_key_from_fact(fact) if key is None: continue - if _is_stale_soi_historic_capital_gains_fact(fact): + if _soi_capital_gains_concept(fact) != _SOI_CAPITAL_GAINS_MODEL_CONCEPT: continue period_key = _period_key(fact) if not _not_after_target_period(period_key, target_period_key): @@ -1550,22 +1610,59 @@ def _soi_capital_gains_active_totals( source_record_id = _source_record_id(fact) if not source_record_id: continue - candidate = _SoiTotalControl( - value=_numeric_value(fact), - source_period=str(_period_value(fact)), - source_record_id=source_record_id, - period_key=period_key, - ) current = controls.get(key) - if current is None or _prefer_candidate( - candidate.period_key, - current.period_key, - target_period_key=target_period_key, - ): - controls[key] = candidate + if current is not None and period_key[:2] == current.period_key[:2]: + tied_record_ids[key].add(source_record_id) + continue + if current is None or period_key[:2] > current.period_key[:2]: + controls[key] = _SoiTotalControl( + value=_numeric_value(fact), + source_period=str(_period_value(fact)), + source_record_id=source_record_id, + period_key=period_key, + ) + tied_record_ids[key] = {source_record_id} + ambiguous = { + key: sorted(record_ids) + for key, record_ids in tied_record_ids.items() + if len(record_ids) > 1 + } + if ambiguous: + raise AmbiguousSoiCapitalGainsControlError( + "SOI capital-gains control is ambiguous: different records tie at " + f"the latest period for {ambiguous}. Deduplicate them upstream; " + "feed order must not pick a rebase control (microcosm#1035)." + ) return controls +def _soi_record_set_family(record_set_id: str) -> str: + """An SOI record set's table family, without period or data vintage. + + ``irs_soi.ty2023.table_1_4`` is ``table_1_4``, + ``irs_soi.ty2022.historic_table_2.state_broad`` is ``historic_table_2`` and + ``irs_soi.ty2023.congressional_district_2022.all_returns`` is + ``congressional_district``. + """ + parts = record_set_id.split(".") + if parts[0] != "irs_soi": + return "" + rest = parts[2:] if len(parts) > 1 and _is_period_token(parts[1]) else parts[1:] + if not rest: + return "" + family = rest[0] + if family.startswith("congressional_district"): + return "congressional_district" + return family + + +def _soi_capital_gains_concept(fact: object) -> str | None: + """The reviewed IRS concept of a capital-gains fact's family, if any.""" + return _SOI_CAPITAL_GAINS_FAMILY_CONCEPTS.get( + _soi_record_set_family(_str_at(fact, "layout", "record_set_id")) + ) + + def _soi_capital_gains_stale_national_totals( facts: tuple[object, ...], ) -> dict[tuple[str, str, str, str], float]: diff --git a/packages/microcosm-build/tests/test_us_fiscal_targets.py b/packages/microcosm-build/tests/test_us_fiscal_targets.py index 3559eb8451..97b1d992a4 100644 --- a/packages/microcosm-build/tests/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/test_us_fiscal_targets.py @@ -1179,15 +1179,16 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ) -> None: """The release surface on the pinned feed: 32,842 compiled targets after Medicaid substitution, 5,694 national_state targets at registry - d315c75804ef, 32 CHIP rows none of them for an M-CHIP state, no + 65e4dde11c83, 32 CHIP rows none of them for an M-CHIP state, no other-income row and no tips return count. Before microcosm#956 it was - 32,867 / 5,719 at d5f9d854fe11, and before decision d179 dropped the - ty2020 tips return count it was 32,843 / 5,695 at 386fac439e77 - (docs/us-chronicle-feed-repin.md).""" + 32,867 / 5,719 at d5f9d854fe11, before decision d179 dropped the ty2020 + tips return count it was 32,843 / 5,695 at 386fac439e77, and before + microcosm#1035 moved the capital-gains control back to Table 1.4 it was + d315c75804ef at the same counts (docs/us-chronicle-feed-repin.md).""" registry, surface, _ = pinned_feed_national_state_surface assert len(registry.specs) == 32_842 assert len(surface.specs) == 5_694 - assert surface.version == "d315c75804ef" + assert surface.version == "65e4dde11c83" chip = [ spec for spec in surface.specs @@ -1214,6 +1215,101 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ) +def test_pinned_feed_capital_gains_rows_take_the_table_1_4_level( + pinned_feed_national_state_surface, +) -> None: + """The 51 Historic Table 2 state capital-gains rows take their level from + Table 1.4 ty2023, the concept capital_gains_gross measures, and not from + the congressional-district US row that shares its period stamp and sorts + first in the feed (microcosm#1035). The return counts sum to 12,289,836, + 99.2% of the national 12,392,020 target, as in the Build P release; with + the CD row as control they summed to 29,599,604.""" + _, surface, _ = pinned_feed_national_state_surface + expected = { + "net_capital_gains_returns": ( + "irs_soi.ty2023.table_1_4.all.net_capital_gains_returns", + "0.406751165649407", + 12_289_835.97216556, + ), + "net_capital_gains_amount": ( + "irs_soi.ty2023.table_1_4.all.net_capital_gains_amount", + "0.771900044145164", + None, + ), + } + for measure, (control_id, factor, total) in expected.items(): + rows = [ + spec + for spec in surface.specs + if ".historic_table_2.state_broad." in spec.name + and spec.metadata.get("source_measure_id") == measure + ] + assert len(rows) == 51 + national = next(spec for spec in surface.specs if spec.name == control_id) + for spec in rows: + assert spec.metadata["uprating_index_source_record_id"] == control_id + assert spec.metadata["uprating_factor"] == factor + assert spec.metadata["soi_source_concept"] == ( + "form_1040_line_7_net_gain_or_loss" + ) + assert spec.metadata["soi_control_concept"] == ( + "schedule_d_taxable_net_gain" + ) + state_total = sum(spec.value for spec in rows) + assert state_total < national.value + if total is not None: + assert math.isclose(state_total, total, rel_tol=1e-12) + + +def test_pinned_feed_no_soi_state_family_sums_past_its_national_target( + pinned_feed_national_state_surface, +) -> None: + """Invariant: the states partition the nation (HT2 state rows omit only + other areas and Puerto Rico), so on national_state no SOI state family may + sum past every national target of the same model quantity and filters. + Before microcosm#1035 the capital_gains_gross return counts, one of the 63 + compared families, summed to 2.39x theirs. Some quantities still carry + both a stale HT2 US row and a newer Table 1.x row, a few percent apart, so + the bound is the larger of them.""" + _, surface, _ = pinned_feed_national_state_surface + + def quantity(spec) -> tuple[object, ...]: + metadata = spec.metadata + return ( + metadata.get("variable"), + metadata.get("measure_mode"), + metadata.get("agi_lower_bound"), + metadata.get("agi_upper_bound"), + metadata.get("filing_status"), + metadata.get("soi_return_universe", "all_returns"), + metadata.get("itemized_only"), + tuple( + sorted( + (key, value) + for key, value in metadata.items() + if key.startswith("ledger_filter_") + ) + ), + ) + + national: dict[tuple[object, ...], list[float]] = {} + states: dict[tuple[object, ...], float] = {} + for spec in surface.specs: + if spec.family != "irs_soi": + continue + if spec.metadata.get("state_fips"): + states[quantity(spec)] = states.get(quantity(spec), 0.0) + spec.value + else: + national.setdefault(quantity(spec), []).append(spec.value) + compared = {key: total for key, total in states.items() if key in national} + assert len(compared) == 63 + assert { + key[:2]: total / max(national[key]) + for key, total in compared.items() + if total > max(national[key]) + } == {} + + def test_reviewed_zero_support_facts_are_not_active_targets() -> None: excluded_source_record_id = ( "hhs_acf_tanf.fy2024.cash_assistance.ar." @@ -4506,6 +4602,384 @@ def test_stale_soi_capital_gains_without_source_total_is_dropped() -> None: assert state_record_id not in source_record_ids +_CG_T14_RETURNS = "irs_soi.ty2023.table_1_4.all.net_capital_gains_returns" +_CG_T14_AMOUNT = "irs_soi.ty2023.table_1_4.all.net_capital_gains_amount" +_CG_CD_RECORD_SET = "irs_soi.ty2023.congressional_district_2022.all_returns" +_CG_HT2_CA_RETURNS = ( + "irs_soi.ty2022.historic_table_2.state_broad.ca.all.net_capital_gains_returns" +) +_CG_HT2_CA_AMOUNT = ( + "irs_soi.ty2022.historic_table_2.state_broad.ca.all.net_capital_gains_amount" +) + + +def _capital_gains_ht2_facts() -> list[dict[str, object]]: + """TY2022 Historic Table 2 line-7 rows: US 100 / 1,000 and CA 25 / 250.""" + return [ + _soi_capital_gains_fact( + 2022, + source_record_id=( + "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_returns" + ), + measure_id="net_capital_gains_returns", + value=100.0, + ), + _soi_capital_gains_fact( + 2022, + source_record_id=( + "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_amount" + ), + value=1_000.0, + ), + _soi_capital_gains_fact( + 2022, + source_record_id=_CG_HT2_CA_RETURNS, + measure_id="net_capital_gains_returns", + geography_level="state", + geography_id="0400000US06", + value=25.0, + ), + _soi_capital_gains_fact( + 2022, + source_record_id=_CG_HT2_CA_AMOUNT, + geography_level="state", + geography_id="0400000US06", + value=250.0, + ), + ] + + +def _capital_gains_cd_us_facts() -> list[dict[str, object]]: + """The congressional-district file's US row: TY2022 line-7 data stamped + ty2023, as on the pinned feed (PolicyEngine/chronicle#117).""" + return [ + _soi_capital_gains_fact( + 2023, + source_record_id=f"{_CG_CD_RECORD_SET}.us.net_capital_gains_returns", + measure_id="net_capital_gains_returns", + layout_record_set_id=_CG_CD_RECORD_SET, + value=98.0, + ), + _soi_capital_gains_fact( + 2023, + source_record_id=f"{_CG_CD_RECORD_SET}.us.net_capital_gains_amount", + layout_record_set_id=_CG_CD_RECORD_SET, + value=925.0, + ), + ] + + +def _capital_gains_t14_facts() -> list[dict[str, object]]: + """Table 1.4 ty2023 Schedule D taxable net gain: 40 returns / 800.""" + return [ + _soi_capital_gains_fact( + 2023, + source_record_id=_CG_T14_RETURNS, + measure_id="net_capital_gains_returns", + layout_record_set_id="irs_soi.ty2023.table_1_4", + value=40.0, + ), + _soi_capital_gains_fact( + 2023, + source_record_id=_CG_T14_AMOUNT, + layout_record_set_id="irs_soi.ty2023.table_1_4", + value=800.0, + ), + ] + + +@pytest.mark.parametrize("cd_first", [True, False], ids=["cd-first", "t14-first"]) +def test_capital_gains_controls_are_table_1_4_in_either_feed_order( + cd_first: bool, +) -> None: + """A congressional-district US row that shares Table 1.4's period never + becomes the control, whichever comes first in the feed (microcosm#1035). + On the pinned feed it sorted first and set the HT2 state returns rows at + 2.39x the national Table 1.4 target of the same model quantity.""" + cd, t14 = _capital_gains_cd_us_facts(), _capital_gains_t14_facts() + facts = [ + *packaged_reference_facts(), + *_capital_gains_ht2_facts(), + *(cd + t14 if cd_first else t14 + cd), + ] + + controls = fiscal_targets._soi_capital_gains_active_totals( + tuple(facts), target_period=2024 + ) + assert {key[0]: control.source_record_id for key, control in controls.items()} == { + "net_capital_gains_returns": _CG_T14_RETURNS, + "net_capital_gains_amount": _CG_T14_AMOUNT, + } + + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + specs = {spec.metadata["ledger_source_record_id"]: spec for spec in registry.specs} + ca_returns = specs[_CG_HT2_CA_RETURNS] + ca_amount = specs[_CG_HT2_CA_AMOUNT] + assert ca_returns.value == 10.0 + assert ca_returns.metadata["uprating_factor"] == "0.4" + assert ca_returns.metadata["uprating_index_source_record_id"] == _CG_T14_RETURNS + assert ca_amount.value == 200.0 + assert ca_amount.metadata["uprating_index_source_record_id"] == _CG_T14_AMOUNT + for spec in (ca_returns, ca_amount): + # The bridge is declared: the row supplies a line-7 share, the level + # comes from the Schedule D concept capital_gains_gross measures. + assert spec.metadata["soi_source_concept"] == ( + "form_1040_line_7_net_gain_or_loss" + ) + assert spec.metadata["soi_control_concept"] == "schedule_d_taxable_net_gain" + + +def test_capital_gains_rows_without_a_concept_matched_control_are_dropped() -> None: + """With only a congressional-district row stamped later, no control + exists, and a line-7 HT2 row cannot stand as a capital_gains_gross level.""" + facts = [ + *packaged_reference_facts(), + *_capital_gains_ht2_facts(), + *_capital_gains_cd_us_facts(), + ] + + assert ( + fiscal_targets._soi_capital_gains_active_totals( + tuple(facts), target_period=2024 + ) + == {} + ) + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + source_record_ids = { + spec.metadata["ledger_source_record_id"] for spec in registry.specs + } + assert _CG_HT2_CA_RETURNS not in source_record_ids + assert _CG_HT2_CA_AMOUNT not in source_record_ids + + +def test_capital_gains_rows_whose_control_predates_them_are_dropped() -> None: + """A control older than the HT2 row would reverse its period, so the row + drops rather than ship its line-7 level.""" + facts = [ + *packaged_reference_facts(), + *_capital_gains_ht2_facts(), + _soi_capital_gains_fact( + 2021, + source_record_id="irs_soi.ty2021.table_1_4.all.net_capital_gains_returns", + measure_id="net_capital_gains_returns", + layout_record_set_id="irs_soi.ty2021.table_1_4", + value=41.0, + ), + ] + + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + source_record_ids = { + spec.metadata["ledger_source_record_id"] for spec in registry.specs + } + assert _CG_HT2_CA_RETURNS not in source_record_ids + + +def test_capital_gains_control_refuses_two_matched_records_at_one_period() -> None: + duplicate = _soi_capital_gains_fact( + 2023, + source_record_id="irs_soi.ty2023.table_1_4.all_returns.net_capital_gains_returns", + measure_id="net_capital_gains_returns", + layout_record_set_id="irs_soi.ty2023.table_1_4", + value=41.0, + ) + facts = (*_capital_gains_t14_facts(), duplicate) + + for ordered in (facts, tuple(reversed(facts))): + with pytest.raises( + fiscal_targets.AmbiguousSoiCapitalGainsControlError, + match="all_returns.net_capital_gains_returns", + ): + fiscal_targets._soi_capital_gains_active_totals(ordered, target_period=2024) + # The same record twice is one candidate, not a tie. + same = (*_capital_gains_t14_facts(), *_capital_gains_t14_facts()) + assert ( + len(fiscal_targets._soi_capital_gains_active_totals(same, target_period=2024)) + == 2 + ) + + +@pytest.mark.parametrize( + ("record_set_id", "family"), + [ + ("irs_soi.ty2023.table_1_4", "table_1_4"), + ("irs_soi.table_1_4", "table_1_4"), + ("irs_soi.ty2022.historic_table_2.us", "historic_table_2"), + ("irs_soi.ty2022.historic_table_2.state_broad", "historic_table_2"), + (_CG_CD_RECORD_SET, "congressional_district"), + ( + "irs_soi.ty2024.congressional_district_2024.all_returns", + "congressional_district", + ), + ("irs_soi.congressional_district_2022.all_returns", "congressional_district"), + ("irs_soi.ty2023.table_4_3.all_returns_excluding_dependents", "table_4_3"), + ("irs_soi.ty2023", ""), + ("cbo.revenue_projection.ty2023", ""), + ("", ""), + ], +) +def test_soi_record_set_family_drops_period_and_data_vintage( + record_set_id: str, family: str +) -> None: + assert fiscal_targets._soi_record_set_family(record_set_id) == family + + +def test_capital_gains_control_is_order_free_latest_and_concept_matched() -> None: + """Property (microcosm#1035): for any set of national capital-gains facts + and any build period, the control per measure is the latest Table 1.4 + fact not after the build period, whatever the feed order, and no + Historic Table 2, congressional-district or unreviewed-family fact is ever + chosen. With no such Table 1.4 fact there is no control.""" + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + record_sets = { + "table_1_4": "irs_soi.ty{period}.table_1_4", + "historic_table_2": "irs_soi.ty{period}.historic_table_2.us", + "congressional_district": ( + "irs_soi.ty{period}.congressional_district_2022.all_returns" + ), + "unreviewed": "irs_soi.ty{period}.table_4_3", + } + measures = ("net_capital_gains_returns", "net_capital_gains_amount") + candidate = st.tuples( + st.sampled_from(sorted(record_sets)), + st.integers(min_value=2018, max_value=2027), + st.sampled_from(measures), + st.floats(min_value=1.0, max_value=1e12, allow_nan=False), + ) + + @settings(max_examples=300, deadline=None) + @given( + st.lists(candidate, max_size=14, unique_by=lambda item: item[:3]), + st.integers(min_value=2019, max_value=2026), + st.randoms(use_true_random=False), + ) + def check(candidates, target_period, rng) -> None: + facts = [ + _soi_capital_gains_fact( + period, + source_record_id=( + f"{record_sets[family].format(period=period)}.us.{measure}" + ), + measure_id=measure, + layout_record_set_id=record_sets[family].format(period=period), + value=value, + ) + for family, period, measure, value in candidates + ] + shuffled = list(facts) + rng.shuffle(shuffled) + + controls = fiscal_targets._soi_capital_gains_active_totals( + tuple(facts), target_period=target_period + ) + assert controls == fiscal_targets._soi_capital_gains_active_totals( + tuple(shuffled), target_period=target_period + ) + chosen = {key[0]: control for key, control in controls.items()} + for measure in measures: + eligible = [ + (period, value) + for family, period, candidate_measure, value in candidates + if family == "table_1_4" + and candidate_measure == measure + and period <= target_period + ] + if not eligible: + assert measure not in chosen + continue + period, value = max(eligible) + control = chosen[measure] + assert control.source_period == str(period) + assert control.value == value + assert control.source_record_id == ( + f"irs_soi.ty{period}.table_1_4.us.{measure}" + ) + + check() + + +def test_rebased_capital_gains_states_never_outgrow_their_control() -> None: + """Property: whatever the HT2 state rows and national total, rebased + state rows keep their HT2 shares and so sum to the control times the + states' share of the HT2 nation; when the HT2 states sum to at most the + HT2 nation, they sum to at most the control.""" + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + states = (("ca", "06"), ("ny", "36"), ("tx", "48")) + + @settings(max_examples=25, deadline=None) + @given( + st.lists( + st.integers(min_value=1, max_value=10_000_000), + min_size=len(states), + max_size=len(states), + ), + st.integers(min_value=0, max_value=5_000_000), + st.integers(min_value=1, max_value=50_000_000), + ) + def check(state_values, other_areas, control_value) -> None: + national = sum(state_values) + other_areas + facts = [ + *packaged_reference_facts(), + _soi_capital_gains_fact( + 2022, + source_record_id=( + "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_returns" + ), + measure_id="net_capital_gains_returns", + value=float(national), + ), + *( + _soi_capital_gains_fact( + 2022, + source_record_id=( + "irs_soi.ty2022.historic_table_2.state_broad." + f"{postal}.all.net_capital_gains_returns" + ), + measure_id="net_capital_gains_returns", + geography_level="state", + geography_id=f"0400000US{fips}", + value=float(value), + ) + for (postal, fips), value in zip(states, state_values, strict=True) + ), + _soi_capital_gains_fact( + 2023, + source_record_id=_CG_T14_RETURNS, + measure_id="net_capital_gains_returns", + layout_record_set_id="irs_soi.ty2023.table_1_4", + value=float(control_value), + ), + ] + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + rebased = [ + spec + for spec in registry.specs + if ".historic_table_2.state_broad." in spec.name + and spec.metadata.get("source_measure_id") == "net_capital_gains_returns" + ] + assert len(rebased) == len(states) + total = sum(spec.value for spec in rebased) + assert math.isclose( + total, control_value * sum(state_values) / national, rel_tol=1e-12 + ) + assert total <= control_value * (1 + 1e-12) + + check() + + def test_cross_period_soi_taxable_interest_agi_slice_without_total_is_dropped() -> None: source_record_id = ( "irs_soi.ty2022.historic_table_2.us.200k_to_500k.taxable_interest_amount" From 6ebfcf7f4f90bc30fbea63cddb40eff67ed97f2d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 27 Sep 2026 05:22:06 -0400 Subject: [PATCH 2/8] Refuse any ambiguous capital-gains control; qualify two claims (#1036 review) The chooser now treats one record id carrying two values or period labels at the winning period as ambiguous too, so no feed order can pick the value. Tests cover that, an equivalent period label, and a tie at a superseded period. The model-concept comment scopes "no loss returns" to the Build P release data, and the doc's AGI-mix median is the median absolute change. Co-Authored-By: Claude Opus 5.5 --- docs/us-soi-capital-gains-concepts.md | 2 +- .../build/us_runtime/fiscal_targets.py | 39 +++++++------- .../tests/test_us_fiscal_targets.py | 51 +++++++++++++++++++ 3 files changed, 74 insertions(+), 18 deletions(-) diff --git a/docs/us-soi-capital-gains-concepts.md b/docs/us-soi-capital-gains-concepts.md index 841d6fdf85..4475329c53 100644 --- a/docs/us-soi-capital-gains-concepts.md +++ b/docs/us-soi-capital-gains-concepts.md @@ -75,7 +75,7 @@ share of Schedule D gain returns. No IRS state product publishes a gain-only count, so this cannot be checked directly. AGI composition is one measurable source of difference: weighting each state's HT2 N01000 by AGI class with Table 1.4's gain share per class would move TY2022 state targets by −2.9% (WV) -to +4.3% (DC), with a median of 1.6% and no state beyond 5%. +to +4.3% (DC), with a median absolute change of 1.6% and no state beyond 5%. ## How the returns control drifted diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 4b5334f83c..acd4751ddb 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -1502,8 +1502,10 @@ class _SoiTotalControl: #: PE-US ``capital_gains`` (short-term plus long-term, i.e. Schedule D, before #: the loss limit); distributions reported without Schedule D are the separate #: ``non_sch_d_capital_gains``. Its indicator therefore counts Table 1.4 col 37 -#: returns and its sum is col 38. The model carries no Schedule D loss returns, -#: so a line-7 count has no model counterpart at any level. +#: returns and its sum is col 38; a line-7 count, which also counts losses and +#: distributions-only returns, is a different population. (PE-US allows +#: negative gains, but the Build P release data carries no tax unit with a +#: net loss, so it could not reach a line-7 count at any level.) _SOI_CAPITAL_GAINS_MODEL_CONCEPT = _SOI_SCHEDULE_D_TAXABLE_NET_GAIN @@ -1590,13 +1592,14 @@ def _soi_capital_gains_active_totals( ``capital_gains_gross`` measures qualifies, so HT2 and congressional-district rows never do, whatever their period stamp. The latest qualifying period not after the build period wins, and the choice - depends on the fact set alone: two different records at the winning - period refuse the compile. Feed order used to break that tie, and the + depends on the fact set alone: two different candidates at the winning + period (different records, or one record with different values or period + labels) refuse the compile. Feed order used to break that tie, and the September re-pin, which re-sorted the feed by content hash, handed the returns control to the congressional-district US row (microcosm#1035). """ controls: dict[tuple[str, str, str], _SoiTotalControl] = {} - tied_record_ids: dict[tuple[str, str, str], set[str]] = {} + tied: dict[tuple[str, str, str], set[tuple[str, float, str]]] = {} target_period_key = _period_key_from_value(target_period) for fact in facts: key = _soi_capital_gains_control_key_from_fact(fact) @@ -1610,26 +1613,28 @@ def _soi_capital_gains_active_totals( source_record_id = _source_record_id(fact) if not source_record_id: continue + candidate = _SoiTotalControl( + value=_numeric_value(fact), + source_period=str(_period_value(fact)), + source_record_id=source_record_id, + period_key=period_key, + ) + identity = (source_record_id, candidate.value, candidate.source_period) current = controls.get(key) if current is not None and period_key[:2] == current.period_key[:2]: - tied_record_ids[key].add(source_record_id) + tied[key].add(identity) continue if current is None or period_key[:2] > current.period_key[:2]: - controls[key] = _SoiTotalControl( - value=_numeric_value(fact), - source_period=str(_period_value(fact)), - source_record_id=source_record_id, - period_key=period_key, - ) - tied_record_ids[key] = {source_record_id} + controls[key] = candidate + tied[key] = {identity} ambiguous = { - key: sorted(record_ids) - for key, record_ids in tied_record_ids.items() - if len(record_ids) > 1 + key: sorted(candidates) + for key, candidates in tied.items() + if len(candidates) > 1 } if ambiguous: raise AmbiguousSoiCapitalGainsControlError( - "SOI capital-gains control is ambiguous: different records tie at " + "SOI capital-gains control is ambiguous: different candidates tie at " f"the latest period for {ambiguous}. Deduplicate them upstream; " "feed order must not pick a rebase control (microcosm#1035)." ) diff --git a/packages/microcosm-build/tests/test_us_fiscal_targets.py b/packages/microcosm-build/tests/test_us_fiscal_targets.py index 97b1d992a4..d6508f4840 100644 --- a/packages/microcosm-build/tests/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/test_us_fiscal_targets.py @@ -4804,6 +4804,57 @@ def test_capital_gains_control_refuses_two_matched_records_at_one_period() -> No ) +def _t14_returns_fact(period: object, *, record_id: str, value: float) -> dict: + fact = _soi_capital_gains_fact( + 2023, + source_record_id=record_id, + measure_id="net_capital_gains_returns", + layout_record_set_id="irs_soi.ty2023.table_1_4", + value=value, + ) + fact["period"] = {**fact["period"], "value": period} + return fact + + +@pytest.mark.parametrize( + "rival", + [ + # One record id carrying two values: order would pick the value. + {"period": 2023, "record_id": _CG_T14_RETURNS, "value": 41.0}, + # Another record at the same period under an equivalent label. + { + "period": "tax_year_2023", + "record_id": "irs_soi.ty2023.table_1_4.v2.net_capital_gains_returns", + "value": 40.0, + }, + ], + ids=["same-id-other-value", "equivalent-period-label"], +) +def test_capital_gains_control_refuses_any_ambiguous_candidate(rival) -> None: + control = _t14_returns_fact(2023, record_id=_CG_T14_RETURNS, value=40.0) + other = _t14_returns_fact( + rival["period"], record_id=rival["record_id"], value=rival["value"] + ) + for ordered in ((control, other), (other, control)): + with pytest.raises(fiscal_targets.AmbiguousSoiCapitalGainsControlError): + fiscal_targets._soi_capital_gains_active_totals(ordered, target_period=2024) + + +def test_capital_gains_tie_at_a_superseded_period_is_not_ambiguous() -> None: + """Only a tie at the winning period matters: a later control supersedes + an older tie, in any order.""" + older = ( + _t14_returns_fact(2022, record_id="irs_soi.ty2022.table_1_4.a", value=1.0), + _t14_returns_fact(2022, record_id="irs_soi.ty2022.table_1_4.b", value=2.0), + ) + latest = _t14_returns_fact(2023, record_id=_CG_T14_RETURNS, value=40.0) + for ordered in ((*older, latest), (latest, *older), (older[0], latest, older[1])): + (control,) = fiscal_targets._soi_capital_gains_active_totals( + ordered, target_period=2024 + ).values() + assert (control.source_record_id, control.value) == (_CG_T14_RETURNS, 40.0) + + @pytest.mark.parametrize( ("record_set_id", "family"), [ From 011a63b1ad45da8364d7e463fabf327c76d757b9 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 27 Sep 2026 17:02:20 -0400 Subject: [PATCH 3/8] Rebase congressional-district capital gains onto Table 1.4; drop mislabeled CD columns (#1038) On the full target surface the congressional-district file's capital-gains rows compiled as Form 1040 line-7 levels onto capital_gains_gross (Schedule D gain): its states and districts each summed to 29,845,710 returns, 2.41x the Table 1.4 target, and $1,522.2B, 1.198x. They now enter as shares of the file's own US row scaled to the Table 1.4 control, as Historic Table 2 rows do since #1036; the US row retires and every share row names its denominator in soi_share_total_source_record_id. The same package reads N18425/A18425 (state and local income taxes) as the limited SALT deduction and N85530/A85530 (additional Medicare tax) as the premium tax credit. US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS drops facts by (measure id, IRS column), replaces the hard-coded CD PTC-amount check, and is listed in the exclusion receipt and the release coverage report. On the pinned feed: 31,376 compiled targets (was 32,842), no CD-file family past its national target on full (was 3), national_state values and membership unchanged (version moves only for the new provenance key). Co-Authored-By: Claude Opus 5.5 --- .../1038-cd-capital-gains-shares.fixed.md | 1 + docs/us-chronicle-feed-repin.md | 11 + docs/us-soi-capital-gains-concepts.md | 109 ++- .../microcosm/build/us_runtime/__init__.py | 2 + .../build/us_runtime/fiscal_targets.py | 239 ++++-- .../engine_free/us/test_us_fiscal_targets.py | 742 +++++++++++++++++- .../microcosm_build/us_fiscal_targets.py | 52 +- tools/build_us_fiscal_refresh_release.py | 11 + 8 files changed, 1069 insertions(+), 98 deletions(-) create mode 100644 changelog.d/1038-cd-capital-gains-shares.fixed.md diff --git a/changelog.d/1038-cd-capital-gains-shares.fixed.md b/changelog.d/1038-cd-capital-gains-shares.fixed.md new file mode 100644 index 0000000000..07af399c0b --- /dev/null +++ b/changelog.d/1038-cd-capital-gains-shares.fixed.md @@ -0,0 +1 @@ +On the `full` target surface, congressional-district capital-gains rows now enter as shares of the file's own US row scaled to the Table 1.4 Schedule D control, as Historic Table 2 rows do; the file's US row retires. Every capital-gains share row names its denominator in `soi_share_total_source_record_id`. A new reviewed source-column exclusion, `US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`, drops congressional-district facts read from IRS columns their measure id does not mean: N18425/A18425 (state and local income taxes) read as the limited SALT deduction, and N85530/A85530 (additional Medicare tax) read as the premium tax credit. It replaces the hard-coded congressional-district PTC-amount check and is listed in the exclusion receipt. On the pinned feed, the congressional-district capital-gains states and districts each move from 29,845,710 returns (2.41x the national target) and $1,522.2B (1.198x) to 12,392,020 and $1,270.9B, and no congressional-district family sums past its national target on `full` (microcosm#1038). diff --git a/docs/us-chronicle-feed-repin.md b/docs/us-chronicle-feed-repin.md index 907fbefc79..8d2dab4ced 100644 --- a/docs/us-chronicle-feed-repin.md +++ b/docs/us-chronicle-feed-repin.md @@ -181,6 +181,17 @@ chosen by concept and period alone ([us-soi-capital-gains-concepts.md](us-soi-capital-gains-concepts.md)). The counts are unchanged and `national_state` is `65e4dde11c83`. +**Follow-up (27 September 2026, microcosm#1038).** On the `full` surface, +the congressional-district file's capital-gains rows are now shares of its US +row scaled to the same control, and the US row retires. The package's four +mislabeled columns are dropped: N18425/A18425 (state and local income taxes), +read as the limited SALT deduction, and N85530/A85530 (additional Medicare +tax), read as the premium tax credit. That leaves 31,376 compiled targets, all of them on +`full` (`059dc56d78db`). `national_state` keeps its 5,694 targets and values. +It moves to `47dce807b412` only because its 102 Historic Table 2 +capital-gains rows now name their share denominator in +`soi_share_total_source_record_id`. + The labelled feed exposed one latent defect in microcosm, fixed in this change: `apply_us_medicaid_enrollment_substitutions` built Rhode Island's substituted spec by cloning a neighbouring state's spec and kept that state's diff --git a/docs/us-soi-capital-gains-concepts.md b/docs/us-soi-capital-gains-concepts.md index 4475329c53..784cdecd72 100644 --- a/docs/us-soi-capital-gains-concepts.md +++ b/docs/us-soi-capital-gains-concepts.md @@ -67,8 +67,13 @@ level. at or after its own period is dropped, because a line-7 level would be a 2.36x mismatch. The HT2 national all-AGI rows retire because the control's own row owns the national concept. -- **Congressional-district rows** are not controls. Their own targets, which - are on the `full` surface only, are not reconciled here. +- **Congressional-district rows** are not controls either, and their own + targets (on the `full` surface only) enter the same way: each state and + district row is its share of the congressional-district file's US row, + scaled to the control and tagged with the same two concept keys, and the + file's US row retires (microcosm#1038, below). Every share row names its + denominator, the HT2 or congressional-district US row, in + `soi_share_total_source_record_id`. The share step assumes that each state's share of line-7 returns equals its share of Schedule D gain returns. No IRS state product publishes a gain-only @@ -118,3 +123,103 @@ crosswalk, then narrowed to `national_state`: `test_pinned_feed_no_soi_state_family_sums_past_its_national_target` holds the invariant across all 63 SOI state families on the surface. + +## Congressional-district rows on the full surface (microcosm#1038) + +The release tool's default `--target-surface` is `full`, which also carries +the congressional-district file's rows: a US row, 51 state rows and 436 +current districts per measure. Before microcosm#1038 they compiled as line-7 +levels onto `capital_gains_gross`. On the pinned feed, their states and their +districts each summed to 29,845,710 returns, 2.41x the national Table 1.4 +target of 12,392,020 on the same surface. Their amounts summed to $1,522.2B +against $1,270.9B, 1.198x. The amount gap is mostly the file's TY2022 data +stamped ty2023 and aged from 2023. The count gap is the concept. + +There were two ways to fix this. Option (a) rebases the rows as shares, like +the HT2 rows. Option (b) drops them with a reviewed exclusion. Microcosm#1038 +takes (a): + +- **The share assumption is already the HT2 one.** Each district's share of + line-7 returns stands in for its share of Schedule D gain returns. AGI + composition is the measurable source of difference. Reweighting each of the + 428 numbered districts' N01000 by AGI class with Table 1.4's per-class gain + share moves its share by a median 2.1% (95th percentile 4.7%, largest + CA-18 +9.7%). Amounts move by a median 0.5% (95th percentile 2.4%). +- **Dropping the rows would discard a much larger signal.** District + capital-gains incidence varies far more than state incidence. N01000 per + return runs from 8.3% to 31.5% between the 5th and 95th percentile + districts, against 13.7% to 23.4% across states. Within a state, the + district share left over after the district's AGI distribution has a log + standard deviation of 0.227 for counts and 0.223 for amounts. The + AGI-composition error of the share proxy is 0.024 and 0.009. +- **The rebased rows agree with the HT2 state rows about as well as other + pairs on `full` already do.** The congressional-district and HT2 state + return counts for capital gains differ by a median 0.6% (95th percentile + 2.1%). The `return_count` pair differs by 1.6% (2.4%). + - Amounts differ more: a median 2.8% (95th percentile 15.4%, Montana + −27.0%). The cause is item suppression in the congressional-district + file's A01000. Montana's is 32.5% below HT2's at the state level. That is + still inside the qualified-dividends amount pair, 11.6% (21.7%). + +The congressional-district file has no rows for other areas, so its US row +equals the sum of its 51 states. The rebased states and districts therefore +each sum to the control itself. HT2's US row includes other areas, so its +rebased states sum to 99.2% of the control for returns and 98.0% for +amounts. + +**Vintage.** A share divides a row by the US row of the same package, so the +package's period stamp cancels. The value lands at the control's period +(2023) and ages from there. It is the same whether 22incd.csv is stamped +ty2023, as on the pinned feed, or truthfully at 2022 +(`test_congressional_district_capital_gains_ignore_the_package_stamp`). + +microcosm#1030 corrects restamped facts to age from their data year, but it +refuses any restamped fact that was rebased. Before it merges with this change, +that refusal must admit a share whose `soi_share_total_source_record_id` is a +restamp of the same package, and leave `uprating_to_period` at the control's +period. Starting the aging at 2022 instead would apply the Table 1.4 +2022-to-2023 fall twice (x0.761). + +### Effect on the pinned feed + +Release compile at 2024, as above: + +| `full` surface | Before | After | +|---|---:|---:| +| Congressional-district capital-gains returns, states and districts (each sum) | 29,845,710 | 12,392,020 | +| Congressional-district capital-gains amounts, states and districts (each sum) | $1,522.19B | $1,270.86B | +| California returns / amount | 3,808,930 / $214.08B | 1,581,478 / $178.73B | +| New York returns / amount | 2,015,540 / $127.96B | 836,858 / $106.83B | +| Compiled targets, all on `full` | 32,842 (`e02123644d42`) | 31,376 (`059dc56d78db`) | +| `national_state` targets | 5,694 (`65e4dde11c83`) | 5,694 (`47dce807b412`) | + +- The 974 congressional-district capital-gains rows move by one factor per + measure: 0.415202720927 for returns and 0.834893818418 for amounts. +- The two US rows retire. +- The Historic Table 2 rows keep their values. The 120 of them (102 on + `national_state`) gain only `soi_share_total_source_record_id`. + +### Two mislabeled columns in the same package + +On the same surface, Chronicle's `congressional_district_2022` package reads +four IRS columns under measure ids whose concept they do not carry. They are +the only differences between its column mapping and the Historic Table 2 +package's (all 56 measures compared): + +| Measure id | Congressional-district column | Doc-guide definition | Column the measure means | +|---|---|---|---| +| `limited_state_local_taxes_returns` / `_amount` | N18425 / A18425 | State and local income taxes (Schedule A line 5a) | N18460 / A18460, limited state and local taxes (line 5e) | +| `premium_tax_credit_returns` / `_amount` | N85530 / A85530 | Additional Medicare tax (Form 8959 line 24) | N85770 / A85770, total premium tax credit | + +The SALT amount rows summed to 1.98x their national target on `full`. The +premium-tax-credit amount was already dropped by a hard-coded check, but the +returns rows compiled as `assigned_aca_ptc` recipients. + +`US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS` in `fiscal_targets.py` now drops +all four, keyed by (measure id, `layout.source_column_id`). It lists them in +the release's exclusion receipt. A corrected package that reads the right +columns passes untouched. + +`test_pinned_feed_no_congressional_district_family_sums_past_its_national_target` +holds the invariant for every congressional-district family on `full`. + diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index 74e4757cba..ac03bb85c4 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -291,6 +291,7 @@ US_FISCAL_TARGET_LEDGER_REFERENCES, US_FISCAL_TARGET_REFERENCES, US_FISCAL_TARGET_REGISTRY, + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS, US_FISCAL_TARGET_SPECS, US_FISCAL_TARGET_SUPPORT_EXCLUSIONS, US_JCT_TAX_EXPENDITURE_REFORMS, @@ -1165,6 +1166,7 @@ "US_FISCAL_TARGET_SUPPORT_EXCLUSIONS", "US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS", "US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES", + "US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS", "US_FISCAL_TARGET_COVERAGE_REQUIREMENTS", "US_FISCAL_TARGET_LEDGER_REFERENCES", "US_JCT_TAX_EXPENDITURE_REFORMS", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index acd4751ddb..6487592936 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -55,6 +55,7 @@ "US_FISCAL_TARGET_SUPPORT_EXCLUSIONS", "US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS", "US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES", + "US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS", "US_FISCAL_LEDGER_PARITY_REGISTRY", "US_FISCAL_LEDGER_PARITY_REPORT", "US_JCT_TAX_EXPENDITURE_REFORMS", @@ -854,6 +855,61 @@ def _us_hierarchy_seed( # every vintage. US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES: dict[str, str] = {} +# Reviewed source-column concept exclusions (microcosm#1038). Each entry is an +# IRS column that a Chronicle package publishes under a measure id whose +# concept the column does not carry, so no fact read from that column +# compiles, whatever its package, geography or vintage, and a package +# corrected to read the right column passes untouched. The key is (measure_id, +# layout.source_column_id) because the IRS column is the definition. The +# congressional-district documentation guide (22incddocguide.docx) defines: +# +# - N18425/A18425 as "State and local income taxes" (Schedule A line 5a) and +# N18460/A18460 as "Limited state and local taxes" (line 5e). salt_deduction, +# like the Historic Table 2 rows of these measure ids, is the limited line 5e +# deduction. +# - N85530/A85530 as "Additional Medicare tax" (Form 8959 line 24) and +# N85770/A85770 as "Total premium tax credit" (Form 8962 line 24; the guide +# prints "8926:24"). N85770/A85770 is what the Historic Table 2 rows of +# these measure ids read and what assigned_aca_ptc measures. The amount was +# dropped before by a hard-coded congressional-district check; the returns +# row compiled as premium-tax-credit recipients. +# +# Chronicle's congressional_district_2022 package reads these four columns +# under the limited_state_local_taxes_* and premium_tax_credit_* measure ids; +# every other measure it shares with Historic Table 2 reads the same column. +US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS: dict[tuple[str, str], str] = { + ("limited_state_local_taxes_returns", "N18425"): ( + "N18425 counts returns with state and local income taxes (Schedule A " + "line 5a), not returns with the limited state and local tax deduction " + "(line 5e, N18460). Chronicle's congressional_district_2022 package " + "reads it under this measure id: TY2022 US 10,842,580 returns against " + "14,777,040 for N18460 in the same file and 14,968,720 in Historic " + "Table 2." + ), + ("limited_state_local_taxes_amount", "A18425"): ( + "A18425 is state and local income taxes before the $10,000 limit " + "(Schedule A line 5a), not the limited deduction (line 5e, A18460) " + "salt_deduction measures. Chronicle's congressional_district_2022 " + "package reads it under this measure id: TY2022 US $250.4B against " + "$121.2B for A18460 in the same file, so on the full surface the " + "congressional-district rows summed to 1.98x the national Historic " + "Table 2 target." + ), + ("premium_tax_credit_returns", "N85530"): ( + "N85530 counts returns with additional Medicare tax (Form 8959 line 24), " + "not returns with the premium tax credit (Form 8962 line 24, N85770) " + "assigned_aca_ptc measures. Chronicle's congressional_district_2022 " + "package reads it under this measure id: TY2022 US 6,848,330 returns " + "against 7,713,000 for N85770 in the same file." + ), + ("premium_tax_credit_amount", "A85530"): ( + "A85530 is additional Medicare tax (Form 8959 line 24), not the premium " + "tax credit (Form 8962 line 24, A85770). Chronicle's " + "congressional_district_2022 package reads it under this measure id: " + "TY2022 US $14.2B against $53.1B for A85770 in the same file." + ), +} + @dataclass(frozen=True) class SimpleTaxExpenditureReform: @@ -1090,6 +1146,9 @@ def us_fiscal_target_exclusion_receipt( vintage included (dropped). - ``m_chip_state_chip_enrollment``: CMS CHIP enrollment facts for an ``_M_CHIP_STATE_FIPS`` state, at every period (dropped). + - ``source_column_concept_exclusion``: SOI facts read from an IRS column + that ``US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`` lists for their + measure id, at every vintage (dropped). - ``allowlisted_vintage_bypass``: facts latest-vintage selection activates at ``target_period`` that are another vintage of an excluded cell and carry a reviewed ``US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES`` entry @@ -1108,6 +1167,7 @@ def us_fiscal_target_exclusion_receipt( ) reviewed: set[str] = set() all_vintage: set[str] = set() + source_column: set[str] = set() m_chip: set[str] = set() for fact in materialized_facts: source_record_id = _source_record_id(fact) @@ -1117,6 +1177,8 @@ def us_fiscal_target_exclusion_receipt( all_vintage.add(source_record_id) elif source_record_id in US_FISCAL_TARGET_SUPPORT_EXCLUSIONS: reviewed.add(source_record_id) + elif _is_source_column_concept_exclusion(fact): + source_column.add(source_record_id) elif _source_name(fact) == "cms_medicaid": reference = _reference_from_ledger_fact(fact, target_period=target_period) if reference is not None and _is_m_chip_state_chip_enrollment( @@ -1144,6 +1206,22 @@ def us_fiscal_target_exclusion_receipt( "entries": sorted(US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS), "source_record_ids": sorted(all_vintage), }, + "source_column_concept_exclusion": { + "action": "dropped", + "scope": "SOI facts read from a listed column under a listed " + "measure id, every vintage", + "entries": [ + { + "measure_id": measure_id, + "source_column_id": source_column_id, + "reason": reason, + } + for (measure_id, source_column_id), reason in sorted( + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS.items() + ) + ], + "source_record_ids": sorted(source_column), + }, "m_chip_state_chip_enrollment": { "action": "dropped", "scope": "CMS CHIP enrollment rows for these states, every period", @@ -1490,7 +1568,8 @@ class _SoiTotalControl: #: TY2022. Its amount, net of losses, is 1.4% below col 38 in TY2022. #: #: A family absent from this register has no reviewed concept and is never a -#: capital-gains control. +#: capital-gains control. A registered family whose concept is not the +#: model's supplies shares only (``_soi_capital_gains_share_family``). _SOI_SCHEDULE_D_TAXABLE_NET_GAIN = "schedule_d_taxable_net_gain" _SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS = "form_1040_line_7_net_gain_or_loss" _SOI_CAPITAL_GAINS_FAMILY_CONCEPTS: dict[str, str] = { @@ -1519,44 +1598,54 @@ def _rebase_stale_soi_capital_gains_distributions( *, target_period: int | str, ) -> TargetRegistry: - """Use Historic Table 2 capital-gains rows as shares of a Table 1.4 level. - - HT2 rows count Form 1040 line 7 (gain or loss), a population - ``capital_gains_gross`` cannot measure, so they never ship as levels. Each - one ships as its share of the HT2 national total, scaled to the - concept-matched control of ``_soi_capital_gains_active_totals`` at or after - its own period, and declares the bridge in ``soi_source_concept`` / - ``soi_control_concept``. Without such a control, the row is dropped. The - national all-AGI HT2 rows retire because the control's own row owns the - national concept. + """Use line-7 capital-gains rows as shares of a Table 1.4 level. + + Historic Table 2 and congressional-district rows count Form 1040 line 7 + (gain or loss), a population ``capital_gains_gross`` cannot measure, so + they never ship as levels. Each one ships as its share of its own family's + national total at its own period (an HT2 row of the HT2 US row, a + congressional-district state or district row of that file's US row), + scaled to the concept-matched control of ``_soi_capital_gains_active_totals`` + at or after that period, and declares the bridge in ``soi_source_concept`` + / ``soi_control_concept``. Without such a control or family total, the row + is dropped. Each family's national all-AGI row retires because the + control's own row owns the national concept. + + One factor per family, quantity and period scales a family's state and + district rows alike, so the district-to-state hierarchy reconciliation that + runs next sees the same proportions. The congressional-district file's + states sum exactly to its US row, so on the full surface its states and + its districts each sum to the control. Before this, they shipped as + line-7 levels, 2.41x the Table 1.4 return count (microcosm#1038). """ controls = _soi_capital_gains_active_totals( facts, target_period=target_period, ) - stale_national_totals = _soi_capital_gains_stale_national_totals(facts) + share_totals = _soi_capital_gains_share_national_totals(facts) specs: list[TargetSpec] = [] for spec in registry.specs: kind = _soi_capital_gains_kind(spec) - if kind is None or not _is_stale_soi_historic_capital_gains_spec(spec): + family = _soi_capital_gains_share_family_from_spec(spec) if kind else None + if kind is None or family is None: specs.append(spec) continue key = _soi_capital_gains_control_key_from_spec(spec) control = controls.get(key) - source_total = stale_national_totals.get((*key, spec.metadata["source_period"])) + source_total = share_totals.get((family, *key, spec.metadata["source_period"])) if control is None or not _period_not_before( control.period_key, _period_key_from_value(spec.metadata["source_period"]) ): continue - if source_total in (None, 0): + if source_total is None or source_total.value == 0: continue if _is_national_all_agi_spec(spec): continue - factor = control.value / source_total + factor = control.value / source_total.value specs.append( replace( spec, @@ -1573,8 +1662,11 @@ def _rebase_stale_soi_capital_gains_distributions( "uprating_index_source_record_id": control.source_record_id, "uprating_factor": _format_float(factor), "stale_distribution_rebased_to_active_total": "true", - "soi_source_concept": _SOI_FORM_1040_LINE_7_NET_GAIN_OR_LOSS, + "soi_source_concept": _SOI_CAPITAL_GAINS_FAMILY_CONCEPTS[family], "soi_control_concept": _SOI_CAPITAL_GAINS_MODEL_CONCEPT, + # The share's denominator: this family's own national + # row, from the same release as the row itself. + "soi_share_total_source_record_id": source_total.source_record_id, }, ) ) @@ -1668,15 +1760,76 @@ def _soi_capital_gains_concept(fact: object) -> str | None: ) -def _soi_capital_gains_stale_national_totals( +def _soi_capital_gains_share_family(record_set_id: str) -> str | None: + """The family of a capital-gains row that may ship only as a share. + + A registered family whose columns count something other than what + ``capital_gains_gross`` measures (Historic Table 2 and the + congressional-district file, both Form 1040 line 7) supplies shares, never + levels. The model's own family and unregistered families return ``None``. + """ + family = _soi_record_set_family(record_set_id) + concept = _SOI_CAPITAL_GAINS_FAMILY_CONCEPTS.get(family) + if concept is None or concept == _SOI_CAPITAL_GAINS_MODEL_CONCEPT: + return None + return family + + +def _soi_capital_gains_share_family_from_spec(spec: TargetSpec) -> str | None: + if spec.family != "irs_soi": + return None + return _soi_capital_gains_share_family( + spec.metadata.get("ledger_layout_record_set_id", "") + ) + + +class AmbiguousSoiCapitalGainsShareTotalError(ValueError): + """One share family carries two different national totals for one period.""" + + +@dataclass(frozen=True) +class _SoiShareTotal: + value: float + source_record_id: str + + +def _soi_capital_gains_share_national_totals( facts: tuple[object, ...], -) -> dict[tuple[str, str, str, str], float]: - totals: dict[tuple[str, str, str, str], float] = {} +) -> dict[tuple[str, str, str, str, str], _SoiShareTotal]: + """Each share family's national all-AGI total, the denominator of its shares. + + Keyed by (family, measure, filing status, universe, period), so an HT2 row + divides by the HT2 US row and a congressional-district row by that file's + US row, never by another family's. Two different values under one key + refuse the compile instead of letting feed order pick the denominator; + equal values under two record ids keep the smaller id, so the named + denominator does not depend on feed order either. + """ + totals: dict[tuple[str, str, str, str, str], _SoiShareTotal] = {} for fact in facts: key = _soi_capital_gains_control_key_from_fact(fact) - if key is None or not _is_stale_soi_historic_capital_gains_fact(fact): + if key is None: + continue + family = _soi_capital_gains_share_family( + _str_at(fact, "layout", "record_set_id") + ) + if family is None: continue - totals[(*key, str(_period_value(fact)))] = _numeric_value(fact) + source_record_id = _source_record_id(fact) + if not source_record_id: + continue + total_key = (family, *key, str(_period_value(fact))) + candidate = _SoiShareTotal(_numeric_value(fact), source_record_id) + current = totals.get(total_key) + if current is not None and current.value != candidate.value: + raise AmbiguousSoiCapitalGainsShareTotalError( + "SOI capital-gains share family has two national totals for " + f"{total_key}: {current.value!r} ({current.source_record_id}) " + f"and {candidate.value!r} ({candidate.source_record_id}). " + "Deduplicate them upstream." + ) + if current is None or candidate.source_record_id < current.source_record_id: + totals[total_key] = candidate return totals @@ -1694,22 +1847,6 @@ def _soi_capital_gains_kind_from_measure(measure_id: str) -> str | None: return None -def _is_stale_soi_historic_capital_gains_spec(spec: TargetSpec) -> bool: - if spec.family != "irs_soi": - return False - if _soi_capital_gains_kind(spec) is None: - return False - return ".historic_table_2." in spec.metadata.get("ledger_layout_record_set_id", "") - - -def _is_stale_soi_historic_capital_gains_fact(fact: object) -> bool: - if _source_name(fact) != "irs_soi": - return False - if _soi_capital_gains_kind_from_measure(_measure_id(fact)) is None: - return False - return ".historic_table_2." in _str_at(fact, "layout", "record_set_id") - - def _soi_capital_gains_control_key_from_fact( fact: object, ) -> tuple[str, str, str] | None: @@ -2622,6 +2759,16 @@ def _period_free_source_record_id(source_record_id: str) -> str: return _normalized_record_set_id(source_record_id) +def _is_source_column_concept_exclusion(fact: object) -> bool: + """Whether an SOI fact reads a column its measure id does not mean.""" + if _source_name(fact) != "irs_soi": + return False + return ( + _measure_id(fact), + _str_at(fact, "layout", "source_column_id"), + ) in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS + + def _is_all_vintage_support_exclusion(source_record_id: str) -> bool: return _period_free_source_record_id(source_record_id) in ( _period_free_source_record_ids(US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS) @@ -2656,18 +2803,6 @@ def _is_soi_congressional_district_record_set(fact: object) -> bool: return ".congressional_district_2022." in record_set_id -def _is_soi_cd_premium_tax_credit_amount_conflict( - fact: object, *, measure_id: str -) -> bool: - # The SOI CD A85530 amount is not the same gross annual PTC control used by - # assigned_aca_ptc. Keep Historic Table 2 national/state PTC amount controls - # until Ledger has an explicit CD reconciliation for this concept. - return ( - measure_id == "premium_tax_credit_amount" - and _is_soi_congressional_district_record_set(fact) - ) - - def _is_period_token(value: str) -> bool: normalized = value.lower().replace("-", "_") if normalized.startswith("month"): @@ -2734,6 +2869,8 @@ def _reference_from_ledger_fact( return None if _is_all_vintage_support_exclusion(source_record_id): return None + if _is_source_column_concept_exclusion(fact): + return None source_name = _source_name(fact) if source_name == "irs_soi": return _soi_reference_from_fact( @@ -2791,8 +2928,6 @@ def _soi_reference_from_fact( and geography_level != "congressional_district" ): return None - if _is_soi_cd_premium_tax_credit_amount_conflict(fact, measure_id=measure_id): - return None variable = SOI_AMOUNT_MEASURE_VARIABLES.get(measure_id) is_count = False if variable is None: diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index cc7037cc82..64d27d4180 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -107,10 +107,14 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: groupby_value_id="hi_01", geography_id="5001700US1501", ), - _soi_congressional_district_fact( + # The pinned feed reads the CD PTC measures from A85530, additional + # Medicare tax, so they never compile (source-column exclusion). + _cd_column_fact( "premium_tax_credit_amount", - 90_000_000, + "A85530", + value=90_000_000, groupby_value_id="hi_01", + geography_level="congressional_district", geography_id="5001700US1501", ), _soi_congressional_district_fact( @@ -164,9 +168,10 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: geography_level="state", geography_id="0400000US15", ), - _soi_congressional_district_fact( + _cd_column_fact( "premium_tax_credit_amount", - 180_000_000, + "A85530", + value=180_000_000, groupby_value_id="hi_total", geography_level="state", geography_id="0400000US15", @@ -734,12 +739,14 @@ def test_exclusion_vintage_scope_registers_are_consistent() -> None: def test_exclusion_receipt_lists_the_ids_each_rule_acts_on(monkeypatch) -> None: - """One synthetic feed exercises all four rules. The M-CHIP and all-vintage + """One synthetic feed exercises all five rules. The M-CHIP and all-vintage rules list every vintage the feed carries (both CMS months even though a 2024 build selects only 2024-12), exact-key hits land under - reviewed_exclusion, and the allowlisted bypass is the fact selection - activates. The receipt is plain JSON.""" - synthetic, expected = _four_rule_synthetic_feed(monkeypatch) + reviewed_exclusion, the source-column rule lists the facts read from a + listed column and not the one read from the right column, and the + allowlisted bypass is the fact selection activates. The receipt is plain + JSON.""" + synthetic, expected = _exclusion_rule_synthetic_feed(monkeypatch) facts = [*packaged_reference_facts(), *synthetic] receipt = us_fiscal_target_exclusion_receipt(facts, target_period=2024) @@ -750,6 +757,7 @@ def test_exclusion_receipt_lists_the_ids_each_rule_acts_on(monkeypatch) -> None: assert {name: rule["action"] for name, rule in rules.items()} == { "reviewed_exclusion": "dropped", "all_vintage_reviewed_exclusion": "dropped", + "source_column_concept_exclusion": "dropped", "m_chip_state_chip_enrollment": "dropped", "allowlisted_vintage_bypass": "allowed", } @@ -761,11 +769,92 @@ def test_exclusion_receipt_lists_the_ids_each_rule_acts_on(monkeypatch) -> None: assert rules["m_chip_state_chip_enrollment"]["state_fips"] == sorted( _M_CHIP_STATE_FIPS ) + assert [ + (entry["measure_id"], entry["source_column_id"]) + for entry in rules["source_column_concept_exclusion"]["entries"] + ] == sorted(US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS) + + +def test_source_column_exclusion_drops_only_the_mislabeled_column() -> None: + """A congressional-district fact labelled limited_state_local_taxes but read + from N18425/A18425 (state and local income taxes, Schedule A line 5a), or + labelled premium_tax_credit but read from N85530 (additional Medicare + tax), never compiles, at any geography; the same measures read from + A18460 and N85770, the columns a corrected package would read, compile + (microcosm#1038).""" + mislabeled = [ + _cd_column_fact("limited_state_local_taxes_returns", "N18425"), + _cd_column_fact("limited_state_local_taxes_amount", "A18425"), + _cd_column_fact( + "limited_state_local_taxes_returns", + "N18425", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ), + ] + corrected = _cd_column_fact( + "limited_state_local_taxes_amount", + "A18460", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + medicare = _cd_column_fact("premium_tax_credit_returns", "N85530") + ptc = _cd_column_fact( + "premium_tax_credit_returns", + "N85770", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + mislabeled = [*mislabeled, medicare] + facts = [*packaged_reference_facts(), *mislabeled, corrected, ptc] + + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + + compiled = {spec.metadata["ledger_source_record_id"] for spec in registry.specs} + assert not compiled & {fact["lineage"]["source_record_id"] for fact in mislabeled} + assert corrected["lineage"]["source_record_id"] in compiled + assert ptc["lineage"]["source_record_id"] in compiled + rule = us_fiscal_target_exclusion_receipt(facts)["rules"][ + "source_column_concept_exclusion" + ] + assert rule["source_record_ids"] == sorted( + fact["lineage"]["source_record_id"] for fact in mislabeled + ) + + +def test_source_column_exclusions_name_the_columns_their_guides_define() -> None: + """Each entry's reason names its own column and the column its measure id + means, as 22incddocguide.docx defines them: N18425/A18425 (state and local + income taxes, Schedule A line 5a) stand in for the limited deduction + N18460/A18460 (line 5e), and N85530/A85530 (additional Medicare tax) for + the premium tax credit N85770/A85770.""" + right_columns = { + ("limited_state_local_taxes_returns", "N18425"): "N18460", + ("limited_state_local_taxes_amount", "A18425"): "A18460", + ("premium_tax_credit_returns", "N85530"): "N85770", + ("premium_tax_credit_amount", "A85530"): "A85770", + } + assert set(US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS) == set(right_columns) + for key, reason in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS.items(): + measure_id, column = key + assert column in reason + assert right_columns[key] in reason + # Counts read N columns and amounts read A columns. + assert ( + column[0] + == right_columns[key][0] + == ("N" if measure_id.endswith("_returns") else "A") + ) def test_exclusion_receipt_is_complete_for_any_subfeed(monkeypatch) -> None: """Invariant (receipt completeness): each rule lists exactly the facts it - drops or allows. For any subset of the four-rule synthetic feed, each rule + drops or allows. For any subset of the five-rule synthetic feed, each rule lists exactly the full feed's ids for that rule that the subset carries. Hypothesis is a workspace dependency; the wheels job installs no test extras, so it skips there.""" @@ -773,7 +862,7 @@ def test_exclusion_receipt_is_complete_for_any_subfeed(monkeypatch) -> None: from hypothesis import given, settings from hypothesis import strategies as st - synthetic, expected = _four_rule_synthetic_feed(monkeypatch) + synthetic, expected = _exclusion_rule_synthetic_feed(monkeypatch) ids = [fiscal_targets._source_record_id(fact) for fact in synthetic] reference = packaged_reference_facts() @@ -859,9 +948,11 @@ def check(years: list[int], reviewed: bool) -> None: def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> None: """On the pinned feed the receipt names the 40 M-CHIP CHIP ids (20 states - x 2 CMS months), the 16 other-income ids (ty2020-ty2023) and the ty2020 - and ty2023 tips return counts, and allows no vintage bypass (decision - d179).""" + x 2 CMS months), the 16 other-income ids (ty2020-ty2023), the ty2020 and + ty2023 tips return counts, and the 1,952 congressional-district SALT and + premium-tax-credit facts read from the wrong IRS column (488 per measure + after the district crosswalk: the US row, 51 states, 436 districts), and + allows no vintage bypass (decision d179, microcosm#1038).""" _, _, receipt = pinned_feed_national_state_surface rules = receipt["rules"] m_chip_ids = rules["m_chip_state_chip_enrollment"]["source_record_ids"] @@ -886,23 +977,44 @@ def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> No ] ) assert rules["allowlisted_vintage_bypass"]["source_record_ids"] == [] + column_ids = rules["source_column_concept_exclusion"]["source_record_ids"] + assert all(".congressional_district_2022." in i for i in column_ids) + assert { + measure: sum( + f".{measure}" in source_record_id for source_record_id in column_ids + ) + for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS + } == {measure: 488 for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS} + assert len(column_ids) == 4 * 488 def test_pinned_feed_national_state_surface_restores_the_fences( pinned_feed_national_state_surface, ) -> None: - """The release surface on the pinned feed: 32,842 compiled targets after - Medicaid substitution, 5,694 national_state targets at registry - 65e4dde11c83, 32 CHIP rows none of them for an M-CHIP state, no - other-income row and no tips return count. Before microcosm#956 it was - 32,867 / 5,719 at d5f9d854fe11, before decision d179 dropped the ty2020 - tips return count it was 32,843 / 5,695 at 386fac439e77, and before - microcosm#1035 moved the capital-gains control back to Table 1.4 it was - d315c75804ef at the same counts (docs/us-chronicle-feed-repin.md).""" + """The release surface on the pinned feed: 31,376 compiled targets after + Medicaid substitution (the full surface, at 059dc56d78db), 5,694 + national_state targets at registry 47dce807b412, 32 CHIP rows none of them + for an M-CHIP state, no other-income row and no tips return count. Before + microcosm#956 it was 32,867 / 5,719 at d5f9d854fe11, before decision d179 + dropped the ty2020 tips return count it was 32,843 / 5,695 at + 386fac439e77, before microcosm#1035 moved the capital-gains control back + to Table 1.4 it was d315c75804ef at the same counts, and before + microcosm#1038 it was 32,842 compiled at 65e4dde11c83 (full e02123644d42): + that change drops 1,466 congressional-district specs (the two + capital-gains US rows and the SALT and premium-tax-credit rows read from + the wrong column) and moves national_state only by one provenance key, + soi_share_total_source_record_id, on the 102 Historic Table 2 + capital-gains rows (docs/us-chronicle-feed-repin.md, + docs/us-soi-capital-gains-concepts.md).""" + from microcosm.calibrate import TargetRegistry + registry, surface, _ = pinned_feed_national_state_surface - assert len(registry.specs) == 32_842 + assert len(registry.specs) == 31_376 + builder = _load_repo_tool("build_us_fiscal_refresh_release") + full, _ = builder._select_target_surface(registry.specs, "full") + assert TargetRegistry(full, country="us").version == "059dc56d78db" assert len(surface.specs) == 5_694 - assert surface.version == "65e4dde11c83" + assert surface.version == "47dce807b412" chip = [ spec for spec in surface.specs @@ -962,6 +1074,9 @@ def test_pinned_feed_capital_gains_rows_take_the_table_1_4_level( national = next(spec for spec in surface.specs if spec.name == control_id) for spec in rows: assert spec.metadata["uprating_index_source_record_id"] == control_id + assert spec.metadata["soi_share_total_source_record_id"] == ( + f"irs_soi.ty2022.historic_table_2.us.all.{measure}" + ) assert spec.metadata["uprating_factor"] == factor assert spec.metadata["soi_source_concept"] == ( "form_1040_line_7_net_gain_or_loss" @@ -986,25 +1101,7 @@ def test_pinned_feed_no_soi_state_family_sums_past_its_national_target( both a stale HT2 US row and a newer Table 1.x row, a few percent apart, so the bound is the larger of them.""" _, surface, _ = pinned_feed_national_state_surface - - def quantity(spec) -> tuple[object, ...]: - metadata = spec.metadata - return ( - metadata.get("variable"), - metadata.get("measure_mode"), - metadata.get("agi_lower_bound"), - metadata.get("agi_upper_bound"), - metadata.get("filing_status"), - metadata.get("soi_return_universe", "all_returns"), - metadata.get("itemized_only"), - tuple( - sorted( - (key, value) - for key, value in metadata.items() - if key.startswith("ledger_filter_") - ) - ), - ) + quantity = _soi_quantity national: dict[tuple[object, ...], list[float]] = {} states: dict[tuple[object, ...], float] = {} @@ -1024,6 +1121,133 @@ def quantity(spec) -> tuple[object, ...]: } == {} +def _soi_quantity(spec) -> tuple[object, ...]: + """The model quantity and filters an SOI target constrains.""" + metadata = spec.metadata + return ( + metadata.get("variable"), + metadata.get("measure_mode"), + metadata.get("agi_lower_bound"), + metadata.get("agi_upper_bound"), + metadata.get("filing_status"), + metadata.get("soi_return_universe", "all_returns"), + metadata.get("itemized_only"), + tuple( + sorted( + (key, value) + for key, value in metadata.items() + if key.startswith("ledger_filter_") + ) + ), + ) + + +def _is_congressional_district_file_spec(spec) -> bool: + return ( + fiscal_targets._soi_record_set_family( + spec.metadata.get("ledger_layout_record_set_id", "") + ) + == "congressional_district" + ) + + +def test_pinned_feed_congressional_district_capital_gains_take_the_table_1_4_level( + pinned_feed_national_state_surface, +) -> None: + """On the full surface the congressional-district file's 51 state and 436 + district capital-gains rows per measure are shares of that file's US row + times the Table 1.4 ty2023 level, and the US row retires + (microcosm#1038). The states, and the districts, each sum to the national + target: 12,392,020 returns, where the line-7 levels summed to 29,845,710 + (2.41x), and the aged Table 1.4 amount, where they summed to $1,522.2B + (1.198x).""" + registry, _, _ = pinned_feed_national_state_surface + specs = {spec.name: spec for spec in registry.specs} + for measure, control_id, cd_us_id, total in ( + ( + "net_capital_gains_returns", + "irs_soi.ty2023.table_1_4.all.net_capital_gains_returns", + f"{_CG_CD_RECORD_SET}.us.net_capital_gains_returns", + 12_392_020.0, + ), + ( + "net_capital_gains_amount", + "irs_soi.ty2023.table_1_4.all.net_capital_gains_amount", + f"{_CG_CD_RECORD_SET}.us.net_capital_gains_amount", + None, + ), + ): + rows = [ + spec + for spec in registry.specs + if spec.metadata.get("source_measure_id") == measure + and _is_congressional_district_file_spec(spec) + ] + levels: dict[str, list] = {} + for spec in rows: + levels.setdefault(spec.metadata["ledger_geography_level"], []).append(spec) + assert {level: len(group) for level, group in levels.items()} == { + "state": 51, + "congressional_district": 436, + } + assert cd_us_id not in specs + national = specs[control_id] + factors = {spec.metadata["uprating_factor"] for spec in rows} + assert len(factors) == 1 + for spec in rows: + assert spec.metadata["uprating_index_source_record_id"] == control_id + assert spec.metadata["soi_share_total_source_record_id"] == cd_us_id + assert spec.metadata["soi_source_concept"] == ( + "form_1040_line_7_net_gain_or_loss" + ) + assert spec.metadata["soi_control_concept"] == ( + "schedule_d_taxable_net_gain" + ) + for level in ("state", "congressional_district"): + level_total = sum(spec.value for spec in levels[level]) + assert math.isclose(level_total, national.value, rel_tol=1e-12) + if total is not None: + assert math.isclose(level_total, total, rel_tol=1e-12) + + +def test_pinned_feed_no_congressional_district_family_sums_past_its_national_target( + pinned_feed_national_state_surface, +) -> None: + """Invariant: the congressional-district file covers a subset of returns + (no Puerto Rico or other areas, ZIP-unmatched returns dropped), so on the + full surface none of its families (its US row, its states, or its + districts) may sum past every national non-district target of the same + model quantity and filters. Before microcosm#1038 three of 38 compared + families did: capital-gains returns at 2.41x, the SALT amount at 1.98x + (read from the uncapped income-tax column A18425) and the capital-gains + amount at 1.198x. 35 of today's 52 families have a national comparator; + the rest (EITC by children, charitable, interest paid, QBI, taxable + interest, filer individuals) have none on this surface.""" + registry, _, _ = pinned_feed_national_state_surface + builder = _load_repo_tool("build_us_fiscal_refresh_release") + full, _ = builder._select_target_surface(registry.specs, "full") + + national: dict[tuple[object, ...], list[float]] = {} + district_file: dict[tuple[object, ...], dict[str, float]] = {} + for spec in full: + if spec.family != "irs_soi": + continue + level = spec.metadata.get("ledger_geography_level") + if _is_congressional_district_file_spec(spec): + sums = district_file.setdefault(_soi_quantity(spec), {}) + sums[level] = sums.get(level, 0.0) + spec.value + elif level == "country": + national.setdefault(_soi_quantity(spec), []).append(spec.value) + compared = {key: sums for key, sums in district_file.items() if key in national} + assert (len(district_file), len(compared)) == (52, 35) + assert { + (key[:2], level): total / max(national[key]) + for key, sums in compared.items() + for level, total in sums.items() + if total > max(national[key]) * (1 + 1e-9) + } == {} + + def test_reviewed_zero_support_facts_are_not_active_targets() -> None: excluded_source_record_id = ( "hhs_acf_tanf.fy2024.cash_assistance.ar." @@ -4556,6 +4780,440 @@ def check(state_values, other_areas, control_value) -> None: check() +def _capital_gains_cd_package_facts( + *, + us: tuple[float, float] = (98.0, 925.0), + hawaii: tuple[float, float] = (4.0, 40.0), + districts: tuple[tuple[float, float], ...] = ((1.5, 10.0), (2.5, 30.0)), +) -> list[dict[str, object]]: + """A congressional-district package's line-7 rows, stamped ty2023 like the + pinned feed: the US row, Hawaii, and Hawaii's two districts, each as + (returns, amount).""" + facts = [] + for index, measure in enumerate( + ("net_capital_gains_returns", "net_capital_gains_amount") + ): + facts.append( + _soi_congressional_district_fact( + measure, + us[index], + groupby_value_id="us", + geography_level="country", + geography_id="0100000US", + ) + ) + facts.append( + _soi_congressional_district_fact( + measure, + hawaii[index], + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + ) + for number, values in enumerate(districts, start=1): + facts.append( + _soi_congressional_district_fact( + measure, + values[index], + groupby_value_id=f"hi_0{number}", + geography_id=f"5001700US150{number}", + ) + ) + return facts + + +def _congressional_district_capital_gains_specs(registry) -> dict[str, object]: + return { + spec.name: spec + for spec in registry.specs + if spec.metadata.get("source_measure_id") + in {"net_capital_gains_returns", "net_capital_gains_amount"} + and ".congressional_district_" in spec.metadata["ledger_layout_record_set_id"] + } + + +def test_congressional_district_capital_gains_rows_are_shares_of_table_1_4() -> None: + """The congressional-district file's N01000/A01000 count Form 1040 line 7, + like Historic Table 2, so its state and district rows ship as shares of + that file's own US row times the Table 1.4 level; the US row retires + (microcosm#1038). On the pinned feed they shipped as line-7 levels, and + the US returns row was 2.41x the Table 1.4 target of the same model + quantity on the full surface.""" + facts = [ + *packaged_reference_facts(), + *_capital_gains_ht2_facts(), + *_capital_gains_cd_package_facts(), + *_capital_gains_t14_facts(), + ] + + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + + cd = _congressional_district_capital_gains_specs(registry) + assert not [ + name + for name, spec in cd.items() + if spec.metadata["ledger_geography_level"] == "country" + ] + for measure, control_id, control, cd_us, hawaii, districts in ( + ("net_capital_gains_returns", _CG_T14_RETURNS, 40.0, 98.0, 4.0, (1.5, 2.5)), + ("net_capital_gains_amount", _CG_T14_AMOUNT, 800.0, 925.0, 40.0, (10.0, 30.0)), + ): + factor = control / cd_us + state = cd[f"{_CG_CD_RECORD_SET}.hi_total.{measure}"] + children = [cd[f"{_CG_CD_RECORD_SET}.hi_0{n}.{measure}"] for n in (1, 2)] + assert math.isclose(state.value, hawaii * factor, rel_tol=1e-12) + for child, raw in zip(children, districts, strict=True): + assert math.isclose(child.value, raw * factor, rel_tol=1e-12) + # One factor scales the state and its districts, so the hierarchy + # reconciliation that follows leaves the proportions alone. + assert child.metadata["hierarchy_reconciliation_factor"] == "1" + for spec in (state, *children): + assert spec.metadata["uprating_factor"] == fiscal_targets._format_float( + factor + ) + assert spec.metadata["uprating_index_source_record_id"] == control_id + assert spec.metadata["soi_source_concept"] == ( + "form_1040_line_7_net_gain_or_loss" + ) + assert spec.metadata["soi_control_concept"] == ( + "schedule_d_taxable_net_gain" + ) + + # Historic Table 2 rows keep dividing by the HT2 US row (100 / 1,000), not + # by the congressional-district US row. + specs = {spec.metadata["ledger_source_record_id"]: spec for spec in registry.specs} + assert specs[_CG_HT2_CA_RETURNS].value == 10.0 + assert specs[_CG_HT2_CA_AMOUNT].value == 200.0 + + +#: The pinned feed's 22incd.csv: TY2022 data that Chronicle stamps ty2023 +#: (PolicyEngine/chronicle#117). A source-vintage correction keys on these. +_CD_2022_SOURCE_SHA256 = ( + "137522878af78d624cfddc0e17cd36ae76a21b7bed166c7bb96ad3243a18a668" +) + + +def _capital_gains_cd_package_at(stamp: int) -> list[dict[str, object]]: + """The congressional-district capital-gains rows of + ``_capital_gains_cd_package_facts``, stamped ``stamp``, carrying the + pinned 22incd.csv's digest and raw key.""" + record_set_id = f"irs_soi.ty{stamp}.congressional_district_2022.all_returns" + facts = [] + for fact in _capital_gains_cd_package_facts(): + row = fact["layout"]["groupby_value_id"] + measure = fact["layout"]["measure_id"] + restamped = _dynamic_ledger_fact( + source_record_id=f"{record_set_id}.{row}.{measure}", + source_name="irs_soi", + measure_id=measure, + value=fact["value"], + period_value=stamp, + geography_level=fact["geography"]["level"], + geography_id=fact["geography"]["id"], + dimensions={"income_range": "all", "filing_status": "all"}, + layout_record_set_id=record_set_id, + groupby_dimension="irs_soi.congressional_district", + groupby_value_id=row, + ) + restamped["source"].update( + { + "source_file": "22incd.csv", + "source_sha256": _CD_2022_SOURCE_SHA256, + "raw_r2_key": ( + "raw/irs_soi/soi-congressional-district-2022/2022/" + f"{_CD_2022_SOURCE_SHA256}/22incd.csv" + ), + "vintage": f"tax_year_{stamp}", + } + ) + facts.append(restamped) + return facts + + +def test_congressional_district_capital_gains_ignore_the_package_stamp() -> None: + """Differential: a congressional-district capital-gains row is its share of + the same package's US row times the Table 1.4 level, so the package's + period stamp cancels. Compiled and aged to 2024, the package stamped + ty2023, as on the pinned feed, gives exactly the values of the same package + stamped truthfully at 2022, and its amounts age from the control's 2023 + period on the CBO net-capital-gain ratio, not from 2022. The facts carry + the pinned file's digest and raw key, so a source-vintage correction that + recognises the restamp sees them: it must leave these values alone + (microcosm#1030, microcosm#1038).""" + cbo = [ + _cbo_income_source_projection_fact(2023, "net_capital_gain", value=981.4e9), + _cbo_income_source_projection_fact(2024, "net_capital_gain", value=1290.9e9), + ] + compiled = {} + for stamp in (2023, 2022): + registry = compile_us_fiscal_target_registry( + [ + *packaged_reference_facts(), + *_capital_gains_cd_package_at(stamp), + *_capital_gains_t14_facts(), + *cbo, + ], + target_period=2024, + age_targets=True, + allow_unaged_dollar_targets=True, + ) + compiled[stamp] = { + ( + spec.metadata["source_measure_id"], + spec.metadata["ledger_geography_id"], + ): (spec) + for spec in _congressional_district_capital_gains_specs(registry).values() + } + + assert set(compiled[2023]) == set(compiled[2022]) + assert len(compiled[2023]) == 6 + ratio = 1290.9e9 / 981.4e9 + for key, spec in compiled[2023].items(): + truthful = compiled[2022][key] + assert math.isclose(spec.value, truthful.value, rel_tol=1e-12), key + assert spec.metadata["uprating_to_period"] == "2023" + if key[0] == "net_capital_gains_amount": + for aged in (spec, truthful): + assert math.isclose( + float(aged.metadata["aging_factor"]), ratio, rel_tol=1e-12 + ) + share = {"0400000US15": 40.0, "5001700US1501": 10.0}.get(key[1], 30.0) + assert math.isclose( + spec.value, share / 925.0 * 800.0 * ratio, rel_tol=1e-12 + ) + + +def test_congressional_district_capital_gains_rows_without_a_control_are_dropped() -> ( + None +): + """Without a Schedule D control no congressional-district capital-gains row + compiles: the file's line-7 counts are never a capital_gains_gross level.""" + facts = [*packaged_reference_facts(), *_capital_gains_cd_package_facts()] + + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + + assert _congressional_district_capital_gains_specs(registry) == {} + + +def test_capital_gains_share_family_refuses_two_national_totals() -> None: + """Two different national totals for one share family, quantity and period + refuse the compile: feed order must not pick a share denominator.""" + rival = _soi_congressional_district_fact( + "net_capital_gains_returns", + 99.0, + groupby_value_id="us_rival", + geography_level="country", + geography_id="0100000US", + ) + facts = [ + *packaged_reference_facts(), + *_capital_gains_cd_package_facts(), + *_capital_gains_t14_facts(), + rival, + ] + + with pytest.raises(fiscal_targets.AmbiguousSoiCapitalGainsShareTotalError): + fiscal_targets._soi_capital_gains_share_national_totals(tuple(facts)) + with pytest.raises(fiscal_targets.AmbiguousSoiCapitalGainsShareTotalError): + compile_us_fiscal_target_registry(facts, allow_unaged_dollar_targets=True) + + +@pytest.mark.parametrize("twin_first", [True, False], ids=["twin-first", "us-first"]) +def test_capital_gains_share_denominator_is_order_free(twin_first: bool) -> None: + """Two records carrying the same national total for one share family are + not ambiguous, and the denominator a share row names is the smaller + record id whichever comes first in the feed.""" + twin = _soi_congressional_district_fact( + "net_capital_gains_returns", + 98.0, + groupby_value_id="us_twin", + geography_level="country", + geography_id="0100000US", + ) + package = _capital_gains_cd_package_facts() + facts = [ + *packaged_reference_facts(), + *((twin, *package) if twin_first else (*package, twin)), + *_capital_gains_t14_facts(), + ] + + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + + hawaii = _congressional_district_capital_gains_specs(registry)[ + f"{_CG_CD_RECORD_SET}.hi_total.net_capital_gains_returns" + ] + assert hawaii.metadata["soi_share_total_source_record_id"] == min( + f"{_CG_CD_RECORD_SET}.us.net_capital_gains_returns", + f"{_CG_CD_RECORD_SET}.us_twin.net_capital_gains_returns", + ) + assert math.isclose(hawaii.value, 4.0 * 40.0 / 98.0, rel_tol=1e-12) + + +@pytest.mark.parametrize( + ("record_set_id", "family"), + [ + ("irs_soi.ty2022.historic_table_2.state_broad.ca", "historic_table_2"), + ( + "irs_soi.ty2023.congressional_district_2022.all_returns", + "congressional_district", + ), + ( + "irs_soi.ty2024.congressional_district_2024.all_returns", + "congressional_district", + ), + ("irs_soi.ty2023.table_1_4", None), + ("irs_soi.ty2023.itemized_all_returns", None), + ("bea.cy2023.regional", None), + ], +) +def test_capital_gains_share_families_are_the_line_7_families( + record_set_id: str, family: str | None +) -> None: + """Only a registered family whose concept is not the model's supplies + shares: Table 1.4 is the model's own concept and an unregistered family + has no reviewed concept.""" + assert fiscal_targets._soi_capital_gains_share_family(record_set_id) == family + + +def test_congressional_district_capital_gains_shares_conserve_the_control() -> None: + """Property: whatever the congressional-district package, the Table 1.4 + level and the feed order, no congressional-district capital-gains row + ships at the country level or without the Schedule D bridge; each state + row is its share of the file's US row times the control; each state's + districts sum to that state row; and the states, and the districts, sum + to the control times the states' share of the US row, so never past the + control. The order is part of the draw, so the result is order-free.""" + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + # Two-district states and one single-district state, whose source row is + # the state total only (22incd.csv carries no district row for them). + two_district_states = (("hi", "15"), ("id", "16"), ("me", "23")) + at_large = ("vt", "50") + measure = "net_capital_gains_returns" + values = st.integers(min_value=1, max_value=10_000_000) + + @settings(max_examples=25, deadline=None) + @given( + districts=st.lists( + st.tuples(values, values), + min_size=len(two_district_states), + max_size=len(two_district_states), + ), + state_excess=st.lists( + st.integers(min_value=0, max_value=1_000_000), + min_size=len(two_district_states), + max_size=len(two_district_states), + ), + at_large_value=values, + other_areas=st.integers(min_value=0, max_value=50_000_000), + control=values, + order=st.randoms(use_true_random=False), + ) + def check(districts, state_excess, at_large_value, other_areas, control, order): + facts = [] + state_values = {} + for (postal, fips), pair, excess in zip( + two_district_states, districts, state_excess, strict=True + ): + state_values[fips] = float(sum(pair) + excess) + facts.append( + _soi_congressional_district_fact( + measure, + state_values[fips], + groupby_value_id=f"{postal}_total", + geography_level="state", + geography_id=f"0400000US{fips}", + ) + ) + for number, value in enumerate(pair, start=1): + facts.append( + _soi_congressional_district_fact( + measure, + float(value), + groupby_value_id=f"{postal}_0{number}", + geography_id=f"5001700US{fips}0{number}", + ) + ) + state_values[at_large[1]] = float(at_large_value) + facts.append( + _soi_congressional_district_fact( + measure, + float(at_large_value), + groupby_value_id=f"{at_large[0]}_total", + geography_level="state", + geography_id=f"0400000US{at_large[1]}", + ) + ) + cd_us = sum(state_values.values()) + other_areas + facts.append( + _soi_congressional_district_fact( + measure, + cd_us, + groupby_value_id="us", + geography_level="country", + geography_id="0100000US", + ) + ) + facts.append( + _soi_capital_gains_fact( + 2023, + source_record_id=_CG_T14_RETURNS, + measure_id=measure, + layout_record_set_id="irs_soi.ty2023.table_1_4", + value=float(control), + ) + ) + order.shuffle(facts) + registry = compile_us_fiscal_target_registry( + [*packaged_reference_facts(), *facts], allow_unaged_dollar_targets=True + ) + + cd = _congressional_district_capital_gains_specs(registry).values() + by_level: dict[str, list] = {} + for spec in cd: + by_level.setdefault(spec.metadata["ledger_geography_level"], []).append( + spec + ) + assert spec.metadata["soi_control_concept"] == ( + "schedule_d_taxable_net_gain" + ) + assert spec.metadata["uprating_index_source_record_id"] == _CG_T14_RETURNS + assert set(by_level) == {"state", "congressional_district"} + assert len(by_level["state"]) == len(state_values) + assert len(by_level["congressional_district"]) == 2 * len(two_district_states) + + factor = control / cd_us + for spec in by_level["state"]: + raw = state_values[spec.metadata["state_fips"]] + assert math.isclose(spec.value, raw * factor, rel_tol=1e-12) + state_by_fips = { + spec.metadata["state_fips"]: spec.value for spec in by_level["state"] + } + district_totals: dict[str, float] = {} + for spec in by_level["congressional_district"]: + fips = spec.metadata["state_fips"] + district_totals[fips] = district_totals.get(fips, 0.0) + spec.value + for fips, total in district_totals.items(): + assert math.isclose(total, state_by_fips[fips], rel_tol=1e-12) + + expected = control * sum(state_values.values()) / cd_us + states_total = sum(state_by_fips.values()) + assert math.isclose(states_total, expected, rel_tol=1e-12) + assert states_total <= control * (1 + 1e-12) + assert sum(district_totals.values()) <= control * (1 + 1e-12) + + check() + + def test_cross_period_soi_taxable_interest_agi_slice_without_total_is_dropped() -> None: source_record_id = ( "irs_soi.ty2022.historic_table_2.us.200k_to_500k.taxable_interest_amount" diff --git a/test_support/microcosm_build/us_fiscal_targets.py b/test_support/microcosm_build/us_fiscal_targets.py index 7f83264f18..4cb6fb7a44 100644 --- a/test_support/microcosm_build/us_fiscal_targets.py +++ b/test_support/microcosm_build/us_fiscal_targets.py @@ -18,6 +18,7 @@ US_FISCAL_TARGET_COVERAGE_REQUIREMENTS, US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES, US_FISCAL_TARGET_REFERENCES, + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS, US_FISCAL_TARGET_SUPPORT_EXCLUSIONS, US_JCT_TAX_EXPENDITURE_REFORMS, US_NONNEGATIVE_SOURCE_OUTPUTS, @@ -160,12 +161,37 @@ def _w2_tips_return_count_fact(tax_year: int) -> dict[str, object]: ) -def _four_rule_synthetic_feed(monkeypatch): +def _cd_column_fact( + measure_id: str, + source_column_id: str, + *, + groupby_value_id: str = "us", + geography_level: str = "country", + geography_id: str = "0100000US", + value: float = 1_000_000.0, +) -> dict[str, object]: + """A congressional-district fact read from IRS column ``source_column_id``, + as every pinned-feed fact records in ``layout.source_column_id``.""" + fact = _soi_congressional_district_fact( + measure_id, + value, + groupby_value_id=groupby_value_id, + geography_level=geography_level, + geography_id=geography_id, + ) + fact["layout"]["source_column_id"] = source_column_id + return fact + + +def _exclusion_rule_synthetic_feed(monkeypatch): """Synthetic facts that exercise every receipt rule, and each rule's ids. The register holds no bypass since d179, so this reviews a synthetic one: a taxable-interest cell excluded at ty2023 with its ty2020 vintage - allowlisted. + allowlisted. The congressional-district SALT facts read the income-tax + column N18425/A18425 that the pinned feed labels as the limited deduction, + and one reads N18460, the column a corrected package would read, which + must compile. """ tanf = ( "hhs_acf_tanf.fy2024.cash_assistance.ar." @@ -217,8 +243,30 @@ def _four_rule_synthetic_feed(monkeypatch): ) ), ] + salt_excluded = [ + _cd_column_fact("limited_state_local_taxes_returns", "N18425"), + _cd_column_fact("limited_state_local_taxes_amount", "A18425"), + _cd_column_fact( + "limited_state_local_taxes_amount", + "A18425", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ), + ] + salt_kept = _cd_column_fact( + "limited_state_local_taxes_returns", + "N18460", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + facts = [*facts, *salt_excluded, salt_kept] expected = { "reviewed_exclusion": sorted([tanf, interest_excluded]), + "source_column_concept_exclusion": sorted( + fact["lineage"]["source_record_id"] for fact in salt_excluded + ), "all_vintage_reviewed_exclusion": sorted( [ *( diff --git a/tools/build_us_fiscal_refresh_release.py b/tools/build_us_fiscal_refresh_release.py index e421049cfc..7cc40f2614 100644 --- a/tools/build_us_fiscal_refresh_release.py +++ b/tools/build_us_fiscal_refresh_release.py @@ -86,6 +86,7 @@ SIPP_2023_VOLUNTARY_FILING_DONOR_SIZE_BYTES, SOI_VARIABLE_MAP, US_FISCAL_TARGET_COVERAGE_REQUIREMENTS, + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS, US_FISCAL_TARGET_SUPPORT_EXCLUSIONS, US_JCT_TAX_EXPENDITURE_REFORMS, US_MEDICAID_ENROLLMENT_TARGET_TABLE, @@ -14687,6 +14688,16 @@ def _main(argv: Sequence[str] | None = None) -> None: US_FISCAL_TARGET_SUPPORT_EXCLUSIONS.items() ) ] + coverage["fiscal_target_source_column_exclusions"] = [ + { + "measure_id": measure_id, + "source_column_id": source_column_id, + "reason": reason, + } + for (measure_id, source_column_id), reason in sorted( + US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS.items() + ) + ] coverage[US_FISCAL_TARGET_EXCLUSION_RECEIPT_KEY] = fiscal_target_exclusion_receipt write_us_source_coverage_diagnostics( coverage, release_dir / "us_source_coverage.json" From a17db0f15279ace288a804b6c3786e0735e27f21 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 12 Oct 2026 08:35:02 -0400 Subject: [PATCH 4/8] Exclude the CD interest-paid column and fail closed on missing columns and totals (#1040 review) The congressional-district package reads interest_paid_deduction_* from N19300/A19300, Schedule A line 8a mortgage interest, while interest_deduction is the line 10 total; add both to US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS. A congressional-district fact of a listed measure with no layout.source_column_id now refuses (MissingSoiSourceColumnError) instead of compiling unchecked, and a capital-gains share row that has a control but no national total of its own family refuses (MissingSoiCapitalGainsShareTotalError) instead of dropping silently. Co-Authored-By: Claude Opus 5.5 --- .../build/us_runtime/fiscal_targets.py | 85 +++++++++++++-- .../engine_free/us/test_us_fiscal_targets.py | 103 ++++++++++++++---- 2 files changed, 155 insertions(+), 33 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 6487592936..d56dfda0cb 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -873,10 +873,20 @@ def _us_hierarchy_seed( # these measure ids read and what assigned_aca_ptc measures. The amount was # dropped before by a hard-coded congressional-district check; the returns # row compiled as premium-tax-credit recipients. +# - N19300/A19300 as "Mortgage interest paid" (Schedule A line 8a). The +# interest deduction interest_deduction measures is line 10: lines 8a-8c +# plus investment interest (line 9). The file carries 8b, 8c and 9 as +# separate columns and no line 10 total. # -# Chronicle's congressional_district_2022 package reads these four columns -# under the limited_state_local_taxes_* and premium_tax_credit_* measure ids; -# every other measure it shares with Historic Table 2 reads the same column. +# Chronicle's congressional_district_2022 package reads these six columns +# under the limited_state_local_taxes_*, premium_tax_credit_* and +# interest_paid_deduction_* measure ids. Of its other 50 measures, the 46 it +# shares with a Historic Table 2 package read the same column there, and the +# other four (charitable contributions A19700, qualified business income +# deduction A04475) match their doc-guide definitions. +# A congressional-district fact of a listed measure that carries no +# layout.source_column_id refuses the compile, so a feed schema change cannot +# silently lift the exclusion. US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS: dict[tuple[str, str], str] = { ("limited_state_local_taxes_returns", "N18425"): ( "N18425 counts returns with state and local income taxes (Schedule A " @@ -908,6 +918,22 @@ def _us_hierarchy_seed( "congressional_district_2022 package reads it under this measure id: " "TY2022 US $14.2B against $53.1B for A85770 in the same file." ), + ("interest_paid_deduction_returns", "N19300"): ( + "N19300 counts returns with home mortgage interest reported on Form " + "1098 (Schedule A line 8a), not returns with the interest deduction " + "(line 10: lines 8a-8c and investment interest on line 9) " + "interest_deduction measures. Chronicle's congressional_district_2022 " + "package reads it under this measure id: TY2022 US 11,352,500 " + "returns; the file has no line 10 column." + ), + ("interest_paid_deduction_amount", "A19300"): ( + "A19300 is home mortgage interest reported on Form 1098 (Schedule A " + "line 8a), one part of the line 10 interest deduction " + "interest_deduction measures. Chronicle's congressional_district_2022 " + "package reads it under this measure id: TY2022 US $140.1B, 85.6% of " + "lines 8a-8c and 9 ($163.7B, A19300 + A19500 + A19530 + A19570) in " + "the same file." + ), } @@ -1607,9 +1633,10 @@ def _rebase_stale_soi_capital_gains_distributions( congressional-district state or district row of that file's US row), scaled to the concept-matched control of ``_soi_capital_gains_active_totals`` at or after that period, and declares the bridge in ``soi_source_concept`` - / ``soi_control_concept``. Without such a control or family total, the row - is dropped. Each family's national all-AGI row retires because the - control's own row owns the national concept. + / ``soi_control_concept``. Without such a control the row is dropped; with + a control but no family total its share is undefined and the compile + refuses. Each family's national all-AGI row retires because the control's + own row owns the national concept. One factor per family, quantity and period scales a family's state and district rows alike, so the district-to-state hierarchy reconciliation that @@ -1640,7 +1667,13 @@ def _rebase_stale_soi_capital_gains_distributions( ): continue if source_total is None or source_total.value == 0: - continue + raise MissingSoiCapitalGainsShareTotalError( + f"SOI capital-gains row {spec.name!r} has a control but its " + f"{family} family has no nonzero national all-AGI total for " + f"{key} at {spec.metadata['source_period']}, so its share is " + "undefined; a line-7 row must never ship as a level " + "(microcosm#1038)." + ) if _is_national_all_agi_spec(spec): continue @@ -1787,6 +1820,10 @@ class AmbiguousSoiCapitalGainsShareTotalError(ValueError): """One share family carries two different national totals for one period.""" +class MissingSoiCapitalGainsShareTotalError(ValueError): + """A share-family row has a control but no national total of its own.""" + + @dataclass(frozen=True) class _SoiShareTotal: value: float @@ -2759,14 +2796,38 @@ def _period_free_source_record_id(source_record_id: str) -> str: return _normalized_record_set_id(source_record_id) +class MissingSoiSourceColumnError(ValueError): + """A congressional-district fact of a listed measure names no IRS column.""" + + def _is_source_column_concept_exclusion(fact: object) -> bool: - """Whether an SOI fact reads a column its measure id does not mean.""" + """Whether an SOI fact reads a column its measure id does not mean. + + A congressional-district fact of a measure the register lists must name + its column: without one the exclusion could not tell a mislabeled column + from a corrected one, so the compile refuses instead of guessing. + """ if _source_name(fact) != "irs_soi": return False - return ( - _measure_id(fact), - _str_at(fact, "layout", "source_column_id"), - ) in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS + measure_id = _measure_id(fact) + if measure_id not in { + measure for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS + }: + return False + column = _str_at(fact, "layout", "source_column_id") + if not column: + if ( + _soi_record_set_family(_str_at(fact, "layout", "record_set_id")) + == "congressional_district" + ): + raise MissingSoiSourceColumnError( + f"{_source_record_id(fact)} is a congressional-district " + f"{measure_id} fact with no layout.source_column_id, so " + "US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS cannot tell which " + "IRS column it reads (microcosm#1038)." + ) + return False + return (measure_id, column) in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS def _is_all_vintage_support_exclusion(source_record_id: str) -> bool: diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index 64d27d4180..5b2354811b 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -129,10 +129,14 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: groupby_value_id="hi_01", geography_id="5001700US1501", ), - _soi_congressional_district_fact( + # N19300 is line 8a mortgage interest, not the line 10 interest + # deduction the measure id means, so it never compiles either. + _cd_column_fact( "interest_paid_deduction_returns", - 23_456, + "N19300", + value=23_456, groupby_value_id="hi_01", + geography_level="congressional_district", geography_id="5001700US1501", ), _soi_congressional_district_fact( @@ -190,7 +194,9 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: if spec.metadata.get("ledger_geography_level") == "congressional_district" ] assert not any( - source_id.endswith("premium_tax_credit_amount") + source_id.endswith( + ("premium_tax_credit_amount", "interest_paid_deduction_returns") + ) and ".congressional_district_2022." in source_id for source_id in source_ids ) @@ -198,7 +204,7 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: assert any( "hi_total.adjusted_gross_income" in source_id for source_id in source_ids ) - assert len(cd_specs) == 11 + assert len(cd_specs) == 10 by_measure = { spec.metadata["source_measure_id"]: spec for spec in cd_specs @@ -242,12 +248,6 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: ) assert qbi_returns.value == 12_345 - interest_returns = by_measure["interest_paid_deduction_returns"] - assert interest_returns.metadata["measure_mode"] == "indicator_sum" - assert interest_returns.metadata["base_variable"] == "interest_deduction" - assert interest_returns.metadata["itemized_only"] == "true" - assert interest_returns.value == 23_456 - charitable_returns = by_measure["charitable_returns"] assert charitable_returns.metadata["measure_mode"] == "indicator_sum" assert charitable_returns.metadata["base_variable"] == "charitable_deduction" @@ -828,28 +828,68 @@ def test_source_column_exclusion_drops_only_the_mislabeled_column() -> None: def test_source_column_exclusions_name_the_columns_their_guides_define() -> None: - """Each entry's reason names its own column and the column its measure id - means, as 22incddocguide.docx defines them: N18425/A18425 (state and local - income taxes, Schedule A line 5a) stand in for the limited deduction - N18460/A18460 (line 5e), and N85530/A85530 (additional Medicare tax) for - the premium tax credit N85770/A85770.""" + """Each entry's reason names its own column and what its measure id means, + as 22incddocguide.docx defines them: N18425/A18425 (state and local income + taxes, Schedule A line 5a) stand in for the limited deduction N18460/A18460 + (line 5e), N85530/A85530 (additional Medicare tax) for the premium tax + credit N85770/A85770, and N19300/A19300 (line 8a mortgage interest) for the + line 10 interest deduction, which the file has no column for.""" right_columns = { ("limited_state_local_taxes_returns", "N18425"): "N18460", ("limited_state_local_taxes_amount", "A18425"): "A18460", ("premium_tax_credit_returns", "N85530"): "N85770", ("premium_tax_credit_amount", "A85530"): "A85770", + ("interest_paid_deduction_returns", "N19300"): None, + ("interest_paid_deduction_amount", "A19300"): None, } assert set(US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS) == set(right_columns) for key, reason in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS.items(): measure_id, column = key assert column in reason - assert right_columns[key] in reason # Counts read N columns and amounts read A columns. - assert ( - column[0] - == right_columns[key][0] - == ("N" if measure_id.endswith("_returns") else "A") - ) + assert column[0] == ("N" if measure_id.endswith("_returns") else "A") + right_column = right_columns[key] + if right_column is None: + assert "line 10" in reason + else: + assert right_column in reason + assert right_column[0] == column[0] + + +def test_source_column_exclusion_refuses_a_district_fact_without_a_column() -> None: + """A congressional-district fact of a listed measure that names no IRS + column refuses the compile and the receipt, because the register could not + tell a mislabeled column from a corrected one. A Historic Table 2 fact of + the same measure without a column (the US package publishes none) still + compiles.""" + unnamed = _soi_congressional_district_fact( + "limited_state_local_taxes_amount", + 250_000_000.0, + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) + ht2 = _soi_capital_gains_fact( + 2022, + source_record_id=( + "irs_soi.ty2022.historic_table_2.us.all.limited_state_local_taxes_amount" + ), + measure_id="limited_state_local_taxes_amount", + value=122_000_000_000.0, + ) + reference = packaged_reference_facts() + + for facts in ([*reference, unnamed], [*reference, ht2, unnamed]): + with pytest.raises(fiscal_targets.MissingSoiSourceColumnError): + compile_us_fiscal_target_registry(facts, allow_unaged_dollar_targets=True) + with pytest.raises(fiscal_targets.MissingSoiSourceColumnError): + us_fiscal_target_exclusion_receipt(facts) + registry = compile_us_fiscal_target_registry( + [*reference, ht2], allow_unaged_dollar_targets=True + ) + assert ht2["lineage"]["source_record_id"] in { + spec.metadata["ledger_source_record_id"] for spec in registry.specs + } def test_exclusion_receipt_is_complete_for_any_subfeed(monkeypatch) -> None: @@ -5000,6 +5040,27 @@ def test_congressional_district_capital_gains_rows_without_a_control_are_dropped assert _congressional_district_capital_gains_specs(registry) == {} +def test_capital_gains_share_row_without_its_family_total_refuses() -> None: + """A congressional-district row with a Schedule D control but no US row of + its own has no defined share. The compile refuses instead of dropping it + silently or letting another family's total stand in (microcosm#1038).""" + facts = [ + *packaged_reference_facts(), + *( + fact + for fact in _capital_gains_cd_package_facts() + if fact["layout"]["groupby_value_id"] != "us" + ), + *_capital_gains_ht2_facts(), + *_capital_gains_t14_facts(), + ] + + with pytest.raises( + fiscal_targets.MissingSoiCapitalGainsShareTotalError, match="hi_total" + ): + compile_us_fiscal_target_registry(facts, allow_unaged_dollar_targets=True) + + def test_capital_gains_share_family_refuses_two_national_totals() -> None: """Two different national totals for one share family, quantity and period refuse the compile: feed order must not pick a share denominator.""" From 4e86ec3580e3e40ef501ac60fa1c16834143742f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 12 Oct 2026 08:52:54 -0400 Subject: [PATCH 5/8] Re-pin #1036 on the 51aa40f feed after merging main main re-pinned the US Chronicle feed to 51aa40f (#940), whose capital-gains cells are identical to c5e5bf8's. Compiled there, this change still moves the same 60 Historic Table 2 return counts by 0.415202720927 and adds the concept keys to the 60 amount specs; national_state goes from e1f9906bf0e0 to 882f4764a248 at 6,102 targets (full f5bd921d6b21 to f09571684885). Co-Authored-By: Claude Opus 5.5 --- docs/us-chronicle-feed-repin.md | 5 ++++- docs/us-soi-capital-gains-concepts.md | 6 ++++++ .../tests/engine_free/us/test_us_fiscal_targets.py | 4 ++-- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/us-chronicle-feed-repin.md b/docs/us-chronicle-feed-repin.md index 0c4787e508..400a897b6f 100644 --- a/docs/us-chronicle-feed-repin.md +++ b/docs/us-chronicle-feed-repin.md @@ -210,7 +210,10 @@ Table 1.4 ty2023. It took over the returns control and raised the 51 Historic Table 2 state return counts from 12,289,836 to 29,599,604. The control is now chosen by concept and period alone ([us-soi-capital-gains-concepts.md](us-soi-capital-gains-concepts.md)). The -counts are unchanged and `national_state` is `65e4dde11c83`. +counts are unchanged and `national_state` is `65e4dde11c83` on this feed. On +the 2026-10-04 `51aa40f` re-pin, whose capital-gains cells are identical, the +same 60 return counts move and `national_state` goes from `e1f9906bf0e0` to +`882f4764a248` at 6,102 targets. The labelled feed exposed one latent defect in microcosm, fixed in this change: `apply_us_medicaid_enrollment_substitutions` built Rhode Island's diff --git a/docs/us-soi-capital-gains-concepts.md b/docs/us-soi-capital-gains-concepts.md index 4475329c53..feefc5d3e8 100644 --- a/docs/us-soi-capital-gains-concepts.md +++ b/docs/us-soi-capital-gains-concepts.md @@ -116,5 +116,11 @@ crosswalk, then narrowed to `national_state`: - The state return counts are back at the Build P values; California's 1,566,996.66 equals the July scorecard's. +On the 2026-10-04 `51aa40f` re-pin (#940), whose capital-gains cells are +identical, the same 60 values move by the same factor and the same 60 amount +specs gain only the concept keys: 33,250 compiled, 6,102 on +`national_state`, which goes from `e1f9906bf0e0` to `882f4764a248` (full +`f5bd921d6b21` to `f09571684885`). + `test_pinned_feed_no_soi_state_family_sums_past_its_national_target` holds the invariant across all 63 SOI state families on the surface. diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index e7656ce424..aaed3a00a3 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -895,7 +895,7 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ) -> None: """The release surface on the pinned feed: 33,250 compiled targets after Medicaid substitution, 6,102 national_state targets at registry - PENDING_1036_NS, 32 CHIP rows none of them for an M-CHIP state, no + 882f4764a248, 32 CHIP rows none of them for an M-CHIP state, no other-income row and no tips return count. microcosm#1035 moved it from e1f9906bf0e0 at the same counts by choosing the capital-gains control by concept and period, which returns the 51 Historic Table 2 state return @@ -911,7 +911,7 @@ def test_pinned_feed_national_state_surface_restores_the_fences( registry, surface, _ = pinned_feed_national_state_surface assert len(registry.specs) == 33_250 assert len(surface.specs) == 6_102 - assert surface.version == "PENDING_1036_NS" + assert surface.version == "882f4764a248" chip = [ spec for spec in surface.specs From d1614b2f5cc62be163c2b3ddb13f4e87b333ad7c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 12 Oct 2026 09:07:39 -0400 Subject: [PATCH 6/8] Address the #1040 review; re-pin on the 51aa40f feed; align the column exclusions with #1224 Review (review-1040.md) follow-ups: - Interest paid stays out of US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS: N19300 (line 8a mortgage interest) is a component of the line 10 deduction, not another concept, and it feeds state_cd's itemized bridge; its share-or-drop decision is a separate chip. The register comment, docs and claims now say 50 of the package's 56 measures have a Historic Table 2 twin. - A share row with no national row of its own family is dropped, as the established rule and test_stale_soi_capital_gains_without_source_total_is_dropped require; the pinned invariant is now two-sided (every compared district family keeps its state and district rows and sits in (0.85, 1] of its national target), so a vanished or partial family fails. - The CD-vs-HT2 tilt (0.8% returns, 2.1% amounts) is stated plainly; the amount agreement is reported on the demeaned comparison with the worst states named. - The Hypothesis property draws Historic Table 2 rows alongside and checks each family divides by, and names, its own US row; comments on CD boundary vintages sharing a family and on a corrected PTC package. The SALT/PTC column exclusions are now #1224 (a bug fix that lands on its own); this branch carries them byte-identical until it merges. Pins on the 51aa40f feed: 31,784 compiled (full 6e1ac91262bc), national_state 6,102 at 84504a7afd57. Co-Authored-By: Claude Opus 5.5 --- .../1038-cd-capital-gains-shares.fixed.md | 2 +- .../1038-cd-source-column-exclusions.fixed.md | 1 + docs/us-acs-local-soi-target-surface.md | 43 ++++-- docs/us-chronicle-feed-repin.md | 19 +-- docs/us-fact-to-target.md | 13 ++ docs/us-soi-capital-gains-concepts.md | 84 +++++++---- .../build/us_runtime/fiscal_targets.py | 74 ++++------ .../engine_free/us/test_us_fiscal_targets.py | 136 +++++++++++++----- tools/us_acs_local_cd_surface.py | 6 +- 9 files changed, 242 insertions(+), 136 deletions(-) create mode 100644 changelog.d/1038-cd-source-column-exclusions.fixed.md diff --git a/changelog.d/1038-cd-capital-gains-shares.fixed.md b/changelog.d/1038-cd-capital-gains-shares.fixed.md index 07af399c0b..cd9ac24eb3 100644 --- a/changelog.d/1038-cd-capital-gains-shares.fixed.md +++ b/changelog.d/1038-cd-capital-gains-shares.fixed.md @@ -1 +1 @@ -On the `full` target surface, congressional-district capital-gains rows now enter as shares of the file's own US row scaled to the Table 1.4 Schedule D control, as Historic Table 2 rows do; the file's US row retires. Every capital-gains share row names its denominator in `soi_share_total_source_record_id`. A new reviewed source-column exclusion, `US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`, drops congressional-district facts read from IRS columns their measure id does not mean: N18425/A18425 (state and local income taxes) read as the limited SALT deduction, and N85530/A85530 (additional Medicare tax) read as the premium tax credit. It replaces the hard-coded congressional-district PTC-amount check and is listed in the exclusion receipt. On the pinned feed, the congressional-district capital-gains states and districts each move from 29,845,710 returns (2.41x the national target) and $1,522.2B (1.198x) to 12,392,020 and $1,270.9B, and no congressional-district family sums past its national target on `full` (microcosm#1038). +On the `full` target surface, congressional-district capital-gains rows now enter as shares of the district file's own US row scaled to the Table 1.4 Schedule D control, as Historic Table 2 rows do; the file's US row retires, and every capital-gains share row names its denominator in `soi_share_total_source_record_id`. On the pinned feed the district file's states and districts each move from 29,845,710 returns (2.41x the national target) and $1,522.2B (1.198x) to 12,392,020 and $1,270.9B, and no congressional-district family sums past its national target on `full` (microcosm#1038). diff --git a/changelog.d/1038-cd-source-column-exclusions.fixed.md b/changelog.d/1038-cd-source-column-exclusions.fixed.md new file mode 100644 index 0000000000..dc49547865 --- /dev/null +++ b/changelog.d/1038-cd-source-column-exclusions.fixed.md @@ -0,0 +1 @@ +The US fiscal-target compiler drops congressional-district SOI facts read from an IRS column their measure id does not mean, through a new reviewed register keyed by (measure id, IRS column), `US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`: N18425/A18425 (state and local income taxes, Schedule A line 5a) published as the limited SALT deduction, and N85530/A85530 (additional Medicare tax, Form 8959 line 24) published as the premium tax credit. It replaces the hard-coded congressional-district PTC-amount check, refuses a listed district fact that names no column, and appears in the exclusion receipt and the release coverage report. On the pinned feed the full surface loses the 976 district SALT specs, which summed to 1.98x the national target, and the 488 district PTC-return specs (microcosm#1038). diff --git a/docs/us-acs-local-soi-target-surface.md b/docs/us-acs-local-soi-target-surface.md index e072362569..80fcd27bb6 100644 --- a/docs/us-acs-local-soi-target-surface.md +++ b/docs/us-acs-local-soi-target-surface.md @@ -79,6 +79,20 @@ their role is `soi_fiscal_distribution`, so `totals` is unchanged: | **Admin specs** | **4,380** | **760** | **31,474** | With the 487 population marginals `state` now calibrates to 4,867 targets. + +**Since microcosm#1038** the compiler drops the district file's facts read +from the wrong IRS column (`US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS` in +`us_runtime/fiscal_targets.py`): the SALT rows read from N18425/A18425 +(state and local income taxes) and the premium-tax-credit rows read from +N85530/A85530 (additional Medicare tax). `state` and `state_cd` are +unchanged. `totals` loses the 487 district-file premium-tax-credit return +rows and `full` loses those and the 974 district-file SALT rows: + +| Family | `state` | `totals` | `full` | +|---|---:|---:|---:| +| `irs_soi` | 4,227 | 120 | 29,860 | +| **Admin specs** | **4,380** | **273** | **30,013** | + The rest of this section describes the 2026-09-22 measurement. The 487 population marginals (51 states and 436 congressional districts) are @@ -104,8 +118,12 @@ congressional-district file. The feed-gated test ### What `totals` contains, and what it does not -All 607 SOI specs that `totals` keeps are ACA premium tax credit measures (547 -returns, 60 amounts), 454 of them at congressional-district geography. +All 607 SOI specs that `totals` kept on 2026-09-22 are ACA premium tax credit +measures (547 returns, 60 amounts), 454 of them at congressional-district +geography. Since microcosm#1038 it keeps 120 (60 returns and 60 amounts, +all Historic Table 2: the 51 states and the 9 at-large district copies), 18 +of them at district geography: the 487 district-file return rows read +N85530, additional Medicare tax, not the premium tax credit. `_soi_target_role` in `us_runtime/fiscal_targets.py` gives named roles only to country-level all-income-range facts, the all-income-range PTC measures and the W-2 tip items; every other SOI fact gets `soi_fiscal_distribution`, @@ -292,8 +310,8 @@ bridged state row records its sibling and bridge factor. | Rows | Specs on the pinned feed | Why | |---|---:|---| -| District-file `limited_state_local_taxes_*`, both geographies | 974 | Column A18425/N18425 is state and local income taxes, not the limited SALT deduction (microcosm#1038) | -| District-file `premium_tax_credit_returns`, both geographies | 487 | Column N85530 is the additional Medicare tax, not the premium tax credit (microcosm#1038) | +| District-file `limited_state_local_taxes_*`, both geographies | 974 | Column A18425/N18425 is state and local income taxes, not the limited SALT deduction (microcosm#1038). Since #1038 the compiler drops them first; `STATE_CD_DEFECTIVE_CD_FILE_MEASURES` stays as a second fence | +| District-file `premium_tax_credit_returns`, both geographies | 487 | Column N85530 is the additional Medicare tax, not the premium tax credit (microcosm#1038); dropped first by the compiler, as above | | District `tax_filer_individual_count` | 436 | No state parent in either vintage, so it could not nest in a bound state target | | District rows of states with one district on the 117th plan (AK, DE, DC, MT, ND, SD, VT, WY) | 459 | The SOI file has no sub-state rows there. These rows are the state total copied, or for Montana split by population, so they carry no district information | | Historic Table 2 rows copied to at-large districts | 360 | The same copies from the other vintage | @@ -301,12 +319,12 @@ bridged state row records its sibling and bridge factor. | District rows of the four out-of-band rebase blocks | 34 | The factor band (above) | | Bridged state rows whose sibling ratio is out of band | 4 | The factor band (above) | -PR #1040 (awaiting a ruling) excludes the same SALT and PTC columns in the -compiler and rescales the district file's capital-gains rows by one national -factor. `state_cd` already takes district capital gains as within-state -shares of Historic Table 2, which a national factor does not move, so their -level changes only when Historic Table 2's own capital-gains rows do -(#1036). +Since microcosm#1038 the compiler excludes the same SALT and PTC columns +(`US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`). PR #1040 (awaiting a ruling) +rescales the district file's capital-gains rows by one national factor. +`state_cd` already takes district capital gains as within-state shares of +Historic Table 2, which a national factor does not move, so their level +changes only when Historic Table 2's own capital-gains rows do (#1036). North Carolina's district rows are bound. Before #1043 the packaged 117th→119th crosswalk was built from NC's 2016 plan rather than the 2019 plan @@ -556,7 +574,10 @@ record one of the four modes, before any release directory exists. calibrate stage's peak is the artifact write. - A checkpoint written before the sparse matrix (measures as dense H5 columns) is refused by `--stage calibrate`; re-run `--stage materialize`. -- To reproduce the 2026-09-22 hours rebuild (#974), pass `--soi-mode totals`. +- To reproduce the 2026-09-22 hours rebuild (#974), pass `--soi-mode totals` + on a compiler from before microcosm#1038; after it, `totals` no longer + carries the 487 district-file premium-tax-credit return rows that run + calibrated to. To reproduce a build that ran as `full` after `b7922b089`, pass `--soi-mode full`. - A checkpoint materialized under an earlier default records its own mode and diff --git a/docs/us-chronicle-feed-repin.md b/docs/us-chronicle-feed-repin.md index e0bb63d9a2..644645937f 100644 --- a/docs/us-chronicle-feed-repin.md +++ b/docs/us-chronicle-feed-repin.md @@ -212,15 +212,16 @@ chosen by concept and period alone ([us-soi-capital-gains-concepts.md](us-soi-capital-gains-concepts.md)). The counts are unchanged and `national_state` is `65e4dde11c83`. -**Follow-up (27 September 2026, microcosm#1038).** On the `full` surface, -the congressional-district file's capital-gains rows are now shares of its US -row scaled to the same control, and the US row retires. The package's four -mislabeled columns are dropped: N18425/A18425 (state and local income taxes), -read as the limited SALT deduction, and N85530/A85530 (additional Medicare -tax), read as the premium tax credit. That leaves 31,376 compiled targets, all of them on -`full` (`059dc56d78db`). `national_state` keeps its 5,694 targets and values. -It moves to `47dce807b412` only because its 102 Historic Table 2 -capital-gains rows now name their share denominator in +**Follow-up (microcosm#1038).** On the `full` surface, the +congressional-district file's capital-gains rows are shares of its US row +scaled to the same control, and the US row retires (#1040). The package's +four mislabeled columns are dropped (#1224): N18425/A18425 (state and local +income taxes), read as the limited SALT deduction, and N85530/A85530 +(additional Medicare tax), read as the premium tax credit. On the `51aa40f` +feed that leaves 31,784 compiled targets, all of them on `full` +(`6e1ac91262bc`). `national_state` keeps its 6,102 targets and values and +moves from `882f4764a248` to `84504a7afd57` only because its 102 Historic +Table 2 capital-gains rows now name their share denominator in `soi_share_total_source_record_id`. The labelled feed exposed one latent defect in microcosm, fixed in this diff --git a/docs/us-fact-to-target.md b/docs/us-fact-to-target.md index e4ae417a8d..af5b185b75 100644 --- a/docs/us-fact-to-target.md +++ b/docs/us-fact-to-target.md @@ -204,6 +204,19 @@ should calibrate, add its id to `US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES` with a reason. `us_source_coverage.json`'s `fiscal_target_exclusion_receipt` lists the ids each rule dropped or allowed in a release. +A fact whose package reads the wrong IRS column for its measure id is a +different defect: every fact read from that column is wrong, at every +geography and vintage, while the same measure read from the right column is +fine. `US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS` is keyed by (measure id, +`layout.source_column_id`) for these, so a package corrected to read the +right column passes untouched; its receipt rule is +`source_column_concept_exclusion` (microcosm#1038). Check the column against +the IRS documentation guide before adding an entry, and list a column there +only when it carries another concept. A column that is a component or a +superset of the measure's concept (the district file's line 8a mortgage +interest under `interest_paid_deduction_*`, its line 7 capital gains) needs a +share-or-drop decision instead. + ## 5. Add a reform-coverage probe Probes are **generated data**: author them in diff --git a/docs/us-soi-capital-gains-concepts.md b/docs/us-soi-capital-gains-concepts.md index 784cdecd72..71ea372072 100644 --- a/docs/us-soi-capital-gains-concepts.md +++ b/docs/us-soi-capital-gains-concepts.md @@ -152,20 +152,31 @@ takes (a): district share left over after the district's AGI distribution has a log standard deviation of 0.227 for counts and 0.223 for amounts. The AGI-composition error of the share proxy is 0.024 and 0.009. -- **The rebased rows agree with the HT2 state rows about as well as other - pairs on `full` already do.** The congressional-district and HT2 state - return counts for capital gains differ by a median 0.6% (95th percentile - 2.1%). The `return_count` pair differs by 1.6% (2.4%). - - Amounts differ more: a median 2.8% (95th percentile 15.4%, Montana - −27.0%). The cause is item suppression in the congressional-district - file's A01000. Montana's is 32.5% below HT2's at the state level. That is - still inside the qualified-dividends amount pair, 11.6% (21.7%). - -The congressional-district file has no rows for other areas, so its US row -equals the sum of its 51 states. The rebased states and districts therefore -each sum to the control itself. HT2's US row includes other areas, so its -rebased states sum to 99.2% of the control for returns and 98.0% for -amounts. +- **The rebased return counts agree with the HT2 state rows as well as any + pair on `full`; the amounts are the loosest pair.** Both files' state rows + stay on `full`. Each of the 45 measures they share pairs two state targets + for every state. After dividing out each pair's common offset (coverage and + aging, which differ by measure), the capital-gains return counts differ by + a median 0.4% and at most 1.4%. The capital-gains amounts differ by a + median 1.8% and up to 28% (Montana), 18% (West Virginia) and 17% + (Mississippi): the widest 95th percentile of the 45 pairs. The cause is item + suppression in the congressional-district file's A01000; Montana's is 32.5% + below HT2's at the state level. Other pairs have worse single states (the + qualified-dividends amount pair: Hawaii, −58%). + +**The two families split the control differently.** The congressional-district +file has no rows for other areas (territories, APO/FPO and foreign +addresses), so its US row equals the sum of its 51 states, and its rebased +states and districts each sum to the control itself. HT2's US row includes +other areas, so its rebased states sum to 99.2% of the control for returns +and 98.0% for amounts. Every congressional-district capital-gains target +therefore sits above the matching HT2 share, by 0.8% for returns and 2.1% +for amounts in aggregate. This is accepted, not corrected. The model has +households in the 50 states and DC only, and the national Table 1.4 row on +the same surface already asks them for the whole control; the other-area +part of HT2's split is a share no model household can carry. The +alternative, scaling the district file by HT2's 51-state share of its own US +row, would mix the two families' totals in one factor. **Vintage.** A share divides a row by the US row of the same package, so the package's period stamp cancels. The value lands at the control's period @@ -174,15 +185,21 @@ ty2023, as on the pinned feed, or truthfully at 2022 (`test_congressional_district_capital_gains_ignore_the_package_stamp`). microcosm#1030 corrects restamped facts to age from their data year, but it -refuses any restamped fact that was rebased. Before it merges with this change, -that refusal must admit a share whose `soi_share_total_source_record_id` is a -restamp of the same package, and leave `uprating_to_period` at the control's -period. Starting the aging at 2022 instead would apply the Table 1.4 -2022-to-2023 fall twice (x0.761). +refuses any restamped fact that was rebased, so as written it refuses every +compile that carries these rows. It must admit a share whose +`soi_share_total_source_record_id` is a restamp of the same package and +stamp: move `source_period` and `uprating_from_period` to the data year, +and leave `uprating_to_period` at the control's period. Starting the aging at +2022 instead would apply the Table 1.4 2022-to-2023 fall twice (x0.761). +Whichever of the two changes merges second runs its CI on a merge ref that +contains the first; the stamp test above then fails until that narrowing is +in place. ### Effect on the pinned feed -Release compile at 2024, as above: +Release compile at 2024 on the `51aa40f` feed, as above. "Before" is #1036 on +main; "after" adds microcosm#1038's changes, including the column exclusions +below (#1224), which remove 1,464 of the 1,466 specs: | `full` surface | Before | After | |---|---:|---:| @@ -190,21 +207,22 @@ Release compile at 2024, as above: | Congressional-district capital-gains amounts, states and districts (each sum) | $1,522.19B | $1,270.86B | | California returns / amount | 3,808,930 / $214.08B | 1,581,478 / $178.73B | | New York returns / amount | 2,015,540 / $127.96B | 836,858 / $106.83B | -| Compiled targets, all on `full` | 32,842 (`e02123644d42`) | 31,376 (`059dc56d78db`) | -| `national_state` targets | 5,694 (`65e4dde11c83`) | 5,694 (`47dce807b412`) | +| Compiled targets, all on `full` | 33,250 (`f09571684885`) | 31,784 (`6e1ac91262bc`) | +| `national_state` targets | 6,102 (`882f4764a248`) | 6,102 (`84504a7afd57`) | - The 974 congressional-district capital-gains rows move by one factor per measure: 0.415202720927 for returns and 0.834893818418 for amounts. - The two US rows retire. - The Historic Table 2 rows keep their values. The 120 of them (102 on - `national_state`) gain only `soi_share_total_source_record_id`. + `national_state`) gain only `soi_share_total_source_record_id`, which is + the whole `national_state` change. ### Two mislabeled columns in the same package On the same surface, Chronicle's `congressional_district_2022` package reads -four IRS columns under measure ids whose concept they do not carry. They are -the only differences between its column mapping and the Historic Table 2 -package's (all 56 measures compared): +four IRS columns under measure ids whose concept they do not carry. Of the +package's 56 measures, 50 have a twin in a Historic Table 2 package, and +these four are the only ones that read a different column from their twin: | Measure id | Congressional-district column | Doc-guide definition | Column the measure means | |---|---|---|---| @@ -218,7 +236,19 @@ returns rows compiled as `assigned_aca_ptc` recipients. `US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS` in `fiscal_targets.py` now drops all four, keyed by (measure id, `layout.source_column_id`). It lists them in the release's exclusion receipt. A corrected package that reads the right -columns passes untouched. +columns passes untouched. A congressional-district fact of a listed measure +that names no column refuses the compile, so a feed schema change cannot +quietly lift the exclusion. + +Of the six measures with no HT2 twin, `interest_paid_deduction_*` reads +N19300 / A19300, mortgage interest paid (Schedule A line 8a), while +`interest_deduction` is the line 10 total (lines 8a-8c and investment +interest on line 9; A19300 is 85.6% of it in the TY2022 US row). That is a +component of the concept, as N01000 is a superset of capital gains, not a +different concept, so it is left out of the register. Whether those rows +should ship as shares of a national line 10 control or be dropped is a +separate decision. It also governs the `state_cd` surface, which bridges +them onto the HT2 basis. `test_pinned_feed_no_congressional_district_family_sums_past_its_national_target` holds the invariant for every congressional-district family on `full`. diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 8aff6578ea..c3f50779f8 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -896,18 +896,21 @@ def _us_hierarchy_seed( # prints "8926:24"). N85770/A85770 is what the Historic Table 2 rows of # these measure ids read and what assigned_aca_ptc measures. The amount was # dropped before by a hard-coded congressional-district check; the returns -# row compiled as premium-tax-credit recipients. -# - N19300/A19300 as "Mortgage interest paid" (Schedule A line 8a). The -# interest deduction interest_deduction measures is line 10: lines 8a-8c -# plus investment interest (line 9). The file carries 8b, 8c and 9 as -# separate columns and no line 10 total. +# row compiled as premium-tax-credit recipients. A package corrected to read +# N85770/A85770 compiles as assigned_aca_ptc targets with no further gate, +# which is intended: those columns are the premium tax credit. # -# Chronicle's congressional_district_2022 package reads these six columns -# under the limited_state_local_taxes_*, premium_tax_credit_* and -# interest_paid_deduction_* measure ids. Of its other 50 measures, the 46 it -# shares with a Historic Table 2 package read the same column there, and the -# other four (charitable contributions A19700, qualified business income -# deduction A04475) match their doc-guide definitions. +# Chronicle's congressional_district_2022 package reads these four columns +# under the limited_state_local_taxes_* and premium_tax_credit_* measure ids; +# the 46 other measures it shares with a Historic Table 2 package read the +# same column there. Of the six it has alone, interest_paid_deduction_* reads +# N19300/A19300, "Mortgage interest paid" (Schedule A line 8a), while +# interest_deduction is the line 10 total (lines 8a-8c plus investment +# interest on line 9; 8a is 85.6% of it in the TY2022 US row). That is a +# component of the measure's concept, not another concept, so it is not +# listed here: like the line 7 capital-gains rows, it needs a share-or-drop +# decision of its own, which also binds the state_cd surface's itemized +# bridge. # A congressional-district fact of a listed measure that carries no # layout.source_column_id refuses the compile, so a feed schema change cannot # silently lift the exclusion. @@ -942,22 +945,6 @@ def _us_hierarchy_seed( "congressional_district_2022 package reads it under this measure id: " "TY2022 US $14.2B against $53.1B for A85770 in the same file." ), - ("interest_paid_deduction_returns", "N19300"): ( - "N19300 counts returns with home mortgage interest reported on Form " - "1098 (Schedule A line 8a), not returns with the interest deduction " - "(line 10: lines 8a-8c and investment interest on line 9) " - "interest_deduction measures. Chronicle's congressional_district_2022 " - "package reads it under this measure id: TY2022 US 11,352,500 " - "returns; the file has no line 10 column." - ), - ("interest_paid_deduction_amount", "A19300"): ( - "A19300 is home mortgage interest reported on Form 1098 (Schedule A " - "line 8a), one part of the line 10 interest deduction " - "interest_deduction measures. Chronicle's congressional_district_2022 " - "package reads it under this measure id: TY2022 US $140.1B, 85.6% of " - "lines 8a-8c and 9 ($163.7B, A19300 + A19500 + A19530 + A19570) in " - "the same file." - ), } @@ -1658,10 +1645,10 @@ def _rebase_stale_soi_capital_gains_distributions( congressional-district state or district row of that file's US row), scaled to the concept-matched control of ``_soi_capital_gains_active_totals`` at or after that period, and declares the bridge in ``soi_source_concept`` - / ``soi_control_concept``. Without such a control the row is dropped; with - a control but no family total its share is undefined and the compile - refuses. Each family's national all-AGI row retires because the control's - own row owns the national concept. + / ``soi_control_concept``. Without such a control or a nonzero national + row of its own family the row is dropped, never shipped as a line-7 level. + Each family's national all-AGI row retires because the control's own row + owns the national concept. One factor per family, quantity and period scales a family's state and district rows alike, so the district-to-state hierarchy reconciliation that @@ -1692,13 +1679,9 @@ def _rebase_stale_soi_capital_gains_distributions( ): continue if source_total is None or source_total.value == 0: - raise MissingSoiCapitalGainsShareTotalError( - f"SOI capital-gains row {spec.name!r} has a control but its " - f"{family} family has no nonzero national all-AGI total for " - f"{key} at {spec.metadata['source_period']}, so its share is " - "undefined; a line-7 row must never ship as a level " - "(microcosm#1038)." - ) + # Unanchored: a share needs its own family's national row. Never + # divided by another family's total, never shipped as a level. + continue if _is_national_all_agi_spec(spec): continue @@ -1845,10 +1828,6 @@ class AmbiguousSoiCapitalGainsShareTotalError(ValueError): """One share family carries two different national totals for one period.""" -class MissingSoiCapitalGainsShareTotalError(ValueError): - """A share-family row has a control but no national total of its own.""" - - @dataclass(frozen=True) class _SoiShareTotal: value: float @@ -1865,7 +1844,10 @@ def _soi_capital_gains_share_national_totals( US row, never by another family's. Two different values under one key refuse the compile instead of letting feed order pick the denominator; equal values under two record ids keep the smaller id, so the named - denominator does not depend on feed order either. + denominator does not depend on feed order either. Congressional-district + packages of different boundary vintages for one tax year share the + ``congressional_district`` family, so their US rows must agree or the + compile refuses. """ totals: dict[tuple[str, str, str, str, str], _SoiShareTotal] = {} for fact in facts: @@ -3212,10 +3194,8 @@ def _is_source_column_concept_exclusion(fact: object) -> bool: return False column = _str_at(fact, "layout", "source_column_id") if not column: - if ( - _soi_record_set_family(_str_at(fact, "layout", "record_set_id")) - == "congressional_district" - ): + # Any congressional-district package, whatever its boundary vintage. + if ".congressional_district_" in _str_at(fact, "layout", "record_set_id"): raise MissingSoiSourceColumnError( f"{_source_record_id(fact)} is a congressional-district " f"{measure_id} fact with no layout.source_column_id, so " diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index 9de16ee6cf..a995f2ed3b 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -131,8 +131,8 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: groupby_value_id="hi_01", geography_id="5001700US1501", ), - # N19300 is line 8a mortgage interest, not the line 10 interest - # deduction the measure id means, so it never compiles either. + # The pinned feed reads it from N19300 (line 8a mortgage interest, a + # component of the line 10 deduction), which is not excluded. _cd_column_fact( "interest_paid_deduction_returns", "N19300", @@ -196,9 +196,7 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: if spec.metadata.get("ledger_geography_level") == "congressional_district" ] assert not any( - source_id.endswith( - ("premium_tax_credit_amount", "interest_paid_deduction_returns") - ) + source_id.endswith("premium_tax_credit_amount") and ".congressional_district_2022." in source_id for source_id in source_ids ) @@ -206,7 +204,7 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: assert any( "hi_total.adjusted_gross_income" in source_id for source_id in source_ids ) - assert len(cd_specs) == 10 + assert len(cd_specs) == 11 by_measure = { spec.metadata["source_measure_id"]: spec for spec in cd_specs @@ -243,6 +241,12 @@ def test_soi_congressional_district_targets_are_always_compiled() -> None: assert qbi.metadata["base_variable"] == "qualified_business_income_deduction" assert qbi.value == 2_000_000_000 + interest_returns = by_measure["interest_paid_deduction_returns"] + assert interest_returns.metadata["measure_mode"] == "indicator_sum" + assert interest_returns.metadata["base_variable"] == "interest_deduction" + assert interest_returns.metadata["itemized_only"] == "true" + assert interest_returns.value == 23_456 + qbi_returns = by_measure["qualified_business_income_deduction_returns"] assert qbi_returns.metadata["measure_mode"] == "indicator_sum" assert ( @@ -833,16 +837,13 @@ def test_source_column_exclusions_name_the_columns_their_guides_define() -> None """Each entry's reason names its own column and what its measure id means, as 22incddocguide.docx defines them: N18425/A18425 (state and local income taxes, Schedule A line 5a) stand in for the limited deduction N18460/A18460 - (line 5e), N85530/A85530 (additional Medicare tax) for the premium tax - credit N85770/A85770, and N19300/A19300 (line 8a mortgage interest) for the - line 10 interest deduction, which the file has no column for.""" + (line 5e) and N85530/A85530 (additional Medicare tax) for the premium tax + credit N85770/A85770.""" right_columns = { ("limited_state_local_taxes_returns", "N18425"): "N18460", ("limited_state_local_taxes_amount", "A18425"): "A18460", ("premium_tax_credit_returns", "N85530"): "N85770", ("premium_tax_credit_amount", "A85530"): "A85770", - ("interest_paid_deduction_returns", "N19300"): None, - ("interest_paid_deduction_amount", "A19300"): None, } assert set(US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS) == set(right_columns) for key, reason in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS.items(): @@ -850,12 +851,8 @@ def test_source_column_exclusions_name_the_columns_their_guides_define() -> None assert column in reason # Counts read N columns and amounts read A columns. assert column[0] == ("N" if measure_id.endswith("_returns") else "A") - right_column = right_columns[key] - if right_column is None: - assert "line 10" in reason - else: - assert right_column in reason - assert right_column[0] == column[0] + assert right_columns[key] in reason + assert right_columns[key][0] == column[0] def test_source_column_exclusion_refuses_a_district_fact_without_a_column() -> None: @@ -1033,12 +1030,12 @@ def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> No def test_pinned_feed_national_state_surface_restores_the_fences( pinned_feed_national_state_surface, ) -> None: - """The release surface on the pinned feed: PENDING_MINE_COMPILED compiled + """The release surface on the pinned feed: 31,784 compiled targets after Medicaid substitution (the full surface, at - PENDING_MINE_FULL), 6,102 national_state targets at registry - PENDING_MINE_NS, 32 CHIP rows none of them for an M-CHIP state, no + 6e1ac91262bc), 6,102 national_state targets at registry + 84504a7afd57, 32 CHIP rows none of them for an M-CHIP state, no other-income row and no tips return count. microcosm#1038 took it from - 33,250 compiled (full PENDING_1036_FULL) and PENDING_1036_NS: it drops + 33,250 compiled (full f09571684885) and 882f4764a248: it drops the congressional-district capital-gains US rows and the congressional-district SALT, premium-tax-credit and interest-paid rows read from the wrong IRS column, all on the full surface, and moves @@ -1058,12 +1055,12 @@ def test_pinned_feed_national_state_surface_restores_the_fences( from microcosm.calibrate import TargetRegistry registry, surface, _ = pinned_feed_national_state_surface - assert len(registry.specs) == PENDING_MINE_COMPILED_INT + assert len(registry.specs) == 31_784 builder = _load_repo_tool("build_us_fiscal_refresh_release") full, _ = builder._select_target_surface(registry.specs, "full") - assert TargetRegistry(full, country="us").version == "PENDING_MINE_FULL" + assert TargetRegistry(full, country="us").version == "6e1ac91262bc" assert len(surface.specs) == 6_102 - assert surface.version == "PENDING_MINE_NS" + assert surface.version == "84504a7afd57" chip = [ spec for spec in surface.specs @@ -1269,9 +1266,16 @@ def test_pinned_feed_no_congressional_district_family_sums_past_its_national_tar model quantity and filters. Before microcosm#1038 three of 38 compared families did: capital-gains returns at 2.41x, the SALT amount at 1.98x (read from the uncapped income-tax column A18425) and the capital-gains - amount at 1.198x. 35 of today's 52 families have a national comparator; - the rest (EITC by children, charitable, interest paid, QBI, taxable - interest, filer individuals) have none on this surface.""" + amount at 1.198x. + + The band is two-sided. Coverage (0.92-0.99 of Historic Table 2) and one + year less of aging keep every compared family above 0.85 of its national + target on this feed (lowest: the taxable Social Security amount, 0.855), + so a family below that has lost rows or reads another concept; each + compared family must also keep its state and district rows. 35 of the 52 + families have a national comparator on this surface; the rest (EITC by + children, charitable, interest paid, QBI deduction, taxable interest, + filer individuals) are compared under other keys or not at all.""" registry, _, _ = pinned_feed_national_state_surface builder = _load_repo_tool("build_us_fiscal_refresh_release") full, _ = builder._select_target_surface(registry.specs, "full") @@ -1289,11 +1293,16 @@ def test_pinned_feed_no_congressional_district_family_sums_past_its_national_tar national.setdefault(_soi_quantity(spec), []).append(spec.value) compared = {key: sums for key, sums in district_file.items() if key in national} assert (len(district_file), len(compared)) == (52, 35) + assert not [ + key[:2] + for key, sums in compared.items() + if not {"state", "congressional_district"} <= set(sums) + ] assert { - (key[:2], level): total / max(national[key]) + (key[:2], level): round(total / max(national[key]), 4) for key, sums in compared.items() for level, total in sums.items() - if total > max(national[key]) * (1 + 1e-9) + if not 0.85 * max(national[key]) <= total <= max(national[key]) * (1 + 1e-9) } == {} @@ -5049,10 +5058,11 @@ def test_congressional_district_capital_gains_rows_without_a_control_are_dropped assert _congressional_district_capital_gains_specs(registry) == {} -def test_capital_gains_share_row_without_its_family_total_refuses() -> None: +def test_capital_gains_share_row_without_its_family_total_is_dropped() -> None: """A congressional-district row with a Schedule D control but no US row of - its own has no defined share. The compile refuses instead of dropping it - silently or letting another family's total stand in (microcosm#1038).""" + its own family has no defined share: it is dropped, never divided by the + Historic Table 2 US row and never shipped as a line-7 level. The HT2 rows + beside it keep their own denominator (microcosm#1038).""" facts = [ *packaged_reference_facts(), *( @@ -5064,10 +5074,16 @@ def test_capital_gains_share_row_without_its_family_total_refuses() -> None: *_capital_gains_t14_facts(), ] - with pytest.raises( - fiscal_targets.MissingSoiCapitalGainsShareTotalError, match="hi_total" - ): - compile_us_fiscal_target_registry(facts, allow_unaged_dollar_targets=True) + registry = compile_us_fiscal_target_registry( + facts, allow_unaged_dollar_targets=True + ) + + assert _congressional_district_capital_gains_specs(registry) == {} + specs = {spec.metadata["ledger_source_record_id"]: spec for spec in registry.specs} + assert specs[_CG_HT2_CA_RETURNS].value == 10.0 + assert specs[_CG_HT2_CA_RETURNS].metadata["soi_share_total_source_record_id"] == ( + "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_returns" + ) def test_capital_gains_share_family_refuses_two_national_totals() -> None: @@ -5159,7 +5175,9 @@ def test_congressional_district_capital_gains_shares_conserve_the_control() -> N row is its share of the file's US row times the control; each state's districts sum to that state row; and the states, and the districts, sum to the control times the states' share of the US row, so never past the - control. The order is part of the draw, so the result is order-free.""" + control. Historic Table 2 rows drawn beside them divide by their own US + row, and every row names the denominator it used. The order is part of the + draw, so the result is order-free.""" pytest.importorskip("hypothesis") from hypothesis import given, settings from hypothesis import strategies as st @@ -5186,10 +5204,35 @@ def test_congressional_district_capital_gains_shares_conserve_the_control() -> N at_large_value=values, other_areas=st.integers(min_value=0, max_value=50_000_000), control=values, + ht2=st.tuples(values, values), order=st.randoms(use_true_random=False), ) - def check(districts, state_excess, at_large_value, other_areas, control, order): - facts = [] + def check( + districts, state_excess, at_large_value, other_areas, control, ht2, order + ): + ht2_state, ht2_rest = ht2 + ht2_us_id = "irs_soi.ty2022.historic_table_2.us.all.net_capital_gains_returns" + ht2_state_id = ( + "irs_soi.ty2022.historic_table_2.state_broad.hi.all." + "net_capital_gains_returns" + ) + facts = [ + _soi_capital_gains_fact( + 2022, + source_record_id=ht2_us_id, + measure_id=measure, + value=float(ht2_state + ht2_rest), + ), + _soi_capital_gains_fact( + 2022, + source_record_id=ht2_state_id, + measure_id=measure, + geography_level="state", + geography_id="0400000US15", + layout_record_set_id="irs_soi.ty2022.historic_table_2.state_broad.hi", + value=float(ht2_state), + ), + ] state_values = {} for (postal, fips), pair, excess in zip( two_district_states, districts, state_excess, strict=True @@ -5247,6 +5290,18 @@ def check(districts, state_excess, at_large_value, other_areas, control, order): [*packaged_reference_facts(), *facts], allow_unaged_dollar_targets=True ) + hawaii_ht2 = next( + spec + for spec in registry.specs + if spec.metadata["ledger_source_record_id"] == ht2_state_id + ) + assert math.isclose( + hawaii_ht2.value, + ht2_state * control / (ht2_state + ht2_rest), + rel_tol=1e-12, + ) + assert hawaii_ht2.metadata["soi_share_total_source_record_id"] == ht2_us_id + cd = _congressional_district_capital_gains_specs(registry).values() by_level: dict[str, list] = {} for spec in cd: @@ -5257,6 +5312,9 @@ def check(districts, state_excess, at_large_value, other_areas, control, order): "schedule_d_taxable_net_gain" ) assert spec.metadata["uprating_index_source_record_id"] == _CG_T14_RETURNS + assert spec.metadata["soi_share_total_source_record_id"] == ( + f"{_CG_CD_RECORD_SET}.us.{measure}" + ) assert set(by_level) == {"state", "congressional_district"} assert len(by_level["state"]) == len(state_values) assert len(by_level["congressional_district"]) == 2 * len(two_district_states) diff --git a/tools/us_acs_local_cd_surface.py b/tools/us_acs_local_cd_surface.py index edd37055fa..1348e0e4fc 100644 --- a/tools/us_acs_local_cd_surface.py +++ b/tools/us_acs_local_cd_surface.py @@ -50,8 +50,10 @@ STATE_CD_VINTAGE_RULE = "state_cd.historic_table_2_level_cd_file_shares.v1" #: District-file measures that read the wrong IRS column (microcosm#1038). -#: PR #1040 excludes the same columns in the compiler; until it lands this -#: register keeps them off the ``state_cd`` surface at both geographies. +#: The compiler's ``US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`` drops these +#: facts by (measure id, IRS column) before this surface sees them; this +#: register stays as a second fence on the ``state_cd`` surface, at both +#: geographies, and records the reason in the receipt. STATE_CD_DEFECTIVE_CD_FILE_MEASURES: Mapping[str, str] = { "limited_state_local_taxes_amount": ( "22incd.csv column A18425 is state and local income taxes (Schedule A " From 90ad595a648bc2c0ce379595a3d749193cb43365 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 12 Oct 2026 09:28:22 -0400 Subject: [PATCH 7/8] Pin the ACS-local SOI counts; record state_cd's capital-gains factors under the share rebase The ACS-local contract test pins totals at 120 SOI specs and full at 29,860 (identical to #1224). With the district file's capital-gains rows as shares of the Table 1.4 control, state_cd's capital-gains amount rebase factors move from 0.77-1.00 to 0.93-1.20 (median 0.98); its counts and factor-band verdicts are unchanged (measured on the pinned 51aa40f feed). Co-Authored-By: Claude Opus 5.5 --- docs/us-acs-local-soi-target-surface.md | 17 +++++++++++------ .../us/test_us_acs_local_release_tool.py | 16 ++++++++++++++++ 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/docs/us-acs-local-soi-target-surface.md b/docs/us-acs-local-soi-target-surface.md index 80fcd27bb6..3f5b7fdcf0 100644 --- a/docs/us-acs-local-soi-target-surface.md +++ b/docs/us-acs-local-soi-target-surface.md @@ -217,8 +217,11 @@ district rows, are mostly 1.00 to 1.10. chained SOI and CBO series (about 1.120). The smallest amount factor, 1.03, is that aging ratio alone. Correcting the district file's stamp (#1030) would remove the aging part only. -- **Two factors reflect known level differences:** taxable interest (2.35 to - 2.70, the Table 4.3 rebase) and capital-gains amounts (0.77 to 1.00). +- **Taxable interest reflects a known level difference** (2.35 to 2.70, the + Table 4.3 rebase). Capital-gains factors sit near 1: since microcosm#1038 + (#1040) both files' capital-gains rows are shares of the same Table 1.4 + control (amounts 0.93 to 1.20, median 0.98; before, 0.77 to 1.00; returns + 0.98 to 1.00). - **The largest single-state factors** were on qualified and ordinary dividends (Hawaii, 2.68 and 2.17), tax-exempt interest (Utah, 1.43) and rental income (New York, 0.72), where the two tables disagree about one @@ -320,11 +323,13 @@ bridged state row records its sibling and bridge factor. | Bridged state rows whose sibling ratio is out of band | 4 | The factor band (above) | Since microcosm#1038 the compiler excludes the same SALT and PTC columns -(`US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`). PR #1040 (awaiting a ruling) -rescales the district file's capital-gains rows by one national factor. -`state_cd` already takes district capital gains as within-state shares of +(`US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`, #1224) and rescales the +district file's capital-gains rows by one national factor per measure +(#1040). `state_cd` takes district capital gains as within-state shares of Historic Table 2, which a national factor does not move, so their level -changes only when Historic Table 2's own capital-gains rows do (#1036). +changes only when Historic Table 2's own capital-gains rows do (#1036). Its +capital-gains rebase factors move by the reciprocal of that factor, and no +factor-band verdict or contract count changes. North Carolina's district rows are bound. Before #1043 the packaged 117th→119th crosswalk was built from NC's 2016 plan rather than the 2019 plan diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py b/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py index 4024b3a260..c948e928ab 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_acs_local_release_tool.py @@ -956,6 +956,22 @@ def test_pinned_feed_soi_surfaces_match_their_contracts() -> None: } assert {spec.metadata["ledger_geography_level"] for spec in soi} == {"state"} assert {spec.name for spec in state_specs} <= set(full_by_name) + # The compiler drops the district file's SALT and premium-tax-credit facts + # read from the wrong IRS column (microcosm#1038), so `totals` keeps only + # the Historic Table 2 premium-tax-credit rows (51 states and 9 at-large + # district copies, returns and amounts) and `full` 29,860 SOI specs. + totals_soi = [spec for spec in totals_registry.specs if spec.family == "irs_soi"] + assert len(totals_soi) == 120 + assert {spec.metadata["source_measure_id"] for spec in totals_soi} == { + "premium_tax_credit_returns", + "premium_tax_credit_amount", + } + assert not [ + spec.name + for spec in totals_soi + if ".congressional_district_" in spec.metadata["ledger_layout_record_set_id"] + ] + assert sum(spec.family == "irs_soi" for spec in full_registry.specs) == 29_860 assert not [ spec From d3a62535043da60ed63f2663b3e19d3cb3c70666 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Mon, 12 Oct 2026 11:41:12 -0400 Subject: [PATCH 8/8] Run the column exclusion before the district crosswalk; address the #1040 re-review nits From the #1224 review (P2/P3), carried here byte-identical: the congressional-district crosswalk sums source districts under the first contributor's layout, so US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS and the missing-column refusal now run on each source fact before translation, in the compile and the receipt (which therefore lists 4 x 480 source ids); tests cover mixed-column merges in both orders, every missing-column order, and a corrected A85770 PTC amount. From the #1040 re-review (APPROVE WITH NITS): the doc says the return counts agree among the tightest pairs (ninth of 45 by median gap) and that the CD-over-HT2 tilt holds in aggregate (Montana's amount is 0.73x); the congressional-district parity-manifest note names the capital-gains shares, the retired US row and the column exclusions. Co-Authored-By: Claude Opus 5.5 --- docs/us-soi-capital-gains-concepts.md | 11 +- .../build/us/target_parity_manifest.json | 2 +- .../build/us_runtime/fiscal_targets.py | 33 ++++- .../engine_free/us/test_us_fiscal_targets.py | 126 +++++++++++++++++- tools/build_us_target_parity_manifest.py | 5 +- 5 files changed, 157 insertions(+), 20 deletions(-) diff --git a/docs/us-soi-capital-gains-concepts.md b/docs/us-soi-capital-gains-concepts.md index bea1ea2e5d..b945d54081 100644 --- a/docs/us-soi-capital-gains-concepts.md +++ b/docs/us-soi-capital-gains-concepts.md @@ -158,8 +158,8 @@ takes (a): district share left over after the district's AGI distribution has a log standard deviation of 0.227 for counts and 0.223 for amounts. The AGI-composition error of the share proxy is 0.024 and 0.009. -- **The rebased return counts agree with the HT2 state rows as well as any - pair on `full`; the amounts are the loosest pair.** Both files' state rows +- **The rebased return counts agree with the HT2 state rows among the + tightest of the pairs on `full`; the amounts are the loosest pair.** Both files' state rows stay on `full`. Each of the 45 measures they share pairs two state targets for every state. After dividing out each pair's common offset (coverage and aging, which differ by measure), the capital-gains return counts differ by @@ -175,9 +175,10 @@ file has no rows for other areas (territories, APO/FPO and foreign addresses), so its US row equals the sum of its 51 states, and its rebased states and districts each sum to the control itself. HT2's US row includes other areas, so its rebased states sum to 99.2% of the control for returns -and 98.0% for amounts. Every congressional-district capital-gains target -therefore sits above the matching HT2 share, by 0.8% for returns and 2.1% -for amounts in aggregate. This is accepted, not corrected. The model has +and 98.0% for amounts. In aggregate, then, the congressional-district +capital-gains targets sit above the matching HT2 shares, by 0.8% for returns +and 2.1% for amounts; state by state they scatter around that (Montana's +amount is 0.73x HT2's, below it, because of the suppression above). This is accepted, not corrected. The model has households in the 50 states and DC only, and the national Table 1.4 row on the same surface already asks them for the whole control; the other-area part of HT2's split is a share no model household can carry. The diff --git a/packages/microcosm-build/src/microcosm/build/us/target_parity_manifest.json b/packages/microcosm-build/src/microcosm/build/us/target_parity_manifest.json index 6de4b30b49..7b4654ff2e 100644 --- a/packages/microcosm-build/src/microcosm/build/us/target_parity_manifest.json +++ b/packages/microcosm-build/src/microcosm/build/us/target_parity_manifest.json @@ -523,7 +523,7 @@ }, "irs_soi.congressional_district_2022": { "status": "compiled", - "note": "One US target surface: national, state, and congressional-district facts compile for every calibrated artifact. Geography is a constraint; artifact record count changes only L0, never target membership. There is no local-versus-national surface. Congressional-district rows inform shape while national and state actuals retain control ownership." + "note": "One US target surface: national, state, and congressional-district facts compile for every calibrated artifact. Geography is a constraint; artifact record count changes only L0, never target membership. There is no local-versus-national surface. Congressional-district rows inform shape while national and state actuals retain control ownership. Its capital-gains rows ship as shares of its own US row scaled to the national Table 1.4 control, and that US row retires (microcosm#1038); facts it reads from the wrong IRS column are excluded (US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS)." }, "irs_soi.filing_season_week47": { "status": "compiled" diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index c3f50779f8..ccb186955b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -1103,7 +1103,7 @@ def compile_us_fiscal_target_registry( auditable waiver instead of raising; every waived target carries ``period_contract_waiver`` metadata in diagnostics. """ - materialized_facts = tuple(facts) + materialized_facts = _without_source_column_concept_exclusions(tuple(facts)) if congressional_district_vintage_crosswalk is not None: materialized_facts = translate_congressional_district_facts_to_current_vintage( materialized_facts, @@ -1186,7 +1186,8 @@ def us_fiscal_target_exclusion_receipt( ``_M_CHIP_STATE_FIPS`` state, at every period (dropped). - ``source_column_concept_exclusion``: SOI facts read from an IRS column that ``US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS`` lists for their - measure id, at every vintage (dropped). + measure id, at every vintage, as the feed carries them, before the + congressional-district crosswalk (dropped). - ``allowlisted_vintage_bypass``: facts latest-vintage selection activates at ``target_period`` that are another vintage of an excluded cell and carry a reviewed ``US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES`` entry @@ -1197,7 +1198,13 @@ def us_fiscal_target_exclusion_receipt( it raises on a vintage bypass nobody reviewed. """ - materialized_facts = tuple(facts) + raw_facts = tuple(facts) + source_column = { + _source_record_id(fact) + for fact in raw_facts + if _is_source_column_concept_exclusion(fact) and _source_record_id(fact) + } + materialized_facts = _without_source_column_concept_exclusions(raw_facts) if congressional_district_vintage_crosswalk is not None: materialized_facts = translate_congressional_district_facts_to_current_vintage( materialized_facts, @@ -1205,7 +1212,6 @@ def us_fiscal_target_exclusion_receipt( ) reviewed: set[str] = set() all_vintage: set[str] = set() - source_column: set[str] = set() m_chip: set[str] = set() for fact in materialized_facts: source_record_id = _source_record_id(fact) @@ -1215,8 +1221,6 @@ def us_fiscal_target_exclusion_receipt( all_vintage.add(source_record_id) elif source_record_id in US_FISCAL_TARGET_SUPPORT_EXCLUSIONS: reviewed.add(source_record_id) - elif _is_source_column_concept_exclusion(fact): - source_column.add(source_record_id) elif _source_name(fact) == "cms_medicaid": reference = _reference_from_ledger_fact(fact, target_period=target_period) if reference is not None and _is_m_chip_state_chip_enrollment( @@ -3206,6 +3210,23 @@ def _is_source_column_concept_exclusion(fact: object) -> bool: return (measure_id, column) in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS +def _without_source_column_concept_exclusions( + facts: tuple[object, ...], +) -> tuple[object, ...]: + """Drop facts read from a listed column, before the district crosswalk. + + The crosswalk sums several source districts into one current district and + keeps one contributor's layout, so the column check must see each source + fact: afterwards an excluded contribution could inherit a corrected + column, and a fact with no column could hide behind one that has it. + Raises ``MissingSoiSourceColumnError`` for a listed district fact with no + column. + """ + return tuple( + fact for fact in facts if not _is_source_column_concept_exclusion(fact) + ) + + def _is_all_vintage_support_exclusion(source_record_id: str) -> bool: return _period_free_source_record_id(source_record_id) in ( _period_free_source_record_ids(US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS) diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index 215b0f5773..21cbceadfc 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -814,8 +814,17 @@ def test_source_column_exclusion_drops_only_the_mislabeled_column() -> None: geography_level="state", geography_id="0400000US15", ) + # The amount twin too: the removed record-set check dropped every + # district PTC amount; the register keeps A85770, the premium tax credit. + ptc_amount = _cd_column_fact( + "premium_tax_credit_amount", + "A85770", + groupby_value_id="hi_total", + geography_level="state", + geography_id="0400000US15", + ) mislabeled = [*mislabeled, medicare] - facts = [*packaged_reference_facts(), *mislabeled, corrected, ptc] + facts = [*packaged_reference_facts(), *mislabeled, corrected, ptc, ptc_amount] registry = compile_us_fiscal_target_registry( facts, allow_unaged_dollar_targets=True @@ -825,6 +834,7 @@ def test_source_column_exclusion_drops_only_the_mislabeled_column() -> None: assert not compiled & {fact["lineage"]["source_record_id"] for fact in mislabeled} assert corrected["lineage"]["source_record_id"] in compiled assert ptc["lineage"]["source_record_id"] in compiled + assert ptc_amount["lineage"]["source_record_id"] in compiled rule = us_fiscal_target_exclusion_receipt(facts)["rules"][ "source_column_concept_exclusion" ] @@ -891,6 +901,107 @@ def test_source_column_exclusion_refuses_a_district_fact_without_a_column() -> N } +#: Two Hawaii source districts merged into one current district, so the +#: crosswalk sums them under one contributor's layout. +_HI_MERGE_CROSSWALK = [ + { + "source_geography_id": "5001700US1501", + "target_geography_id": "5001900US1501", + "weight": 1.0, + }, + { + "source_geography_id": "5001700US1502", + "target_geography_id": "5001900US1501", + "weight": 1.0, + }, +] + + +def _hi_salt_district_fact(number: int, column: str | None, value: float): + if column is None: + return _soi_congressional_district_fact( + "limited_state_local_taxes_amount", + value, + groupby_value_id=f"hi_0{number}", + geography_id=f"5001700US150{number}", + ) + return _cd_column_fact( + "limited_state_local_taxes_amount", + column, + value=value, + groupby_value_id=f"hi_0{number}", + geography_level="congressional_district", + geography_id=f"5001700US150{number}", + ) + + +@pytest.mark.parametrize( + "columns", + [("A18460", "A18425"), ("A18425", "A18460")], + ids=["kept-first", "excluded-first"], +) +def test_source_column_exclusion_runs_before_the_crosswalk_merges(columns) -> None: + """The crosswalk sums source districts into one current district under one + contributor's layout, so the column check runs on each source fact first. + In either order the A18425 contribution never reaches the merged target, + which carries only the A18460 district's value, and the receipt names the + excluded source fact (#1224 review).""" + facts = [ + _hi_salt_district_fact(number, column, value) + for number, column, value in zip((1, 2), columns, (100.0, 200.0), strict=True) + ] + feed = [*packaged_reference_facts(), *facts] + + registry = compile_us_fiscal_target_registry( + feed, + congressional_district_vintage_crosswalk=_HI_MERGE_CROSSWALK, + allow_unaged_dollar_targets=True, + ) + receipt = us_fiscal_target_exclusion_receipt( + feed, congressional_district_vintage_crosswalk=_HI_MERGE_CROSSWALK + ) + + merged = [ + spec.value + for spec in registry.specs + if spec.metadata.get("source_measure_id") == "limited_state_local_taxes_amount" + and spec.metadata.get("ledger_geography_level") == "congressional_district" + ] + assert merged == [100.0 if columns[0] == "A18460" else 200.0] + excluded = next(f for f in facts if f["layout"]["source_column_id"] == "A18425") + assert receipt["rules"]["source_column_concept_exclusion"]["source_record_ids"] == [ + excluded["lineage"]["source_record_id"] + ] + + +@pytest.mark.parametrize( + "columns", + [("A18460", None), (None, "A18460"), ("A18425", None), (None, "A18425")], +) +def test_missing_column_refuses_before_the_crosswalk_merges(columns) -> None: + """A district fact of a listed measure with no column refuses in any merge + order: the crosswalk cannot hide it behind a contributor that has one, nor + behind an excluded one (#1224 review).""" + feed = [ + *packaged_reference_facts(), + *( + _hi_salt_district_fact(number, column, 100.0) + for number, column in zip((1, 2), columns, strict=True) + ), + ] + + with pytest.raises(fiscal_targets.MissingSoiSourceColumnError): + compile_us_fiscal_target_registry( + feed, + congressional_district_vintage_crosswalk=_HI_MERGE_CROSSWALK, + allow_unaged_dollar_targets=True, + ) + with pytest.raises(fiscal_targets.MissingSoiSourceColumnError): + us_fiscal_target_exclusion_receipt( + feed, congressional_district_vintage_crosswalk=_HI_MERGE_CROSSWALK + ) + + def test_exclusion_receipt_is_complete_for_any_subfeed(monkeypatch) -> None: """Invariant (receipt completeness): each rule lists exactly the facts it drops or allows. For any subset of the five-rule synthetic feed, each rule @@ -988,10 +1099,11 @@ def check(years: list[int], reviewed: bool) -> None: def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> None: """On the pinned feed the receipt names the 40 M-CHIP CHIP ids (20 states x 2 CMS months), the 16 other-income ids (ty2020-ty2023), the ty2020 and - ty2023 tips return counts, and the 1,952 congressional-district SALT and - premium-tax-credit facts read from the wrong IRS column (488 per measure - after the district crosswalk: the US row, 51 states, 436 districts), and - allows no vintage bypass (decision d179, microcosm#1038).""" + ty2023 tips return counts, and the 1,920 congressional-district SALT and + premium-tax-credit facts read from the wrong IRS column, as the feed + carries them before the district crosswalk (480 per measure: the US row, + 51 states, 428 source districts), and allows no vintage bypass (decision + d179, microcosm#1038).""" _, _, receipt = pinned_feed_national_state_surface rules = receipt["rules"] m_chip_ids = rules["m_chip_state_chip_enrollment"]["source_record_ids"] @@ -1023,8 +1135,8 @@ def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> No f".{measure}" in source_record_id for source_record_id in column_ids ) for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS - } == {measure: 488 for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS} - assert len(column_ids) == 4 * 488 + } == {measure: 480 for measure, _ in US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS} + assert len(column_ids) == 4 * 480 def test_pinned_feed_national_state_surface_restores_the_fences( diff --git a/tools/build_us_target_parity_manifest.py b/tools/build_us_target_parity_manifest.py index 0b8548ebe3..ade3f8b4aa 100644 --- a/tools/build_us_target_parity_manifest.py +++ b/tools/build_us_target_parity_manifest.py @@ -537,7 +537,10 @@ def _fence(origin: str, purpose: str, verdict_basis: str) -> dict[str, str]: "artifact record count changes only L0, never target membership. " "There is no local-versus-national surface. " "Congressional-district rows inform shape while national and state " - "actuals retain control ownership." + "actuals retain control ownership. Its capital-gains rows ship as shares " + "of its own US row scaled to the national Table 1.4 control, and that US " + "row retires (microcosm#1038); facts it reads from the wrong IRS column " + "are excluded (US_FISCAL_TARGET_SOURCE_COLUMN_EXCLUSIONS)." ), }