diff --git a/changelog.d/us-w2-items-tax-year-2020.fixed.md b/changelog.d/us-w2-items-tax-year-2020.fixed.md new file mode 100644 index 0000000000..9c02bbb981 --- /dev/null +++ b/changelog.d/us-w2-items-tax-year-2020.fixed.md @@ -0,0 +1 @@ +Stamp the IRS SOI Form W-2 item facts with the tax year IRS published them for. The pinned US Chronicle feed carried five cells of Table 4.B for Tax Year 2020 (`20in04w2all.xlsx`) with period 2023 and vintage `tax_year_2023`: the Box 7 tips amount, return count and taxpayer count, the Box 12 D 401(k) elective deferrals and the Box 12 AA designated Roth contributions. Latest-vintage selection took the ty2023 tips amount, so the 2024 target aged one year on the CBO wages ratio ($28,280,884,269) where the TY2020 cell needs the SOI wage chain from 2020 ($34,287,530,779, +21.2%). The feed is re-pinned to Chronicle `f98acf4` (`c5e5bf8` plus the W-2 package's literal TY2020 labels, tag `microcosm-us-feed-w2-ty2020-v1`): 39,155 rows, `a81cbcc5…`, scope 585 pairs. No other target changes name, value or metadata; the registry stays at 32,842 targets and `national_state` at 5,694 (`d315c75804ef` to `61a081ae55bb`). The #451 tips return-count exclusion is keyed to the ty2020 id and still holds at every vintage (decision d179). The compile and the exclusion receipt now refuse a W-2 item fact whose period or `tax_year` vintage is not the one tax year its source table's title names. `tools/route_a/route_a.sh` restates the new feed digest, and a test fails if it falls behind the pin. Tax Year 2020 is the latest Table 4.B IRS has published, so there is no TY2023 table to add. diff --git a/docs/us-acs-local-soi-target-surface.md b/docs/us-acs-local-soi-target-surface.md index dbca25b164..375927b15a 100644 --- a/docs/us-acs-local-soi-target-surface.md +++ b/docs/us-acs-local-soi-target-surface.md @@ -54,10 +54,14 @@ surface's specs. ## The surfaces on the pinned feed Measured on 2026-09-22 by calling `state_admin_specs(feed, ["snap", -"medicaid", "soi"], soi_mode=...)` on the pinned -`consumer_facts_us_c5e5bf8.jsonl` (sha256 `b8543739…`, the `facts_sha256` in -`packages/microcosm-build/src/microcosm/build/us/chronicle_feed.json`). The -Chronicle `b571381` consumer artifact gives the same counts. +"medicaid", "soi"], soi_mode=...)` on +`consumer_facts_us_c5e5bf8.jsonl` (sha256 `b8543739…`, the feed +`packages/microcosm-build/src/microcosm/build/us/chronicle_feed.json` pinned +then). The Chronicle `b571381` consumer artifact gives the same counts, and so +does the feed pinned since 10 October 2026 (`consumer_facts_us_f98acf4.jsonl`, +which differs only in its Form W-2 item rows): +`test_pinned_feed_soi_surfaces_match_their_contracts` and +`test_pinned_feed_state_cd_surface_matches_its_contract` pass on it. | Family | `state` | `totals` | `full` | |---|---:|---:|---:| diff --git a/docs/us-chronicle-feed-repin.md b/docs/us-chronicle-feed-repin.md index 2c68ed03e3..5087d01885 100644 --- a/docs/us-chronicle-feed-repin.md +++ b/docs/us-chronicle-feed-repin.md @@ -13,6 +13,11 @@ any other feed, and `test_us_chronicle_feed.py` fails if the parity resources, the generator and the pin disagree, or if the scope file changes without a new pin. +The pin has moved twice. September 2026 replaced an unlabelled feed with the +first labelled export (the next section onward). October 2026 moved only the +Form W-2 item rows, which the September feed stamped with the wrong tax year +("The W-2 item tables carry Tax Year 2020" below). + ## Why the feed moved `_validate_chronicle_hierarchy_labels` @@ -28,25 +33,32 @@ the first labelled US export. | Field | Value | |---|---| -| Chronicle commit | `c5e5bf8aa84960c1a200ee47303b19c953092d0f` | -| Feed file | `consumer_facts_us_c5e5bf8.jsonl`, 39,158 rows, 164,603,204 bytes | -| `facts_sha256` | `b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801` | +| Chronicle commit | `f98acf4edcc9488343446fda46db77914de8611f` (tag `microcosm-us-feed-w2-ty2020-v1`): `c5e5bf8` plus one package file | +| Feed file | `consumer_facts_us_f98acf4.jsonl`, 39,155 rows, 164,591,409 bytes | +| `facts_sha256` | `a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c` | | Consumer fact schema | `chronicle.consumer_fact.v3`, schema file sha256 `bdb51e2a…` (unchanged from the UK pin) | -| Scope | 586 (record set, period) pairs; 62 package runs over build years 2020 to 2029 | +| Scope | 585 (record set, period) pairs; 61 package runs over build years 2020 to 2029 | | Consumer artifact | none: refused at this commit, see below | +The September pin was Chronicle `c5e5bf8aa84960c1a200ee47303b19c953092d0f`: +`consumer_facts_us_c5e5bf8.jsonl`, 39,158 rows, 164,603,204 bytes, +`facts_sha256` `b8543739…f4839801`, 586 pairs from 62 package runs. The +compile now refuses that feed (`_check_w2_item_fact_tax_years`). + The feed is too large for the repository. Its home on the build machine is -`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_c5e5bf8.jsonl`, +`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_f98acf4.jsonl`, beside the previous pins; `tools/build_us_target_parity_manifest.py` reads it there by default. A holder of the Chronicle commit regenerates it byte for byte with the commands below. ## Scope rule -The feed keeps exactly the 586 (record set, period) pairs of the previous -pin, so the family surface the release parity gate reviews does not move in -this change: 32 compiled families and 52 reviewed exclusions before and after, -the same 81 feed families. Widening the feed to Chronicle's newer US packages +The September feed kept exactly the 586 (record set, period) pairs of the +previous pin, so the family surface the release parity gate reviews did not +move: 32 compiled families and 52 reviewed exclusions before and after, +the same 81 feed families. The October change swaps the three W-2 item pairs +stamped ty2023 for their TY2020 record sets, one of which the scope already +had, leaving 585 pairs and the same families. Widening the feed to Chronicle's newer US packages is a separate, deliberate change with its own compile-or-fence review per family. Scoping by record set alone is wrong: it pulls extra Medicaid months into the snapshot families. @@ -72,9 +84,11 @@ minutes where a targeted package run takes one or two seconds. uv run python tools/build_us_chronicle_feed.py --chronicle-root ~/PolicyEngine/chronicle --out /tmp/us-feed --replace --skip-artifact ``` -The checkout must be clean at the scope's commit. The tool runs one +The checkout must be clean at the scope's commit +(`git checkout microcosm-us-feed-w2-ty2020-v1`; the commit is not on +Chronicle main). The tool runs one `chronicle build-bundle --year Y --source …` per build year (ten -runs, 62 packages), reads each package's own `consumer_facts.jsonl`, keeps a +runs, 61 packages), reads each package's own `consumer_facts.jsonl`, keeps a row only from the run its pair is scoped to, refuses a pair with no row or two rows with one `aggregate_fact_key` and different bytes, sorts by `aggregate_fact_key`, and writes `consumer_facts.jsonl` plus `receipt.json` @@ -181,11 +195,131 @@ and target id, and the register entry carries a reviewed `state_name` (`_geography_fallback_label` is UK-only, so the label cannot be derived from the FIPS code). +## The W-2 item tables carry Tax Year 2020 + +10 October 2026. The September feed carried five facts with period 2023 and +vintage `tax_year_2023` whose `source.source_file` is `20in04w2all.xlsx` and +whose `source.source_table` is "Table 4.B. Summary of Items for Taxpayers with +Form W-2, by Return and Earner Type, Tax Year 2020": + +| `source_record_id` (as stamped) | Value | Workbook cell | +|---|---:|---| +| `irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.amount` | 26,786,522,000 | D13 | +| `…form_w2_social_security_tips.box_7_social_security_tips.return_count` | 6,038,613 | B13 | +| `…form_w2_social_security_tips.box_7_social_security_tips.taxpayer_count` | 6,105,713 | C13 | +| `irs_soi.ty2023.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount` | 277,859,181,000 | D22 | +| `irs_soi.ty2023.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount` | 32,302,509,000 | D41 | + +The three tips rows repeated the feed's correctly stamped `irs_soi.ty2020` +tips facts cell for cell. + +**The source.** Checked on 10 October 2026: + +- IRS's Form W-2 statistics page + (`irs.gov/statistics/soi-tax-stats-individual-information-return-form-w2-statistics`, + last reviewed 27 January 2026) links Table 4 for 2020 and 2019 only. + `21in04w2all.xlsx` to `24in04w2all.xlsx` return 404. There is no Table 4.B + for Tax Year 2021, 2022 or 2023. The prior-releases page carries tax years + 2008 to 2018 in one workbook (`18inallw2.xls`), so 2020 is the latest year + IRS has published. +- A fresh download of `20in04w2all.xlsx` has the sha256 Chronicle pins + (`1178d776…cfc442f`). Cell A1 of sheet "Table 4.B" is the title above, and + rows 13, 22 and 41 hold the five values (money amounts in thousands). + +So every one of the five values is a Tax Year 2020 value. + +**Why the feed stamped them 2023.** At Chronicle `c5e5bf8` the +`soi-w2-statistics-2020` package sets `artifact_year: 2020`, which makes every +build read that one workbook, but rendered its period, record ids, vintage and +legal vintage from `--year`. The scope built it at `--year 2023` for three +record sets, as the July feed's pairs said to. chronicle#292 (merged 28 +September) made the labels literal 2020 on Chronicle main. + +**The pin.** Chronicle `f98acf4` is `c5e5bf8` plus only that package file as +#292 left it (branch `feed/us-c5e5bf8-w2-2020`, tag +`microcosm-us-feed-w2-ty2020-v1`). The scope drops the three ty2023 W-2 pairs +and adds the TY2020 401(k) and Roth record sets; it already had the TY2020 +tips record set. Against the September feed +(`experiments/us-w2-table-year-repin/`): + +- 39,153 rows are byte-identical, the three `irs_soi.ty2020` tips rows among + them; +- the five rows above are gone, and no row carries their ids (a deliberate + supersession: the ids named a tax year IRS has not published); +- two rows are new, the 401(k) and Roth amounts under their `irs_soi.ty2020` + ids, each with the value and the source cells of its ty2023 twin. The + three ty2023 tips rows need no replacement: their TY2020 twins were + already in the feed. + +`tools/build_us_chronicle_feed.py` built the feed from the new scope, and an +independent splice (the September feed without the W-2 package's rows, plus +that package's rows at `f98acf4`, sorted by `aggregate_fact_key`) gives the +same bytes. + +A pin on Chronicle main is a larger change. At main (`3dce46a6`) seven more of +the September scope's pairs are no longer declared: the congressional-district +file, the two IRA tables and the four `state_2022` US rows, which #292 gave +literal 2022 labels. Moving those changes the year every district dollar +target ages from; microcosm#1030 measures that and lists the decisions it +needs. + +**What moved.** Compiled as the release does (target period 2024, district +crosswalk, aging on, Medicaid substitutions), before on main `7639ef8b0` with +the September feed and after on this change: + +| | Before | After | +|---|---|---| +| Compiled targets | 32,842 | 32,842 | +| `national_state` targets | 5,694 | 5,694 | +| `national_state` registry | `d315c75804ef` | `61a081ae55bb` | +| W-2 Box 7 tips amount target | $28,280,884,269 | $34,287,530,779 | +| Its name | `irs_soi.ty2023.form_w2_social_security_tips.…amount` | `irs_soi.ty2020.form_w2_social_security_tips.…amount` | +| Its aging factor | 1.05578784246075 (CBO wages 2024 / 2023) | 1.28002921689246 (SOI Table 1.4 wages 2023 / 2020, then CBO wages 2024 / 2023) | + +The tips target rises $6,006,646,510 (21.2%), the SOI wage growth from 2020 to +2023 that the ty2023 stamp skipped. It is the only target that changes: the +other 32,841 keep their name, value and every metadata field. Latest-vintage +selection had picked the ty2023 row over its TY2020 twin, so the target aged +one year where `target_aging.py` chains the wages series from 2020. The +September re-pin introduced this: it added the ty2023 tips amount and return +count (the row count of `irs_soi.form_w2_social_security_tips` went 4 to 6, +above; it is now 3). The July feed already carried the ty2023 401(k), Roth and +tips taxpayer-count rows, none of which compiles to a target. The Route A release `populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z` +recorded the $28,280,884,269 target +(`tests/fixtures/us_route_a_target_loss_weights.json`). + +The 401(k) and Roth amounts stay reviewed exclusions (`input_side`), so they +move no target. Anything that anchors on them later (microcosm#1191 names the +401(k) amount) is anchoring on a Tax Year 2020 level. + +**The exclusion register.** The #451 tips return-count entry in +`US_FISCAL_TARGET_SUPPORT_EXCLUSIONS` and +`US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS` is keyed to the ty2020 id. +Decision d179 is unchanged: the cell is excluded at every vintage. The receipt +lists one tips id where it listed two. + +**The check.** `_check_w2_item_fact_tax_years` runs at the start of +`compile_us_fiscal_target_registry` and of +`us_fiscal_target_exclusion_receipt`. For a fact of the three W-2 item +families (`irs_soi.form_w2_item` layout) it requires the source table's title +to name exactly one tax year, and that year to be the year target aging reads +from the fact's period and the year in its `tax_year_` vintage. It +refuses the September feed and names the five rows. +`test_us_w2_item_tax_year.py` holds the tests. Chronicle's +`tests/test_chronicle_source_table_year.py` checks the same thing at the +source for every pinned package whose title names one tax year. + +The parity resources were regenerated: 32 compiled families, 52 reviewed +exclusions and 81 feed families, as before. `tools/route_a/route_a.sh` restates +the new `FEED_SHA`; `test_us_chronicle_feed.py` now fails if it falls behind +the pin. + ## The consumer artifact is refused at this commit `chronicle build-consumer-artifact` validates every row against `consumer_fact.v3`, whose `concept_alignment` object has required `authority` -since Chronicle `cafc583`. `build-bundle` at `c5e5bf8` emits 994 rows whose +since Chronicle `cafc583`. `build-bundle` at `c5e5bf8`, and at `f98acf4`, which +differs from it in one W-2 package file, emits 994 rows whose `concept_alignment` has `relation: source_label` and no `authority`, in three packages: `cms-medicaid-chip-monthly-enrollment-dataset` (515), `census-b01001-female-age-2023` (468) and `jct-tax-expenditures-2024` (11). diff --git a/docs/us-fact-to-target.md b/docs/us-fact-to-target.md index 5a4f56a557..38d9d149eb 100644 --- a/docs/us-fact-to-target.md +++ b/docs/us-fact-to-target.md @@ -74,6 +74,14 @@ the release PROGRESS notes). Re-pin the sha everywhere it is enforced: the launch scripts' `FACTS_SHA` guards, `--ledger-facts-sha256`, and the PROGRESS doc. The builder refuses to start on a mismatch. +An id the package no longer emits leaves the feed without a replacement. The +October 2026 re-pin dropped five `irs_soi.ty2023` Form W-2 item ids that way: +they named a tax year IRS has not published for that table +(`docs/us-chronicle-feed-repin.md`). Check a new fact's period against the +year in its source table's title before it ships. For the W-2 item tables the +compile enforces it: `_check_w2_item_fact_tax_years` refuses a fact whose +period or `tax_year` vintage is not the one tax year its title names. + ## 2. Run the support oracle BEFORE wiring anything A target with no model support cannot bind; a target with thin support binds diff --git a/experiments/us-w2-table-year-repin/README.md b/experiments/us-w2-table-year-repin/README.md new file mode 100644 index 0000000000..0859f48354 --- /dev/null +++ b/experiments/us-w2-table-year-repin/README.md @@ -0,0 +1,58 @@ +# W-2 item tables: re-pin the US feed to the tax year IRS published + +10 October 2026. The pinned US Chronicle feed stamped five cells of IRS SOI +Table 4.B for Tax Year 2020 as ty2023. `docs/us-chronicle-feed-repin.md` +("The W-2 item tables carry Tax Year 2020") has the account; this folder holds +the scripts and their outputs. + +| File | What it does | +|---|---| +| `relabel_scope.py` | Rewrites `us/chronicle_feed_scope.json`: the three ty2023 W-2 pairs out, the TY2020 401(k) and Roth record sets in, the new Chronicle commit. Idempotent. | +| `splice_feed.py` | Predicts the new feed from the old one and one run of the W-2 package (`docs/us-fact-to-target.md`, step 1). | +| `apply_pin.py` | Writes the rebuilt feed's digests into `us/chronicle_feed.json` and the parity generator's feed name, from the builder's receipt. | +| `compare_feeds.py` | Row-by-row difference between two feeds. Output: `feed_compare.json`. | +| `measure_surface.py` | Compiles a feed as the release does and lists every target. | +| `compare_surfaces.py` | Target-by-target difference between two `measure_surface.py` reports. Output: `compare.json`. | +| `receipt.json` | The receipt `tools/build_us_chronicle_feed.py` wrote for the pinned feed. | + +## Order of work + +```bash +python experiments/us-w2-table-year-repin/relabel_scope.py packages/microcosm-build/src/microcosm/build/us/chronicle_feed_scope.json f98acf4edcc9488343446fda46db77914de8611f +``` + +```bash +uv run python tools/build_us_chronicle_feed.py --chronicle-root --out --replace --skip-artifact +``` + +```bash +python experiments/us-w2-table-year-repin/apply_pin.py . f98acf4edcc9488343446fda46db77914de8611f consumer_facts_us_f98acf4.jsonl +``` + +Then copy `/consumer_facts.jsonl` to +`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_f98acf4.jsonl` and run +`uv run python tools/build_us_target_parity_manifest.py`. + +## Results + +`feed_compare.json`: of the old feed's 39,158 rows, 39,153 are byte-identical +in the new feed; the five ty2023 W-2 ids are gone; two ids are new (the +401(k) and Roth amounts at ty2020), each with the value and source cells of +its ty2023 twin. The new feed has 39,155 rows. + +`compare.json`: "before" is main `7639ef8b0` compiling the old feed, "after" +is this change compiling the new one. Both compile 32,842 targets, 5,694 of +them on `national_state`. One target differs: the W-2 Box 7 tips amount is +`irs_soi.ty2023.…amount` at $28,280,884,269.41 before and +`irs_soi.ty2020.…amount` at $34,287,530,778.93 after. The other 32,841 match +on value and on a digest of all their metadata. + +"Before" has to run on main's code, because this change refuses the old +feed. It ran from `git archive origin/main packages/microcosm-build/src tools` +with that tree first on `PYTHONPATH`; the report's `compiler` field recorded +which `fiscal_targets.py` did the compile. + +The splice and the builder agree: `splice_feed.py` on the old feed and one +`chronicle build-bundle --year 2020 --source packages/irs_soi/w2_statistics_2020` +run at `f98acf4` gives the bytes `tools/build_us_chronicle_feed.py` writes +from the new scope (`facts_sha256` `a81cbcc5…23df04c`). diff --git a/experiments/us-w2-table-year-repin/apply_pin.py b/experiments/us-w2-table-year-repin/apply_pin.py new file mode 100644 index 0000000000..4a2ec19132 --- /dev/null +++ b/experiments/us-w2-table-year-repin/apply_pin.py @@ -0,0 +1,81 @@ +"""Point the US Chronicle feed pin at the rebuilt feed. + +Usage: python apply_pin.py + +Rewrites ``us/chronicle_feed.json`` (commit, build text, row count, facts and +scope digests) and ``DEFAULT_FEED_NAME`` in +``tools/build_us_target_parity_manifest.py`` from the receipt +``tools/build_us_chronicle_feed.py`` wrote into ````. The scope file +must already name the commit; the consumer-fact schema versions are checked +unchanged (the commit is c5e5bf8 plus one package file). +""" + +from __future__ import annotations + +import hashlib +import json +import re +import sys +from pathlib import Path + +worktree, feed_dir, commit, feed_name = ( + Path(sys.argv[1]), + Path(sys.argv[2]), + sys.argv[3], + sys.argv[4], +) +us = worktree / "packages/microcosm-build/src/microcosm/build/us" +scope_bytes = (us / "chronicle_feed_scope.json").read_bytes() +scope_sha256 = hashlib.sha256(scope_bytes).hexdigest() +assert json.loads(scope_bytes)["source_commit"] == commit +receipt = json.loads((feed_dir / "receipt.json").read_text()) +facts = (feed_dir / "consumer_facts.jsonl").read_bytes() +facts_sha256 = hashlib.sha256(facts).hexdigest() +rows = [json.loads(line) for line in facts.splitlines() if line.strip()] +versions = sorted({row["schema_version"] for row in rows}) +assert receipt["source_commit"] == commit, receipt +assert receipt["facts_sha256"] == facts_sha256, receipt +assert receipt["row_count"] == len(rows), receipt +assert receipt["scope_sha256"] == scope_sha256, receipt + +pin_path = us / "chronicle_feed.json" +pin = json.loads(pin_path.read_text()) +assert versions == pin["consumer_fact_schema_versions"], versions +pin["source_commit"] = commit +pin["build"] = ( + "tools/build_us_chronicle_feed.py --skip-artifact: one `chronicle build-bundle " + "--year Y --source ` per build year in chronicle_feed_scope.json, rows " + "kept for exactly the scope's (record set, period) pairs and sorted by " + "aggregate_fact_key. The commit is c5e5bf8 plus only the W-2 item package's " + "literal Tax Year 2020 labels (tag microcosm-us-feed-w2-ty2020-v1, the " + "package file as chronicle#292 left it on main). Bare feed: `chronicle " + "build-consumer-artifact` refuses it at this commit (994 concept_alignment " + "rows lack `authority`); see docs/us-chronicle-feed-repin.md." +) +pin["fact_row_count"] = len(rows) +pin["facts_sha256"] = facts_sha256 +pin["scope_sha256"] = scope_sha256 +pin_path.write_text(json.dumps(pin, indent=2) + "\n") + +tool_path = worktree / "tools/build_us_target_parity_manifest.py" +tool, count = re.subn( + r'DEFAULT_FEED_NAME = "[^"]+"', + f'DEFAULT_FEED_NAME = "{feed_name}"', + tool_path.read_text(), +) +assert count == 1 +tool_path.write_text(tool) +print( + json.dumps( + { + key: pin[key] + for key in ( + "source_commit", + "fact_row_count", + "facts_sha256", + "scope_sha256", + ) + }, + indent=1, + ) +) diff --git a/experiments/us-w2-table-year-repin/compare.json b/experiments/us-w2-table-year-repin/compare.json new file mode 100644 index 0000000000..568cccb746 --- /dev/null +++ b/experiments/us-w2-table-year-repin/compare.json @@ -0,0 +1,169 @@ +{ + "after": { + "compiled_families": 14, + "compiled_targets": 32842, + "fact_count": 39155, + "feed": "consumer_facts.jsonl", + "feed_sha256": "a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c", + "national_state_targets": 5694, + "national_state_version": "61a081ae55bb" + }, + "before": { + "compiled_families": 14, + "compiled_targets": 32842, + "fact_count": 39158, + "feed": "consumer_facts_us_c5e5bf8.jsonl", + "feed_sha256": "b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801", + "national_state_targets": 5694, + "national_state_version": "d315c75804ef" + }, + "exclusion_receipt_w2_ids_after": { + "all_vintage_reviewed_exclusion": [ + "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.return_count" + ], + "allowlisted_vintage_bypass": [], + "m_chip_state_chip_enrollment": [], + "reviewed_exclusion": [] + }, + "exclusion_receipt_w2_ids_before": { + "all_vintage_reviewed_exclusion": [ + "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.return_count", + "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.return_count" + ], + "allowlisted_vintage_bypass": [], + "m_chip_state_chip_enrollment": [], + "reviewed_exclusion": [] + }, + "shared_metadata_changes": [], + "shared_targets": 32841, + "shared_value_changes": [], + "targets_only_after": { + "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.amount": { + "family": "irs_soi", + "in_national_state": true, + "metadata": { + "aged_to": "2024", + "aging_factor": "1.28002921689246", + "aging_factor_source": "chained:irs_soi.ty2023.table_1_4.all.wages_salaries_amount+cbo.revenue_projection.ty2024.income_by_source.wages_and_salaries.projected_amount", + "basis": "projection", + "ledger_source_record_id": "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.amount", + "source_period": "2020", + "target_role": "w2_social_security_tips_total" + }, + "metadata_sha256": "4db643fa9b3f093231dc8573da956f5e04a8ed60669a63bfcdbae2effe0264a2", + "value": 34287530778.932606 + } + }, + "targets_only_before": { + "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.amount": { + "family": "irs_soi", + "in_national_state": true, + "metadata": { + "aged_to": "2024", + "aging_factor": "1.05578784246075", + "aging_factor_source": "cbo.revenue_projection.ty2024.income_by_source.wages_and_salaries.projected_amount", + "basis": "projection", + "ledger_source_record_id": "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.amount", + "source_period": "2023", + "target_role": "w2_social_security_tips_total" + }, + "metadata_sha256": "f02da6c7cc3fd20c1d24b210a358cfdcf59ce4a7ef9610598525418e6e950fba", + "value": 28280884269.407326 + } + }, + "w2_item_facts_after": [ + { + "period": 2020, + "source_record_id": "irs_soi.ty2020.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 277859181000, + "vintage": "tax_year_2020" + }, + { + "period": 2020, + "source_record_id": "irs_soi.ty2020.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 32302509000, + "vintage": "tax_year_2020" + }, + { + "period": 2020, + "source_record_id": "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.amount", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 26786522000, + "vintage": "tax_year_2020" + }, + { + "period": 2020, + "source_record_id": "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.return_count", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 6038613, + "vintage": "tax_year_2020" + }, + { + "period": 2020, + "source_record_id": "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.taxpayer_count", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 6105713, + "vintage": "tax_year_2020" + } + ], + "w2_item_facts_before": [ + { + "period": 2020, + "source_record_id": "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.amount", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 26786522000, + "vintage": "tax_year_2020" + }, + { + "period": 2020, + "source_record_id": "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.return_count", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 6038613, + "vintage": "tax_year_2020" + }, + { + "period": 2020, + "source_record_id": "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.taxpayer_count", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 6105713, + "vintage": "tax_year_2020" + }, + { + "period": 2023, + "source_record_id": "irs_soi.ty2023.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 277859181000, + "vintage": "tax_year_2023" + }, + { + "period": 2023, + "source_record_id": "irs_soi.ty2023.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 32302509000, + "vintage": "tax_year_2023" + }, + { + "period": 2023, + "source_record_id": "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.amount", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 26786522000, + "vintage": "tax_year_2023" + }, + { + "period": 2023, + "source_record_id": "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.return_count", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 6038613, + "vintage": "tax_year_2023" + }, + { + "period": 2023, + "source_record_id": "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.taxpayer_count", + "source_table": "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020", + "value": 6105713, + "vintage": "tax_year_2023" + } + ] +} diff --git a/experiments/us-w2-table-year-repin/compare_feeds.py b/experiments/us-w2-table-year-repin/compare_feeds.py new file mode 100644 index 0000000000..89ab2a50e7 --- /dev/null +++ b/experiments/us-w2-table-year-repin/compare_feeds.py @@ -0,0 +1,104 @@ +"""What moving the US Chronicle feed pin changed, row by row. + +Usage: python compare_feeds.py + +Rows are compared as bytes, then by ``lineage.source_record_id``: ids only in +the old feed, ids only in the new feed, and ids in both whose bytes differ. +For each new-only id the report names the old-only id with the same record +set family, source row and source column (its twin under another year stamp) +and lists the fields that differ between the two. +""" + +from __future__ import annotations + +import hashlib +import json +import re +import sys +from pathlib import Path + +old_path, new_path, out_path = sys.argv[1:4] +PERIOD_TOKEN = re.compile(r"\.(?:ty|cy|fy)\d{4}(?=\.)") + + +def load(path: str) -> tuple[dict[str, bytes], str, int]: + payload = Path(path).read_bytes() + rows: dict[str, bytes] = {} + for line in payload.splitlines(keepends=True): + if not line.strip(): + continue + source_record_id = json.loads(line)["lineage"]["source_record_id"] + assert source_record_id not in rows, source_record_id + rows[source_record_id] = line + return rows, hashlib.sha256(payload).hexdigest(), len(payload) + + +def flat(value: object, prefix: str = "") -> dict[str, object]: + if isinstance(value, dict): + out: dict[str, object] = {} + for key, item in value.items(): + out.update(flat(item, f"{prefix}.{key}" if prefix else key)) + return out + return {prefix: value} + + +def cell(row: dict) -> tuple[str, str, str]: + layout = row["layout"] + return ( + PERIOD_TOKEN.sub("", "." + layout["record_set_id"]).lstrip("."), + str(layout.get("source_row_id")), + str(layout.get("source_column_id")), + ) + + +old, old_sha256, old_bytes = load(old_path) +new, new_sha256, new_bytes = load(new_path) +only_old = sorted(set(old) - set(new)) +only_new = sorted(set(new) - set(old)) +changed = sorted(i for i in set(old) & set(new) if old[i] != new[i]) +old_cells = {cell(json.loads(old[i])): i for i in only_old} +twins = {} +for source_record_id in only_new: + row = json.loads(new[source_record_id]) + twin_id = old_cells.get(cell(row)) + entry: dict[str, object] = {"old_source_record_id": twin_id} + if twin_id is not None: + before, after = flat(json.loads(old[twin_id])), flat(row) + entry["value_equal"] = before["value"] == after["value"] + entry["source_cell_keys_equal"] = ( + json.loads(old[twin_id])["lineage"]["source_cell_keys"] + == row["lineage"]["source_cell_keys"] + ) + entry["fields_changed"] = { + key: [before.get(key), after.get(key)] + for key in sorted(set(before) | set(after)) + if before.get(key) != after.get(key) + } + twins[source_record_id] = entry +report = { + "old": {"rows": len(old), "facts_sha256": old_sha256, "bytes": old_bytes}, + "new": {"rows": len(new), "facts_sha256": new_sha256, "bytes": new_bytes}, + "rows_byte_identical": len(set(old) & set(new)) - len(changed), + "ids_in_both_with_different_bytes": changed, + "ids_only_in_old": only_old, + "ids_only_in_new": only_new, + "old_only_ids_with_no_new_twin": sorted( + set(only_old) - {entry["old_source_record_id"] for entry in twins.values()} + ), + "new_only_rows": twins, + "new_feed_sorted_by_aggregate_fact_key": [ + json.loads(line)["aggregate_fact_key"] for line in new.values() + ] + == sorted(json.loads(line)["aggregate_fact_key"] for line in new.values()), +} +Path(out_path).write_text(json.dumps(report, indent=1, sort_keys=True) + "\n") +print( + json.dumps( + { + key: (len(value) if isinstance(value, list | dict) else value) + for key, value in report.items() + if key not in {"old", "new"} + }, + indent=1, + ) +) diff --git a/experiments/us-w2-table-year-repin/compare_surfaces.py b/experiments/us-w2-table-year-repin/compare_surfaces.py new file mode 100644 index 0000000000..9dffc6562c --- /dev/null +++ b/experiments/us-w2-table-year-repin/compare_surfaces.py @@ -0,0 +1,61 @@ +"""Diff two measure_surface.py reports target by target. + +Usage: python compare_surfaces.py + +Lists targets only in one report, targets whose value changed and targets +whose reported metadata changed, and restates the W-2 rows of both. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +before_path, after_path, out_path = sys.argv[1:4] +before = json.loads(Path(before_path).read_text()) +after = json.loads(Path(after_path).read_text()) +old, new = before["targets"], after["targets"] +shared = sorted(set(old) & set(new)) +summary_keys = ( + "feed", + "feed_sha256", + "fact_count", + "compiled_targets", + "compiled_families", + "national_state_targets", + "national_state_version", +) +report = { + "before": {key: before[key] for key in summary_keys}, + "after": {key: after[key] for key in summary_keys}, + "targets_only_before": {name: old[name] for name in sorted(set(old) - set(new))}, + "targets_only_after": {name: new[name] for name in sorted(set(new) - set(old))}, + "shared_targets": len(shared), + "shared_value_changes": [ + {"name": name, "before": old[name]["value"], "after": new[name]["value"]} + for name in shared + if old[name]["value"] != new[name]["value"] + ], + "shared_metadata_changes": [ + {"name": name, "before": old[name], "after": new[name]} + for name in shared + if {k: v for k, v in old[name].items() if k != "value"} + != {k: v for k, v in new[name].items() if k != "value"} + ], + "w2_item_facts_before": before["w2_item_facts"], + "w2_item_facts_after": after["w2_item_facts"], + "exclusion_receipt_w2_ids_before": before["exclusion_receipt_w2_ids"], + "exclusion_receipt_w2_ids_after": after["exclusion_receipt_w2_ids"], +} +Path(out_path).write_text(json.dumps(report, indent=1, sort_keys=True) + "\n") +print( + json.dumps( + { + key: (len(value) if isinstance(value, list | dict) else value) + for key, value in report.items() + if key.startswith(("targets_only", "shared")) + }, + indent=1, + ) +) diff --git a/experiments/us-w2-table-year-repin/feed_compare.json b/experiments/us-w2-table-year-repin/feed_compare.json new file mode 100644 index 0000000000..da96a3aa1f --- /dev/null +++ b/experiments/us-w2-table-year-repin/feed_compare.json @@ -0,0 +1,127 @@ +{ + "ids_in_both_with_different_bytes": [], + "ids_only_in_new": [ + "irs_soi.ty2020.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount", + "irs_soi.ty2020.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount" + ], + "ids_only_in_old": [ + "irs_soi.ty2023.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount", + "irs_soi.ty2023.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount", + "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.amount", + "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.return_count", + "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.taxpayer_count" + ], + "new": { + "bytes": 164591409, + "facts_sha256": "a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c", + "rows": 39155 + }, + "new_feed_sorted_by_aggregate_fact_key": true, + "new_only_rows": { + "irs_soi.ty2020.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount": { + "fields_changed": { + "aggregate_fact_key": [ + "ledger.aggregate_fact.v2:8f327e6f8743a570967d6c6e", + "ledger.aggregate_fact.v2:51d2feb7b4c4ddfe18eb53e2" + ], + "label": [ + "United States 2023 tax year sum irs soi form w2 401k elective deferrals for tax unit [irs_soi Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020 20in04w2all.xlsx tax_year_2023]", + "United States 2020 tax year sum irs soi form w2 401k elective deferrals for tax unit [irs_soi Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020 20in04w2all.xlsx tax_year_2020]" + ], + "layout.record_set_id": [ + "irs_soi.ty2023.form_w2_401k_elective_deferrals", + "irs_soi.ty2020.form_w2_401k_elective_deferrals" + ], + "layout.record_set_spec_hash": [ + "8e7516eb16b98e31575928d7", + "0d8f03f75753fe74a7c3d7f7" + ], + "legacy_fact_key": [ + "ledger.fact.v1:4196175a325b82b1742c136e", + "ledger.fact.v1:8f42ed2ce6cc255ccc5bd3ae" + ], + "lineage.source_record_id": [ + "irs_soi.ty2023.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount", + "irs_soi.ty2020.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount" + ], + "period.value": [ + 2023, + 2020 + ], + "semantic_fact_key": [ + "ledger.semantic_fact.v2:0eb672dc84ef89b509d6dd07", + "ledger.semantic_fact.v2:5148171c03407554ec9edad4" + ], + "source.vintage": [ + "tax_year_2023", + "tax_year_2020" + ], + "source_release_key": [ + "ledger.source_release.v2:e160bbe5850bccc8157b4753", + "ledger.source_release.v2:249f35d435e1b0c55f4e0606" + ] + }, + "old_source_record_id": "irs_soi.ty2023.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount", + "source_cell_keys_equal": true, + "value_equal": true + }, + "irs_soi.ty2020.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount": { + "fields_changed": { + "aggregate_fact_key": [ + "ledger.aggregate_fact.v2:389b3202578b67596ebc0284", + "ledger.aggregate_fact.v2:f1737fc1e1b8ddbe91b0d32a" + ], + "label": [ + "United States 2023 tax year sum irs soi form w2 designated roth 401k contributions for tax unit [irs_soi Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020 20in04w2all.xlsx tax_year_2023]", + "United States 2020 tax year sum irs soi form w2 designated roth 401k contributions for tax unit [irs_soi Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and Earner Type, Tax Year 2020 20in04w2all.xlsx tax_year_2020]" + ], + "layout.record_set_id": [ + "irs_soi.ty2023.form_w2_designated_roth_401k_contributions", + "irs_soi.ty2020.form_w2_designated_roth_401k_contributions" + ], + "layout.record_set_spec_hash": [ + "aa782f50bae2d6dcb9592090", + "90656a57a04039035f81f102" + ], + "legacy_fact_key": [ + "ledger.fact.v1:f2768e74a0c730466ab6195b", + "ledger.fact.v1:b9cd6ee65bd8bd5fa9a49c1f" + ], + "lineage.source_record_id": [ + "irs_soi.ty2023.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount", + "irs_soi.ty2020.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount" + ], + "period.value": [ + 2023, + 2020 + ], + "semantic_fact_key": [ + "ledger.semantic_fact.v2:53522560d80a79fad85bd5b0", + "ledger.semantic_fact.v2:eea093242a6a4233755bfc2e" + ], + "source.vintage": [ + "tax_year_2023", + "tax_year_2020" + ], + "source_release_key": [ + "ledger.source_release.v2:e160bbe5850bccc8157b4753", + "ledger.source_release.v2:249f35d435e1b0c55f4e0606" + ] + }, + "old_source_record_id": "irs_soi.ty2023.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount", + "source_cell_keys_equal": true, + "value_equal": true + } + }, + "old": { + "bytes": 164603204, + "facts_sha256": "b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801", + "rows": 39158 + }, + "old_only_ids_with_no_new_twin": [ + "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.amount", + "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.return_count", + "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.taxpayer_count" + ], + "rows_byte_identical": 39153 +} diff --git a/experiments/us-w2-table-year-repin/measure_surface.py b/experiments/us-w2-table-year-repin/measure_surface.py new file mode 100644 index 0000000000..32ee3e566f --- /dev/null +++ b/experiments/us-w2-table-year-repin/measure_surface.py @@ -0,0 +1,123 @@ +"""Compile a US Chronicle feed as the release does and report the W-2 rows. + +Usage: python measure_surface.py + +The compile is the release's: ``compile_us_fiscal_target_registry`` at target +period 2024 with the packaged congressional-district crosswalk and aging on, +then the Medicaid enrollment substitutions, then +``--target-surface national_state``. The report lists every compiled target +(name, value, aging metadata and a digest of all its metadata) so two runs can +be compared target by target. +""" + +from __future__ import annotations + +import hashlib +import importlib.util +import json +import sys +from pathlib import Path + +from microcosm.build.ledger_artifact import load_ledger_consumer_artifact +from microcosm.build.us_runtime import ( + apply_us_medicaid_enrollment_substitutions, + compile_us_fiscal_target_registry, + default_congressional_district_vintage_crosswalk_path, + load_congressional_district_vintage_crosswalk, + us_fiscal_target_exclusion_receipt, +) +from microcosm.calibrate import TargetRegistry + +feed_path, feed_sha256, out_path = sys.argv[1:4] +REPO = Path(__file__).resolve().parents[2] +AGING_KEYS = ( + "source_period", + "aged_to", + "aging_factor", + "aging_factor_source", + "basis", + "target_role", + "ledger_source_record_id", +) + + +def _load_tool(name: str): + spec = importlib.util.spec_from_file_location(name, REPO / "tools" / f"{name}.py") + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +facts = load_ledger_consumer_artifact( + feed_path, expected_facts_sha256=feed_sha256, expected_manifest_sha256=None +).facts +crosswalk = load_congressional_district_vintage_crosswalk( + default_congressional_district_vintage_crosswalk_path() +) +registry = compile_us_fiscal_target_registry( + facts, + target_period=2024, + congressional_district_vintage_crosswalk=crosswalk, + age_targets=True, +) +registry, _ = apply_us_medicaid_enrollment_substitutions(registry) +builder = _load_tool("build_us_fiscal_refresh_release") +surface_specs, _ = builder._select_target_surface(registry.specs, "national_state") +surface = TargetRegistry(surface_specs, country="us") +receipt = us_fiscal_target_exclusion_receipt( + facts, target_period=2024, congressional_district_vintage_crosswalk=crosswalk +) +surface_names = {spec.name for spec in surface.specs} + + +def _record(spec) -> dict[str, object]: + return { + "value": spec.value, + "family": spec.family, + "in_national_state": spec.name in surface_names, + "metadata_sha256": hashlib.sha256( + json.dumps(dict(spec.metadata), sort_keys=True, default=str).encode() + ).hexdigest(), + "metadata": { + key: spec.metadata[key] + for key in sorted(spec.metadata) + if key in AGING_KEYS + }, + } + + +report = { + "feed": Path(feed_path).name, + "feed_sha256": feed_sha256, + "compiler": str(Path(compile_us_fiscal_target_registry.__code__.co_filename)), + "fact_count": len(facts), + "compiled_targets": len(registry.specs), + "compiled_families": len({spec.family for spec in registry.specs}), + "national_state_targets": len(surface.specs), + "national_state_version": surface.version, + "w2_item_facts": sorted( + ( + { + "source_record_id": fact["lineage"]["source_record_id"], + "period": fact["period"]["value"], + "vintage": fact["source"]["vintage"], + "source_table": fact["source"]["source_table"], + "value": fact["value"], + } + for fact in facts + if fact["layout"].get("groupby_dimension") == "irs_soi.form_w2_item" + ), + key=lambda row: row["source_record_id"], + ), + "w2_item_targets": { + spec.name: _record(spec) for spec in registry.specs if "form_w2" in spec.name + }, + "exclusion_receipt_w2_ids": { + rule: [i for i in body["source_record_ids"] if "form_w2" in i] + for rule, body in receipt["rules"].items() + }, + "targets": {spec.name: _record(spec) for spec in registry.specs}, +} +Path(out_path).write_text(json.dumps(report, indent=1, sort_keys=True) + "\n") +print(json.dumps({k: v for k, v in report.items() if k != "targets"}, indent=1)) diff --git a/experiments/us-w2-table-year-repin/receipt.json b/experiments/us-w2-table-year-repin/receipt.json new file mode 100644 index 0000000000..37abb1e93f --- /dev/null +++ b/experiments/us-w2-table-year-repin/receipt.json @@ -0,0 +1,136 @@ +{ + "scope": "packages/microcosm-build/src/microcosm/build/us/chronicle_feed_scope.json", + "scope_sha256": "e61ae1d7e4eca9abbbb895436864c10e3f8c8a97421f786abd027768be78cd50", + "source_commit": "f98acf4edcc9488343446fda46db77914de8611f", + "chronicle_root": "/Users/maxghenis/PolicyEngine/_worktrees/chronicle-feed-w2-2020", + "runs": [ + { + "year": 2020, + "packages": [ + "packages/bea/nipa_total_wages_salaries", + "packages/cms_medicaid/chip_monthly_enrollment_dataset", + "packages/irs_soi/table_1_4", + "packages/irs_soi/w2_statistics_2020" + ], + "command": "uv run --frozen chronicle build-bundle --year 2020 --source packages/bea/nipa_total_wages_salaries --source packages/cms_medicaid/chip_monthly_enrollment_dataset --source packages/irs_soi/table_1_4 --source packages/irs_soi/w2_statistics_2020 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2020 --replace" + }, + { + "year": 2021, + "packages": [ + "packages/irs_soi/table_1_4" + ], + "command": "uv run --frozen chronicle build-bundle --year 2021 --source packages/irs_soi/table_1_4 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2021 --replace" + }, + { + "year": 2022, + "packages": [ + "packages/irs_soi/historic_table_2", + "packages/irs_soi/historic_table_2_state_agi_2022", + "packages/irs_soi/historic_table_2_state_broad_2022", + "packages/irs_soi/historic_table_2_state_eitc_2022", + "packages/irs_soi/table_1_1", + "packages/irs_soi/table_1_4", + "packages/irs_soi/table_2_5_eitc_agi_children_2022" + ], + "command": "uv run --frozen chronicle build-bundle --year 2022 --source packages/irs_soi/historic_table_2 --source packages/irs_soi/historic_table_2_state_agi_2022 --source packages/irs_soi/historic_table_2_state_broad_2022 --source packages/irs_soi/historic_table_2_state_eitc_2022 --source packages/irs_soi/table_1_1 --source packages/irs_soi/table_1_4 --source packages/irs_soi/table_2_5_eitc_agi_children_2022 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2022 --replace" + }, + { + "year": 2023, + "packages": [ + "packages/bea/nipa_pension_contributions", + "packages/bea/nipa_personal_income_components", + "packages/bea/nipa_personal_income_disposition", + "packages/bea/nipa_total_wages_salaries", + "packages/bea/regional_personal_income_state", + "packages/cbo/individual_income_tax_receipts_2026_02", + "packages/cbo/revenue_projections_income_by_source_2026_02", + "packages/census/b01001_female_15_44_2023", + "packages/census/population_projections_2023", + "packages/census/stc_individual_income_tax", + "packages/cms_nhe/historical_service_source", + "packages/cms_nhe/table_24", + "packages/federal_reserve/z1_household_net_worth", + "packages/hhs_acf_liheap/fy2023_national_profile", + "packages/irs_soi/congressional_district_2022", + "packages/irs_soi/ira_roth_contributions_2022", + "packages/irs_soi/ira_traditional_contributions_2022", + "packages/irs_soi/state_2022", + "packages/irs_soi/table_1_1", + "packages/irs_soi/table_1_2", + "packages/irs_soi/table_1_4", + "packages/irs_soi/table_2_1", + "packages/irs_soi/table_2_5", + "packages/irs_soi/table_2_5_eitc_agi_children_2023", + "packages/irs_soi/table_4_3" + ], + "command": "uv run --frozen chronicle build-bundle --year 2023 --source packages/bea/nipa_pension_contributions --source packages/bea/nipa_personal_income_components --source packages/bea/nipa_personal_income_disposition --source packages/bea/nipa_total_wages_salaries --source packages/bea/regional_personal_income_state --source packages/cbo/individual_income_tax_receipts_2026_02 --source packages/cbo/revenue_projections_income_by_source_2026_02 --source packages/census/b01001_female_15_44_2023 --source packages/census/population_projections_2023 --source packages/census/stc_individual_income_tax --source packages/cms_nhe/historical_service_source --source packages/cms_nhe/table_24 --source packages/federal_reserve/z1_household_net_worth --source packages/hhs_acf_liheap/fy2023_national_profile --source packages/irs_soi/congressional_district_2022 --source packages/irs_soi/ira_roth_contributions_2022 --source packages/irs_soi/ira_traditional_contributions_2022 --source packages/irs_soi/state_2022 --source packages/irs_soi/table_1_1 --source packages/irs_soi/table_1_2 --source packages/irs_soi/table_1_4 --source packages/irs_soi/table_2_1 --source packages/irs_soi/table_2_5 --source packages/irs_soi/table_2_5_eitc_agi_children_2023 --source packages/irs_soi/table_4_3 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2023 --replace" + }, + { + "year": 2024, + "packages": [ + "packages/bea/nipa_total_wages_salaries", + "packages/cbo/revenue_projections_income_by_source_2026_02", + "packages/census/pep_2024_national_age_sex", + "packages/census/pep_2024_state_age_sex", + "packages/cms_aca/oep_state_level", + "packages/cms_medicaid/chip_monthly_enrollment_december_2024", + "packages/cms_medicare/medicare_trustees_report_2025", + "packages/cms_nhe/table_24", + "packages/hhs_acf/tanf_caseload_2024", + "packages/hhs_acf/tanf_financial_2024", + "packages/hhs_acf_liheap/fy2024_national_profile", + "packages/irs_soi/filing_season_week47_2024", + "packages/jct/tax_expenditures_2024", + "packages/kff/marketplace_effectuated_enrollment", + "packages/ssa/annual_statistical_supplement_2025", + "packages/ssa/ssi_monthly_statistics_2024_12", + "packages/ssa/ssi_table_7b1_2024", + "packages/usda_snap/fy69_to_current" + ], + "command": "uv run --frozen chronicle build-bundle --year 2024 --source packages/bea/nipa_total_wages_salaries --source packages/cbo/revenue_projections_income_by_source_2026_02 --source packages/census/pep_2024_national_age_sex --source packages/census/pep_2024_state_age_sex --source packages/cms_aca/oep_state_level --source packages/cms_medicaid/chip_monthly_enrollment_december_2024 --source packages/cms_medicare/medicare_trustees_report_2025 --source packages/cms_nhe/table_24 --source packages/hhs_acf/tanf_caseload_2024 --source packages/hhs_acf/tanf_financial_2024 --source packages/hhs_acf_liheap/fy2024_national_profile --source packages/irs_soi/filing_season_week47_2024 --source packages/jct/tax_expenditures_2024 --source packages/kff/marketplace_effectuated_enrollment --source packages/ssa/annual_statistical_supplement_2025 --source packages/ssa/ssi_monthly_statistics_2024_12 --source packages/ssa/ssi_table_7b1_2024 --source packages/usda_snap/fy69_to_current --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2024 --replace" + }, + { + "year": 2025, + "packages": [ + "packages/cbo/revenue_projections_income_by_source_2026_02" + ], + "command": "uv run --frozen chronicle build-bundle --year 2025 --source packages/cbo/revenue_projections_income_by_source_2026_02 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2025 --replace" + }, + { + "year": 2026, + "packages": [ + "packages/cbo/revenue_projections_income_by_source_2026_02", + "packages/jct/obbba_revenue_estimates_2025" + ], + "command": "uv run --frozen chronicle build-bundle --year 2026 --source packages/cbo/revenue_projections_income_by_source_2026_02 --source packages/jct/obbba_revenue_estimates_2025 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2026 --replace" + }, + { + "year": 2027, + "packages": [ + "packages/jct/obbba_revenue_estimates_2025" + ], + "command": "uv run --frozen chronicle build-bundle --year 2027 --source packages/jct/obbba_revenue_estimates_2025 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2027 --replace" + }, + { + "year": 2028, + "packages": [ + "packages/jct/obbba_revenue_estimates_2025" + ], + "command": "uv run --frozen chronicle build-bundle --year 2028 --source packages/jct/obbba_revenue_estimates_2025 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2028 --replace" + }, + { + "year": 2029, + "packages": [ + "packages/jct/obbba_revenue_estimates_2025" + ], + "command": "uv run --frozen chronicle build-bundle --year 2029 --source packages/jct/obbba_revenue_estimates_2025 --out /private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/bundles/2029 --replace" + } + ], + "pair_count": 585, + "rows_per_pair_min": 1, + "rows_per_pair_max": 26880, + "row_count": 39155, + "facts_path": "/private/tmp/claude-501/-Users-maxghenis-PolicyEngine-microcosm--claude-worktrees-exciting-joliot-e8c36c/75ec5d70-ebad-4af4-8643-6ff4e43b9391/scratchpad/us-feed-w2-run1/consumer_facts.jsonl", + "facts_sha256": "a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c", + "artifact": null +} diff --git a/experiments/us-w2-table-year-repin/relabel_scope.py b/experiments/us-w2-table-year-repin/relabel_scope.py new file mode 100644 index 0000000000..8c0e7c5e71 --- /dev/null +++ b/experiments/us-w2-table-year-repin/relabel_scope.py @@ -0,0 +1,65 @@ +"""Move the US Chronicle feed scope's W-2 item pairs to the tax year IRS published. + +Usage: python relabel_scope.py +Idempotent: re-running with the same commit rewrites the same bytes. + +The scope built ``soi-w2-statistics-2020`` at ``--year 2023`` for three record +sets. That package reads one pinned workbook, IRS SOI Table 4.B for Tax Year +2020 (``20in04w2all.xlsx``), so those builds stamped TY2020 cells as ty2023. +At the commit this script is given the package's labels are literal 2020, so +the three ty2023 pairs go and the two TY2020 record sets the scope lacked +(401(k) elective deferrals, designated Roth contributions) come in. The TY2020 +tips pair was already in the scope. +""" + +from __future__ import annotations + +import json +import sys + +PACKAGE = "packages/irs_soi/w2_statistics_2020" +PACKAGE_ID = "soi-w2-statistics-2020" +ITEM_RECORD_SETS = ( + "form_w2_401k_elective_deferrals", + "form_w2_designated_roth_401k_contributions", + "form_w2_social_security_tips", +) +TABLE_TAX_YEAR = 2020 +RULE = ( + "The (record_set_id, period) pairs of the pinned feed " + "consumer_facts_buildn_v9_4.jsonl, with the three W-2 item record sets it " + "stamped ty2023 replaced by the TY2020 record sets of the table they are " + "read from (IRS SOI Table 4.B, Tax Year 2020; IRS publishes no later " + "year); each built from the named package with " + "`chronicle build-bundle --year --source `." +) + +path, commit = sys.argv[1], sys.argv[2] +assert len(commit) == 40, commit +scope = json.load(open(path)) +pairs = [pair for pair in scope["pairs"] if pair["package"] != PACKAGE] +dropped = [pair for pair in scope["pairs"] if pair["package"] == PACKAGE] +assert {pair["package_id"] for pair in dropped} == {PACKAGE_ID}, dropped +for record_set in ITEM_RECORD_SETS: + pairs.append( + { + "record_set_id": f"irs_soi.ty{TABLE_TAX_YEAR}.{record_set}", + "period_type": "tax_year", + "period_value": str(TABLE_TAX_YEAR), + "package_id": PACKAGE_ID, + "package": PACKAGE, + "build_year": TABLE_TAX_YEAR, + } + ) +pairs.sort(key=lambda e: (e["record_set_id"], e["period_type"], e["period_value"])) +runs: dict[str, set[str]] = {} +for pair in pairs: + runs.setdefault(str(pair["build_year"]), set()).add(pair["package"]) +scope["source_commit"] = commit +scope["rule"] = RULE +scope["pair_count"] = len(pairs) +scope["runs"] = {year: sorted(runs[year]) for year in sorted(runs)} +scope["pairs"] = pairs +with open(path, "w") as handle: + handle.write(json.dumps(scope, indent=1) + "\n") +print(len(pairs), sum(len(v) for v in scope["runs"].values())) diff --git a/experiments/us-w2-table-year-repin/splice_feed.py b/experiments/us-w2-table-year-repin/splice_feed.py new file mode 100644 index 0000000000..ad0fd962aa --- /dev/null +++ b/experiments/us-w2-table-year-repin/splice_feed.py @@ -0,0 +1,64 @@ +"""Predict the re-pinned feed from the old pin and one package run. + +Usage: python splice_feed.py + +``docs/us-fact-to-target.md`` (step 1) describes a new feed as the previous +feed plus the new rows, deduplicated on ``lineage.source_record_id``. This +applies that rule to the W-2 item package: drop every old row the +``soi-w2-statistics-2020`` package emitted, add the rows the package emits at +the new commit, and sort by ``aggregate_fact_key`` as the feed builder does. +The result is an independent prediction of what +``tools/build_us_chronicle_feed.py`` writes from the new scope; the two must +be byte-identical (see README.md). +""" + +from __future__ import annotations + +import hashlib +import json +import sys + +W2_SOURCE_FILE = "20in04w2all.xlsx" + +old_path, package_path, out_path = sys.argv[1:4] +kept: dict[str, bytes] = {} +dropped: list[str] = [] +with open(old_path, "rb") as stream: + for line in stream: + if not line.strip(): + continue + row = json.loads(line) + if row["source"].get("source_file") == W2_SOURCE_FILE: + dropped.append(row["lineage"]["source_record_id"]) + continue + kept[row["aggregate_fact_key"]] = line +added: list[str] = [] +seen_ids: set[str] = set() +with open(package_path, "rb") as stream: + for raw in stream: + if not raw.strip(): + continue + line = raw if raw.endswith(b"\n") else raw + b"\n" + row = json.loads(line) + source_record_id = row["lineage"]["source_record_id"] + assert source_record_id not in seen_ids, source_record_id + seen_ids.add(source_record_id) + assert row["aggregate_fact_key"] not in kept, source_record_id + kept[row["aggregate_fact_key"]] = line + added.append(source_record_id) +payload = b"".join(kept[key] for key in sorted(kept)) +with open(out_path, "wb") as stream: + stream.write(payload) +print( + json.dumps( + { + "rows": len(kept), + "facts_sha256": hashlib.sha256(payload).hexdigest(), + "bytes": len(payload), + "dropped": sorted(dropped), + "added": sorted(added), + "superseded_without_replacement": sorted(set(dropped) - set(added)), + }, + indent=1, + ) +) diff --git a/packages/microcosm-build/src/microcosm/build/us/chronicle_feed.json b/packages/microcosm-build/src/microcosm/build/us/chronicle_feed.json index acc37281e1..164b30bac4 100644 --- a/packages/microcosm-build/src/microcosm/build/us/chronicle_feed.json +++ b/packages/microcosm-build/src/microcosm/build/us/chronicle_feed.json @@ -3,15 +3,15 @@ "country": "us", "scope": "us_fiscal_targets", "source_repo": "PolicyEngine/chronicle", - "source_commit": "c5e5bf8aa84960c1a200ee47303b19c953092d0f", - "build": "tools/build_us_chronicle_feed.py --skip-artifact: one `chronicle build-bundle --year Y --source ` per build year in chronicle_feed_scope.json, rows kept for exactly the scope's (record set, period) pairs and sorted by aggregate_fact_key. Bare feed: `chronicle build-consumer-artifact` refuses it at this commit (994 concept_alignment rows lack `authority`); see docs/us-chronicle-feed-repin.md.", + "source_commit": "f98acf4edcc9488343446fda46db77914de8611f", + "build": "tools/build_us_chronicle_feed.py --skip-artifact: one `chronicle build-bundle --year Y --source ` per build year in chronicle_feed_scope.json, rows kept for exactly the scope's (record set, period) pairs and sorted by aggregate_fact_key. The commit is c5e5bf8 plus only the W-2 item package's literal Tax Year 2020 labels (tag microcosm-us-feed-w2-ty2020-v1, the package file as chronicle#292 left it on main). Bare feed: `chronicle build-consumer-artifact` refuses it at this commit (994 concept_alignment rows lack `authority`); see docs/us-chronicle-feed-repin.md.", "artifact_schema_version": null, "consumer_fact_schema_versions": [ "chronicle.consumer_fact.v3" ], "consumer_fact_schema_sha256": "bdb51e2a8115634633ba7448c4005930fd9c0bfbade5e1b079b6bc24da485d3d", - "fact_row_count": 39158, - "facts_sha256": "b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801", + "fact_row_count": 39155, + "facts_sha256": "a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c", "manifest_sha256": null, - "scope_sha256": "b769887d716e90612518788700c5c01dba8b411c5c75f3d47bb8b1273550ddc1" + "scope_sha256": "e61ae1d7e4eca9abbbb895436864c10e3f8c8a97421f786abd027768be78cd50" } diff --git a/packages/microcosm-build/src/microcosm/build/us/chronicle_feed_scope.json b/packages/microcosm-build/src/microcosm/build/us/chronicle_feed_scope.json index 9afd1dfabf..77393cc597 100644 --- a/packages/microcosm-build/src/microcosm/build/us/chronicle_feed_scope.json +++ b/packages/microcosm-build/src/microcosm/build/us/chronicle_feed_scope.json @@ -2,9 +2,9 @@ "version": 1, "country": "us", "source_repo": "PolicyEngine/chronicle", - "source_commit": "c5e5bf8aa84960c1a200ee47303b19c953092d0f", - "rule": "Exactly the (record_set_id, period) pairs of the pinned feed consumer_facts_buildn_v9_4.jsonl; each built from the named package with `chronicle build-bundle --year --source `.", - "pair_count": 586, + "source_commit": "f98acf4edcc9488343446fda46db77914de8611f", + "rule": "The (record_set_id, period) pairs of the pinned feed consumer_facts_buildn_v9_4.jsonl, with the three W-2 item record sets it stamped ty2023 replaced by the TY2020 record sets of the table they are read from (IRS SOI Table 4.B, Tax Year 2020; IRS publishes no later year); each built from the named package with `chronicle build-bundle --year --source `.", + "pair_count": 585, "runs": { "2020": [ "packages/bea/nipa_total_wages_salaries", @@ -49,8 +49,7 @@ "packages/irs_soi/table_2_1", "packages/irs_soi/table_2_5", "packages/irs_soi/table_2_5_eitc_agi_children_2023", - "packages/irs_soi/table_4_3", - "packages/irs_soi/w2_statistics_2020" + "packages/irs_soi/table_4_3" ], "2024": [ "packages/bea/nipa_total_wages_salaries", @@ -3010,6 +3009,22 @@ "package": "packages/hhs_acf/tanf_financial_2024", "build_year": 2024 }, + { + "record_set_id": "irs_soi.ty2020.form_w2_401k_elective_deferrals", + "period_type": "tax_year", + "period_value": "2020", + "package_id": "soi-w2-statistics-2020", + "package": "packages/irs_soi/w2_statistics_2020", + "build_year": 2020 + }, + { + "record_set_id": "irs_soi.ty2020.form_w2_designated_roth_401k_contributions", + "period_type": "tax_year", + "period_value": "2020", + "package_id": "soi-w2-statistics-2020", + "package": "packages/irs_soi/w2_statistics_2020", + "build_year": 2020 + }, { "record_set_id": "irs_soi.ty2020.form_w2_social_security_tips", "period_type": "tax_year", @@ -4322,30 +4337,6 @@ "package": "packages/irs_soi/congressional_district_2022", "build_year": 2023 }, - { - "record_set_id": "irs_soi.ty2023.form_w2_401k_elective_deferrals", - "period_type": "tax_year", - "period_value": "2023", - "package_id": "soi-w2-statistics-2020", - "package": "packages/irs_soi/w2_statistics_2020", - "build_year": 2023 - }, - { - "record_set_id": "irs_soi.ty2023.form_w2_designated_roth_401k_contributions", - "period_type": "tax_year", - "period_value": "2023", - "package_id": "soi-w2-statistics-2020", - "package": "packages/irs_soi/w2_statistics_2020", - "build_year": 2023 - }, - { - "record_set_id": "irs_soi.ty2023.form_w2_social_security_tips", - "period_type": "tax_year", - "period_value": "2023", - "package_id": "soi-w2-statistics-2020", - "package": "packages/irs_soi/w2_statistics_2020", - "build_year": 2023 - }, { "record_set_id": "irs_soi.ty2023.roth_ira_contributions.all_taxpayers", "period_type": "tax_year", diff --git a/packages/microcosm-build/src/microcosm/build/us/target_parity_feed_families.json b/packages/microcosm-build/src/microcosm/build/us/target_parity_feed_families.json index 6a8c987fda..1c19b97930 100644 --- a/packages/microcosm-build/src/microcosm/build/us/target_parity_feed_families.json +++ b/packages/microcosm-build/src/microcosm/build/us/target_parity_feed_families.json @@ -1,6 +1,6 @@ { - "feed": "consumer_facts_us_c5e5bf8.jsonl", - "feed_sha256": "b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801", + "feed": "consumer_facts_us_f98acf4.jsonl", + "feed_sha256": "a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c", "families": { "bea_nipa.business_current_transfer_receipts": 1, "bea_nipa.defined_contribution_actual_contributions": 1, @@ -59,7 +59,7 @@ "irs_soi.filing_season_week47": 34, "irs_soi.form_w2_401k_elective_deferrals": 1, "irs_soi.form_w2_designated_roth_401k_contributions": 1, - "irs_soi.form_w2_social_security_tips": 6, + "irs_soi.form_w2_social_security_tips": 3, "irs_soi.historic_table_2": 5654, "irs_soi.roth_ira_contributions": 2, "irs_soi.state_2022": 4, diff --git a/packages/microcosm-build/src/microcosm/build/us/target_parity_manifest.json b/packages/microcosm-build/src/microcosm/build/us/target_parity_manifest.json index 904f8bf512..376a1f7699 100644 --- a/packages/microcosm-build/src/microcosm/build/us/target_parity_manifest.json +++ b/packages/microcosm-build/src/microcosm/build/us/target_parity_manifest.json @@ -1,8 +1,8 @@ { "schema_version": 1, "reference": { - "feed": "consumer_facts_us_c5e5bf8.jsonl", - "feed_sha256": "b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801", + "feed": "consumer_facts_us_f98acf4.jsonl", + "feed_sha256": "a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c", "target_period": "2024", "registry_compile": "compile_us_fiscal_target_registry(age_targets=True, canonical CD vintage crosswalk) + apply_us_medicaid_enrollment_substitutions; one national + state + congressional-district surface for every artifact (microcosm#449)", "us_data_source": "retired us-data pipeline (archived): utils/loss.py (eCPS loss matrix), db/etl_national_targets.py, db/etl_*.py, utils/national_target_parity.py", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index 1cf1d0ec54..222a75a5af 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -10,6 +10,7 @@ from __future__ import annotations import json +import re from collections.abc import Iterable, Mapping from dataclasses import dataclass, replace from functools import cache @@ -28,6 +29,9 @@ from microcosm.build.us_runtime.congressional_district_vintage import ( translate_congressional_district_facts_to_current_vintage, ) +from microcosm.build.us_runtime.target_aging import ( + _period_year as _aging_period_year, +) from microcosm.build.us_runtime.target_aging import ( age_us_dollar_targets, enforce_period_contract, @@ -343,6 +347,18 @@ def _us_hierarchy_seed( "box_7_social_security_tips", } ) +# An IRS SOI table title ends with the tax year it is for ("..., Tax Year +# 2020"). The span is the year or years that follow "Tax Year(s)", so a title +# for several years ("Tax Years 2019 and 2020") is told apart from one year. +_TITLE_YEAR = r"(? int | None: + """The one tax year a source table's title names, else ``None``. + + ``None`` covers both a title that names no tax year and a title that names + more than one: neither says which year a cell of the table is for. + """ + + years = { + int(year) + for span in _SOURCE_TABLE_TAX_YEAR_SPAN.findall(source_table) + for year in re.findall(_TITLE_YEAR, span) + } + return years.pop() if len(years) == 1 else None + + +def _is_soi_form_w2_item_fact(fact: object) -> bool: + return ( + _str_at(fact, "layout", "groupby_dimension") + == _SOI_FORM_W2_ITEM_LAYOUT_DIMENSION + ) + + +def _w2_item_tax_year_mismatches(facts: Iterable[object]) -> tuple[str, ...]: + """Describe each Form W-2 item fact its source table does not back. + + A fact passes when its source table's title names exactly one tax year + and that is the year target aging reads from the fact's period; a + ``tax_year_`` source vintage must name it too. The result is + sorted, so it does not depend on feed order. + """ + + problems: list[str] = [] + for fact in facts: + if not _is_soi_form_w2_item_fact(fact): + continue + identity = _source_record_id(fact) or "a fact with no source_record_id" + source_table = _str_at(fact, "source", "source_table") + table_year = _source_table_tax_year(source_table) + if table_year is None: + problems.append( + f"{identity}: source table {source_table!r} names no single tax year" + ) + continue + stamped = [f"period {_period_value(fact)!r}"] + agrees = _aging_period_year(_period_value(fact)) == table_year + vintage = _str_at(fact, "source", "vintage") + vintage_year = _TAX_YEAR_VINTAGE.fullmatch(vintage) + if vintage_year is not None: + stamped.append(f"vintage {vintage!r}") + agrees = agrees and int(vintage_year.group(1)) == table_year + if not agrees: + problems.append( + f"{identity}: {' and '.join(stamped)}, but source table " + f"{source_table!r} is for tax year {table_year}" + ) + return tuple(sorted(problems)) + + +def _check_w2_item_fact_tax_years(facts: Iterable[object]) -> None: + """Refuse Form W-2 item facts stamped with a year their table is not for. + + Target aging starts from a fact's period, so a W-2 item fact stamped with + a later year than its table loses the growth in between. The feed pinned + at Chronicle c5e5bf8 stamped Table 4.B for Tax Year 2020 as ty2023, and + the Box 7 tips amount aged one year to 2024 where it needed four + (docs/us-chronicle-feed-repin.md). The check covers the three W-2 item + families (``irs_soi.form_w2_item`` layout): tips, 401(k) elective + deferrals and designated Roth contributions. + """ + + problems = _w2_item_tax_year_mismatches(facts) + if problems: + raise ValueError( + "Form W-2 item facts whose tax year disagrees with their source " + "table: " + + "; ".join(problems) + + ". A W-2 item fact's period, and its tax_year vintage, must be " + "the tax year its source table's title names. Rebuild the feed " + "from a Chronicle commit that stamps the published year " + "(docs/us-chronicle-feed-repin.md)." + ) + + def _zero_value_means_missing_for_latest_selection( reference: LedgerTargetReference, ) -> bool: diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_chronicle_feed.py b/packages/microcosm-build/tests/engine_free/us/test_us_chronicle_feed.py index 5b90126482..132b45b962 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_chronicle_feed.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_chronicle_feed.py @@ -13,6 +13,7 @@ import hashlib import importlib.util import json +import re import subprocess import sys import textwrap @@ -31,8 +32,8 @@ _TEST_PATHS = paths_for("microcosm-build") _ROOT = _TEST_PATHS.repository -_CHRONICLE_COMMIT = "c5e5bf8aa84960c1a200ee47303b19c953092d0f" -_FACTS_SHA256 = "b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801" +_CHRONICLE_COMMIT = "f98acf4edcc9488343446fda46db77914de8611f" +_FACTS_SHA256 = "a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c" _SCHEMA_SHA256 = "bdb51e2a8115634633ba7448c4005930fd9c0bfbade5e1b079b6bc24da485d3d" @@ -55,7 +56,7 @@ def test_pin_records_the_rebuilt_bare_feed() -> None: assert pin.source_repo == "PolicyEngine/chronicle" assert pin.source_commit == _CHRONICLE_COMMIT assert pin.scope == "us_fiscal_targets" - assert pin.fact_row_count == 39158 + assert pin.fact_row_count == 39155 assert pin.facts_sha256 == _FACTS_SHA256 assert pin.consumer_fact_schema_versions == ("chronicle.consumer_fact.v3",) assert pin.consumer_fact_schema_sha256 == _SCHEMA_SHA256 @@ -131,14 +132,33 @@ def test_parity_resources_and_generator_restate_the_pin() -> None: assert manifest["reference"]["reviewed_exclusions"] == "52" +def test_route_a_driver_restates_the_pin() -> None: + """The Route A driver refuses a build commit whose pin is not its + ``FEED_SHA``, so a re-pin that leaves the driver behind blocks every run. + Its example environment names the feed file the generator reads.""" + pin = load_us_chronicle_feed() + generator = _load_tool("build_us_target_parity_manifest") + driver = (_ROOT / "tools" / "route_a" / "route_a.sh").read_text(encoding="utf-8") + assert re.findall(r"(?m)^FEED_SHA=(\S+)$", driver) == [pin.facts_sha256] + (feed_row,) = re.findall(r"(?m)^release_ledger_facts\|.*$", driver) + assert f"{pin.fact_row_count:,} rows" in feed_row + assert f"Chronicle {pin.source_commit[:7]}" in feed_row + example = (_ROOT / "tools" / "route_a" / "route_a.env.example").read_text( + encoding="utf-8" + ) + assert re.findall(r"consumer_facts_us_\w+\.jsonl", example) == [ + generator.DEFAULT_FEED_NAME + ] + + def test_scope_names_every_pinned_pair_once() -> None: builder = _load_tool("build_us_chronicle_feed") scope_path = builder.default_scope_path() scope = builder.load_scope(scope_path) raw = scope["raw"] assert scope["commit"] == load_us_chronicle_feed().source_commit - assert len(scope["pairs"]) == 586 == raw["pair_count"] - assert sum(len(p) for p in scope["runs"].values()) == 62 + assert len(scope["pairs"]) == 585 == raw["pair_count"] + assert sum(len(p) for p in scope["runs"].values()) == 61 assert set(scope["runs"]) == { 2020, 2021, diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py index d9dfb311d3..559eaaa4d3 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_targets.py @@ -570,11 +570,11 @@ def fact(tax_year: int, source_record_id: str) -> dict[str, object]: def test_tips_return_count_is_excluded_at_every_vintage() -> None: """Decision d179: the #451 W-2 Box 7 return-count exclusion holds at every - vintage. The ty2020 fact carries the same 6,038,613 returns as the - excluded ty2023 row and used to calibrate in its place as a reviewed - bypass. Now no vintage compiles, the receipt lists each one under the - all-vintage rule, nothing is allowed through, and a sibling Table 1.4 row - still compiles as the control.""" + vintage. The entry is keyed to ty2020, the latest vintage IRS has + published; a later table's count would be dropped too. No vintage + compiles, the receipt lists each one under the all-vintage rule, nothing + is allowed through, and a sibling Table 1.4 row still compiles as the + control.""" tips_ids = [_W2_TIPS_RETURN_COUNT.format(year=year) for year in (2020, 2023, 2024)] control = "irs_soi.ty2023.table_1_4.all.taxable_interest_amount" facts = [ @@ -663,7 +663,9 @@ def test_all_vintage_scope_is_vintage_invariant_and_cell_exact() -> None: from hypothesis import strategies as st entries = sorted(US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS) - assert all(entry.split(".")[1] == "ty2023" for entry in entries) + # at_vintage swaps the second token, so it must be each entry's tax year + # (ty2023 for the Table 1.4 rows, ty2020 for the W-2 tips return count). + assert all(re.fullmatch(r"ty\d{4}", entry.split(".")[1]) for entry in entries) def at_vintage(entry: str, token: str) -> str: source, _, rest = entry.split(".", 2) @@ -705,7 +707,8 @@ def test_exclusion_vintage_scope_registers_are_consistent() -> None: vintage of exactly one register key that is not all-vintage scoped (else it could never be selected) and says why it may calibrate. The all-vintage set is the four #564 other-income rows plus the #451 tips return count - (decision d179), and no bypass remains.""" + (decision d179, keyed to the ty2020 id IRS published), and no bypass + remains.""" period_free = fiscal_targets._period_free_source_record_id assert US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS <= set( US_FISCAL_TARGET_SUPPORT_EXCLUSIONS @@ -715,7 +718,7 @@ def test_exclusion_vintage_scope_registers_are_consistent() -> None: f"irs_soi.ty2023.table_1_4.all.{measure}" for measure in _OTHER_INCOME_TABLE_1_4_MEASURES ), - _W2_TIPS_RETURN_COUNT.format(year=2023), + _W2_TIPS_RETURN_COUNT.format(year=2020), } assert US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES == {} for source_record_id, reason in US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES.items(): @@ -860,8 +863,8 @@ def check(years: list[int], reviewed: bool) -> None: def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> None: """On the pinned feed the receipt names the 40 M-CHIP CHIP ids (20 states x 2 CMS months), the 16 other-income ids (ty2020-ty2023) and the ty2020 - and ty2023 tips return counts, and allows no vintage bypass (decision - d179).""" + tips return count, the only W-2 vintage the feed carries, and allows no + vintage bypass (decision d179).""" _, _, receipt = pinned_feed_national_state_surface rules = receipt["rules"] m_chip_ids = rules["m_chip_state_chip_enrollment"]["source_record_ids"] @@ -882,7 +885,6 @@ def test_pinned_feed_exclusion_receipt(pinned_feed_national_state_surface) -> No for measure in _OTHER_INCOME_TABLE_1_4_MEASURES ), _W2_TIPS_RETURN_COUNT.format(year=2020), - _W2_TIPS_RETURN_COUNT.format(year=2023), ] ) assert rules["allowlisted_vintage_bypass"]["source_record_ids"] == [] @@ -893,15 +895,17 @@ def test_pinned_feed_national_state_surface_restores_the_fences( ) -> None: """The release surface on the pinned feed: 32,842 compiled targets after Medicaid substitution, 5,694 national_state targets at registry - d315c75804ef, 32 CHIP rows none of them for an M-CHIP state, no + 61a081ae55bb, 32 CHIP rows none of them for an M-CHIP state, no other-income row and no tips return count. Before microcosm#956 it was 32,867 / 5,719 at d5f9d854fe11, and before decision d179 dropped the - ty2020 tips return count it was 32,843 / 5,695 at 386fac439e77 + ty2020 tips return count it was 32,843 / 5,695 at 386fac439e77. The + registry was d315c75804ef while the feed stamped the W-2 tips amount + ty2023; the same 5,694 targets now carry it as ty2020 (docs/us-chronicle-feed-repin.md).""" registry, surface, _ = pinned_feed_national_state_surface assert len(registry.specs) == 32_842 assert len(surface.specs) == 5_694 - assert surface.version == "d315c75804ef" + assert surface.version == "61a081ae55bb" chip = [ spec for spec in surface.specs @@ -923,7 +927,7 @@ def test_pinned_feed_national_state_surface_restores_the_fences( # The fence drops the return count only: the tips amount row of the same # record set still calibrates. assert ( - "irs_soi.ty2023.form_w2_social_security_tips.box_7_social_security_tips.amount" + "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.amount" in {spec.name for spec in surface.specs} ) @@ -4448,7 +4452,7 @@ def test_soi_form_w2_social_security_tips_return_count_targets_tip_income( "irs_soi.ty2024.form_w2_social_security_tips." "box_7_social_security_tips.return_count" ) - # Decision d179 drops every vintage of this cell (the #451 ty2023 + # Decision d179 drops every vintage of this cell (the #451 ty2020 # exclusion is all-vintage scoped). This test pins only the variable # mapping, so it lifts that scope and allowlists its own id, which the # compile would otherwise refuse as another vintage of the exclusion. @@ -4456,7 +4460,7 @@ def test_soi_form_w2_social_security_tips_return_count_targets_tip_income( fiscal_targets, "US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS", US_FISCAL_TARGET_ALL_VINTAGE_SUPPORT_EXCLUSIONS - - {_W2_TIPS_RETURN_COUNT.format(year=2023)}, + - {_W2_TIPS_RETURN_COUNT.format(year=2020)}, ) monkeypatch.setitem( US_FISCAL_TARGET_EXCLUSION_VINTAGE_BYPASSES, @@ -4466,16 +4470,7 @@ def test_soi_form_w2_social_security_tips_return_count_targets_tip_income( registry = compile_us_fiscal_target_registry( [ *packaged_reference_facts(), - _dynamic_ledger_fact( - source_record_id=source_record_id, - source_name="irs_soi", - measure_id="return_count", - value=6_038_613, - period_value=2024, - layout_record_set_id="irs_soi.ty2024.form_w2_social_security_tips", - groupby_dimension="irs_soi.form_w2_item", - groupby_value_id="box_7_social_security_tips", - ), + _w2_tips_return_count_fact(2024), ], allow_unaged_dollar_targets=True, ) @@ -4499,16 +4494,7 @@ def test_soi_form_w2_social_security_tips_amount_targets_tip_income() -> None: registry = compile_us_fiscal_target_registry( [ *packaged_reference_facts(), - _dynamic_ledger_fact( - source_record_id=source_record_id, - source_name="irs_soi", - measure_id="amount", - value=26_786_522_000, - period_value=2024, - layout_record_set_id="irs_soi.ty2024.form_w2_social_security_tips", - groupby_dimension="irs_soi.form_w2_item", - groupby_value_id="box_7_social_security_tips", - ), + _w2_item_fact(2024), ], allow_unaged_dollar_targets=True, ) @@ -4531,16 +4517,7 @@ def test_age_targets_chains_w2_tips_through_soi_wages_bridge() -> None: ) facts = [ *packaged_reference_facts(), - _dynamic_ledger_fact( - source_record_id=source_record_id, - source_name="irs_soi", - measure_id="amount", - value=26_786_522_000, - period_value=2020, - layout_record_set_id="irs_soi.ty2020.form_w2_social_security_tips", - groupby_dimension="irs_soi.form_w2_item", - groupby_value_id="box_7_social_security_tips", - ), + _w2_item_fact(2020), _dynamic_ledger_fact( source_record_id="irs_soi.ty2020.table_1_4.all.wages_salaries_amount", source_name="irs_soi", diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_w2_item_tax_year.py b/packages/microcosm-build/tests/engine_free/us/test_us_w2_item_tax_year.py new file mode 100644 index 0000000000..f30325aee4 --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/us/test_us_w2_item_tax_year.py @@ -0,0 +1,462 @@ +"""Form W-2 item facts carry the tax year their source table is for. + +IRS SOI Table 4.B (Form W-2 items) is published for one tax year per workbook +and says so in its title. Target aging starts from a fact's period, so a W-2 +item fact stamped with a later year than its table loses the growth in +between. The feed pinned at Chronicle c5e5bf8 stamped the Tax Year 2020 table +as ty2023, and the Box 7 tips target aged one year to 2024 where it needed +four (docs/us-chronicle-feed-repin.md). + +Invariants under test, for the three W-2 item families (tips, 401(k) elective +deferrals, designated Roth contributions): + +- the compile and the exclusion receipt refuse a W-2 item fact unless its + source table's title names exactly one tax year and that year is the fact's + period and its ``tax_year_`` vintage; +- facts outside those families are never refused by this check; +- the verdict does not depend on feed order; +- an honest fact and its restamped twin age differently by exactly the growth + the restamp skips (the differential that makes the refusal matter). +""" + +# ruff: noqa: F403, F405 +from test_support.microcosm_build.us_fiscal_targets import * +from test_support.microcosm_build.us_fiscal_targets import ( + _W2_TABLE_4B_TITLE, + _cbo_income_source_projection_fact, + _dynamic_ledger_fact, + _load_repo_tool, + _soi_taxable_interest_fact, + _w2_item_fact, +) + +_W2_ITEM_FAMILIES = ( + ("form_w2_social_security_tips", "box_7_social_security_tips", "amount"), + ("form_w2_social_security_tips", "box_7_social_security_tips", "return_count"), + ("form_w2_social_security_tips", "box_7_social_security_tips", "taxpayer_count"), + ("form_w2_401k_elective_deferrals", "box_12_d_401k_elective_deferrals", "amount"), + ( + "form_w2_designated_roth_401k_contributions", + "box_12_aa_designated_roth_401k_contributions", + "amount", + ), +) +# What IRS printed in Table 4.B for Tax Year 2020 (20in04w2all.xlsx, sheet +# "Table 4.B", rows 13, 22 and 41; money amounts are in thousands), read from +# the workbook on 2026-10-10. +_PUBLISHED_TY2020 = { + "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.return_count": 6_038_613, + "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.taxpayer_count": 6_105_713, + "irs_soi.ty2020.form_w2_social_security_tips.box_7_social_security_tips.amount": 26_786_522_000, + "irs_soi.ty2020.form_w2_401k_elective_deferrals.box_12_d_401k_elective_deferrals.amount": 277_859_181_000, + "irs_soi.ty2020.form_w2_designated_roth_401k_contributions.box_12_aa_designated_roth_401k_contributions.amount": 32_302_509_000, +} +_TIPS_AMOUNT = ( + "irs_soi.ty{year}.form_w2_social_security_tips.box_7_social_security_tips.amount" +) + + +# --- the title reader -------------------------------------------------------- + + +@pytest.mark.parametrize( + ("title", "expected"), + [ + (_W2_TABLE_4B_TITLE.format(year=2020), 2020), + ("Table 4.B, Tax Year 2020 (revised January 2026)", 2020), + ("Table 1, Tax Year 2020, and Table 2, Tax Year 2020", 2020), + ("table 4.b, tax year\n2019", 2019), + ("Individual returns, Tax Years 2019 and 2020", None), + ("Individual returns, Tax Years 2019-2020", None), + ("Individual returns, tax years 2019, 2020 and 2021", None), + ("Individual returns, Tax Year 2020 through 2022", None), + ("Table 1, Tax Year 2019, and Table 2, Tax Year 2020", None), + ("SPI Tables 3.1 to 3.11, tax year 2023-24", None), + ("Tax Year 20201", None), + ("Estimates of Federal Tax Expenditures for Fiscal Years 2024-2028", None), + ("Calendar Years 1987-2024", None), + ("2020 Form W-2 statistics", None), + ("irs_soi table", None), + ("", None), + ], +) +def test_source_table_tax_year_reads_irs_titles(title: str, expected) -> None: + assert fiscal_targets._source_table_tax_year(title) == expected + + +def test_source_table_tax_year_properties() -> None: + """One named year reads back whatever surrounds it; a title that names + two different years, or no tax year, never reads as one.""" + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + reader = fiscal_targets._source_table_tax_year + years = st.integers(1900, 2099) + # Surrounding text with no digits, so it can name no other year. + filler = st.text( + alphabet=st.characters( + whitelist_categories=("Lu", "Ll", "Zs", "Po", "Pd", "Ps", "Pe") + ), + max_size=30, + ).filter(lambda text: "tax year" not in " ".join(text.lower().split())) + connectors = st.sampled_from( + [" and ", "-", "–", "—", " to ", " through ", ", ", " & "] + ) + markers = st.sampled_from(["Tax Year", "tax year", "TAX YEAR", "Tax Year"]) + + @settings(max_examples=400, deadline=None) + @given(prefix=filler, suffix=filler, year=years, marker=markers) + def single_year_reads_back(prefix, suffix, year, marker) -> None: + # A suffix opening with a dash or slash then a digit would abbreviate a + # second year; the filler has no digits, so it cannot. + assert reader(f"{prefix} {marker} {year}{suffix}") == year + + @settings(max_examples=400, deadline=None) + @given( + prefix=filler, + suffix=filler, + first=years, + second=years, + connector=connectors, + ) + def two_years_never_read_as_one(prefix, suffix, first, second, connector) -> None: + title = f"{prefix} Tax Years {first}{connector}{second}{suffix}" + assert reader(title) == (first if first == second else None) + + @settings(max_examples=200, deadline=None) + @given(text=filler, year=years) + def no_marker_means_no_tax_year(text, year) -> None: + assert reader(text) is None + assert reader(f"{text} Calendar Year {year}") is None + assert reader(f"{year} {text}") is None + + single_year_reads_back() + two_years_never_read_as_one() + no_marker_means_no_tax_year() + + +# --- the mismatch finder ----------------------------------------------------- + + +def test_w2_item_mismatches_flag_exactly_the_unbacked_facts() -> None: + """Property: over generated feeds, the finder flags a fact if and only if + it is a W-2 item fact whose title names no single tax year, or names one + that is not its period or its tax_year vintage. The oracle is restated + here from the fact's fields, not from the finder.""" + pytest.importorskip("hypothesis") + from hypothesis import given, settings + from hypothesis import strategies as st + + years = st.integers(2015, 2026) + titles = st.one_of( + years.map(lambda year: (_W2_TABLE_4B_TITLE.format(year=year), year)), + st.sampled_from( + [ + ("irs_soi table", None), + ("Form W-2 items, Tax Years 2019 and 2020", None), + ("Form W-2 items, tax year 2020-21", None), + ("", None), + ] + ), + ) + + @st.composite + def facts_with_verdicts(draw): + facts, expected = [], set() + for index in range(draw(st.integers(0, 8))): + record_set, item, measure_id = draw(st.sampled_from(_W2_ITEM_FAMILIES)) + period = draw(years) + title, table_year = draw(titles) + vintage_year = draw(st.one_of(st.just(period), years)) + vintage = draw( + st.sampled_from([f"tax_year_{vintage_year}", str(vintage_year), ""]) + ) + is_w2_item = draw(st.booleans()) + fact = _w2_item_fact( + period, + record_set=record_set, + item=item, + measure_id=measure_id, + source_table=title, + source_vintage=vintage, + ) + source_record_id = f"{fact['lineage']['source_record_id']}.{index}" + fact["lineage"]["source_record_id"] = source_record_id + if not is_w2_item: + fact["layout"]["groupby_dimension"] = "us:statutes/26/62#agi" + backed = ( + table_year is not None + and period == table_year + and (not vintage.startswith("tax_year_") or vintage_year == table_year) + ) + if is_w2_item and not backed: + expected.add(source_record_id) + facts.append(fact) + return facts, expected, draw(st.randoms(use_true_random=False)) + + @settings(max_examples=300, deadline=None) + @given(case=facts_with_verdicts()) + def check(case) -> None: + facts, expected, rng = case + problems = fiscal_targets._w2_item_tax_year_mismatches(facts) + flagged = {problem.split(": ", 1)[0] for problem in problems} + assert flagged == expected + assert len(problems) == len(expected) + shuffled = list(facts) + rng.shuffle(shuffled) + assert fiscal_targets._w2_item_tax_year_mismatches(shuffled) == problems + if expected: + with pytest.raises(ValueError, match="Form W-2 item facts whose tax year"): + fiscal_targets._check_w2_item_fact_tax_years(facts) + else: + fiscal_targets._check_w2_item_fact_tax_years(facts) + + check() + + +def test_mismatch_finder_reads_the_year_target_aging_reads() -> None: + """The period is compared as the year target aging starts from, so a + labelled or string period that ages from the table's year passes.""" + for period in (2020, "2020", "tax_year_2020", "ty2020"): + fact = _w2_item_fact(2020) + fact["period"]["value"] = period + assert fiscal_targets._w2_item_tax_year_mismatches([fact]) == () + for period in (2023, "2023", "ty2023", None, "all_years"): + fact = _w2_item_fact(2020) + fact["period"]["value"] = period + assert len(fiscal_targets._w2_item_tax_year_mismatches([fact])) == 1 + + +# --- the compile ------------------------------------------------------------- + + +@pytest.mark.parametrize(("record_set", "item", "measure_id"), _W2_ITEM_FAMILIES) +def test_compile_refuses_a_w2_item_fact_stamped_with_another_tax_year( + record_set: str, item: str, measure_id: str +) -> None: + """The pinned-feed shape: a Tax Year 2020 cell with ty2023 labels. The + compile and the receipt both refuse it and name the fact; its honest + TY2020 twin compiles.""" + restamped = _w2_item_fact( + 2023, record_set=record_set, item=item, measure_id=measure_id, table_year=2020 + ) + source_record_id = restamped["lineage"]["source_record_id"] + assert source_record_id.startswith("irs_soi.ty2023.") + message = ( + f"{source_record_id}: period 2023 and vintage 'tax_year_2023', but source " + f"table {_W2_TABLE_4B_TITLE.format(year=2020)!r} is for tax year 2020" + ) + facts = [*packaged_reference_facts(), restamped] + with pytest.raises(ValueError, match=re.escape(message)): + compile_us_fiscal_target_registry(facts, allow_unaged_dollar_targets=True) + with pytest.raises(ValueError, match=re.escape(message)): + us_fiscal_target_exclusion_receipt(facts) + + honest = _w2_item_fact( + 2020, record_set=record_set, item=item, measure_id=measure_id + ) + compile_us_fiscal_target_registry( + [*packaged_reference_facts(), honest], allow_unaged_dollar_targets=True + ) + + +def test_compile_refuses_a_w2_item_fact_whose_table_names_no_tax_year() -> None: + """Without a year in the title the period cannot be checked, so the fact + is refused, not waved through.""" + for title in ("irs_soi table", "Form W-2 items, Tax Years 2019 and 2020", ""): + fact = _w2_item_fact(2020, source_table=title) + with pytest.raises(ValueError, match="names no single tax year"): + compile_us_fiscal_target_registry( + [*packaged_reference_facts(), fact], allow_unaged_dollar_targets=True + ) + + +def test_compile_refuses_a_w2_item_vintage_that_disagrees_with_the_table() -> None: + fact = _w2_item_fact(2020, source_vintage="tax_year_2023") + with pytest.raises( + ValueError, match=re.escape("period 2020 and vintage 'tax_year_2023'") + ): + compile_us_fiscal_target_registry( + [*packaged_reference_facts(), fact], allow_unaged_dollar_targets=True + ) + + +def test_check_leaves_other_families_alone() -> None: + """The check is scoped to the W-2 item layout: another SOI fact whose + table title names a different year than its period still compiles (a + table for one year can carry a prior-year column).""" + control = "irs_soi.ty2023.table_1_4.all.taxable_interest_amount" + fact = _soi_taxable_interest_fact( + 2023, + source_record_id=control, + value=240_000_000_000, + layout_record_set_id="irs_soi.ty2023.table_1_4", + ) + fact["source"]["source_table"] = "Table 1.4. All Returns, Tax Year 2020" + registry = compile_us_fiscal_target_registry( + [*packaged_reference_facts(), fact], allow_unaged_dollar_targets=True + ) + assert control in { + spec.metadata["ledger_source_record_id"] for spec in registry.specs + } + + +def _tips_aging_facts(tips_fact: dict[str, object]) -> list[dict[str, object]]: + wages = {2020: 8_416_495_535_000, 2023: 10_000_000_000_000} + return [ + *packaged_reference_facts(), + tips_fact, + *( + _dynamic_ledger_fact( + source_record_id=f"irs_soi.ty{year}.table_1_4.all.wages_salaries_amount", + source_name="irs_soi", + measure_id="wages_salaries_amount", + value=value, + period_value=year, + dimensions={"income_range": "all", "filing_status": "all"}, + layout_record_set_id=f"irs_soi.ty{year}.table_1_4", + groupby_dimension="us:statutes/26/62#adjusted_gross_income", + groupby_value_id="all", + ) + for year, value in wages.items() + ), + _cbo_income_source_projection_fact( + 2023, "wages_and_salaries", value=10_200_000_000_000 + ), + _cbo_income_source_projection_fact( + 2024, "wages_and_salaries", value=10_700_000_000_000 + ), + ] + + +def test_restamped_tips_fact_would_age_one_year_where_the_table_needs_four( + monkeypatch, +) -> None: + """Differential: what the refusal prevents. With the check lifted, a + TY2020 tips cell stamped ty2023 ages by the CBO 2023-to-2024 ratio alone; + its honest twin also gets the SOI wage growth from 2020 to 2023. The two + targets differ by exactly that growth.""" + soi_growth = 10_000_000_000_000 / 8_416_495_535_000 + cbo_growth = 10_700_000_000_000 / 10_200_000_000_000 + + def tips_target(fact: dict[str, object]): + registry = compile_us_fiscal_target_registry( + _tips_aging_facts(fact), + target_period=2024, + age_targets=True, + allow_unaged_dollar_targets=True, + ) + (spec,) = ( + spec + for spec in registry.specs + if spec.metadata.get("target_role") == "w2_social_security_tips_total" + ) + return spec + + honest = tips_target(_w2_item_fact(2020)) + assert honest.metadata["source_period"] == "2020" + assert honest.metadata["aging_factor_source"].startswith("chained:") + assert honest.value == pytest.approx(26_786_522_000 * soi_growth * cbo_growth) + + restamped_fact = _w2_item_fact(2023, table_year=2020) + with pytest.raises(ValueError, match="is for tax year 2020"): + tips_target(restamped_fact) + monkeypatch.setattr( + fiscal_targets, "_check_w2_item_fact_tax_years", lambda facts: None + ) + restamped = tips_target(restamped_fact) + assert restamped.metadata["source_period"] == "2023" + assert not restamped.metadata["aging_factor_source"].startswith("chained:") + assert restamped.value == pytest.approx(26_786_522_000 * cbo_growth) + assert honest.value / restamped.value == pytest.approx(soi_growth) + + +# --- the pinned feed --------------------------------------------------------- +# These run only where the pinned Chronicle feed sits at the target-parity +# generator's default path; CI has no feed and skips them, so the compile-time +# check above is the enforcement and these pin the outcome. + + +@pytest.fixture(scope="module") +def pinned_feed_facts(): + feed_path = _load_repo_tool("build_us_target_parity_manifest").DEFAULT_FEED_PATH + if not feed_path.exists(): + pytest.skip(f"pinned feed not present at {feed_path}") + from microcosm.build.ledger_artifact import load_ledger_consumer_artifact + from microcosm.build.us_runtime.chronicle_feed import load_us_chronicle_feed + + return load_ledger_consumer_artifact( + feed_path, + expected_facts_sha256=load_us_chronicle_feed().facts_sha256, + expected_manifest_sha256=None, + ).facts + + +def test_pinned_feed_w2_item_facts_are_the_published_tax_year_2020_cells( + pinned_feed_facts, +) -> None: + """Registry test: every W-2 item fact in the pinned feed is one of the + five Table 4.B cells IRS published for Tax Year 2020, stamped 2020 in its + period, vintage, record id and record set, with the published value.""" + w2_facts = [ + fact + for fact in pinned_feed_facts + if fact["layout"].get("groupby_dimension") == "irs_soi.form_w2_item" + ] + assert { + fact["lineage"]["source_record_id"]: fact["value"] for fact in w2_facts + } == _PUBLISHED_TY2020 + assert len(w2_facts) == len(_PUBLISHED_TY2020) + assert fiscal_targets._w2_item_tax_year_mismatches(pinned_feed_facts) == () + for fact in w2_facts: + assert fact["period"] == {"type": "tax_year", "value": 2020} + assert fact["source"]["vintage"] == "tax_year_2020" + assert fact["source"]["source_table"] == _W2_TABLE_4B_TITLE.format(year=2020) + assert fact["source"]["source_file"] == "20in04w2all.xlsx" + assert fact["layout"]["record_set_id"].startswith("irs_soi.ty2020.form_w2_") + # No fact anywhere in the feed still reads that workbook under another year. + assert not [ + fact["lineage"]["source_record_id"] + for fact in pinned_feed_facts + if fact["source"].get("source_file") == "20in04w2all.xlsx" + and fact["period"]["value"] != 2020 + ] + + +def test_pinned_feed_tips_target_ages_from_tax_year_2020( + pinned_feed_facts, pinned_feed_national_state_surface +) -> None: + """Differential: the compiled Box 7 tips target equals the published + TY2020 amount times a factor recomputed here from four feed facts, the + SOI Table 1.4 wage totals for 2020 and 2023 and the CBO wage projections + for 2023 and 2024. It was $28,280,884,269 while the feed stamped the cell + ty2023 and aged it on the CBO ratio alone.""" + by_id = {fact["lineage"]["source_record_id"]: fact for fact in pinned_feed_facts} + soi_2020 = by_id["irs_soi.ty2020.table_1_4.all.wages_salaries_amount"]["value"] + soi_2023 = by_id["irs_soi.ty2023.table_1_4.all.wages_salaries_amount"]["value"] + cbo = "cbo.revenue_projection.ty{year}.income_by_source.wages_and_salaries.projected_amount" + cbo_2023 = by_id[cbo.format(year=2023)]["value"] + cbo_2024 = by_id[cbo.format(year=2024)]["value"] + expected = 26_786_522_000 * (soi_2023 / soi_2020) * (cbo_2024 / cbo_2023) + + registry, surface, _ = pinned_feed_national_state_surface + w2_specs = [spec for spec in registry.specs if "form_w2" in spec.name] + assert [spec.name for spec in w2_specs] == [_TIPS_AMOUNT.format(year=2020)] + (tips,) = w2_specs + assert tips.name in {spec.name for spec in surface.specs} + assert tips.value == pytest.approx(expected, rel=1e-12) + assert tips.value == pytest.approx(34_287_530_778.93, abs=0.01) + assert tips.metadata["source_period"] == "2020" + assert tips.metadata["aged_to"] == "2024" + assert float(tips.metadata["aging_factor"]) == pytest.approx( + 1.28002921689246, rel=1e-12 + ) + assert tips.metadata["aging_factor_source"] == ( + "chained:irs_soi.ty2023.table_1_4.all.wages_salaries_amount+" + + cbo.format(year=2024) + ) + # What the ty2023 stamp cost: the 2020-to-2023 wage growth. + assert tips.value / (26_786_522_000 * cbo_2024 / cbo_2023) == pytest.approx( + soi_2023 / soi_2020 + ) diff --git a/test_support/microcosm_build/us_fiscal_targets.py b/test_support/microcosm_build/us_fiscal_targets.py index 7f83264f18..264a3e71b1 100644 --- a/test_support/microcosm_build/us_fiscal_targets.py +++ b/test_support/microcosm_build/us_fiscal_targets.py @@ -145,21 +145,55 @@ def _other_income_table_1_4_facts(tax_years) -> list[dict[str, object]]: "irs_soi.ty{year}.form_w2_social_security_tips." "box_7_social_security_tips.return_count" ) +# IRS titles the Form W-2 item table with its tax year; the compile refuses a +# W-2 item fact whose period is not that year. +_W2_TABLE_4B_TITLE = ( + "Table 4.B. Summary of Items for Taxpayers with Form W-2, by Return and " + "Earner Type, Tax Year {year}" +) -def _w2_tips_return_count_fact(tax_year: int) -> dict[str, object]: +def _w2_item_fact( + tax_year: int, + *, + record_set: str = "form_w2_social_security_tips", + item: str = "box_7_social_security_tips", + measure_id: str = "amount", + value: float = 26_786_522_000, + table_year: int | None = None, + source_table: str | None = None, + source_vintage: str | None = None, +) -> dict[str, object]: + """A Form W-2 item fact stamped ``tax_year``. + + ``table_year`` is the year its source table is for (default: the same + year, an honest fact); ``source_table`` overrides the whole title. + """ + table_year = tax_year if table_year is None else table_year return _dynamic_ledger_fact( - source_record_id=_W2_TIPS_RETURN_COUNT.format(year=tax_year), + source_record_id=f"irs_soi.ty{tax_year}.{record_set}.{item}.{measure_id}", source_name="irs_soi", - measure_id="return_count", - value=6_038_613, + measure_id=measure_id, + value=value, period_value=tax_year, - layout_record_set_id=f"irs_soi.ty{tax_year}.form_w2_social_security_tips", + layout_record_set_id=f"irs_soi.ty{tax_year}.{record_set}", groupby_dimension="irs_soi.form_w2_item", - groupby_value_id="box_7_social_security_tips", + groupby_value_id=item, + source_table=( + _W2_TABLE_4B_TITLE.format(year=table_year) + if source_table is None + else source_table + ), + source_vintage=( + f"tax_year_{tax_year}" if source_vintage is None else source_vintage + ), ) +def _w2_tips_return_count_fact(tax_year: int) -> dict[str, object]: + return _w2_item_fact(tax_year, measure_id="return_count", value=6_038_613) + + def _four_rule_synthetic_feed(monkeypatch): """Synthetic facts that exercise every receipt rule, and each rule's ids. @@ -1085,8 +1119,11 @@ def _dynamic_ledger_fact( layout_record_set_id: str | None = None, dimensions: dict[str, object] | None = None, universe_constraints: list[dict[str, object]] | None = None, + source_table: str | None = None, + source_vintage: str | None = None, ) -> dict[str, object]: fact_id = _fact_id(source_record_id, period_value) + source_table = f"{source_name} table" if source_table is None else source_table fact_dimensions = dict(dimensions or {}) hierarchy_dimensions = dict(fact_dimensions) if groupby_dimension and groupby_value_id: @@ -1135,15 +1172,17 @@ def _dynamic_ledger_fact( }, "observed_measure": { "source_name": source_name, - "source_table": f"{source_name} table", + "source_table": source_table, "source_measure_id": measure_id, "source_concept": measure_id, "unit": "usd", }, "source": { "source_name": source_name, - "source_table": f"{source_name} table", - "vintage": str(period_value), + "source_table": source_table, + "vintage": ( + str(period_value) if source_vintage is None else source_vintage + ), "url": f"https://example.org/{fact_id}", }, } diff --git a/tools/build_us_target_parity_manifest.py b/tools/build_us_target_parity_manifest.py index 57d3238ba9..2527bfc764 100644 --- a/tools/build_us_target_parity_manifest.py +++ b/tools/build_us_target_parity_manifest.py @@ -61,7 +61,7 @@ MANIFEST_PATH = US_PACKAGE_DIR / "target_parity_manifest.json" FEED_FAMILIES_PATH = US_PACKAGE_DIR / "target_parity_feed_families.json" -DEFAULT_FEED_NAME = "consumer_facts_us_c5e5bf8.jsonl" +DEFAULT_FEED_NAME = "consumer_facts_us_f98acf4.jsonl" DEFAULT_FEED_PATH = ( Path.home() / "PolicyEngine" / "_buildh-runtime" / "inputs" / DEFAULT_FEED_NAME ) diff --git a/tools/route_a/route_a.env.example b/tools/route_a/route_a.env.example index e9b3250457..c14388b9b0 100644 --- a/tools/route_a/route_a.env.example +++ b/tools/route_a/route_a.env.example @@ -15,7 +15,7 @@ : "${AGENT_SECRET:=$HOME/bin/agent-secret}" : "${STORAGE:=$PE/policyengine-us-data/policyengine_us_data/storage}" : "${EDU:=$PE/_buildm-runtime/inputs/asec_education}" -: "${FEED:=$PE/_buildh-runtime/inputs/consumer_facts_us_c5e5bf8.jsonl}" +: "${FEED:=$PE/_buildh-runtime/inputs/consumer_facts_us_f98acf4.jsonl}" # d713: CT-fixed ladder; route_a.sh verifies its digest and size. : "${LADDER:=$PE/_build_artifacts/us-ct-cbsa/us_block_ladder_2020.npz}" : "${SSI:=$PE/_buildo-runtime/inputs/attempt6_basis_schema3_seed.json}" diff --git a/tools/route_a/route_a.sh b/tools/route_a/route_a.sh index 79dc61c704..205e20a1b9 100755 --- a/tools/route_a/route_a.sh +++ b/tools/route_a/route_a.sh @@ -50,13 +50,15 @@ LOG=$RUN_ROOT/route_a.log # Inputs, resolved and hashed on this machine on 2026-09-23 (see status.md). # Format: role|path|sha256|bytes|where main pins it # --------------------------------------------------------------------------- -FEED_SHA=b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801 +FEED_SHA=a81cbcc504caa71e4eb23b4d70a11528c8e6db6523e41283f8482569a23df04c # The base stage reads the feed only for the SOI congressional-district return # counts. On 2026-09-23 the pinned feed and the 2026-09-16 base feed # (consumer_facts_builde_aging_v5.jsonl, a5d34d4a...) gave the same 436-row # distribution through main's congressional_district_distribution_from_ledger_facts; # after the vintage crosswalk, 73 weights differ by at most 2.2e-16 relative. # The pinned feed is used for both stages so the lineage has one feed identity. +# The feed pinned since the W-2 relabel (Chronicle f98acf4) differs from that +# one only in its Form W-2 item rows, so every district row is the same bytes. BASE_LEDGER_FACTS=${BASE_LEDGER_FACTS:-$FEED} BASE_LEDGER_FACTS_SHA=${BASE_LEDGER_FACTS_SHA:-$FEED_SHA} # d713 (Max 2026-10-03): the next certified build uses the Connecticut-fixed ladder from microcosm#1072 @@ -96,7 +98,7 @@ asec_education_2023_zip|$EDU/asecpub24csv.zip|cdb39cdac34bef99dd0940ab28e306f692 asec_education_2024_zip|$EDU/asecpub25csv.zip|318845a2b5e0034eb2973898de1738f4df0025727de38499e7669cb9c0deef0b|147271429|us_runtime/education_assistance_source.py base_ledger_facts|$BASE_LEDGER_FACTS|$BASE_LEDGER_FACTS_SHA|-|us/chronicle_feed.json when it is the pinned feed block_ladder_npz|$LADDER|6840b990acdfa2003d7723da5595e3cfff7205a4fa3d8d6206ed835c85c3f233|18991218|d713: Connecticut-fixed ladder rebuilt at microcosm#1072 (af98853e); _build_artifacts/us-ct-cbsa/verify_vs_route_a.json -release_ledger_facts|$FEED|$FEED_SHA|164603204|us/chronicle_feed.json facts_sha256 (Chronicle c5e5bf8, bare feed, 39,158 rows) +release_ledger_facts|$FEED|$FEED_SHA|164591409|us/chronicle_feed.json facts_sha256 (Chronicle f98acf4 = c5e5bf8 plus the W-2 items' TY2020 labels, bare feed, 39,155 rows) ${TAIL:+qrf_tail_exclusions|$TAIL|${QRF_TAIL_EXCLUSIONS_SHA:-9bd497dc0d03dd793b53a228979e99408e62fdd2430bdd0d04df5fc7576e6229}|-|operator-supplied register; the release records its sha in diagnostics} ssi_take_up_prior_basis|$SSI|$SSI_SHA|4782|--ssi-take-up-prior-weight-basis-sha256 scf_summary_extract|$SCF|6b8dd2d935a76ed225ddebc80fb2db22a467f0c80d9a1acaa67b4584aa4bafd1|24904185|us/source_stages.json, us_runtime/scf_wealth.py