Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/719-work-experience-columns.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
US release person records carry `detailed_industry_recode` (CPS `WEIND`), `major_industry_recode` (`WEMIND`) and `worked_last_year` (`WKSWORK > 0`) next to `detailed_occupation_recode`: ASEC vintages restore `WEIND`/`WEMIND` by exact `PERIDNUM` join before pooling, and the `org_wages` carry checks their industry identities. Both the release predictor join and native ACS input mapper reuse the same `INDP`/`WKWN` crosswalk (version 2). ACS local staging and packaging refuse incomplete canonical job inputs on donor or final person rows (#719).
13 changes: 13 additions & 0 deletions docs/agent-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,14 @@ module or dependency. CI tests the merge ref, so merge main and re-pin rather
than hunting for an environment leak. Editable installs hide packaging breaks;
if you touch packaging, build wheels locally before pushing.

For an intentional US source-stage change, regenerate the release input
coverage manifest with `tools/build_us_release_input_coverage_manifest.py`,
then follow the bundle/pin/coverage sequence documented in
`tools/repin_us_spec_engine_contracts.py`. The repinner derives frozen source
hashes, compiler field inventories and their test expectations; its `--check`
mode refuses drift. Full bundle generation needs the US engine environment.
These commands do not rebuild survey data or certify a release.

The `integration-uk` job in `.github/workflows/test.yml` runs the
`integration/uk/` directory serially on Python 3.13 whenever the unified
changed-path selection includes the UK, and on every push to `main`. It skips
Expand Down Expand Up @@ -335,6 +343,11 @@ and the ACS local-area `stored_inputs` gate, each naming the built-with engine
on which an SPM unit the built-with engine measures has no classified adult,
and binds the `spm_measurement_composition` gate the publisher requires
(microcosm#1263; `microcosm.build.us_runtime.spm_measurement_gate`).
The same package stage requires complete occupation, industry and
worked-last-year inputs on every final person row and binds that verdict to
the H5 hash. ACS staging checks its preserved ASEC donor before acquisition;
historical donors and staging files missing these inputs must be rebuilt
through the exact ASEC restoration/fiscal carry and native ACS mapping.
US household ladders carry both `county_fips` (2020 counties for the locked
policyengine-us rules) and `county_fips_2023` (3,144 county-equivalents for
calibration, including Connecticut's nine planning regions). The latter is
Expand Down
50 changes: 25 additions & 25 deletions docs/evidence/spec-engine/us-f0-coverage.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
"version": 1
},
"country": "us",
"documentation_sha256": "44f40b4e8aaa2e6be67a7bc9680cc29f88e185c10573ffb9c3f9dfbd3ad1081f",
"documentation_sha256": "801a99da54b8a4834a8c5e6c4b013fdae1e7576e0132de7e87cb1ef098f1ad8c",
"field_usage": {
"authored_normative_field_count": 32437,
"authored_normative_field_count": 32448,
"claim_count": 49,
"claims": [
{
Expand Down Expand Up @@ -108,8 +108,8 @@
],
"mode": "legacy_behavior",
"pointer_class": "all",
"pointer_count": 312,
"pointer_sha256": "9bad55b4945af1c4213c510942e7b9d22d104bf75b0d9804e4a183e1656c8393",
"pointer_count": 315,
"pointer_sha256": "13227bf16227b7027c6ccd1220bf2b451b6c8be2b17dbe63b0c5500dc32433a2",
"rationale": null,
"relative_sink_prefix": "/calibration_contract",
"source_prefix": "/authored/spec~1calibration.yaml",
Expand Down Expand Up @@ -514,8 +514,8 @@
],
"mode": "legacy_behavior",
"pointer_class": "all",
"pointer_count": 1749,
"pointer_sha256": "d0979722ea5153bd152e5e1c1201f5d530d11b21aecad18aca38473962caa062",
"pointer_count": 1757,
"pointer_sha256": "29c137db9d9e522c4280fbad42a3d3528411453853d2c7c5d1d347662e7b2076",
"rationale": null,
"relative_sink_prefix": "/source_manifest/stages",
"source_prefix": "/authored/spec~1sources.yaml/stages",
Expand Down Expand Up @@ -772,20 +772,20 @@
"verifier": "vintages"
}
],
"configuration_field_count": 42218,
"consumed_field_count": 42218,
"configuration_field_count": 42229,
"consumed_field_count": 42229,
"generation0_effect_counts": {
"legacy_behavior": 38531,
"legacy_behavior": 38542,
"no_generation0_effect": 3687
},
"mode_counts": {
"compiler_semantic": 27718,
"front_end_validation": 354,
"identity_only": 103,
"legacy_behavior": 14043
"legacy_behavior": 14054
},
"multiple_primary_use_field_count": 0,
"pointer_inventory_sha256": "a0da3562ad5559cbbcdf5ee14ebd21db2f067c90e7e470e24d4e3762d7b90848",
"pointer_inventory_sha256": "0a2951f4484a0dd9459a474505b5a0caa0216b80ad13009a286dd85e2165e7e8",
"resolved_binding_field_count": 9781,
"unused_field_count": 0
},
Expand Down Expand Up @@ -824,7 +824,7 @@
"stacked_checkpoint_full_components": 13,
"stacked_checkpoint_pool_code_components": 19,
"stacked_checkpoint_static_components": 10,
"tail_control_fields": 934,
"tail_control_fields": 937,
"take_up_pipeline_steps": 28,
"take_up_programs": 17,
"typed_artifacts": 84,
Expand Down Expand Up @@ -917,13 +917,13 @@
"legacy_adapter.stacked_checkpoint_static_components"
],
"expected": {
"sha256": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c",
"terminal_fields": 934
"sha256": "256947cd7c1bbd85f5d1fa3911a07a170ae87830f098110a40ed25d93f78347c",
"terminal_fields": 937
},
"failures": [],
"observed": {
"sha256": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c",
"terminal_fields": 934
"sha256": "256947cd7c1bbd85f5d1fa3911a07a170ae87830f098110a40ed25d93f78347c",
"terminal_fields": 937
},
"status": "covered"
},
Expand Down Expand Up @@ -1361,12 +1361,12 @@
],
"expected": {
"producer_count": 38,
"sha256": "cbac7c8fbdbb82f0f06ab356d0c9f0d3645ca930550b44af2272d93315f2e934"
"sha256": "b555f811aaf1b65f588a3aa0a6cee2d311b1bc40b1735749c527749b5b3d0da1"
},
"failures": [],
"observed": {
"producer_count": 38,
"sha256": "cbac7c8fbdbb82f0f06ab356d0c9f0d3645ca930550b44af2272d93315f2e934"
"sha256": "b555f811aaf1b65f588a3aa0a6cee2d311b1bc40b1735749c527749b5b3d0da1"
},
"status": "covered"
},
Expand Down Expand Up @@ -1775,12 +1775,12 @@
"legacy_adapter.imputation.late_producer_resource_semantics"
],
"expected": {
"sha256": "629a0712659a26b2d6aa6c0e5672421f415d625e0599d6baa0a834d118953c6a",
"sha256": "87b1e124969ec11b836740c027d28dc2a9d319d279ed55be1d9fc1523390f634",
"stages": 38
},
"failures": [],
"observed": {
"sha256": "629a0712659a26b2d6aa6c0e5672421f415d625e0599d6baa0a834d118953c6a",
"sha256": "87b1e124969ec11b836740c027d28dc2a9d319d279ed55be1d9fc1523390f634",
"stages": 38
},
"status": "covered"
Expand Down Expand Up @@ -1887,7 +1887,7 @@
"alpha",
"zeta"
],
"sha256": "e5da85693fab1f1cba9c37beba5cdf91477962d466bc21e9f29125f2f90b7e79"
"sha256": "9fb7350a66a0ec9f7a4dbc879ccafaf1e6cb51486722938b3a8e6f5c57ae9e06"
},
"failures": [],
"observed": {
Expand All @@ -1911,7 +1911,7 @@
"alpha",
"zeta"
],
"sha256": "e5da85693fab1f1cba9c37beba5cdf91477962d466bc21e9f29125f2f90b7e79"
"sha256": "9fb7350a66a0ec9f7a4dbc879ccafaf1e6cb51486722938b3a8e6f5c57ae9e06"
},
"status": "covered"
},
Expand Down Expand Up @@ -1975,7 +1975,7 @@
"take_up_contract",
"us_qbi_reconciliation_contract"
],
"sha256": "fb863f18a79610dc1005089a32e1535c502c5b9f7e5ba589db0d6025b01e3c82"
"sha256": "b7669fd0f941d7ca68fbdf5966d5ae8751fda598dc64f6d0d7420c0c33b8e9e2"
},
"status": "covered"
},
Expand Down Expand Up @@ -2599,7 +2599,7 @@
"country": "us",
"schema_id": "country_spec",
"schema_version": 1,
"spec_sha256": "77df608c9b343219cd2717aa08af5198c0ba69b70c61691fb2459f7a886528ba"
"spec_sha256": "53fb8de63585a27fc86a10a94eeae4ca546a4ded2d528514c64e4df574b430f8"
}
},
"report_schema_version": 3,
Expand All @@ -2609,7 +2609,7 @@
"country": "us",
"schema_id": "country_spec",
"schema_version": 1,
"spec_sha256": "77df608c9b343219cd2717aa08af5198c0ba69b70c61691fb2459f7a886528ba"
"spec_sha256": "53fb8de63585a27fc86a10a94eeae4ca546a4ded2d528514c64e4df574b430f8"
},
"status": "pass"
}
6 changes: 6 additions & 0 deletions docs/us-asec-census-person-columns.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,12 @@ the column.
`CENSUS_TAX_ID` was never appended: the extractor derives it, and the Census
files do not carry it.

Beyond that receipt, `WEIND` and `WEMIND` (industry of the longest job last
year) are restored for #719, read by `org_wages.derive_us_org_occupation_inputs`.
No H5 vintage carries them, so every vintage gets them from its member, and
each vintage is checked against the industry identities. See
[`us-work-experience-columns.md`](us-work-experience-columns.md).

## Real-data evidence (2026-09-23, commit `39b8e7b63`)

The real-data run used route A's exact base inputs and flags
Expand Down
133 changes: 133 additions & 0 deletions docs/us-work-experience-columns.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Industry and worked-last-year on every US person record (#719)

The county-file person schema carries a working indicator, industry and
occupation on every person record. Occupation already shipped
(`detailed_occupation_recode`, CPS ASEC `POCCU2`). This change adds three
PolicyEngine-US person inputs (defined in policyengine-us#9311, released in
1.820.2, locked at 2.2.1):

| column | meaning | CPS ASEC source | ACS 2024 source |
|---|---|---|---|
| `detailed_industry_recode` | industry of the longest job last year, 23 detailed groups | `WEIND` | `INDP` via `ACS_INDP_TO_WEIND` |
| `major_industry_recode` | the same job, 15 major groups | `WEMIND` | `WEIND_TO_WEMIND` of the above |
| `worked_last_year` | worked at any time last year | `WKSWORK > 0` | `WKWN > 0` |

Codes (`WEIND`): 0 = not in universe (under 15; ACS: under 16), 1–21 =
civilian industry groups, 22 = Armed Forces, 23 = age 15+ (ACS: 16+) and did
not work last year. `WEMIND` nests them: 0 = not in universe, 1–13 civilian
major groups, 14 = Armed Forces, 15 = did not work.

## Why these CPS columns

`WEIND`/`WEMIND` describe the longest job last year, the same job `POCCU2`
describes, and the same reference year as the income the file carries. The
survey-week recodes `A_DTIND`/`A_MJIND` describe the current job instead.
`worked_last_year` uses `WKSWORK > 0` rather than `WORKYN = 1`: in all three
pinned members `WEIND` carries a worker code exactly where `WKSWORK > 0` (zero
exceptions), while `WORKYN = 1` misses 642 / 572 / 594 allocation rows per
year that have weeks worked and an industry.

## Where each spine gets them

- **ASEC rows (and their PUF-support clones).** No `census_cps` H5 vintage
carries `WEIND`/`WEMIND`, not even 2024. `asec_census_person_columns`
(#720's restoration) appends both to every vintage from its pinned Census
person member by exact `PERIDNUM` join before pooling. They sit outside the
2026-08-23 offline-fix receipt (`ASEC_CENSUS_PERSON_COLUMNS_BEYOND_OFFLINE_FIX`).
`WKSWORK` is already in every H5.
- **ACS rows.** The release predictor join (`acs_release_predictors`,
crosswalk version 2) materializes `WEIND`, `WEMIND` and `WKSWORK` for ACS
people from the pinned ACS 2024 person archive, the same boundary that maps
`OCCP` to `POCCU2`. The newer ACS local-area native-input mapper uses the
same `crosswalk_acs_occupation_and_work_experience` helper and writes the
canonical occupation, industry and worked-last-year inputs directly. Its
source loader retains the required `OCCP`, `INDP`, `WKWN` and `ESR` fields.
- **Pooled release rows.** The release-time `org_wages` carry
(`derive_us_org_occupation_inputs`) turns them into the three columns next
to `detailed_occupation_recode`.

## The ACS crosswalk

ACS 2024 `INDP` uses the 2022 Census industry code list, as does the 2025 ASEC
member (`pppub25.csv`), where nonzero `INDUSTRY` determines `WEIND` exactly
(259 codes). The 2023/2024 members use the 2017 list; the 225 codes the lists share
map to the same group in both. `ACS_INDP_TO_WEIND` takes every civilian code's
group from `pppub25.csv` and reviews the eight ACS-only codes:

- 9670, 9680, 9690, 9770, 9780, 9790, 9870 (Army, Air Force, Navy, Marines,
Coast Guard, branch not specified, Reserves or National Guard) → 22. CPS
collapses them into 9890, which the members map to 22.
- 9920 ("unemployed, no work experience in the last 5 years or never
worked") → 23, as `OCCP` 9920 maps to `POCCU2` 53.

Blank `INDP` follows the `OCCP` rule: 0 below age 16 (age 15 keeps the
out-of-universe sentinel rather than inventing CPS code 23), 23 from age 16,
where only ESR 6 may be blank. `WKSWORK` is `WKWN` (weeks worked in the past 12
months) or 0.

Reference periods differ, as they already do for occupation: an observed ACS
`INDP` describes the most recent job in the past five years, and `WKWN` counts
the 12 months before interview; the CPS columns describe the prior calendar
year. So on ACS rows an industry can sit on a person with
`worked_last_year = False` (a job held one to five years ago).

## Checked identities

| identity | ASEC rows | ACS rows | where checked |
|---|---|---|---|
| `WEMIND = WEIND_TO_WEMIND[WEIND]` | yes | yes | restoration (per vintage), ACS join (by construction), `org_wages` carry and gate |
| `WKSWORK > 0` ⇒ `WEIND` in 1–22 | yes | yes | restoration, ACS join, `org_wages` carry and gate |
| `WEIND` in 1–22 ⇒ `WKSWORK > 0` | yes | no (5-year `INDP`) | restoration only, where rows are known to be ASEC |

## Real-data evidence (2026-09-24)

ASEC members `pppub23/24/25.csv` (pins in `spm_role_source.ASEC_SPM_ROLE_SOURCES`):

- Nonzero `INDUSTRY` → `WEIND` and → `WEMIND` are functions in every vintage
(263 / 263 / 259 codes); `INDUSTRY = 0` covers both `WEIND = 0` and the
did-not-work group 23; `WEIND` → `WEMIND` is the same function in all three.
- `WEIND` in 1–22 iff `INDUSTRY > 0` iff `WKSWORK > 0`: zero violations in
each vintage. `WEIND = 0` exactly for ages 0–14; `WEIND = 23` only from 15.
- Weighted (`A_FNLWGT`) shares: nonzero `WEIND` 0.8217 / 0.8220 / 0.8244;
worked last year 0.5200 / 0.5232 / 0.5228.

ACS 2024 one-year person archive (SHA-256 `afdc6d90…`, the release pin), all
3,422,888 persons run through `_crosswalk_people`:

- No refusal fires; all 265 observed `INDP` codes are mapped.
- `INDP` and `OCCP` are blank on exactly the same rows; a blank at 16+ occurs
only with ESR 6.
- `WKWN > 0` iff `WKL = 1` (zero disagreements); every person with weeks
worked has a worker `WEIND` code.
- 264,806 persons carry an industry without weeks worked, exactly the count
with `WKL = 2` (last worked one to five years ago).
- Weighted (`PWGTP`) shares: nonzero `WEIND` 0.8123; worked last year 0.5385.

The `org_wages` gate bands (`detailed_industry_recode` and
`major_industry_recode` nonzero 0.70–0.92, `worked_last_year` 0.40–0.62)
bracket both spines.

## What this does not do

It changes code and manifests only. Released files carry the columns once the
pooled base and multispine pool are rebuilt with the restoration and a new
release runs (#922 sequencing step 1); until then the release input-coverage
gate is red on these three columns by design.

The ACS local-area staging lane also preserves its ASEC donor rows. It now
refuses a donor missing complete canonical occupation, industry or
worked-last-year inputs before acquiring ACS data. Historical donors need
the exact ASEC restoration and fiscal `org_wages` carry before that lane can
produce a complete stacked export. Packaging rechecks the final H5 person
rows and binds the result to its hash, so an older green staging summary
cannot bypass this requirement. The assembly-first nullable pool remains
an intermediate input; its fiscal release applies that carry to every row.

The 2026-10-23 branch refresh preserves main's current stage and graph
structure, including its six ESI source fields alongside the industry
restoration. Main's independent refusal of mixed ACS inputs with null ESI
premiums remains in place; this refresh adds no ESI producer.

The real-data measurements above are historical evidence; the
refresh validates synthetic contracts without rebuilding ASEC, ACS or a
release, so it does not certify any existing or newly built data artifact.
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@
)
from .schemas import load_schema_registry

EXPECTED_AUTHORED_FIELD_COUNT = 32_437
EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_781
EXPECTED_CONFIGURATION_FIELD_COUNT = 42_218
EXPECTED_AUTHORED_FIELD_COUNT = 32_448
EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9781
EXPECTED_CONFIGURATION_FIELD_COUNT = 42229


class FieldUsageError(AssertionError):
Expand Down Expand Up @@ -361,8 +361,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]:
"36564747627fb0058bb99e8e8c4397e41d8ecc472a20c49d4b6d7df1003a80ec",
),
"calibration": (
312,
"9bad55b4945af1c4213c510942e7b9d22d104bf75b0d9804e4a183e1656c8393",
315,
"13227bf16227b7027c6ccd1220bf2b451b6c8be2b17dbe63b0c5500dc32433a2",
),
"catalog_columns": (
1_673,
Expand Down Expand Up @@ -485,8 +485,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]:
"d6782c5de5bbed1bdc6bf653c4a6d4aadcad4ccc72d35e1092e130fcb04680a3",
),
"source_stages": (
1_749,
"d0979722ea5153bd152e5e1c1201f5d530d11b21aecad18aca38473962caa062",
1_757,
"29c137db9d9e522c4280fbad42a3d3528411453853d2c7c5d1d347662e7b2076",
),
"spine_assembly_household_mass_shares": (
2,
Expand Down
Loading
Loading