Repository navigation
Fix Populace release and export gates - #37
Merged
Merged
Conversation
…e producer The releases already on policyengine/populace-us disagree with each other: 1abddeb has no build_manifest.json and an unversioned release_manifest schema, while later releases carry schema_version 1. Every consumer was left to implement its own defensive filter. validate_release_dir() is the single gate: required files, manifest schema version, and build-id agreement between both manifests and the directory name — raising ReleaseContractError that names every violation at once. Verified against the real 9f1260b release (passes) and the real 1abddeb release (rejected with six named failures). Fixes #11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Hub repo had no way to say which release is current: consumers listed the releases/ tree and guessed, and same-day builds (the ids end in a date) have no defined ordering. publish_release() validates the release directory against the contract (an invalid release uploads nothing), uploads its files, and writes the latest.json pointer LAST so a reader never sees a pointer to files that are not there yet. latest_release() is the consumer side: one call that returns the typed pointer. The Hub client is injected so the suite tests the real ordering against a fake, network-free. Fixes #9. Stacked on #11's contract branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he artifact CalibrationResult already carries everything an auditor needs — per-target initial/final estimates and tolerance verdicts, the per-epoch loss trajectory, every skipped target with its reason, and the solver options actually used — but none of it left the build machine: the build pushed diagnostics to telemetry and dropped them, and the published .npz kept only closing scalars. "Skipped and reported, never dropped silently" is only true if the report ships. diagnostics_payload() renders the result as strict JSON (non-finite floats become null; the writer passes allow_nan=False so anything that escapes the scrub fails loudly), and write_calibration_diagnostics() writes the calibration_diagnostics.json a release directory publishes next to its manifests. Fixes #10. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jun 14, 2026
Closed
MaxGhenis
added a commit
to TheAxiomFoundation/axiom-oracles
that referenced
this pull request
Oct 15, 2026
The July entry eitc-earned-income-8614-partnership-scorp-split selected every tax unit's EITC rows (case_id_prefix "ecps-tax_unit_"), and its mechanism was wrong. It said PolicyEngine-US 1.729.0 predates #8614, but 1.729.0 contains #8614. Every EITC row of the 2026-10-14 run is now attributed by counterfactual. Each mechanism is one change to Axiom's inputs that makes Axiom see what PolicyEngine-US 1.729.0 sees, and a row belongs to the smallest set of changes whose re-run clears it. The 17,424 EITC rows split into five entries, each listing its rows by output (case_ids_by_output) and bound to them (selector_binding): - 7,627 rows, bridge_artifact: the bridge's 32(c)(2) self-employment projection (axiom-oracles#640). - 4,618 rows, bridge_artifact: the Populace artifact stores partnership_se_income, which 1.729.0 no longer reads (PolicyEngine/microcosm#35). - 4,086 rows, bridge_artifact: the Populace artifact stores a dividend_income that disagrees with its components (PolicyEngine/microcosm#37). - 890 rows, upstream_engine_gap: 1.729.0's 32(i)(2) investment-income definition (PolicyEngine/policyengine-us#9635 and #9567; Axiom debt rulespec-us#1416). - 203 rows, upstream_engine_gap: 1.729.0's childless age test counts dependents (PolicyEngine/policyengine-us#9131; Axiom debt rulespec-us#1776). The tax-before-credits, capital-gain and CTC entries are narrowed to their 2,131 rows the same way, and each row's signature is re-verified. Evidence, tooling and tests: - The evidence report (reports/fiit-ecps-eitc-attribution-2026-10-14.json) records every row's stage attributions. The FULL v2 report the dispositions bind to is committed beside it. - scripts/fiit_ecps_eitc_attribution/ holds the three stage scripts, which now share minimal_sets.attribute. Re-applied to the variants stages 1 and 2 stored, that function reproduces their attributions exactly. - scripts/build_fiit_ecps_eitc_dispositions.py rebuilds the evidence byte for byte from the stage outputs. Its --check mode audits the evidence, the dispositions and the bound report against each other, and tests/test_fiit_ecps_eitc_attribution.py runs it with mutation cases. - Stage 3's base check now accepts rows only the counterfactual has, and records them. The PE-Axiom ratchet rises by one, from 81 to 82, as an approved and recorded debt_raises entry. The grandfathered entry is closed, and the two PolicyEngine attributions declare their Axiom side as debt instead. docs/federal-income-tax-comparison.md now gives the October residuals and the suite's manual lane. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
to TheAxiomFoundation/axiom-oracles
that referenced
this pull request
Oct 17, 2026
…ctual attribution (#635) * Run fiit-ecps against this checkout's bridge and a commit-pinned PolicyEngine-US 1.729.0 PyPI no longer serves policyengine-us 1.729.0, so the suite pins its release commit's archive (byte-identical to the PyPI wheel) and the runner checks the installed version against the label before a population run. The tax-ecps runner now overlays this checkout's bridge via PYTHONPATH, as the UK EFRS runner does, so a run validates the current bridge rather than the axiom-oracles commit axiom-encode pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bind non-v2 harness reports through a full v2 report; republish a run with --from-report tax-populace-compare writes its own JSON, so the reports/ file a premerged dashboard copy pointed at was not a v2 report and apply_dispositions.py --check could never accept it. The runner now publishes the adapted v2 report (every mismatch row, no cases) beside the raw one and binds the dashboard copy to it. --from-report rebuilds a dashboard copy from an earlier real run after a dispositions edit, executing nothing, and stamp_selector_bindings.py binds against the report the dashboard names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Correct fiit-ecps's #8614 rationale: 1.729.0 carries #8614; the pinned artifact stores the pre-#8614 column PE-US 1.729.0's release commit (180410fe) descends from #8614's merge (1de4bff6) and reads partnership_self_employment_net_earnings. The pinned Populace artifact still stores partnership_se_income, so PolicyEngine reads $0 while the bridge feeds Axiom the old column. Document the runner's overlay, commit-pinned source, full v2 report and --from-report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Let a population run raise the tax bridge's engine timeouts One engine call evaluates a whole surface over the selected population. On a loaded host the CTC call over all 87,519 Populace units outlasted the fixed 600-second run timeout. AXIOM_ENGINE_RUN_TIMEOUT_SECONDS and AXIOM_ENGINE_COMPILE_TIMEOUT_SECONDS raise them; the defaults are unchanged and a malformed value is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Request the engine's fast mode from the tax bridge Fast returns exactly what explain returns except the trace, and the bridge reads only output values. Since axiom-rules-engine#131, explain builds each query's trace by scanning every cached value in the request, so a population request's cost grows with the square of its tax units: fiit-ecps's CTC surface ran past 36 minutes over 87,519 units. The Medicaid and SNAP Populace bridges already request fast. Log the engine's fallback reason when it runs explain instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin fiit-ecps to the engine line rulespec-us builds its artifacts with Engine main cannot compile rulespec-us main's 26 USC 1(h), 1(j), 32 and 25A programs (module.source_verification.values, rulespec-us#1050). The release/v0.1 engine rulespec-us pins as axiom_artifact_rules_engine_ref (98af0dce, v0.1.2) compiles all thirteen fiit-ecps surfaces, so the suite declares that revision and the runner refuses any other checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep the full v2 binding report to the rows the binding check reads fiit-ecps's adapted report carries one observed_outputs row per compared value (about 3.9 million rows, 1 GB). apply_dispositions.py --check re-derives a premerged dashboard copy only from summary and mismatches, so the published binding report omits observed_outputs as well as cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Select disposition rows by output, so one case's rows can carry different mechanisms fiit-ecps's EITC concept covers eleven compared outputs, so one tax unit carries several rows under it, and the rerun shows units whose rows have different mechanisms (an investment-income eligibility row beside a partnership earned-income row). Case-level selectors would give all of a unit's rows to the first matching entry. The tax-ecps adapter now names each row's output; case_selector.case_ids_by_output selects exact (output, case) pairs; the premerged binding check keys retained rows by output too. Selector membership sets are built once per selector. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Publish the full v2 binding report only when a dashboard copy needs a binding A suite whose dashboard copy keeps every mismatch row never binds to a source, so writing a -v2.json beside each non-v2 raw report published an extra file for every such run (four provenance tests counted the published files). The dashboard writer now writes it only for a premerged-slim copy whose named source is not already this suite's v2 report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep population-scale observed outputs out of published fiit-ecps copies A full fiit-ecps run records one observed-output row per compared value, 3,881,635 rows. The tax-ecps adapter has kept them since #368, so the first full run after it wrote a 1.2 GB dashboard copy and a 999 MB overview. Past 10,000 rows, the dashboard copy and the FULL v2 binding report now drop observed_outputs whole and record the count in observed_outputs_omitted. Each output's comparison count stays in output_summary. Attestation only credits an output when the report body shows its returned values. So dropping the rows also empties the stamp's per-output claims and sets outputs_complete to false. A copy with the rows gone and the stamp left as it was fails attestation, and a test pins that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Split fiit-ecps's blanket #8614 EITC disposition by per-row attribution The July entry eitc-earned-income-8614-partnership-scorp-split selected every tax unit's EITC rows (case_id_prefix "ecps-tax_unit_"), and its mechanism was wrong. It said PolicyEngine-US 1.729.0 predates #8614, but 1.729.0 contains #8614. Every EITC row of the 2026-10-14 run is now attributed by counterfactual. Each mechanism is one change to Axiom's inputs that makes Axiom see what PolicyEngine-US 1.729.0 sees, and a row belongs to the smallest set of changes whose re-run clears it. The 17,424 EITC rows split into five entries, each listing its rows by output (case_ids_by_output) and bound to them (selector_binding): - 7,627 rows, bridge_artifact: the bridge's 32(c)(2) self-employment projection (axiom-oracles#640). - 4,618 rows, bridge_artifact: the Populace artifact stores partnership_se_income, which 1.729.0 no longer reads (PolicyEngine/microcosm#35). - 4,086 rows, bridge_artifact: the Populace artifact stores a dividend_income that disagrees with its components (PolicyEngine/microcosm#37). - 890 rows, upstream_engine_gap: 1.729.0's 32(i)(2) investment-income definition (PolicyEngine/policyengine-us#9635 and #9567; Axiom debt rulespec-us#1416). - 203 rows, upstream_engine_gap: 1.729.0's childless age test counts dependents (PolicyEngine/policyengine-us#9131; Axiom debt rulespec-us#1776). The tax-before-credits, capital-gain and CTC entries are narrowed to their 2,131 rows the same way, and each row's signature is re-verified. Evidence, tooling and tests: - The evidence report (reports/fiit-ecps-eitc-attribution-2026-10-14.json) records every row's stage attributions. The FULL v2 report the dispositions bind to is committed beside it. - scripts/fiit_ecps_eitc_attribution/ holds the three stage scripts, which now share minimal_sets.attribute. Re-applied to the variants stages 1 and 2 stored, that function reproduces their attributions exactly. - scripts/build_fiit_ecps_eitc_dispositions.py rebuilds the evidence byte for byte from the stage outputs. Its --check mode audits the evidence, the dispositions and the bound report against each other, and tests/test_fiit_ecps_eitc_attribution.py runs it with mutation cases. - Stage 3's base check now accepts rows only the counterfactual has, and records them. The PE-Axiom ratchet rises by one, from 81 to 82, as an approved and recorded debt_raises entry. The grandfathered entry is closed, and the two PolicyEngine attributions declare their Axiom side as debt instead. docs/federal-income-tax-comparison.md now gives the October residuals and the suite's manual lane. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Close the Sol review of #635: output in digests, per-merge selector sets, a merge-faithful audit Fixes the four major findings and the minor one from the round-1 review (~/reviews/axiom-oracles-635/review-r1.md). - Both binding digests now include each row's output when the row names one. Before this, two equal-valued rows of one case could trade entries without changing selector_binding or the premerged assignment digest. Output-less rows hash exactly as before, so the bindings of us-tariff-panel and fiit-taxsim-ecps are unchanged. fiit-ecps's bindings are restamped. - A selector's membership sets are built once per merge pass and dropped after it, instead of being cached across merges by list identity and length. A same-length, in-place edit between merges is now seen. - build_fiit_ecps_eitc_dispositions.py --check now does three more things: validates the dispositions document; replays the production merge over the bound report, which must give every row its evidence entry with nothing expired, orphaned or unexplained; and requires each entry's selector_binding to be its rows' binding. Mutation tests cover a changed kind, a changed concept, a corrupted digest, a removed binding and a removed mechanism. - --from-report takes the publication lock before its first read and adapts the one snapshot it parsed, so a run replacing the report meanwhile cannot pair one run's results with another's provenance. - The investment-definition entry no longer says its rows clear "only through" a definition switch. 171 of the 890 tie with the Populace dividend artifact alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Report malformed audit inputs instead of raising; match outputs as strings Two follow-ups from the round-2 review of #635 (~/reviews/axiom-oracles-635/review-r2.md), which approved 0363f2d: - build_fiit_ecps_eitc_dispositions.py --check reports a committed file of the wrong shape as a problem, not a traceback. Tests cover a missing run block, a list in place of the rows, a row without its value and a string binding. - case_ids_by_output compares outputs as strings, as the binding digest identifies them. Every current output is a string, so no selection changes. The assignment-digest sort nit is left as it is. It can only reorder rows whose entry ids are empty, which validation never admits, and changing it would re-digest every premerged suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Reject unresolved FIIT EITC counterfactual attributions Require supported clearing labels in every applicable downstream stage and passing baseline/model checks before building or auditing dispositions. Check original mismatches while allowing counterfactual-only extra rows. Cover corrupt labels, failed checks, missing stage-3 labels, and an honestly unresolved stage-2 run. Preserve saved evidence, counts, and row bindings. Clarify selector expiry and correct the TAXSIM suite count in the PR body. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Correct two descriptions of fiit-ecps refresh and v2 report contents The unattended regeneration scripts do attempt fiit-ecps; the runner refuses an engine checkout that is not at the pinned commit. The v2 report drops observed_outputs only above 10,000 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the live-Populace cleanup after retiring Microplex/eCPS from the production repo surface. It folds the release contract/latest-pointer/calibration-diagnostics draft PRs into one branch and adds release/export gates that should block bad artifacts before publication.
Fixes #9, #10, #11, #18, #24, #25, and #36.
Refs #27, #28, #29, #33, and #34.
Supersedes draft PRs #12, #13, and #14 once merged.
What changed
build_manifest.json,release_manifest.json, andcalibration_diagnostics.jsonbefore publicationlatest.jsonlast, validate exact contract paths on read, and reject partial/invalid uploads before touching the HubNaNin release contract filesrepo_idpolicyengine-us==1.729.0; keeppolicyengine-us-data<1.115.4because 1.115.4 still pins the older engine metadataPolicyEngine/populace-benchmarksValidation
47 passeduv sync --python 3.13 --all-packages --all-extras --dev;ruff check .;pytest=>459 passed, 2 skipped459 passed, 2 skipped410 passed, 6 skippedeachgit diff --checkrg -i "microplex|\\becps\\b|enhanced_cps|enhanced CPS|sound_ecps|mp[-_]?ecps|populace-score-work|score_driver" ./cycleread-only review: final pass had no actionable findingsNotes
This intentionally does not merge PR #21's eCPS parity runner or PR #22's old
build_us_candidate.pypatch. #21 is benchmark-harness material now, and #22 targets a deleted historical build path; the partnership/S-corp values work should be ported to the current Populace build/profile surface separately.