Skip to content

U2: commit the pre-registration evidence record and test it as present - #577

Merged
MaxGhenis merged 4 commits into
masterfrom
u2-preregistration-evidence-record
Oct 25, 2026
Merged

MaxGhenis merged 4 commits into
masterfrom
u2-preregistration-evidence-record

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

This adds U2's pre-registration evidence record at docs/design/u2_preregistration_evidence.json, the path #556 made loader.PREREGISTRATION_EVIDENCE_PATH. From here on the record's presence is the intended state. Four checks asserted that the record was absent, so they now treat it as present. Every refusal branch is still tested: absence is forced through an explicit missing path, and the record is never removed.

The record (35954275)

  • A byte-for-byte copy of the record that the §20 step-4 pass wrote under d1724 at 1e739d328b73c99d259a72952b3452546721bfb8 (master Tests run 39502262845). It holds hashes only.
  • SHA-256 7bc5c3c26ff014271061a8228068c6e71103430ec299048462570476a510b845, 13,946 bytes. Checked equal at the source, at the destination (cmp byte-identical) and in git show HEAD:docs/design/u2_preregistration_evidence.json. It equals the step-4 pass's MANIFEST pin and loader.PREREGISTRATION_EVIDENCE_SHA256.
  • Value-free scan: CLEAN. The scan report's SHA-256 is d1f1b1bc3ca1e1f4ee8a899c2ddede2858f2cf3b54b6138179127e0e47872dd0.

Tests treat the record as present (2faf0356)

  1. tests/track_u2/test_u2_loader.py: test_the_committed_evidence_record_is_absent_so_registration_refuses is replaced in place by test_the_committed_evidence_record_is_the_pinned_one. It asserts:
    • the file exists;
    • its SHA-256 equals loader.PREREGISTRATION_EVIDENCE_SHA256 and the literal pin;
    • read_preregistration_evidence() accepts it;
    • read_preregistration_evidence(tmp_path / "absent.json") still refuses with "no pre-registration";
    • a copy one hex digit away refuses as "not the committed record". That copy is still well-formed (check_preregistration_evidence accepts it), so only the pin refuses it.
  2. test_a_frozen_load_needs_the_committed_record keeps its "committed record differs" case.
    • The "no record" case now patches loader.read_preregistration_evidence with a wrapper that reads a missing path. Patching PREREGISTRATION_EVIDENCE_PATH would not work, because the default argument is bound at definition.
    • It still refuses before any PSID path is resolved or any file opened.
  3. tests/track_u2/test_u2_scripts.py::test_preflight_refuses_while_the_evidence_record_is_absent uses the same wrapper for the registered preflight.
  4. scripts/track_u2_dry_run.py: registered_preflight_without_evidence now reads an explicit path inside a fresh temporary directory, and checks that the path does not exist first. The check raises rather than asserts, so it holds under python -O. The output keys are unchanged, and test_dry_run_records_every_branch_and_refusal passes unchanged.

Not touched: docs/design/boomers2004_1946_55_comparison.md (the registration binds its SHA-256), loader.py and scripts/run_track_u2_registered.py. No test is added or removed. .test_durations is not edited here, so it still lists the replaced test's old name.

Test results

All runs used an empty PSID root (POPULACE_DYNAMICS_PSID_DIR and HOME pointed at empty directories), -n 4, Python 3.14.7 and a .[dev] install (pandas 3.1.0, numpy 2.5.4).

  • At 2faf0356: tests/track_u2 plus tests/estimates/test_birth_evidence_artifact.py (which lists the dry-run script): 1,432 passed, 254 skipped, 0 failed.
  • tests/track_u2 compared with 0e462efa (before the record existed, 1,419 passed, 254 skipped): per-test outcomes are identical, except that the replaced test passes under its new name.
  • Tier counts (pytest --collect-only -q -m <tier>), unchanged before and after the edits and equal to tests/tier_counts.json:
    • unit 18,125
    • artifact 4,261
    • integration_psid 1,450
    • reproduction_legacy 520
    • oracle_policyengine 308
    • 24,664 in total
  • Black and Ruff are clean on the three changed Python files.

Exact U1 differential at 2faf0356 (c9c0f4a2)

docs/design/u2_m2_u1_differential_2faf035.json (SHA-256 4d19093b56fe52555388cff7ee721b80e8302c7a859bd6a18d077e100229c891) is the output of scripts/u2_u1_differential.py, run the way #556 ran it and in the frame environment the evidence record pins.

Setup

  • base 9cee2423f048683e15838aa5fee330afd072f85f and candidate 2faf03565cb55ab56256317aebefab2a2ce82efd, as clean detached checkouts;
  • Python 3.14.7, pandas 3.0.6 and numpy 2.5.4 (the step-4 pass's versions), with future.infer_string True;
  • seed 20260924 and an empty PSID root.

The report's environment block records these settings. It was read from the same interpreter after the run. With the block removed, the file is byte-identical to the harness's output.

Results

  • all_equal true and all_checks_passed true.
  • Refusal expectations met on both sides.
  • loader_file_accesses_empty true, and dry_run_skipped false.
  • The contract-check digest (49e47726…) equals the one in u2_m2_u1_differential_14d9576.json, and in every earlier record.
  • The dry-run digests are equal between base and candidate. They differ from earlier records' digests, as those differ from one another, because the harness's temporary output directory is part of the compared run.command.

Under pandas 3.1.0, which a fresh .[dev] install and CI now resolve, the results are the same. That report was committed at this path in a0fd18cc (SHA-256 d85499e5…, numpy 2.5.4). It differs from the 3.0.6 report only in the interpreter path, the temporary directory in run.command, and the dry-run digests that include that directory.

Carry-forward for step 9: the registered run's pandas

The registered one-shot must run under pandas==3.0.6 with future.infer_string True: the frame_environment the evidence record pins.

  • Why it matters: a fresh .[dev] install now resolves pandas 3.1.0.
  • What happens under another pandas: loader.load_u2_inputs(evidence=…) calls check_frame_environment. That check compares {pandas_version, future_infer_string} with the record and raises U2FrozenIdentityMismatch before any PSID path is resolved. scripts/run_track_u2_registered.py calls that loader. runner.run_track_u2 checks the same environment again and raises U2RunError.
  • numpy: the record does not pin it; the step-4 pass ran numpy 2.5.4.
  • Install used here for the differential: uv pip install -e ".[dev]" "pandas==3.0.6" "numpy==2.5.4" under Python 3.14.7.
  • The test suite: CI's pandas 3.1.0 is fine for it.

Review and CI at c9c0f4a2

  • Independent review: APPROVE, no findings. Done by GPT-6.1 Sol (Subfleet job 20261024-121913-md577-review-c9c0f4a) in its own clone at this head, with an empty PSID root. The reviewer ran every check itself:
    • Scope: five files and four commits; the protected files and tests/tier_counts.json are unchanged.
    • Mutations: pointing either wrapper at the committed path makes its test fail. Under python -O the dry-run helper still raises before the read.
    • Dry-run output: the key paths of result.json equal those at 0e462efa (29,004 paths).
    • Tests: 1,432 passed, 254 skipped, 0 failed. The five tier counts are unchanged, and the only node-ID change is the renamed test.
    • Lint: Black and Ruff are clean.
    • Differential: the reviewer reran it independently under pandas 3.0.6. It found no field differences after normalizing paths and the temp directory, and it reproduced both committed dry-run digests by changing only the temp-directory string in run.command.
    • Frame-environment claim: checked against the code and by direct probes.
  • CI: Tests run 40110842940 succeeded at c9c0f4a2, and every check passes.

axiom: n/a: committed evidence record (hashes) and test updates, no policy rule

🤖 Generated with Claude Code

… by value-free scan)

Adds docs/design/u2_preregistration_evidence.json, a byte-for-byte copy
of the record the section 20 step-4 pass wrote under d1724 at 1e739d3.
Its SHA-256 equals loader.PREREGISTRATION_EVIDENCE_SHA256
(7bc5c3c26ff014271061a8228068c6e71103430ec299048462570476a510b845).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
social-security-model Ready Ready Preview Oct 24, 2026 4:18pm UTC

Request Review

MaxGhenis and others added 2 commits October 24, 2026 12:06
The committed pre-registration evidence record is now the intended
state. Every refusal branch stays tested, with absence forced through an
explicit missing path; the record itself is never removed.

- test_the_committed_evidence_record_is_absent_so_registration_refuses
  is replaced in place by test_the_committed_evidence_record_is_the_pinned_one.
  It checks that the file exists and that its SHA-256 equals both
  loader.PREREGISTRATION_EVIDENCE_SHA256 and the literal pin 7bc5c3c2...b845.
  It checks that read_preregistration_evidence() accepts the record and
  that an explicit missing path still refuses ("no pre-registration").
  A copy one hex digit away, still well-formed, refuses as "not the
  committed record".
- test_a_frozen_load_needs_the_committed_record and
  test_preflight_refuses_while_the_evidence_record_is_absent force the
  no-record case by patching loader.read_preregistration_evidence to read
  an explicit missing path.
- scripts/track_u2_dry_run.py: registered_preflight_without_evidence now
  reads an explicit missing path in a fresh temporary directory, checked
  absent first. The output keys are unchanged.

No test is added or removed, and the five tier counts are unchanged
(18125/4261/1450/520/308). The loader, the registered runner and the
specification are untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/u2_u1_differential.py ran with the U1 base 9cee242 and the
candidate 2faf035 as clean detached checkouts, one Python 3.14.7
interpreter (pandas 3.1.0, numpy 2.5.4), seed 20260924, and an empty
PSID root. The result has all_equal true and all_checks_passed true,
and refusal expectations are met on both sides. The contract-check
digest equals the one in u2_m2_u1_differential_14d9576.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis MaxGhenis changed the title U2: commit the pre-registration evidence record (hashes only) U2: commit the pre-registration evidence record and test it as present Oct 24, 2026
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 24, 2026 16:11
The pre-registration evidence record pins pandas 3.0.6 with
future.infer_string True. scripts/u2_u1_differential.py was rerun in
that environment, using the same clean checkouts as before: base
9cee242 and candidate 2faf035. The environment was Python 3.14.7,
pandas 3.0.6 and numpy 2.5.4 (the step-4 pass's versions), with seed
20260924 and an empty PSID root.

Results:
- all_equal and all_checks_passed are true.
- The expected refusals hold on both sides.
- The contract-check digest is unchanged.

The report gains an `environment` block after `python`, read from that
interpreter after the run. With the block removed, the file is
byte-identical to the harness's output.

The pandas 3.1.0 run committed in a0fd18c gave the same results. It
differs only in the interpreter path, the temporary output directory
named in run.command and the dry-run digests that include it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Closing and reopening only to rerun CI on current master (4e20c6e4, after #582). The head is unchanged.

@MaxGhenis MaxGhenis closed this Oct 25, 2026
@MaxGhenis MaxGhenis reopened this Oct 25, 2026
@MaxGhenis
MaxGhenis merged commit 9967dfd into master Oct 25, 2026
22 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

The dynamics coordinator merged this at its reviewed head c9c0f4a20a1d37526acca7346851747e10e611eb (merge commit 9967dfd6eb26860f562696e82b1ab03f305a1c81), with --merge --match-head-commit.

Gates.

Independent review. GPT-6.1 Sol (20261024-121913-md577-review-c9c0f4a, the U2 lane's job): APPROVE at c9c0f4a2, no findings. It confirmed:

  • the scope is exactly five files and four commits over 0e462efa;
  • the protected files and the tier counts are unchanged;
  • the dry-run output at HEAD and at 0e462efa has matching JSON key paths.

Next. U2 §20 step 8, the #42 registration, is Max's decision. The registration binding is computed at this merge commit before the comment goes to him.

This branch was successfully deployed

1 active deployment
Preview — c9c0f4a2 Deployed Oct 24, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant