Skip to content

probe: confirm flagged opt-in rows at fresh seeds before they fail the run - #54

Merged
oshaughnessy-junior merged 2 commits into
rift_O4dfrom
rift_O4d_probe_confirm
Aug 8, 2026
Merged

probe: confirm flagged opt-in rows at fresh seeds before they fail the run#54
oshaughnessy-junior merged 2 commits into
rift_O4dfrom
rift_O4d_probe_confirm

Conversation

@oshaughnessy-junior

Copy link
Copy Markdown
Owner

Applies #49's confirm-on-fail policy to the flag-ON probe, which was the one gate component still able to fail a PR on a coin flip.

Problem

probe_portfolio_optin_flags.py reuses the shape gate's thresholds and its evaluate(), so it inherits the same near-threshold realization sensitivity -- but unlike compare_shape_results.py it had no confirmation step.

Measured: adaptive_alloc ON / d4_n1_s303 reported base=PASS flag=FAIL on four consecutive gate runs during #51, reproduced identically against an unrelated base checkout, and the same cell has read n_eff 422 (PASS) and 46 (STARVED) on identical code. Re-running the same seed reproduces the same false verdict; only fresh seeds separate "the flag broke this" from "this cell responds to its realization".

What it does

A flagged row is re-run at fresh seeds with the flag off and on, and reported as a regression only if the flag arm is worse in a majority. Fails closed, matching confirm_regressions.py:

  • a flag arm that produces no record counts AGAINST the flag (crashing is worse than passing);
  • too few usable pairs is INCONCLUSIVE with a nonzero exit, never a silent clear;
  • --no-confirm restores the previous immediate-fail behaviour.

Flags: --confirm-repeats (default 5), --confirm-seeds, --confirm-min-valid, --no-confirm.

One rule, not two

The probe's regression rule is now a single is_probe_regression() used by both the summary and the confirmation. It deliberately differs from compare_shape_results.is_blocking: this probe tolerates flag=STARVED, since an opt-in path may trade efficiency on a target the default already resolves. That is exactly why it is defined once here rather than copied -- two representations of one rule drifting apart produced most of the findings in the #47/#51 series.

Verified live

The row that had failed every run for four rounds now clears, and the two arms are bit-identical at every fresh seed -- which independently confirms the flag never touched that cell:

# flagged rows: 1
  adaptive_alloc ON d4_n1_s303
     seed 988654: base=STARVED flag=STARVED (n_eff 36 vs 36)
     seed 989654: base=STARVED flag=STARVED (n_eff 84 vs 84)
     seed 990654: base=FAIL    flag=FAIL    (n_eff 131 vs 131)
     -> NOT CONFIRMED (realization noise) (0 worse / 3 not-worse)
# confirmed opt-in regressions: 0          [probe exit 1 -> 0]

At seed 990654 both arms FAIL, so that target is marginal on its own merits rather than because of the flag -- consistent with the d6_n3_s303 finding in FOLLOWUPS.md.

Tests

test_probe_confirm.py, 6 checks, all aimed at the direction that ships a bug (a false clear of a real opt-in regression; a false block only costs a rerun): noise clears, a reproducible flag failure confirms, a flag arm producing no record counts against the flag, zero valid pairs is inconclusive, a minority-worse seed does not confirm, and the rule tolerates STARVED but not FAIL.

Also

Adds FOLLOWUPS.md recording the outstanding integrator items, since issues are disabled on this fork -- the mis-budgeted GMM mix_d6_n3_s303 strict row (starves 4 of 5 seeds; needs a budget/strict-set decision, not code) and an audit of _rvs consumers that prefer a cached column over the canonical components. Happy to move these to the upstream tracker or drop the file if issues get enabled here.

…e run

The flag-ON probe reuses the shape gate's thresholds and its evaluate(), so it
inherits the same near-threshold realization sensitivity -- but unlike the
comparison it had NO confirmation step, so a single noisy row failed a PR.

Measured: `adaptive_alloc ON / d4_n1_s303` reported base=PASS flag=FAIL on four
consecutive gate runs during PR #51, reproduced identically against an unrelated
base checkout, and the same cell has read n_eff 422 (PASS) and 46 (STARVED) on
IDENTICAL code.  Re-running the same seed reproduces the same false verdict; only
fresh seeds separate "the flag broke this" from "this cell responds to its
realization".

Flagged rows are now re-run at fresh seeds with the flag off and on, and reported
as a regression only if the flag arm is worse in a majority.  Fails closed, like
confirm_regressions.py: a flag arm that produces no record counts AGAINST the
flag, and too few usable pairs is INCONCLUSIVE with a nonzero exit rather than a
silent clear.  --no-confirm restores the previous immediate-fail behaviour.

The probe's regression rule is now ONE function (is_probe_regression) used by
both the summary and the confirmation.  It deliberately differs from
compare_shape_results.is_blocking -- this probe tolerates flag=STARVED, since an
opt-in path may trade efficiency on a target the default already resolves -- so
it is defined once here rather than duplicated, which is the failure mode that
produced most of the findings in the #47/#51 series.

Verified live: the d4_n1_s303 row clears at 3 fresh seeds with the two arms
BIT-IDENTICAL (36/36, 84/84, 131/131), probe exit 1 -> 0.  At one seed both arms
FAIL, so the cell is marginal on its own merits, not because of the flag.

Adds test_probe_confirm.py (6 checks, all aimed at the direction that ships a bug
-- a false clear of a real opt-in regression).  Also records the outstanding
integrator follow-ups in FOLLOWUPS.md, since issues are disabled on this fork.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants