Warm-start coverage honesty + genuinely cold MC-error replicas (PR 161 blockers 1 and 2)

[oshaughnessy-junior]

Addresses the two remaining PR 161 blockers (cover_frac coverage, MC-error replica independence). Stacked on the merged #45.

[1] cover_frac does not provide full-prior support — confirmed

A FINITE set of uniform points occupies only the bins it lands in, so a seeded grid is not a superset of a cold start. Fraction of the [-5,5]^d prior box actually covered (cold = 1.0):

cover_frac	0.0	0.2	0.5	0.9
d=2	0.027	0.634	0.982	1.000
d=4	0.0015	0.028	0.104	0.620
d=6	6.3e-05	0.00087	0.0033	0.0287

At d=6 even cover_frac=0.9 leaves 97% of the box unsampled, so the docstring's claim that a warm-started integral "can never be MORE biased than a cold one" was false. Corrected in place with these numbers.

What actually protects production, which was undocumented. A portfolio warm start also narrowed the backstop member (V=1 -> 0.0033). I fixed that and then measured — and could NOT reproduce a bias in the default [AV, GMM] portfolio either way: with a deliberately displaced seed, |lnZ bias| <= 0.05 in every d=4/d=6 run in both arms. The reason is the GMM member's explicit uniform defensive component (gmm_defensive_frac, default 0.05) plus Gaussian tails, so q_mix never vanishes. In an ALL-AV portfolio, where every component is a hard-edged box, the same seed gives -1.0 to -6.8 nats.

So the invariant is "some member has support everywhere", not "member 0 is cold". AV is marked has_unbounded_support = False; a member is held cold only when every member has compact support. [AV, GMM] is unchanged.

Be clear about the limits. The cold backstop is cheap insurance, not a rescue: in the all-AV test it still gave -1.1 to -4.2 nats (n_eff 3-9), because a uniform member finds a sharp 6-D peak too rarely to carry the integral within budget. Coverage in principle is necessary, not sufficient. Detection of a mismatched seed is separate work, in progress.

[2] MC-error replicas were not adaptation-independent — fixed

Neither mcsamplerPortfolio, mcsamplerAdaptiveVolume nor mcsamplerEnsemble defines reset_sampling (only the AC/GPU sampler does), so the replica loop's reset was a no-op for them; and since portfolio.integrate_log does not call setup(), each replica reran on the previous replica's contracted grid and fitted GMM. Replicas that share the adaptation whose failure they exist to detect understate the MC error. The loop now calls clear_warm_state() first. AV live volume across 3 replicas:

before: 2.16e-07 -> 4.08e-08 -> 1.42e-08   (monotone contraction; not independent)
after:  2.16e-07 -> 2.17e-07 -> 2.04e-07   (each contracts independently)

Standalone AV was already cold, as its comment claimed — its integrate_log calls setup().

Gate

Shape gate vs rift_O4d @ 87d66b75: COMPARE_EXIT=0, 0 blocking regressions; flag-ON probe 0 regressions. 15 portfolio unit tests + 6 gate tests pass.

An earlier revision of this branch was rejected by the gate for disabling the AV warm start in [AV, GMM] (3 blocking portfolio_warm regressions, reading AV member V=1.000, live bins=1). That is what drove the corrected invariant above, and both directions are now pinned by tests.

Unrelated gate defect noticed

GMM mix_d6_n3_s303 is order-dependent: 66 / 119 / 104 across runs of the same unchanged base checkout (full matrix vs isolated), sitting right on the n_eff = 100 starvation floor. Not caused by this branch (nothing here touches mcsamplerEnsemble), but it can flip a blocking verdict at random and should be re-budgeted or dropped from the strict set. Filed separately rather than fixed here.

[oshaughnessy-junior]

Both findings fixed in 9d3f21b.

[P1] Full-support detection could discard the only backstop — fixed

Reproduced your exact case. [unrestricted AV, restricted GMM] reports _full_support_members == [0], yet the restricted GMM set _has_broad, so member 0 was warm-started and contracted to V=0.095 — nothing covering the prior box, which is precisely the silent low bias the mechanism exists to prevent.

Both of your requirements implemented: has_unbounded_support now defaults False and must be declared by each sampler, and only members that are both declared broad and still full-range (per _full_support_members) are counted.

mcsamplerEnsemble declares it as a property keyed on gmm_defensive_frac > 0, not as a flat True. The uniform defensive component is the actual guarantee — Gaussian tails underflow to exactly zero far from the mode, so they cannot be relied on — which means gmm_defensive_frac=0 correctly reports False and keeps a cold backstop. Verified at the decision point: 0.05 -> True, 0.0 -> False. AV declares False.

[P1] Replicas retained portfolio-level adaptation — fixed

You listed the fields precisely. Adds reset_adaptation(): clear_warm_state() plus a restore of portfolio_weights, portfolio_quality, portfolio_quality_nobs, portfolio_probe_ptr, portfolio_draw_iteration, portfolio_member_ness_history and the breakpoint schedule. Restores to each field's post-setup value, snapshotted during setup(), so an explicit initial weighting or breakpoint schedule is preserved rather than replaced by a hard-coded uniform. The replica loop calls it in place of clear_warm_state().

Includes the test you asked for: all portfolio-level state identical across replicas.

Verification

19 tests; the four added this round each verified to fail when its defect is reintroduced:

CAUGHT restricted-broad: member 0 was contracted even though the only other member is range-restricted
CAUGHT portfolio-state:  reset_adaptation left portfolio state carried over: {'portfolio_weights': ('[0.5, 0.5]', '[0.9, 0.1]'...

Gate

All portfolio rows PASS; flag-ON probe 0 regressions. One blocking row was reported — GMM mix_d6_n3_s303 — and it is not from this branch. Confirmed with the new confirm-on-fail step at 5 fresh seeds, where the two arms are bit-identical:

seed 988654: base=STARVED cand=STARVED (n_eff 93 vs 93)
seed 989654: base=STARVED cand=STARVED (n_eff 80 vs 80)
seed 990654: base=PASS    cand=PASS    (n_eff 119 vs 119)
seed 991654: base=STARVED cand=STARVED (n_eff 95 vs 95)
seed 992654: base=STARVED cand=STARVED (n_eff 96 vs 96)
-> NOT CONFIRMED (0 worse / 5 not-worse)

Worth noting for its own sake: that cell starves in 4 of 5 seeds, so its PASS at the default seed is the lucky draw and it has been a coin-flip gate on every branch. Tooling and the case for re-budgeting it are in #49; I have not changed the strict set or budgets here.

Flagged, not fixed here

mcsamplerEnsemble.bootstrap_from_samples rebuilds its integrator without the caller's gmm_defensive_frac, so a warm start silently restores the 0.05 default. Same class as the gmm_dict configuration loss fixed in #45. Left out to keep this PR scoped; it interacts with the capability check above, so worth doing next.

[oshaughnessy-junior]

Status + handoff — a decision is needed before this merges

Do not merge this PR until the question at the bottom is answered. The fixes are done and tested, but the last commit changes a shared sampler's default behaviour, and that is not mine to decide unilaterally.

The remaining P1 is fixed, and it was worse than reported

Confirmed exactly as described: add_defensive_component() is called only by fit_gmm_adaptive; the fixed-component paths (mcsamplerEnsemble.py:369-374, MonteCarloEnsemble.py:394-398) fit GMM.gmm(...) directly, and gmm_adaptive defaults to None (off). So the default configuration requested a defensive component and never installed one, while the portfolio contracted its AV member believing it was covered.

Additionally: gmm.score() floors its return at 1e-300 (gaussian_mixture_model.py:621). The member therefore always looks like it has density everywhere. That floor is a guard against log(0), not coverage — a sample landing there carries weight L*p/q ~ 1e300. Far corner of the prior box, fixed-component fit to a tight cloud:

	default (floored)	with defensive component
d=2	2.9e-273	1.3e-04
d=6	1.0e-300 (the floor itself)	7.6e-10

This retracts a claim I made earlier in this PR. I wrote that "the GMM member's unbounded support is what has protected production." It is not unbounded support, it is a numerical floor. The |lnZ bias| <= 0.05 I measured with a displaced seed means the far region was never sampled in those runs — not that it was covered. The integrator lore entry asserting the same thing needs the same correction.

What 34d3e770 does

1. add_defensive_component sets model.defensive_frac as a verifiable marker, so presence is checked rather than inferred.
2. Both fixed-component fit paths install it.
3. has_unbounded_support inspects the installed models: False if any trained group lacks the component, or if gmm_defensive_frac <= 0. While untrained it trusts the config — sound only because every fit path now installs it, which test_every_fit_path_installs_the_defensive_component pins so a new fit path cannot silently reintroduce the gap.

20 unit tests pass; the four from the previous round each verified to fail when their defect is reintroduced.

THE DECISION

34d3e770 changes the default GMM proposal: fixed-component fits now carry the 5% defensive component that gmm_defensive_frac's default has always promised but never delivered. This affects standalone-GMM users, not just the portfolio.

Arguments for: it is what the docstring intends, it bounds the importance weights (the Hesterberg rationale the function was written for), and without it the portfolio's coverage argument is false rather than merely unproven.

Arguments against: it is a shared sampler, and a 5% defensive weight dilutes a well-fitted proposal, costing some efficiency in well-behaved cases.

The alternative, if that is too invasive: drop 34d3e770, keep the accurate has_unbounded_support from 26dfcf98, and accept that the default [AV, GMM] portfolio then reports no full-support member and keeps its AV member cold on a warm start. Safe, but it costs the AV warm start entirely, and the gate's portfolio_warm case would need rethinking since it asserts the AV seed IS installed.

I have implemented the first option because it keeps the system coherent, and flagged it here rather than letting it pass as routine. Reverting to the second is one git revert plus a gate case update.

Gate

Running against rift_O4d @ 87d66b75 with the changed default; result to follow in this thread. The GMM rows are the ones to watch, since they now exercise a different proposal.

Still open, not started

• mcsamplerEnsemble.bootstrap_from_samples rebuilds its integrator without the caller's gmm_defensive_frac, silently restoring the 0.05 default. Same class as the gmm_dict loss fixed in helper: SNR-scaled chunk + cubic time interpolation; portfolio interval-narrowing plumbing; truth-known SNR study #45, and it now interacts with the capability check above.
• The GMM mix_d6_n3_s303 cell starves in 4 of 5 seeds and is mis-budgeted for the strict set — tooling and evidence in gate: confirm blocking regressions at fresh seeds before they block #49, decision left to review.

[oshaughnessy-junior]

The gate answered the question I asked — the global default change is too expensive

34d3e770 (defensive component on every fixed-component fit) failed the gate with 2 blocking regressions, and unlike the previous dispute these are genuinely mine. The effect is systematic across GMM rows, not a threshold flip:

row	base n_eff	with defensive component
d6_n1_s101	2036	1565
d6_n3_s202	39	15
d6_n3_s303	119	75 (blocks)
d8_n1_s303	448	210 (blocks)
d2/d4 rows	—	unchanged within scatter

A 5% broad component spends 5% of draws where the likelihood is negligible; at d>=6 that is a real efficiency loss. So the answer to the decision I posted above is no — do not change the shared default.

Superseded by e17a9c4: opt-in, portfolio-scoped

The install now sits behind gmm_defensive_all_paths, default False. mcsamplerPortfolio.setup() turns it on for its own members, because a portfolio member may be the mixture's only full-support component and the guarantee has to be real there. A standalone GMM user gets exactly the previous behaviour and none of the cost.

has_unbounded_support additionally returns False when neither install path is active (no opt-in and gmm_adaptive off), so it cannot promise coverage the configuration will not build.

Verified directly:

standalone GMM:   all_paths=False -> declares coverage False
portfolio member: all_paths=True  -> declares coverage True

20 unit tests pass. The gate for this shape is running; its result is not yet in. The numbers above price the superseded variant, not this one. Do not merge until that lands — if the portfolio rows regress, the fallback is to drop the defensive work entirely and have the portfolio hold its AV member cold, at the cost of the AV warm start.

State of the branch

commit	contents	gate
e459ec03	hold a member cold only when every member has compact support	green
26dfcf98	declare full-support explicitly; reset portfolio-level adaptation	green
34d3e770	defensive component on all paths (global default change)	2 blocking
e17a9c4	same, made opt-in and portfolio-scoped	pending

Also still open, unchanged: mcsamplerEnsemble.bootstrap_from_samples rebuilds its integrator without the caller's gmm_defensive_frac (and now without gmm_defensive_all_paths), so a warm start can silently drop both. Same class as the gmm_dict loss fixed in #45. It interacts directly with the capability check, so it should be the next thing done.

[oshaughnessy-junior]

Gate result for e17a9c4 (the opt-in, portfolio-scoped version): GREEN

COMPARE_EXIT=0
# blocking regressions (strict=['AV','GMM','portfolio_seq','portfolio_seq_nobs','portfolio_warm']): 0
PROBE_EXIT=0
# opt-in regressions: 0

The important check is that the shared sampler is genuinely untouched. 23 of the 24 standalone-GMM rows are bit-identical between arms, including every row the previous variant regressed:

row	base	candidate	previous (global) variant
d6_n1_s101	2036	2036	1565
d6_n3_s202	39	39	15
d6_n3_s303	119	119	75 (blocked)
d8_n1_s303	448	448	210 (blocked)

The single exception is d8_n3_s303 (26 vs 21) — both far below the n_eff=100 floor, BOTH-STARVED, non-blocking. I am not claiming perfect identity: that row is consistent with the worker-pool context sensitivity documented in #49, not with a code path difference, but I have not proven that and it does not affect the verdict.

All warm/sequential rows PASS, so the portfolio member still gets its coverage guarantee:

portfolio_warm     mix_d6_n1_s101/202/303   n_eff 3159 / 3368 / 5707   PASS
portfolio_seq_nobs mix_d2_n1_s101/202/303   n_eff 2032 / 1603 /  811   PASS
portfolio_seq      mix_d2_n1_s101_o+2.00    n_eff 6326                 PASS
AV_seq             mix_d2_n1_s101_o+2.00    n_eff 2432                 PASS

Summary of the whole PR

commit	contents	gate
e459ec03	hold a member cold only when every member has compact support	green
26dfcf98	declare full-support explicitly; reset portfolio-level adaptation	green
34d3e770	defensive component on all paths (global default change)	2 blocking — superseded
e17a9c4	same, opt-in and portfolio-scoped	green

All four review findings across three rounds are addressed, each with a regression test verified to fail when its defect is reintroduced. 20 portfolio unit tests + 6 gate tests pass.

34d3e770 is left in history rather than squashed away: it is the commit whose gate numbers justify the opt-in design, and losing that evidence would make the current shape look arbitrary.

Still open (not started)

mcsamplerEnsemble.bootstrap_from_samples rebuilds its integrator without the caller's gmm_defensive_frac or the new gmm_defensive_all_paths, so a warm start can silently drop both — which would quietly revoke the coverage guarantee this PR establishes. Same class as the gmm_dict configuration loss fixed in #45. This is the natural next task.

[oshaughnessy-junior]

Both lifecycle findings fixed, base conflict resolved, gate clean. One probe result needs explaining, and it is not from this branch.

[P1] Warm bootstrap disabled the portfolio's defensive opt-in — fixed

bootstrap_from_samples() re-runs setup() to rebuild the integrator as a single full-dim group, and did so bare: gmm_defensive_all_paths reset to False and the warm GMM refitted with no defensive component, after the portfolio had already decided on the strength of that flag that it was safe to contract AV. It now carries the defensive config forward from the existing integrator.

[P1] Updates absorbed the defensive component — fixed

Confirmed with the drift you predicted. _merge() blends component i with fitted component order[i] for every i and never consults self.adapt, so the broad component -- marked adapt=False precisely so it would be left alone -- was dragged toward the fitted cloud each update while defensive_frac stayed set and has_unbounded_support kept reporting coverage.

Measured far-field density at the box corner, d=4:

	before fix	after fix
after 1 update cycle	3.09e-07 -> 2.04e-08	3.09e-07
after 4 updates	(continues collapsing)	3.09e-07, constant

update() now detaches the defensive component, updates the real ones, and reinstates it. The defensive component's weight, mean and covariance are unchanged across 4 updates.

Tests

Your point that the tests only covered installation was right, and it is exactly why this survived. The two added here exercise the warm-bootstrap path and repeated updates, and assert on the actual far-field density and on the defensive component's own parameters -- deliberately not on the defensive_frac marker, since the marker is what stayed true while the guarantee evaporated. Both verified to fail when their defect is reintroduced.

Base conflict — resolved

Conflicted with #48 (escaped-mass diagnostic), which landed on rift_O4d. Structural rather than semantic: both sides added a method immediately before bootstrap_from_samples and the signatures differed. Resolution keeps both bodies with the single signature carrying keep_backstop_cold. 22 portfolio tests and all 5 of the diagnostic's own tests pass on the merged result.

Gate

COMPARE_EXIT=0, 0 blocking regressions. All warm/sequential rows PASS.

The flag-ON probe reported 1 opt-in regression (d4_n1_s303, PASS -> STARVED across most flag configs). It is not from this branch. Re-running the probe against the pre-lifecycle commit 999e85af reproduces it exactly -- same row, same verdict, # opt-in regressions: 1 -- while the gate run at that same commit had reported PROBE_EXIT=0. Same code, two runs, opposite verdicts, with weight_clip ON reading n_eff 422 in one and ~46 in the other.

I also tested and discarded my own first explanation: I suspected the now-persistent defensive component was costing efficiency, but forcing the opt-in on gave n_eff 116 versus 36 with it off -- it helps there, not hurts.

So this is the same near-threshold realization sensitivity documented in #49, now showing up in the probe. The probe has no confirm-on-fail step; on this evidence it needs one, and I would rather add that than let a coin-flip row sit against this PR. Filed as follow-up rather than widened into this branch.