integrators: stabilize the MC error estimate (tail diagnostic, replicas, disclosed budgets)

[oshaughnessy-junior]

The reported sigma_lnL is the pooled weight sample-variance — algebraically 1/ESS_hat − 1/n restated, computed from the SAME weights as the integral. It is tail-blind: a run that misses the likelihood peak reports BOTH a low lnL AND a small sigma, so the error bar fails conditionally on the run being wrong. A controlled lognormal-weight study shows the median underestimate grows 1.0→2.4× (tau 1→5), 90% coverage collapses to 0.24, and the reported sigma is ANTI-correlated with the actual error; dslice exports corroborate (sigmaL understates redraw scatter 3–18× at high SNR).

Changes

• statutils: pareto_khat_from_log (Zhang–Stephens GPD tail fit, PSIS xi convention), ess_from_log_weights, block_scatter_sigma (delete-one jackknife over per-chunk lnZ), bootstrap_lnZ_quantiles.
• mcsampler / mcsamplerGPU: record per-chunk lnZ; at finalize emit pareto_khat, n_ESS, sigma_lnZ_block, and (relative error > 0.3) bootstrap lnZ quantiles via dict_return. Fixes the _rvs[weights] append-onto-joint_s_prior bug.
• mcsamplerAdaptiveVolume: adds the missing Var(lnV) (stochastic live volume) and trunc_p^2 terms to the returned relative variance, components disclosed.
• ILE batchmode: floors sigma at the between-chunk scatter; prints k-hat (>0.7 ⇒ sigma is a lower bound) and bootstrap CI; opt-in --mc-error-replicas N (+sigma/khat/ESS triggers) reruns cold replicas in-process and reports the LINEAR mean with sigma = max(propagated, replica scatter). Default off — production behavior unchanged unless enabled.
• util_CleanILE: combines repeated evaluations by the ntot-weighted LINEAR mean with sigma = max(propagated, between-replica scatter). The previous 1/sigma^2 weighting overweighted exactly the replica that missed the peak (toy: −0.86 nat combine bias vs −0.26 linear).
• mcsamplerGPU adaptive-proposal bias fix (ff0a04ba) — root-caused and closed the "−0.23 nat invisible bias" flagged in the first version of this PR. The 1D marginals of the drawn samples matched the claimed p_s exactly, but E[prior/p_s] = 0.37 on fresh draws: ~63% of the prior volume sat in histogram bins with exactly zero proposal probability. A zero bin is an absorbing state (it can never be re-drawn), so the sampled support shrinks irreversibly and the integral silently loses the mass outside it — invisible to every within-run error estimate. Four changes:
  • compute_hist: clamp the uniform-mixture floor to HIST_FLOOR_LEVEL_MIN = 1e-2 so no bin reaches zero (production ILE's --adapt-floor-level 0.1 already exceeds this; the clamp only binds for smaller/unset floors).
  • integrate_log: adaptation weights are now the stored tempered importance weights exp(tempering_exp*lnL + ln p − ln p_s), so the weighted histogram estimates the FIXED target L^beta * prior — the contract integrate() and the ILE driver comment always documented. The old lnL + max(maxlnL,200) weights ignored tempering_exp and 1/p_s; being near-flat they made each histogram replay the previous proposal's sampling noise, a multiplicative random walk that collapsed the proposal onto a comb of surviving bins (61/100 per dim) and drove the truncation.
  • integrate_log: the n_adapt freeze test double-multiplied by n (already scaled at parse time), so adaptation never froze at any setting; no-kwarg default scaled consistently.
  • pdf_from_hist: bin-index clamp off-by-one (right-edge sample indexed out of range).

Validation

• 19/19 in the roboto-paper demos/error_estimate/run_validation.py (k-hat recovers GPD k=0.3/0.7 to ±0.04; all samplers emit diagnostics; CleanILE matches the new formula; non-adaptive AC unbiased).
• Adaptive AC bias (2D Gaussian, n=2000, neff=1000, 16 runs/config): −0.321 ± 0.011 → −0.003 ± 0.004 for all n_adapt in {5..100} and tempering_exp in {0, 0.1, 1.0}; final proposal support 56% → 100%; E[prior/p_s] on fresh draws 0.373 → 1.0002; runs now reach neff≈1000 instead of exhausting nmax at neff≈350. T4b now reports delta = −0.016 against an honest sigma_block ≈ 0.009.
• CI gate test_mcsamplerEnsemble_extended --as-test passes both variants unchanged.

🤖 Generated with Claude Code

[oshaughnessy-junior]

Auto-review: not clear to merge yet. This is a core integrator change (MC-error estimator paths in the standard, AV, and GPU samplers), so the mandatory CPU base-vs-candidate shape-recovery test is required. I verified a clean merge, changed-Python compilation, and test_mcsamplerEnsemble_extended.py --as-test locally.

However, this PR’s base predates the hard shape-gate suite and still has the eager optional-NF entry-point import issue, which prevents a valid strict portfolio gate without torch. GitHub also shows no CI checks for this head. Please rebase/depend on the branch with the hard gate and optional-plugin isolation, then provide a green CI run and standard base/candidate shape result before merge.

[oshaughnessy-junior]

Superseded by #38: same single commit rebased onto rift_O4d (which now carries the hard shape-gate suite and optional-plugin isolation), retargeted to rift_O4d per review, with the required base-vs-candidate shape-recovery gate attached there (0 blocking regressions; base 8 strict failures vs PR 4 -- the delta is honest error bars widening sigma-scaled tolerances, biases byte-identical; details in #38). Closing in favor of #38.

🤖 Generated with Claude Code