integrators: stabilize the MC error estimate (tail diagnostic, replicas, disclosed budgets)

[oshaughnessy-junior]

integrators: stabilize the MC error estimate (tail diagnostic, replicas, disclosed budgets)

Supersedes #37, rebased onto rift_O4d per review (base now includes the hard shape-gate suite and optional-plugin isolation). Same single commit, cherry-picked; one trivial conflict vs the AV warm-start block, resolved keeping both.

What this changes

The reported sigma/L is the pooled weight sample-variance -- algebraically 1/ESS_hat - 1/n restated, computed from the SAME weights as the integral. It is tail-blind and fails conditionally on the run being wrong (a run that misses the peak reports BOTH low lnL AND small sigma). Changes:

• statutils: pareto_khat_from_log (Zhang-Stephens GPD tail fit, PSIS xi convention), ess_from_log_weights, block_scatter_sigma (delete-one jackknife over per-chunk lnZ), bootstrap_lnZ_quantiles.
• mcsampler / mcsamplerGPU: per-chunk lnZ recorded; finalize emits pareto_khat, n_ESS, sigma_lnZ_block, lnZ_ci90 via dict_return; fixes _rvs['weights'] appending onto joint_s_prior.
• mcsamplerAdaptiveVolume: adds the missing Var(lnV) (live volume is a stochastic product of per-cycle binomial survival fractions) and trunc_p^2 terms to the returned relative variance, disclosed in dict_return.
• ILE batchmode: floors sigma at the between-chunk scatter; prints k-hat and bootstrap CI; opt-in --mc-error-replicas (+ sigma/khat/ESS triggers) reruns cold replicas in-process, combined by the LINEAR mean with scatter-based error. Default off.
• util_CleanILE: replaces inverse-variance combination (which overweights exactly the replica that missed the peak: correlated low-lnL/low-sigma) with the ntot-weighted linear mean and max(propagated, scatter) error.

Merge-gate evidence (this head = junior/rift_O4d @ cbb67cc + 1 commit)

• Fast CI: test_mcsamplerEnsemble_extended.py --as-test --n-max 100000 exit 0, and --use-lnL variant exit 0 (CPU, CUDA_VISIBLE_DEVICES="").
• Shape-recovery gate (standard preset, 96 runs, CPU, same seeds both sides):
  • base rift_O4d: 8 strict failures, 5 warn-only, 45 starved
  • this PR: 4 strict failures, 5 warn-only, 45 starved
  • compare_shape_results.py: 0 blocking regressions (strict = AV, GMM); 5 IMPROVED(fail->pass); 4 PREEXISTING-FAIL (fail on both).
• Honest reading of the "improvement": the per-target AV lnZ biases are byte-identical between base and PR (same seeds; the sampling path is untouched) -- e.g. mix_d8_n1_s101 bias +0.642 on both sides. What changed is the tolerance, which the gate scales by the reported sigma: with the previously-missing Var(lnV) + trunc_p^2 terms disclosed, several borderline targets now pass because the error bar is honest, not because the bias shrank. The residual AV lnZ biases (d4_n1_s202 -0.289, d6_n1_s101 +0.528, d8_n1_s101 +0.642) are pre-existing and are exactly the class of error this PR's k-hat flag + replicate-when-loud machinery is designed to expose downstream.
• Pre-existing (both sides, per policy ticketed not blocking): GMM mix_d2_n1_s101 shape failure; AV lnZ biases above.
• Gate artifacts on CIT: ~/RIFT_develUWM/src/research-projects-RIT/.claude/worktrees/{base_shape.json,pr_shape.json,base_shape.log,pr_shape.log} (run via run_shape_recovery.sh, SHAPE_JOBS=8).

Validation beyond the gate (paper repo demos/error_estimate/): 19/19 unit/e2e checks incl. GPD k-hat recovery on known tails; log-normal toy quantifying the conditional underestimate (coverage 0.90->0.24 over tau=1..5; inverse-variance combine bias -0.86 vs -0.26 nats for linear mean).

🤖 Generated with Claude Code

[oshaughnessy-junior]

2026-08-16 00:42 UTC read-only interhost monitoring: PR #94 (b775c6de over rift_O4d@364a22fd), PR #96 (652731b6 over the same base), and PR #97 (2db91e67 over rift_O4d@d904e72d) are draft OPEN/CLEAN. Each exact head has successful private dispatch and CI (#94 31901722562/31901723554; #96 31823833402/31823833384; #97 31907523507/31907523919), but no exact-head review. The Stage-8 plan-only observer outcome is still absent; active-developer interlock preserved. No route, queue, worker, source, checkout, review, push, merge, permission, UUID, or diagnostic mutation occurred. Evidence: activities/2026-08-06-PRReviewSessionRouter/runs/2026-08-16-interhost-pr94-pr96-pr97-monitoring-0042.md. Next: monitor only new heads or exact-head review outcomes; do not run the diagnostic until the observer outcome is recorded.