Catch rift_O4d up with oshaughnessy-junior/rift_O4d (integrator/calmarg/convergence payload + CHANGES.rst since rc1) - #170
Merged
Conversation
--limit-declination and --limit-inclination were SILENTLY IGNORED whenever the
corresponding cosine sampler was in use. In integrate_likelihood_extrinsic_batchmode
the parsed limits were stored in param_limits[...] as radians, but only the plain
(angle) branches consulted them: the `--declination-cosine-sampler` and
`--inclination-cosine-sampler` branches hardcoded left_limit=-1, right_limit=1 and
never looked at param_limits. No error, no warning, no narrowing -- the run simply
sampled the whole sphere / whole iota range. Since the cosine samplers are the
production default in the extrinsic-export campaign, a requested sky box collapsed
to RA-only narrowing, which for a high-SNR event is often not enough to pull the
integrator out of the eff_samp~1 regime the box was meant to fix.
The fix transforms the limits into the coordinate that is actually sampled. The
conventions are read off the likelihood closures in the same file
(`dec = pi/2 - arccos(z)`, `iota = arccos(z)`):
declination: sampled variable is z = sin(dec) (polar_theta = pi/2 - dec, uniform
in cos(polar_theta)). sin() is INCREASING on [-pi/2, pi/2], so the limit order
is PRESERVED: [lo, hi] -> [sin(lo), sin(hi)].
inclination: sampled variable is z = cos(iota). cos() is DECREASING on [0, pi],
so the limit order SWAPS: [lo, hi] -> [cos(hi), cos(lo)].
Prior semantics are preserved exactly. The cosine branches keep the FULL-RANGE
constant prior density 1/2 in z (previously they reused the sampling pdf object as
the prior, which was the same thing only because the range was [-1,1]); it is
deliberately not renormalized over the box, so the prior mass removed by a box is
0.5*(z_hi - z_lo), identical to the non-cosine branch where prior_pdf is
0.5*cos(dec) resp. 0.5*sin(iota) over the same physical box. The isotropic prior
is therefore unchanged and the two samplers give the same posterior and the same
lnZ for the same physical box.
Also in this change:
* The non-cosine dec/iota branches now use a properly TRUNCATED sampling pdf and
cdf_inv when a limit is given. Previously they set left/right_limit but kept the
full-range dec_samp_vector / cos_samp_vector, so mcsampler / mcsamplerGPU (which,
unlike the AV sampler, do use pdf and cdf_inv) would still draw outside the box.
With no limit requested the legacy functions are used unchanged.
* Limits are validated: a malformed 'LO,HI', an empty or inverted range, or a range
that does not overlap the physical domain now fails loudly instead of producing a
degenerate sampler. Dec/iota ranges are clipped to [-pi/2,pi/2] resp. [0,pi].
* --limit-right-ascension / --limit-declination are now rejected together with
--internal-sky-network-coordinates: that option samples RA/dec in a rotated,
network-aligned frame, so an equatorial sky box would silently select the wrong
patch of sky.
* Fixed an adjacent silent bug in the scalar time-marginalized likelihood closure,
which zipped the RAW sampled `inclination` instead of the converted `incl`; under
--inclination-cosine-sampler that fed cos(iota) to the waveform as an angle.
* Help strings updated: the "Assumes the plain (non-cosine) dec/incl samplers"
caveat is dropped, and the options now document that limits are always given in
radians of the physical angle.
New helpers live in RIFT/integrators/mcsampler.py (clip_angle_limits,
cosine_sampler_limits, ret_dec_samp_vector, ret_dec_samp_cdf_inv_vector,
ret_cos_samp_vector, ret_cos_samp_cdf_inv_vector) so they are importable and
testable independently of the backend the run selects.
test/test_limit_cosine_samplers.py (22 tests) covers: the dec order-preserving and
the iota order-SWAPPING transform (with an assertion that fails for the naive
[cos(lo), cos(hi)]); clipping; empty/inverted/non-finite ranges raising; the
truncated samplers reducing to the legacy ones over the full range; identical
restricted support and identical prior mass in both samplers; AV-style
(volume x prior) equivalence and identical posterior shape; end-to-end
MCSampler.integrate agreement between the two parameterizations for both a flat
and a peaked integrand; and a source-level guard against reintroducing the
hardcoded [-1,1] range.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nly when conditioning Follow-up to 02e668e, addressing review on the upstream PR: the odd-to-even resize of the ChooseFDModes grid is unconditional, so the removal of the surviving unpaired -fNyq bin must not be guarded by not(no_condition) -- otherwise no_condition=True could still exhibit (l,±m) asymmetry for models with support at Nyquist. Guard on the truncation itself instead. Verified: conditioned and no_condition paths both at machine precision (3.3e-16 / 3.5e-16 worst conjugate-pair residual, XHM nonprec control). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e run The flag-ON probe reuses the shape gate's thresholds and its evaluate(), so it inherits the same near-threshold realization sensitivity -- but unlike the comparison it had NO confirmation step, so a single noisy row failed a PR. Measured: `adaptive_alloc ON / d4_n1_s303` reported base=PASS flag=FAIL on four consecutive gate runs during PR #51, reproduced identically against an unrelated base checkout, and the same cell has read n_eff 422 (PASS) and 46 (STARVED) on IDENTICAL code. Re-running the same seed reproduces the same false verdict; only fresh seeds separate "the flag broke this" from "this cell responds to its realization". Flagged rows are now re-run at fresh seeds with the flag off and on, and reported as a regression only if the flag arm is worse in a majority. Fails closed, like confirm_regressions.py: a flag arm that produces no record counts AGAINST the flag, and too few usable pairs is INCONCLUSIVE with a nonzero exit rather than a silent clear. --no-confirm restores the previous immediate-fail behaviour. The probe's regression rule is now ONE function (is_probe_regression) used by both the summary and the confirmation. It deliberately differs from compare_shape_results.is_blocking -- this probe tolerates flag=STARVED, since an opt-in path may trade efficiency on a target the default already resolves -- so it is defined once here rather than duplicated, which is the failure mode that produced most of the findings in the #47/#51 series. Verified live: the d4_n1_s303 row clears at 3 fresh seeds with the two arms BIT-IDENTICAL (36/36, 84/84, 131/131), probe exit 1 -> 0. At one seed both arms FAIL, so the cell is marginal on its own merits, not because of the flag. Adds test_probe_confirm.py (6 checks, all aimed at the direction that ships a bug -- a false clear of a real opt-in regression). Also records the outstanding integrator follow-ups in FOLLOWUPS.md, since issues are disabled on this fork. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eights cache Follow-up sweep from PR #51. Two real defects, both of the one-of-two-paths kind. [1] 'log_weights' does not mean one thing. mcsamplerPortfolio stores the true importance weight (lnL + ln p - ln p_s); mcsamplerGPU stores tempering_exp*lnL + ln p - ln p_s -- the ADAPTATION weight. That exponent is not 1 in production: helper_LDG_Events sets it from the SNR (helper_LDG_Events.py:1472/1477), and --no-adapt drives it to 0, removing the likelihood from the column entirely. The .dgrid and calibration-posterior exporters both PREFERRED that cache, so with the GPU/AC sampler they reweighted their output by L^(e-1) -- and by 1/L under --no-adapt. The extrinsic-proposal fit in the same file already derived for exactly this reason, with a comment naming the hazard; these two sites never got the same treatment. Adds ln_weights_from_rvs() as the single definition -- log components, then the linear mcsamplerEnsemble form with out-of-support rows at -inf, and an explicit exception when neither is present, because a loud failure beats a plausible wrong number in a science output. Five call sites now share it, including the two inline copies added in #51. The cache is not read on any weight path. [2] mcsamplerGPU.py:1194 appended new weights onto joint_s_prior instead of weights, corrupting that column from the second chunk on. mcsampler.py:571 carries the identical block WITH the fix and a BUGFIX comment; the GPU copy never received it. Reachable -- mcsamplerGPU:1536 reads _rvs['weights']. Fixed, with a comment naming its twin. Adds test_rvs_weight_derivation.py (5 checks): derives from components rather than the cache and asserts the cache was materially different; --no-adapt loses the likelihood entirely; a portfolio-style cache agrees but is still not read; the linear form sends out-of-support rows to -inf, not NaN; a cache-only record raises instead of guessing. Full verdict table for every writer and reader in RVS_CACHE_AUDIT.md, including one divergence NOT chased: mcsampler.py:1138 notes _rvs['weights'] is sorted as a side effect, and two callers already work around it by recomputing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_rvs audit: derive importance weights instead of reading the ambiguous log_weights cache
…al_convention=) Style follow-up from review of the upstream PR: the ascending center-DC grid for ChooseFDModes output was built inline in hlmoft, so any future consumer (e.g. a diagnostic script) would reinvent it or fall into the same trap. Add lal_convention=True/False to evaluate_fvals -- default False keeps the existing RIFT reversed packing byte-identical; True returns the LAL generator packing f[k] = deltaF*(k - npts//2), exact for even and odd lengths -- and document when each applies. hlmoft now calls evaluate_fvals(..., lal_convention=True). No behavior change: XHM nonprec conjugate-pair residual still 3.3e-16; default-convention output verified byte-identical on even-length series. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
probe: confirm flagged opt-in rows at fresh seeds before they fail the run
…inal draw Replace the weighted replace=False export draw (numpy successive sampling: indices return in draw order with the head enriched in high-weight points, so the first-N truncation biased the export at any N) with systematic resampling on the weight CDF: expected counts are exactly N*w_i/sum(w) at any N, the duplication is the minimum possible, and the output order is shuffled so truncation stays fair. New --posterior-unique-draw caps the draw at floor(sum(w)/max(w)) -- the exact bound below which a fair draw can be duplicate-free -- so the flagged output is simultaneously fair AND unique, with honest undersupply reported in a new +annotation_export.dat sidecar (n_requested, n_delivered, n_distinct, bound). pseudo_pipe adds the flag to the final CIP argument group only: the whole group, so a convergence-test abort partway through still publishes a unique fair draw; Z lines carry the flag into the run-to-convergence subdag; a final G group is left alone (the Gaussian-resampling executable has a strict parser without the option); the AMR arg-list path is skipped entirely. Internal iterations keep the fair draw with duplicates allowed, so successive iterations feed an unbiased convergence test. Supersedes #44 (unconditional replace=True put duplicates in the published product) and reworks #46 (kept the biased draw for the one product that is published, and its rewrite flagged G lines). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…did anything The third cached-column divergence flagged in the #55 audit, now identified. mcsampler.py:1138 and mcsamplerGPU.py:1559 carried the note "rvs['weights'] is *sorted* (side effect?), breaking test. Recalculated weights are not. Use explicitly calculated weights until sorting effect identified." The effect is real and its source is the save-P thresholding block (mcsampler.py:739-752): it reindexes EVERY _rvs column by cumulative-weight order, so surviving rows come out sorted by weight ascending. convergence_test_NormalSubIntegrals then splits them into `ncopies` CONTIGUOUS segments and assumes those are independent -- but sorted rows put the smallest weights in the first segment and the largest in the last. Measured on a reconstructed record, max/min segment mass is 1.15e3 weight-ordered versus 1.83 in draw order: the sub-integrals differ by construction and the normality check is meaningless. The workaround does NOT help, and this is the part worth recording: the components were permuted by the same reindexing, so recomputing from them reproduces the identical order. Measured, cached and recomputed weights are both 100% ascending. The comment's claim that recalculated weights are unsorted is simply false. What survives the permutation is `sample_n`, written immediately before that block precisely as an iteration number. Both copies of the test now reorder by it before segmenting. SEVERITY, stated honestly: every production caller of this test is commented out (util_ConstructIntrinsicPosterior_GenericCoordinates.py:693 and two siblings); the only live caller is test/test_like_and_samp.py. So this was a latent trap rather than a live defect -- but a misleading comment plus an ineffective workaround is exactly how one survives, and re-enabling the test would have silently produced nonsense. Adds test_convergence_sample_order.py (3 checks), including one that pins WHY the old workaround was ineffective so it is not reinstated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… cast Review [P2] on PR #56: the helpers cast weights to float64 before normalizing, so finite longdouble weights above float64's range (RIFT builds export weights in extended precision on x86_64) overflowed to inf -- the bound could then fail on a NaN-to-int conversion and the resample could see invalid probabilities. The pre-#56 code normalized in the original dtype precisely to avoid this. Restore that discipline in one place (_normalized_probabilities): scale by the maximum weight in the input dtype so every value lies in [0,1], validate finite/nonnegative weights with a positive maximum, and only then cast the normalized probabilities to float64. Regression tests: an extended-precision case with weights above DBL_MAX (skipped on platforms whose longdouble has no extra range) and explicit rejection of empty/all-zero/negative/non-finite weights. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d it; deprecate the skymap branch Review of #57 was right, and measuring it turned up a second problem in the same place, so the source hunks are withdrawn. The diff against upstream in mcsampler.py / mcsamplerGPU.py is now COMMENT-ONLY. [P2, confirmed] A reused sampler appends rows to the weight columns while sample_n still has the previous length -- measured 3000 weights against 1500 ids -- and numpy fancy-indexing then silently returns the shorter array, dropping every new sample. [not raised, found while checking] Even after the threshold block re-runs and the lengths agree, reordering by sample_n does NOT restore draw order: it is (re)created as arange(len(...)) against an ALREADY-permuted array, so it encodes the order at the start of that call. Measured after two integrate() calls, 0.752 ascending (draw order is ~0.5) and a segment ratio of 373 rather than 1.83. So the proposed fix would have been partly ineffective even with a length guard. A real fix needs stable ids assigned where samples are APPENDED, in every sampler. This test has no live callers -- every production call site is commented out, only test/test_like_and_samp.py uses it -- so that plumbing is not justified. What ships instead: the false claim is removed (the old note said recomputing from components avoids the sorting; measured, cached and recomputed are BOTH 100% ascending), and the caveat now states what is actually true, including what anyone re-enabling the test must do first. Tests keep the evidence: 5 checks now, including both reuse failure modes. The draw-order test is written so it FAILS if the samplers ever grow stable append-time ids, which is the signal to revisit the caveat. Separately: flags the bayestar skymap branch as deprecated. It is unused in production for ~a decade and is the only place n-chunk drops to 500, making it the least-tested regime for anything scaling with chunk size -- notably --portfolio-weight-clip, whose tau = C*sqrt(n_chunk)*mean(w) is ~4.5x more aggressive there than at the n_chunk=1e4 every clip test used. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review [P3] on PR #56: floor(1/max(p)) took a float64 reciprocal of the normalized maximum, which rounds the exact bound down by one whenever roundoff lands just below an integer (93 equal weights -> 92), needlessly undersupplying the final draw and mislabeling the annotated bound. Compute floor(sum(w/max(w))) directly in the input dtype instead; validation moves to a shared _validated_scaled_weights helper. Regression: equal-weight bounds at n=3/93/1000 and the longdouble beyond-DBL_MAX case must all return n exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ests prove The module header still said sample_n 'is what survives' the permutation -- written before the reuse tests showed it is stale-and-short between calls and does not encode draw order after one. It now states both failed repairs, the measurements, and that this module is documentation only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…info skip Review [P2] on PR #56: the equal-weights exactness test evaluated np.longdouble(10.0)**400 unconditionally; on platforms where longdouble has only float64 range that is inf, unique_draw_bound correctly raises, and the test fails on platforms RIFT supports. The beyond-DBL_MAX assertion now lives in the extended-precision test after its finfo skip; the np.ones assertions stay unconditional. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_unique_final CIP: fair posterior export via systematic resampling + unique capped final draw
…ive-alloc regression
BUDGET. GMM mix_d6_n3_s303 sits on the n_eff=100 floor at the matrix budget, so
as a merge-BLOCKING row it is close to a coin flip. Measured over 8 fresh seeds:
budget n_eff min/med/max clears 100 median |bias|
x1 59 / 105 / 159 5/8 0.014
x2 105 / 169 / 214 8/8 0.010
x4 209 / 293 / 473 8/8 0.012
Bias is flat, so this is threshold margin rather than a defect.
Neither option previously proposed was expressible: the matrix budget is
nmax_per_dim*d for EVERY cell and --strict-samplers is per-SAMPLER, so "re-budget
this cell" and "drop this row from strict" both needed a mechanism that did not
exist. Adds CELL_BUDGET_MULT, in the same shape as the existing per-case
WARM_CASES budgets, and sets this cell to x4. x2 clears 8/8 but its MINIMUM
(105) is 5% above the floor -- not a margin worth trusting for a row that has
read 66 and 119 on unchanged code. x4 gives min 209 (2.1x) for ~4% of the gate's
evaluations, being one cell of ~96.
ADAPTIVE-ALLOC. Re-running the probe confirmation on the FIXED probe reverses a
claim I made repeatedly across #47/#51/#55: the `adaptive_alloc ON / d4_n1_s303`
row is a CONFIRMED opt-in regression, not realization noise. The earlier "not
confirmed" came from the probe's patching bug, which compared the flag against
itself and could not report "worse". At 5 fresh seeds the flag arm fails at
EVERY seed, frequently with HIGHER n_eff than the default arm -- so it degrades
posterior SHAPE, not efficiency. Opt-in, default off, never set by the pipeline,
so production is unaffected; recorded as FOLLOWUPS item 4 with the scoping work
needed before the flag could ever be promoted.
Also marks FOLLOWUPS items 1-3 resolved with their outcomes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…anary Fix container canaries after setuptools 84
In the per-iteration cip-args-list loop, a G-prefixed group set cip_exe_here (worker exe) to opts.cip_exe_G but never updated cip_exe_here_master, which is what the master .sub file is written with. write_CIP_sub falls back to util_ConstructIntrinsicPosterior_GenericCoordinates.py when exe is None, so without --cip-explode-jobs a G group silently ran the standard CIP instead of util_ConstructIntrinsicPosterior_GaussianResampling.py; with explode jobs the (--not-worker) master also ran the wrong exe. Set cip_exe_here_master alongside cip_exe_here for G groups. GaussianResampling accepts the master-only flags (--not-worker, --n-eff, --n-max, --n-output-samples), so both explode and non-explode modes are safe. Regression check: test-build.sh section 4 renders a --use-gauss-early pseudo_pipe config and asserts the G group's master AND worker subs both use GaussianResampling while non-G groups keep the standard CIP. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_budget convergence test: document the weight-order trap (documentation only; no functional change)
Review follow-up (PR #61): the unconditional cip_exe_here_master override defeated --cip-explode-jobs-flat, whose documented behavior is a /bin/true no-op master (no shared fit to save). Restore the flat-mode else branch that was active when the G option landed (79a5627) and commented out in 9d68201, ordered after the G assignment so flat wins for every group -- matching the non-per-iteration CIP.sub, which already goes /bin/true in flat mode. test-build.sh section 4 now covers both regimes: 4a non-flat (ini variant with cip-fit-method=gp; the ini section overrides the command line, so --cip-fit-method cannot be used): G master AND worker use GaussianResampling, non-G master keeps the standard CIP. 4b flat (ref ini default rf fit): every master is /bin/true (CIP.sub doubles as the flat witness), the G exe appears only in the worker sub. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ster_sub BUG: G-group CIP master sub used the standard CIP exe, not --cip-exe-G
…ling adaptive_alloc probe rows Two review findings on the CELL_BUDGET_MULT change: * The override was applied in shape_recovery.main() only, so the pytest entry point (test_shape_recovery.py, which the suite documents as an equivalent way to run the matrix) still used nmax_per_dim*ndim and left the very cell the table exists to fix starved. Both paths now go through one cell_budget() helper; verified they agree. * It also rescaled an EXPLICIT --nmax-per-dim, whose help text promises nmax = this * ndim. The x1/x2/x4 budget study that produced the x4 figure passes --nmax-per-dim, so it would have silently been measuring x4/x8/x16. An explicit --nmax-per-dim now disables the table and says so; --no-cell-budget-mult disables it at preset defaults. Separately, the confirmed --portfolio-adaptive-alloc shape regression (FOLLOWUPS item 4) is contained by commenting its two rows out of the probe, so the probe keeps working as a detector for the flags that pass. FLAG_CONFIGS hoisted to module level so the exclusion is greppable and so the confirmation-machinery tests inject a synthetic config list instead of coupling to whichever flags ship. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
At high network SNR the extrinsic export died with
===> FAILED ANALYSIS <====
zero-size array to reduction operation CUPY_CUB_MAX which has no identity
Probable reasons: SEOB nyquist or starting frequency limit or signal duration
on 11 of 12 replicates at rho_net 146.8, 5/12 at 102.8 and 4/129 at 72.1. The
attribution was wrong in every case: nothing about the waveform is involved.
ROOT CAUSE (instrumented reproduction, RIFT_AV_TRACE=1). Two defects, stacked.
1. The production likelihood UNDERFLOWS: exp() of a lnL more than ~745 nats below
the peak is 0 in float64, so it returns -inf. Over a cold extrinsic prior at
rho 147 that is 99996 of 100000 draws, so AV's live set is born holding a
handful of samples -- sometimes one:
[AV trace] cycle 1: drawn=100000 finite=4 neginf=99996 ninj=4
2. get_likelihood_threshold could then return a threshold >= max(lkl). With a
small live set and one dominant weight BOTH of its terms degenerate:
prob_stop_thr saturates at the MAXIMUM (every other weight underflows to
exactly 0, so the discard_prob quantile IS the top sample) while lkl_stop_thr
falls through the `len > nsel` test to the array MINIMUM. integrate_log
applies that as a STRICT `allloglkl > loglkl_thr`, discarding at least one
sample per cycle regardless of merit, so the live set ratchets to 1 and then
to 0 -- and the empty array reaches `xpy_here.max(allloglkl)`.
Both routes are reproduced deterministically in the new test, and both are
backend-independent: numpy raises the identical ValueError from the same line.
The traceback names CUPY_CUB_MAX only because production runs on the GPU.
CHANGES
RIFT/integrators/mcsamplerAdaptiveVolume.py
* get_likelihood_threshold: raise a named LiveVolumeCollapse on empty input,
and CLAMP the threshold strictly below max(lkl) so it can never discard the
entire live volume. A threshold at/above the max encloses zero probability,
contradicting the enc_prob=0.999 the function exists to maintain.
* integrate_log: admit only FINITE samples (matching the screen already in
update_sampling_prior_selfish); treat a cycle with no finite in-volume sample
as recoverable rather than fatal; keep a defensive guard against an emptied
live set; fail with a cause-naming error if nothing finite is ever found.
* live_volume_collapse_verdict(): report a degenerate contraction instead of
silently exporting one sample. dict_return gains live_volume_collapsed /
collapse_reason / n_live_final, and the run prints an [AV COLLAPSE] block.
* RIFT_AV_TRACE=1: opt-in per-cycle trace of the contraction, which is what
made the diagnosis possible.
bin/integrate_likelihood_extrinsic_batchmode
* the "Probable reasons: SEOB nyquist ..." hint was printed for EVERY exception
in the block. Make it conditional and name the integrator collapse when that
is the cause.
WHY THE VERDICT DOES NOT GATE ON k-hat. k-hat exceeds its nominal 0.70
"unresolved tail" threshold even in HEALTHY runs on this problem -- the 12
converged rho=51.4 replicates measure 0.819-1.605 -- so gating on it would have
false-flagged 12 of 12 good exports. It is also not always computable once the
live set is tiny. ESS separates the regimes with nothing in between (16.3-34.3
healthy vs 1.0-2.0 collapsed), so the verdict keys on ESS and live-set size and
merely quotes k-hat.
VALIDATION (12 replicates per arm, zero-noise injections at a fixed intrinsic
point, production extrinsic config, A100):
rho_net crashed before -> after exports collapse reported
146.8 11/12 (92%) -> 0/12 12/12 11/12
102.8 5/12 (42%) -> 0/12 12/12 12/12
51.4 0/12 -> 0/12 12/12 0/12
No false alarms in the regime that already works: every rho=51.4 replicate
converged (eff_samp 8.0-9.7, ESS 16.3-34.3) and none was flagged. The one
unflagged rho=146.8 replicate converged cold for real (ESS 14.3, terminating on
the n_eff target at 2.25M evals). A rescue arm at rho 146.8
(--sampler-warmstart-retry-neff 5, seeds 9124-9135, 9124 being the seed on record
as crashing identically under rescue) is also 0/12 crashed.
INERT ON HEALTHY RUNS: five well-conditioned integrals at a fixed seed reproduce
lnZ, eff_samp, ESS, k-hat AND ntotal bit-identically against the pinned pre-fix
tree (ba2b38d). The posterior shape-recovery merge gate required by
RIFT/integrators/TESTING.md reports 0 blocking regressions with all 25 AV rows
bit-identical.
KNOWN GAP: the verdict covers the COLD pass only. The L0 warm-start rescue pass
aborts inside sampler.integrate on a pre-existing cupy/numpy defect
("[L0 auto-rescue] skipped ( Implicit conversion to a NumPy array is not allowed
... )") before reaching any diagnostics, so a degenerate warm start still passes
silently. Not a regression from this branch; tracked separately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two strands, developed in parallel in the same tree and validated together. === PR #63 review findings === 1. [P1] The production caller ignored the collapse verdict. The sampler set live_volume_collapsed but the ILE never read it, so a known-degenerate export entered downstream posterior assembly as an ordinary likelihood row -- strictly worse than the pre-fix crash, which at least kept it out. The ILE now reads the verdict, prints "*** LIVE VOLUME COLLAPSED ***", feeds it to the existing --mc-error-replicas trigger list, and offers --reject-collapsed-live-volume to drop the event entirely. That gate is OFF by default on purpose: dropping the event silently thins the posterior in an SNR-dependent way, which is the failure this whole branch exists to remove. Exported-and-loud is the better default; the flag is there when a contaminated point is worse than a missing one. 2. [P1] The empty-cycle check used the CUMULATIVE live-set size. `ninj` is measured after old survivors and new samples are combined, so `ninj == 0` only ever caught LEADING empty chunks. Once one sample survived, a later chunk contributing nothing sailed past it and re-thresholded the recycled live set. Reproduced before fixing: 20 live points decaying 19, 18, 17, 16, ... with ln V falling -0.05, -0.11, -0.16, -0.22, ... over chunks that each returned ZERO finite samples -- contraction, and a biased lnZ, on no new evidence at all. Now the admissions from the current chunk are counted before the append and contraction is skipped when that count is zero. Pinned by an invariant test: extra chunks that contribute nothing must not move lnZ or the live-set size. 3. [P2] The legacy empty-reduction classifier was too broad. Any exception whose text contained "zero-size array" and "no identity" was labelled an AV collapse, although the enclosing handler also covers waveform generation, data conditioning and the whole likelihood stack. The named LiveVolumeCollapse is now matched by isinstance, and the string fallback additionally requires the traceback to name mcsamplerAdaptiveVolume. === The L0 warm-pass gap (documented as known in the PR body) === Root-caused: integrate() writes an 'integrand' key into _rvs AFTER integrate_log returns -- i.e. after the arrays have been moved to the host and, if a fair draw ran, truncated. On a SECOND integrate() on the same sampler (exactly the warm-start rescue) that key is stale in both length and backend, integrate_log repopulates every other key but not it, and the fair-draw gather then indexes a host array with a device index array: "Implicit conversion to a NumPy array is not allowed". The pass aborted mid-way, so `res, var, neff, dict_return = sampler.integrate(...)` never completed and the ILE reported the COLD pass's lnZ/k-hat/ESS beside the WARM pass's exported samples. * drop the stale key on entry (as mcsamplerPortfolio.integrate_log already does); * build the fair-draw weights on the sampler's own backend and gather on the host; * snapshot _rvs rather than aliasing it, and restore the cold pass in full if the warm pass raises -- the L0 handler now says "*** FAILED ***", not "skipped". With the pass no longer aborting, the verdict reaches it -- and finds the OPPOSITE failure, which none of the existing rules could see: seeded from too few points the grid contracts onto a sliver, the integrand is flat across it, and the pass terminates in one cycle looking excellent (n_eff at target, small k-hat) while lnZ is short by the mass outside the sliver. Measured over 12 rho_net=146.8 rescue replicates: eleven seeded from 2000 puffed points warm-started at V = 7.5e-9..1.5e-8 and returned ln(Z/Lmax) = -27.0..-30.6; the one seeded from 2 points warm-started at V = 9.2e-36 and returned -80.7 -- about 50 nats low, with eff_samp 9789 of 10010. So the verdict gains a geometric rule, n_warm_seed <= ndim+1 (fewer points than a simplex cannot define a volume), with seed provenance carried through bootstrap_from_samples/save_state/ load_state, and the report distinguishes an over-contracted warm start from underflow because the two need opposite advice. === Validation === * test_av_empty_live_volume.py: 41 passed, 2 GPU-only skips. * Healthy-run bit-identity re-verified against the pinned pre-fix tree (ba2b38d) after both strands: all five well-conditioned integrals reproduce lnZ, eff_samp, ESS, k-hat and ntotal to every digit. * Shape-recovery merge gate (RIFT/integrators/TESTING.md): 0 blocking regressions, all 25 AV rows bit-identical to base, same 4/24/29 strict/warn/starved baseline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two P2 test defects from re-review, each verified by reintroducing the defect: * test_adaptive_alloc_is_excluded_from_the_probe_configs inspected the SOURCE of PB.main(), which after the FLAG_CONFIGS hoist contains only `configs = FLAG_CONFIGS` -- so it could never see a re-enabled row. Confirmed vacuous: appending the known-bad config to FLAG_CONFIGS left the old form passing. It now inspects FLAG_CONFIGS itself, and additionally asserts the probe still has >= 2 opt-in arms, since an emptied list would satisfy "adaptive_alloc absent" while testing nothing. * test_shape_recovery.py unpacked evaluate() as `ok, reasons` and asserted `ok`. evaluate() has returned a STATUS STRING since 6467ac9 -- "FAIL", "STARVED" and "ERROR" are all truthy, so the pytest entry point has passed on every outcome it exists to catch since the day it was written. Verified: the old assertion passes on ERROR, STARVED and all four FAIL paths. On STARVED the reviewer asked for a strict `status == "PASS"`. Deviating deliberately: the gate defines STARVED as non-blocking in absolute terms and gating only differentially (6467ac9, after whole d=8 rows legitimately starved at production budgets), and strict equality immediately reds the DEFAULT preset -- GMM d4_n2_s101 reads n_eff=42 at quick's 200k budget. FAIL and ERROR assert; STARVED skips with the n_eff in the message, so it stays visible in the pytest summary and is not counted as a pass. Absolute-vs-base starvation gating remains compare_shape_results.py's job. Recorded as FOLLOWUPS item 5. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second round of PR #63 review. [P1] Collapsed results were still indistinguishable downstream. The verdict only reached stdout; both --reject-collapsed-live-volume and --mc-error-replicas default off, after which a collapsed run writes an ordinary .dat/.grid/XML/posterior with no marker on it. The .dat and .grid schemas are POSITIONAL and read by CIP, so the marker cannot be a new column without breaking every reader. It goes in a sidecar instead: <output>_<indx>_integrator_status.json written on EVERY run beside the other artifacts, carrying collapsed (bool), collapse_reason, lnL/sigma_lnL/neff/ntotal and whichever of pareto_khat, n_ESS, n_live_final, n_empty_cycles, n_live_collapses, n_warm_seed(_rank) the sampler reported. Written unconditionally so "collapsed": false is an explicit statement rather than an inference from a missing file -- downstream can require the file and fail loudly against an integrator too old to write one. A failure to write it can never abort a run that otherwise succeeded. Verified on the real pipeline: a rho_net=146.8 export writes it beside the .dat and .xml.gz with "collapsed": true and the reason; a rho_net=51.4 export writes "collapsed": false with ESS 18.6, k-hat 1.31, 1000 live points. Rejection is still not the default, for the reason the branch exists: dropping the event silently thins the posterior in an SNR-dependent way. With the sidecar the default is now exported-and-labelled rather than exported-and-silent. [P2] Warm-seed adequacy was a row count, which is neither necessary nor sufficient. Duplicated or collinear points span the same degenerate subspace two points do and produce the identical near-zero-volume failure however many rows there are; and the n <= ndim+1 boundary flagged a simplex, which is exactly enough to define a volume in ndim dimensions. Both directions were wrong. The invariant is the AFFINE RANK of the seed cloud: the rank of the mean-centred points over the adaptive axes, each scaled by its box extent so the test is unit-free (a distance in Mpc and an angle in radians must not get different tolerances). _grid_from_points now records n_seed_rank and n_seed_dim alongside n_seed, they travel through save_state/load_state, and the verdict tests rank < dim. Rank subsumes the count anyway, since n points span at most n-1 affine dimensions. Where rank is absent (state written before this) the count fallback now uses the correct simplex boundary, n <= dim. Tests: a 500-point cloud confined to a plane in 6-D is measured rank 2 and flagged; a full-rank cloud is not; ndim+1 independent points are not; duplicated and collinear seeds are flagged whatever their row count; rank survives save/load. Two tests that had encoded the old ndim+1 boundary are corrected. 52 passed, 2 GPU-only skips. Healthy-run bit-identity re-verified against ba2b38d (lnZ, eff_samp, ESS, k-hat, ntotal identical on all five well-conditioned integrals); shape-recovery merge gate 0 blocking regressions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gate: per-cell budget for the mis-budgeted GMM row; exclude the confirmed adaptive-alloc regression from the probe
P1 (merge blocker): PrecomputeLikelihoodTerms no longer changes its return arity. The two fused-calmarg self-term cross-term structures are returned ONLY when the new keyword return_calibration_crossterms=True; the default is the historical 6-tuple, so integrate_likelihood_extrinsic[_batchmode_lisa], ile_postproc_add_time, jax_ile and test/* keep working unchanged. The batchmode ILE and the calmarg tests opt in. P2: the per-realization self-term fix now applies for n_cal==1 too (a single applied calibration draw still carries rho_sq_c=<C_0 h|C_0 h>). Previously guarded to n_cal>1, so --calibration-n-realizations 1 silently used the cal-independent <h|h>. test_selfterm_reduction gains an n_cal==1 case: the fixed path matches the own-U brute force exactly and differs from the no-fix baseline by ~4.5 nats. P3: removed the dangling analyses/calmarg_selfterm_bias/NOTE.md path from the --calibration-conjugate-phase CLI help and in-code comments (the note lives in a sibling analysis repo, not here); updated the DiscreteFactoredLogLikelihoodViaArrayVectorNoLoop docstring and the ComputeModeIPTimeSeries comment to state that the template self-term is per-realization (rho_sq_c), not calibration-independent, when the |C_c|^2-weighted cross terms are supplied. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…|h> route
The per-realization self-term <C_c h|C_c h> corrects an amplitude bias whose evidence
impact scales as ~0.5 rho^4 sigma_A^2 -- negligible at low/moderate SNR unless the amplitude
envelope is wide. In that regime the complete route's extra precompute (SVD of {|C_c|^2}
plus rank-M(M+1)/2 weighted template blocks, both O(N_band)) can dominate the per-intrinsic-point
cost for long low-mass templates and large N_cal. --calibration-global-norm skips that precompute
entirely and uses the calibration-independent <h|h> for every realization (the calibration is still
applied to the data, so the marginalization is otherwise unchanged).
Implementation: a new PrecomputeLikelihoodTerms(calibration_self_term=True) gates only the
self-term build (return_calibration_crossterms still controls the return arity, so callers keep
unpacking the 8-tuple; the trailing structures come back None when the build is skipped). Verified
on GW240426: the flag reproduces the base-worktree global-norm lnZ within the AV Monte-Carlo scatter,
and toggling it moves lnZ by the self-term. Existing calmarg tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…within them The draw randomises which n_min points each worker file contributes and their order WITHIN that file, then appends each file's block in file order, so the merged output is [file0][file1]... That is invisible to a consumer that reads the whole file. The nested ILE does not: it reads a prefix. Measured on a live S240629by run, ILE evaluates rows 0-2999 of a 20,000-row merge, so with 25 workers x 800 it saw workers 0-3 and nothing from the other 21; at the more common 6 x 3334 the whole prefix sits inside worker 0. Which is exactly the failure the file header warns about -- "Important when merging files from many workers, to avoid accidentally using only the output from one of them." The hyperpipeline branch of write_joingrids_sub already gets this right, concatenating every shard and piping through shuf with the comment "shuffle so spokes are interleaved". This brings the XML path into line with its sibling rather than introducing a new policy. Verified on 25 real CIP worker files: row count unchanged at 20,000, no duplicates introduced, and contiguous same-worker runs go 25 -> 19,248 with all 25 workers represented in the first 3000 rows instead of 4. --preserve-block-order restores the previous behaviour exactly (25 runs, 4 workers) for reproducing earlier runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…osterior scale
The rescue built its warm seed from the cold pass's points within
--sampler-sequential-warmstart-deltalnL of the peak and puffed only when
len(seed) < 2. A count cannot see the failure it was standing in for: measured on
zero-noise injections, a 5-point seed at rho_net 102.8 had affine rank 2-4 of 6 and
a 2-point seed at rho_net 146.8 had rank 0 of 6. Both passed the count test and
both warm-started a live volume collapsed onto a degenerate subspace (V ~ 3e-06 and
~9e-36 against a healthy ~1e-08), which reports a fine n_eff over a sliver of the
support. The [AV COLLAPSE] report already printed that rank; it just did not act
on it.
Widening the seed window does NOT fix this and is not the answer: at 20x the
default the seed stayed at 2-5 points, because a collapsed cold pass never drew
more than a handful of finite-likelihood points for a window to admit.
* seed_affine_rank is now the ONE definition of the test, shared by the grid
builder (which records it) and the rescue (which now acts on it), so the guard
and the diagnostic cannot drift apart.
* build_warm_seed applies it and AUGMENTS the seed -- the real points are the only
direct evidence of where the peak is, and a real point outside the puff widens
the seeded volume to include it.
* The puff width was a hardcoded 1/200 of each PRIOR range, which knows nothing
about a posterior that narrows as 1/rho; on a known-lnZ 6-D target it truncated
by 0.8-8.3 nats. warm_seed_scale_from_finite_points instead recovers the
posterior covariance -- correlations included -- from every finite lnL the
collapsed pass already drew, via cov * (d+2)/(2D) for the underflow level set.
Recovered sigma/true: 1.01-1.23 per axis.
* Both tails are wrong, so the safety factor is small and measured: x2 gives mean
lnZ error +0.08 nats at ESS 52, against -8.5 at x0.5 (truncation) and -30.0 at
x12 (a cold start in all but name, which re-collapses).
New: --sampler-l0-rescue-puff-{scale,width-frac,factor}. puff-scale fixed with
width-frac 0.005 and factor 1 reproduces the previous puff exactly.
…aw subset Root cause, upstream of the rank test. integrate_log's fair draw takes n_extr = min(n_extr, 1.5*eff_samp, 1.5*neff) rows WITH REPLACEMENT and rebinds every self._rvs key to that subset -- a resample built for EXPORT. The L0 rescue then read _rvs for its seed. On the collapsed pass the rescue exists for, eff_samp ~ 1, so it was seeding from ONE row: measured at rho_net 146.8, "Fairdraw size : 1" while the live set held 1000. At rho_net 102.8 it is five rows, several of them the same point drawn twice -- which is exactly how a "5-point seed" came back with affine rank 2, and a "2-point seed" with rank 0 (two copies of one point). So the earlier reading of this failure -- that a collapsed cold pass never sampled more than a handful of finite-likelihood points -- was wrong. The points were drawn and retained; a resample for export threw them away before the rescue could look. It also explains why widening --sampler-sequential-warmstart-deltalnL did nothing at 20x: there were only n_extr rows left to admit at any window. integrate_log now stashes a bounded uniform subsample of the retained points (plus the peak row) before the overwrite, cleared on entry so a pass that raises cannot leave the previous point's peak behind for the next point's rescue to seed from. The rescue prefers it and falls back to _rvs for any sampler that does not keep one.
Covers the three defects and the boundaries around them, no GPU: rank (not count) decides; duplicated rows from a with-replacement fair draw really do have affine rank 0; the real points are AUGMENTED rather than replaced and can widen the seeded volume; the puff is clipped into the box; the scale estimator recovers the posterior width from the underflow shell and declines rather than guess from too few points; the retained points survive a fair draw that keeps one row, carry the peak, and cannot leak from one point to the next. Also pins that widening without limit is NOT safe (V -> O(1) is a cold start in all but name) and that --sampler-l0-rescue-puff-scale fixed reproduces the historical puff exactly.
…#78, P1) mcsamplerPortfolio.integrate_log drives its members through draw_simplified(), never through their integrate_log(), so NO MEMBER EVER BUILDS a reserve -- and the portfolio did not build one either. Every portfolio rescue therefore reached the _rvs fallback, after the portfolio's own pruning and fair draw had already cut it to ~1.5*n_eff rows sampled WITH REPLACEMENT: precisely the starvation this branch exists to end. Confirmed in the campaign logs that shipped with the branch. The two portfolio replicates whose rescue fired at rho_net 146.8 print no "seeding from N retained sample(s)" line at all, seed from 3 and 5 rows, and fall back from the 'auto' puff scale to the fixed prior fraction -- because three rows cannot define a 6-D covariance: [L0 auto-rescue] cold n_eff 5.0 < 5.0; re-running warm ... (2005 pts) [L0 auto-rescue] seed of 5 point(s) had affine rank 4/6: PUFFED ... (fixed scale, x2) [L0 auto-rescue] cold n_eff 2.2 < 5.0; re-running warm ... (2003 pts) [L0 auto-rescue] seed of 3 point(s) had affine rank 1/6: PUFFED ... (fixed scale, x2) And the failure mode is QUIET, not loud: those two were caught only because their tiny subsets happened to come back rank-deficient. A fair-drawn subset of d+1 distinct rows is full rank, sails through the guard, skips puffing entirely, and seeds another sliver that then reports a healthy n_eff. The portfolio now takes its reserve from the aggregate _rvs BEFORE the pruning block and the fair draw, and clears it on entry so a pass that raises cannot leave the previous point's peak for the next point's rescue. make_warm_seed_reserve is the one builder, shared with AV, so an L0 rescue gets the identical record whichever sampler ran; it also carries the two prior components, which the reserve needed anyway to be a complete record of the retained set. Tests: the portfolio really does leave a reserve larger than its fair-drawn _rvs; it is taken before both pruning and the fair draw; it cannot leak between points; and the premise itself is pinned -- if the portfolio ever starts routing members through integrate_log, the test that asserts it does not will say so.
…orker_merge util_RandomizeOverlapOrder: interleave across worker files (ILE was seeing only 4 of 25 workers)
…ate RNG (review #78) P1 -- THE CAP WAS DESTROYING WHAT IT WAS KEEPING. AV reaches make_warm_seed_reserve with only retained (finite) rows, but the PORTFOLIO's _rvs holds EVERY draw, and on the collapsed pass this machinery exists for essentially all of them are -inf. A uniform subsample over all rows keeps the finite ones in proportion -- almost none. Reproduced: 10 finite rows among 1,000,000 at a 20,000 cap survived as 2, the forced peak plus one lucky draw. build_warm_seed then sees a rank-0 core, warm_seed_scale_from_finite_points declines for want of points, and the puff falls back to the fixed prior fraction: the 0.8-8.3 nat truncation, with a healthy-looking ESS, that this branch exists to remove. Non-finite rows are now dropped outright before the cap. They are ballast for every consumer -- the seed core is `lnL > max - deltalnL`, the scale estimator filters to finite, and _lnZ_of_rvs filters to finite before averaging -- so this leaves that lnZ bit-identical while making the cap mean what it says. 10 of 10 now survive. No second stratum, deliberately: once the finite rows ARE the population, the cap only binds on a run with >n_max finite samples, i.e. a healthy one where the peak window is proportionally represented anyway (a 20,001-row reserve out of 4e6 gave a 239-point seed core). A top-lnL stratum would buy nothing there and would bias both the covariance estimate and any lnZ taken from the reserve, since neither would be a uniform sample of the level set. The peak row stays force-appended -- one row in n_max. P2 -- AN OPT-IN RESCUE THAT IS OFF MUST NOT MOVE A SEEDED RUN. The reserve is built unconditionally, including when --sampler-warmstart-retry-neff is unset and nothing will ever read it, and its subsample was drawn from the global numpy stream -- advancing it before the fair draw, before the exported posterior, and before every later event and replica. Confirmed by seeding numpy and comparing the next draw across a build. It now uses a private, deterministic RandomState: no global state consumed, and the reserve is itself reproducible. Four tests, each failing on the pre-fix code: rare finite rows survive the cap; the cap still binds and still keeps the peak on a healthy run; building the reserve leaves the global stream untouched (asserting first that the subsample path really ran, so the test cannot pass vacuously); and the subsample is reproducible.
…ild gate) P2a: the SVD self-term basis cache could return a stale basis when the PSD/band weights changed -- the fingerprint sampled base_weights2side only every 503 bins, so two different weight arrays agreeing on the sampled bins collided. Key on the FULL weights digest (blake2b) instead, and store a weakref to the draws so a hit re-verifies object identity (an id reused after GC cannot return another array's basis). P2b: --calibration-burn-in-neff sets n_cal=1 but keeps the full N_cal cross-term arrays, which made rho_sq_cal += rho_sq_det_cal a (1,npts) += (N_cal,npts) shape mismatch (caught, so the burn-in was silently discarded). Slice the weighted blocks to the first n_cal realizations in the reduction, so the self-term stays consistent with the rholm blocks the n_cal reduction actually reads. P2c: build the (potentially dominant) self-term cross terms only when the caller will receive them AND wants the complete route -- gate on return_calibration_crossterms as well as calibration_self_term -- so a six-value-API caller supplying calibration realizations no longer pays the SVD + rank integrations just to discard them. Verified: cache no longer reuses across weights (different-PSD basis differs; same-PSD hit returns the same object); n_cal=1 with N_cal cross-term arrays no longer raises and equals the realization-0 self-term exactly; alignment/reduction/backtest regressions pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Catch calmarg / self-term regressions automatically. New .travis/test-calmarg.sh runs the CPU gate -- precompute time-alignment + identity-cal cross terms, the loop/fused reduction vs a brute-force reference (incl. n_cal==1), the low-rank SVD self-term basis vs a direct band integral, and the cal-reduction backtest (default + distance-marg) -- wired as a 'calmarg-check' job in GitHub Actions and a 'calmarg_check' job in GitLab. .travis/test-calmarg-gpu.sh exercises the fused CUDA kernels (in_loop_C, default + distmarg) and runs on the GitLab gpu runner (appended to gpu_integration). Any nonzero exit fails the job; verified the CPU gate green end-to-end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fterm-fix # Conflicts: # .gitlab-ci.yml
…-rank L0 rescue: seed from the points the pass retained, and judge the seed by rank
…m-fix calmarg fused self-term fix: per-realization rho_sq_c = <C_c h|C_c h>
…se suites in CI P1 (review). A capped reserve is a Horvitz-Thompson sample: its LINEAR total is unbiased, but its LOGARITHM is not, and the L0 reject gate is a comparison of logarithms against a 0.5-nat threshold. When a few rare rows carry the weight, lnZ moves in discrete jumps depending on whether the subsample caught them. With two equally dominant rows among 200,000 at n_max=2000 only the force-appended peak is certain; the other is missed ~99% of the time, putting lnZ low by log(2) = 0.69 -- above the threshold. A cold reserve that fits under the cap, compared against an otherwise identical warm reserve that does not, would then reject a valid warm pass on nothing but subsample luck. The exact finite-population total is the one quantity a bounded record cannot rebuild afterwards, so it is now captured at build time, before the cap, as ln_sum_w_finite. The gate reads it (see the stacked branch) and never sees cap sampling error at all. Absent rather than wrong when the prior components were not supplied. P2 (review). None of these suites was reachable from CI. Neither .github/workflows/ci.yml nor .travis/test-integrate.sh invoked test_l0_rescue_seed.py -- and the same was true of test_av_empty_live_volume.py and of test_portfolio_fairdraw_backend.py, which shipped with an ALREADY-MERGED PR. So every required check could stay green while the AV collapse-detection family, this rescue's seed accounting, or the portfolio fair-draw backend regressed. These are precisely the failures that do not announce themselves: an evidence normalization or an inclusion probability that is wrong still returns a plausible-looking number. All three are now run in the sampler job, which is the one matrixed over both numpy lanes. They are CPU-only and cost seconds, so no split was needed.
…xport base
The transverse-spin investigation needs four features at once, and until now they
lived in two different trees: the merged fair export (--posterior-unique-draw) and
the boundary-reflecting puff (--reflect-parameter) were on this branch's history,
while the tail-sensitive convergence test and the tail-guard puff append existed
only as uncommitted working-tree edits in RIFT_develUWM. The decisive re-run needs
all of them together, so transplant the latter two here.
convergence_test_samples.py --method js_lame: 'lame' on the unbounded, non-circular
parameters, a boundary-reflected JS on each bounded transverse parameter, and a
lagged upper-quantile drift test over --drift-window iterations. The observed
failure mode is slow monotone tail drift that sits inside the noise floor of any
one-step statistic, so a lag window is required to see it. Also factor the
per-file field derivation into read_and_prepare() (the lagged files need the same
treatment), and make an unknown --method say so loudly instead of silently
returning inf and never converging.
util_ParameterPuffball.py --append-with-random-parameter: append uniform-random
transverse draws (azimuth redrawn, magnitude capped by the chi1/chi2 downselect)
and SHUFFLE the output rows. The shuffle is not cosmetic: nested ILE truncates
the puffball to its first 3000 rows, so unshuffled appends are silently dropped.
helper_LDG_Events.py / util_RIFT_pseudo_pipe.py wiring: pass
--internal-test-convergence-method through, and let js_lame auto-enable the
transverse-tails bundle.
Verified together on this base: --posterior-unique-draw, --reflect-parameter,
--append-with-random-parameter, --internal-test-convergence-method, and
--internal-cip-transverse-tails all resolve in one tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ct sample count
A fixed threshold is correct at exactly one sample size. The measured null floor (two
independent draws of the SAME converged posterior, K=400 bootstrap pairs) moves fast: JS and
lame fall as 1/n, upper-quantile drift as 1/sqrt(n). The shipped defaults (js 0.002,
quantile-tolerance 0.02, lame 0.02) sit 15-50x BELOW the floor at the honest per-worker supply
of ~800 distinct samples, so at that supply every component fires on pure noise and the gate
can never report convergence -- uninformative in the opposite direction from the original bug,
but just as broken.
--js-lame-auto-threshold derives each threshold as a multiple (default 1.5) of the fitted p95
floor at the distinct count actually supplied. Distinct, not row count: CIP pads its export
with duplicates once the request exceeds the honest supply, so a 20000-row posterior can carry
~800 distinct points and a row-count-keyed threshold would land ~25x too low. At n=5000 and
n=20000 the derived values reproduce the study's hand-computed recommendations.
Two things that made the gate fail quietly are now loud, independent of how thresholds are set:
* BLINDNESS. Noise is lag-independent while a monotone drift compounds, so the gate sees a
reference drift of d/iteration only if the quantile threshold is below (1+d)**W - 1. When
it is not, the test can stop the loop mid-widening; it now says so and names the window
length that would fix it.
* AN UNPOPULATED WINDOW. At sub-iteration 2-3 there are no lagged posteriors, so the test
silently degrades to the one-step statistic that provably cannot see the drift at honest
supply -- which is exactly where the shipped gate stopped the nested loop. This is now
reported, and --js-lame-require-lags refuses to certify convergence until the window is
populated. A --samples path that does not match the posterior_samples-N.dat naming used to
locate lagged files is also now reported instead of silently disabling the window.
Strictly additive: with the new flags off, both 'lame' and 'js_lame' return values identical to
the pre-change script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…thod help argparse %-formats help strings, so a literal "90% CI" makes --help raise ValueError: unsupported format character 'C'. Caught by the help-check CI job, which runs --help over every bin/ script. The neighbouring --internal-ile-n-chunk help already escapes its percentages as "88%%->50%%"; this one did not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t-total warm-seed reserve: exact pre-cap weight total, and route these suites into CI (follow-up to #78)
…ntegrated js_lame: sample-size-aware convergence gate + tail-guard puff (opt-in, off by default)
CHANGES.rst was last touched at the rc1 mark (027cc21); the ~315 commits since then are undocumented. Add one subsystem-grouped block covering that span, with no rc label -- the next release-candidate number is not assumed here. Grouping is by subsystem rather than merge order, since the same feature (portfolio draw allocation, the L0 warm-start rescue, the shape gate) was revised across several fork PRs. Per-feature detail and the measurements behind each claim stay in fork PRs #19-#84 of oshaughnessy-junior/research-projects-RIT. Recorded negative results and retractions alongside the positive ones, so a reader of CHANGES.rst does not re-derive claims the study already withdrew. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oshaughnessy-junior
had a problem deploying
to
private-review-dispatch-rift-upstream
August 13, 2026 02:40 — with 
GitHub Actions
Failure
GitHub Actions
Failure
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
oshaughn
had a problem deploying
to
private-review-dispatch-rift-upstream
August 13, 2026 02:49 — with
GitHub Actions
Failure
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Catch
oshaughn/rift_O4dup withoshaughnessy-junior/rift_O4d, and land the CHANGES.rst entryfor the 0.0.18.0 development that has accumulated since the rc1 mark.
oshaughn/rift_O4d@200dd60coshaughnessy-junior/rift_O4d@0ae3f48fExactly one commit was authored for this catch-up: the CHANGES.rst entry. Everything else is
already-reviewed work landed on the fork through the PRs tabulated below.
What is in the payload (fork PRs, by subsystem)
Each row is a merged PR on
oshaughnessy-junior/research-projects-RITagainst itsrift_O4d.Review threads, measurements, and the negative/retracted results live there; this PR is the
transport.
--sampler-method portfolioon GPUrho_sq_c = <C_c h|C_c h>;--calibration-global-normfallback; CPU+CUDA calmarg CI gates--limit-right-ascension/-declination/-inclination/-psiwere silently ignored under the cosine samplersutil_RandomizeOverlapOrderinterleaves across worker files (ILE was seeing only 4 of 25 workers)dintegrand on its native backend_rvsimportance weights instead of reading the ambiguouslog_weightscache; do notlog()lnL under--internal-use-lnL--posterior-unique-draw)--method js_lamesample-size-aware convergence gate, tail-guard puff,--internal-cip-transverse-tailsbundle (opt-in, off by default)--cip-exe-GBehavior-changing defaults in the payload are limited to the bug fixes above
(#61, #62, #63, #76, #77, #82) and the crash/correctness fixes they carry. The new
capabilities —
js_lame, the transverse-tails bundle, the AV anisotropic bins, portfolioweight clipping, portfolio adaptive draw allocation — are all opt-in and default off.
The CHANGES.rst commit
CHANGES.rstwas last touched at the rc1 mark (027cc21d). The ~315 commits since then —this payload plus the two earlier catch-ups, #155
and #161 — were undocumented.
This PR adds one block to the
0.0.18.0section covering that whole span. Two deliberatechoices:
** (unreleased; development since rc1). The nextrelease-candidate number is yours to assign; nothing here assumes it.
allocation, the L0 rescue, the shape gate) was revised across several fork PRs, and a
chronological list would read as churn rather than as a feature.
Negative results and retractions are recorded next to the positive ones —
n_effdoes notcertify correctness, the Pareto k-hat diagnostic does not catch confidently-wrong runs from
support mismatch, and the L0 "doubles landed fraction" and cap24 lnZ-bias claims are withdrawn —
so a later reader does not re-derive a claim the study already dropped.
Not included
container_imageselector truncation) is still a separate branch and is not in thispayload; it remains independently reviewable.
#169 are the
rift_O4cbackports of Baryon mass factor corrected and RePrimAnd accuracy scale option added #76 and Tabular EOS project: pipeline and helper updates + bug fixes #82 respectively. They carry the same payload to the release line and
are not superseded by this PR.
🤖 Generated with Claude Code