integrators: probe for a CUDA device before selecting cupy; fail loudly when GMM refit never succeeds

[oshaughnessy-junior]

Regression

The May–June 2026 GPU-port commits (5d55c04/9dd7188a, f44c7a8) made the GMM integrator stack select cupy at module import time whenever import cupy succeeds. On a cupy-installed node without a usable GPU (shared install, or CUDA_VISIBLE_DEVICES masked — a configuration that has repeatedly bitten production workers), every GMM refit dies in gaussian_mixture_model._near_psd_impl through the import-time binding _xpy_eigvals = cupy.linalg.eigvalsh (cudaErrorNoDevice). MonteCarloEnsemble.integrate swallows the exception ("Error training, resetting..."), the proposal never adapts, and integrate() returns n_eff ≈ 1 with no error flag.

Found by the shape-recovery merge gate's O4c back-check (#31): the O4c code recovers the mix_d6_n1_s101 discriminator target with n_eff ≈ 3047 in the identical environment where O4d returns n_eff ≈ 1.05. Full bisect, probe table and probe script live in the RIFT_roboto_paper worktree under analyses/integrator_shape_gate/gmm_regression_bisect/.

Fix

1. Device probe at import (MonteCarloEnsemble, gaussian_mixture_model, mcsamplerEnsemble): gate cupy selection on cupy.cuda.runtime.getDeviceCount() > 0. Broad except because the probe raises CUDARuntimeError, not ImportError, on GPU-less nodes.
2. Call-time eigen dispatch (gaussian_mixture_model): drop the import-time _xpy_eigvals/_xpy_eig globals; resolve eigvalsh/eigh through the caller's xpy, mirroring how _xpy_logsumexp already dispatches.
3. Fail loudly (MonteCarloEnsemble.integrate): after 5 consecutive refit failures (max_consecutive_train_failures kwarg) raise RuntimeError and set self.integration_error instead of silently returning n_eff ≈ 1. A separate integration_error string is also set on the existing max-error exit path.
4. Drive-by: mcsamplerEnsemble passed tripwire_epsion= (typo) into integrate(), so caller-supplied tripwire_epsilon values were silently discarded; now passed under the real name.

Verification (ldas-pcdev11, env ~/RIFT_develUWM, OMP_NUM_THREADS=1, CUDA_VISIBLE_DEVICES="")

• Discriminator probe mix_d6_n1_s101: n_eff 1.05 → 2037.0, js_max 0.63 → 0.00017, no error (matches the validated files_f28ad8d0_eigvals_np bisect point exactly).
• Shape-recovery quick preset (base 6950de8 vs this branch, same seeds): AV/NF/portfolio rows bit-identical (their pre-existing quick-budget WARN/FAIL untouched); GMM improves — d2: JSmax 0.0021 → 0.0004, n_ESS 1103 → 6418; d4: n_eff 9 → 42 (still STARVED at quick budget, non-blocking).
• Tripwire: with gmm.fit forced to raise (emulating cudaErrorNoDevice) on the d6 target, integrate() now raises RuntimeError: GMM proposal refit failed 5 consecutive times; proposal never adapted instead of returning n_eff ≈ 1. On an easy target that legitimately converges before adaptation matters, the tripwire correctly does not fire.
• GPU-visible check: all three modules still select cupy when a device is present (cupy_ok=True), identical to base. Full on-GPU end-to-end could not be exercised on pcdev11: the env's cupy predates the Blackwell GPUs and fails NVRTC compilation on both base and this branch identically (pre-existing environment limitation, unrelated to this change).

Known residual (out of scope)

A ~1.5× GMM adaptation slowdown relative to O4c remains, introduced by f28ad8d (correct log-domain weight handling): n_eff ≈ 2037 vs O4c's ≈ 3047 on the discriminator at equal budget. That is a correctness-motivated change and is tracked separately from this regression fix.

🤖 Generated with Claude Code

[oshaughnessy-junior]

Auto-review completed the mandatory integrator shape-recovery gate. The standard CPU-only matrix ran on the refreshed base and this candidate (96 runs each, SHAPE_JOBS=10, OMP_NUM_THREADS=1); the supplied comparator reported # blocking regressions (strict=['AV', 'GMM']): 0.\n\nAlso passed: merge simulation, diff/compilation checks, CPU-forced backend import (all three touched modules selected NumPy), and the quick shape wrapper (4 passed). This workstation has no CuPy installation, so I could not independently reproduce the CuPy-installed/no-device discriminator; the PR's documented external probe remains the evidence for that configuration. Existing portfolio failures are pre-existing/warn-only under the gate policy.

[oshaughnessy-junior]

2026-08-04 18:41 UTC heartbeat checkpoint: PR #43 remains OPEN/CLEAN/MERGEABLE with all 19 CI checks successful. The isolated Issue #32 worktree has only the known excluded pixi.lock modification, and rift_O4d...oshaughn/rift_O4d remains 0/0; the unsafe designated checkout was not touched.

Roadblock: owner merge decision, followed by required human-accessible validation.

Next: detect merge or actionable PR change; after merge, verify the Sphinx integration and record the human-validation gate.