Conversation
initPool() ran from a static initializer, so the first CUDA runtime calls of the process happened before main(). On a host where such an early call fails (here: one degraded GPU, cudaGetDeviceCount -> "invalid device ordinal") the runtime stays in the error state, stream_pool is never filled, and Stream::init later indexes assigner_idx with a garbage device id and segfaults. Build the pool on first use from Stream::init instead. No change in behaviour on healthy hosts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
initPool()runs from a static initializer insrc/CudaUtils.cu, so the process's first CUDA runtime calls (cudaGetDeviceCount,cudaSetDevice,cudaStreamCreateWithFlags) happen beforemain(). When one of those calls fails, the CUDA runtime stays in the error state,stream_poolis never filled, and the firstStream::initindexesassigner_idxwith a garbage device id and segfaults. We hit this on a 2x H200 machine with one GPU in a degraded state (invalid device ordinal); on a healthy host nothing is visible.This change builds the pool on first use, from
Stream::init, through a function-local static (still one-time and thread-safe). No API change, and no change in behaviour on healthy hosts.Tested on v2.1.3 (786c760) with CUDA 13.0 / driver 580 on H200: the degraded host went from segfault to running, and our bootstrapped CKKS workloads have been running on this patch since 2026-08-31. The commit also applies cleanly on
OpenFHECompatTests.Fable 5.1 on behalf of Seyfal