Skip to content

Fix: GPU compatibility for V100, with CUDA 12.x and sm_70. - #45

Open
ericxiesg wants to merge 1 commit into
CAPS-UMU:mainfrom
ericxiesg:gpu-dev
Open

ericxiesg wants to merge 1 commit into
CAPS-UMU:mainfrom
ericxiesg:gpu-dev

Conversation

@ericxiesg

Copy link
Copy Markdown

What

Gets FIDESlib building and running on a V100 (sm_70) with CUDA 12.x. Two parts:

  1. CMake picks the CUDA archs for you. If FIDESLIB_ARCH isn't set, it reads the GPUs from nvidia-smi and the toolkit version from nvcc --version, then builds for the GPUs that nvcc actually supports. With no GPU visible, you get the old fat list clipped to what nvcc accepts. -DFIDESLIB_ARCH=70-real (or any list) still overrides all of it.
  2. Fixes for the bugs that showed up once it ran on a GV100. Mostly out-of-bounds reads that surfaced as illegal-address errors far from where they happened, plus a few API correctness fixes picked up along the way.

Why

The default arch list starts at 80-real, so a V100 got no kernel image at all. No single hardcoded list works everywhere: CUDA 13 dropped sm_70, and toolkits older than 12.8 don't know sm_100/sm_120.

Once it did run, these came up:

Symptom on the GV100 Root cause Fix
NaN out of EvalMultByI, then an illegal memory access, then cudaMemcpy 'invalid argument' (N = 2^16, dnum = 3) L + K > MAXP, so the special primes get written past the end of the [MAXP] constant tables. Nothing checked it. SetupConstants now throws with the actual counts and what to change (depth, dnum or MAXP)
LTdotProductPtBatch reading 35 limbs from a plaintext that holds 34 (bootstrap, FIXEDMANUAL) OpenFHE encodes the CtS/StC diagonals one level below where our ModRaise leaves the ciphertext EvalCoeffsToSlots drops the ciphertext to each layer's diagonal level before the product, which is what OpenFHE's AdjustLevelsAndDepth does implicitly
Illegal access reported from a GPUfree several calls later multMonomial can index past limb on a pooled polynomial that wasn't grown to its level Explicit limb-count check that throws right there

LTdotProductPtBatch also checks every operand's limb count against the launch size now, so a level mismatch fails with a readable message instead of a garbage device pointer.

API fixes in api/CryptoContext.cpp:

  • EvalSub(scalar, ct): on GPU this returned ct - scalar. Dropped the extra multScalar(-1.0).
  • EvalMult(ct, pt) / EvalMultInPlace(ct, pt), CPU fallback: these any_cast to ConstPlaintext, but pt->cpu always holds a Plaintext, so they threw bad_any_cast.
  • ~CryptoContextImpl: it called the no-arg ClearEvalMultKeys() / ClearEvalAutomorphismKeys(), which wipe OpenFHE's global key maps, i.e. every other live context's keys too. Removed. OpenFHE's own CryptoContextImpl doesn't do this either.

Also, KeySwitchingKey held a Context& that dangled once LoadContext moved its local Context into std::any. It's now a std::weak_ptr<ContextData> behind a context() accessor. It's deliberately not a shared_ptr: keys live inside ContextData, so owning it would make a cycle and no context would ever be freed.

Why it is safe

  • All the new checks run on the host before a launch and cost O(number of operands). They only throw where the old code would have read out of bounds.
  • The LTdotProductPtBatch check skips nullptr diagonals, which the kernel skips as well. An earlier version of the check didn't, and that broke a working bootstrap.
  • The level alignment in EvalCoeffsToSlots only ever drops the ciphertext, and only when it sits above that layer's diagonals. When the levels already match, it's a no-op.
  • Arch detection only kicks in when FIDESLIB_ARCH isn't set. Explicit values behave exactly as before.

Testing

The bugs above were hit and fixed on a GV100 (CUDA 12.x, sm_70). A full fideslib-test run on the V100 is in progress; I'll post the results here.

Notes

  • Configuring on a machine with a GPU now builds only that GPU's arch (-real, no PTX). That's great locally, but not what you want for a package or CI build, so pass FIDESLIB_ARCH explicitly there. Happy to make auto-detection opt-in instead if you'd prefer that.
  • The level-budget-1 bootstrap path (EvalLinearTransform) doesn't get the alignment yet. If the same one-limb gap exists there, it now fails with the limb-count error instead of reading out of bounds.
  • AddBootstrapPlaintexts prints one diagnostic line to stderr per bootstrap setup. It was handy while chasing the level gap, and I can put it behind a flag.
  • .gitignore: added __pycache__/, *.pyc and build-*/.

@ericxiesg
ericxiesg marked this pull request as ready for review September 11, 2026 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant