[FEA]: add workaround for loading broken cufile wheel

[gforsyth]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this request and that I agree to the Code of Conduct

Area

cuda.pathfinder

Is your feature request related to a problem? Please describe.

In rapidsai/kvikio#1000 we ran across a bug in nvidia-cufile, where the upstream cufile wheel is improperly linked and fails to load librt.so.1 and complains about missing shm_copy.

Note: this is definitely NOT a pathfinder bug, just reporting it in case you want to incorporate the workaround we've added in kvikio to help other users (while we wait to see if it's fixed and backported upstream)

Environment

Running on an arm64 machine using this container: rapidsai/citestwheel:26.08-cuda13.0.3-rockylinux8-py3.12

Install cufile and cuda-python: pip install nvidia-cufile==1.15.1.6.* cuda-python

Loading error

Attempting to load cufile errors:

Python 3.12.13 (main, Jul  1 2026, 20:40:39) [GCC 8.5.0 20210514 (Red Hat 8.5.0-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda.pathfinder import load_nvidia_dynamic_lib
>>> load_nvidia_dynamic_lib('cufile')
Traceback (most recent call last):
  File "/pyenv/versions/3.12.13/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 218, in load_with_abs_path
    handle = _load_lib(desc, found_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.12.13/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 155, in _load_lib
    return ctypes.CDLL(filename, cdll_mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.12.13/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /pyenv/versions/3.12.13/lib/python3.12/site-packages/nvidia/cu13/lib/libcufile.so.0: undefined symbol: shm_open

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/pyenv/versions/3.12.13/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 303, in load_nvidia_dynamic_lib
    return _load_lib_no_cache(libname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.12.13/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 175, in _load_lib_no_cache
    return LOADER.load_with_abs_path(desc, find.abs_path, find.found_via)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.12.13/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 220, in load_with_abs_path
    raise RuntimeError(f"Failed to dlopen {found_path}: {e}") from e
RuntimeError: Failed to dlopen /pyenv/versions/3.12.13/lib/python3.12/site-packages/nvidia/cu13/lib/libcufile.so.0: /pyenv/versions/3.12.13/lib/python3.12/site-packages/nvidia/cu13/lib/libcufile.so.0: undefined symbol: shm_open

Workaround

Manually loading librt.so.1 first takes care of the issue:

Python 3.12.13 (main, Jul  1 2026, 20:40:39) [GCC 8.5.0 20210514 (Red Hat 8.5.0-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.CDLL("librt.so.1", mode=ctypes.RTLD_GLOBAL)
<CDLL 'librt.so.1', handle 342d23b0 at 0xefed3f4b8b90>
>>> from cuda.pathfinder import load_nvidia_dynamic_lib
>>> load_nvidia_dynamic_lib('cufile')
LoadedDL(abs_path='/pyenv/versions/3.12.13/lib/python3.12/site-packages/nvidia/cu13/lib/libcufile.so.0', was_already_loaded_from_elsewhere=False, _handle_uint=875329216, found_via='site-packages')

We haven't seem this on all of our arm runners (this issue first popped up in CI), so including some system info in case it's of use:

>>> cuda.core.system.get_driver_branch()
'r596_25'
>>> cuda.core.system.get_kernel_mode_driver_version()
(595, 71, 5)
>>> cuda.core.system.get_user_mode_driver_version()
(13, 2)

[root@b0cf708dccaa /]# ldd --version
ldd (GNU libc) 2.28
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
[root@b0cf708dccaa /]# uname -a
Linux b0cf708dccaa 6.8.0-111-generic #111-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 11 22:59:23 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux

Describe the solution you'd like

I'll defer to you whether you want to take action on this or not.

One solution is to add the librt.so.1 loading it, which might help out some downstream users, but you also might not want to take on that workaround into your codebase.

Describe alternatives you've considered

Do nothing and wait for upstream to be fixed.

Additional context

No response

[leofang]

cc @rwgk for vis