Skip to content

update rapids version for 2606 release#1396

Merged
cjac merged 3 commits into
GoogleCloudDataproc:mainfrom
nvliyuan:rapids-v2606
Jul 21, 2026
Merged

update rapids version for 2606 release#1396
cjac merged 3 commits into
GoogleCloudDataproc:mainfrom
nvliyuan:rapids-v2606

Conversation

@nvliyuan

Copy link
Copy Markdown
Contributor

Update the spark-rapids jar version after plugin release.

CC @sameerz @viadea @jayadeep-jayaraman

Verification

  • Confirmed com.nvidia:rapids-4-spark_2.12:26.06.0 is published in Maven metadata.
  • Ran bash -n spark-rapids/spark-rapids.sh.

nvliyuan and others added 2 commits May 28, 2026 16:14
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@nvliyuan
nvliyuan marked this pull request as ready for review June 24, 2026 02:41

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the default Spark Rapids version from 26.04.2 to 26.06.0 in the spark-rapids.sh script. There are no review comments, and I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@nvliyuan

Copy link
Copy Markdown
Contributor Author

CC @cjac @jayadeep-jayaraman please help review thx

@cjac

cjac commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@nvliyuan

Copy link
Copy Markdown
Contributor Author

Hi @cjac could you help merge? thx

@cjac

cjac commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@cjac cjac left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simple enough. Running tests now

@cjac

cjac commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Uh oh. Failing presubmit. I'll poke at it today. I also want to finish that patch to make it run faster on -ml or custom images with drivers pre-installed.

@sameerz

sameerz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Hi @cjac any update on this?

@cjac

cjac commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

I've been meaning to get it working. I'll look now.

@cjac

cjac commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

I checked gs://dataproc-108de5de-43c2-4a4b-979a-adebc15a58a8-us-central1/google-cloud-dataproc-
metainfo/5a49fddd-fee9-4b93-b61c-6a1d5fe2d95f/test-rapids-single-2-2-20260715-102646-td5r-m/dataproc-
initialization-script-0_output .

Here are the key findings from that log file:

  1. Initialization Parameters Recorded:
    • OS Name / Version: ubuntu (22.04)
    • Dataproc Image: 2.2-ubuntu22
    • Kernel Version: 6.8.0-1060-gcp
    • Selected RAPIDS / Driver Defaults: SPARK_RAPIDS_VERSION=26.06.0, CUDA_VERSION=12.4.1,
    NVIDIA_DRIVER_VERSION=550.54.15

  2. Error Trace in Log (Lines 800–825 & 841–870):
    Loading new nvidia-550.54.15 DKMS files...
    Building for 6.8.0-1060-gcp
    Building for architecture x86_64
    Building initial module for 6.8.0-1060-gcp
    ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/nvidia-dkms-550-open.0.crash'
    Error! Bad return status for module build on kernel: 6.8.0-1060-gcp (x86_64)
    Consult /var/lib/dkms/nvidia/550.54.15/build/make.log for more information.
    dpkg: error processing package nvidia-dkms-550-open (--configure):
    installed nvidia-dkms-550-open package post-installation script subprocess returned error exit status 10
    dpkg: dependency problems prevent configuration of nvidia-driver-550-open:
    nvidia-driver-550-open depends on nvidia-dkms-550-open (<= 550.54.15-1); however:
    Package nvidia-dkms-550-open is not configured yet.
    ...
    E: Sub-process /usr/bin/dpkg returned an error code (1)

  3. Conclusion:
    The log confirms that spark-rapids.sh failed during apt-get install -y nvidia-driver-550-open because
    nvidia-550.54.15 open-gpu-kernel-modules failed DKMS compilation against the guest kernel (6.8.0-1060-gcp),
    causing 10 consecutive retry loops to fail and exiting with status 1.

@cjac

cjac commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@nvliyuan

Copy link
Copy Markdown
Contributor Author

I had previously addressed this Ubuntu 22 / Linux 6.8 NVIDIA DKMS issue in #1394.

That fix makes Ubuntu 22 use the online NVIDIA repository instead of the older driver embedded in the local CUDA repository, and selects GCC/G++ 12 for CUDA 12 driver builds. I validated it on Dataproc 2.2.82-ubuntu22 with kernel 6.8.0-1058-gcp; cluster creation succeeded and nvidia-smi reported driver 550.163.01.

I have now merged #1394 into this PR and reverted b5ab3cb. The RAPIDS version remains 26.06.0.

While checking b5ab3cb, I also reproduced two issues with that commit:

  • nvidia-driver-cuda has no installation candidate in the CUDA 13.1 local Ubuntu repository.
  • The non-default test combination CUDA 12.4.0 + driver 590.48.01 points to a nonexistent NVIDIA installer URL.

@nvliyuan

Copy link
Copy Markdown
Contributor Author

verified worked
image

@cjac

cjac commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@cjac

cjac commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

I patched my changes out

@cjac
cjac merged commit aa3c100 into GoogleCloudDataproc:main Jul 21, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants