From 66f3db1b772f0d0b6c306c83c9cd673cbc327b58 Mon Sep 17 00:00:00 2001 From: shivansh193 Date: Sat, 5 Sep 2026 07:52:49 +0530 Subject: [PATCH] fix: restore sm_60 (P100) PyTorch kernels in GPU image The Colab GPU base image ships PyTorch built against a CUDA wheel index (currently cu128) whose compute-capability list is sm_70/75/80/86/90/100/120 -- it drops sm_60 (Pascal, e.g. the Tesla P100 Kaggle's scheduler still assigns as a free GPU option). Any real GPU op on a P100 then fails with "CUDA error: no kernel image is available for execution on the device", even for plain fp16 model.generate() calls with no quantization involved. PyTorch's cu126 wheels are still built for {50,60,70,75,80,86,90} (verified against pytorch/pytorch's .ci/manywheel/build_env_setup.py arch table), so for GPU images only, reinstall the *same* torch/torchvision/torchaudio version from the cu126 index instead of whatever the base image pulled. This restores sm_60 while keeping every GPU Kaggle currently offers (T4, sm_75) working. The tradeoff is losing sm_100/sm_120 (Blackwell) kernels, which Kaggle does not currently offer as a notebook accelerator. Fixes #1546 --- Dockerfile.tmpl | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/Dockerfile.tmpl b/Dockerfile.tmpl index 8d72c7a5..cd02fc1e 100644 --- a/Dockerfile.tmpl +++ b/Dockerfile.tmpl @@ -16,6 +16,28 @@ RUN cat /kaggle_requirements.txt >> /requirements.txt # Install Kaggle packages RUN uv pip install --system --no-cache -r /requirements.txt +{{ if eq .Accelerator "gpu" }} +# b/342143152, Kaggle/docker-python#1546: the Colab GPU base image's PyTorch +# wheel is built against a CUDA index (currently cu128) whose compute-capability +# list is sm_70/75/80/86/90/100/120 -- it does not include sm_60 (Pascal, e.g. +# the Tesla P100 that Kaggle's scheduler still hands out as a free GPU option). +# Any real GPU op on a P100 then fails with: +# torch.AcceleratorError: CUDA error: no kernel image is available for +# execution on the device +# PyTorch's cu126 wheels are still built for {50,60,70,75,80,86,90} (confirmed +# against pytorch/pytorch's .ci/manywheel/build_env_setup.py arch table), so +# reinstalling the *same* torch/torchvision/torchaudio version from the cu126 +# index restores sm_60 while keeping every GPU currently offered on Kaggle +# (e.g. T4, sm_75) working. The known tradeoff is losing sm_100/sm_120 +# (Blackwell) kernels, which Kaggle does not currently offer as a notebook +# accelerator. +RUN TORCH_VERSION=$(python -c "import torch; print(torch.__version__.split('+')[0])") && \ + uv pip install --system --no-cache --force-reinstall \ + "torch==${TORCH_VERSION}" torchvision torchaudio \ + --index-url https://download.pytorch.org/whl/cu126 \ + --extra-index-url https://pypi.org/simple +{{ end }} + # Install manual packages: # b/183041606#comment5: the Kaggle data proxy doesn't support these APIs. If the library is missing, it falls back to using a regular BigQuery query to fetch data. RUN uv pip uninstall --system --no-cache google-cloud-bigquery-storage