# Runtime Error: Cuda Initialization **URL:** \ **Category:** 🤗Transformers\ **Created:** [March 20, 2025, 2:18pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649 "2025-03-20T14:18:52Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![hxn137](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@hxn137](https://discuss.huggingface.co/u/hxn137)\ **Post date:** [March 20, 2025, 2:18pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/1 "2025-03-20T14:18:52Z")
Hi all, I am trying to run a simple script to analyze attention scores of Llama-2-7B-32K model on Jetstream2 HPC cluster. Until yesterday, I was able to run scripts but when I suddenly started seeing the following error: ```auto Traceback (most recent call last): File "/home/exouser/Squeezed-Attention/offline_clustering.py", line 8, in from utils.model_parse import ( File "/home/exouser/Squeezed-Attention/utils/model_parse.py", line 1, in from transformers import AutoModelForCausalLM, LlamaForCausalLM, OPTForCausalLM File "", line 1055, in _handle_fromlist File "/home/exouser/Squeezed-Attention/transformers/src/transformers/utils/import_utils.py", line 1500, in __getattr__ value = getattr(module, name) File "/home/exouser/Squeezed-Attention/transformers/src/transformers/utils/import_utils.py", line 1499, in __getattr__ module = self._get_module(self._class_to_module[name]) File "/home/exouser/Squeezed-Attention/transformers/src/transformers/utils/import_utils.py", line 1511, in _get_module raise RuntimeError( RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback): cudaErrorInitializationError: initialization error ``` I am quite new to this area but I was not sure the root cause of this error and could not find anything that is relevant online. Does anyone have any suggestions on how to tackle this? ---
**Author:** ![aaraya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aaraya/32/48313_2.png) [@aaraya](https://discuss.huggingface.co/u/aaraya)\ **Post date:** [March 20, 2025, 4:46pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/2 "2025-03-20T16:46:38Z")
It seems like there’s a CUDA initialization error. Are you sure you have access to a GPU on the node you’re running on? Try running: ```auto nvidia-smi ``` If that doesn’t work, you’re likely on a node without GPU or the environment isn’t set up correctly. Make sure to run on a GPU node and check that CUDA and drivers are properly loaded. Adrian Araya Machine Learning Engineer at RidgeRun.ai Contact us: [support@ridgerun.ai](mailto:support@ridgerun.ai) > **[RidgeRun AI | Artificial Intelligence Services | AI Consultant | United States](https://www.ridgerun.ai)** > > AI, Computer Vision and Machine Learning for your Audio and Video Projects. RidgeRun Artificial Intelligence provides cutting-edge machine learning consulting services and products for audio and video analytics that make your product unique. ---
**Author:** ![hxn137](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@hxn137](https://discuss.huggingface.co/u/hxn137)\ **Post date:** [March 20, 2025, 5:02pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/3 "2025-03-20T17:02:45Z")
Hi Adrian, Thank you for your response. Here is the output of `nvidia-smi` command ```auto (fixedprompt) (base) exouser@possibly-right-crawdad:~/Squeezed-Attention/LongBench$ nvidia-smi Thu Mar 20 16:58:13 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A100-SXM4-40GB Off | 00000000:04:00.0 Off | 0 | | N/A 25C P0 52W / 400W | 1MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ``` It does seem like I am running this on a GPU node. `nvcc --version` returns ```auto nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Thu_Jun__6_02:18:23_PDT_2024 Cuda compilation tools, release 12.5, V12.5.82 Build cuda_12.5.r12.5/compiler.34385749_0 ``` If I understand correctly, there is no incompatibility between cuda version and driver version. However, below command indicates that the cuda is not available even though `nvidia-smi` shows that there is a node available? ```auto (fixedprompt) (base) exouser@possibly-right-crawdad:~/Squeezed-Attention/LongBench$ python Python 3.9.21 | packaged by conda-forge | (main, Dec 5 2024, 13:51:40) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> print(torch.cuda.is_available()) /home/exouser/miniconda3/envs/fixedprompt/lib/python3.9/site-packages/torch/cuda/ __init__.py:129: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.) return torch._C._cuda_getDeviceCount() > 0 False ``` ---
**Author:** ![aaraya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aaraya/32/48313_2.png) [@aaraya](https://discuss.huggingface.co/u/aaraya)\ **Post date:** [March 20, 2025, 5:09pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/4 "2025-03-20T17:09:34Z")
Mmm, it looks like PyTorch in your environment might not have GPU support. To check, run: ```auto import torch print(torch.version.cuda) ``` If it prints None, it means you installed the CPU-only version of PyTorch. In that case, reinstall with GPU support using: ```auto conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia ``` * * * Adrian Araya Machine Learning Engineer at [RidgeRun.ai](http://RidgeRun.ai) Contact us: [support@ridgerun.ai](mailto:support@ridgerun.ai) ---
**Author:** ![hxn137](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@hxn137](https://discuss.huggingface.co/u/hxn137)\ **Post date:** [March 20, 2025, 5:22pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/5 "2025-03-20T17:22:53Z")
Below is the output ```auto >>> import torch >>> print(torch.version.cuda) 12.4 >>> print(torch.cuda.is_available()) /home/exouser/miniconda3/envs/fixedprompt/lib/python3.9/site-packages/torch/cuda/ __init__.py:129: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.) return torch._C._cuda_getDeviceCount() > 0 False ``` It does print out the version of cuda installed with the right version in my environment. However, it still shows me that it is not available. ---
**Author:** ![aaraya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aaraya/32/48313_2.png) [@aaraya](https://discuss.huggingface.co/u/aaraya)\ **Post date:** [March 20, 2025, 8:45pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/6 "2025-03-20T20:45:44Z")
Mmm, maybe the environment isn’t picking up the right driver. Try this just to be sure: ```bash module purge module load cuda/12.4 conda activate fixedprompt python -c "import torch; print(torch.cuda.is_available())" ``` Sometimes that resets things and makes PyTorch see the GPU. * * * Adrian Araya Machine Learning Engineer at [RidgeRun.ai](http://RidgeRun.ai) Contact us: [support@ridgerun.ai](mailto:support@ridgerun.ai) ---
**Author:** ![hxn137](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@hxn137](https://discuss.huggingface.co/u/hxn137)\ **Post date:** [March 21, 2025, 1:45pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/7 "2025-03-21T13:45:36Z")
I ran the following command but I still see the same error. I tried by restarting from scratch including creating a new conda environment, installing packages, and then ran the script. For the first couple of times, it actually ran fine without seeing initialization error but at certain point I started seeing initialization error. The only thing I noticed was that the RAM of the root disk of the cluster spiked but that is the only thing. This also happened when I created a new python script and shell script to perform attention analysis. I do not know if this is related however. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 21, 2025, 2:22pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/8 "2025-03-21T14:22:44Z")
The most common reasons for this are that the default version of the library on the server side has changed, or that there is a mismatch with the new version of the library that was installed when the server was restarted. The most suspicious thing is always PyTorch, but you might want to try downgrading Transformers, etc. as well. ```auto pip install transformers==4.48.3 ``` ---
**Author:** ![hxn137](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@hxn137](https://discuss.huggingface.co/u/hxn137)\ **Post date:** [March 21, 2025, 2:42pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/9 "2025-03-21T14:42:30Z")
I am currently using transformers=4.40.dev right now. Could this version be the issue? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 21, 2025, 3:01pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/10 "2025-03-21T15:01:04Z")
No, if you’ve already fixed the version, it would be more suspicious to update some other library that hasn’t been fixed. If it’s a Hugging Face’s Transformers-around library… ```py pip install -U transformers peft accelerate huggingface_hub numpy<2 sentencepiece torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124 ``` Also, this is probably not related to the outside of Hugging Face, but there have been cases recently where it works if you re-specify the **HF\_HOME** environment variable. This is because libraries such as Transformers and **huggingface\_hub** refer to it and work. > **[Environment variables](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. Oh, but the .dev version may be a bit suspicious, as there are multiple versions even if the version number is the same. ---
**Author:** ![hxn137](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@hxn137](https://discuss.huggingface.co/u/hxn137)\ **Post date:** [March 21, 2025, 3:38pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/11 "2025-03-21T15:38:54Z")
The error still persists after re-installing packages and re-set HF\_HOME to `~/.cache/huggingface` ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 21, 2025, 4:05pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/12 "2025-03-21T16:05:42Z")
Hmmm… perhaps Jetstream2 HPC cluster issue…? > > > I have the same issue with my cuda 12.1 run under wsl2 with both python 310-312 …tested > > nvcc: NVIDIA (R) Cuda compiler driver > Copyright (c) 2005-2023 NVIDIA Corporation > Built on Mon\_Apr\_\_3\_17:16:06\_PDT\_2023 > Cuda compilation tools, release 12.1, V12.1.105 > Build cuda\_12.1.r12.1/compiler.32688072\_0 > > I got the error like: > \[E 12/20/24 08:45:40.689 2984\] \[cuda\_driver.h:operator()@92\] CUDA Error CUDA\_ERROR\_NO\_DEVICE: no CUDA-capable device is detected while calling init (cuInit) > > However I can run in windows with cuda device working properly. but rendering is not supported, > ---
**Author:** ![hxn137](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@hxn137](https://discuss.huggingface.co/u/hxn137)\ **Post date:** [March 24, 2025, 12:32pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/13 "2025-03-24T12:32:33Z")
I am beginning to think that it is a cluster issue but I am not sure. When I create a new instance, I can run the code without running into this issue for a bit and then, after a while, it just starts running into initialization issue. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 24, 2025, 1:54pm UTC](https://discuss.huggingface.co/t/runtime-error-cuda-initialization/146649/14 "2025-03-24T13:54:53Z")
Hmm… > **[GPU FAQs - Jetstream2 Documentation](https://docs.jetstream-cloud.org/faq/gpu/)** > > Jetstream2 Documentation