AttributeError: /opt/conda/bin/python: undefined symbol: cudaRuntimeGetVersion

[0xdevalias]

Describe the bug

AttributeError: /opt/conda/bin/python: undefined symbol: cudaRuntimeGetVersion

Reproduction

%cd /workspace

!git clone https://github.com/huggingface/diffusers.git

%cd /workspace/diffusers/examples/dreambooth

!pwd

pip install -U -r requirements.txt

!pip install bitsandbytes

!git config --global credential.helper store

from huggingface_hub import notebook_login
notebook_login()

MODEL_NAME_OR_PATH="CompVis/stable-diffusion-v1-4"
# MODEL_NAME_OR_PATH="runwayml/stable-diffusion-v1-5"
# MODEL_NAME_OR_PATH="runwayml/stable-diffusion-inpainting"
INSTANCE_DIR="./instance_images"
CLASS_DIR="./class_images"
OUTPUT_DIR="./trained_model"
INSTANCE_PROMPT="a photo of sks dog"
CLASS_PROMPT="a photo of dog"
NUM_CLASS_IMAGES=200
MAX_TRAIN_STEPS=800
SEED=1337

# Ensure the directories have been created
!mkdir -p $INSTANCE_DIR
!mkdir -p $CLASS_DIR
!mkdir -p $OUTPUT_DIR

!accelerate launch train_dreambooth.py \
  --pretrained_model_name_or_path="$MODEL_NAME_OR_PATH"  \
  --instance_data_dir="$INSTANCE_DIR" \
  --class_data_dir="$CLASS_DIR" \
  --output_dir="$OUTPUT_DIR" \
  --with_prior_preservation --prior_loss_weight=1.0 \
  --instance_prompt="$INSTANCE_PROMPT" \
  --class_prompt="$CLASS_PROMPT" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=2 --gradient_checkpointing \
  --use_8bit_adam \
  --learning_rate=5e-6 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --num_class_images="$NUM_CLASS_IMAGES" \
  --max_train_steps="$MAX_TRAIN_STEPS" \
  --seed=$SEED

Logs

Traceback (most recent call last):
  File "train_dreambooth.py", line 668, in <module>
    main(args)
  File "train_dreambooth.py", line 454, in main
    import bitsandbytes as bnb
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/autograd/_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cextension.py", line 113, in <module>
    lib = CUDASetup.get_instance().lib
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cextension.py", line 109, in get_instance
    cls._instance.initialize()
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cextension.py", line 59, in initialize
    binary_name, cudart_path, cuda, cc, cuda_version_string = evaluate_cuda_setup()
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cuda_setup/main.py", line 125, in evaluate_cuda_setup
    cuda_version_string = get_cuda_version(cuda, cudart_path)
  File "/opt/conda/lib/python3.7/site-packages/bitsandbytes/cuda_setup/main.py", line 45, in get_cuda_version
    check_cuda_result(cuda, cudart.cudaRuntimeGetVersion(ctypes.byref(version)))
  File "/opt/conda/lib/python3.7/ctypes/__init__.py", line 377, in __getattr__
    func = self.__getitem__(name)
  File "/opt/conda/lib/python3.7/ctypes/__init__.py", line 382, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))

AttributeError: /opt/conda/bin/python: undefined symbol: cudaRuntimeGetVersion

### System Info

diffusers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• diffusers version: 0.7.2
• Platform: Linux-5.15.0-52-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.13
• PyTorch version (GPU?): 1.13.0+cu117 (True)
• Huggingface_hub version: 0.10.1
• Transformers version: 4.24.0
• Using GPU in script?: RTX 3090
• Using distributed or parallel set-up in script?: No

[0xdevalias]

Just found this, so maybe it's an issue with bitsandbytes:

• undefined symbol: cudaRuntimeGetVersion Error bitsandbytes-foundation/bitsandbytes#85

Was able to workaround this with setenv LD_LIBRARY_PATH /usr/lib/x86_64-linux-gnu which is where my libcudart.so is (Ubuntu 22.04). Not sure why it misses that.

Originally posted by @tshadley in bitsandbytes-foundation/bitsandbytes#85 (comment)

I had this error and was stumped by it until I realised that my CUDA toolkit had been installed ~incorrectly and was in a place where bitsandbytes couldn't find it. So when I installed PyTorch and the toolkit via conda instead of pip, the library was able to locate the toolkit and this error went away.

Originally posted by @65536william in bitsandbytes-foundation/bitsandbytes#85 (comment)

[0xdevalias]

I did a lit bit of exploring and playing around, and think I have got it figured out.

You can see my process in this issue comment:

• undefined symbol: cudaRuntimeGetVersion Error bitsandbytes-foundation/bitsandbytes#85 (comment)

But basically it seemed like to use bitsandbytes for the 8bit adam optimiser, I also needed to conda install cudatoolkit as well, which wasn't mentioned here:

• https://github.com/huggingface/diffusers/tree/main/examples/dreambooth#training-on-a-16gb-gpu

Technically it is mentioned here, but that's a bit further removed than the "just pip install bitsandbytes" suggestion in the example docs:

• https://github.com/TimDettmers/bitsandbytes#requirements--installation

So I think the example docs may just need to be updated to point out this dependency requirement more explicitly?

[patrickvonplaten]

Hey @0xdevalias,

Yes indeed I think this is a bitsandbytes issue

[0xdevalias]

In my exploration yesterday (#1207 (comment)) I found that it's less of a bitsandbytes issue, and more of a 'the example docs in this repo don't mention a required dependency of bitsandbytes when telling you to install it, and without that, the example code isn't runnable issue.

So as suggested above:

I think the example docs may just need to be updated to point out this dependency requirement more explicitly

[0xdevalias]

Just opened a PR to fix the example docs:

• add 'conda install cudatoolkit' to dreambooth 'training on 16GB' example #1229

[meanna]

What works for me (cuda 11.6, Python 3.9) is

- !pip install torch
- install packages requires by LLM.int8()
  - !pip install --quiet bitsandbytes
  - !pip install --quiet git+https://github.com/huggingface/transformers.git # Install latest version of transformers
  - !pip install --quiet accelerate
  - !pip install --quiet sentencepiece
- !conda install -y cudatoolkit
- check the path of libcudart.so
  - !find /opt/conda/ -name libcudart.so
    - /opt/conda/lib/libcudart.so
    - /opt/conda/pkgs/cudatoolkit-11.3.1-h2bc3f7f_2/lib/libcudart.so
- The problem is bitsandbytes can not find libcudart.so, so we have to set
  - ```
    import os
    os.environ['LD_LIBRARY_PATH'] = '/opt/conda/lib/'
    ```

Using export LD_LIBRARY_PATH='/opt/conda/lib/' may not work in some settings.