"MisconfigurationException: No supported gpu backend found!" with multi gpu training in jupyter notebooks

[vacmar01]

Bug description

When trying to train on two GPUs in a jupyter notebooks environment on jarvislabs.ai with ddp_notebooks I get the following error "MisconfigurationException: No supported gpu backend found!".

I'm trying to train on two RTX 5000 GPUs. On a Kaggle GPU the same code runs without any problem.

Any ideas?

How to reproduce the bug

trainer = pl.Trainer(
    max_epochs=2, 
    accelerator="gpu",
    devices=2,
    precision=16,
    accumulate_grad_batches=2
)
trainer.fit(model, train_dl, val_dl)

Error messages and logs

"MisconfigurationException: No supported gpu backend found!"

Environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0): 1.7.7
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 1.10): 1.11
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version: V11.6.55
#- GPU models and configuration: 2x RTX 5000
#- How you installed Lightning(`conda`, `pip`, source): pip
#- Running environment of LightningApp (e.g. local, cloud): jarvislabs.ai 

More info

No response

cc @justusschock @awaelchli

[rohitgr7]

cc: @awaelchli

[awaelchli]

@vacmar01 Was your PyTorch installed with GPU support? I suspect that it was not. Please check what

import torch
print(torch.cuda.is_available())

returns for you. If not, please install it like so: pip3 install torch --extra-index-url https://download.pytorch.org/whl/cu116.

[vacmar01]

It is installed with GPU support: torch.cuda.is_available() returns True and torch.cuda.device_count() returns 2. But after I call these functions I can't start training anymore because of the "process forking" error.

[awaelchli]

@vacmar01 I cannot reproduce on our multi-GPU machine with Jupyter notebook. I haven't tried with jarvislabs.ai but I don't think it is related. There was some issue in 1.7.7 with precision=16 and the "process forking" error that you described that we fixed.

Would you mind checking again by installing our development version from master to see if your GPUs get properly detected? To install from master, simply run:

pip install https://github.com/Lightning-AI/lightning/archive/refs/heads/master.zip -U

[vacmar01]

I suppose it has something to do with the GPU or the CUDA version, since on Kaggle the exact same code ran with no problem.

I will install the development version of lightning and try again. Thank you!

[vacmar01]

Okay so I installed the dev version of lightning and I get a new error (with precision=32 or precision=16, doesn't matter):

RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!

Whats funny is that now starting the trainer logs the following:

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
[W CUDAFunctions.cpp:112] Warning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (function operator())

So it seems to recognize the GPU but it still doesn't work.

[vacmar01]

Any updates on this?

[awaelchli]

[W CUDAFunctions.cpp:112] Warning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (function operator())

This warning is from torch. Can you make sure that you have the latest driver installed? Perhaps a different cuda version works?

Have you run any pytorch examples? I'm sure you will get the same error there too.

[RylanSchaeffer]

@awaelchli I'm getting the same error, albeit with plain Python (no jupyter).

$ python3
Python 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.12.1+cu113'
>>> import pytorch_lightning as pl
>>> pl.__version__
'1.8.6'

Your suggested command print(torch.cuda.is_available()) prints True, but I still get:

lightning_lite.utilities.exceptions.MisconfigurationException: No supported gpu backend found!

I'm using Cuda 11.3. Any suggestions for identifying the cause?

[RylanSchaeffer]

Downgrading to PL Lightning 1.7.7 works for me. I don't know what the cause of the problem is!