RuntimeError: CUDA error: invalid device ordinal in pytorch_lightning version:1.2.7

[zhhao1]

🐛 Bug

An error is reported when using multiple nodes and multiple GPUs on slurm. It seems it always initialize on only one node.
I only meet this question in version 1.2.7, but all is ok in version 1.1.2.
LOCAL_RANK: 0 -CUDA_VISIBLE_DEVICES: [0,1]
initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/4
LOCAL_RANK: 0 -CUDA_VISIBLE_DEVICES: [0,1]
initializing ddp: GLOBAL_RANK: 3, MEMBER: 4/4
LOCAL_RANK: 0 -CUDA_VISIBLE_DEVICES: [0,1]
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/4
LOCAL_RANK: 0 -CUDA_VISIBLE_DEVICES: [0,1]
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/4
my trainer is:
trainer = Trainer(gpus=2, accelerator='ddp', num_nodes=2)
my slrum script is:
#SBATCH --nodes=2
#SBATCH --gres=gpu:2
#SBATCH --ntasks-per-node=2

• PyTorch Version: 1.7.1
• OS: Centos
• How you installed PyTorch: pip
• Python version: 3.7.0
• CUDA/cuDNN version: 10.1

Additional context

[awaelchli]

Is it working on 1.2.8? I tried to fix it here: #6941
And I got confirmation from another slurm user that it's working now.

[zhhao1]

Is it working on 1.2.8? I tried to fix it here: #6941
And I got confirmation from another slurm user that it's working now.

Is there any flags need to be set in Trainer?

[awaelchli]

no, just use sbatch to submit your job as usual. I think the only thing you need to make sure is that you set the same number of nodes and gpus in the Trainer as in the submission script.
And lightning should detect that you run with slurm based on the env vars (e.g. SLURM_PROCID etc.)

[awaelchli]

any update here? have you managed to get it working with Lightning >= 1.2.8?

[zhhao1]

any update here? have you managed to get it working with Lightning >= 1.2.8?

Sorry, there are still some problems, but you mentioned that someone has successfully used it. I guess this may be my own problem. I'm trying to find the problem.