Code stuck on "initalizing ddp" when using more than one gpu

[JosephGatto]

🐛 Bug

I am trying to run a pytorch lightning model on a 4-GPU node. In my trainer, if I specify

pl.Trainer(gpus=[0])

It runs fine. However, once I add another GPU

pl.Trainer(gpus=[0,1,2,3])

I get this output:

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/4
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/4
initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/4
initializing ddp: GLOBAL_RANK: 3, MEMBER: 4/4

And the model just hangs there forever. I have tried this with only 2 GPUs and get the same behavior.

Any idea why this may happen? I have tried with both ddp and ddp_spawn.

• PyTorch Version-- tried both 1.4 and 1.7
• OS-- Linux
• Installed with pip
• Python version: 3.8.5
• CUDA/cuDNN version: 10.1
• GPU models and configuration: NVIDIA K80s

[github-actions]

Hi! thanks for your contribution!, great first issue!

[edenlightning]

Hey! Can you try to reproduce using our simple boring model?

Just to verify the bug isn't something in your model.

[JosephGatto]

Thanks for the response @edenlightning. This morning (about 12 hrs after my last attempt), I ran the simple boring model and it ran. I also added gpu=[0] and it worked. When I added gpu=[0,1] it worked the first time I ran it. I successfully ctrl+c'd out of it, then tried to run it again and it hangs forever. nvidia-smi shows no processes running and I can continually run the single gpu approach after this. However, anytime I do gpu > 1 from now on it does not run.

Thus, it seems like something bad happens when I ctrl+c a ddp process that blocks it from happening again. Any idea what that might be?

[JosephGatto]

Update: 1) Have had my system admin restart this GPU node and it didn't run again. Not sure how it got through that one time. 2) None of the accelerators ['ddp', 'ddp_spawn', 'dp'] run for me when using gpu=[0,1] on boring model.

Just curious, could this be a related issue? pytorch/pytorch#1637 (comment) I am working on a 4-K80 node.

[JosephGatto]

@edenlightning any ideas about what the problem might be? ddp still isnt working on the boring model and ddp_spawn gives the pickle error on the boring model ... i am very stuck.

[edenlightning]

@justusschock mind taking a look?

[justusschock]

Hi @JosephGatto I am sorry that I cannot reproduce this, since I don't have these kind of GPUs. But I can try to guide you during troubleshooting.

Yes pytorch/pytorch#1637 (comment) seems to be related. Have you tried the steps mentioned there to track down the problem?

[JosephGatto]

Hi @justusschock thanks for offering your help! Sadly, this did not work. Any other ideas?

[SohamTamba]

Hi,
I had this problem to while running PL on my university's SLURM cluster. I was trying to use DDP on 4 GPUs.

Following the tutorial, I set tasks per node to 4 (number of GPUs) and it hung on Initializing DDP.

I solved this by setting tasks per node to 8.