Training stuck at 0% after few epochs while training with DDP

[HareshKarnan]

🐛 Bug

I recently updated to pytorch_lightning 1.1.7 and noticed that after a few epochs of training, the training % is stuck at 0% and never progresses. When I switched back to 1.1.4, this strange behavior does not occur. I do not know the root cause of this issue.

• PyTorch Version (e.g., 1.0): 1.1.7
• OS (e.g., Linux): Linux, U18
• How you installed PyTorch (conda, pip, source): pip install
• Build command you used (if compiling from source):
• Python version: 3.6
• CUDA/cuDNN version: 10.0
• GPU models and configuration: RTX 2080 x3
• Any other relevant information:

[ndrplz]

Related? I'm facing a similar issue, not sure if those might help

• Code stuck on "initalizing ddp" when using more than one gpu  #4612
• NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled system error, NCCL version 2.4.8 pytorch/pytorch#38702

[matyushinleonid]

Hi, @HareshKarnan @ndrplz. Folks, may I ask you to run your pipelines with NCCL_ASYNC_ERROR_HANDLING=1?
Seems like we have the same problem. I use 1.2.0.dev0 lightning version, so this issue may migrate to the new release... My error trace:

https://pastebin.com/LNxG2JF6

[HareshKarnan]

@ndrplz i dont know if it is related, but the problem here is that training does happen for first few epochs - in my case, it ran for 13 epochs and then gets stuck at 0% by epoch 14.

[HareshKarnan]

https://pastebin.com/QhanSNrK

Here is my output. It gets stuck at 0% while training in epoch 16

[matyushinleonid]

https://pastebin.com/QhanSNrK

Here is my output. It gets stuck at 0% while training in epoch 16

run your script with NCCL_ASYNC_ERROR_HANDLING=1 flag. Like NCCL_ASYNC_ERROR_HANDLING=1 python train.py. Without this flag, I do not get any error like you since, by default, this async error does not raise an Exception (at least, for me). If we have the same error traces, it will help community to fix the bug.

[HareshKarnan]

I ran the script again as NCCL_ASYNC_ERROR_HANDLING=1 python train.py and got no difference in the output :

https://pastebin.com/jtLyuXTN

This time it got stuck at epoch 11

[genghisun]

same problem here, stuck at 0% at epoch 18

[Borda]

Mind share code ideally in Colab to reproduce?

[genghisun]

In my case, it stuck at 0% at epoch 18 with 2 gpus ddp before.
Then I try to use only 1 gpu, currently trained for 100+ epochs without any problem.