c10::error during DistributedDataParallel training

[edraizen]

Hi, I am coming across this strange error when I train a MinkUNet using Pytorch Lightning/DistributedDataParallel on 1 node with 4 K80 gpus. The cluster I am using also has P100 and V100 gpus, but the model works fine on those (with 1 node).

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: the launch timed out and was terminated
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f1be4ca61e2 in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7f1be4ef4f92 in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f1be4c949cd in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: std::vector<c10d::Reducer::Bucket, std::allocator<c10d::Reducer::Bucket> >::~vector() + 0x25a (0x7f1c30d825da in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: c10d::Reducer::~Reducer() + 0x28a (0x7f1c30d7785a in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: std::_Sp_counted_ptr<c10d::Reducer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x12 (0x7f1c30d57102 in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x46 (0x7f1c30554b56 in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0xa6bdcb (0x7f1c30d57dcb in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x273fc0 (0x7f1c3055ffc0 in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x27520e (0x7f1c3056120e in /project/ppi_workspace/py37/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #18: __libc_start_main + 0xe7 (0x7f1c32b71b97 in /lib/x86_64-linux-gnu/libc.so.6)

I tried again using horovod (with pytorch, ME, nccl, infiniband, etc installed inside a singularity container). All GPU types with 1 node succeed, but then they all fail when I increase number of nodes to 2 with the same error as above.

According to pytorch/pytorch#13541, it has to do with GLIBCXX_USE_CXX11_ABI set to 1, but I cannot seem to turn it off.

I also posted this to pytorch discuss (https://discuss.pytorch.org/t/setting-glibcxx-use-cxx11-abi-to-0/98054) since I thought it might be more relevant to pytorch. However, I also saw a few people with same error here (#205, #193) so I thought I'd ask here as well.

If you have any ideas what could be causing this, I would would really appreciate it.

Thanks!

[chrischoy]

Hmm, I am using ME with DDP without a problem. What pytorch version, CUDA version are you using?

[edraizen]

10.1 and 10.2. DDP works without a problem on all GPUs except the K80, which is why I am very confused!

[chrockey]

@edraizen
Hi, I have plan to reorganize my code into PyTorch-lightning.
Is it okay to use PyTorch-lightning and MinkowskiEngine together? (especially about synchronized batch normalization)
If you share your experience, it will be greatly helpful!

Thank you!

[chrischoy]

Please refer to the lightning multi gpu example on https://github.com/NVIDIA/MinkowskiEngine/blob/master/examples/multigpu_lightning.py.

Closing the issue since I cannot reproduce this error on multi gpu training.

[anandcu3]

Hi, I'm facing a similar error when using inference in multi-GPUs. @chrischoy Any ideas about this?

terminate called after throwing an instance of 'c10::CUDAError'
 what(): CUDA error: driver shutting down
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from query at /opt/conda/conda-bld/pytorch_1646756402876/work/aten/src/ATen/cuda/CUDAEvent.h:95 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f89cc6bb1bd in /opt/conda/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x11a (0x7f8a0a5496ea in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda_cpp.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x50 (0x7f8a0a54bcd0 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda_cpp.so)
frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x145 (0x7f8a0a54cf65 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda_cpp.so)
frame #4: <unknown function> + 0xcdaa3 (0x7f8a85db1aa3 in /opt/conda/lib/python3.9/site-packages/pyarrow/../../../libstdc++.so.6)
frame #5: <unknown function> + 0x76db (0x7f8a975136db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x3f (0x7f8a9688f61f in /lib/x86_64-linux-gnu/libc.so.6)

This happens after the entire inference script has completed and just before the python script ends. I am not sure if it's related to MinkowskiEngine, but just trying to check if it's related to this old issue. This also doesn't happen on every run and hence not easy to reproduce the error