Actor hangs with `ray` using PL v>=1.7

[erezinman]

When invoking a training from within a Ray actor, the training process hangs. The reason for this is that Ray hangs (happens on multiple Ubuntu machines) when forking a process the way you did when calling device_parser.num_cuda_devices. This doesn't happen on v1.6.5.

To sum-up (before delving into the details), the hanging occurs because of the following change between versions:
In v1.6.5 the gpu (changed to cuda in the newer version) accelerator just calls torch.cuda.device_count() > 0 directly, while in 1.7.0 and above, this function torch.cuda.device_count is called in a forked process (utilities/device_parser.py, line 346, in num_cuda_devices):

with multiprocessing.get_context("fork").Pool(1) as pool:
    return pool.apply(torch.cuda.device_count)

The way I deduced the problem is by listening to system-signals and printing the traceback when getting SIGUSR1 within the hanged process. The traceback we obtained was (starting after calling pl.Trainer(...))

File "<__CONDA__ENV__>/lib/python3.9/site-packages/pytorch_lightning/utilities/argparse.py", line 345, in insert_env_defaults
  return fn(self, **kwargs)
File "<__CONDA__ENV__>/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 433, in __init__
  self._accelerator_connector = AcceleratorConnector(
File "<__CONDA__ENV__>/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 212, in __init__
  self._set_parallel_devices_and_init_accelerator()
File "<__CONDA__ENV__>/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 525, in _set_parallel_devices_and_init_accelerator
  if not self.accelerator.is_available():
File "<__CONDA__ENV__>/lib/python3.9/site-packages/pytorch_lightning/accelerators/cuda.py", line 91, in is_available
  return device_parser.num_cuda_devices() > 0
File "<__CONDA__ENV__>/lib/python3.9/site-packages/pytorch_lightning/utilities/device_parser.py", line 346, in num_cuda_devices
  return pool.apply(torch.cuda.device_count)
File "<__CONDA__ENV__>/lib/python3.9/multiprocessing/pool.py", line 736, in __exit__
  self.terminate()
File "<__CONDA__ENV__>/lib/python3.9/multiprocessing/pool.py", line 654, in terminate
  self._terminate()
File "<__CONDA__ENV__>/lib/python3.9/multiprocessing/util.py", line 224, in __call__
  res = self._callback(*self._args, **self._kwargs)
File "<__CONDA__ENV__>/lib/python3.9/multiprocessing/pool.py", line 729, in _terminate_pool
  p.join()
File "<__CONDA__ENV__>/lib/python3.9/multiprocessing/process.py", line 149, in join
  res = self._popen.wait(timeout)
File "<__CONDA__ENV__>/lib/python3.9/multiprocessing/popen_fork.py", line 43, in wait
  return self.poll(os.WNOHANG if timeout == 0.0 else 0)
File "<__CONDA__ENV__>/lib/python3.9/multiprocessing/popen_fork.py", line 27, in poll
  pid, sts = os.waitpid(self.pid, flag)

The hanging is reproduced by forking from within a Ray worker:

import ray
import multiprocessing

@ray.remote
def f():
    with multiprocessing.get_context("fork").Pool(1) as pool:
        return pool.apply(int)

print(ray.get(f.remote()))

Thanks!

cc @tchaton @rohitgr7 @justusschock @kaushikb11 @awaelchli @akihironitta

[abustamante-coveo]

I just realized this was created a few hours before. We have the same problem #14305

[awaelchli]

Hi @abustamante-coveo Thanks for also reporting it. I'm going to close your issue as duplicate and paste your comments here as I understand the issue is identical:

From @abustamante-coveo:

We are currently using ray tune. Ray tune generates trials according to our hyper-parameter grid. In each of of those trials we're training a pytorch model using lightning's Trainer. Starting version 1.7.0, The Trainer fails/gets stuck (we're not sure) and ray tune's trials never finish. Last version of lightning that worked for us is 1.6.5
If I saw correctly, after 1.6.5 the accelerators.cuda.CUDAAccelerator.is_available method was changed from using torch's device_count to using a new module, device_parser. device_parser.num_cuda_devices() is then used and I have tracked the error to the with block: with multiprocessing.get_context("fork").Pool(1) as pool. That with block either never returns or throws an error (we have a hard time debugging this). It fails in such way that ray tune does not understand it has failed so it stays forever waiting for the trial to begin. This happens in the normal ray tune way of working (a separate process created per trial). If I configure ray tune to run "locally" meaning, all the trials in the same process (not creating a separate process for each trial) then there's no problem. Ray tune's local mode is only meant for debugging and we cannot use it in prod.

[awaelchli]

We call torch.cuda.device_count in this special way

with multiprocessing.get_context("fork").Pool(1) as pool:
    return pool.apply(torch.cuda.device_count)

because calling any torch.cuda functions creates a CUDA context. We want to avoid this to support forking in PL (#13405). This trick to fork a process to evaluate the function prevents the CUDA context to be created in the main process.

However, we also have a check there that, if forking is not supported by the system, we just call the torch.cuda.device_count normally. So it really surprises me that you see these problems. Could you help me understand why it is not possible to fork processes in a ray worker? I guess we just need to find a way to detect that whether it is ok to fork or not.

[abustamante-coveo]

We call torch.cuda.device_count in this special way

with multiprocessing.get_context("fork").Pool(1) as pool:
    return pool.apply(torch.cuda.device_count)

because calling any torch.cuda functions creates a CUDA context. We want to avoid this to support forking in PL (#13405). This trick to fork a process to evaluate the function prevents the CUDA context to be created in the main process.

However, we also have a check there that, if forking is not supported by the system, we just call the torch.cuda.device_count normally. So it really surprises me that you see these problems. Could you help me understand why it is not possible to fork processes in a ray worker? I guess we just need to find a way to detect that whether it is ok to fork or not.

We're having a hard time getting more details as to why the with block hangs, as it never returns. I also ignore why forking inside a ray worker has that effect in this case. I can say that I tried to fix it this way:

if I explicitly close/join de pool, the with block returns and the problem is solved.

[erezinman]

How about using multithreading? Would it create a Cuda context? @awaelchli