SLURM mpirun/openmpi training selects horovod automatically when not requested

[marcelschilling]

🐛 Bug

I tried to do a hyperparameter search using wandb sweep function on a SLURM cluster by mpirun (openmpi 4.0).
However, during training always horovod as distributed backend is tried to be selected though the trainer config is accelerator=None/distributed_backend=None/gpus=1.

Error:
'Requested accelerator="horovod", but Horovod is not installed.'
"Install with \n $HOROVOD_WITH_PYTORCH=1 pip install horovod[pytorch]"

I use mpirun like this

mpirun
--display-map
--display-allocation
--map-by ppr:2:socket:pe=19
bash -c 'export CUDA_VISIBLE_DEVICES=${OMPI_COMM_WORLD_LOCAL_RANK}; wandb agent SWEEPID'

Also tried to install $HOROVOD_WITH_PYTORCH=1 pip install horovod[pytorch]. But in this case, training won't start and hangs in a never ending loop (but no error occurs).

Using

CUDA_VISIBLE_DEVICES=0 wandb agent $SWEEPID &
CUDA_VISIBLE_DEVICES=1 wandb agent $SWEEPID &
CUDA_VISIBLE_DEVICES=2 wandb agent $SWEEPID &
CUDA_VISIBLE_DEVICES=3 wandb agent $SWEEPID

works fine.

Environment

• PyTorch Lightning Version: 1.2.10
• PyTorch Version: 1.8
• Python version: 3.8.0
• OS (e.g., Linux): Linux on SLURM cluster
• openmpi 4.0
• CUDA/cuDNN version: 11.2
• GPUs: one node with 4 Tesla A100
• How you installed PyTorch (conda, pip, source): conda

Do you have any idea, why horovod is implictly selected instead of doing 4 "normal trainings in parallel" with different parameter settings?

Thanks in advance!

Best,
Marcel

[awaelchli]

The reason is probably this line of code where we check for the presence of OMPI_COMM_WORLD_RANK and in your mpirun command it probably gets set.

But if you set the accelerator explicitly, you should be able to prevent this. Mind trying this out?

[marcelschilling]

Thank you very much @awaelchli. I tried to set it explicitly to None (accelerator and distributed_backend) this lead to the problem. However, using the setting of accelerator="dp" fixed it. Thanks a lot.

Is there a difference/disadvantage using dp algorithm with one GPU compared to a normal training with one GPU or is it exactly the same in case of gpu=1?

Do you have an idea concerning reasons for not starting the training in case of using horovod on SLURM with pyTorch lightning?

[awaelchli]

@marcelschilling actually no, I believe you did everything right the first time. I submitted a fix to prevent Lightning from selecting horovod when running with mpirun and horovod is not installed.

For the time being, you can set accelerator="ddp" which should have no performance impact and workaround the issue until this is fixed.

[marcelschilling]

@awaelchli Thank you so much! :)