AttributeError: module 'horovod.torch' has no attribute 'nccl_built'

[daniellepintz]

The PL.pytorch-lightning (GPUs) CI job is failing with AttributeError: module 'horovod.torch' has no attribute 'nccl_built'

https://dev.azure.com/PytorchLightning/pytorch-lightning/_build/results?buildId=61024&view=logs&j=3afc50db-e620-5b81-6016-870a6976ad29&t=059bb337-5de1-5e6a-4c56-42f55a328c0e

cc @tchaton @rohitgr7 @akihironitta @carmocca @Borda

[awaelchli]

There is this message:

Extension horovod.torch has not been built: /usr/local/lib/python3.7/dist-packages/horovod/torch/mpi_lib/_mpi_lib.cpython-37m-x86_64-linux-gnu.so not found
If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error.
Warning! MPI libs are missing, but python applications are still available.

There seems to be a missing MPI package.
The GPU uses the docker container from this file.
We can try to add it there. Horrible

[akihironitta]

@awaelchli @Borda I'll try to fix this in the next hour(s). If you're already finishing up your fix (or you're working on it now), let me now here or in the chat to avoid conflict in pushing fixed docker image to hub :)

[awaelchli]

@akihironitta any ideas and help are very welcome. Feel free to give it a try.

[carmocca]

Could it be because Horovod gets built without MPI? https://github.com/PyTorchLightning/pytorch-lightning/blob/bc812077c41896c93205fe6e9e3d18485d83a2e5/dockers/base-cuda/Dockerfile#L109

Has anyone tried switching it to WITH_MPI?

[akihironitta]

@carmocca Yes, I have tried it locally before, but got the same result... I think Jirka also tried it in #12318, but it seems like didn't work either? @Borda

---

Kind of hakcy, but looks like reinstalling it somehow resolved the issue in my PR #12353. It's now building the docker image and publishing to the docker hub. If that succeeds, we'll first merge the PR, and I'll investigate it further then.