Resolve several Horovod test failures

[carmocca]

🐛 Bug

We are currently skipping several Horovod tests. We should eventually address each one.

• Coverage issue: https://github.com/PyTorchLightning/pytorch-lightning/blob/b9475c1ae913b647c18cbf5f445c53c2f793a7f5/tests/models/test_horovod.py#L52
• Unclear feature: https://github.com/PyTorchLightning/pytorch-lightning/blob/b9475c1ae913b647c18cbf5f445c53c2f793a7f5/tests/models/test_horovod.py#L154-L156
• Flaky test on ubuntu 1. Produces a segfault: https://github.com/PyTorchLightning/pytorch-lightning/blob/b9475c1ae913b647c18cbf5f445c53c2f793a7f5/tests/models/test_horovod.py#L243
• Flaky test on ubuntu 2. Produces a segfault: https://github.com/PyTorchLightning/pytorch-lightning/blob/9402c052ed5a8acddce1b3afaafa5c881ed0c207/tests/models/test_horovod.py#L380
• Permissions issue 1: https://github.com/PyTorchLightning/pytorch-lightning/blob/b9475c1ae913b647c18cbf5f445c53c2f793a7f5/tests/models/test_horovod.py#L276
• Permissions issue 2: https://github.com/PyTorchLightning/pytorch-lightning/blob/b9475c1ae913b647c18cbf5f445c53c2f793a7f5/tests/models/test_horovod.py#L326

cc: @Borda @tgaddair

[Borda]

@tgaddair could you pls check these issues? 🐰

[edenlightning]

@tgaddair friendly ping :)