Added Horovod on Ray accelerator for remote distributed training

[tgaddair]

What does this PR do?

At Uber, we're moving a lot of our deep learning training infrastructure to Ray. As a result, being able to launch Horovod jobs using Horovod on Ray with PyTorch Lightning will greatly simplify the user experience for our customers wishing explore different backends.

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In in short, see following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified; Bugfixes should be including in bug-fix release milestones (m.f.X) and features should be included in (m.X.b) releases.

Did you have fun?

Make sure you had fun coding 🙃

[tgaddair]

cc @richardliaw

[williamFalcon]

hey guys, this is awesome!

However, let's pause on this for a second as we are currently on a third-party integrations hold since it's creating a lot of overhead for us to maintain. I'd prefer to make this an upstream support on the horovod or ray repo? that would look like implementing the accelerator there and hosting on that end.

let's sync up on slack first! cc @edenafek

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need further help see our docs: https://pytorch-lightning.readthedocs.io/en/latest/CONTRIBUTING.html#pull-request or ask the assistance of a core contributor here or on Slack. Thank you for your contributions.

[stale]

This pull request is going to be closed. Please feel free to reopen it create a new from the actual master.