Add automatic GPU choice to trainer

[ahendriksen]

This commit adds the gpu_choice parameter to Trainer. By default,
this parameter is set to 'manual' which causes no observable
difference in behavior.

When gpu_choice is set to "auto" and gpus is an int, then the
trainer will automatically allocate the first available GPU.
This is especially useful when GPUs are configured to be in "exclusive
mode", which means that only one process at a time can use them.

Before submitting

• Was this discussed/approved via a Github issue? Automatically pick available GPU #951
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?
• If you made a notable change (that affects users), did you update the CHANGELOG?

What does this PR do?

Fixes #951

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Comments:

1. I am not sure if the pick_gpu functions should be placed in distrib_parts.py. I chose this because determine_root_gpu_device` is defined there.
2. What is the idiomatic way to you specify that a test should only run when CUDA is available?
3. I am unsure what possible interactions with distributed training there are and if so how I should handle them.
4. I added a docstring, does that count as adding docs?

Did you have fun?

Make sure you had fun coding 🙃

[pep8speaks]

Hello @ahendriksen! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-04-10 08:00:40 UTC

[williamFalcon]

I don't think the backoff would work because lightning scopes the visible GPUs no? so this needs to work with our auto-gpu scoping.

ie: do backoff first to id the GPUs... THEN set the cuda flags

https://github.com/PyTorchLightning/pytorch-lightning/blob/9754c5da55059dd89cf0a4fd582fe5df9449bbe5/pytorch_lightning/trainer/distrib_data_parallel.py#L267

[ahendriksen]

If I understand correctly, this PR does backoff first to id the GPUs and THEN sets the cuda flags.

set_nvidia_flags is called after the gpus have been determined.

GPUs are determined here: https://github.com/PyTorchLightning/pytorch-lightning/blob/2d5db37eb7ade5ae1c4a861bfb231c1db08e512e/pytorch_lightning/trainer/trainer.py#L404

Nvidia flags are set a couple lines later:
https://github.com/PyTorchLightning/pytorch-lightning/blob/2d5db37eb7ade5ae1c4a861bfb231c1db08e512e/pytorch_lightning/trainer/trainer.py#L437

Is there something I am missing?

[williamFalcon]

@ahendriksen makes sense!

[williamFalcon]

what about just
auto_select_gpus=T|F
?

[mergify]

This pull request is now in conflict... :(

[codecov]

Codecov Report

Merging #1426 into master will decrease coverage by 0%.
The diff coverage is 60%.

@@          Coverage Diff           @@
##           master   #1426   +/-   ##
======================================
- Coverage      92%     91%   -0%     
======================================
  Files          66      66           
  Lines        3509    3542   +33     
======================================
+ Hits         3213    3232   +19     
- Misses        296     310   +14     

[williamFalcon]

really cool! @ahendriksen

[ahendriksen]

Nice! Thanks for merging. I had fun 🎉