all_gather raises NotImplementedError when no Accelerator defined in Trainer

[8greg8]

🐛 Bug

When no Accelerator is defined in Trainer, all_gather function in LightningModule raises NotImplementedError.

Please reproduce using the BoringModel and post here

https://colab.research.google.com/drive/1VPEIaQ-aN5KVA70VtvGk24AVkTYPvMoY?usp=sharing

To Reproduce

Expected behavior

• All gather should return a value instead.

Environment

• CUDA:
  • GPU:
    • Tesla P100-PCIE-16GB
  • available: True
  • version: 10.1
• Packages:
  • numpy: 1.19.4
  • pyTorch_debug: True
  • pyTorch_version: 1.7.0+cu101
  • pytorch-lightning: 1.1.2rc1
  • tqdm: 4.41.1
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.6.9
  • version: Proposal for help #1 SMP Thu Jul 23 08:00:38 PDT 2020

Additional context

[awaelchli]

Fyi your colab notebook is not public, it can't be accessed :)

[8greg8]

@awaelchli sorry my bad. Corrected and changed the link in bug description. You should be able to access now.

[awaelchli]

@tchaton we should provide implementations for all_gather on cpu and single gpu to make code device agnostic. Does that make sense?

[tchaton]

Hey @awaelchli,

Yes, we should.

Best,
T.C