AttributeError: 'float' object has no attribute 'clone' - when logging float values with sync_dist=True and ddp

[stachu86]

🐛 Bug

self.log('d_acc_dummy' , 100.0, on_step=True, on_epoch=True, sync_dist=True) 

results in

/usr/local/lib/python3.6/dist-packages/pytorch_lightning/core/step_result.py in log(self, name, value, prog_bar, logger, on_step, on_epoch, reduce_fx, tbptt_reduce_fx, tbptt_pad_token, enable_graph, sync_dist, sync_dist_op, sync_dist_group, sync_fn)
    138             is_dist_initialized = torch.distributed.is_available() and torch.distributed.is_initialized()
    139             # TODO: Find a way to make the reduction only once, so we don't need to clone.
--> 140             value = value.clone() if is_dist_initialized else value
    141             value = sync_fn(value, group=sync_dist_group, reduce_op=sync_dist_op)
    142 

AttributeError: 'float' object has no attribute 'clone'

Float is allowed type (from step_result.py)

 if sync_dist and isinstance(value, (torch.Tensor, numbers.Number)):
            is_dist_initialized = torch.distributed.is_available() and torch.distributed.is_initialized()
            # TODO: Find a way to make the reduction only once, so we don't need to clone.
            value = value.clone() if is_dist_initialized else value
            value = sync_fn(value, group=sync_dist_group, reduce_op=sync_dist_op)

Please reproduce using [the BoringModel and post here]

Simple gan example

To Reproduce

Expected behavior

Logging of float values

Environment

• CUDA:
  • GPU:
    • Tesla T4
  • available: True
  • version: 10.1
• Packages:
  • numpy: 1.18.5
  • pyTorch_debug: True
  • pyTorch_version: 1.7.0+cu101
  • pytorch-lightning: 1.0.8
  • tqdm: 4.41.1
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.6.9
  • version: Proposal for help #1 SMP Thu Jul 23 08:00:38 PDT 2020

Additional context

[github-actions]

Hi! thanks for your contribution!, great first issue!

[Borda]

I think you shall be using Tensor, @SkafteNicki?

[SkafteNicki]

@Borda not metric package related, logging related.
But I do agree that manually casting the number to tensor should get it working for now, but we should probably do this internally.

[awaelchli]

value = torch.tensor(value, device=.... dtype=...)
value = value.clone()

should be sufficient, but perhaps the conversion can be done earlier and cloning should not be necessary in case value was a float. @stachu86 are you interested in working on this?

[stachu86]

@awaelchli yes, it looks to me like a good first contribution. I was little discouraged by the #TODO comment that suggests somebody will rewrite this soon, but from my understanding clone() is necessary here because torch.distributed.all_reduce operates in-place. However, isn't value.detach().clone() would be more appropriate in the logging context?