Do not force `sync_dist=True` on epoch end

[carmocca]

Proposed refactor

https://github.com/PyTorchLightning/pytorch-lightning/blob/fb40cbce2ea4afbcf5842e9ec524f665a3faa95d/pytorch_lightning/trainer/connectors/logger_connector/result.py#L524-L530

Metrics logged inside epoch_end hooks are forcefully synced. This makes the flag irrelevant and causes communication overrhead that might be undesired.

Motivation

This was added #7966

With the reasoning that users would likely forget to set this themselves. Currently, for extra control, using torchmetrics.Metrics is the suggested solution.

Pitch

Do not force this sync and show a warning for 1 epoch suggesting setting sync_dist=True for such metrics.

Additional context

Proposed by @rohitgr7

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @Borda @justusschock @awaelchli @rohitgr7 @carmocca @edward-io @ananthsub @kamil-kaczmarek @Raalsky @Blaizzy

[chongxiaoc]

The above code block changes the log behavior flag on the fly, and users even don't realize that all epoch-end metrics are allreduced after tracing the source code.

I agree with turning off this change.

[awaelchli]

I agree too

[krishnakalyan3]

I would like to work on this.

[carmocca]

One issue from doing this change is that when one logs on step and on epoch, the sync_dist will apply to both cases but one might want to only sync on epoch

That is what would happen currently with log("key", ..., on_step=True, on_epoch=True, sync_dist=False) today because it is forced on epoch end.

Whereas now the user would need to split them

log("key_step", ..., on_step=True, on_epoch=False, sync_dist=False)
log("key_epoch", ..., on_step=False, on_epoch=True, sync_dist=True)