ddp no_sync

[ruotianluo]

🚀 Feature

DDP has this no_sync context manager.
https://pytorch.org/docs/stable/nn.html?highlight=no_sync#torch.nn.parallel.DistributedDataParallel.no_sync

Could be good to add it when doing gradient accumulation.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!