powersgd

[titusnicolae]

Powersgd paper shows promising scaling for distributed training.
https://arxiv.org/abs/1905.13727
I'm interested in porting this to pytorch-lightning, do you think it's a good idea?
Thanks

[jeffling]

@titusnicolae I'm a big fan of this. I'm not sure if it needs to be a framework level feature, or if we should find a way to make it easy to plug in. But I would love to see a lightning-compatible implementation of this, whether in a PR or as a separate plugin.

[williamFalcon]

@titusnicolae @soumith this might be better in PyTorch?

[Borda]

@titusnicolae @jeffling any update here?

[jeffling]

@titusnicolae any luck on porting this over?

If it could be done in a callback, it should be. otherwise, I think if there is enough proof / demand behind this it would make sense to be in the framework.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.