Skip to content

Add ZeroRedundancyOptimizer Recipe #1375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Feb 26, 2021
Merged

Add ZeroRedundancyOptimizer Recipe #1375

merged 2 commits into from
Feb 26, 2021

Conversation

@mrshenli
Copy link
Contributor

@mrshenli mrshenli commented Feb 25, 2021

No description provided.

@netlify
Copy link

netlify bot commented Feb 25, 2021 •
edited
Loading

Deploy preview for pytorch-tutorials-preview ready!

Built with commit 7b4e3bc

https://deploy-preview-1375--pytorch-tutorials-preview.netlify.app

blefaudeux
blefaudeux reviewed Feb 25, 2021
View reviewed changes
recipes_source/zero_redundancy_optimizer.rst Outdated
----------------------------------

The idea of `ZeroRedundancyOptimizer` comes from the
`DeepSpeed/ZeRO project <https://github.com/microsoft/DeepSpeed>`_ that shards
Copy link

@blefaudeux blefaudeux Feb 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can add another reference here, I can look that up, the idea is older than that actually and it was in a public NLP framework

Copy link
Contributor Author

@mrshenli mrshenli Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @blefaudeux, let me know which reference link we should add here. thanks!

Copy link

@blefaudeux blefaudeux Feb 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has been in Marian for a while it seems, before Zero (I learnt that recently)
https://github.com/marian-nmt/marian-dev

https://github.com/marian-nmt/marian-dev/blob/737f43014a939a3ded2806b00cbaa661fbcc5f49/src/training/communicator_nccl.h#L228

Copy link
Contributor Author

@mrshenli mrshenli Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

blefaudeux
blefaudeux reviewed Feb 25, 2021
View reviewed changes
blefaudeux
blefaudeux approved these changes Feb 25, 2021
View reviewed changes
Copy link

@blefaudeux blefaudeux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except for one sentence, minor

@mrshenli mrshenli force-pushed the zero branch 2 times, most recently from 6e7720a to 99ec1b8 Compare February 25, 2021 22:34
@mrshenli
Copy link
Contributor Author

mrshenli commented Feb 25, 2021

screencapture-deploy-preview-1375-pytorch-tutorials-preview-netlify-app-recipes-zero-redundancy-optimizer-html-2021-02-25-17_49_23

@mrshenli
Copy link
Contributor Author

mrshenli commented Feb 26, 2021

Hey @brianjo, this recipe is ready and the test failures look irrelevant. Shall we land?

@brianjo
Copy link
Contributor

brianjo commented Feb 26, 2021

Let me see if we can get that fixed, and I'll merge it tomorrow. Nikita is looking into it. Thanks!

@brianjo brianjo merged commit ef58ce1 into pytorch:master Feb 26, 2021
gaoteng-git pushed a commit to gaoteng-git/tutorials that referenced this pull request Mar 2, 2021
@mrshenli @brianjo @gaoteng-git
Add ZeroRedundancyOptimizer Recipe (pytorch#1375)
19218cf 
Co-authored-by: Brian Johnson <[email protected]>
rodrigo-techera pushed a commit to Experience-Monks/tutorials that referenced this pull request Nov 29, 2021
@mrshenli @brianjo
Add ZeroRedundancyOptimizer Recipe (pytorch#1375)
66d636d 
Co-authored-by: Brian Johnson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@blefaudeux blefaudeux blefaudeux approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mrshenli @brianjo @blefaudeux @facebook-github-bot