Add descriptions to accelerator broadcast function/clean up all_gather #6044
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SeanNaren
requested review from
awaelchli,
justusschock,
tchaton and
williamFalcon
as code owners
carmocca
approved these changes
Feb 17, 2021
carmocca
reviewed
Feb 17, 2021
| The results of the last training/testing run will be cached within the training type plugin. | ||
| In distributed training, we make sure to transfer the results to the appropriate master process. | ||
| """ | ||
| # TODO: improve these docs |
Contributor
There was a problem hiding this comment.
Improved enough already to remove this?
Contributor
Author
There was a problem hiding this comment.
I don't like rando todos, so let me remove it :)
Contributor
There was a problem hiding this comment.
my comment, but I don't remember what I wanted to improve 🤣
Borda
reviewed
Feb 18, 2021
| def barrier(self, name: Optional[str] = None) -> None: | ||
| self.training_type_plugin.barrier(name=name) | ||
|
|
||
| def broadcast(self, obj: object, src: int = 0) -> object: |
Collaborator
There was a problem hiding this comment.
shall we then rather call source_rank?
Contributor
Author
There was a problem hiding this comment.
Its called source in the lower level pytorch call that is commonly used outside of TPU/some training type plugin:
https://pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast
So I think it might be alright to keep it as is!
Co-authored-by: Jirka Borovec <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #6044 +/- ##
======================================
Coverage 93% 93%
======================================
Files 160 160
Lines 11392 11397 +5
======================================
+ Hits 10578 10590 +12
+ Misses 814 807 -7 |
awaelchli
approved these changes
Feb 18, 2021
Borda
approved these changes
Feb 18, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Address's @Borda comment of adding a better docstring to the broadcast function in #6039
Before submitting
PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:
Did you have fun?
Make sure you had fun coding 🙃