Add img to collective docs #4038
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
TCChenlong
reviewed
Nov 10, 2021
|
|
||
| 假设将NxM的参数矩阵切分到两个设备device_0和device_1。那么每个设置上的参数矩阵为(N/2+1)行和M列。device_0上，输入x中的值如果介于[0, N/2-1]，则其值保持不变；否则值变更为N/2，经过embedding映射为全0值。类似地，device_1上，输入x中的值V如果介于[N/2, N-1]之间，那么这些值将变更为(V-N/2)；否则，值变更为N/2，经过embedding映射为全0值。最后，使用all_reduce_sum操作汇聚各个卡上的结果。 | ||
|
|
||
| 单卡Embedding情况如下图所示 |
Collaborator
There was a problem hiding this comment.
- Embedding 单卡 是不是做成乘法的方式好一点 in * [0, 0, 1, ... 0] -> out？并行同理，这样，比较好描述下面的 00....0是怎么来的？
Contributor
Author
There was a problem hiding this comment.
embedding不能单纯的看做乘法，它是类似于查表操作，split embedding相当于将这个表格分成几份放在不同位置，然后对input进行查表的时候，只会对有这个input的表格拿出对应feature，其他未存放这个input的就输出0
Collaborator
|
有个统一的问题是，文档描述里都是进程怎么怎么样，但是图里都是GPU怎么怎么样，这两个概念最好统一下 |
TCChenlong
reviewed
Nov 12, 2021
Collaborator
TCChenlong
left a comment
TCChenlong
left a comment
There was a problem hiding this comment.
统一 GPU、rank 的使用为 rank；
每篇API文档首次出现rank时，需要括号标注下rank(即GPU，后文统一用rank)。
TCChenlong
reviewed
Nov 16, 2021
| :align: center | ||
|
|
||
| 情形3：列并行Linear | ||
| Linear操作的参数是个NxM的矩阵，行数为N，列数为M。列并行Linear情形下，参数切分到num_partitions个设备，每个设备上的参数是N行、M/num_partitions列的矩阵。 |
TCChenlong
approved these changes
Nov 18, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add illustrations to some collective communication operators docs.