Add 2d-2d support to MXFP8 Grouped GEMM #4816
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for pytorch-fbgemm-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
dc99695 to
069b945
Compare
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
069b945 to
b008808
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
b008808 to
953bbe8
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
953bbe8 to
484cea5
Compare
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
484cea5 to
ec96115
Compare
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
ec96115 to
5d9c6dd
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
5d9c6dd to
b0c77bb
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
b0c77bb to
018b5d2
Compare
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
018b5d2 to
66e7a12
Compare
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
66e7a12 to
08430a6
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
b34875f to
bc0c554
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
bc0c554 to
812d712
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
812d712 to
4b3651c
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
4b3651c to
4914467
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
danielvegamyhre
added a commit
to danielvegamyhre/FBGEMM
that referenced
this pull request
Sep 4, 2025
Summary: Pull Request resolved: pytorch#4816 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
4914467 to
0755452
Compare
Summary: X-link: facebookresearch/FBGEMM#1846 ## MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API - Add support for 2d-2d inputs with dynamic groups along K dimension - Added tests to ensure correct numerics for both 2d-2d and 2d-3d cases, with randomly group sizes - Add benchmarks for both 2d-3d and 2d-2d cases Reviewed By: ngimel, cthi Differential Revision: D81362680
danielvegamyhre
force-pushed
the
export-D81362680
branch
from
0755452 to
f809b74
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D81362680 |
Contributor
|
This pull request has been merged in c6a8daf. |
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
Sep 6, 2025
…ump (#162209) ## Summary - We just landed 2d-2d support for mxfp8 grouped gemm in FBGEMM: pytorch/FBGEMM#4816 - This is needed for backward pass of mxfp8 MoE training with grouped gemms - Changes: - Add dispatching + input validation for mxfp8 grouped gemm in `torch._scaled_grouped_mm` - Add meta registration input validation for mxfp8 grouped gemm, for composability with compile - Add unit tests exercising torch._scaled_grouped_mm with mxfp8 inputs - Bump FBGEMM third party submodule to include: - pytorch/FBGEMM#4816 - pytorch/FBGEMM#4820 - pytorch/FBGEMM#4821 - pytorch/FBGEMM#4823 #### How fbgemm dependency was bumped Documenting this since I haven't found it documented elsewhere: - `cd ~/pytorch/third_party/fbgemm` - `git fetch` - `git checkout <hash>` - `cd ~/pytorch` - `git add third_party/fbgemm` ## Test plan #### Test build ``` USE_FBGEMM_GENAI=1 python -m pip install --no-build-isolation -v -e . ... Successfully installed torch-2.9.0a0+gitf5070f3 ``` [full build log](https://www.internalfb.com/phabricator/paste/view/P1933787581) #### Unit tests ``` pytest test/test_matmul_cuda.py -k test_mxfp8_scaled_grouped_mm_ ... test/test_matmul_cuda.py ......... [100%] ============================================================== 9 passed, 1668 deselected in 5.34s =============================================================== ``` Pull Request resolved: #162209 Approved by: https://github.com/ngimel
daisyden
pushed a commit
to daisyden/pytorch
that referenced
this pull request
Sep 8, 2025
…ump (pytorch#162209) ## Summary - We just landed 2d-2d support for mxfp8 grouped gemm in FBGEMM: pytorch/FBGEMM#4816 - This is needed for backward pass of mxfp8 MoE training with grouped gemms - Changes: - Add dispatching + input validation for mxfp8 grouped gemm in `torch._scaled_grouped_mm` - Add meta registration input validation for mxfp8 grouped gemm, for composability with compile - Add unit tests exercising torch._scaled_grouped_mm with mxfp8 inputs - Bump FBGEMM third party submodule to include: - pytorch/FBGEMM#4816 - pytorch/FBGEMM#4820 - pytorch/FBGEMM#4821 - pytorch/FBGEMM#4823 #### How fbgemm dependency was bumped Documenting this since I haven't found it documented elsewhere: - `cd ~/pytorch/third_party/fbgemm` - `git fetch` - `git checkout <hash>` - `cd ~/pytorch` - `git add third_party/fbgemm` ## Test plan #### Test build ``` USE_FBGEMM_GENAI=1 python -m pip install --no-build-isolation -v -e . ... Successfully installed torch-2.9.0a0+gitf5070f3 ``` [full build log](https://www.internalfb.com/phabricator/paste/view/P1933787581) #### Unit tests ``` pytest test/test_matmul_cuda.py -k test_mxfp8_scaled_grouped_mm_ ... test/test_matmul_cuda.py ......... [100%] ============================================================== 9 passed, 1668 deselected in 5.34s =============================================================== ``` Pull Request resolved: pytorch#162209 Approved by: https://github.com/ngimel
markc-614
pushed a commit
to markc-614/pytorch
that referenced
this pull request
Sep 17, 2025
…ump (pytorch#162209) ## Summary - We just landed 2d-2d support for mxfp8 grouped gemm in FBGEMM: pytorch/FBGEMM#4816 - This is needed for backward pass of mxfp8 MoE training with grouped gemms - Changes: - Add dispatching + input validation for mxfp8 grouped gemm in `torch._scaled_grouped_mm` - Add meta registration input validation for mxfp8 grouped gemm, for composability with compile - Add unit tests exercising torch._scaled_grouped_mm with mxfp8 inputs - Bump FBGEMM third party submodule to include: - pytorch/FBGEMM#4816 - pytorch/FBGEMM#4820 - pytorch/FBGEMM#4821 - pytorch/FBGEMM#4823 #### How fbgemm dependency was bumped Documenting this since I haven't found it documented elsewhere: - `cd ~/pytorch/third_party/fbgemm` - `git fetch` - `git checkout <hash>` - `cd ~/pytorch` - `git add third_party/fbgemm` ## Test plan #### Test build ``` USE_FBGEMM_GENAI=1 python -m pip install --no-build-isolation -v -e . ... Successfully installed torch-2.9.0a0+gitf5070f3 ``` [full build log](https://www.internalfb.com/phabricator/paste/view/P1933787581) #### Unit tests ``` pytest test/test_matmul_cuda.py -k test_mxfp8_scaled_grouped_mm_ ... test/test_matmul_cuda.py ......... [100%] ============================================================== 9 passed, 1668 deselected in 5.34s =============================================================== ``` Pull Request resolved: pytorch#162209 Approved by: https://github.com/ngimel
This was referenced Sep 17, 2025
mansiag05
pushed a commit
to mansiag05/pytorch
that referenced
this pull request
Sep 22, 2025
…ump (pytorch#162209) ## Summary - We just landed 2d-2d support for mxfp8 grouped gemm in FBGEMM: pytorch/FBGEMM#4816 - This is needed for backward pass of mxfp8 MoE training with grouped gemms - Changes: - Add dispatching + input validation for mxfp8 grouped gemm in `torch._scaled_grouped_mm` - Add meta registration input validation for mxfp8 grouped gemm, for composability with compile - Add unit tests exercising torch._scaled_grouped_mm with mxfp8 inputs - Bump FBGEMM third party submodule to include: - pytorch/FBGEMM#4816 - pytorch/FBGEMM#4820 - pytorch/FBGEMM#4821 - pytorch/FBGEMM#4823 #### How fbgemm dependency was bumped Documenting this since I haven't found it documented elsewhere: - `cd ~/pytorch/third_party/fbgemm` - `git fetch` - `git checkout <hash>` - `cd ~/pytorch` - `git add third_party/fbgemm` ## Test plan #### Test build ``` USE_FBGEMM_GENAI=1 python -m pip install --no-build-isolation -v -e . ... Successfully installed torch-2.9.0a0+gitf5070f3 ``` [full build log](https://www.internalfb.com/phabricator/paste/view/P1933787581) #### Unit tests ``` pytest test/test_matmul_cuda.py -k test_mxfp8_scaled_grouped_mm_ ... test/test_matmul_cuda.py ......... [100%] ============================================================== 9 passed, 1668 deselected in 5.34s =============================================================== ``` Pull Request resolved: pytorch#162209 Approved by: https://github.com/ngimel
cleonard530
pushed a commit
to cleonard530/pytorch
that referenced
this pull request
Sep 22, 2025
…ump (pytorch#162209) ## Summary - We just landed 2d-2d support for mxfp8 grouped gemm in FBGEMM: pytorch/FBGEMM#4816 - This is needed for backward pass of mxfp8 MoE training with grouped gemms - Changes: - Add dispatching + input validation for mxfp8 grouped gemm in `torch._scaled_grouped_mm` - Add meta registration input validation for mxfp8 grouped gemm, for composability with compile - Add unit tests exercising torch._scaled_grouped_mm with mxfp8 inputs - Bump FBGEMM third party submodule to include: - pytorch/FBGEMM#4816 - pytorch/FBGEMM#4820 - pytorch/FBGEMM#4821 - pytorch/FBGEMM#4823 #### How fbgemm dependency was bumped Documenting this since I haven't found it documented elsewhere: - `cd ~/pytorch/third_party/fbgemm` - `git fetch` - `git checkout <hash>` - `cd ~/pytorch` - `git add third_party/fbgemm` ## Test plan #### Test build ``` USE_FBGEMM_GENAI=1 python -m pip install --no-build-isolation -v -e . ... Successfully installed torch-2.9.0a0+gitf5070f3 ``` [full build log](https://www.internalfb.com/phabricator/paste/view/P1933787581) #### Unit tests ``` pytest test/test_matmul_cuda.py -k test_mxfp8_scaled_grouped_mm_ ... test/test_matmul_cuda.py ......... [100%] ============================================================== 9 passed, 1668 deselected in 5.34s =============================================================== ``` Pull Request resolved: pytorch#162209 Approved by: https://github.com/ngimel
dsashidh
pushed a commit
to dsashidh/pytorch
that referenced
this pull request
Sep 26, 2025
…ump (pytorch#162209) ## Summary - We just landed 2d-2d support for mxfp8 grouped gemm in FBGEMM: pytorch/FBGEMM#4816 - This is needed for backward pass of mxfp8 MoE training with grouped gemms - Changes: - Add dispatching + input validation for mxfp8 grouped gemm in `torch._scaled_grouped_mm` - Add meta registration input validation for mxfp8 grouped gemm, for composability with compile - Add unit tests exercising torch._scaled_grouped_mm with mxfp8 inputs - Bump FBGEMM third party submodule to include: - pytorch/FBGEMM#4816 - pytorch/FBGEMM#4820 - pytorch/FBGEMM#4821 - pytorch/FBGEMM#4823 #### How fbgemm dependency was bumped Documenting this since I haven't found it documented elsewhere: - `cd ~/pytorch/third_party/fbgemm` - `git fetch` - `git checkout <hash>` - `cd ~/pytorch` - `git add third_party/fbgemm` ## Test plan #### Test build ``` USE_FBGEMM_GENAI=1 python -m pip install --no-build-isolation -v -e . ... Successfully installed torch-2.9.0a0+gitf5070f3 ``` [full build log](https://www.internalfb.com/phabricator/paste/view/P1933787581) #### Unit tests ``` pytest test/test_matmul_cuda.py -k test_mxfp8_scaled_grouped_mm_ ... test/test_matmul_cuda.py ......... [100%] ============================================================== 9 passed, 1668 deselected in 5.34s =============================================================== ``` Pull Request resolved: pytorch#162209 Approved by: https://github.com/ngimel
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
MXFP8 grouped GEMM updates to (1) handle 2d-2d case, and (2) have a PyTorch compliant API
Reviewed By: ngimel, cthi
Differential Revision: D81362680