This repository was archived by the owner on Nov 1, 2024. It is now read-only.

Description
This is just an issue to track that we do not have unit tests added yet for the BF16 optimizer.
Main tests would be:
a - verify AdamW and BF16 set to FP32 dtypes produce matching results. (BF16 degrades to AdamW w/o Kahan and dtypes to FP32).
b - simple checks for confirming internal dtypes match expected (i.e. set variance to BF16 but momentum to FP32, verify both are in proper dtype).