[dreambooth] fix applying clip_grad_norm_ #686
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The documentation is not available anymore as the PR was closed or merged. |
|
so this has been fixed? I'm confused lol. i didn't when it was set on "no" didnt have enough cuda memory" when i changed it to fp16 i had a different error.. but i sadly don't remember atm. Also, i would love to be able to run this on something other than Colab. Like Vast.ai or RunPod. Since Google Doesn't provide Colab Pro World Wide. i tried to use it on Vast i was hit with a error of it not be able to install files |
prathikr
pushed a commit
to prathikr/diffusers
that referenced
this pull request
Oct 26, 2022
fix applying clip grad norm
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix gradient clipping when doing gradient accumulation. The gradients needs to clipped after they are synchronized when doing accumulation. cf huggingface/accelerate#641 (comment)
fixes #668
Although not sure why it only got triggered only in half-precision.