Skip to content

Conversation

@carmocca
Copy link
Contributor

@carmocca carmocca commented Sep 2, 2021

What does this PR do?

Fixes #9599, #9330, #9205, #9694

Before:

closure:
    zero_grad()
    forward()
    backward()
    <-- MISSING UNSCALE
    track_norm()
    clip_grad()
optimizer.step()
    closure()
    unscale()
    actual_step()

After:

closure:
    zero_grad()
    forward()
    backward()
optimizer.step()
    closure()
    unscale()
    actual_step()
track_norm()
clip_grad()

Zero will happen on the next step so the gradients will not have been zero-ed out

Does your PR introduce any breaking changes? If yes, please list them.

None

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • [n/a] Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

@carmocca carmocca added the bug Something isn't working label Sep 2, 2021
@carmocca carmocca added this to the v1.5 milestone Sep 2, 2021
@carmocca carmocca self-assigned this Sep 2, 2021
@carmocca carmocca changed the title [WIP] Fix gradient norm tracking and gradient clipping [WIP] Fix gradient norm tracking and gradient clipping [2/2] Sep 2, 2021
@carmocca carmocca added the priority: 0 High priority task label Sep 6, 2021
@carmocca carmocca changed the base branch from master to bugfix/plugin-closure-execution September 6, 2021 21:30
Base automatically changed from bugfix/plugin-closure-execution to master September 7, 2021 11:52
@carmocca carmocca modified the milestones: v1.5, v1.4.x Sep 7, 2021
Copy link
Contributor

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGMT !

@mergify mergify bot added the ready PRs ready to be merged label Oct 27, 2021
@carmocca carmocca enabled auto-merge (squash) October 27, 2021 14:10
@rohitgr7 rohitgr7 disabled auto-merge October 28, 2021 05:19
@rohitgr7 rohitgr7 enabled auto-merge (squash) October 28, 2021 05:19
@mergify mergify bot removed the has conflicts label Oct 28, 2021
@rohitgr7 rohitgr7 merged commit 03f01fb into master Oct 28, 2021
@rohitgr7 rohitgr7 deleted the bugfix/track-grad-norm branch October 28, 2021 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: 0 High priority task ready PRs ready to be merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clip norm after scaler.unscale_ in native fp16 training

5 participants