Skip to content

Change accelerator in backward to use DDP-wrapped model #4415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
+1 −1
Closed

Change accelerator in backward to use DDP-wrapped model #4415

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pytorch_lightning/accelerators/accelerator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ def backward(self, closure_loss, optimizer, opt_idx, *args, **kwargs):
)
else:
# do backward pass
model = self.trainer.get_model()
model = self.trainer.model
Copy link
Contributor

@williamFalcon williamFalcon Oct 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ummm... this will break the other accelerators no?
DP will be wrapped, and so will the ddp one?

Copy link
Contributor Author

@ananthsub ananthsub Oct 28, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The motivating factor is skipping parameter syncs in DDP (#4301). While accumulating gradients,

  • training_step_and_backward calls training_loop backward
  • training_loop backward calls accelerator backward here
  • accelerator backward reaches into the DP/DDP model and extracts the module inside and calls backward on that
  • which ignores the flags here:

How should we respect those settings in the backwards pass here?

model.backward(closure_loss, optimizer, opt_idx, *args, **kwargs)

# once backward has been applied, release graph
Expand Down