-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Description
Proposed refactor
Remove these two calls to profile model_forward:
https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/optimization/optimizer_loop.py#L417
https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/optimization/manual_loop.py#L110
Motivation
We profile training_step a few lines after we profile model_forward:
https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/optimization/optimizer_loop.py#L417-L424
There is a lot of overlap between model_forward and training_step, with training_step taking up the majority of model_forward's time.
see lines 11-12 in this example profiler output: https://gist.github.com/daniellepintz/7b37ab4807fab345248bf60c72e46ae8
Additional Context
Discussed in #12015