"Advanced" profiler not working

[bnaman50]

🐛 Bug

Hello Guys,

I am having issues with the advanced profiling option in lightning. Here is the colab file documenting the issue on a simple model.

The issue happens, whenever I set stochastic_weight_avg=True.

Also, I have my complicated setup where I faced issue regarding Advanced profiler but it was a different one. When I tried to reproduce it, I faced the above mentioned issue. Thus, I am just mentioning my actual error over here.

Here is how I call my trainer -

trainer = pl.Trainer.from_argparse_args(args,
                                            callbacks=[LoggingCallback(), checkpoint_callback],
                                            profiler="advanced",
                                            )

And here is the error stack -

Traceback (most recent call last):
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 71, in profile
    yield action_name
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1102, in call_hook
    trainer_hook(*args, **kwargs)
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/trainer/callback_hook.py", line 35, in on_before_accelerator_backend_setup
    callback.on_before_accelerator_backend_setup(self, model)
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/callbacks/swa.py", line 142, in on_before_accelerator_backend_setup
    self._average_model = deepcopy(pl_module)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/nzb0040/.pyenv/versions/3.8.6/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'Profile' object
Exception ignored in: <function AdvancedProfiler.__del__ at 0x7f045ad834c0>
Traceback (most recent call last):
  File "/home/nzb0040/.pyenv/versions/intersection_train/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 280, in __del__
    if self.output_file:
AttributeError: 'AdvancedProfiler' object has no attribute 'output_file'

[justusschock]

Hey @bnaman50 ,

Thanks for the issue. You're right, there seem to be some pickling issues with this profiler.
#6552 fixes this for me. Can you try that one? :)

[bnaman50]

Hey, I am still facing the same issue even with the new version (1.2.4).

P.S. - Reference code is there in the same colab.

Thanks,
Naman

[justusschock]

@bnaman50 ,

Yes the 1.2.4 release does not include that fix, since the PR is not yet merged. You could try to install from that specific git branch. Hopefully, the PR will be merged by the next release.