Model summarize displayed twice before training starts

[ananthsub]

🐛 Bug

Model summarize is called in two spots, which results in duplication in the logs:

• It's called once in the training loop: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/training_loop.py#L160

And I'm unsure how it's called again

The first time is with the logger.info(...) since it's formatted. The second time it appears to be logged with print(...) since the table formatting doesn't handle newlines

To Reproduce

Run any training with weights_summary set on the trainer

Expected behavior

We only call summarize once

[rohitgr7]

Tried on master. Working fine. Is it happening with ddp backend only?

[ananthsub]

Yes this happens with DDP backend

[awaelchli]

on my windows machine, single gpu, happens too. but the logs also appear twice, including all warnings.

TPU available: False, using: 0 TPU cores
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0]
CUDA_VISIBLE_DEVICES: [0]

¨
which means it is logging related, not model summary.
I will investigate

[JVGD]

Just started migration of our codebase to Lightning and the first trainer I write have this issue... Could this be something related with duplicated handlers in logger? It might be that the logger.getLogger() which is a singleton is called twice, so it might be that 2 handlers are initialized, the root handler and another stream handler

This is my case:

[awaelchli]

yes, I think that's the case. does my PR not resolve the problem for you? which os are you on?

[JVGD]

I am using Ubuntu 18.04.

I tried you PR but unfortunately it does not solve the issue. Maybe I am doing something wrong?

I think this is what might be happening: https://jdhao.github.io/2020/06/20/python_duplicate_logging_messages/

[awaelchli]

@JVGD could you try again with my updated branch. sorry, I don't have much experience with this python logging, for me the duplicate messages are resolved locally. I need you as the tester 😄
I tried my windows and ubunut multi-gpu

[JVGD]

Unfortunately is still showing duplicated logs. I am going to do further testing, but as far as I know I think the pytroch_lightning logger adds an extra StreamHandler to the base root logger. Maybe that is why there are duplicates

[awaelchli]

indeed, if we remove the streamhandler, this also solves the problem (for me). can you confirm?
https://github.com/PyTorchLightning/pytorch-lightning/pull/3514/files

[JVGD]

Amazing! it was just deleting that line and now it works like a charm. By the way, I did it in the pytorch-lightning==0.9.0 version, I was in my development environment. So I haven't been able to test it on your branch. But tomorrow I will!

Thank you for the support. Now it looks great:

I am not sure if that change alone suffices for avoiding duplicated logging in other platforms, but in my case it is indeed.