argument for detect_anomaly on for debug

[ruotianluo]

🚀 Feature

Add an arg for trainer to turn on detect_anomaly. So that when gradients get nan, the model will automatically detect.

Motivation

For easier debugging.

[ananthsub]

@ruotianluo - there is terminate_on_nan on the Trainer constructor: https://github.com/PyTorchLightning/pytorch-lightning/blob/4928dc55790eec4ee4a558f494a6fc3d49a5e78a/pytorch_lightning/trainer/trainer.py#L304-L305

[ruotianluo]

It is actually different. detect anomaly tells you where you got the nan during back propagation instead of just terminate.

[ananthsub]

ah do you mean support for PyTorch's torch.autograd.detect_anomaly detection here? https://pytorch.org/docs/stable/_modules/torch/autograd/anomaly_mode.html#detect_anomaly

[ruotianluo]

yes

[ananthsub]

@awaelchli @carmocca what do you think? we could enable torch.autograd.set_detect_anomaly(trainer.terminate_on_nan) inside of the training loop.

this could repurpose the existing trainer argument

[awaelchli]

Yes, nice Idea. @ruotianluo would you like to give this a try?

@ananthsub could it also be applied as a ctx manager over the fit_loop.run() call?
We probably also want to make sure it gets deactivated once the fit ends.

[ananthsub]

@awaelchli - one concern is the slowdown that will occur when using terminate_on_nan with vs. without anomaly detection. my assumption is that people use terminate_on_nan for debugging, but i'm not sure if that's true in all cases.

[awaelchli]

Yes, I would recommend strictly for debugging. I thought it was already there but we should add it to our debugging section in the docs.

Perhaps terminate_on_nan should be replaced by anomaly detection entirely and the trainer argument renamed? According to the pytorch docs, their anomaly detection already terminates with an error. The difference is that PL stops also when the loss is NaN.

[carmocca]

Agree that this would be a great addition.

it was already there but we should add it to our debugging section in the docs.

Also mention it in the Trainer's description of the argument

(from the docs)

Any backward computation that generate “nan” value will raise an error.

Does detect_anomaly also check for +-inf?

[ananthsub]

fyi @daniellepintz - this would be another good issue & feature as we're looking at error handling & debugging capabilities

[daniellepintz]

sg! could someone to assign to me? unless @ruotianluo wants to take it on