Save checkpoint and validate every n steps

[ruotianluo]

❓ Questions and Help

How to save checkpoint and validate every n steps.

I saw there is a val_check_interval, but it seems it's not for that purpose.

[nanaHa1003]

Set check_val_every_n_epoch=n in Trainer may do the trick.

[ruotianluo]

Won’t that do validation every n epochs?

[nanaHa1003]

Oh, you are right, I misunderstand your question.

[rohitgr7]

As per my understanding, you want to validate the model after every n steps in the same epoch. If I am correct then val_check_interval does the same:
https://github.com/PyTorchLightning/pytorch-lightning/blob/7ef73f242ad4e5f14e6c967c8639c3a65285a048/pytorch_lightning/trainer/evaluation_loop.py#L52-L68

[ruotianluo]

Not really.
First val_check_interval can't be bigger than the number of training batch.
Second, it is the number of steps in one epoch. that is, do when batch_idx % val_check_interval == 0 instead of global_step % val_check_interval == 0.

[ruotianluo]

My current workaround is:
set check_val_every_n_epoch to be float('inf').

And add such callback:

class ValEveryNSteps(pl.Callback):
    def __init__(self, every_n_step):
        self.every_n_step = every_n_step

    def on_batch_end(self, trainer, pl_module):
        if trainer.global_step % self.every_n_step == 0 and trainer.global_step != 0:
            trainer.run_evaluation(test_mode=False)

[rohitgr7]

Ok, your n is global_steps.

[awaelchli]

Let's make this issue into a feature request, allowing val_check_interval > len(train_dataloader)?

[andrewjong]

This is an important feature. Especially for large datasets where an epoch may take a whole day, we might want to save a checkpoint in between epochs in case something goes wrong. We need a way to checkpoint based on steps, or in between epochs.

[andrewjong]

Saving a checkpoint every N steps should really not be tied to validation.

For some models it doesn't make sense to monitor a decreasing validation loss. For example: vanilla GANs expect a constantly shifting loss value between generator and discriminator. We need independent N-steps checkpointing.