wandb logger problem with on_step log on validation

[andreaRapuzzi]

🐛 Bug

When logging on the validation_step with on_step=True and on_epoch=False the following happens:

• wandb warnings are generated to alert about a step numbering problem (probably confusing the validation step number which seems cyclical with the overall step which is always increasing)

• wandb charts for training (by step) is shrunk on the x dimension (like the number of steps for the whole training were less). We tested 2 training runs: the first (blue in the image below) with on_step=False and on_epoch=True on validation_step, the second with on_step=True and on_epoch=False (red in the image below). As you can see the training chart is affected by this:

• an error is issued at the end of the second training run:

• two new (unrequested) panels appear at the top to the wandb project (this is the weirdest of the lot :-))

Please reproduce using the colab link at the top of this article

To Reproduce

Just change the validation_step logging like this:

def validation_step(self, batch, batch_idx):
    x, y = batch
    logits = self(x)
    loss = F.nll_loss(logits, y)

    # validation metrics
    preds = torch.argmax(logits, dim=1)
    acc = accuracy(preds, y)
    self.log('val_loss', loss, on_step=True, on_epoch=False, prog_bar=True)
    self.log('val_acc', acc, on_step=True, on_epoch=False, prog_bar=True)
    return loss

[github-actions]

Hi! thanks for your contribution!, great first issue!

[awaelchli]

This is not surprising. What exactly are you trying to log in each step of validation?
It is not meaningful to log per batch in validation as there is no real progression and the global step does not increase.
If you need a validation accuracy, certainly you should average it over the whole validation set which is what self.log does by default when you set epoch=True.
Hope this answer helps.

[andreaRapuzzi]

Hi @awaelchli, thanks for the prompt answer!
While I agree that the per epoch validation info has the most value, still there's value in logging validation step info to monitor how 'noisy' (variance) is a metric inside a single epoch.
So my initial validation logging had on_step=True and on_epoch=True which had similar problems. To simplify the problem reporting I described a simpler config (on_step=True, on_epoch=False).
Logging validation just on the epoch level is a sensible and useful default, but it should not be the only way (or otherwise the library interface should constraint this behaviour).
Thanks a lot, again!!

[borisdayma]

My suggestion in this case would be to gather your validation metrics in an array and log it as an histogram at the end.

[andreaRapuzzi]

Thanks @borisdayma !!