Validation not called when using an IterableDataset and `limit_train_batches` flag

[DeWittmm]

When using an IterableDataset the limit_train_batches trainer flag will prevent the validation loop from being called. Here is a simplified notebook to reproduce the issue. Note this error may manifest as:

ValueError: cannot convert float NaN to integer

if only limit_train_batches is set, but can also interfere with model checkpointing if the limit_val_batches flag is set:

MisconfigurationException: ModelCheckpoint(monitor='val_log') not found in the returned metrics: ['loss']. HINT: Did you call self.log('val_log', tensor) in the LightningModule?

Possible Workaround:
Interestingly if the IterableDataset has __len__ defined than neither flag is an issue.

Environment Info:

• PyTorch Version (e.g., 1.0): 1.7.1+cu101
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.8.2
• Any other relevant information: 32 cores, 128GB mem
• Pytorch Lightning Version: 1.1.5

[melaanya]

I have the exact same issue with pytorch-ligtning==1.2.4 , but using a Dataset class (with __len__ defined)

[hudeven]

limit_train_batches is very useful for integration tests or quick experiments. Currently, I have to workaround it by setting monitor to metric logged in training_step.
@ananthsub would this issue be fixed by #6672 ? cc: @shuyingsunshine21

[ananthsub]

@DeWittmm - I'm unable to reproduce the failure case from your notebook with this test on the current master branch: https://gist.github.com/ananthsub/c795351be6e44af1ddcd251cc256c240

[shuyingsunshine21]

limit_train_batches is very useful for integration tests or quick experiments. Currently, I have to workaround it by setting monitor to metric logged in training_step.
@ananthsub would this issue be fixed by #6672 ? cc: @shuyingsunshine21

@hudeven , had a draft PR: #6671 , let me rebase and refine a bit, will publish