val_dataloader is called twice in each worker

[undertherain]

🐛 Bug

I'm trying a LightningDataModule class to manage the data.
Using horovod backend, if that matters.
I've noticed that each rank is calling train_dataloader once, but val_dataloader two times somehow.

To Reproduce

run LIghtning with Dataclass and horovod, add some debug print on when val_dataloader is called

soemthing like

    def train_dataloader(self):
        print(f"\n#####worker {hvd.rank()} of {hvd.size()} creating train_loader\n")
        return load_ds_from_dir(os.path.join(self.path, "train"), self.batch_size)

    def val_dataloader(self):
        print(f"\n#####worker {hvd.rank()} of {hvd.size()} creating val\n")
        return load_ds_from_dir(os.path.join(self.path, "validation"), self.batch_size)

Expected behavior

expect val loader to be called only once...

Environment

* CUDA:
	- GPU:
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
	- available:         True
	- version:           10.2
* Packages:
	- numpy:             1.19.1
	- pyTorch_debug:     False
	- pyTorch_version:   1.6.0
	- pytorch-lightning: 0.9.0
	- tensorboard:       2.2.0
	- tqdm:              4.46.1
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- ELF
	- processor:         x86_64
	- python:            3.8.2
	- version:           #1 SMP Fri Apr 20 16:44:24 UTC 2018

[undertherain]

I see... it is doing run_sanity_check before training and then resets a val dataloader
that's surprising..... why not be more explicit about this functionality in the tutorial or better make it false by default?

[undertherain]

okey, num_sanity_val_steps=0, solved it for me
but I'll keep this open so that devs have a chance to see and maybe consider making it zero by default, or at least leave a word of warning in the docs. I mean, I was not using any sort of lazy loading data, but reading terabytes of tensors to memory on extra time of val loader create...

[awaelchli]

This is not a bug, it is intentionally on by default. The sanity check is there to report exceptions early instead of only after a long training epoch. We can add a note to the docs that the dataloader gets reloaded.
I'm removing the bug label, as this is expected behavior as far as I know.

[undertherain]

Thanks for considering to update the docs!

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!