Trainer.validate(model, dataloader=dl) does not use passed dataloader

[jwallat]

🐛 Bug

I wanted to validate my model on a subset of the original validation set. I found the Trainer.validate() function which takes one or multiple dataloaders as one of its arguments. However, the model will use the model's default dataloader and not the one I passed to .validate().

Please reproduce using the BoringModel

https://colab.research.google.com/drive/1KMgKeugv9VqDgT4Mj8S-GObc8-3vS_H_?usp=sharing

To Reproduce

Run the colab example. You will see that despite passing another dataloader to trainer.validate(), it still uses the initial dataloader.

Expected behavior

If no new dataloader is passed to validate(), then it should use the existing dataloader of the model.
If a new dataloader is passed to validate(), then the passed dataloader should be used.

Environment

• PyTorch Lightning Version (e.g., 1.3.8):
• PyTorch Version (e.g., 1.8)
• Python version:
• OS (e.g., Linux):
• CUDA/cuDNN version:
• GPU models and configuration:
• How you installed PyTorch (conda, pip, source):
• If compiling from source, the output of torch.__config__.show():
• Any other relevant information:

Additional context

[awaelchli]

Related: #8435

[awaelchli]

I believe this is user error. Let me know if I'm wrong.

@jwallat you are passing a dataloader into another dataloader:

    other_val = RandomDataset(size=1, num_samples=20)
    other_val = DataLoader(val, batch_size=1)

where val is defined a few lines above as:

val = DataLoader(val, batch_size=1)

Could you confirm whether this is your intention or not?

[tchaton]

Dear @jwallat,

Any updates ? Otherwise, we will close the issue as it seems to be a user error related issue.

Best,
T.C

[awaelchli]

@jwallat Closing for now.
Feel free to come back anytime and let us know if it is still not working.
I added a test in #8442 to make sure the val_dataloader for validate gets used.