stopped calling datamodule.train_dataloader() when tuning the batch_size using pl.trainer.tune()

[randommm]

version 1.3.1 pytorch_lightining stopped calling datamodule.train_dataloader() when tuning the batch_size using pl.trainer.tune()

I don't know if this was reported already, but since version 1.3.1 pytorch_lightining stopped calling datamodule.train_dataloader() when tuning the batch_size using pl.trainer.tune(), so the batch_size effectivelly stays fixed at 2, see the attached files, in particular the print statements of

    def train_dataloader(self):
        print("THIS SHOULD BE CALLED!!!!!", self.batch_size)

and

    def training_step(self, train_batch, batch_idx):
        inputv, target = train_batch
        output = self.forward(inputv)
        print("actual batch_size", len(inputv))

when tune is called.

code_examples.zip

[Lucklyric]

Hi @randommm, I think the problem is caused by
train and val dataloaders only reset when they are none. We can change it to if self.trainer.train_dataloader is None or self.trainer.status == TrainerStatus.RUNNING:
https://github.com/PyTorchLightning/pytorch-lightning/blob/e126649d19e85c449b007008361f10374878f2f4/pytorch_lightning/trainer/training_loop.py#L184-L195

[randommm]

Thanks @Lucklyric ! Was it like that on versions prior to 1.3.0/1.3.1?

[Lucklyric]

Hi, @randommm. This section was recently updated. In fact, there are numerous changes in 1.3 regarding refactoring in the tuner and trainer. As a result, we'll have to wait for more feedback to validate this fix. However, I am certain that your problem is caused by not calling the reset function.

[Lucklyric]

@randommm , sorry this might not be the solution, I thought it is TunerStatus.RUNNING, but it is TrainerStatus.RNNING which means it is always True. However, I tried to check trainer.tunning == True, it fixes the issue but fails one test case. I am keeping investigating it.

[tchaton]

Hey @randommm, @Lucklyric,

I am not entirely sure, but I think this bug could be resolved by deleting dataloaders reference to the trainer as the end of the tuner tune call.

I think it could be done right there with a teardown function: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/tuner/tuning.py#L56, deleting reference on the trainer to the dataloader.

Does anyone of you wants to give it a try ?

Best,
T.C