Missing cleanup after trainer.fit() and trainer.test()

[ananthsub]

🐛 Bug

The Lightning trainer holds references to the LightningModule/DataModule after fit/test complete. This can leads to different behavior in calls likeL

Please reproduce using the BoringModel and post here

def test_x(tmpdir):
    # validation checks do not regularly since we don'tre-instantiate the trainer inside each loop
    for i in range(2):
      trainer = pl.Trainer(max_epochs=4, check_val_every_n_epoch=2, logger=False, checkpoint_callback=False)
      test_module = BoringModel()
      trainer.fit(test_module)


reuse_trainer = pl.Trainer(max_epochs=4, check_val_every_n_epoch=2, logger=False, checkpoint_callback=False)
def test_reuse(tmpdir):
    # validation checks do not run on the second loop since we don't re-instantiate the trainer inside each loop
    for i in range(2):
      test_module = BoringModel()
      reuse_trainer.fit(test_module)

To Reproduce

Expected behavior

The latter test_reuse should re-bind the model + hooks when calling fit again inside each loop

[jabertuhin]

Recently, I have faced this kind of issue.
For a kaggle competition, with same configuration(same trainer, datamodule and model) but with different variable names I was training my model in two separate notebook cells. But for two runs train and validation losses were different.
I tried to replicate these runs result by restarting the kernel and running all the cells.
Two versions of the notebook resulted same result, so it might be because of the issue mentioned here.

So, I would like to work on this issue.

[edenlightning]

@jabertuhin would be great if you can contribute! Let us know if you need any help.

[jabertuhin]

@edenlightning
First, I thought at first the issue was with trainer. I went through the trainer code, and it seemed to trainer has no issue.
Then I was suspecting LightningDataModule(for my case), now I think it all goes down to Dataloader.

I set random seed in my notebook with this:

from pytorch_lightning import seed_everything
seed_everything(42)

Here is an example, I am creating two new datamodule objects, and getting different images for first batch.

And then I restarted the kernel and ran them again:

Is there any other better way to debug this issue? Or is it even an issue or expected behavior ?