Can't get attribute '_gpus_arg_default'

[sun-peach]

🐛 Bug

when I use the code
model = model.load_from_checkpoint(ckpt_path)
It pops out a problem: Can't get attribute '_gpus_arg_default' on <module 'pytorch_lightning.utilities.argparse' from '/opt/conda/envs/lasaft/lib/python3.8/site-packages/pytorch_lightning/utilities/argparse.py'>

To Reproduce

Very hard to describe the reproduction, because I could not load model.

Expected behavior

Succeed in loading the model

Environment

• PyTorch Version (e.g., 1.0):1.7.1
• OS (e.g., Linux): Ubuntu 18.04
• How you installed PyTorch (conda, pip, source): conda
• Build command you used (if compiling from source):
• Python version: 3.8.5
• CUDA/cuDNN version: 10.2
• GPU models and configuration: V100
• Any other relevant information: PL version is 1.3.0rc1

[quancs]

I met the same problem when I upgrade from 1.2.2 to 1.2.8 and from 1.2.6 to 1.2.8

[quancs]

And when I debugged it, I found that the “pytorch_lightning.utilities.argparse” in 1.2.8 does not have "_gpus_arg_default", but “pytorch_lightning.utilities.argparse” in 1.2.6 does have "_gpus_arg_default".

[SeanNaren]

I haven't been able to reproduce this issue.

Taking the boring model and running this:

import torch
from torch.utils.data import DataLoader, Dataset

from pytorch_lightning import Trainer, LightningModule


class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class BoringModel(LightningModule):

    def __init__(self):
        """
        Testing PL Module

        Use as follows:
        - subclass
        - modify the behavior for what you want

        class TestModel(BaseTestModel):
            def training_step(...):
                # do your own thing

        or:

        model = BaseTestModel()
        model.training_epoch_end = None

        """
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def loss(self, batch, prediction):
        # An arbitrary loss to have a loss that updates the model weights during `Trainer.fit` calls
        return torch.nn.functional.mse_loss(prediction, torch.ones_like(prediction))

    def step(self, x):
        x = self(x)
        out = torch.nn.functional.mse_loss(x, torch.ones_like(x))
        return out

    def training_step(self, batch, batch_idx):
        output = self(batch)
        loss = self.loss(batch, output)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        output = self(batch)
        loss = self.loss(batch, output)
        return {"x": loss}

    def test_step(self, batch, batch_idx):
        output = self(batch)
        loss = self.loss(batch, output)
        return {"y": loss}

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(self.layer.parameters(), lr=0.1)
        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)
        return [optimizer], [lr_scheduler]

    def train_dataloader(self):
        return DataLoader(RandomDataset(32, 64))

    def val_dataloader(self):
        return DataLoader(RandomDataset(32, 64))

    def test_dataloader(self):
        return DataLoader(RandomDataset(32, 64))


if __name__ == '__main__':
    model = BoringModel()

    trainer = Trainer(
        fast_dev_run=True,
        gpus=-1,
    )
    trainer.fit(model)
    trainer.save_checkpoint('model.pt')

    BoringModel.load_from_checkpoint('model.pt')

Works on 1.2.8. I then tried saving a checkpoint using 1.2.6, and then loading it in 1.2.8 as such:

BoringModel.load_from_checkpoint('model.pt')

And this worked fine. Could you give more details as to how this bug appears so I can reproduce @quancs @sun-peach?

[awaelchli]

I think the problem is that the args were passed into the model and saved as hyper parameters in 1.2.6 but in 1.2.8 we removed the _gpus_arg_default function, so pickle can't find the source code for that :(
Change was made here: #6898

You can probably fix it by doing right before you load your checkpoint.
Workaround:

from pytorch_lightning.utilities import argparse 
setattr(argparse, "_gpus_arg_default", lambda x: 0)

my bad. I didn't think of this. shoot.
we should re-introduce this function and keep it there so old checkpoints who have that flaw can still be loaded.

[awaelchli]

This was reproducible for me when I passed in the argparse args into the model and they get saved.
I verified that this is fixed in my PR #7043, checkpoint loads without problem.

import argparse

import torch
from torch.utils.data import DataLoader, Dataset

from pytorch_lightning import Trainer, LightningModule


class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class BoringModel(LightningModule):

    def __init__(self, **kwargs):
        super().__init__()
        self.save_hyperparameters()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def loss(self, batch, prediction):
        # An arbitrary loss to have a loss that updates the model weights during `Trainer.fit` calls
        return torch.nn.functional.mse_loss(prediction, torch.ones_like(prediction))

    def step(self, x):
        x = self(x)
        out = torch.nn.functional.mse_loss(x, torch.ones_like(x))
        return out

    def training_step(self, batch, batch_idx):
        output = self(batch)
        loss = self.loss(batch, output)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        output = self(batch)
        loss = self.loss(batch, output)
        return {"x": loss}

    def test_step(self, batch, batch_idx):
        output = self(batch)
        loss = self.loss(batch, output)
        return {"y": loss}

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(self.layer.parameters(), lr=0.1)
        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)
        return [optimizer], [lr_scheduler]

    def train_dataloader(self):
        return DataLoader(RandomDataset(32, 64))

    def val_dataloader(self):
        return DataLoader(RandomDataset(32, 64))

    def test_dataloader(self):
        return DataLoader(RandomDataset(32, 64))


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser = Trainer.add_argparse_args(parser)
    args = parser.parse_args()
    model = BoringModel(**vars(args))
    trainer = Trainer.from_argparse_args(args)
    trainer.fit(model)
    
    # comment the line below on < 1.2.8
    trainer.save_checkpoint('model.pt')
    
    # comment the line below on 1.2.8
    BoringModel.load_from_checkpoint('model.pt')