Loading a model from PL 1.2 that was saved in PL 1.1 breaks

[nicola-decao]

🐛 Bug

I saved a model trained with PL 1.1 from an environment with PL 1.2 and it breaks. There are some PL specific objects that get pickled into the checkpoint. This shouldn't happen. See error below:

Traceback (most recent call last):
  File "scripts/train_bart_seq2seq_augmented_kilt.py", line 45, in <module>
    model = BartSeq2SeqAugmented(**vars(args))
  File "/home/ndecao/modify-transformers-memory/src/models/bart_seq2seq_augmented_kilt.py", line 67, in __init__
    self.model = BartSeq2Seq.load_from_checkpoint(self.hparams.model_checkpoint)
  File "/home/ndecao/.anaconda3/envs/kilt37/lib/python3.7/site-packages/pytorch_lightning/core/saving.py", line 134, in load_from_checkpoint
    checkpoint = pl_load(checkpoint_path, map_location=lambda storage, loc: storage)
  File "/home/ndecao/.anaconda3/envs/kilt37/lib/python3.7/site-packages/pytorch_lightning/utilities/cloud_io.py", line 32, in load
    return torch.load(f, map_location=map_location)
  File "/home/ndecao/.anaconda3/envs/kilt37/lib/python3.7/site-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/ndecao/.anaconda3/envs/kilt37/lib/python3.7/site-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
AttributeError: Can't get attribute '_gpus_arg_default' on <module 'pytorch_lightning.utilities.argparse_utils'

Expected behavior

The model should load without any error.

Environment

The model was trained and saved using PL 1.1.6 and loaded from PL 1.2.1

• PyTorch Version (e.g., 1.0): 1.7.1
• OS (e.g., Linux): Linux
• Python version: 3.9

[Borda]

mind share you model details, as we have legacy tests in place and testing against many past versions...
https://github.com/PyTorchLightning/pytorch-lightning/blob/a312219d422709cc38dea0f5604c531f31b7ab28/tests/checkpointing/test_legacy_checkpoints.py#L31-L58

[nicola-decao]

@Borda I'll try to reproduce using the BoringModel

[boyuzz]

Meet the same issue. Failed to load model in PL 1.2.4 where the model is trained using PL 1.1.6.

[Borda]

Meet the same issue. Failed to load model in PL 1.2.4 where the model is trained using PL 1.1.6.

@nicola-decao @boyuzz could you please share some code/example to reproduce?

[Borda]

@nicola-decao @boyuzz how is it going, any update on the example?

[boyuzz]

@nicola-decao @boyuzz how is it going, any update on the example?

Well, finally I retrained the model in PL 1.2.4. Currently I'm unable to provide the code/example to reproduce. Sorry.

[nicola-decao]

@Borda @boyuzz same here. I am just fixing the version to 1.1.6 for my project and I won't update it. I cannot spend time debugging this right now as my code works if I do not update. Sorry.

[nicola-decao]

@Borda btw this is my hparams.yaml. I think the problem might be in the tpu_cores?

accelerator: ddp
accumulate_grad_batches: 1
amp_backend: native
amp_level: O2
auto_lr_find: false
auto_scale_batch_size: false
auto_select_gpus: false
automatic_optimization: null
batch_size: 256
benchmark: false
check_val_every_n_epoch: 1
checkpoint_callback: true
default_root_dir: null
deterministic: false
dev_data_path: /datastore/shared/kilt/datasets/fever-dev-kilt.jsonl
dirpath: models/bert_binary_augmented_fever
distributed_backend: null
divergences: kl
enable_pl_optimizer: null
fast_dev_run: false
flush_logs_every_n_steps: 100
gpus: 4
gradient_clip_val: 0
limit_test_batches: 1.0
limit_train_batches: 1.0
limit_val_batches: 1.0
log_every_n_steps: 50
log_gpu_memory: null
logger: true
lr: 0.0003
lr_alpha: 0.1
margin_kl_max: 0.1
margin_kl_min: 0.001
margin_l2_max: 1.0e-06
margin_l2_min: 1.0e-09
max_epochs: 1000
max_length: 32
max_scale: 1
max_steps: 50000
min_epochs: 1
min_steps: null
model_checkpoint: models/bert_binary_fever/version_0/checkpoints/model-epoch=02-valid_acc=0.7708.ckpt
model_name: bert-base-uncased
move_metrics_to_cpu: false
num_nodes: 1
num_processes: 1
num_sanity_val_steps: 2
num_workers: 32
overfit_batches: 0.0
plugins: null
precision: 32
prepare_data_per_node: true
process_position: 0
profiler: null
progress_bar_refresh_rate: 1
reload_dataloaders_every_epoch: false
replace_sampler_ddp: true
resume_from_checkpoint: null
save_top_k: 10
seed: 0
sync_batchnorm: false
terminate_on_nan: false
total_num_updates: 200000
tpu_cores: !!python/name:pytorch_lightning.utilities.argparse_utils._gpus_arg_default ''
track_grad_norm: -1
train_data_path: /datastore/shared/kilt/datasets/fever-train-kilt.jsonl
truncated_bptt_steps: null
use_views: true
val_check_interval: 1.0
warmup_updates: 1000
weights_save_path: null
weights_summary: top

[nicola-decao]

@Borda Look at this. in Hyperparameters I have a function where there should be a primitive