After trainer resume_from_checkpoint checkpoint_callback doesn't restore best metric value

[Vozf]

After I restore training with

    pl_model = LightningModel.load_from_checkpoint(str(ckpt_path))

    trainer = Trainer(
        resume_from_checkpoint=str(ckpt_path.name),
        logger=instantiate(cfg.logger, experiment_key=cfg.experiment_id),
        checkpoint_callback=instantiate(cfg.checkpoint.model_checkpoint),
        callbacks=[instantiate(callback) for callback in cfg.callbacks],
    )
    trainer.fit(pl_model)

instantiate from facebook/hydra
The training process doesnt restore the best metric value and starts from scratch so I'm getting on the first restored epoch

INFO:lightning:Epoch 1129: val_mae reached 0.10418 (best 0.10418), saving model to

When there was a better checkpoint before interruption

[tchaton]

Hey @Vozf,

Could you mind reproducing the bug with our https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report_model.py.

But my guess is a misuse of Hydra instantiate :) Will re-created ModelCheckpoint without reloading previously trained state instantiate(cfg.checkpoint.model_checkpoint)

Best, T.C

[Vozf]

ModelCheckpoint should be reloaded manually? Currently I create it with instantiate and expect trainer to restore its state? Isn't it an expected way of state restoring for ModelCheckpoint?

[awaelchli]

Yes, I believe this is related to the bug #4014 and #4336 will fix that.

[awaelchli]

@Vozf this should be fixed now. make sure to pass your modelcheckpoint instance to the callbacks, i.e.:

trainer = Trainer(
        resume_from_checkpoint=str(ckpt_path.name),
        logger=instantiate(cfg.logger, experiment_key=cfg.experiment_id),
        # don't pass in an object here
        # checkpoint_callback=instantiate(cfg.checkpoint.model_checkpoint),
        # user-defined checkpoint callback must go into this list:
        callbacks=[instantiate(callback) for callback in cfg.callbacks] + [instantiate(cfg.checkpoint.model_checkpoint)],
    )

It's probably best if you change your configuration so that the model checkpoint is directly in cfg.callbacks.

[Vozf]

Thanks for advice. But what is the difference when passsing modelCheckpoint as callback vs chekpoint_callback?

[awaelchli]

there isn't supposed to be a difference, however, the latter is now deprecated and will be unsupported.
all callbacks must be passed into the callbacks list going forward.

EDIT: I meant to say the latter

[LeCongThuong]

@Vozf Can you solve your problem?
I have the same problem as you despite passing modelCheckpoint to callbacks list.
I am using pl version 1.2.6

[Vozf]

As I remember pl update solved it for me, but as you're using the latest version I don't really know. You could try opening a new issue with the latest version. I'm using 1.2.0 now and it's working fine