Why all files are deleted from model checkpoint directory when save_top_k=1?

[celsofranssa]

checkpoint_callback = ModelCheckpoint(
        monitor="val_loss",
        dirpath=hparams.model_checkpoint.dir,
        filename=hparams.model.name+"_"+hparams.data.name+"_{epoch:02d}",
        save_top_k=1,
        mode="min"
    )
#UserWarning: Checkpoint directory With save_top_k=1, all files in this directory will be deleted when a checkpoint is saved!

Shouldn't it delete only the latest checkpoints of the same name?
Or should I have a directory for each model I train?

[rohitgr7]

It won't delete any existing file if it's not created by the ModelCheckpoint instance passed in the Trainer. This warning seems redundant. Mind send a PR to remove this warning in ModelCheckpoint??

Just need to confirm this once more cc: @carmocca

[carmocca]

Yes, this warning is not correct. We can keep it and just warn the user that the directory is not empty.

[heng-yuwen]

Hi, I noticed that if we resume the training, no file will be deleted if we have more than k checkpoints. For example, set save_top_k=3, then train for a few epochs, we get 3 saved checkpoints. Then we interrupt the code, and resume from last saved checkpoint. After training, we can have more than 3 saved checkpoints.

[rohitgr7]

yeah.... that's a bug. I also noticed it a day before.

[awaelchli]

so the fix for the warning is an easy one, who wants to send it? 😃

the second observation about resuming is simply because the model checkpoint does not save the top-k dict into it's state_dict when saving the checkpoint. When the state gets loaded, the dict is empty.
We can save it as part of the state, but then we have to deal with the special cases when the user overrides things, like the checkpoint location or the save_top_k (lower value or higher value), in order to keep it well-defined. So let's discuss it first?

[rohitgr7]

good points here by @awaelchli

like the checkpoint location or the save_top_k (lower value or higher value), in order to keep it well-defined.

I believe for dirpath if it's changed then clear that dict. To get the old dirpath, we can either save that too or just extract it from the filenames present in the dict. For save_top_k, it's pretty easy, if it's less, just keep the top k(new value) otherwise keep it as it is. Any other edge cases??

[carmocca]

We can save it as part of the state, but then we have to deal with the special cases when the user overrides things, like the checkpoint location or the save_top_k (lower value or higher value), in order to keep it well-defined. So let's discuss it first?

We have a similar discussion about this at #4911

[rohitgr7]

cool, let's discuss it there.