-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Closed
Labels
bugSomething isn't workingSomething isn't workingcallbackhelp wantedOpen to be worked onOpen to be worked onpriority: 1Medium priority taskMedium priority task
Milestone
Description
🐛 Bug
When training is resumed with resume_from_checkpoint, a warning is given about missing callbacks even if the expected callbacks are included.
I got the following:
UserWarning: Be aware that when using ``resume_from_checkpoint``,
callbacks used to create the checkpoint need to be provided.
Please, add the following callbacks: ['BetterModelCheckpoint', 'FreezeUnfreeze']
when the relevant variables had these values:
current_callbacks_type: {<class 'src.callbacks.model_checkpoint.BetterModelCheckpoint'>,
<class 'src.callbacks.freeze_unfreeze.FreezeUnfreeze'>, ...}
saved_callbacks_type: {'BetterModelCheckpoint', 'FreezeUnfreeze'}
From a quick look, it seems that the source of the bug is that in current_callbacks_type there are types, and in saved_callbacks_type there are strings from Callback.state_id (which is .__qualname__ of the type).
Sorry I didn't follow the template for reporting bugs. If this isn't enough, I'll be able to update the issue in a few days.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingcallbackhelp wantedOpen to be worked onOpen to be worked onpriority: 1Medium priority taskMedium priority task