[Discussion] Callback interface/architechture

[SkafteNicki]

PR #1544 opened the discussion that aspects of the callback interface needs to be rethink. This issue will keep track of future discussion. From the PR, these points were made:

1. Trainer callback arguments: Currently there are 3 arguments in trainer (callback, early_stopping_callback and checkpoint_callback). It should be discussed what the user can pass to the different arguments. Mostly it seems that people are in favor of only allowing bool arguments for early_stopping_callback and checkpoint_callback, which will add a default version of the respective callback. Anything else should be passed on callback

2. Callback order: As pointed out early stopping needs to be called before model checkpoint, because modelcheckpoint save early stopping callback stats. This implies that some form of dependency tree should be implemented to callback interface

[justusschock]

What about the following:

1.) We will make callbacks a dict (if users still pass a sequence they can be converted automatically for backward compatibility). This has the advantage, that each callback has a unique specifier.

2.) Each callback can have a list of keys it depends on (identifiers from the dict).

3.) We then internally order them based on their dependencies (for same dependency level, order will be based on keys) and add them into an OrderedDict.

4.) If set to True, early_stopping and checkpoint overwrite their callback (add a certain debug log).

This way we would have unique specifiers for all callbacks, we would have an order and we could adjust the order to reflect the dependency tree...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[moi90]

Any progress on this? I think this is important because it relieves users of the need to know the correct parameter for each callback.