Connectors Refactoring Discussion #4510
Description
This is a brief summary of our refactoring discussion with @tchaton @awaelchli .
The easy part
Refactor all connectors to be completely self-contained and establish call hierarchy.
This means:
- Connectors should not call other connectors
- Connectors should not have a trainer reference or change anything on trainer
- Trainer properties will just gather information from respective connector.
A potential call hierarchy could be like this:
Trainer -> Loops -> Connectors -> Accelerators -> Plugins
That also means, that every class can only call types that are below it in the hierarchy and nothing that is on the same level or higher. I.e. Accelerators cannot call other accelerators, trainer, loops or connectors but only plugins.
The more difficult part
Refactor all Accelerators and Plugins to further separate them.
Accelerators should only contain hardware specific stuff, whereas plugins include changes in the training routine.
So we would end up with something about 3-5 accelerators and everything else (like DDP, DDP2, DDP-Spawn, DP, AMP etc.) would become a Plugin. This has the advantages that
1.) it is cleaner to implement once the things are more separated
2.) It is easier to look at specific parts since currently the DDP stuff for example is still scattered across the plugin and the accelerators
3.) Every new plugin should apply to all/most accelerators with no change.
How to do this
The first part can be changed slowly while fixing bugs and implementing features.
The second part however needs to be done more carefully since this requires breaking changes in our internals.
So I'll start there with a draft and see, how hard it would be and how the result would look like before we apply this to all the accelerators and plugins.