DeepSpeed: Multi-GPU don't seem to converge with accumulate_grad_batches > 1

[tchaton]

Discussed in #8019

^{Originally posted by thomas-happify June 17, 2021}

seed_everything(43, worker=True)

Trainer(gpus=1, accumulate_grad_batches=16, accelerator='ddp', plugin='deepspeed',  max_epoch=20)
Trainer(gpus=2, accumulate_grad_batches=8, accelerator='ddp', plugin='deepspeed', max_epoch=20)

shouldn't these two args have similar training results?
when I use gpus=2 and accumulate_grad_batches=8, the model can't converge.

[tchaton]

@SeanNaren Mind checking this out ?

[Borda]

can we also check that the gpus=2, accumulate_grad_batches=16 converges since I guess that the accumulation is per GPU so lowering it with nb GPUs does not make much sense... @awaelchli?

[SeanNaren]

We have tests to ensure convergence with accumulation, so for us to go further we need a reproduce of some sort!