Trainer should invoke torch.use_deterministic_algorithms()

[cowwoc]

When Trainer(deterministic=True) then the implementation should invoke torch.use_deterministic_algorithms(True) under the hood, otherwise there is a chance that the underlying code will be non-deteministic.

Source: https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html

[ananthsub]

@cowwoc there is an open PR for this: #9121

The issue blocking that PR's merge is setting use_deterministic_algorithms can raise a runtime error, which is not backwards compatible behavior with the current setting