Add support for torch.use_deterministic_algorithms

[ananthsub]

What does this PR do?

https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html
Sets the flag for using deterministic algorithms based on the trainer flag deterministic

Fixes #9107
Fixes #9544

Does your PR introduce any breaking changes? If yes, please list them.

Yes. This will raise a runtime error if there are no deterministic algorithms available. Previously, this was best-effort, and no error would be raised.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

[ananthsub]

test failures for parity tests are related due to the new flag. will take a deeper look

[codecov]

Codecov Report

Merging #9121 (9ab9bbd) into master (8c9cb0c) will decrease coverage by 4%.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #9121    +/-   ##
=======================================
- Coverage      93%     89%    -4%     
=======================================
  Files         177     177            
  Lines       15456   15460     +4     
=======================================
- Hits        14317   13698   -619     
- Misses       1139    1762   +623     

[tchaton]

LGTM !

[justusschock]

Actually, I am not sure if we can switch it that easily, Before deterministic was more like a "make it as deterministic as possible but still run it". Changing that to raising issues is quite a breaking change here even though this is more what one would expect.

Personally, I have a lot of code using that flag and relying on it not to fail.

[ananthsub]

@justusschock I agree, this would be a breaking change. Even some of our tests have been updated to avoid the runtime error raised.

I am not sure how to square this with new expectations PyTorch provides around these deterministic checks. The new API is more comprehensive and offers stricter guarantees around reproducibility.

Options:

• go with the current approach in this PR, where deterministic follows the strictest interpretation currently offered. This leads to a breaking change with potential runtime exceptions now raised if deterministic implementations aren't available. To opt out, users don't set the flag on the trainer, and instead configure the cudnn flag independently (as seen in the parity test change). The runtime error could be seen as a good thing if we're trying to provide more visibility & guarantees around determinism. But it's obviously not great dealing with newly raised exceptions.
• or ask users to set torch's use deterministic algorithms themselves, and deal with runtime errors, while the trainer does only safe changes that don't risk runtime errors raised. This is cleaner from the exception handling POV. But it's misleading because the guarantees the Lightning Trainer can make around determinism are inherently weaker. It's also the framework not leveraging PyTorch's capabilities to the fullest.

Are there other paths you see?

[carmocca]

Are there other paths you see?

deterministic=False: Not enabled
deterministic=True: Same as today. Prints info message about deterministic="strict" for discoverability
deterministic="strict": Uses torch.use_deterministic_algorithms

IMO torch should've provided a function or flag that doesn't raise a RuntimeError so that cudnn.benchmark can be easily replaced, since the new function also "will make other PyTorch operations [than CUDA] behave deterministically".

For context: pytorch/pytorch#15359

[carmocca]

@kurtamohler (author of use_deterministic_algorithms) Looks like the initial design included a mechanism to default to a warning but was eventually removed: pytorch/pytorch#38683 (comment). Where was the final decision made? Do you have any suggestions for how this PR should proceed?

Thanks :)

[kurtamohler]

I'm not sure at the moment where the discussion was, I'll do some searching.

To me, it seems like we would have to add a warn-only option to use_deterministic_algorithms. Would you mind opening an issue in pytorch and tag me in it? Otherwise, I can open an issue next time I'm at my computer

[awaelchli]

+1 for the current approach. IMO if our Trainer has an argument "deterministic" it should do what it says and the stricter version introduced here makes sense to me.

[tchaton]

LGTM !