[bug]: Pytorch versions after 1.11 have a massive 50% performance reduction (CLI)

[n00mkrad]

Is there an existing issue for this?

• I have searched the existing issues

OS

Windows

GPU

cuda

VRAM

24 GB

What happened?

Pytorch 1.11.0:
python -m pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 --extra-index-url https://download.pytorch.org/whl/cu113

=> 22 it/s at 512x512 with SD 1.5 fp16

Latest Pytorch 1.13.1:
pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/cu117

=> 11 it/s at 512x512 with SD 1.5 fp16

Screenshots

No response

Additional context

No response

Contact Details

No response

[n00mkrad]

To reproduce simply install 1.11.0 with the command I posted above and watch your performance double.

My testing was done with SD 1.5, 30 steps, half-precision, k_euler_a.

Hardware is Ryzen 5900X, RTX 4090, Drivers 531.18 with CUDA 12.1

[ebr]

Interesting find. I will try to also reproduce this.

What CUDA version (if any) have you got installed on the host?

Edit: nvm, I see this is Windows; ignore the last question, and I will try to reproduce on Linux.

[psychedelicious]

I cannot figure out the right set of dependencies needed to run on pytorch 1.11.0. On Linux.

I can confirm, however, that my speeds on a 4090 are the same as what you indicated for pytorch 1.13.1 (~11it/s), and I have suspected this isn't quite right since I got the card. I expected about twice that, with the possibility to hit 40it/s with the right combination of python packages, CUDA, and drivers. Just haven't gotten around to exploring this with InvokeAI.

[psychedelicious]

Ok, so I just doubled my speeds by updating to the latest pytorch and CUDA:

• pip uninstall invokeai
• pip install InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu118
• pip uninstall xformers

This installs, for better or worse, pytorch 2.0.0. I think this is unintentional and due to an overly permissive dependency on torch>=1.13.1. Better, because this combination doubles speeds. Worse, because maybe this will cause problems on some systems as it is a major version bump.

Having xformers enabled totally breaks it (and we do not need it on pytorch 2.0.0 anyways).

Edit: 4090 now getting ~33it/s on SD1.5 512x512, see #2962

[n00mkrad]

I can confirm that, without altering anything else about my setup, upgrading straight from 1.11 to 2.0.0 "fixes" the issue and even gives me some extra speed. 25 it/s without further optimizations.

[github-actions]

There has been no activity in this issue for 14 days. If this issue is still being experienced, please reply with an updated confirmation that the issue is still being experienced with the latest release.