cpu_offload

[dblunk88]

Describe the bug

I tried CPU offloading and found a few issues:

1. if safety is set to None, it will error out. An if-else in the enable_sequential_cpu_offload should fix that
2. multi-GPU use (I think) is not supported since it just offloads to cuda. Could add in an optional device passthrough to tell it where to load off
3. I generator is set via cuda, it will give a runtime error that it expected a cpu type, but cuda was found
4. Community pipe mega still needs support
5. File completion goes from 10 seconds to 40 minutes, which leads me to believe that instead of CPU offloading, it actually just uses the CPU

Reproduction

No response

Logs

No response

System Info

Name: diffusers
Version: 0.7.0.dev0

Name: accelerate
Version: 0.13.2

Name: torch
Version: 1.12.1+cu116

10 GPUs

[patrickvonplaten]

Hey @dblunk88,

Thanks a lot for the issue - I think you're right about a lot of things here - especially 5.) Related PR: #1177
The weighs are actually by themselves already over 1.7GB in half precision so I think it's close to impossible to get lower than 2GB VRAM => so 2.8GB which prob includes 800MB for just starting the CUDA kernels makes sense!

A bit of False advertisement here - sorry!

[patrickvonplaten]

1.) should be solved as well now

Still need to test 3.) and 4.)

[patrickvonplaten]

5.) is fixed as well, but it requires more GPU now. Regarding 3.) I actually think we should just create it via CPU in this case. In any case it's an experimental feature, so maybe we don't need to over-optimize it just yet.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.