sequential cpu offload causes > 3x slowdown

[keturn]

Using StableDiffusionPipeline.enable_sequential_cpu_offload causes massive slowdown. e.g. from 6 it/s to 2.

Reproduction

Standard pipeline:

pipe = diffusers.StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    revision="fp16", torch_dtype=torch.float16,
)
pipe.enable_sequential_cpu_offload()
x = pipe("a jockey riding a dragon on mars")

System Info

• diffusers version: 0.12.1
• Platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
• Python version: 3.10.6
• PyTorch version (GPU?): 1.13.1+cu117 (True)
• Huggingface_hub version: 0.12.0
• Transformers version: 0.16.0
• Accelerate version: not installed env lied, it is 0.16.0
• xFormers version: 0.0.16
• Using GPU in script?: RTX 3060

[keturn]

my hunch is that this is because accelerate.cpu_offload works by wrapping a hook around the model's forward call.

so when our sampling loop calls forward 50 times, it's offloading and re-loading each time.

[patrickvonplaten]

Super nice issue - thanks a lot @keturn!
Think your hunch in spot on. @pcuenca is looking into it.

Maybe we'll do a temporary fix in diffusers and try to longer term add a nice UX directly to accelerate's cpu_offload method.

[pcuenca]

I guess we can close this issue after #2285 was merged. Is that ok, @keturn?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.