0.5.1 k schedulers

[tonetechnician]

@patil-suraj This PR is related to conversions that have been happening for this PR #636.

It ports @hlky's implementation of k-diffusion schedulers up to 0.5.1 of diffusers.

Currently the output is working for txt2img pipeline (although not matching results from k-diffusion output), but there are still issues with img2img pipeline output which I'm resolving and some of that can be followed in this issue report #901.

Also some further work needs to be done for DPM2 and Heun solvers which are meant to accept the model and use that to determine the next sample step https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/sampling.py#L114. I believe @hlky implemented an approximation of this which is why we currently do get reasonable output.

I have some code that I'm wanting to push up soon that allows the model to passed to the step scheduler, but still solving some issues there.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[tonetechnician]

Here is some example output that I've got with settings:

prompt: "A dense Cyberpunk city street"
num_inference_steps: 20
guidance_scale: 7.5
seed: 0

EULER:

EULER-A:

DPM2:

DPM2-A:

HEUN:

LMS:

PNDM:

DDIM:

DDPM:

[andrea-gatto]

Thanks a lot @tonetechnician for your work!
JFYI I tested your branch with the euler schedulers and I wasn't able to replicate the images above. Also the images produced with these two schedulers look more grainy wrt the images generated using @hlky euler schedulers

[tonetechnician]

Thanks for checking it out @andrea-gatto

There does seem to be a difference between 0.3.0 and 0.4.0 as mentioned here #902 which could be related to what you're experiencing.

That being said, you should be getting the same output as me with my PR branch, so something is off. I'll dig in a bit more and see what that might be. Would you be able to post your output for reference?

Thanks!

[andrea-gatto]

Thanks @tonetechnician for the heads up
Moving to 0.3.0 doesn't have any effect on the quality for me.
As you can see below my generations are similar to yours (especially Euler), but with much lower quality.
This is the code I've used to replicate your results

import torch
from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler, EulerAncestralDiscreteScheduler

device = "cuda"
model_id = "CompVis/stable-diffusion-v1-4"
scheduler = EulerAncestralDiscreteScheduler()
pipe = StableDiffusionPipeline.from_pretrained(model_id, use_auth_token=True, scheduler=scheduler)
pipe = pipe.to(device)
generator = torch.Generator(device).manual_seed(0)

prompt = "A dense Cyberpunk city street"
with torch.autocast("cuda"):
    image = pipe(prompt, guidance_scale=7.5, num_inference_steps=20, generator=generator).images[0]

image.save("test.png")

EULER

EULER-A

[tonetechnician]

Thanks for reference to images and code @andrea-gatto.

Sorry, I should've posted my scheduler settings:

scheduler = EulerAncestralDiscreteScheduler(
    beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear"
)
self.pipe.scheduler = scheduler
output = self.pipe([prompt], generator=generator, guidance_scale=guidance_scale, num_inference_steps=num_inference_steps, seed=seed)

Might be that the scheduler is defaulting to linear in yours?

[andrea-gatto]

thanks a lot @tonetechnician
ha I see, I thought you were using the default values :)
updating your values I'm able to replicate your euler result, still different for euler-a (though the quality has improved dramatically)
EULER

EULER-A

[sausax]

Great work @tonetechnician . One thing I noticed that this takes little more memory than the main branch. I was getting CUDA memory error on a 8gb card. Worked fine on more memory card.

[patil-suraj]

Amazing work @tonetechnician ! I would suggest to split this PR into two.

1. for euler schedulers
2. for DPM and Heun

The euler schedulers are already working great and would be awesome to have them merged soon! Then we can start discussing the design for DPM & Heun in other PR to so this won't get blocked by it.

[patil-suraj]

Hey @tonetechnician , let us know if we could help in any way, would be cool to have the euler schedulers merged soon!

[tonetechnician]

Hey @patil-suraj

All good for the meanwhile, just had some backlog work that prevented me from splitting this out. Should be able to do so today and make a PR

[patil-suraj]

No worries ! And awesome, looking forward to it, this is going to be really nice addition!

[hlky]

@patil-suraj #1019

[patil-suraj]

Thank you @hlky will take a look soon :)

[tonetechnician]

Solved here --> #1019

[patrickvonplaten]

Should we open a new PR for the 2nd order schedulers? We're still definitely be interested in adding those if you'd like :-)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

I think all the 2nd order schedulers have now been added. Thanks a mille for you first draft here @tonetechnician - it has been super useful to get the other schedulers correctly added.

[tonetechnician]

Great work everyone! SO EPIC to see this + 2.0 + depth2img. Unfortunately been MIA on another project that has taken my attention for the last few weeks but back in today :) Pulled latest changes and was happy to see the second order samplers!

Thanks @patrickvonplaten for the mention! But great work to everyone for getting this in, so good :) Look forward to checking them all out. Feel free to close this PR @patrickvonplaten :)