Progressive distillation

[bglick13]

Starting to work on implementing PROGRESSIVE DISTILLATION FOR FAST SAMPLING OF DIFFUSION MODELS

This is very much a draft PR. So far I've included a toy example in a notebook.

First, it trains an unconditional image diffusion model on a single image. Then, it implements the distillation training procedure - training a student model to produce the same output in N // 2 steps.

TODOs include:

• Verify the equation block in the distillation function is correct
• Test on multiple iterations of distillation (i.e., N = 1000 -> 500 -> 250...)
• Test on a larger image dataset
• Clean up changes in DDIM scheduler
• Convert distillation process to a pipeline(?) or script

[bglick13]

@lukovnikov sorry for the delayed response. I think that's an important meta decision to make with the Diffusers team! I've seen a similar approach e.g., in the Imagic PR. For now I'm still working on getting the distillation results as good as the paper's!

[lukovnikov]

@bglick13 I've also implemented distillation in my fork: https://github.com/lukovnikov/diffusers/blob/mine/examples/unconditional_image_generation/distill_unconditional.py
However, I'm facing some strange issues where the contrast of the generated images is increased after every distillation phase. Did you have similar issues and did you manage to reproduce the paper's results?

[Thomas-MMJ]

lukovnikov - at a guess the clip text encoder tokens are also being trained and are becoming 'longer' (attention focuses more on longer vectors which are concepts that the text encoder knows well, which is why popular celebrities can get an overbaked appearance in default models) which causes the images to be overbaked.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[adhikjoshi]

@patrickvonplaten do you think this is something to give it a try again?