feat: add repaint

[Revist]

Added repaint pipeline based on https://arxiv.org/pdf/2201.09865.pdf @anton-l

In tests i was not sure where you take numbers to compare against from so I run the pipeline and pasted what it produces for me:
[0.14537135, 0.10728511, 0.08822048, 0.15828621, 0.11806837, 0.11007798, 0.15231332, 0.1214554, 0.15475643]

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[anton-l]

Hi @Revist! Excited to see a working RePaint port, thank you so much for working on it!

Overall the implementation is super nice, just left a couple of suggestions to bring it closer to the Stable Diffusion inpainting API (which got added after my original repaint PR).

Now I wonder if it's possible to move the noising & masking logic from RePaintScheduler to RePaintPipeline, so that it can be used with other schedulers like PNDM or K-LMS? By using just step() and add_noise(), similar to the unsupervised inpainting pipeline for Stable Diffusion: https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint_legacy.py#L46
What do you think, would that be possible to do here?

[Revist]

Hi @anton-l !

Was doing some reading as i was not that familiar with these, but it seems that although theoretically that should be possible (after all RePaint just repeats some steps) in practice you have some problems like here:
https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_pndm.py#L209
where the step depends on some counter and not only on timestep itself. To include PNDM first thing would be to remove such dependency.
Then one could build repaint timesteps and ask with step() and add_noise() to go forward and backward, where how to go forward would be recognized by timestep value and scheduler name.

[Revist]

Reading further in to make a step with PNDM you do not need a single model_output at single time but at t, t+delta/2 and t+delta (from different inputs in fact so we have 4 passes) so that changes logic here:
https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pndm/pipeline_pndm.py#L85
where you would like to give a single model_output to step, you would have to pass model (unet) and image (x_t) instead.
You skipped this problem by creating virtual steps in this scheduler which are not really steps, then using this counter. But i would say the problems arise from this (and makes code complex imo).
In other words it ought to be true that if self.timesteps contains some step then this is really a step in time and not some placeholder for making calculations.

[Revist]

@anton-l I am not sure if you agree with the above comments. If so I could take a look at how to change the code to the PNDM (and possibly other) schedulers however that seems like significant changes across code.

[anton-l]

@Revist I've tried refactoring it a bit and completely share your concerns now, let's leave the scheduler as is for now!

Pinging @patrickvonplaten @patil-suraj for a second review, as this is a new pipeline and scheduler :)

[patrickvonplaten]

Ok for now, but let's make sure to pass to numpy image tests in the very near future

[anton-l]

Will fetch the numpy images for all integration tests in a bulk PR 👍

[patrickvonplaten]

Looks good to me ! Before merging let's please make sure to add it to the docs as well and maybe reach out to the original author here?

[anton-l]

Added the docs for the pipeline: https://moon-ci-docs.huggingface.co/docs/diffusers/pr_974/en/api/pipelines/repaint
And the scheduler: https://moon-ci-docs.huggingface.co/docs/diffusers/pr_974/en/api/schedulers#diffusers.RePaintScheduler

Gently pinging the RePaint author @andreas128 for an optional review and any comments, in case out email thread gets lost 🤗

[anton-l]

@Revist the pipeline will be included in today's diffusers==0.7.0 release! 🥳

If you're interested, maybe we could work together on a Spaces demo similar to SD-inpainting https://huggingface.co/spaces/runwayml/stable-diffusion-inpainting to promote the pipeline further, let me know what you think!
Also would be interesting to try integrating OpenAI's improved-diffusion models to completely reproduce the RePaint paper 🔥

[Revist]

@anton-l Sounds interesting. What models would you like to integrate from improved diffusion? I was thinking of taking a look into #894 as this may be of use to our group in the near future. Maybe we could speak on some other communicator?

[anton-l]

@Revist ping me ( @anton-l ) on the HF discord: https://discord.gg/G7tWnz98XR
We have a diffusers core-contributors channel there, where we can discuss everything further :)