[Feature Request] Documentation around multiple pipelines reusing model weights

[theahura]

Is your feature request related to a problem? Please describe.
If I have a StableDiffusionPipeline, a StableDiffusionImg2ImgPipeline, and a StableDiffusionInpaintPipeline, these will all load separate weights to the GPU even though they are all using the same model. This makes creating applications that use the full breadth of SD difficult to create.

Describe the solution you'd like
It seems like it is possible to reuse weights, according to https://discuss.huggingface.co/t/access-clip-from-stablediffusionpipeline-and-use-the-same-models-for-multiple-pipelines/24813. However, this is not at all clear from the API docs and it's also unclear whether we need to do things like route to cuda multiple times.

Describe alternatives you've considered
NA

Additional context
NA

[victordibia]

I encountered a similar issue while attempting to use the pipelines for an app (peacasso) that allows multiple model behaviours (text2img, img2img, inpainting +).

I ended up writing my own pipeline that "stitches" in all of these functionality into a single pipeline with multiple parameters . The caveat here is that I have to update the pipeline each time I upgrade diffusers (as the api may have changed ... and has changed frequently over the last few weeks). I am going with this approach as I have a few experimental behaviors I am working on that will need custom changes to the pipeline.

Another (easier) solution is to use the) community pipeline ( stable_diffusion_mega ) written by @patrickvonplaten which integrates all three flows.

Based on the documentation here, you can use it as follows:

#!/usr/bin/env python3
from diffusers import DiffusionPipeline
import PIL
import requests
from io import BytesIO
import torch


def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")

pipe = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", custom_pipeline="stable_diffusion_mega", torch_dtype=torch.float16, revision="fp16")
pipe.to("cuda")
pipe.enable_attention_slicing()


### Text-to-Image

images = pipe.text2img("An astronaut riding a horse").images

### Image-to-Image

init_image = download_image("https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg")

prompt = "A fantasy landscape, trending on artstation"

images = pipe.img2img(prompt=prompt, init_image=init_image, strength=0.75, guidance_scale=7.5).images

### Inpainting

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
init_image = download_image(img_url).resize((512, 512))
mask_image = download_image(mask_url).resize((512, 512))

prompt = "a cat sitting on a bench"
images = pipe.inpaint(prompt=prompt, init_image=init_image, mask_image=mask_image, strength=0.75).images

[patrickvonplaten]

Hey @theahura,

That's a good point - we should definitely add this somewhere to the docs. Where do you think would be the best place to discover it for newcomers? Somewhere under "using-diffusers" ? here: https://huggingface.co/docs/diffusers/main/en/using-diffusers/

[theahura]

I personally was looking in the README, and I think it's basically one paragraph. It would fit right below the section where you describe the different pipelines (e.g. img2img, inpaint):

It's possible to reuse weights between pipelines. For example, if you wanted to use StableDiffusion and StableDiffusionImg2Img, you could initialize the weights for the first and apply them to the latter:

SD = diffusers.StableDiffusionPipeline('url', ...params...)
SD_IMG2IMG = diffusers.StableDiffusionPipelineImg2Img('url', unet=SD.unet, scheduler=SD.scheduler, token_extractor=SD.token_extractor, ...)

SD_PIPE = SD.to("cuda")
SD_IMG2IMG_PIPE = SD_IMG2IMG.to("cuda")

Now you can call both pipelines and they will reuse the weights.

[theahura]

I believe these weights can also be reused in community pipelines? but not 100% sure

[patrickvonplaten]

Cool! Would you like to open a PR to add it to the docs? :-)

[theahura]

Sure thing, see: #1221

[WASasquatch]

https://huggingface.co/docs/diffusers/main/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.components

For outside reference (though to note, it does say img2text which is odd for a StableDiffusionPipeline text2img pipeline -- though I'm dyslexic and would do the same thing... :P)

[patrickvonplaten]

Good catch @WASasquatch - thanks :-) Here a PR to fix it: #1329

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.