Skip to content

Commit bcdb3d5

Browse files
Community pipeline img2img inpainting (#1114)
* adds image to image inpainting with `PIL.Image.Image` inputs the base implementation claims to support `torch.Tensor` but seems it would also fail in this case. * `make style` and `make quality` * updates community examples readme Co-authored-by: Patrick von Platen <[email protected]>
1 parent 72eae64 commit bcdb3d5

File tree

2 files changed

+498
-1
lines changed

2 files changed

+498
-1
lines changed

examples/community/README.md

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,8 @@ If a community doesn't work as expected, please open an issue and ping the autho
1717
| Wild Card Stable Diffusion | Stable Diffusion Pipeline that supports prompts that contain wildcard terms (indicated by surrounding double underscores), with values instantiated randomly from a corresponding txt file or a dictionary of possible values | [Wildcard Stable Diffusion](#wildcard-stable-diffusion) | - | [Shyam Sudhakaran](https://github.com/shyamsn97) |
1818
| Composable Stable Diffusion| Stable Diffusion Pipeline that supports prompts that contain "&#124;" in prompts (as an AND condition) and weights (separated by "&#124;" as well) to positively / negatively weight prompts. | [Composable Stable Diffusion](#composable-stable-diffusion) | - | [Mark Rich](https://github.com/MarkRich) |
1919
| Seed Resizing Stable Diffusion| Stable Diffusion Pipeline that supports resizing an image and retaining the concepts of the 512 by 512 generation. | [Seed Resizing](#seed-resizing) | - | [Mark Rich](https://github.com/MarkRich) |
20-
2120
| Imagic Stable Diffusion | Stable Diffusion Pipeline that enables writing a text prompt to edit an existing image| [Imagic Stable Diffusion](#imagic-stable-diffusion) | - | [Mark Rich](https://github.com/MarkRich) |
21+
| Image to Image Inpainting Stable Diffusion | Stable Diffusion Pipeline that enables the overlaying of two images and subsequent inpainting| [Image to Image Inpainting Stable Diffusion](#image-to-image-inpainting-stable-diffusion) | - | [Alex McKinney](https://github.com/vvvm23) |
2222

2323

2424
To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.
@@ -501,3 +501,37 @@ res = pipe_compare(
501501
image = res.images[0]
502502
image.save('./seed_resize/seed_resize_{w}_{h}_image_compare.png'.format(w=width, h=height))
503503
```
504+
505+
### Image to Image Inpainting Stable Diffusion
506+
507+
Similar to the standard stable diffusion inpainting example, except with the addition of an `inner_image` argument.
508+
509+
`image`, `inner_image`, and `mask` should have the same dimensions. `inner_image` should have an alpha (transparency) channel.
510+
511+
The aim is to overlay two images, then mask out the boundary between `image` and `inner_image` to allow stable diffusion to make the connection more seamless.
512+
For example, this could be used to place a logo on a shirt and make it blend seamlessly.
513+
514+
```python
515+
import PIL
516+
import torch
517+
518+
from diffusers import StableDiffusionInpaintPipeline
519+
520+
image_path = "./path-to-image.png"
521+
inner_image_path = "./path-to-inner-image.png"
522+
mask_path = "./path-to-mask.png"
523+
524+
init_image = PIL.Image.open(image_path).convert("RGB").resize((512, 512))
525+
inner_image = PIL.Image.open(inner_image_path).convert("RGBA").resize((512, 512))
526+
mask_image = PIL.Image.open(mask_path).convert("RGB").resize((512, 512))
527+
528+
pipe = StableDiffusionInpaintPipeline.from_pretrained(
529+
"runwayml/stable-diffusion-inpainting",
530+
revision="fp16",
531+
torch_dtype=torch.float16,
532+
)
533+
pipe = pipe.to("cuda")
534+
535+
prompt = "Your prompt here!"
536+
image = pipe(prompt=prompt, image=init_image, inner_image=inner_image, mask_image=mask_image).images[0]
537+
```

0 commit comments

Comments
 (0)