StableDiffusionXLImg2ImgPipeline not working as expected when using with stable-diffusion-xl-base-1.0 checkpoint

[yiyixuxu]

Describe the bug

I can't get the classic img2img example to work with sdxl 1.0
this is the output I get using the script I attached in reproduction section

as a reference, here is output for sd1.5 with same inputs (when resize image to 768 x 512)

Reproduction

import requests
import torch
from PIL import Image
from io import BytesIO

from diffusers import StableDiffusionImg2ImgPipeline, StableDiffusionXLImg2ImgPipeline, AutoencoderKL

# inputs

device = "cuda"

url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg"

response = requests.get(url)
prompt = "A fantasy landscape, trending on artstation"

## pipe sdxl
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((1024, 1024))
model_id_or_path = "stabilityai/stable-diffusion-xl-base-1.0"

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    model_id_or_path, 
    vae=vae,
    torch_dtype=torch.float16)
pipe = pipe.to(device)

generator = torch.manual_seed(0)
images = pipe(prompt=prompt, image=init_image, strength=0.75, guidance_scale=7.5, generator=generator).images
images[0].save("fantasy_landscape_sdxl.png")

Logs

No response

System Info

• diffusers version: 0.21.0.dev0
• Platform: Linux-4.19.0-24-cloud-amd64-x86_64-with-glibc2.28
• Python version: 3.10.8
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Huggingface_hub version: 0.13.2
• Transformers version: 4.27.3
• Accelerate version: 0.17.0.dev0
• xFormers version: 0.0.20
• Using GPU in script?: YES
• Using distributed or parallel set-up in script?: NO

Who can help?

@patrickvonplaten

[yiyixuxu]

The result looking good now with strength= 0.99 !

[yiyixuxu]

comparing euler vs ddim at different strength values, I would tell if one is better than another

strength = 0.75

euler	ddim

strength = 0.80

euler	ddim

strength = 0.86

euler	ddim

strength = 0.91

euler	ddim

strength = 0.97

euler	ddim

strength = 1.

euler	ddim

[bghira]

for anyone who comes along later, try Inverse DDIM, too!