Potential regression in deterministic outputs

[pcuenca]

Describe the bug

I've started noticing different outputs in the latest version of diffusers starting from diffusers 0.4.0 when compared against 0.3.0. This is my test code (extracted from a notebook):

import diffusers
from diffusers import StableDiffusionPipeline, DDIMScheduler
import torch
from IPython.display import display

def run_tests(pipe):
    torch.manual_seed(1000)
    display(pipe("A photo of Barack Obama smiling with a big grin").images[0])
    torch.manual_seed(1000)
    display(pipe("Labrador in the style of Vermeer").images[0])

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe = pipe.to("cuda")
run_tests(pipe)

The first prompt produces identical results. The second one, however, results in different outputs:

0.3.0

main@a3efa433eac5feba842350c38a1db29244963fb5

Using DDIM, both prompts generate different images.

scheduler = DDIMScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", scheduler=scheduler)
pipe = pipe.to("cuda")
run_tests(pipe)

DDIM 0.3.0

DDIM main

DDIM 0.3.0

DDIM main

In addition, there's this post from a forum user with very different results in the img2img pipeline: https://discuss.huggingface.co/t/notable-differences-between-other-implementations-of-stable-diffusion-particularly-in-the-img2img-pipeline/24635/5. They opened another issue recently #901. Cross-referencing, may or may not be related to this issue.

Reproduction

As explained above.

Logs

No response

System Info

diffusers: main @ a3efa433eac5feba842350c38a1db29244963fb5 vs v0.3.0

[pcuenca]

Update regarding the img2img pipeline: there are small differences using DDIM, I didn't notice any with PNDM or k-LMS. All tests done in full precision for now.

DDIM v0.3.0

DDIM main

[patrickvonplaten]

Thanks a lot @pcuenca !

Do you think we could try to narrow down even more when the differences started happening? E.g. I guess it's between 0.3.0 and 0.4.0 , but then it'd also be nice to find out exactly which PR is the origin :-)

Otherwise happy to look into it myself in ~2 days!

[pcuenca]

Yes, I'll investigate a bit :)

[pcuenca]

Update: 0.4.0 seems to suffer from the same behavior as 0.6.0.

[changjonathanc]

I tried it on 0.3.0 and some 0.5.0 fork I have. And in both version the dog looks like the second one. Can you reproduce it with docker?

[pcuenca]

I tried it on 0.3.0 and some 0.5.0 fork I have. And in both version the dog looks like the second one. Can you reproduce it with docker?

That's interesting. I haven't tried docker, but all the tests were done in the same system just updating the version of diffusers.

[ydshieh]

@pcuenca Could you provide the system info? GPU type, python/torch/torchvision versions, CUDA/CUDNN versions?

[ydshieh]

I can't reproduce with T4/V100 GPU on GCP for the default scheduler. Both commits give the second dog image. Also google colab gives the same image.

[ydshieh]

For dog prompt with DDIM: running from commit 3f55d13 to 9bca402, there are 4 commits where image difference occurs

074-a9fdb3d => 075-6bd005e: 0.09812037150065105
099-3b747de =>100-bd8df2d: 0.0812060038248698
116-a7058f4 => 117-9ebaea5: 0.22810236612955728
132-1070e1a =>133-f1484b8: 0.17296854654947916

The last number is the ratio of equal values over the total number of values in the numpy arrays
I upload the images to this repo.

For the default scheduler, also these 4 commits give some difference in the numpy arrays, but the ratio > 0.998, so visually we can't see any difference.

[ydshieh]

[pcuenca]

Hi @ydshieh, thanks a lot for looking into this! I've done a few more experiments and these are my results:

My workstation

- GPU: 3090, CUDA 11.4
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.31
- Python version: 3.9.12
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Huggingface_hub version: 0.10.0
- Transformers version: 4.22.2

0.3.0 fp32

0.6.0 fp16 (Note different ear)

0.6.0 fp32 (Blue hat)

Colab T4

- GPU: Tesla T4, CUDA 11.2
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Huggingface_hub version: 0.10.1
- Transformers version: 4.23.1

0.3.0 fp32

0.6.0 fp16

0.6.0 fp32

Colab A100

- GPU: A100 (40 GB) CUDA 11.2
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Huggingface_hub version: 0.10.1
- Transformers version: 4.23.1

0.3.0 fp32

0.6.0 fp16

0.6.0 fp32

As you can see, results differ across versions. Even if we ignore the fp16 tests, results in the A100 Colab are different to those in the T4 environment. The A100 fp32 versions are consistent between diffusers 0.3.0 and 0.6.0 (no blue hat), and T4 fp32 are also consistent but different (blue hat). In my computer, 0.3.0 has no blue hat but 0.6.0 has.

Any idea about what might be going on?