Runtime error when running Stable Diffusion Mega pipeline (inpaint)

[kovtcharov]

When attempting to execute the Stable Diffusion Mega pipeline snippet, the following runtime error was triggered.

This seems to be a datatype mismatch, a workaround was found by commenting out torch_dtype and revision of the pipeline.

Reproduction

Execute the Stable Diffusion mega pipeline found here.

Logs

Traceback (most recent call last):
  File "c:\Users\k\Work\BE\libs\stablediffusion.py", line 177, in <module>
    images = pipe.inpaint(prompt=prompt, init_image=init_image, mask_image=mask_image, strength=0.75).images
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\k/.cache\huggingface\modules\diffusers_modules\git\stable_diffusion_mega.py", line 139, in inpaint
    return StableDiffusionInpaintPipelineLegacy(**self.components)(
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_inpaint_legacy.py", line 397, in __call__
    image, has_nsfw_concept = self.safety_checker(images=image, clip_input=safety_checker_input.pixel_values)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\diffusers\pipelines\stable_diffusion\safety_checker.py", line 38, in forward
    pooled_output = self.vision_model(clip_input)[1]  # pooled_output
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\transformers\models\clip\modeling_clip.py", line 845, in forward
    return self.vision_model(
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\transformers\models\clip\modeling_clip.py", line 776, in forward
    hidden_states = self.embeddings(pixel_values)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\transformers\models\clip\modeling_clip.py", line 133, in forward
    patch_embeds = self.patch_embedding(pixel_values)  # shape = [*, width, grid, grid]
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\nn\modules\conv.py", line 446, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Users\k\Anaconda3\envs\eveserver\lib\site-packages\torch\nn\modules\conv.py", line 442, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

System Info

Following packages are used:

Name: diffusers
Version: 0.6.0

Name: torch
Version: 1.10.1+cu113

[kovtcharov]

Also tried it with the latest version of pytorch, same issue:

Name: torch
Version: 1.13.0+cu117

[patrickvonplaten]

Hey @kovtcharov,

Thanks a lot for the bug! It should be solved in #1137 :-)

[kovtcharov]

Hey @kovtcharov,

Thanks a lot for the bug! It should be solved in #1137 :-)

Excellent, thank you for being on top of this!