Loading a SDXL model using from_single_file forces the use of the StableDiffusionXLPipeline

[Mystfit]

When loading a SDXL model using from_single_file(), the returned pipeline is always an instance of StableDiffusionXLPipeline. This means that if you try to load a model into a StableDiffusionXLImg2ImgPipeline the returned pipeline is always StableDiffusionXLPipeline instead.

The culprint is this line - https://github.com/huggingface/diffusers/blob/b024ebb96501b410ff21c8158cd075ceb3856404/src/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py#L1527C32-L1527C32

Other instances in the same file respect the pipeline_class argument passed into the download_from_original_stable_diffusion_ckpt function which is in charge of inferring the pipeline type from the input model. The logic in the linked line will always return the aforementioned pipeline instead.

Can be fixed by instantiating the pipeline using pipeline_class instead but I imagine that some safety checks might need to be performed in order to verify that the new pipeline supports the text_encoder_2 and tokenizer_2 arguments.

[patrickvonplaten]

Great catch, @Mystfit would you like to try to open a PR to fix it?

[Mystfit]

Great catch, @Mystfit would you like to try to open a PR to fix it?

Thanks! I caught it in my Unreal Engine plugin when my img2img code kept failing when using the single file pruned weights version of SDXL but dug around a bit and thought that something didn't look right.

I'll draft a PR since I've been using the pipeline_class fix on my own local code and it's fixed the problem.

For the safety checks, I've been thinking about how to check if pipeline_class supports text_encoder_2 and tokenizer_2. Would it make sense to just instantiate pipeline_class and then only set those two additional parameters externally by running hasattr on the returned pipeline instance?

I should add a test in test_stable_diffusion_xl_img2img.py as well just to verify that the correct pipeline type is being returned.

[patrickvonplaten]

Nice!

Adding a test to test_stable_diffusion_xl_img2img.py would be great. I think all _xl will always (at least) optionally support both text_encoder and text_encoder_2 so I don't think we have to add too many safety checks there, but it might also be easiest to discuss in a PR :-)