Diffusers->SD CKPT script doesn't work for V2 Model.

[d8ahazard]

Describe the bug

Extracting the V2-768 model with the current script (updated today) and then compiling again does not result in a working model.

Reproduction

Extract the V2-768 checkpoint using convert_original_stable_diffusion_to_diffusers.py

Rebuild to ckpt with convert_diffusers_to_original_stable_diffusion.py

Load with automatic1111 and the vanilla v2-inference-v.yaml config file.

Use txt2img to generate...anything. Result for "a photo of a dog":

Logs

No response

System Info

Diffusers version: 0.9.0
OS: Windows 11
GPU: RTX 3090

[WASasquatch]

It's a different architecture, that should be expected. Need a new script for v-prediction.

[d8ahazard]

It's a different architecture, that should be expected. Need a new script for v-prediction.

nvm its the other way around, seems likely we will get the reverse soon enough

Hence why I opened a report, to give a gentle nudge. ;)

FWIW, it appears as if @bmalthais got it working over here:

https://github.com/bmaltais/kohya_ss/blob/master/train_db_fixed.py

But even borrowing from his code, I'm not getting a viable output ckpt.

[lawfordp2017]

Please try my version posted here:
convert_diffusers_to_original_stable_diffusion_v2.py

[d8ahazard]

Please try my version posted here: convert_diffusers_to_original_stable_diffusion_v2.py

No such luck. Were you compiling diffusers from the hub, or extracted checkpoint.

[lawfordp2017]

I used the dreambooth training script:
https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth.py
which I assume starts with the diffusers version of the 768 v-diffusion model up on huggingface. Then I used the conversion script I linked earlier to convert to checkpoint and ran inference on automatic1111. Output matched what I was expecting.
Should also mention that I trained both the U-Net and text encoder.
The other direction (original SD -> diffusers) is still incomplete so I can't do round-trip testing.

Okay, sorry, I misread your initial comment.
I see you want to do round-trip but the orig.SD -> diffusers script is still partly broken. I had originally thought you wanted to convert trained dreambooth models for use on other inference platforms.

[d8ahazard]

Ah, that could be my issue then. I'll double-check with the hub version and see if that creates a valid checkpoint. Then I've got at least half the battle won. ;)

[nerdyrodent]

Script worked here - thanks :)

[lawfordp2017]

I've updated so both directions should work now for the 768-v model.

[d8ahazard]

I've updated so both directions should work now for the 768-v model.

Sorry, took me a bit of time to integrate with my setup and test, but I can confirm, your script will successfully compile a working V2.0 Checkpoint. Thank you so much!

Also, as a side note, I did a diff between the weights pulled down from the hub versus the ones generated by the SD->Diff script, and can confirm that the outputs are identical. Well, nearly. Generated configs have the filename for the bin saved to them. But, that has zero bearing on the actual model data.

So, great work. I'd definitely suggest submitting a PR so this gets pulled into main. ;)

[patrickvonplaten]

We'd be happy about a PR to add this new conversion :-)

[lawfordp2017]

I'll try to create one later today.

[patrickvonplaten]

Just merged your PR, sorry for the delay 🙏

Super nice that you put in the effort @lawfordp2017 :-)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.