LoRA for Dreambooth

[polavishnu4444]

Hi,
Have come across LoRA for diffusion models.
It might be right for diffusers to collaborate and later get to one single diffusers SDK as Dreambooth is supported here as well in the current diffusers' library.
Interested to learn about your thoughts on this. 😃

Link: https://github.com/cloneofsimo/lora

[patrickvonplaten]

Yes, very interested in adding this to diffusers. Would love to review a PR here. Note that it could be as simple to copy the correct training script from: https://huggingface.co/spaces/ysharma/Low-rank-Adaptation/tree/main into examples/lora

[patrickvonplaten]

Also cc @pcuenca @williamberman @patil-suraj

[patil-suraj]

LMK if anyone is interested in adding this, very happy to help :)

[Thomas-MMJ]

@cloneofsimo

[cloneofsimo]

Ah sorry we've actually talked about this on the slack channel as well. Just finished final exam. I'll work on it today

[dhruvrnaik]

Happy to help with it @cloneofsimo

[polavishnu4444]

Interested as well if you can teach me 😃

[shirayu]

Just for your information:

This repository includes another LoRA implementation, however I haven't tried.
https://github.com/kohya-ss/sd-scripts

Description in Japanese:
https://github.com/kohya-ss/sd-scripts/blob/main/train_network_README-ja.md

They stated that the main difference between the cloneofsimo's implementation at this time is more application of LoRA to MLP in Text Encoder, FFN in U-Net, and in/out projection in Transformer.

[patrickvonplaten]

Starting working on it: #1884 -> should be ready end of week :-)

[cloneofsimo]

I'm sorry for not updating this earlier. And thank you @patrickvonplaten ! I just checked this PR as well : #1639 this would make LoRA and Paint with words implementation much easier. (But would probably need refactor as well)

I've been trying to make LoRA better before implementing here, and with contributors, we've recently made 4 major progresses and improved fidelity considerably more. I'll try my best to implement them in the examples when I find time (or anyone can jump in)

[yasyf]

@patil-suraj @patrickvonplaten I started working on the Flax version of this a while ago and got stuck. My work so far is in #1894!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patil-suraj]

LoRA is now supported cf #1884