Cannot load multiple LoRAs. Need to revert PR #3649

[JemiloII]

Describe the bug

#3649 Introduced a bug. It automatically reverts any lora loaded when calling load_lora_weights. I can no longer load multiple loras. Rather than automatically unloading loras, we should have a method to unload them individually and a method to unload them all. This change has completely broke my app and I have to downgrade or revert the files in PR #3649

Reproduction

Just use load_lora_weights multiple times and note that only a single lora is in use.

Logs

No response

System Info

diffusers v0.17.1

[sayakpaul]

Were you observing expected outputs with multiple LoRAs? If so, I would appreciate some concrete examples here. I am asking this because we actually don't have the technical implementation to support operating with multiple LoRAs.

Also, if you checked the PR #3649, you'd have noticed that we had to do it to avoid latching into recursive calls to monkey-patching.

So, the broader question here is supporting multiple LoRAs in a streamlined manner (which we are internally brainstorming already).

[JemiloII]

I think the reason I didn't run into the monkey patch issue is that I would load which LoRAs I wanted to use and leave them loaded in memory. I still see this as a bug as there is no option to not disable the auto-remove feature added. If I needed to change, I would restart my app. So if I'm understanding correctly, from this PR and the other I commented on.

We want the following:

• Load LoRA & LyCORIS
• To Unload LoRAs & LyCORIS
• To Allow Multiple LoRAs & LyCORIS loaded
• Does not require an app restart

Anything else I'm missing before I get started? I am already updating mine to add an option to load_lora_weights to not auto remove/monkey patch. But I can add some more features to it if it means that it can be merged into the project without me having to constantly merge updates to it.

[sayakpaul]

I still see this as a bug as there is no option to not disable the auto-remove feature added

See #3689. Specifically #3689 (comment)

Regarding supporting other formats and multiple variants of them, we don't have anything immediate. We will continue to monitor it and look for opportunities here and happy to accept community contributions after some discussions with the potential contributors.

A good example is: #3437 (by @takuma104).

Cc: @patrickvonplaten

[takuma104]

@JemiloII If you want to use multiple LoRAs at the same time right now, feel free to use the hook version I wrote before. Although this code couldn't be directly merged into Diffusers, it's what I personally use. The usage is written in the comments, so please refer to that.
https://gist.github.com/takuma104/e38d683d72b1e448b8d9b3835f7cfa44

[JemiloII]

@takuma104 Yeah, this works very well. I was able to reuse my original LoRA scales/weights from before and generate the same images.

[sayakpaul]

@JemiloII thanks for trying out Takuma's version.

With #3781 (comment) in mind, maybe we can close the issue?

As mentioned, we will revisit the design to support multiple LoRAs from diffusers preferably using peft.

[JemiloII]

Sounds good to me. Thanks again.