[diffusers]: Model Manager Web UI

[keturn]

Model Manager UI needs some changes to support diffusers models:

• Description: this stays the same!
• Config: remove; a model's config file is always in a standard location in its directory.
• Model Location: needs to offer two options:
  • path for a local filesystem path, OR
  • repo_id for a model hosted on 🤗, e.g. CompVis/stable-diffusion-v1-4
• VAE Location: needs the same path or repo_id option as Model Location.
  • There might be a use case for subfolder as well, as some repositories are only a VAE like stabilityai/sd-vae-ft-mse, but you might also want to load just the VAE from a repository that has one in a subdirectory, like runwayml/stable-diffusion-v1-5 subfolder=vae. Or maybe we define some syntax (like a colon separator) to make that part of a single string?
• Width and Height: remove; we get these from the model's own settings.

plus maybe some indication of whether a model hosted with a repo_id is locally cached or not?

[blessedcoolant]

This will need rewriting of how the models.yaml is handled on the backend. Once @lstein does it on the backend, I'll make the necessary changes to the frontend.

[lstein]

VAE Location: needs the same path or repo_id option as Model Location.
There might be a use case for subfolder as well, as some repositories are only a VAE like stabilityai/sd-vae-ft-mse, but you might also want to load just the VAE from a repository that has one in a subdirectory, like runwayml/stable-diffusion-v1-5 subfolder=vae. Or maybe we define some syntax (like a colon separator) to make that part of a single string?

I'm writing the support for vae subfolders. Here is what I understand the task to be. Please confirm.

Case 1: No VAE entry (already supported)

stable-diffusion-2.1:
  description: Stable Diffusion version 2.1 diffusers model (5.21 GB)
  repo_id: stabilityai/stable-diffusion-2-1
  format: diffusers

In this case, we just load the diffuser model and whatever vae model is in the vae subdirectory will be used by default.

Case 2: VAE-specific model entry (already supported)

stable-diffusion-2.1:
  description: Stable Diffusion version 2.1 diffusers model (5.21 GB)
  repo_id: stabilityai/stable-diffusion-2-1
  format: diffusers
  vae:
      repo_id: stabilityai/sd-vae-ft-mse

In this case, we download the vae, load it, and replace the main model's VAE with this one.

Case 3: VAE in a subfolder of another model (not supported)

stable-diffusion-2.1:
  description: Stable Diffusion version 2.1 diffusers model (5.21 GB)
  repo_id: stabilityai/stable-diffusion-2-1
  format: diffusers
  vae:
      repo_id: runwayml/stable-diffusion-1.5
      subfolder: vae

In this case, we're going to load the model located in runwayml/stable-diffusion-1.5 subdirectory vae and replace the main model's VAE with this one.

Is this the correct behavior?

[keturn]

Yep, that describes the places to find VAE models. 👍

[aleyan]

repo_id for a model hosted on 🤗, e.g. CompVis/stable-diffusion-v1-4
Is it possible to have a repo_url instead? For 🤗 domains, it would use diffusers native functionality and for other domains it would download the model and pass the path to diffusers.

[lstein]

The model_manager backend does accept a URL for ckpt-based models when calling add_model(). It downloads and then installs.

[lstein]

@blessedcoolant - Are you working on this? It's the last must-have feature before we merge.

[blessedcoolant]

@blessedcoolant - Are you working on this? It's the last must-have feature before we merge.

I'm a little busy for the next 2-3 days but I'll try to see if I can find the time in between. If not, I can do it after.

But maybe in the meanwhile, you could still merge it to main and I can hotfix the model manager UI afterwards.

That way people can test out the rest of the stuff if they wish to.

[blessedcoolant]

More or less done. Might need to clean up how the VAE's are found but the rest of it is good to go I think. Added to the diffusers PR to try. Not built yet. So need to launch in dev mode.

[keturn]

done, diffusers models can be added and edited with the web model manager.

Some functions, like converting model formats, are not yet in the web UI.