[diffusers]: import CompVis-flavored ckpt models

[keturn]

As I watch the new users roll in with their suitcases full of ckpt files, I'm realizing that a thing I hoped I could hand-wave away will need to be addressed for the diffusers release: handling ckpt model weights saved from some CompVis-derived code.

The ideal UX would be to just load them seamlessly, transparently rewriting the references from the CompVis structure to the 🧨diffusers structure on the fly. I think that may be technically possible, but y'all aren't paying me enough to put that much work in to a legacy format with only a couple months of legacy behind it.

The other option would be to have an import command that converts them. It means duplicating the storage space if they are attached to keeping their old copies, but it's a one-time operation that doesn't add any burden to the rest of the process. Should be able to convert them over to safetensors while we're at it.

I'm not sure how resource-intensive the conversion process is. Saving to safetensors shouldn't be too bad? But if, for some reason, that turns out to have explosive RAM requirements, the other other option would be to search a model hub on huggingface or someplace and see if there's a host already serving a diffusers-flavored version of that model.

[damian0815]

wait, we can’t just point diffusers at a .ckpt file and it just works? or is it only certain kinds of ckpt files that don’t load? pretty sure i saw diffusers had an option to load ckpt files….

[keturn]

there are conversion scripts: https://github.com/huggingface/diffusers/tree/main/scripts

[Dekker3D]

As a new user with a suitcase full of ckpt files, which I add to every tool I use via symlinks... having to convert or redownload them to use them with InvokeAI is a real pain. It means doubling the size of my models folder, which is already weighing in at 54 gb. I want to keep it on an SSD to keep load times low, otherwise I'd put it on the much larger hdd and it'd be no problem.

[damian0815]

oh boy. this needs to be shouted really loudly in the invoke discord so that everyone who might have an opinion knows about it.

[keturn]

data point: @blessedcoolant ran one of the conversion scripts and it took 40 seconds and 10 GB of system RAM.

That seems acceptable for a one-time conversion cost, but not at all encouraging as something to do on the fly whenever we load a model.

[blessedcoolant]

I've run the test now across various v1 and v2 models. On my system the average time for the conversion is around 40-50 seconds. Which would not be an issue except that the converted model folder is quite large too. Almost as large as the original checkpoint or even larger.

This'll be an absolute deal breaker for most users. @damian0815 has opened a new issue on the diffusers repo asking for a feature to be implemented that allows the usage of the checkpoint files directly.
huggingface/diffusers#1595

Let us see what they say.

[Baughn]

I have about 270 GB of various checkpoint files, hardlinked into various UIs and branched repositories. The idea of converting all of them is... hoo boy. Deleting the originals isn't an option right now. I think I'd be forced to use a different UI.

Just my two cents.

[blessedcoolant]

I have about 270 GB of various checkpoint files, hardlinked into various UIs and branched repositories. The idea of converting all of them is... hoo boy. Deleting the originals isn't an option right now. I think I'd be forced to use a different UI.

Just my two cents.

In the initial transition phases, we are planning to retain support for both ckpt files and diffusers. But eventually, diffusers will be the way to go. Almost every single DB trained model out there has a diffusers version of the checkpoint.

The advantage of moving to diffusers outweigh the cons of the shift by a long margin. In the long term, this is the best choice to make for Invoke to keep evolving and stay up to date with the latest features and updates.

[keturn]

done, thanks to work from lstein and damian.

See !import and !convertcommands.