new OffloadingDevice loads one model at a time, on demand

[keturn]

Trying something different in place of #2542.

Fixes #2326.

[keturn]

The accelerate feature the previous PR used does this terribly clever thing where the parameters are defined on the meta device until their module's forward method is called.

That often resulted in foo.device returning meta, which is useless for trying to prepare tensors you want to pass to that model, but at least it's obviously useless upon inspection.

In contrast, having models shuffled off to cpu means that foo.device is never the unusable meta, but it does mean that the foo.device you see now isn't necessarily what foo.device will be when you foo(), so we still have to be cautious about relying on .device.

[keturn]

It's minimally working now, tested for txt2img in text mode only.

There are still a few other things to clean up, e.g. web runs in to an error in lowres_estimated_image.

[keturn]

added a workaround for the estimated image function, then moved on to testing inpainting and that has its own bucket of problems.

I assumed that vae.decode was a problem because its forward method was encode and so decode was the "extra" method, but no. Its forward does both for some reason, so neither encode nor decode get the hooks.

And then it's got the usual set of "tensors prepared using vae.device end up on cpu" problems, which we have otherwise but are multiplied because inpainting deals with more stuff (input image, mask, etc).

I'm coming to believe that we can't do this completely transparently like this, with the Pipeline class not aware of how the model devices are managed. I think it needs a way to discover what their actual execution device is, whether that's through inquiring with an Offloader instance or some property we add to the models.

[keturn]

This is starting to shape up now, and the usage of it doesn't require nearly so many awkward workarounds as the previous attempt.

It's working in many cases, but my troubleshooting is currently impeded by something that's swallowing all the console output.

[damian0815]

i hit resolve on both of those conversations, thanks for making the naming change

[lstein]

Neat! Very elegant solution.