[bug]: `free_gpu_mem` does not offload diffusers models

[keturn]

To Do

• avoid throwing an exception if free_gpu_mem is on and a diffusers pipeline is loaded
• connect free_gpu_mem configuration flag to diffusers_pipeline "sequential CPU offload" option, and make sure that all still works in conjunction with ModelManager's CPU caching. didn't work, see fix free_gpu_memory option for diffusers models [using accelerate cpu_offload] #2542

Traceback (most recent call last):
  File "InvokeAI/backend/invoke_ai_web_server.py", line 1216, in generate_images
    self.generate.prompt2image(
  File "InvokeAI/ldm/generate.py", line 463, in prompt2image
    if self.free_gpu_mem and self.model.cond_stage_model.device != self.model.device:
AttributeError: 'WeightedPromptFragmentsToEmbeddingsConverter' object has no attribute 'device'

Additional context

I think there are a couple different memory-related flags?

If this is the one that just forces garbage collection before doing the decode, I think that's cheap enough to do that we might as well always do it and spare the trouble of documenting a configuration flag for it.

But do we also have some other flag that should be mapped to the diffusers sequential CPU offload option?

Update: No, those weren't separate things in Invoke, free_gpu_mem did both, so that's the option to map to sequential CPU offload.

InvokeAI/ldm/invoke/ckpt_generator/txt2img.py

Lines 58 to 63 in 251a409

	if self.free_gpu_mem:
	self.model.model.to('cpu')
	self.model.cond_stage_model.device = 'cpu'
	self.model.cond_stage_model.to('cpu')
	gc.collect()
	torch.cuda.empty_cache()