[diffusers]: EmbeddingManager #1778
Description
Textual Inversion, concept embeddings, whatever we're calling it: that thing where a prompt can include some tokens that aren't included in the base model, and we load the vectors for them from some auxiliary data source and include them in the embeddings we pass to the model for conditioning.
EmbeddingManager has no direct counterpart in the diffusers model, nor do the model configurations for diffusers models have any personalization_config element.
#1583 currently includes a crude patch adding an EmbeddingManager, but I'm not sure if it is sufficient or appropriate.
We need, at the very least, some test cases we can use to evaluate whether that's working. They should include single-token/single-vector embeddings as well as the multi-token or multi-vector kind.
The good news is that both the legacy implementation and diffusers-flavored models use the same CLIP models from transformers for tokenizing and embedding, so none of this should require much new code.
Related reading: