Define a metadata format for textual inversion concepts

[allo-]

Is your feature request related to a problem? Please describe.
The textual inversion script currently creates a dict with one term and its associated embedding. Usual scripts load one or more embeddings using the keys as token for prompt.

This format does not allow for metadata associated with the key(s). The longer the format exists, the more it will be used everywhere, so it would be good to define a more extendable format as soon as possible.

Describe the solution you'd like
Change the format from:

{
  "term": embedding_tensor
}

to something like

=== First version (leaving room for additional data for each term) ===

{
  "term": {
    "embedding": embedding_tensor
  }
}

or even

=== Second version (leaving room for additional data for each term and data for the whole file) ===

{
  "terms": [
    "term": {
      "embedding": embedding_tensor
    }
  }
}

which leaves room for adding metadata that not every script needs to understand like

=== First version ===

{
  "term": {
    "embedding": embedding_tensor
    "training_steps": 2000 # will be shown in the loading screen
    "training_files": ["hash1", "hash2", "hash3"] # is used to continue training
    "some_other_metadata": "for_the_embedding"
  }
}

=== Second version ===

{
  "terms": [
    "term": {
      "embedding": embedding_tensor
      "training_steps": 2000 # will be shown in the loading screen
      "training_files": ["hash1", "hash2", "hash3"] # is used to continue training
      "some_other_metadata": "for_the_embedding"
    }
  },
  "author": "My Name", # can be extracted for attribution
  "license": "CC-0"
  "some_other_metadata": "for_the_file"
}

The additional keys can be read or can be ignored (which means front-end developers can agree on a standard later on and convert between different formats), but moving the tensor into a sub-structure is required to make it possible to add additional data, which can be distinguished from the actual embedding data, without breaking other people's codes.

Describe alternatives you've considered
With the current structure it would require workarounds like a special key "metadata" which is skipped by scripts loading the embeddings. Another option would be to store the metadata in a separate file.

Additional context
It would often be useful to provide a embedding in different versions and it should be easy to bundle the associated metadata with the embedding.

[allo-]

Automatic1111 already uses an own format here even when it still is able to load the plain textual inversion files generated by the example script in this repo.

[patrickvonplaten]

cc @patil-suraj could you take a look here?

[patrickvonplaten]

Hey @allo- ,

I think this might be a bit too much high-level customization for an example script of diffusers at the moment.

[keturn]

Who maintains https://huggingface.co/sd-concepts-library ? Is that a 🤗 project or led by an external team? In any case, that seems like the right place to set a precedent for this. And it would be nice if diffusers an example script of how to work with one of those concept embeddings.

Or I guess maybe technically that part might be better for Transformers, because that's where CLIPTokenizer and CLIPTextModel live?

But diffusers could use an example of how you use a from_pretrained diffusion model, that constructs its tokenizer and text_encoder submodels from the distribution's model_index.json, and extends them so that it can use the concepts library.

[allo-]

It's too high-level for an example script, but somewhere should be a standard defined and it should be done before there are too many implementations.

That said ... I think the diffusers example script is one source and seems to be mainly used by sd-concepts-library (and that's the only large library I know of) and most other people use the automatic1111 UI that uses a more advanced format and can read the diffusers embeddings.

So maybe automatic1111 already defined a de-facto standard and the only thing remaining would be to ask if people are interested in formalizing a standard everyone can implement.

[patrickvonplaten]

nice if diffusers an example script of how to work with one of those concept embeddings.

cc'ing @apolinario here :-)

[allo-]

nice if diffusers an example script of how to work with one of those concept embeddings.

One of the things I want to make a PR for, when #781 is finally merged.

You basically need this code:
#781 (comment)

In my textual inversion script, the training script loads existing embeddings to continue training, what is probably also a nice example.

[patrickvonplaten]

I think a doc page would also be very nice here cc @patil-suraj

[apolinario]

Who maintains https://huggingface.co/sd-concepts-library ? Is that a 🤗 project or led by an external team? In any case, that seems like the right place to set a precedent for this. And it would be nice if diffusers an example script of how to work with one of those concept embeddings.

Hi @keturn , I maintain that, happy to set this precedent, I love the idea! I also love the idea of including an example on how to load the embeds into the model; I'll work on it! We can also chat about the precedent part and how to adapt the library