[Feature request] Support soft_prompt or inputs_embeds?

[xiaocaoxu]

In Fastertransformer, t5 supports inputs_embeds as an input and gpt supports soft_prompt, it's necessary when using llm as part of a multimodal model.

[symphonylyh]

@xiaocaoxu we're working on multi-modal support that can take embedding from other modalities and feed to LLM (OPT/LLaMa/Encoder-decoder). Stay tuned.

[xiaocaoxu]

Okay, thank you.

[panzeyu2013]

@xiaocaoxu we're working on multi-modal support that can take embedding from other modalities and feed to LLM (OPT/LLaMa/Encoder-decoder). Stay tuned.

I'm wondering if there is any alternative solution for inputs_embeds if I want to use Tensor-RT or Tensor-RT-LLM

[symphonylyh]

@panzeyu2013 did you mean how to use input_embeds instead of input_ids as the input? Yes, a good example would be using BLIP2's prompt tuning as a way to pass in input_embeds: https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/blip2/run.py#L81. If you're coming from multi-modal models, we will have a better general multi-modal support soon.

[panzeyu2013]

Thanks for your reply, I was working on a project of multi-modal model which utilize Llama as a crucial component. It would be efficient if I can compile the language model by Tensor-RT

[symphonylyh]

@panzeyu2013 makes sense. Stay tuned as we'll release general multi-modal support (flexible choices of visual encoders + flexible choices of language models) after the holiday season

[symphonylyh]

@panzeyu2013 @xiaocaoxu we now released the multi-modal support for BLIP w/ OPT or T5, and LLaVA. Please take a look :)
Annoucement: #847

Feel free to re-open this if you have encountered any issue. But since this issue is a feature request, would better file a new issue regarding any specific problem.