Provide an interface similar to OpenAI API

[Pevernow]

Could you please provide a simple interface similar to OpenAI API?

[juney-nvidia]

@Pevernow
Can you elaborate more about your request?
Thanks
June

[Pevernow]

The most basic thing is just a simple chat/completions function similar to OpenAI.

The purpose is to facilitate access to existing applications that use the chatgpt API.

Currently, many open-source LLM projects have implemented this feature, such as the famous oobabooga/text generation webui

[merrymercy]

Users want something like this https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md, so they can switch their apps from OpenAI models to TRT-LLM models easily without code change.

[Pevernow]

@juney-nvidia Please take a look here, thank you

[gesanqiu]

Right now Python API still have a lot of issues to be fixed, I encapsulated one OpenAI API, but met #283, so you still need use C++ runtime, which means you need Triton. Spent weeks on TRT-LLM, it difficult to develop on python runtime.

@juney-nvidia What's the position of TRT-LLM's Python runime? I mean, python is easier than C++, and batch manager doesn't open source right now. Most developors may not use Triton since they won't meet that large commercial demand.

[chrjxj]

mark

[juney-nvidia]

Sorry for replying late due to being trapped by other things.

Users want something like this https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md, so they can switch their apps from OpenAI models to TRT-LLM models easily without code change.

@Pevernow @merrymercy well received, will discuss with prod about this. @ncomly-nvidia for vis.

What's the position of TRT-LLM's Python runime? I mean, python is easier than C++, and batch manager doesn't open source right now. Most developors may not use Triton since they won't meet that large commercial demand.

@gesanqiu we have already released the Python binding of C++ runtime, including batch manager, does this fulfill your requirement here?

[binarycrayon]

vote up, monitoring