Return log probabilities for tokens

[yunfeng-scale]

Hi, we'd like to have log probabilities for tokens returned from the model, in addition to the token ids. Can you help with this feature request?

[byshiue]

Here is a flag to output log probs of generated tokens https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/runtime/generation.py#L280. The log probs of input tokens are not supported now, it will be supported in near future.

[yunfeng-scale]

@byshiue thanks. Possible to include in the C++ implementation too? https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/include/tensorrt_llm/batch_manager/callbacks.h#L32 currently inflight batching is supported through C++ runtime

[Shmdtalha]

Is it possible to return top N logprobs? Currently I see only 1 logrpobbper token when logprobs= True

[SStarRoad]

Is it possible to return top N logprobs? Currently I see only 1 logrpobbper token when logprobs= True

same request. 🙏