Add automatic reuse of common key value cache blocks between requests #620
Description
Imagine two requests sharing same prefix comes to the trt-llm generation runtime, it would be awesome if those requests can share key value cache blocks between each other.
During generation with paged attention in fact each block corresponds to particular prefix of the input, so all of the blocks can be made prefix addressable (i.e. it should be possible to keep a map from prefix to block, or from additional prefix and previous block to a new block). This way automatic reuse of blocks between generation requests with shared prefixes will be done "out of box".
This potentially can be beneficial for speculative execution as well.