[Feature Request] Mixtral Offloading

[shixianc]

There's a new cache technique mentioned in the paper https://arxiv.org/abs/2312.17238. (github: https://github.com/dvmazur/mixtral-offloading)
They introduced LRU cache to cache experts based on patterns they found, and also took speculative guess to pre-load experts before the computation of the next layer. The result looks quite promising. Can we support it for Mixtral? This helps a lot to run on smaller GPUs.

[ncomly-nvidia]

Thanks for highlighting this - that's a very good suggestion to save memory. We'll evaluate what it would take to support it in TRT-LLM

[shiqingzhangCSU]

May also consider MoE-Infinity, a cost-efficient mixture-of-expert (MoE) serving system that realizes activation-aware expert offloading. :)

[karljang]

@shixianc Thank you for this feature request on Mixtral offloading!

I'm sorry for the delayed response, but there have been updates in terms of MoE model supports. TensorRT-LLM now has extensive MoE support with significant optimizations that have been developed since you opened this issue.

Could you please try the current MoE implementation and let us know if it meets your offloading needs? The examples in /examples/models/core/mixtral/ provide comprehensive guidance for deployment.

[shixianc]

@karljang thanks for the reply -
A lot of things are invented and things have been changing these 2 years for MoEs. Guess people are focusing on multinode EP instead of getting it offloaded to CPU lol. Feel free to close it.

[karljang]

Thanks for your comment!