核心信息
腾讯AI的FlexKV针对长上下文推理中的瓶颈：即便KV缓存命中，从GPU外部读取缓存仍可能让GPU闲置。FlexKV采用逐层恢复方式，让计算与数据加载重叠进行。
要点
- FlexKV按层恢复KV缓存：前层计算的同时后层加载数据，减少GPU等待时间。
- 预取机制提前启动缓存传输，异步写回则让缓存I/O与推理过程重叠。
- 该工作聚焦于提升长上下文场景下“缓存命中”的实际收益，应对KV缓存规模增长带来的挑战。
FlexKV：腾讯AI加速长上下文缓存命中的新方法
原始推文
A cache is only worth what it hits. The community's having a KV cache moment. Here's the corner we work in: In long-context serving, a cache hit can still leave the GPU waiting for data. When KV lives outside GPU memory, how you bring it back matters. That's the problem we set out to solve. FlexKV restores it layer by layer: earlier layers compute while later layers load. Prefetching starts transfers early, while asynchronous writeback helps overlap cache I/O with inference. Making those hits faster goes hand in hand with making more of them possible. FlexKV compresses KV losslessly, expands cache capacity with CPU RAM, SSDs, and remote storage, reuses prefixes across the cluster, and routes requests to wherever the cache already lives. It sits under your inference engine, so there’s nothing to rewire. Works across SGLang, vLLM, TensorRT-LLM, and Dynamo. Up to 70% lower TTFT, +16% QPM. https://t.co/Rvhgh7TqqF