Key Info

LiteLLM now supports cache-aware routing for self-hosted models running on Kubernetes. The router keeps each agent session pinned to the same GPU pod so vLLM can reuse its prompt cache.

Highlights

  • Cache-aware routing targets self-hosted model deployments on Kubernetes
  • Each agent session stays on the same GPU pod
  • Pod affinity lets vLLM reuse its prompt cache across requests
  • A blog post with details is linked