Key Info
Requesty's gateway now uses latency-based routing that sends each new session to the fastest provider for a given model, ranked by time to first token and throughput and corrected for errors. Sessions stay pinned so caches remain warm.
Highlights
- Fastest provider for a model can change throughout the day, so routing is dynamic per new session.
- Ranking is based on time to first token and throughput, with error correction.
- Sessions are pinned to a provider to keep the cache warm.
- Aimed at teams running AI in production where latency and cache efficiency matter.