Key Info

LiteLLM announced Lens Profiles, a feature that turns production traffic into task-level model evaluations by automatically discovering what customers use agents for and attributing gateway-recorded spend to those tasks.

Highlights

  • Automatically discovers agent tasks without manually labeling every run
  • Attributes gateway-recorded spend to specific tasks to surface cost drivers
  • Compares model deployments by request volume, average cost, and p95 latency per task
  • Builds evals from task categories and real production runs against actual customer workloads
  • Implementation PRs are referenced for gateway cost attribution