Key Info

LiteLLM announced the launch of agent evals in the LiteLLM SDK, enabling users to compare models and harnesses on their own tasks using litellm.agent_eval().

Highlights

  • New capability: agent evals built into the LiteLLM SDK
  • Users can compare different models on their own custom tasks
  • Users can also compare harnesses, not just models
  • Invoked through the litellm.agent_eval() function