Key Info
LiteLLM announced the launch of agent evals in the LiteLLM SDK, enabling users to compare models and harnesses on their own tasks using litellm.agent_eval().
Highlights
- New capability: agent evals built into the LiteLLM SDK
- Users can compare different models on their own custom tasks
- Users can also compare harnesses, not just models
- Invoked through the litellm.agent_eval() function