Key Info
LiteLLM Lens CLI lets developers compare models across different agent harnesses (Claude Agent SDK, OpenAI Agents SDK, LangGraph, or custom agents) from the terminal, running the same tasks and producing comparative metrics.
Highlights
- Specify which model × harness combinations to evaluate via your local coding agent
- Compare pass rates by model and harness
- Track total cost and cost per successful task, plus cost vs. quality
- Measure latency and get failure breakdowns (wrong answers, tool misuse, errors)
- Replaces manual eval loops and spreadsheet-based result collection