Key Info
Airbnb Engineering benchmarked Gemma 4 E2B for policy classification, serving it on a single NVIDIA RTX Pro 6000 GPU and reporting quality that matches top frontier APIs at a fraction of the cost.
Highlights
- Max F1 rose from 0.563 to 0.902, matching top frontier API quality
- Median latency of 35ms (85ms p99) versus roughly 900ms for frontier models
- Inference cost about 1% of frontier API spend at high volume
- Gains attributed to careful evals, domain data, and choosing a specialized small model over a massive prompt