Key Info

Airbnb Engineering benchmarked Gemma 4 E2B for policy classification, serving it on a single NVIDIA RTX Pro 6000 GPU and reporting quality that matches top frontier APIs at a fraction of the cost.

Highlights

  • Max F1 rose from 0.563 to 0.902, matching top frontier API quality
  • Median latency of 35ms (85ms p99) versus roughly 900ms for frontier models
  • Inference cost about 1% of frontier API spend at high volume
  • Gains attributed to careful evals, domain data, and choosing a specialized small model over a massive prompt