Key Info
A developer trained a 380M specialist decision model for curating web-scale data, fine-tuned from UltraBERT-380M, and released the model, training data and full recipe.
Highlights
- The small 380M model runs cheaply at scale: 1M FineWeb pages took 19 minutes and about $5 on Hugging Face Jobs
- It classified roughly a quarter of those pages as product pages
- 6% passed a simple "educational, not spam" filter, and 1 page in 1,020,000 was rated "outstanding"
- The model, training data (with source, labeller and licence per row) and full recipe are shared openly