Qwen Launches RecreationBench: A Hybrid Benchmark for Computer-Use Agents

Qwen Developers 53 min ago

Key Info

Qwen unveiled RecreationBench, a benchmark for hybrid computer-use agents with 250 application-recreation tasks spanning Ubuntu, macOS, Windows, Android, and Web, covering domains like productivity, development, graphics, multimedia, and science. Agents must explore a running reference app, recreate it in code, and pass both programmatic tests and VLM-based visual evaluation.

Highlights

RecreationBench goes beyond GUI-only or terminal-only benchmarks by forcing agents to switch between using applications and writing code.
The 250 tasks span five major platforms and multiple domains, making it a broad testbed for real-world agent capability.
Evaluation combines automated programmatic tests with VLM-based visual checks for deeper verification.
The playground and dataset are already available for researchers and developers to build on.

Original post

Coding an app is one thing. Using one is another. A real computer-use agent should be able to do both, and know when to switch between them. That’s what we’re exploring with RecreationWorld

Quoted post

Qwen introduces RecreationBench, a benchmark for Hybrid Computer-Use Agents with 250 application-recreation tasks across Ubuntu, macOS, Windows, Android, and Web, spanning domains such as productivity, development, graphics, multimedia, and science. Unlike GUI-only or terminal-only benchmarks, agents must explore a running reference app, recreate it in code, and pass both programmatic tests and VLM-based visual evaluation. The playground is ready. Let’s build! 🚀Dataset: https://t.co/mcJSwoEHnX

#RecreationBench#computer-use agents#AI benchmark

View original on X