Key Info
A maintainer of OpenCode argues that benchmarks are meaningless at this point and that anyone publishing benchmark results should reach out to the products being tested beforehand. They say a lot was set up incorrectly in an earlier evaluation, including reasoning being dropped, yet the result was reposted widely; after the fix, OpenCode went from last to first.
Highlights
- The maintainer says the original benchmark setup was incorrect, with reasoning dropped from the evaluation.
- Despite the error, the flawed result was reposted across many places.
- After the setup was fixed, OpenCode moved from last place to first.
- The maintainer urges benchmarkers to contact the products they evaluate before publishing.
- The quoted post notes OpenCode 2 far outscores OpenCode 1.19, going from 8th of 9 harnesses to tied for first across all three models on the Hugging Face Harness Leaderboard.