Key Info
An experiment attempted to optimize the Pi harness for the Qwen model series by tweaking various harness components, and found that despite early gains on ALE (Agents Last Exam), results on Terminal-Bench Lite showed little improvement.
Highlights
- The Pi harness was tuned component-by-component for Qwen models
- Early findings showed improvement on ALE (Agents Last Exam)
- Terminal-Bench Lite showed little to no improvement
- The author questions why the two benchmarks diverge
- Experiment setup and findings are shared in a thread