Key Info

Ahmad Awais explains that DeepSeek's poor tool calling was due to a harness problem, not the model. By adding a tool-input repair layer to their open-source CLI CommandCodeAI, DeepSeek v4 Pro beat Opus 4.7 on internal evals 6/10 times.

Highlights

  • Identified that tool calling failures often stem from the harness, not the model itself.
  • Implemented a repair layer after observing DeepSeek-flash fail on simple tasks due to unreadable error formats.
  • DeepSeek v4 Pro outperformed Opus 4.7 in 6 out of 10 internal evaluation runs.