65% Pass Rate, 25% Reliability: The Agent Benchmark Gap No One Measured

Most agent benchmarks measure whether a model can find a correct trajectory. Thinkingbox measures whether it can find one consistently. That distinction collapses the apparent progress in agent capability.

The core problem is structural. Completing a business workflow is not the same as producing a valid tool call or a plausible response. An agent handling a hotel cancellation or an insurance claim must gather missing information across multiple conversational turns, respect domain-specific policies that constrain what actions are even legal, coordinate tools whose outputs depend on each other, and arrive at a final backend state that is exactly right, with no missing effects and no collateral ones. Current benchmarks, built around code repair, web navigation, or API function calling, do not test this combination. They test whether the model can find any path to a correct answer. Thinkingbox tests whether it finds the right path, every time, in a stateful world where wrong intermediate actions leave permanent traces.

The sandbox works by isolating each agent session in an MCP-compatible environment that records complete execution traces and evaluates outcomes against terminal backend state, not against model outputs. Think of it as the difference between grading an exam by reading the student's reasoning versus checking whether the answer in the gradebook is correct after the test is turned in. Each of the 507 workflows in Thinkingbox-bench is conditioned on domain policies, spanning retail, hospitality, auto insurance, neobank internal IT, and consulting IT and HR support. Task-specific executable checks accept any valid trajectory that reaches the correct final state, but reject wrong effects, missing effects, and extra effects equally. Some tasks additionally verify properties of the final response to the user. The benchmark does not reward partial credit for clean-looking failures.

The numbers expose the gap directly. The strongest model across both proprietary and open-weight evaluations reaches 65.36% pass@1. That same model scores 25.25% on pass^20, meaning it produces a reliable, repeatable correct trajectory only one in four attempts across 20 tries. The failure mode is not obvious errors. Many failed trials show clean termination and valid state-changing actions, which means response-level and tool-call-level signals are not reliable proxies for whether the task actually completed correctly. For teams shipping agents into production business workflows, the takeaway is direct: a model that passes your evaluation suite once is not a model you can trust to pass it the next time a real user triggers the same task.

We're thinking: We keep seeing the same pattern: benchmark scores improve, deployment confidence rises, and then production reveals a different story. Thinkingbox makes the mechanism of that gap explicit. The 40-percentage-point spread between pass@1 and pass^20 is not a quirk of these specific tasks. It is what happens when evaluation finally measures the thing that matters in production: not peak performance, but consistent performance under policy constraints, across turns, with persistent state. The more pointed implication is that teams using pass@1 or single-attempt evals to gate agent deployments are measuring something close to best-case luck. Any agent system touching real business data, where a wrong state transition is not easily undone, needs reliability metrics, not capability ceilings.

Key takeaways:

• Thinkingbox evaluates agents against terminal backend state across full multi-turn sessions, with executable checks that reject wrong, missing, or extra effects, separating trajectory plausibility from actual task completion.
• The strongest evaluated model scores 65.36% pass@1 but only 25.25% pass^20 across 507 policy-conditioned workflows; many failures show clean termination, making tool-call-level signals an unreliable proxy for success.
• Teams deploying agents into stateful business workflows should replace or supplement single-attempt evals with repeated-trial reliability metrics before treating benchmark scores as deployment readiness signals.

Source: Thinkingbox: Sandbox and Benchmark for Agents in Stateful Business Workflows