The Scaffold Beats the Model: Benchmarking LLMs at Harness Optimization

Most teams chasing capability gains treat the model as the variable. Swap in a stronger checkpoint, rerun evals, ship. HarnessOpt-Bench exposes the flaw in that instinct: across 111 scored runs, the choice of optimizer model separates performance far more than the coding harness it acts through, and gains from improving the scaffold around a fixed model routinely exceed what weight upgrades deliver.

The harness is everything that is not the weights: the system prompt, the tool definitions, the control flow, the memory scheme, the orchestration code. In production, these components are where most of the performance is actually won or lost. Yet until now there was no common protocol for measuring how well a frontier LLM can improve this scaffold automatically. HarnessOpt-Bench fills that gap by treating harness optimization as a first-class, measurable capability. An optimizer, itself an LLM paired with a coding harness, receives a seed harness for a target agent, graded evaluation feedback, and a fixed budget of target-evaluation calls. It edits the harness iteratively, nominates a final candidate, and that candidate is scored on a held-out test partition the optimizer never sees during search. A trusted execution environment enforces the evaluation boundary, meters resource use, and preserves candidate versions for audit. The design mirrors real deployment constraints: evaluations are expensive, stochastic, and finite.

Four downstream tasks, five frontier LLMs tested both under a shared coding harness and under their native harnesses, 111 scored runs total. Three findings stand out. First, optimizer models separate more than the harnesses they act through, meaning the model's capacity to reason about and edit orchestration code is the binding constraint, not which scaffolding style it uses. Second, native harnesses are not consistently superior to the shared coding harness, which collapses the assumption that each model's proprietary scaffolding gives it a structural edge. Third, gains vary substantially across tasks and seed regimes, confirming that harness optimization is not a uniform lift but a task-sensitive capability with large headroom for improvement. For ML engineers and agent platform teams, the takeaway is direct: before buying a more expensive model tier, audit whether the harness around your current model has been systematically optimized, because that is where the benchmark shows the larger variance lives.

We're thinking: We find the native-harness result particularly telling. The assumption that a model's proprietary scaffolding is tuned to extract its best performance turns out to be empirically shaky. That has a direct implication for teams evaluating frontier models: benchmarks that test models under their native harnesses may be measuring harness quality as much as model quality, and the two are now separable. The more consequential shift is organizational. If harness optimization is a measurable, discriminative capability, then the team that builds the best automated harness optimizer, not the team with the largest model budget, may hold the durable production advantage. That reframes the build-vs-buy calculus for anyone running agents at scale.

Key takeaways:

• Harness optimization, the automated iterative improvement of prompts, tools, control flow, and orchestration code around a fixed model, is now a formally benchmarked capability with a held-out evaluation protocol that prevents score inflation during search.
• Across 111 scored runs on 4 tasks and 5 frontier LLMs, optimizer model identity drives more performance separation than harness style, and native harnesses show no consistent superiority over a shared coding harness; gains vary enough across tasks that no single optimizer dominates all regimes.
• Teams running agentic systems should treat harness optimization as a distinct engineering investment: measure normalized gain over a seed harness on held-out data before concluding that a model upgrade is the right next step.

Source: HarnessOpt-Bench: Evaluating LLMs at Harness Optimization