A Stronger Model Can Teach a Weaker One at Test Time, No Retraining Required

The standard assumption in model improvement is that capability flows through weights. You distill, you fine-tune, you update parameters, and the smaller model gets better. This paper breaks that assumption directly: a stronger model can transfer cognitive structure to a weaker one entirely at inference time, with no parameter updates on either side.

The mechanism is inference-time harness design. A "builder" model, given 5% of a benchmark's data as a validation set, iteratively constructs a harness: a scaffold of deterministic code, task-specific routing logic, and strict answer-format enforcement that wraps around the weaker "target" model at inference time. The builder refines this harness across multiple rounds, testing each version against the validation split, before the final harness runs on the full test set. Think of it as the stronger model writing a structured exam protocol that the weaker model follows, rather than the weaker model being retrained to internalize new skills.

The design insight is where the gains actually come from, and it is not what most practitioners would guess. The harness does not prompt the target model to reason more, sample more broadly, or chain more steps together. The gains come from offloading unstable model reasoning into deterministic code, from routing different subtask types to appropriate handling paths, and from enforcing output formats that prevent the target model from producing ambiguous or malformed answers. The weaker model's reasoning is constrained, not expanded.

Across four Theory-of-Mind benchmarks, average target-model performance lifts from 0.49 to 0.91. That is nearly a doubling. The analysis also finds that builder-model reasoning effort improves harness quality monotonically, that platform effects are modest compared to the builder's own capability level, and that weaker target models receive the largest absolute gains. For teams deploying smaller models in production, the takeaway is direct: if you have API access to a strong frontier model, you may be able to close much of the capability gap on structured tasks through harness engineering alone, before ever touching a fine-tuning budget.

We're thinking: We find the cost implication here more significant than the accuracy numbers. If harness design by a strong builder model can nearly double a weak target model's performance on structured reasoning tasks, the decision calculus for small-model deployment shifts. The question is no longer only "can we afford to fine-tune a capable model" but "can a frontier model write the scaffolding that makes a cheap model good enough." The catch is scope: Theory-of-Mind benchmarks are structured, bounded, and amenable to deterministic routing in ways that open-ended generation tasks are not. Teams should treat this as a strong signal for task-specific deployments, not a general-purpose distillation replacement.

Key takeaways:

• Inference-time harnesses built by a strong model transfer capability to a weaker model by replacing unstable model reasoning with deterministic code, task routing, and format enforcement, with no weight updates on either side.
• Average target-model performance lifts from 0.49 to 0.91 across four Theory-of-Mind benchmarks; gains are largest for the weakest target models, but results are scoped to structured reasoning tasks where deterministic scaffolding is tractable.
• Teams deploying smaller models on well-defined, structured tasks should prototype harness-based scaffolding using a strong builder model before committing fine-tuning resources, particularly where task routing and output format control are feasible.

Source: AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses