Weak Teachers Can Improve Stronger Students: Frontier Distillation Without a Bigger Model

Standard on-policy distillation assumes a stronger teacher exists. At the frontier, that assumption collapses: there is no larger model to pull from, no consolidated expert that outperforms the student on the target task. The field has treated this as a hard ceiling. It is not.

Weak-to-Strong On-Policy Distillation (W2S-OPD) sidesteps the ceiling by never asking a weak model to directly teach a stronger one. Instead, it asks two weak models to expose a direction. The method constructs a proxy teacher entirely in logit space by taking the difference between a "positive" model (one that has the capability) and a "negative" model (one that lacks it), then adding that logit difference to the student's own base model. The result is a proxy teacher that points toward the target capability while staying distributionally close enough to the student that on-policy distillation remains stable.

Three contrast pair configurations make this concrete. First, a post-RL expert versus its pre-RL initialization isolates exactly what reinforcement learning instilled, stripped of everything else. Second, a larger base model versus a smaller one extracts the signal from scale without requiring the student to distill the larger model directly. Third, a small base model given a correct solution hint versus a wrong one isolates the instance-level direction toward a correct answer. Each contrast pair is cheap to obtain, smaller than the student, and targets a distinct capability axis. The student then minimizes per-token reverse KL against the proxy teacher on its own rollouts, keeping the training on-policy throughout.

Across four math benchmarks and three code benchmarks, W2S-OPD outperforms standard on-policy distillation and, in several settings, pushes the student past the domain teacher entirely. Analysis of the three contrast types reveals that the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes procedural solving steps. These are complementary signals, not redundant ones. For teams training at the capability edge where no stronger teacher is available, the takeaway is direct: capability direction, not capability magnitude, is what distillation actually needs to transfer.

We're thinking: The deeper implication here is about what "teacher" means in distillation. We have been conflating two things: a model that produces better outputs, and a model that points in a better direction. W2S-OPD shows these are separable. A pair of weak models that bracket a capability can encode a more precise directional signal than a single strong model whose logits carry noise from every capability it has, not just the one you want. The practical consequence is that teams training specialist frontier models can now construct targeted proxy teachers from small, cheap contrast pairs rather than waiting for a larger general model to exist. That changes the economics of frontier training in a specific, actionable way.

Key takeaways:

• W2S-OPD constructs a proxy teacher from the logit difference of a positive and negative weak model pair, adding that capability direction to the student's base model rather than distilling any single teacher directly.
• Across seven benchmarks (four math, three code), W2S-OPD outperforms standard on-policy distillation and enables the student to surpass its domain teacher; the method holds even when every supervision source is weaker than the student, though benchmark coverage is currently limited to math and code.
• Teams training at the capability frontier, where no stronger teacher model exists, should evaluate W2S-OPD's contrast pair construction as a first step before defaulting to self-improvement or synthetic data pipelines.

Source: Weak-to-Strong On-Policy Distillation