RLVR Trains One Skill by Erasing the Trajectories Needed for the Next

Training a model to follow instructions makes it worse at finding correct math solutions later. Not because the model forgets math. Because on-policy RL with verifiable rewards physically removes the trajectory variety needed to learn the next thing.

The standard assumption in sequential RLVR training is that each stage improves the model and hands a better base to the next stage. What this work shows instead: each stage reshapes which response openings the model samples, and that reshaping can starve the next verifier of the diverse trajectories it needs to assign credit. The failure is silent. Endpoint metrics improve. The curriculum appears to be working. The damage is only visible when you measure sampling coverage rather than single-shot accuracy.

The mechanism is concentrated in the first few tokens of each response. On-policy RLVR does not teach the model new solution strategies. It reranks the opening patterns already present in the base policy, promoting the ones that score well under the current verifier and suppressing everything else. Because autoregressive generation is path-dependent, an opening token or phrase causally constrains the rest of the response. Math-RLVR promotes step-by-step openings; IF-RLVR shifts responses toward direct answers. When IF-RLVR runs first, it compresses the opening distribution toward direct-answer starts, and those starts are structurally worse entry points for the kind of exploratory, multi-step reasoning that Math-RLVR needs to find and reinforce. The model still contains the capability. It just can't reach it within a fixed rollout budget.

Controlled opening interventions confirm this causality: forcing a step-by-step opening on a model trained with IF-RLVR recovers math searchability. The opening is not a stylistic detail. It is the branching point that determines whether successful trajectories are reachable at all.

The numbers make the divergence concrete. On IFEval with Qwen3-8B-Base, Math-RLVR lifts pass@1 by 6.5 percentage points and drops best@32 by 9.8 percentage points. The same pattern holds across both model families tested and across multiple instruction-following benchmarks. Reward variation for downstream Math-RLVR also falls after IF-RLVR, meaning the later training stage has less signal to work with. Tested mitigations, including reference-policy constraints, routing priors, and on-policy distillation, preserve cross-task support only partially. MathIF and ReasonIF results show that marginal endpoint gains translate only partly into responses that satisfy both correctness and constraint-following simultaneously. For teams building multi-stage RLVR curricula, the takeaway is direct: a rising pass@1 at each stage is not evidence that the next stage will train successfully.

We're thinking: We find the framing here more alarming than the numbers alone suggest. The failure mode is not that RLVR plateaus or overfits in a detectable way. It is that RLVR actively prunes the reachable trajectory set, and it does so while every logged metric looks healthy. A team running a sequential math-then-instruction or instruction-then-math curriculum has no obvious loss signal to catch this. The practical implication is that best@k across sampling budgets, not just pass@1, should be a required diagnostic in any RLVR training run, checked after each stage before the next begins. The deeper issue is that on-policy optimization is not a neutral transformation of the model. It is a distribution edit, and distribution edits compound.

Key takeaways:

• On-policy RLVR reshapes response-opening distributions rather than building new capabilities, and because openings causally constrain full trajectories, each training stage can eliminate the sampling support the next stage depends on.
• Math-RLVR on Qwen3-8B-Base raises pass@1 by 6.5 points while dropping best@32 by 9.8 points; the divergence holds across two model families and multiple benchmarks, and tested mitigations (reference-policy constraints, routing priors, distillation) only partially recover cross-task coverage.
• Teams running sequential RLVR curricula should instrument best@k at multiple sampling budgets after every training stage, not just pass@1, before proceeding to the next objective.

Source: Verifier-Induced Support Reshaping in On-Policy Optimization