Fixed Reward Weights Keep Training on Already-Solved Problems

The standard recipe for multi-reward post-training collapses multiple reward signals into a single weighted sum before computing group-relative advantages. That ordering sounds harmless. It is not.

When you scalarize first, rollouts with completely different reward profiles can land on the same advantage score. A run that aces formatting but fails at correctness and a run that does the opposite can receive identical gradient signals. Worse, fixed relative weights mean the optimization budget stays locked to its original allocation even after one objective is effectively solved. Training keeps pushing on a reward the model has already saturated, while the harder objective starves.

SA-MRPO (Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization) fixes the ordering problem at the source. Instead of scalarizing rewards before standardization, it standardizes each reward objective independently, then discounts each objective's contribution according to a batch-level saturation estimate. When an objective's variance collapses, indicating the model has largely solved it, its weight in the combined advantage shrinks. Gradient budget flows toward objectives that still have headroom. The analogy is a study schedule that drops topics you have mastered and doubles down on the ones where you are still making errors, recalculated every batch.

One finding worth pausing on: saturation-aware reweighting can reverse the sign of an update, not just rescale its magnitude. That is a qualitatively different kind of correction. A fixed-weight baseline that is pulling the policy in the wrong direction on a saturated objective cannot self-correct. SA-MRPO can.

Across 15 benchmark comparisons on mathematical reasoning with two- and three-objective reward setups, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 cases, with gains reaching 5% on AIME24. On adaptive reasoning tasks, accuracy improves across all five benchmarks, averaging 3.8% and peaking at 9.2% on AMC23. Coding pass rates improve by up to 2.3%. Crucially, the easier, already-satisfied objectives hold near their prior levels throughout. For teams running multi-objective RLHF on reasoning or coding tasks, the takeaway is direct: if your reward mix includes one hard objective and one easier one, fixed scalarization is likely suppressing signal on the objective that matters most.

We're thinking: We find the sign-reversal result more consequential than the headline accuracy numbers. The accuracy gains on AIME24 and AMC23 are real and worth having, but they are recoverable through other means. A gradient update that points in the wrong direction because a saturated objective is dominating the advantage signal is a structural failure that fixed-weight scalarization cannot diagnose or correct by design. SA-MRPO exposes that this failure mode is common enough to matter across 15 benchmarks, which suggests most multi-reward RLHF pipelines in production today are systematically misallocating compute during training without any visible signal that it is happening. The saturation estimate is cheap to compute and the method slots into existing GRPO-style training without architectural changes, which makes the cost of not adopting it hard to justify once you have read the ablations.

Key takeaways:

• SA-MRPO standardizes each reward objective independently and applies a batch-level saturation discount, dynamically reallocating gradient budget toward under-optimized objectives rather than those already near ceiling.
• Across 15 math reasoning benchmark comparisons it beats GDPO on the harder correctness objective in 12 of 15 cases, with up to 5% on AIME24 and 9.2% on AMC23; the caveat is that evaluations focus on reasoning and coding, so generalization to RLHF reward mixes involving style or safety objectives remains to be tested.
• Teams training language models with two or more reward objectives, especially where one objective (format, length) saturates faster than another (correctness, reasoning), should replace fixed-weight scalarization with per-objective standardization and saturation discounting before the next training run.

Source: SA-MRPO: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization