PPO's Clipping Creates a Derivative Cliff. OR Removes It.

The standard assumption in LLM post-training is that PPO's clipped surrogate objective is a stable, well-understood baseline. The clipping is treated as a safety mechanism, a trust region that keeps policy updates bounded. What it also creates, quietly, is an abrupt discontinuity in the objective's derivative at the clip boundary, a hard edge where gradient information simply stops.

OR (Output Reset) replaces that hard edge with a squared-margin loss applied in rollout-relative token log-ratio space. Instead of cutting the gradient off at a fixed clip boundary, OR uses a smooth one-sided saturation rule: tokens that have already crossed the favorable margin contribute zero direct OR residual, but the transition is continuous rather than abrupt. The advantage sign still determines update direction. The structural difference is that the loss surface no longer has a kink where the optimizer has to guess what to do next. Think of it as replacing a step function with a ramp: the information content at the boundary is preserved rather than discarded.

The comparison runs on Llama-3.2-1B-Instruct with the Anthropic hh-rlhf dataset, one shared reward model, and three seeds per method. That is a small-scale controlled experiment, not a production sweep, and the results reflect training-time reward-model scores, not held-out human-preference evaluations. Under generalized advantage estimation (GAE), PPO-OR posts a mean final reward-model score 0.305 higher than PPO-clip. Under group-relative advantages, GRPO-OR does not beat GRPO on mean score at G=2, but shows tighter across-seed spread, near-zero terminal OR residual, and a declining overshoot fraction, while the matched GRPO trace stays variable. For teams running LLM RLHF pipelines, the takeaway is direct: the clipping mechanism you inherited from proximal policy optimization carries an optimization artifact that a smooth alternative measurably reduces, at least under GAE.

We're thinking: We find the GAE vs. group-relative divergence more telling than the headline number. OR lifts mean reward under GAE but fails to translate that into a reward gain under GRPO at G=2. The paper flags group size as a potential explanation and leaves it open. That asymmetry suggests the derivative discontinuity matters most when advantage estimates are higher quality, which is exactly what GAE provides over two-sample group-relative estimation. If that read is correct, OR's practical ceiling depends on your advantage estimation regime, not just your clipping rule, and teams running GRPO with small groups may need to fix the advantage estimator before the loss surface change pays off.

Key takeaways:

• OR replaces PPO/GRPO's hard clip boundary with a smooth squared-margin loss in token log-ratio space, eliminating the derivative discontinuity at the trust region edge.
• PPO-OR beats PPO-clip by 0.305 mean reward-model score under GAE across three seeds; GRPO-OR shows no mean gain at G=2 but tighter variance, with results limited to training-time reward-model measurements on a 1B model.
• Teams running PPO-style RLHF with GAE should test OR as a drop-in clip replacement; teams using GRPO should treat G=2 results as inconclusive and wait for larger-group evaluations before switching.

Source: OR Else: A Differentiable Trust Region for Policy Optimization