PPO · clipped surrogate objective

Where does PPO’s clip switch the gradient off?

Take one sample with advantage \(A\). Set the clip width \(\varepsilon\), move the probability ratio \(r\), and watch which term the \(\min\) picks.

$$L^{\text{CLIP}}(r)=\min\!\Big(r\,A,\ \operatorname{clip}(r,\,1-\varepsilon,\,1+\varepsilon)\,A\Big)$$
$$r=\frac{\pi_\theta(a\mid s)}{\pi_{\theta_\text{old}}(a\mid s)}$$
\(L^{\text{CLIP}}\) (the min) \(rA\) \(\operatorname{clip}(r,1-\varepsilon,1+\varepsilon)\,A\) zero gradient

Slope of the objective, \(\partial L^{\text{CLIP}}/\partial r\)

Drag on either plot to move \(r\).

\(rA\)min
\(\operatorname{clip}(r,1-\varepsilon,1+\varepsilon)\,A\)min
\(L^{\text{CLIP}}\)
\(\partial L^{\text{CLIP}}/\partial r\)

The whole \((r, A)\) plane

Arrows: the direction gradient ascent moves \(r\), length \(\propto |A|\). Red: zero gradient. Drag to move the point.

All six cases

\(r<1-\varepsilon\)\(1-\varepsilon\le r\le 1+\varepsilon\)\(r>1+\varepsilon\)
\(A>0\) \(rA\)slope \(A\): push \(r\) up \(rA\)slope \(A\): push \(r\) up \((1+\varepsilon)A\)slope 0
\(A<0\) \((1-\varepsilon)A\)slope 0 \(rA\)slope \(A\): push \(r\) down \(rA\)slope \(A\): push \(r\) down

The current state is outlined. Each sign of \(A\) has exactly one flat side: the side the advantage is asking \(r\) to move toward.

Try this

Why it works this way

The \(\min\) always takes the more pessimistic of the two terms. The clipped term is constant outside \([1-\varepsilon,1+\varepsilon]\), so it has slope 0 there.

If \(r\) has moved past the clip edge in the direction \(A\) favors, the clipped term is the smaller one. The \(\min\) picks it, and the slope is 0. If \(r\) has moved past the edge the other way, \(rA\) is the smaller one. The \(\min\) picks it, and the full slope \(A\) remains. So the clip removes the incentive to go further, but it never stops the correction of a bad move.

The gradient with respect to the policy parameters goes through \(r\):

$$\nabla_\theta L^{\text{CLIP}}=\frac{\partial L^{\text{CLIP}}}{\partial r}\;r\,\nabla_\theta\log\pi_\theta(a\mid s)$$

Here \(\partial L^{\text{CLIP}}/\partial r\) is \(A\) when the \(rA\) term is active. It is \(0\) in exactly two cases: \(A>0\) with \(r>1+\varepsilon\), and \(A<0\) with \(r<1-\varepsilon\). In those cases this sample adds nothing to the update.

At the start of each PPO update, \(\theta=\theta_\text{old}\), so \(r=1\) for every sample and the gradient equals the plain policy gradient \(A\,\nabla_\theta\log\pi_\theta\). The clip only acts in later epochs and minibatches, after \(r\) has drifted. Samples that reach the flat region then drop out of the gradient; the rest keep training.

At exactly \(r=1\pm\varepsilon\) the two terms tie, and the slope is not defined. This is a single point; autodiff libraries return one side or the other.