Gradient clip in dqn

First look at this https://stackoverflow.com/questions/36462962/loss-clipping-in-tensor-flow-on-deepminds-dqn

DQN As mentioned in the article clip Not at all gradient clip.

Let's look at it first tensorflow 1 Medium huber_loss, Make d = 1.

  0.5 * x^2                  if |x| <= d
  0.5 * d^2 + d * (|x| - d)  if |x| > d

The derivative is

f'(x) = x    if x in [-1, 1]
f'(x) = +1    if x > +1
f'(x) = -1    if x < -1

$$\begin{gathered} loss=f(y_i-\widehat{y_i}) \\ los{s}^{\prime }=f^{\prime }(y_i-\widehat{y_i})\cdot y_i^{\prime }=f^{\prime }(z)\cdot y_i^{\prime } \end{gathered}$$

DQN Mentioned in the text

We also found it helpful to clip the error term from the update to be between −1 and 1. Because the absolute value loss function |x| has a derivative of −1 for all negative values of x and a derivative of 1 for all positive values of x, clipping the squared error to be between −1 and 1 corresponds to using an absolute value loss function for errors outside of the (−1,1) interval. This form of error clipping further improved the stability of the algorithm.

But this means $f^{\prime }(z)$ be in -1 and 1 Between . And the gradient clip Then it is $f^{\prime }(z)\cdot y_i^{\prime }$ The whole is in -1 To 1 Between .