Activate function

Why? tanh Than Sigmoid Convergence is faster ？

1. Zuo Wei Sigmoid：
Sigmoid Is a commonly used nonlinear activation function , His mathematical form is as follows ：
Picture description here
It can put the continuous real value of input ” Compress “ To 0-1 Between .
Special , If it's a very large negative number , So the output is 0; If it's a very large positive number , Output is 1.
Sigmoid Functions have been used a lot , But now , Fewer and fewer people use it . Mainly because of some of its shortcoming ：
（1）Sigmoids saturate and kill gradients, This is the gradient vanishing problem we often mention .sigmoid There is a very fatal flaw , When the input is very large or very small （saturation）, The gradient of these neurons is close to 0 Of . If your initial value is very large , Most neurons may be in saturation And put gradient kill fall , This will make the Internet difficult to learn .
（2）Sigmoid Of output No 0 mean value . This is not desirable , Because this will cause the neurons in the latter layer to get the non output of the upper layer 0 Mean signal as input .

2. Right for tanh;
tanh yes Sigmoid Deformation of ：
Picture description here
And Sigmoid The difference is ,tanh yes 0 Mean . therefore , Practical application ,tanh than Sigmoid Better .

The corresponding derivative ：
Picture description here
Picture description here
You know , The range of is （0,1）
The range of is （0,1/4）.
Sum up ,tanh(x) Gradient vanishing problem ratio Sigmoid Be light , So convergence should be fast .
 ————————————————  
Copyright notice ： This paper is about CSDN Blogger 「Peanut_ Fan 」 The original article of , follow CC 4.0 by-sa Copyright agreement , For reprint, please attach the original source link and this statement .
Link to the original text ：https://blog.csdn.net/u013841196/article/details/80473654