Comparison between relu and SIGMOD

relu The advantages of ：

First of all , Prevent gradient dispersion ,sigmod The function is only in 0 There is good activation nearby , When approaching the positive and negative saturation region , Too slow to change , Derivative tends to 0, Cause gradient dispersion , and relu The function is greater than 0 Part of the gradient is constant , Therefore, there will be no gradient dispersion phenomenon ;

second , sparsity ,Relu It's going to make the output of some of the neurons zero 0, This results in the sparsity of the network , The problem of overfitting is alleviated ,relu The derivative of the function in the negative half region is 0 , So once the neuron activation value enters the negative half region , Then the gradient will be 0, In other words, this neuron will not undergo training ;

Third , Speed up the calculation ,relu The derivative of a function is calculated faster , Program implementation is a if-else sentence , and sigmoid The function performs four floating-point operations .