当前位置：网站首页>Machine learning notes temperature+softmax

Machine learning notes temperature+softmax

2022-06-28 19:23:00 【UQI-LIUWJ】

1 Introduce

belt temperature Of Softmax, Describe... With a formula , It can be expressed as

$S_\tau(x,y)=[\frac{exp(a_1/\tau)}{\sum exp(a_i/\tau)},\frac{exp(a_2/\tau)}{\sum exp(a_i/\tau)},\cdots,\frac{exp(a_i/\tau)}{\sum exp(a_i/\tau)}]$

Feel it directly

import numpy as np

def exp_tem(x,tau):
    return np.exp(x/tau)/sum(np.exp(x/tau))

print(exp_tem(np.array([1,2,3]),2))
#[0.18632372 0.30719589 0.50648039]
print(exp_tem(np.array([1,2,3]),1))
#[0.09003057 0.24472847 0.66524096]
print(exp_tem(np.array([1,2,3]),0.5))
#[0.01587624 0.11731043 0.86681333]

It's not hard to find out ,t The bigger it is , The smaller the gap between the classes , Result more “ smooth ”;t The smaller it is , The greater the gap between the categories , Result more “ Sharp ”.

2 temperature The role of

I think it can be compared to reinforcement learning to some extent ε-greedy, If temperature The setting is relatively large , Then there is little difference between the classes , There is a high probability of choosing different classes , Got some exploration Space ; If temperature It's set smaller , Then the class with the greatest probability The resulting value will “ stand head and shoulders above others ”, So basically, this class is selected .

So we can set temperature $\tau=\frac{c}{1+logT}$ , As the model is updated ,temperature It's getting smaller , And gradually from exploration to turn to exploitation 了 .