How to understand metrics in keras

Keras The library provides a method to calculate and report a set of standard indicators when training deep learning models . In addition to providing standard indicators for classification and regression problems ,Keras It also allows you to define and report your own custom metrics during in-depth learning . If you want to better capture the performance metrics of the model during training , This will be particularly useful .

      In this tutorial , You will learn how to use built-in metrics , And how to Keras Define and use your own indicators when training the deep learning model . After completing this tutorial , You will learn ：

• Keras How indicators work and how to use them when training models .
• How to use working examples in Keras Regression and classification indicators are used in .
• How to use working examples in Keras Define and use your own custom metrics .

1 Keras indicators

      Keras Allows you to list indicators to be monitored during model training . You can specify by “metrics” Parameters are... On the model compile() Function provide function name （ Or function name alias ） List to do this . for example ：

Log in and copy

      
      

       
       model.compile(..., metrics=['mse'])
      
      
      
      

       
       
1.
      
      

      The specific indicators you list can be Keras Name of function （ Such as mean_squared_error） Or string aliases of these functions （ Such as “mse”）.

      The measurements are recorded on each of the training data sets epoch At the end of . If a validation data set is also provided , The recorded metrics are also calculated for the validation dataset .

      All indicators are output and called in detail fit() Report in the history object returned by the function . In both cases , The name of the measurement function is used as the key of the measurement . For the indicators of the validation data set ,“val_” The prefix is added to the key .

      Loss function and clearly defined Keras Indicators can be used as training indicators .

2 Keras Regression index

      Here is what you can do in Keras List of indicators used for regression problems in .

• Mean square error ：mean_squared_error、MSE or
• Mean absolute error ：mean_absolute_error, MAE, mae
• Mean absolute percentage error ：mean_absolute_percentage_error、MAPE、mape
• Cosine proximity ：cosine_proximity, cosine

        The above four indicators , The smaller the value. , The better the fitting degree of the model , But it does not mean that the prediction effect of the model is better , It can only be assumed that the model with good fitting effect has better prediction effect . In other words, it's not too rigorous but easy to remember ： The smaller the above index value , The better the model predicts .

      The following example demonstrates this of a simple artificial regression problem 4 Built in regression indicators .

Log in and copy

      
      

       
       from numpy import array
       
       
from keras.models import Sequential
       
       
from keras.layers import Dense
       
       
from matplotlib import pyplot
       
       
#  Prepare the model 
       
       
X = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
       
       
# create model
       
       
model = Sequential()
       
       
model.add(Dense(2, input_dim=1))
       
       
model.add(Dense(1))
       
       
model.compile(loss='mse', optimizer='adam', metrics=['mse', 'mae', 'mape', 'cosine'])
       
       
#  Training models 
       
       
history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)
       
       
#  Draw pictures to see indicators 
       
       
pyplot.plot(history.history['mean_squared_error'])
       
       
pyplot.plot(history.history['mean_absolute_error'])
       
       
pyplot.plot(history.history['mean_absolute_percentage_error'])
       
       
pyplot.plot(history.history['cosine_proximity'])
       
       
pyplot.show()
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
       
       
7.
       
       
8.
       
       
9.
       
       
10.
       
       
11.
       
       
12.
       
       
13.
       
       
14.
       
       
15.
       
       
16.
       
       
17.
       
       
18.
       
       
19.
      
      

      Be careful ： Your results may vary due to the randomness of the algorithm or evaluation process or differences in numerical accuracy . Consider running the example several times and comparing the average results .

      Running this example will run on each epoch Print measurements at the end .

Log in and copy

      
      

       
       ...
       
       
Epoch 96/100
       
       
0s - loss: 1.0596e-04 - mean_squared_error: 1.0596e-04 - mean_absolute_error: 0.0088 - mean_absolute_percentage_error: 3.5611 - cosine_proximity: -1.0000e+00
       
       
Epoch 97/100
       
       
0s - loss: 1.0354e-04 - mean_squared_error: 1.0354e-04 - mean_absolute_error: 0.0087 - mean_absolute_percentage_error: 3.5178 - cosine_proximity: -1.0000e+00
       
       
Epoch 98/100
       
       
0s - loss: 1.0116e-04 - mean_squared_error: 1.0116e-04 - mean_absolute_error: 0.0086 - mean_absolute_percentage_error: 3.4738 - cosine_proximity: -1.0000e+00
       
       
Epoch 99/100
       
       
0s - loss: 9.8820e-05 - mean_squared_error: 9.8820e-05 - mean_absolute_error: 0.0085 - mean_absolute_percentage_error: 3.4294 - cosine_proximity: -1.0000e+00
       
       
Epoch 100/100
       
       
0s - loss: 9.6515e-05 - mean_squared_error: 9.6515e-05 - mean_absolute_error: 0.0084 - mean_absolute_percentage_error: 3.3847 - cosine_proximity: -1.0000e+00
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
       
       
7.
       
       
8.
       
       
9.
       
       
10.
       
       
11.
      
      

Then create during training 4 A line chart of indicators .

Please note that , The indicator is to use the string alias value [‘mse‘, ‘mae‘, ‘mape‘, ‘cosine‘] designated , And use their extension function names as history Key value reference on object .

      We can also use the extended name to specify the indicator , As shown below ：

Log in and copy

      
      

       
       model.compile(loss='mse', optimizer='adam', metrics=['mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'cosine_proximity'])
      
      
      
      

       
       
1.
      
      

      If the function name is imported into the script , We can also specify them directly .

Log in and copy

      
      

       
       from keras import metrics
       
       
model.compile(loss='mse', optimizer='adam', metrics=[metrics.mean_squared_error, metrics.mean_absolute_error, metrics.mean_absolute_percentage_error, metrics.cosine_proximity])
      
      
      
      

       
       
1.
       
       
2.
      
      

      You can also use the loss function as a measure . for example , You can put the mean square logarithmic error （mean_squared_logarithmic_error,MSLE or msle） The loss function is used as a measure , As shown below ：

Log in and copy

      
      

       
       model.compile(loss='mse', optimizer='adam', metrics=['msle'])
      
      
      
      

       
       
1.
      
      

3 Keras Classification index

      Here is what you can do in Keras List of indicators used to classify problems in .

• Binary precision ：binary_accuracy,acc
• Classification accuracy ：categorical_accuracy, acc
• Sparse classification accuracy ：sparse_categorical_accuracy
• Top k Categorical Accuracy：top_k_categorical_accuracy（ You need to specify one k Parameters ）
• sparse Top k Classification accuracy ：sparse_top_k_categorical_accuracy（ You need to specify k Parameters ）

      Accuracy is special . Whether your problem is a two class problem or a multi class problem , You can specify “accuracy” Indicators to report accuracy .

      The following is an example of a binary classification problem , It demonstrates the built-in accuracy index .

Log in and copy

      
      

       
       from numpy import array
       
       
from keras.models import Sequential
       
       
from keras.layers import Dense
       
       
from matplotlib import pyplot
       
       
#  Prepare the data 
       
       
X = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
       
       
y = array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
       
       
#  Creating models 
       
       
model = Sequential()
       
       
model.add(Dense(2, input_dim=1))
       
       
model.add(Dense(1, activation='sigmoid'))
       
       
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
       
       
#  Training models 
       
       
history = model.fit(X, y, epochs=400, batch_size=len(X), verbose=2)
       
       
#  Draw indicators 
       
       
pyplot.plot(history.history['accuracy'])
       
       
pyplot.show()
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
       
       
7.
       
       
8.
       
       
9.
       
       
10.
       
       
11.
       
       
12.
       
       
13.
       
       
14.
       
       
15.
       
       
16.
       
       
17.
      
      

      Be careful ： Your results may vary due to the randomness of the algorithm or evaluation process or differences in numerical accuracy . Consider running the example several times and comparing the average results .

      Running this example will report the accuracy at the end of each training period .

Log in and copy

      
      

       
       ...
       
       
Epoch 396/400
       
       
0s - loss: 0.5934 - acc: 0.9000
       
       
Epoch 397/400
       
       
0s - loss: 0.5932 - acc: 0.9000
       
       
Epoch 398/400
       
       
0s - loss: 0.5930 - acc: 0.9000
       
       
Epoch 399/400
       
       
0s - loss: 0.5927 - acc: 0.9000
       
       
Epoch 400/400
       
       
0s - loss: 0.5925 - acc: 0.9000
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
       
       
7.
       
       
8.
       
       
9.
       
       
10.
       
       
11.
      
      

4 stay Keras Custom indicators in

      You can also call compile() Function and define your own indicators in “metrics” Specify the function name in the function list of the parameter .

I haven't used custom indicators yet , I haven't even seen others use . Therefore, there are few guesses about the application of custom indicators . When you choose Custom indicators , Ask yourself at least three questions ： Can common indicators really not meet the demand ？ Are the commonly used indicators used to measure the model selected correctly , Is there a classification index to measure the regression problem ？ My math is good enough to understand correctly “metrics” What functions in the function list of parameters are used for ？ At least I was blocked by the first question . In the future, I will add some custom indicators .^_^