21. Support Vector Machine - Introduction to Kernel Functions

Main content

1. The concept of kernel function
2. Classification of kernel functions

I. The concept of kernel function

1.1 Kernel function definition

• In order to better classify, the support vector machine maps the input low-dimensional space X (a subset or discrete set of Euclidean space R^n) to a high-dimensional feature space H (Hilbert space), if there is K(x, y), x, y ∈ Χ in the low-dimensional space, such that K(x, y) = ϕ(x) ϕ(y), then K(x, y) is the kernel function, where ϕ(x)·ϕ(y) is the inner product mapped to the feature space by ϕ(x)·ϕ(y), and ϕ(x) is X→Hthe mapping function.
• Visual understanding: Think of a kernel function as a wrapper or interface that converts data from a hard-to-handle form to an easier-to-handle form
• Kernel tricks or nuclear transformation: One of the benefits of SVM optimization is that all operations can be written in the form of inner products. We can replace the inner product operations with kernel functions for processing.This process is called nuclear technology or nuclear transformation

The role of the 1.2 kernel function

• After spatial transformation through the kernel function, we can solve linear problems in high-dimensional space, which is equivalent to solving nonlinear problems in low-dimensional space
• The dimension of the target feature space H is generally high, and may even be infinite, so it is difficult to find the inner product. When using it, only the kernel function is defined, and the mapping function ϕ is not explicitly defined, so only the transformed inner product is involved.product without transforming the value.In this way, on the one hand, it can solve the linear inseparable problem, and on the other hand, it avoids the "curse of dimensionality" and reduces the amount of computation.
• Kernel functions are not only used in support vector machines, many other machine learning algorithms also use kernel functions

Second, the classification of kernel functions

• Introduction to common kernel functions in the collection

2.1 Linear Kernel

• Linear kernel function is the simplest kernel function, mainly used for linearly separable cases
• expression for linear kernel function:

​

​ where c is a optional constant

• The linear kernel function is the inner product of the original input space, that is, The dimension of the feature space and the input space is the same, with fewer parameters and faster operation speed
• In general, when the number of features is very large relative to the number of samples, a linear kernel function is suitable.

2.2 Polynomial Kernel

• The expression of the polynomial kernel function is:

  

  • α represents the adjustment parameter, d represents the highest order number, and c is an optional constant

• The polynomial kernel function has many parameters. When the polynomial order is high, the complexity will be very high. For the orthogonally normalized data, the polynomial kernel function can be preferred

2.3 Radial Basis Function Kernel

• The radial basis kernel function is also called Gaussian kernel function because it is similar to the Gaussian function

• The expression of the Radial Basis Kernel Function is:

  

  • The larger the α2 is, the smoother the Gaussian kernel function becomes, that is, a model that changes slowly with the input x, The model has large deviation and variance, poor generalization ability, and is easy to overfit
  • The smaller the α2, the more severe the change of the Gaussian kernel function, the smaller the deviation and variance of the model, and the more sensitive the model is to noise samples.

• Radial basis kernel functions are flexible and widely used

• Compared with the polynomial kernel function, it has fewer parameters, so in most cases, it has better performance; When you are not sure which kernel function to use, it can be preferredVerify the Gaussian kernel function

2.4 Sigmoid Kernel

• Sigmoid kernel function expression is as follows:

  

  • α represents the adjustment parameter, c is an optional constant, generally, c is 1/n, and n is the data dimension

• The Sigmoid kernel function is derived from the activation function in MLP, and SVM uses Sigmoid to be equivalent to a two-layer perceptron network

If you think the article is helpful to you, you can like and encourage me, welcome to collect the article and learn together
Follow me, we will learn and make progress together!!!