Mathematical Ideas in AI

The mathematical knowledge involved in AI mainly involves three aspects: linear algebra, calculus, and probability theory. Below we describe the mathematical knowledge involved in the entire AI process. In the final analysis, degree learning is learning how to fit a function, this function maps the input to the output, which is essentially a mathematical modeling problem.

Assumption space:

Hypothesis space, also known as function space, how to choose a function requires some prior knowledge. We are accustomed to dividing house price prediction as a regression problem, because the output value is a continuous real value; whether it will rain tomorrow, the result is onlyThis problem is called classification problem; for such problems, it is easy for us to choose functions, but for complex problems, such as image pattern recognition, ordinary regression cannot complete the task (in machineIn the era of learning, this requires manual extraction of features and input them into the machine learning algorithm), which requires a neural network. Although it is a black box, we don’t know what’s going on inside, but it needs several hidden layers.The so-called function space means that when the network model is fixed, the network parameters are initialized according to a certain probability distribution, and then the function space is optimized during the training process, but there is a special case, Dropout, which is similar to the Bagging idea, the existence of Dropout makes the network model more complex, but this complexity effectively prevents overfitting.

Objective function:

Since it is a fitting problem, there is an indispensable tool for evaluating the fitting effect. This is the role of the objective function, which is used to measure the gap between the output of the fitting function and the real label, and continuously reduce by changing the network parameters.Small gap, this is the optimization problem in mathematics. In mathematics, the derivative of a function represents the rate of change of the dependent variable when the independent variable changes. Ideally, the point where the derivative is 0 is called the extreme point.The parameters of is the final result we require. This ideal function is called a convex function, but the problem in real life is often not a convex optimization problem. The point where the derivative is 0 may be a local extreme point or a saddle point. Faced with thisTo solve this kind of problem, you need the gradient descent method to find the optimal solution, initialize the starting point first, and then move in the direction with the largest gradient at each step, this method may still only find the local optimal solution, the Adam and other methods proposed by scientists are for the purpose ofOptimize the solution process.

The optimization process depends on the characteristics of the loss function: it is differentiable everywhere, the linear regression model chooses the mean square error as the loss function, while the classification problem chooses the cross entropy as the loss function; the function fitted by deep learning is a very complex function, we can think of it as a deep composite function, how to derive a composite function?The answer is the chain derivation rule; for the case where the input and output are both scalars, the derivation process is very simple, but for the CV field, the input is a matrix, the input in the NLP field is a vector, and the output may be a vector or matrix.is a scalar. In this case, the matrix derivation rule can be used to find the extreme value of the loss function.

When it comes to matrix operations, it can be solved according to the properties of the matrix, for example, the inverse of the matrix, the eigenvalue decomposition of the matrix, the singular value decomposition of the matrix, the matrix determinant, etc.