20. Support vector machine - knowledge of mathematical principles

Main content

1. Inner product of vectors
2. Using the properties of vector inner product to understand SVM
3. SVM's method for selecting better decision boundaries

First, the inner product of vectors

1.1 Definition and geometric meaning of inner product

​

• If there are two vectors u and v , u^Tv is called the inner product between the vectors u and v
• Geometric meaning: The inner product of vectors is equivalent to the product of projection lengths

1.2 Euclidean length (norm)

• If there is a vector u, ∥u∥ represents the norm norm of u, that is, the Euclidean length of the vector u, which is a real number

• According to the Pythagorean theorem, the norm is calculated as follows:

  

1.3 Two ways to calculate inner product

• (1) u^Tv = u1 × v1 + u2 × v2 = v^Tu

• (2) First, project v to the u vector, and record its length as p (positive and negative, positive in the same direction as u, negative in the opposite direction, scalar), then the inner product of the two vectors:

  u^Tv = ||u|| · ||v|| · cosθ = ||u|| · p

• Note: If the angle between the two vectors is greater than 90°, then p is negative, and the inner product of the two vectors is also negative

Second, use the properties of vector inner product to understand SVM

• If C is set very large and A is minimized to 0, then the cost function of SVM will be simplified as shown in the following figure:

  

• For ease of understanding, let's simplify the function expression: let θ0 = 0, and then only have two parameters, θ1 and θ2

  

  • **What the support vector machine does is: **minimize the square of the parameter vector norm, or the square of the length

• According to the inner product calculation formula, there is θ^Tx = p · ||θ||, where p is the projection of x on θ.Use p^(i) ⋅ ∥θ∥ instead of θ^Tx(i)

  

Three, SVM method to select a better decision boundary

• We assume that the decision boundary is shown as the green line in the left figure above, and we can know that the parameter vector θ is perpendicular to the boundary (the proof process can be found in another blog post of mine)

• It is found that for each sample x(1) and x(2), their projection lengths on θ are very small, then in order to satisfy the condition p(i)·||θ|| ≥1 or p(i)·||θ||≤-1, then ||θ|| should take a large value, which contradicts the previous minimization of the cost function (1/2||θ||2);

• The support vector machine tries to make p(i) (the distance from the training sample to the decision boundary) large enough to make the norm of θ small (such as the decision boundary in the right figure above - green line)) to minimize the cost function;
  This is how SVM produces the phenomenon of large-spacing classification;

• Simplifying by letting θ0 = 0 means that we let the decision circle pass through the origin. If θ0 ≠ 0, the decision boundary is no more than the origin , and the conclusion that SVM produces a classifier with a large gap is also true (in the case of a particularly large C).

If you think the article is helpful to you, you can give me a thumbs up and encourage me. Welcome friends to collect articles and study
Follow me, let's learn together and make progress together!!!