Using information entropy to construct decision tree

today , Let's continue to share the relevant content of the decision tree . Through the last article , We know that the basic principle of decision tree is based on the tree structure , From top to bottom , adopt if/else The process of finding a decision path and finally obtaining the decision result .

The core part of the decision tree algorithm is How to choose an appropriate feature attribute to divide , So as to gradually construct our tree structure .

The quantitative evaluation criteria are different according to our selection of features , There are many different construction algorithms for decision trees , In the last article, we introduced the use of Gini（ The gini coefficient ） Construct a decision tree as a splitting criterion , We call this decision tree algorithm CART Algorithm .

today , We From the perspective of information gain, we will further explore other algorithms for constructing decision trees . at present , The commonly used decision tree construction algorithm is as follows ：

First , Before specific discussion , Let's first understand some basic concepts ：

What is information entropy ？

The concept of information entropy was introduced by Claude · Shannon in 1948 in . Information is a very abstract concept , Suppose we take an article as an example , The amount of information in an article does not entirely depend on the number of words in the article , Sometimes even a small amount of content can contain a large amount of information . Information entropy is proposed to solve the problem of quantitative evaluation of information .

Information entropy H(X) The calculation formula of is as follows ：