Some understanding of inductive bias

Inductive bias

   Inductive bias is a very subtle concept in machine learning ： In machine learning , Many algorithms make assumptions about some problems , These assumptions are called inductive biases .

• inductive ： Look for commonalities in some examples , generalization , Form a more general rules and regulations
• bias ： Our preference for models

therefore , Inductive bias can be understood as , A certain rule concluded from the phenomena observed in real life , Then make certain constraints on the model , So as to play “ Model selection ” The role of , That is to choose a model more in line with the real rules from the hypothetical space .
Inductive bias is almost everywhere in machine learning ：

• Okam razor ： If not necessary , Do not add entities.
• KNN It is assumed that adjacent samples in the feature space tend to belong to the same class
• SVM The classifier should maximize the class boundary distance

The same is true in deep learning . Take neural networks as an example , Various network structures 、 Components 、 The mechanism often comes from inductive bias . stay CNN in , We assume that features have local properties , That is, when we put some adjacent features together , It will be easier to get the solution ; stay RNN in , We assume that the calculation at each moment depends on the historical calculation results ; And attention mechanisms , It is also based on human intuition 、 The rules summed up by life experience .

CNN Of inductive bias yes locality and spatial invariance, That is, those with similar space grid elements Connected but far from , And weight sharing
RNN Of inductive bias yes sequentiality and time invariance, That is... In sequence timesteps Connection and invariance of time transformation

summary ：

All in all ,inductive bias It is our task that has these difficulties , The model we chose can solve these difficulties , Therefore, the ability to solve these difficulties is a model inductive bias