Cognitive fallacy: what is dimensional curse

The more detailed your data , The less insight it has . Add only to the drawing 1 Additional parameters will cause the volume of the graph to increase exponentially , Scatter the contained data points and delete meaningful associations between them .

The phenomenon of dimensional curse appears in numerical analysis 、 sampling 、 Combinatorics 、 machine learning 、 Data mining, database and other fields . The common theme of these issues is , As dimensions increase , The growth of volume and space is so fast , So that the available data becomes sparse . In order to obtain reliable results , The amount of data required usually grows exponentially with the dimension .

The phrase , Due to Richard Bellman, Is to express the use of brute force （ Also known as grid search ） To optimize functions with too many input variables .

In today's big data world , It can also refer to several other potential problems when your data has a large number of dimensions .

1. If we have more features than observations , We run the risk of large-scale over fitting models -- This usually leads to poor off sample performance .
2. When we have too many characteristics , Observations will become more difficult to cluster -- Believe it or not , Too many dimensions will cause each observation in your data set to be equidistant from other observations . Because clustering uses distance measurement methods such as Euclidean distance to quantify the similarity between observations , So this is a big problem . If the distances are approximately equal , Then all the observations look the same （ The same difference ）, Can't form meaningful clustering .

Refer to machine learning PCIA