Dimension disaster dimension disaster suspense

Distance measurement problem

For distance based models KNN,K-means Come on . Effective dimensionality reduction is needed , Or a lot of data training , Discover the low dimensional manifold space of data .

Theorem[Beyer et al.99]:Fix $ϵ$ >0 and N,If data is “truly high-dimensional”,the under fairly weak assumptions on the distrbution of the data:
$$\underset{D\to \infty }{\lim }Pr[d_{max}(N,D)\le (1+ϵ)d_{min}(N,D)]=1$$

$$With\; dimensionsDAn\; increase\; in\; ,\; The\; difference\; between\; the\; maximum\; distance\; and\; the\; minimum\; distance\; between\; data\; will\; be\; infinitely\; small\; ,\; Using\; distance\; will\; not\; effectively\; distinguish\; data$$

The difference of Euclidean distance is not obvious

With dimensions d rising , The relative difference between the maximum and minimum Euclidean distances approaches 0, therefore KNN The convergence rate is very slow :
$$\underset{d\to \infty }{\lim }E(\frac{dis{t}_{max}(d)-dis{t}_{min}(d)}{dis{t}_{min}(d)})\to 0$$

Related papers

Gaussian distribution . If we look at the Gaussian density of the three-dimensional input space , It's like a hypersphere , The probability decreases with the average radius , This usually applies to more than 3 Dimensions . The volume fraction occupied by the shell on the hypersphere increases with the increase of data dimension

high dimension, there is no such thing as interpolatioIn high dimension, everything isextrapolation

The problem of data demand

For linear classifiers , We need to constantly increase dimensions

More training data is needed

Space size problem

Volume of unit hypersphere

$$\begin{array}{c}{V}_n=\frac{{\pi }^{n/2}}{\Gamma \left(\frac{n}{2}+1\right)}\end{array}$$

        So more volume is in the space around the hypersphere , With dimensions n An increase in , The proportion of hypersphere tends to 0.

Computer vision fool's book :https://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/ Chinese translation

Reference and more

You know ： How to understand Curse of Dimensionality（ Dimension disaster ）?
Dimension disaster
Curse_of_dimensionality
Hypersphere volume calculation

Why do dimensional disasters occur ？ How to solve ？

Concentration of spherical Gaussians
Why is the Gaussian distribution in high-dimensional space like a soap bubble

Like the relationship between the distribution of low and high dimensions , The center of high-dimensional space collapses .
In high-dimensional space, with the increase of dimension, the ratio of the center of space will shrink to zero, or even the positive looks like a soap bubble (where it is less dense in the center and more dense at the edge) instead of a bold of mold where it is more dense in the center.
In high-dimensional space, with the increase of dimension, the ratio of Space Center will shrink to zero, and even the Gaussian distribution looks like a soap bubble (low center density, high edge density), rather than a model with high center density.

Information theory

k - Nearest neighbor classification
Another effect of higher dimensions on distance functions involves using distance functions to build from datasets k Nearest neighbor ( k -NN) chart . As the dimensions increase ,k -NN The penetration distribution of a digraph becomes skewed , The peak appears on the right , Because there is a disproportionate number of hubs , That is, it appears in more others k -NN The data points in the list are higher than the average . This phenomenon will affect various classification techniques （ Include k-NN classifier ）、 Semi supervised learning and clustering ,[19], It also affects information retrieval .[20]

Anomaly detection
stay 2012 In a survey in ,Zimek wait forsomeone . The following problems were found when searching for exceptions in high-dimensional data ： [13]

Concentration of fractions and distances ： Derived values such as distance become numerically similar
Irrelevant properties ： In high dimensional data , A large number of attributes may be irrelevant
Definition of reference set ： For local methods , Reference sets are usually based on nearest neighbors
Incomparable scores of different dimensions ： Different subspaces produce incomparable scores
The interpretability of scores ： Scores usually no longer convey semantics
Index search space ： The search space can no longer be scanned by the system
Data snooping bias ： Given a large search space , For the meaning of each expectation , Can find a hypothesis
Hubness： Some objects appear more frequently in the neighbor list than others .
Many specialized methods of analysis solve one or the other of these problems , But there are still many unsolved research problems .

Dimensional blessing
It's amazing , Despite the expected “ Curse of dimensions ” difficult , But common sense heuristics based on the most direct method are useful for high-dimensional problems “ It can produce almost certainly the optimal result ”.[21] 1990 In the late S “ Dimensional blessing ” The word" .[21] Donohue is in his “ Millennium Declaration ” It clearly explains why “ Dimensional blessing ” It will become the foundation of data mining in the future .[22] The influence of dimensional blessing has been found in many applications , And found their basis in the concentration of measurement phenomena .[23] An example of dimensionality is the linear separability of a random point and a large finite random set , Even if this set is exponential , It's also possible ： The number of elements in this random set can increase exponentially with the dimension . Besides , This linear functional can be chosen as the simplest linear Fisher Discriminant . This separability theorem has been proved to be applicable to a wide range of probability distributions ： Generally uniform logarithmic concave distribution 、 Product distributions in cubes and many other families （ Recently [23] Commented on ）.

“ Dimensional blessing and dimensional curse are two aspects of the same coin .” [24] for example , In essence, the typical property of high-dimensional probability distribution in high-dimensional space is ： The square distance from the random point to the selected point is likely to be close to the average （ Or median ） Square distance . This attribute significantly simplifies the expected geometry of the data and the indexing of high-dimensional data （ blessing ）,[25], But at the same time , It makes the similarity search in high dimension difficult or even useless （ Damnation ）.[26]

Zimek et al .[13] Pointed out that , Although the typical formalization of dimensional curse will affect iid data , But even in high dimensions , The data separated in each attribute will also become easier , And that the signal-to-noise ratio is very important ： The data in each attribute becomes easier to add the attribute of the signal , Only add noise to the data （ Irrelevant errors ） The properties of are more difficult . Especially for unsupervised data analysis , This effect is called swamping .