[data mining] Chapter 2 understanding data

data mining

Chapter two Knowledge data

The future world is a ternary world ： Physical space ; Human social space ; Information space

2.1 Type of data

give an example ：

• Trading data
• Document data
• network data
• Gene sequence
• Environmental data

What is data ？

Data is a collection of objects and their attributes , It can be expressed as a matrix . Data can be understood as a point in high-dimensional space

Attribute types

• Classified
  * Nominal ： male or Woman
  * Ordinal number ： Undergraduate , master , Doctor
• Numerical
  * Section ： Today's temperature is 15~28 degree
  * ratio ： today A The stock market fell 2.26%

Normalization 、 distance 、 angle

• Normalization ： Normalize the data , Standardization and other operations . For example, subtract the average divided by the standard deviation
• Euclidean distance ： Is the most commonly used distance .
• Cosine distance ： Cosine of the angle between two vectors , The bigger the value is. , The smaller the angle , It shows that two vectors are almost in the same direction , The more relevant . It can be used to judge the correlation between two vectors , For example, for face recognition .

Probability view

For example, analyze the probability distribution of an attribute .

• Probability distribution function
  • Probability density function ：pdf
  • Probability mass function ：pmf, The probability that the sample is some discrete value
• Common probability distribution
  • Bernoulli distribution
  • The binomial distribution

Data quality

• Noise and outliers
• Missing value
• Duplicate data

sampling

• Simple random sampling
  • The probability of selecting any object is the same
  • No return sampling
  • There is a return sample
• Stratified sampling
• Sample size

Property transfer

• Nonlinear functions ： power function , Exponential function , Logarithmic function
• Standardization

Dimension disaster and dimension reduction

Purpose ：

• Avoid dimensional disaster
• Reduce the time and memory required by data mining algorithms
• Make data easier to visualize
• Help eliminate irrelevant features or reduce noise

Method ：

• Principal component analysis (PLA)
• Singular value decomposition （SVD）
• TSNE： Put high-dimensional data in 2,3 Dimensional visualization

Selection of feature subset

It is also a data dimensionality reduction

post-processing

• visualization ： intuitive , The human eye is a powerful analytical tool