1.4 regression of machine learning methods

Regression analysis is used to predict the relationship between input variables and output variables , Especially when the value of the input variable changes , The value of the output variable also changes .

1. Linear regression

The linear regression algorithm assumes that the characteristics and results satisfy the linear relationship . This means that you can multiply the input by some constants , Add up the results to get the output .

• Model
  Select the fitting function form $h_{\theta }(x)={\sum }_{i=0}^n{\theta }_i{X}_i={\theta }^{\top }X$
  To describe the components in a feature , such as $x_1$ The area of the room ,$x_2$ The orientation of the room , wait , Make an estimation function ：
  $$h(x)=h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2$$

• Strategy
  Determine the form of the loss function ：
  $$J(\theta )=\frac{1}{2}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2$$
  $$\underset{\theta }{\min }J(\theta )$$

• Algorithm
  Gradient descent method . First of all, $\theta$ assignment , This value can be random , Can also let $\theta$ It's an all zero vector .
  change $\theta$ Value , bring $J(\theta )$ Decrease in the direction of gradient descent , The end of the algorithm will be in $\theta$ Down to the point where it can't continue to fall .
  

1.1 Least square method

See 《 Linear regression least square method for machine learning 》

1.2 Ridge return

Ridge return ( English name ：ridge regression, Tikhonov regularization) It is a biased estimation regression method for collinearity data analysis , In essence, it is an improved least squares estimation method , By giving up the unbiasedness of least squares , To lose some information 、 It is more practical to obtain the regression coefficient at the cost of reducing the accuracy 、 More reliable regression methods , The fitting of ill-conditioned data is better than the least square method .

1.3 Lasso Return to

Lasso Regression is a compressed estimate . It gets a more refined model by constructing a penalty function , Make it compress some coefficients , At the same time, set some coefficients to zero . Therefore, the advantage of subset contraction is retained , It is a biased estimator for processing data with complex collinearity .
Applicable scenario ： Small sample size , But there are many indicators . Applicable to high-dimensional statistics , Traditional methods cannot deal with such data . also Lasso Feature selection can be carried out .
Fundamental theorem .Lasso Parameter estimation is defined as follows

1.3.1 Back to the case ： Financial revenue forecast of a city

On the basis of existing research Lasso The method of feature selection studies the factors that affect local fiscal revenue , stay Lasso Based on feature selection , Use support vector regression SVR Model , Regression analysis of the selected features , Get the prediction model of fiscal revenue . The case code is based on python+pandas+numpy+scikit-learn Realized .

Basic information of fiscal revenue data . Name of each feature ： Number of social employees x1、 Total wages of on-the-job employees x2、 Total retail sales of consumer goods x3、 Per capita disposable income of urban residents x4、 Per capita consumption expenditure of urban residents x5、 Total Population x6、 Investment in fixed assets of the whole society x7、 Gross Regional Product x8、 Output value of primary industry x9、 tax x10、 Consumer price index x11、 The ratio of the output value of the tertiary industry to that of the secondary industry x12、 Consumption level of residents x13.