"Statistical learning methods (2nd Edition)" Li Hang Chapter 17 latent semantic analysis LSA LSI mind mapping notes and after-school exercise answers (detailed steps) Chapter 17

Mind mapping ：

17.1

Trial plot 17.1 For latent semantic analysis , And observe the results .

import numpy as np

X = np.array([[2, 0, 0, 0],
              [0, 2, 0, 0],
              [0, 0, 1, 0],
              [0, 0, 2, 3],
              [0, 0, 0, 1],
              [1, 2, 2, 1]])

U, Sigma, VT = np.linalg.svd(X)
print(f' word - The topic matrix is ：\n{
      U[:, :4]}')
print(f' topic of conversation - The text matrix is ：\n{
      Sigma * VT}')

 word - The topic matrix is ：
[[-7.84368672e-02 -2.84423033e-01  8.94427191e-01 -2.15138396e-01]
 [-1.56873734e-01 -5.68846066e-01 -4.47213595e-01 -4.30276793e-01]
 [-1.42622354e-01  1.37930417e-02 -1.25029761e-16  6.53519444e-01]
 [-7.28804669e-01  5.53499910e-01 -2.24565656e-16 -1.56161345e-01]
 [-1.47853320e-01  1.75304609e-01  8.49795536e-18 -4.87733411e-01]
 [-6.29190197e-01 -5.08166890e-01 -1.60733896e-16  2.81459486e-01]]
 topic of conversation - The text matrix is ：
[[-7.86063931e-01 -9.66378217e-01 -1.27703091e+00 -7.78569971e-01]
 [-1.75211118e+00 -2.15402591e+00  7.59159661e-02  5.67438984e-01]
 [ 4.00432027e+00 -1.23071666e+00 -4.47996189e-16  6.41436023e-17]
 [-5.66440763e-01 -6.96375947e-01  1.53734473e+00 -6.74757960e-01]]

about U matrix , Only the former 4 Column , Because in the end, only 4 Singular values （ topic of conversation ）. in addition , Scientific counting is somewhat difficult to observe , We reserve the 2 Take a look at the decimal places U matrix ：

You can see , First column , The more important thing is 4 And the 6 Word , namely apple and produce, Guess this is about Apple or apple agricultural products ;
Second column , The more important thing is 2,4,6 Word , namely aircraft,apple,produce, This topic may have a certain probability problem , Whereas aircraft, It may be the topic of Apple product air transportation ;
The third column , The most important thing is the first word , The second is relatively important , namely airplane,airplane, Guess the topic is aviation related ;
The fourth column , Here's the important thing computer,fruit,aircraft, This is confusing , It may indicate that there is not enough data , The classification is still biased , Maybe this singular value or principal component is no longer important .
Then take a look at Sigma matrix ：

You can find 4 There is no magnitude difference between the two topics , Then it may indeed be divided into these four topics rather than less .
Finally, take a look at VT matrix ：

Analysis of each column shows , The probability of the first sample is 3 Topics , Corresponding to the aviation topic in the third column ;
The second sample probability is the second topic , It may involve the transportation of Apple products , But the actual situation may not be so ;
The third sample probability is the fourth topic , Combined with its large projection component in the first topic , Guess it's related to Apple products ;
The fourth sample probability is the first and fourth topic .
In short, it can be found that the analysis is not so accurate .

17.2

The nonnegative matrix decomposition is given when the loss function is divergence loss （ Latent semantic analysis ） The algorithm of .

Imitation of this topic book 335 page , Algorithm 17.1 It's a good way to do it , Just change the update rule from square loss to divergence loss .
Input ： word - Text matrix $X\ge 0$, Number of topics in the text set $k$, Maximum number of iterations $t$;
Output ： word - Topic matrix $W$, topic of conversation - Text matrix $H$.
(1) initialization ：
$W\ge 0$, Also on $W$ Each column of is normalized ;
$H\ge 0$
(2) iteration ：
The number of iterations is determined by 1 To $t$ Follow these steps ：
i. to update $W$ The elements of ,
$$W_{il}*W_{il}\frac{{\sum }_j{H}_{lj}{X}_{ij}/(WH{)}_{ij}}{{\sum }_j{W}_{lj}}$$
among ,$i$ from 1 To $m$（ Number of words ）,$l$ from 1 To $k$.
ii. to update $H$ The elements of ,
$$H_{lj}*H_{lj}\frac{{\sum }_i{W}_{il}{X}_{ij}/(WH{)}_{ij}}{{\sum }_i{W}_{il}}$$
among ,$j$ from 1 To $n$（ Number of texts ）,$l$ from 1 To $k$.

17.3

The computational complexity of two algorithms for latent semantic analysis is given , Including singular value decomposition method and non negative matrix decomposition method .

First , General number of words $m$ Greater than the number of texts $n$, For singular value decomposition algorithm , Its complexity is $O(m^3)$, For nonnegative matrix, its complexity is $O(t{m}^2)$

17.4

List the similarities and differences between latent semantic analysis and principal component analysis .

identical ：
For latent semantic analysis using singular value decomposition LSA, Its essence is exactly the same as principal component analysis ;
For using nonnegative matrix decomposition LSA, Its idea is the same as that of truncated singular value decomposition .

Different ：
For using singular value decomposition LSA, The sample in this book is listed , features / Words are lines , And PCA The process of one is different PCA To normalize the data （ Put the origin at the position of data mean , Variance can be normalized, but not normalized ）, in addition ,PCA Of SVD We should transpose the original matrix in this book , therefore PCA Of SVD The right singular matrix in the algorithm corresponds to LSA Left singular matrix of , Another less important point is PCA Of SVD Is the sample PCA For the whole PCA Estimation , It uses the unbiased estimation that the sample covariance is the total covariance , The sample covariance is different from that just mentioned LSA It's one transpose away , There will be more $\frac{1}{n-1}$ The coefficient of ;
PCA The original matrix of is generally not a sparse matrix ,LSA The matrix of is generally sparse ;
Personally feel ,LSA Try something like PCA Do that to normalize the data , Because different words may be chosen to be the same scaling There will be different results , Maybe it will be closer to the real situation , But I haven't practiced , Please leave a message and let me know if someone has tested it , I'm curious ;
LSA Nonnegative matrix decomposition of NMF Obviously and SVD There are differences in Algorithm , The advantage of being positive is that it looks better to explain .