Machine learning: the difference between random gradient descent (SGD) and gradient descent (GD) and code implementation.

If you want to understand in detail ：-》 Gradient descent method

Gradient descent method （GD）

Hypothesis function fx, Cost function cost, It has the following expression ：
$$\begin{array}{r}f\left(x\right)=w_1{x}_1+w_2{x}_2+b\\ cost\left(w\right)=\frac{1}{n}\sum _{i=1}^n\left(f（x_i\right)-y_i)\\ w_1=w_{1old}-\alpha \frac{\partial cost\left(w\right)}{\partial w_1}cost\left(w\right)\\ w_2=w_{2old}-\alpha \frac{\partial cost\left(w\right)}{\partial w_2}cost\left(w\right)\end{array}$$
From the above formula , We come to the conclusion that ：
1. Parameters w,b Every update , It is necessary to calculate the partial derivative of all data to the corresponding parameters , This calculation is very large , The convergence speed of the function will be very slow when there is a large amount of data .
2. And SGD Different , Every time the parameter changes , Can guarantee cost It moves towards the global minimum .
3. If cost Nonconvex functions , Functions may fall into local optima .

And then the gradient goes down （SGD）

The formula is as follows ：
$$f\left(x\right)=w_1{x}_1+w_2{x}_2+b$$
$$\begin{gathered} for(i=0,i<=n,i++) \\ cost\left(w\right)=(f(x_i)-y_i) \\ {w}_1=w_{1old}-\alpha \frac{\partial cost\left(w\right)}{\partial w_1}cost\left(w\right) \\ {w}_2=w_{2old}-\alpha \frac{\partial cost\left(w\right)}{\partial w_2}cost\left(w\right) \end{gathered}$$

From the above formula , Come to the following conclusion ：

1. SGD Every time the parameter is updated , Only calculated 1 individual batch Gradient of （ The above formula assumes batch=1）, It greatly accelerates the convergence speed of the function .
   2.SGD Only one data is considered for each parameter update , It may not move in the direction of global optimization every time , Eventually, it may not converge to the minimum , But it will solve the problem of falling into local optimization .

Code implementation

Take Boston house price forecast as an example
Import data

import numpy as np
path = 'Desktop/ Boston prices /trian.csv'
data = np.loadtxt(path, delimiter = ",", skiprows=1)
data.shape

Split data

train = data[:int(data.shape[0]*0.8)]
test = data[int(data.shape[0]*0.8):]
print(train.shape, test.shape)
train_x = train[:,:-1]
train_y = train[:,13:]
test_x = test[:,:-1]
test_y = test[:,13:]
print(train_x.shape, train_y.shape)

class Network:

    def __init__(self, num_weights):
        self.num_weights = num_weights
        self.w = np.random.rand(num_weights, 1)
        self.b = 0

    def forward(self, x):
        z = np.dot(x, self.w) + self.b 
        return z

    def loss(self, z, y):
        cost = (z-y)*(z-y)
        cost = np.mean(cost)
        return cost

    def gradient(self, z, y):
        w = (z-y)*train_x
        w = np.mean(w, axis=0)
        w = np.array(w).reshape([13, 1])
        b = z-y
        b = np.mean(b)
        return w, b

    def update(self, gradient_w, gradient_b, eta):
        self.w = self.w - eta*gradient_w
        self.b = self.b - eta*gradient_b
# gradient descent 
    def train_GD(self, items, eta):
        for i in range(items):
            z = self.forward(train_x)
            loss = self.loss(z, train_y)
            gradient_w, gradient_b = self.gradient(z, train_y)
            self.update(gradient_w, gradient_b, eta)
            # if i % 100 == 0:
            test_loss = self.test()
            print('item:', i, 'loss:', loss, 'test_loss:', test_loss)
# And then the gradient goes down 
    def train_SGD(self, num_epochs, batchsize, eta):
        for epoch_id in range(num_epochs):
            np.random.shuffle(train)
            losses = []
            for i in range(0, len(train), batchsize):
                # print(i, batchsize+i)
                mini_batchs = train[i:i + batchsize]
                for iter_id, mini_batch in enumerate(mini_batchs):
                    # print(mini_batch)
                    x = mini_batch[:-1]
                    y = mini_batch[-1]
                    z = self.forward(x)
                    loss = self.loss(z, y)
                    gradient_w, gradient_b = self.gradient(z, y)
                    self.update(gradient_w, gradient_b, eta)
                    losses.append(loss)
            sum = 0
            for i in losses:
                sum += i
            loss_mean = sum/len(losses)
            print('Epoch{}, loss{}, loss_mean{}'.format(epoch_id, loss, loss_mean))

    def test(self):
        z = self.forward(test_x)
        loss = self.loss(z, test_y)
        return loss


net = Network(13)
net.train_GD(100, eta=1e-9)
net.train_SGD(100, 5, 1e-9)