Pytorch main modules

Basic steps of machine learning and deep learning

The basic process of machine learning is as follows ·：

The basic process of in-depth learning is as follows ：

pytorch Basic process of model training

Basic parameter settings

import os
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import torch.optim as optimizer
# Initialize the super parameter 
batch_size = 16 #batch size
lr = 1e-4 # Initial learning rate 
max_epochs = 100 # Maximum training times 
#GPU Set up 
#  Scheme 1 ： Use os.environ, If you use GPU You don't have to set it 
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
#  Option two ： Use “device”, For subsequent use GPU For variables of .to(device) that will do 
device = torch.device("cuda:1" if torch.cuda.is_available() else "cpu")

Data read in

Custom data class

Customize Dataset Class needs to inherit PyTorch Self Dataset class . It mainly contains three functions ：

• __init __: Used to pass in external parameters to the class , Also define the sample set
• __getitem __: Used to read the elements in the sample set one by one , You can make some changes , And will return to training / Data required for validation
• __len __: The number of samples used to return the dataset

for example ：

class MyDataset(Dataset):
    def __init__(self, data_dir, info_csv, image_list, transform=None):
        """ Args: data_dir: path to image directory. info_csv: path to the csv file containing image indexes with corresponding labels. image_list: path to the txt file contains image names to training/validation set transform: optional transform to be applied on a sample. """
        label_info = pd.read_csv(info_csv)
        image_file = open(image_list).readlines()
        self.data_dir = data_dir
        self.image_file = image_file
        self.label_info = label_info
        self.transform = transform

    def __getitem__(self, index):
        """ Args: index: the index of item Returns: image and its labels """
        image_name = self.image_file[index].strip('\n')
        raw_label = self.label_info.loc[self.label_info['Image_index'] == image_name]
        label = raw_label.iloc[:,0]
        image_name = os.path.join(self.data_dir, image_name)
        image = Image.open(image_name).convert('RGB')
        if self.transform is not None:
            image = self.transform(image)
        return image, label

    def __len__(self):
        return len(self.image_file)

Read data locally

from torchvision import datasets
# train_path = '' # Training set path 
# val_path = '' #  Test set path 
train_data = datasets.ImageFolder(train_path, transform=data_transform)
val_data = datasets.ImageFolder(val_path, transform=data_transform)
#  or 
train_data = MyDataset(train_path, transform=data_transform)
val_data = MyDataset(val_path, transform=data_transform)

Data is loaded in batches

train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, num_workers=4, shuffle=True, drop_last=True)
val_loader = torch.utils.data.DataLoader(val_data, batch_size=batch_size, num_workers=4, shuffle=False)
# batch_size： The number of samples read in each batch 
# num_workers： How many processes are used to read data 
# shuffle： Whether to disturb the read data 
# drop_last： For samples whose last part of the sample does not reach the number of batches , Stop them from training 

Picture data view

import matplotlib.pyplot as plt
images, labels = next(iter(val_loader))
print(images.shape)
plt.imshow(images[0].transpose(1,2,0))
plt.show()

model building

Building a deep learning neural network mainly through torch.nn modular .torch.nn It mainly includes the following parts ：

classification	modular	Sub module	explain
Parameters	parameter	Parameter	Model parameters ,Tensor
		UninitializedParameter	There is no need to initialize parameters
		UninitializedBuffer	No initialization is required Tensor
Basic unit	Containers	Module	Build the basic unit of neural network
		Sequential	Connect different modules to form a neural network model
		ModuleList/Dict	Module Composed of List/Dict, No sequential connection
		ParameterList/Dict	Parametric List/Dict
Base layer	Convolution Layers （ Convolution layer ）	nn.Conv1d nn.Conv2d nn.Conv3d	1、2、3 Dimensional signal convolution
		nn.ConvTranspose1d nn.ConvTranspose2d nn.ConvTranspose3d	1、2、3 Dimensional image transpose convolution
		nn.LazyConv1d nn.LazyConv2d nn.LazyConv3d	Use the first input initialization parameter 1、2、3 Dimensional signal convolution
		nn.LazyConvTranspose1d nn.LazyConvTranspose2d nn.LazyConvTranspose3d	Use the first input initialization parameter 1、2、3 Dimensional image transpose convolution
		nn.Unfold	Extract elements from sliding windows
		nn.Fold	Restore the elements in the sliding window to Tensor
	Pooling layers （ Pooling layer ）	nn.MaxPool1d nn.MaxPool2d nn.MaxPool3d	1、2、3 Dimension maximization
		nn.MaxUnpool1d nn.MaxUnpool2d nn.MaxUnpool3d	1、2、3 Dimension maximum pooling plus 0 Restore
		nn.AvgPool1d nn.AvgPool2d nn.AvgPool3d	1、2、3 Dimensional mean pooling
		nn.FractionalMaxPool2d nn.FractionalMaxPool3d	2、3 Dimensional fractional order maximization
		nn.LPPool1d nn.LPPool2d	1、2 Dimensional power average pooling
		nn.MaxPool1d nn.MaxPool2d nn.MaxPool3d	1、2、3 Dimension maximization
		nn.AdaptiveMaxPool1d nn.AdaptiveMaxPool2d nn.AdaptiveMaxPool3d nn.AdaptiveAvgPool1d nn.AdaptiveAvgPool2d nn.AdaptiveAvgPool3d	1、2、3 Dimensional adaptive maximum / The average pooling
	Padding Layers	nn.ReflectionPad1d nn.ReflectionPad2d nn.ReflectionPad3d	Use the reflection of the input boundary （ Take the boundary as the axisymmetric element ） Fill in the input tensor
		nn.ReplicationPad1d nn.ReplicationPad2d nn.ReplicationPad3d	Fill the input tensor with input boundary elements
		nn.ZeroPad2d	use 0 Input tensor
		nn.ConstantPad1d nn.ConstantPad2d nn.ConstantPad3d	Fill the input tensor with the specified constant
	Non-linear Activations （ Nonlinear activation function ）	nn.Softmax nn.Sigmoid nn.ReLU nn.Tanh etc.	Details refer to Official website
	Linear Layers （ Linear layer ）	nn.Identity nn.Linear nn.Bilinear nn.LazyLinear	Linear variation
	Normalization Layers	nn.BatchNorm1d nn.BatchNorm2d nn.BatchNorm3d nn.LazyBatchNorm1d nn.LazyBatchNorm2d nn.LazyBatchNorm3d	A data batch Normalize in , Details refer to The paper
		nn.InstanceNorm1d nn.InstanceNorm2d nn.InstanceNorm3d nn.LazyInstanceNorm1d nn.LazyInstanceNorm2d nn.LazyInstanceNorm3d	Normalization in one channel
		nn.LayerNorm	One layer is normalized
		nn.GroupNorm	A set of data （mini-batch） Normalize in
		nn.SyncBatchNorm	Normalization within a set of specified dimensional data
		nn.LocalResponseNorm	Specify local normalization around the data
	Recurrent Layers	nn.RNNBase nn.RNN nn.LSTM nn.GRU nn.RNNCell nn.LSTMCell nn.GRUCell	Related structure layer of recurrent neural network
	Transformer Layers	nn.Transformer	Transformer Model
		nn.TransformerEncoder nn.TransformerDecoder	It consists of multiple coding layers （ Decoding layer ） The encoder consists of （ decoder ）
		nn.TransformerEncoderLayer	It is composed of self attention network and feedforward neural network
		nn.TransformerDecoderLayer	By the self attention network 、multi-head Self attention network and feedforward neural network
	Dropout Layers	nn.Dropout nn.Dropout2d nn.Dropout3d	Press during training Bernoulli The distribution will probability p The data of is randomly changed to 0（ Prevent over fitting ）
		nn.AlphaDropout nn.FeatureAlphaDropout	dropout The process keeps the mean 、 The standard deviation remains unchanged
	Sparse Layers	nn.Embedding	Embed vector
		nn.EmbeddingBag	take embedding Group sum 、 Mean value calculation
function	distance function	nn.CosineSimilarity	Cosine similarity
		nn.PairwiseDistance	p Normal form pairwise distance
	Loss function	nn.L1Loss nn.MSELoss nn.CrossEntropyLoss nn.KLDivLoss etc.	Details refer to Official website
other	Vision Layers	nn.PixelShuffle nn.PixelUnshuffle	Pixel reorganization / Restore
		nn.Upsample nn.UpsamplingNearest2d nn.UpsamplingBilinear2d	On the sampling
	Shuffle Layers	nn.ChannelShuffle	Channel data is disordered
	DataParallel Layers	nn.DataParallel nn.parallel.DistributedDataParallel	many GPU Parallel computing
	Utilities	from torch.nn.utils import...	Details refer to Official website

Module Building neural networks

Take multilayer perceptron as an example ：

import torch
from torch import nn
# Define a MLP class 
class MLP(nn.Module):
  #  Declare layers with model parameters , Two full connection layers are declared here 
  def __init__(self, **kwargs):
    super(MLP, self).__init__(**kwargs) # call MLP Parent class Block Constructor to do the necessary initialization 
    self.hidden = nn.Linear(784, 256)
    self.act = nn.ReLU()
    self.output = nn.Linear(256,10)
    
   #  Define the forward calculation of the model , That is, how to input x The calculation returns the required model output 
  def forward(self, x):
    o = self.act(self.hidden(x))
    return self.output(o)  
# Instantiation 
X = torch.rand(2,784)
net = MLP()
net(X)

Build your own Layer

• Without model parameter layer

class MyLayer(nn.Module):
    def __init__(self, **kwargs):
        super(MyLayer, self).__init__(**kwargs)
    def forward(self, x):
        return x - x.mean()

• Including model parameter layer

class MyListDense(nn.Module):
    def __init__(self):
        super(MyListDense, self).__init__()
        self.params = nn.ParameterList([nn.Parameter(torch.randn(4, 4)) for i in range(3)])
        self.params.append(nn.Parameter(torch.randn(4, 1)))

    def forward(self, x):
        for i in range(len(self.params)):
            x = torch.mm(x, self.params[i])
        return x


class MyDictDense(nn.Module):
    def __init__(self):
        super(MyDictDense, self).__init__()
        self.params = nn.ParameterDict({
    
                'linear1': nn.Parameter(torch.randn(4, 4)),
                'linear2': nn.Parameter(torch.randn(4, 1))
        })
        self.params.update({
    'linear3': nn.Parameter(torch.randn(4, 2))}) #  newly added 

    def forward(self, x, choice='linear1'):
        return torch.mm(x, self.params[choice])

• 2 Convolution layer

def corr2d(X, K): 
    h, w = K.shape
    X, K = X.float(), K.float()
    Y = torch.zeros((X.shape[0] - h + 1, X.shape[1] - w + 1))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            Y[i, j] = (X[i: i + h, j: j + w] * K).sum()
    return Y

#  Two dimensional convolution 
class Conv2D(nn.Module):
    def __init__(self, kernel_size):
        super(Conv2D, self).__init__()
        # Random initialization 
        self.weight = nn.Parameter(torch.randn(kernel_size))
        self.bias = nn.Parameter(torch.randn(1))

    def forward(self, x):
        return corr2d(x, self.weight) + self.bias

• Pooling layer

def pool2d(X, pool_size, mode='max'):
    p_h, p_w = pool_size
    Y = torch.zeros((X.shape[0] - p_h + 1, X.shape[1] - p_w + 1))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
            if mode == 'max':
                Y[i, j] = X[i: i + p_h, j: j + p_w].max()
            elif mode == 'avg':
                Y[i, j] = X[i: i + p_h, j: j + p_w].mean()
    return Y

Build a model

• LeNet Model

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        #  The input image channel：1; Output channel：6;5x5 Convolution kernel 
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # 2x2 Max pooling
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        #  If it's a square matrix , You can use only one number to define 
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  #  Remove all other dimensions of the batch dimension 
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

• AlexNet

class AlexNet(nn.Module):
    def __init__(self):
        super(AlexNet, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 96, 11, 4), # in_channels, out_channels, kernel_size, stride, padding
            nn.ReLU(),
            nn.MaxPool2d(3, 2), # kernel_size, stride
            #  Reduce the convolution window , Use padding as 2 To make the height and width of input and output consistent , And increase the number of output channels 
            nn.Conv2d(96, 256, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(3, 2),
            #  continuity 3 Convolution layers , And use smaller convolution windows . Except for the final convolution layer , Further increase the number of output channels .
            #  After the first two convolution layers, no pooling layer is used to reduce the input height and width 
            nn.Conv2d(256, 384, 3, 1, 1),
            nn.ReLU(),
            nn.Conv2d(384, 384, 3, 1, 1),
            nn.ReLU(),
            nn.Conv2d(384, 256, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(3, 2)
        )
         #  Here, the output ratio of the full connection layer is LeNet A few times as big as . Use the discard layer to alleviate over fitting 
        self.fc = nn.Sequential(
            nn.Linear(256*5*5, 4096),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(4096, 4096),
            nn.ReLU(),
            nn.Dropout(0.5),
            #  Output layer . Because of the use of Fashion-MNIST, So the number of categories is 10, Not in the paper 1000
            nn.Linear(4096, 10),
        )

    def forward(self, img):
        feature = self.conv(img)
        output = self.fc(feature.view(img.shape[0], -1))
        return output

Model initialization

Different initialization methods shall be selected for different structures ,pytorch The initialization function is in torch.nn.init in , For details, please refer to Official website .
Traverse , Initialize all module parameters of the model ：

def initialize_weights(self):
	for m in self.modules():
		#  Judge whether it belongs to Conv2d
		if isinstance(m, nn.Conv2d):
			torch.nn.init.xavier_normal_(m.weight.data)
			#  Judge whether there is bias 
			if m.bias is not None:
				torch.nn.init.constant_(m.bias.data,0.3)
		elif isinstance(m, nn.Linear):
			torch.nn.init.normal_(m.weight.data, 0.1)
			if m.bias is not None:
				torch.nn.init.zeros_(m.bias.data)
		elif isinstance(m, nn.BatchNorm2d):
			m.weight.data.fill_(1) 		 
			m.bias.data.zeros_()	

Loss functions are commonly used

name	function	The formula	application
Two class cross entropy loss function	torch.nn.BCELoss(weight=None, size_average=None, reduce=None, reduction=‘mean’)	$$\ell (x,y)=\{\begin{array}{ll}mean(L)& \text{reduction=’mean’}\\ sum(L)& \text{reduction=’sum’}\end{array}$$	Calculate the cross entropy of two classification tasks
Cross entropy loss function	torch.nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction=‘mean’)	$$\mathrm{loss}(x,\text{ class })=-\log \left(\frac{\exp (x[\text{ class }])}{{\sum }_j\exp (x[j])}\right)=-x[\text{ class }]+\log \left(\sum _j\exp (x[j])\right)$$	Many classification
L1 Loss function	torch.nn.L1Loss(size_average=None, reduce=None, reduction=‘mean’)	$$L_n=abs(x_n-y_n)$$	The return question , Returns the absolute value of the error
MSE Loss function	torch.nn.MSELoss(size_average=None, reduce=None, reduction=‘mean’)	$$l_n={\left(x_n-y_n\right)}^2$$	The return question , Returns the square of the error
smooth L1 Loss function	torch.nn.SmoothL1Loss(size_average=None, reduce=None, reduction=‘mean’, beta=1.0)	$$\mathrm{loss}(x,y)=\frac{1}{n}\sum _{i=1}^n{z}_i$$ $$z_i=\{\begin{array}{ll}0.5{\left(x_i-y_i\right)}^2& \text{abs(xi-yi)<1}\\ abs(x_i-y_i)-0.5& \text{else}\end{array}$$	L1 Smooth output , It can reduce the impact of outliers
Negative log likelihood loss of target Poisson distribution	torch.nn.PoissonNLLLoss(log_input=True, full=False, size_average=None, eps=1e-08, reduce=None, reduction=‘mean’)	$$\mathrm{loss}(x,y)=\{\begin{array}{ll}{e}^{x_n}-x_n{y}_n& \text{log\_input=True}\\ x_n-y_n\cdot \log \left(x_n+\text{ eps }\right)& \text{log\_input=False}\end{array}$$	The negative log likelihood loss function of Poisson distribution
KL The divergence	torch.nn.KLDivLoss(size_average=None, reduce=None, reduction=‘mean’, log_target=False)	$$D_{\mathrm{K}\mathrm{L}}(P,Q)=\sum _{i=1}^{n}P\left(x_i\right)\left(\log P\left(x_i\right)-\log Q\left(x_i\right)\right)$$	Distance measure for continuous distribution , And it is usually useful to regress the discrete continuous output spatial distribution
MarginRankingLoss	torch.nn.MarginRankingLoss(margin=0.0, size_average=None, reduce=None, reduction=‘mean’)	$$\mathrm{loss}(x1,x2,y)=\max (0,-y\ast (x1-x2)+\mathrm{margin})$$	Used to sort tasks
Multi label boundary loss function	torch.nn.MultiLabelMarginLoss(size_average=None, reduce=None, reduction=‘mean’)	$$\mathrm{loss}(x,y)=\sum _{ij}\frac{\max (0,1-x[y[j]]-x[i])}{x\cdot \mathrm{size}(0)}$$	Multi label classification
Two class loss function	torch.nn.SoftMarginLoss(size_average=None, reduce=None, reduction=‘mean’)	$$\mathrm{loss}(x,y)=\sum _i\frac{\log (1+\exp (-y[i]\cdot x[i]))}{x\cdot \mathrm{nelement}()}$$	Loss function of binary logic
Multi category folding loss	torch.nn.MultiMarginLoss(p=1, margin=1.0, weight=None, size_average=None, reduce=None, reduction=‘mean’)	$$\mathrm{loss}(x,y)=\frac{{\sum }_i\max (0,\mathrm{margin}-x[y]+x[i]{)}^p}{x\cdot \mathrm{size}(0)}$$	Multiple classification problem
Triplet loss	torch.nn.TripletMarginLoss(margin=1.0, p=2.0, eps=1e-06, swap=False, size_average=None, reduce=None, reduction=‘mean’)	$$L(a,p,n)=\max \left\{d\left(a_i,p_i\right)-d\left(a_i,n_i\right)+\mathrm{margin},0\right\}$$	Triple similarity
HingeEmbeddingLoss	torch.nn.HingeEmbeddingLoss(margin=1.0, size_average=None, reduce=None, reduction=‘mean’)	$${\ell }_n=\{\begin{array}{ll}{x}_n& \text{yn=1}\\ \max \left\{0,\Delta -x_n\right\}& \text{yn=-1}\end{array}$$x Is the absolute value of the difference between the two inputs	Judge the similarity between two inputs
Cosine similarity	torch.nn.CosineEmbeddingLoss(margin=0.0, size_average=None, reduce=None, reduction=‘mean’)	$${\ell }_n=\{\begin{array}{ll}1-\cos \left(x_1,x_2\right)& \text{yn=1}\\ \max \left\{0,\cos \left(x_1,x_2\right)-\text{ margin }\right\}& \text{yn=-1}\end{array}$$	Two vectors do cosine similarity
CTC Loss function	torch.nn.CTCLoss(blank=0, reduction=‘mean’, zero_infinity=False)	/	Time series classification problem

model training 、 Verification and testing

In the process of model training, the parameters of back propagation can be modified ; And verification / Testing process , The parameters are the same .

Training process

• technological process ：

  • Declare the training process
  • Reading data （ It can be put in batches to GPU Conduct ）
  • Zero the optimizer's gradient
  • Training
  • Calculate the loss function
  • take loss Back propagation back to the network
  • Use the optimizer to update model parameters

• Code ：

def train(epoch):
    model.train()
    train_loss = 0
    for data, label in train_loader:
        data, label = data.cuda(), label.cuda() # Put to GPU
        optimizer.zero_grad() # Zero the optimizer's gradient 
        output = model(data) # take data Into the model for training 
        loss = criterion(label, output) # Calculate the loss function 
        loss.backward() # take loss Back propagation back to the network 
        optimizer.step() # Use the optimizer to update model parameters 
        train_loss += loss.item()*data.size(0)
    train_loss = train_loss/len(train_loader.dataset)
		print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch, train_loss))

verification / Testing process

• technological process ：

  • Declare the validation process
  • Reading data （ It can be put in batches to GPU Conduct ）
  • Training
  • Calculate the loss function

• Code ：

def val(epoch):       
    model.eval()
    val_loss = 0
    with torch.no_grad():
        for data, label in val_loader:
            data, label = data.cuda(), label.cuda()
            output = model(data)
            preds = torch.argmax(output, 1)
            loss = criterion(output, label)
            val_loss += loss.item()*data.size(0)
            running_accu += torch.sum(preds == label.data)
    val_loss = val_loss/len(val_loader.dataset)
    print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch, val_loss))

Optimizer

• pytorch library ：torch.optim

Optimizer	explain
LBFGS	Quasi Newton method
SGD	Stochastic gradient descent
ASGD	The average random gradient decreases
Adagrad	Adaptive learning rate , Increase the second-order momentum
Adadelta	Adagrad An extension of , Don't rely on global learning rates
Rprop	Elastic back propagation
RMSprop	Adadelta special case , about RNN The effect is very good
Adam	First order momentum + Second order momentum
Adamax	The boundary range ratio of learning rate Adam Simple
NAdam	with Nesterov Of momentum terms Adam
SparseAdam	For sparse tensors Adam
RAdam	Provide automated variance attenuation , Eliminated during training warmup The need for manual tuning involved
AdamW	Adam+L2 Regular

• application

from torch import optim
from torchvision.models import resnet18
# Model 
net = resnet18()
# Optimizer parameters for different layers 
optimizer = optim.SGD([{
    'params':net.fc.parameters()},
    {
    'params':net.layer4[0].conv1.parameters(),'lr':1e-2}], 
    lr=1e-5)
for epoch in range(EPOCH):
	...
	optimizer.zero_grad()  # Zero gradient 
	loss = ...             # Calculation loss
	loss.backward()        #BP Back propagation 
	optimizer.step()       # Gradient update 

example ：FashionMNIST Fashion classification

Basic library preparation

import os
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader

device = torch.device("cuda:1" if torch.cuda.is_available() else "cpu")

batch_size = 256
num_workers = 4   
lr = 1e-4
epochs = 20

Data loading

Define data format conversion

from torchvision import transforms

image_size = 28
data_transform = transforms.Compose([
    transforms.ToPILImage(),   #  This step depends on how the subsequent data is read , If you use a built-in dataset, you don't need 
    transforms.Resize(image_size),
    transforms.ToTensor()
])

Data read in

• Method 1 ： Remote dataset download

from torchvision import datasets

train_data = datasets.FashionMNIST(root='./', train=True, download=True, transform=data_transform)
test_data = datasets.FashionMNIST(root='./', train=False, download=True, transform=data_transform)

• Method 2 ：csv Local data loading

# csv Data download link ：https://www.kaggle.com/zalando-research/fashionmnist
class FMDataset(Dataset):
    def __init__(self, df, transform=None):
        self.df = df
        self.transform = transform
        self.images = df.iloc[:,1:].values.astype(np.uint8)
        self.labels = df.iloc[:, 0].values
        
    def __len__(self):
        return len(self.images)
    
    def __getitem__(self, idx):
        image = self.images[idx].reshape(28,28,1)
        label = int(self.labels[idx])
        if self.transform is not None:
            image = self.transform(image)
        else:
            image = torch.tensor(image/255., dtype=torch.float)
        label = torch.tensor(label, dtype=torch.long)
        return image, label

train_df = pd.read_csv("./FashionMNIST/fashion-mnist_train.csv")
test_df = pd.read_csv("./FashionMNIST/fashion-mnist_test.csv")
train_data = FMDataset(train_df, data_transform)
test_data = FMDataset(test_df, data_transform)

Data loading

train_loader = DataLoader(train_data, batch_size=batch_size, shuffle=True, num_workers=num_workers, drop_last=True)
test_loader = DataLoader(test_data, batch_size=batch_size, shuffle=False, num_workers=num_workers)

data validation

import matplotlib.pyplot as plt

image, label = next(iter(train_loader))
print(image.shape, label.shape)
plt.imshow(image[0][0], cmap="gray")

CNN model building

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 32, 5),
            nn.ReLU(),
            nn.MaxPool2d(2, stride=2),
            nn.Dropout(0.3),
            nn.Conv2d(32, 64, 5),
            nn.ReLU(),
            nn.MaxPool2d(2, stride=2),
            nn.Dropout(0.3)
        )
        self.fc = nn.Sequential(
            nn.Linear(64*4*4, 512),
            nn.ReLU(),
            nn.Linear(512, 10)
        )
        
    def forward(self, x):
        x = self.conv(x)
        x = x.view(-1, 64*4*4)
        x = self.fc(x)
        # x = nn.functional.normalize(x)
        return x

model = Net()
model = model.cuda()

model training

Define the loss function

criterion = nn.CrossEntropyLoss()

Define optimizer

optimizer = optim.Adam(model.parameters(), lr=0.001)

Training and verification

def train(epoch):
    model.train()
    train_loss = 0
    for data, label in train_loader:
        data, label = data.cuda(), label.cuda()
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, label)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()*data.size(0)
    train_loss = train_loss/len(train_loader.dataset)
    print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch, train_loss))

def val(epoch):       
    model.eval()
    val_loss = 0
    gt_labels = []
    pred_labels = []
    with torch.no_grad():
        for data, label in test_loader:
            data, label = data.cuda(), label.cuda()
            output = model(data)
            preds = torch.argmax(output, 1)
            gt_labels.append(label.cpu().data.numpy())
            pred_labels.append(preds.cpu().data.numpy())
            loss = criterion(output, label)
            val_loss += loss.item()*data.size(0)
    val_loss = val_loss/len(test_loader.dataset)
    gt_labels, pred_labels = np.concatenate(gt_labels), np.concatenate(pred_labels)
    acc = np.sum(gt_labels==pred_labels)/len(pred_labels)
    print('Epoch: {} \tValidation Loss: {:.6f}, Accuracy: {:6f}'.format(epoch, val_loss, acc))

for epoch in range(1, epochs+1):
    train(epoch)
    val(epoch)

Save the model

save_path = "./FahionModel.pkl"
torch.save(model, save_path)

Reference resources

1. datawhale： Explain profound theories in simple language pytorch
2. pytorch Official website