Self supervised learning (SSL)

Catalog

supervise 、 Unsupervised 、 Self supervised learning

Self supervised learning

The core idea

The main method

Mainstream classification

generative methods

contrastive methods

---

supervise 、 Unsupervised 、 Self supervised learning

The main difference between supervised and unsupervised is whether the model needs Manual marking Of Tag information .

The main purpose of self supervised learning is to learn richer semantic representations .

Assess the ability of self supervised learning , Mainly through Pretrain-Fintune The pattern of .

Supervised Pretrain - Finetune technological process ：

1. From a lot of Tagged data Training on , Get the pre training model ,

2. For new downstream tasks （Downstream task）, The parameters we will learn （ For example, the parameters of the layer before the output layer ） Migration , On new tagged tasks 「 fine-tuning 」, So we can get a network that can adapt to the new task .

Self supervised Pretrain - Finetune technological process ：

1. From a large number of Unlabeled data Pass through pretext Training network （ Automatically construct supervision information in data ）, Get the pre training model

2. For new downstream tasks , Like supervised learning , Transfer the learned parameters and fine tune it .

Therefore, the ability of self supervised learning is mainly reflected by the performance of downstream tasks .

supervised learning Characteristics ：

1. For every picture , The machine predicts a category Or is it bounding box
2. Training data are manually labeled
3. Each sample can only provide very little information ( such as 1024 individual categories Only 10 bits Information about )

self-supervised learning Characteristics ：

1. For a picture , The machine can predict any part （ Automatically build supervision signals ）
2. For video , Can predict future frames
3. Each sample can provide a lot of information

Self supervised learning

The core idea

 Self-Supervised Learning First, the parameters are trained from a blank sheet of paper to the initial shape , Then from preliminary shaping training to complete shaping .1. Training to the initial shape of things , Visual Representation.

2. Then according to the downstream tasks (Downstream Tasks) Different to use Tagged datasets Train the parameters to full shape ,

Then the amount of data set used at this time is not too much , Because the parameter has passed the second step 1 The training stage is almost the same .

The first phase does not involve any downstream tasks , Is to pre train with a pile of unlabeled data , There are no specific tasks , This word is expressed in the official language called ：in a task-agnostic way.

The second phase involves downstream tasks , Is to take a pile of labeled data to the downstream task Fine-tune, This word is expressed in the official language called ：in a task-specific way.

  In the field

Self-Supervised Learning Not just in NLP field , stay CV, There are also many classic works in the field of voice , It can be divided into 3 class ：Data Centric, Prediction ( Also called Generative) and Contrastive.

The main method

1. Based on context （Context based）

2. Based on timing （Temporal Based）

3. Based on contrast （Contrastive Based）

Mainstream classification

generative methods

Can rebuild => Can extract good feature expression

eg：MAE、BERT
 

contrastive methods

Distinguish different inputs in the feature space

Reference link ：

Self supervised learning | (1) Self-supervised Learning introduction _CoreJT The blog of -CSDN Blog _ Self supervised learning

Self supervised learning | (2) Article to read Self-Supervised Learning_CoreJT The blog of -CSDN Blog