Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Last update: Nov 30, 2022

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

"Towards the Generalization of Contrastive Self-Supervised Learning", Weiran Huang, Mingyang Yi and Xuyang Zhao, arXiv:2111.00743, 2021.

The most critical argument we made in our paper is that the quality of data augmentation exhibits great impact on the quality of contrastive-learned encoder. The data augmentation with sharper intra-class concentration enables the model to have better generalization on downstream tasks. We verify it through a variety of experiments in this repository.

Supported methods

SimCLR
Barlow Twins
MoCo
SimSiam

Installation

pip install -r requirement.txt

Dependencies

torch==1.4.0
torchvision==0.5.0
pytorch-lightning==1.3.8
lightly==1.0.8 (important!)

Evaluation

KNN evaluation protocol. Code from here.

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

Example: python main.py --model=twins --epoch=800 --batch=512 --round=3 --augs=abcde

(a)	(b)	(c)	(d)	(e)	SimCLR	Barlow Twins
✓	✓	✓	✓	✓	89.92 ± 0.05	83.93 ± 0.57
✓	✓	✓	✓	×	88.41 ± 0.11	83.37 ± 0.43
✓	✓	✓	×	×	83.62 ± 0.19	73.70 ± 0.99
✓	✓	×	×	×	62.91 ± 0.25	49.56 ± 0.11
✓	×	×	×	×	62.37 ± 0.09	48.54 ± 0.29

Augmentation operations include:

(a) random cropping with a scaling factor chosen in [0.08, 1.0];

(b) random Gaussian blur with a probability 0.5;

(d) color distortion with a probability of 0.8 and with strength of [0.4, 0.4, 0.4, 0.1];

(e) random horizontal flipping with a probability of 0.5.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Example: python main.py --model=simclr --epoch=800 --batch=512 --round=3 --augs=color --strength=1

Color Distortion Strength	SimCLR	Barlow Twins
1/8	73.60 ± 0.11	61.13 ± 2.81
1/4	76.25 ± 0.16	68.30 ± 0.15
1/2	78.49 ± 0.09	72.76 ± 1.50
1	82.64 ± 0.57	78.79 ± 0.54

Acknowledgement

This code is based on:

Citation

If you find our work useful in your research, please consider citing:

@misc{huang2021generalization,
      title={Towards the Generalization of Contrastive Self-Supervised Learning}, 
      author={Weiran Huang and Mingyang Yi and Xuyang Zhao},
      year={2021},
      eprint={2111.00743},
      archivePrefix={arXiv}
}

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Nyström Attention Implementation of Nyström Self-attention, from the paper Nyströmformer. Yannic Kilcher video Install $ pip install nystrom-attention

95 Jan 2, 2023

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

112 Dec 16, 2022

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow

201 Dec 21, 2022

Official pytorch implementation of paper

Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Related tags

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

Supported methods

Installation

Dependencies

Evaluation

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Acknowledgement

Citation

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Implementation of Barlow Twins paper

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

Functional TensorFlow Implementation of Singular Value Decomposition for paper Fast Graph Learning

Releases(v0.2.1)

v0.2.1(Jun 16, 2022)

v0.2.0(Jun 4, 2022)

v0.1.0(Jun 4, 2022)

Owner

Weiran Huang

A curated list of awesome Machine Learning frameworks, libraries and software.

Official implementation for paper: Feature-Style Encoder for Style-Based GAN Inversion

A Joint Video and Image Encoder for End-to-End Retrieval

🐸STT integration examples

An algorithmic trading bot that learns and adapts to new data and evolving markets using Financial Python Programming and Machine Learning.

百度2021年语言与智能技术竞赛机器阅读理解Pytorch版baseline

Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

VisionKG: Vision Knowledge Graph

🕵 Artificial Intelligence for social control of public administration

Neural Koopman Lyapunov Control

This is a Python wrapper for TA-LIB based on Cython instead of SWIG.

classify fashion-mnist dataset with pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

A paper using optimal transport to solve the graph matching problem.

MDETR: Modulated Detection for End-to-End Multi-Modal Understanding

Vis2Mesh: Efficient Mesh Reconstruction from Unstructured Point Clouds of Large Scenes with Learned Virtual View Visibility ICCV2021

Real-CUGAN - Real Cascade U-Nets for Anime Image Super Resolution

Repository for "Toward Practical Monocular Indoor Depth Estimation" (CVPR 2022)

[CVPR2021] DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets

Code from the paper "High-Performance Brain-to-Text Communication via Handwriting"