[NIPS 2021] UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration.

Last update: Jun 29, 2022

Related tags

Deep Learning uota

Overview

UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

This repository is the official PyTorch implementation of UOTA (Unsupervised OuTlier Arbitration).

0 Requirements

Python 3.6
PyTorch install = 1.6.0
torchvision install = 0.7.0
CUDA 10.1
Apex with CUDA extension
Other dependencies: opencv-python, scipy, pandas, numpy

1 Pretraining

We release a demo to pretrain ResNet50 on ImageNet1K with SwAV+UOTA pretrained models.

1.1 SwAV+UOTA pretrain

To train SwAV+UOTA on a single node with 4 gpus for 200 epochs, run:

DATASET_PATH="path/to/ImageNet1K/train"
EXPERIMENT_PATH="path/to/experiment"

python -m torch.distributed.launch --nproc_per_node=4 main_uota.py \
--data_path ${DATASET_PATH} \
--nmb_crops 2 6 \
--size_crops 224 96 \
--min_scale_crops 0.14 0.05 \
--max_scale_crops 1. 0.14 \
--crops_for_assign 0 1 \
--use_pil_blur true \
--epochs 200 \
--warmup_epochs 0 \
--batch_size 64 \
--base_lr 0.6 \
--final_lr 0.0006 \
--uota_tau 350. \
--epoch_uota_starts 100 \
--wd 0.000001 \
--use_fp16 true \
--dist_url "tcp://localhost:40000" \
--arch uota_r50 \
--sync_bn pytorch \
--dump_path ${EXPERIMENT_PATH}

2 Linear Evaluation

To train a linear classifier on frozen features out of deep network pretrained via various self-supervised pretraining methods, run:

DATASET_PATH="path/to/ImageNet1K"
EXPERIMENT_PATH="path/to/experiment"
LINCLS_PATH="path/to/lincls"

python -m torch.distributed.launch --nproc_per_node=4 eval_linear.py \
--data_path ${DATASET_PATH} \
--arch resnet50 \
--lr 1.2 \
--dump_path ${LINCLS_PATH} \
--pretrained ${EXPERIMENT_PATH}/swav_uota_r50_e200_pretrained.pth \
--batch_size 64 \
--num_classes 100 \

3 Results

To compare with SwAV fairly, we provide a SwAV+UOTA model with ResNet-50 architecture pretrained on ImageNet1K for 200 epochs, and release the pretrained model and the linear classier.

method	epochs	batch-size	multi-crop	ImageNet1K top-1 acc.	pretrained model	linear classifier
SwAV	200	256	2x224 + 6x96	72.7	/	/
SwAV + UOTA	200	256	2x224 + 6x96	73.5	pretrained	linear

4 Citation

@InProceedings{wang2021NeurIPS,
  title={Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration},
  author={Wang, Yu and Lin, Jingyang and Zou, Jingjing and Pan, Yingwei and Yao, Ting and Mei, Tao},
  booktitle={NeurIPS},
  year={2021},
}

You might also like...

PyTorch implementation of spectral graph ConvNets, NIPS’16

Graph ConvNets in PyTorch October 15, 2017 Xavier Bresson http://www.ntu.edu.sg/home/xbresson https://github.com/xbresson https://twitter.com/xbresson

287 Jan 4, 2023

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

Value Iteration Networks in PyTorch Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. Value Iteration Networks. Neural Information Processing

75 Nov 24, 2022

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

VIN: Value Iteration Networks A quick thank you A few others have released amazing related work which helped inspire and improve my own implementation

297 Dec 26, 2022

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

16 Nov 4, 2020

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation This repository is the official implementation of CVPR 2021 paper:

9 Nov 14, 2022

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

[NIPS 2021] UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration.

Related tags

Overview

UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

0 Requirements

1 Pretraining

1.1 SwAV+UOTA pretrain

2 Linear Evaluation

3 Results

4 Citation

You might also like...

PyTorch implementation of spectral graph ConvNets, NIPS’16

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

A gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor.

Outlier Exposure with Confidence Control for Out-of-Distribution Detection

Releases(v1.0.0)

v1.0.0(Oct 26, 2021)

Owner

Make your master artistic punk avatar through machine learning world famous paintings.

Semantic code search implementation using Tensorflow framework and the source code data from the CodeSearchNet project

Suite of 500 procedurally-generated NLP tasks to study language model adaptability

Action Segmentation Evaluation

Code for EMNLP 2021 main conference paper "Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification"

Session-aware Item-combination Recommendation with Transformer Network

Code for our paper "SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization", ACL 2021

KwaiRec: A Fully-observed Dataset for Recommender Systems (Density: Almost 100%)

ROMP: Monocular, One-stage, Regression of Multiple 3D People, ICCV21

StyleGAN - Official TensorFlow Implementation

This is a collection of our NAS and Vision Transformer work.

Assessing the Influence of Models on the Performance of Reinforcement Learning Algorithms applied on Continuous Control Tasks

Face Mask Detector by live camera using tensorflow-keras, openCV and Python

A PyTorch implementation of "Pathfinder Discovery Networks for Neural Message Passing"

A U-Net combined with a variational auto-encoder that is able to learn conditional distributions over semantic segmentations.

Video Matting via Consistency-Regularized Graph Neural Networks

Lua-parser-lark - An out-of-box Lua parser written in Lark

LTR_CrossEncoder: Legal Text Retrieval Zalo AI Challenge 2021

Attentive Implicit Representation Networks (AIR-Nets)

R interface to fast.ai