PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Last update: Dec 19, 2022

Related tags

Deep Learning SAQ

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

2021.11.23: We release the source code of SAQ.

Setup the environments

Clone the repository locally:

git clone https://github.com/zhuang-group/SAQ

Install pytorch 1.8+, tensorboard and prettytable

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install tensorboard
pip install prettytable

Data preparation

ImageNet

Download the ImageNet 2012 dataset from here, and prepare the dataset based on this script.
Change the dataset path in link_imagenet.py and link the ImageNet-100 by

python link_imagenet.py

CIFAR-100

Download the CIFAR-100 dataset from here.

After downloading ImageNet and CIFAR-100, the file structure should look like:

dataset
├── imagenet
    ├── train
    │   ├── class1
    │   │   ├── img1.jpeg
    │   │   ├── img2.jpeg
    │   │   └── ...
    │   ├── class2
    │   │   ├── img3.jpeg
    │   │   └── ...
    │   └── ...
    └── val
        ├── class1
        │   ├── img4.jpeg
        │   ├── img5.jpeg
        │   └── ...
        ├── class2
        │   ├── img6.jpeg
        │   └── ...
        └── ...
├── cifar100
    ├── cifar-100-python
    │   ├── meta
    │   ├── test
    │   ├── train
    │   └── ...
    └── ...

Training

Fixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train low-precision models.

To train low-precision ResNet-20 on CIFAR-100, run:

sh script/train_qsam_cifar_r20.sh

To train low-precision ResNet-18 on ImageNet, run:

sh script/train_qsam_imagenet_r18.sh

Mixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train the configuration generator.

To train the configuration generator of ResNet-20 on CIFAR-100, run:

sh script/train_generator_cifar_r20.sh

To train the configuration generator on ImageNet, run:

sh script/train_generator_imagenet_r18.sh

After training the configuration generator, run following commands to fine-tune the resulting models with the obtained bitwidth configurations on CIFAR-100 and ImageNet.

sh script/finetune_cifar_r20.sh

sh script/finetune_imagenet_r18.sh

Results on CIFAR-100

Network	Method	Bitwidth	BOPs (M)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-20	SAQ	4	674.6	68.7	91.2
ResNet-20	SAMQ	MP	659.3	68.7	91.2
ResNet-20	SAQ	3	392.1	67.7	90.8
ResNet-20	SAMQ	MP	374.4	68.6	91.2
MobileNetV2	SAQ	4	1508.9	75.6	93.7
MobileNetV2	SAMQ	MP	1482.1	75.5	93.6
MobileNetV2	SAQ	3	877.1	74.4	93.2
MobileNetV2	SAMQ	MP	869.5	75.5	93.7

Results on ImageNet

Network	Method	Bitwidth	BOPs (G)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-18	SAQ	4	34.7	71.3	90.0
ResNet-18	SAMQ	MP	33.7	71.4	89.9
ResNet-18	SAQ	2	14.4	67.1	87.3
MobileNetV2	SAQ	4	5.3	70.2	89.4
MobileNetV2	SAMQ	MP	5.3	70.3	89.4

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Acknowledgement

This repository has adopted codes from SAM, ASAM and ESAM, we thank the authors for their open-sourced code.

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

119 Nov 24, 2022

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

1.4k Jan 1, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

360 Dec 10, 2022

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

9.2k Jan 2, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

359 Jan 5, 2023

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

0 Jul 13, 2021

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

8 Nov 21, 2022

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

157 Dec 11, 2022

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

6 Mar 17, 2022

Comments

Quantize_first_last_layer

Hi! I noticed that in your code, you set bits_weights=8 and bits_activations=32 for first layer as default, it's not what is claimed in your paper " For the first and last layers of all quantized models, we quantize both weights and activations to 8-bit. " And I see an accuracy drop if I adjust the bits_activations to 8 for the first layer, could u please explain what is the reason? Thanks!

opened by mmmiiinnnggg 0
代码问题请求帮助

你好，带佬的代码写的很好，有部分代码不太懂，想请教一下， parser.add_argument( "--arch_bits", type=lambda s: [float(item) for item in s.split(",")] if len(s) != 0 else "", default=" ", help="bits configuration of each layer",

if len(args.arch_bits) != 0: if args.wa_same_bit: set_wae_bits(model, args.arch_bits) elif args.search_w_bit: set_w_bits(model, args.arch_bits) else: set_bits(model, args.arch_bits) show_bits(model) logger.info("Set arch bits to: {}".format(args.arch_bits)) logger.info(model) 这个arch_bits主要是做什么的呢，卡在这里有段时间了

opened by LKAMING97 0

PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Related tags

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

Setup the environments

Data preparation

ImageNet

CIFAR-100

Training

Fixed-precision quantization

Mixed-precision quantization

Results on CIFAR-100

Results on ImageNet

License

Acknowledgement

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

Comments

Quantize_first_last_layer

代码问题请求帮助

Releases(v0.1.1)

v0.1.1(Nov 23, 2021)

v0.1(Nov 23, 2021)

Owner

Zhuang AI Group

Face Detection and Alignment using Multi-task Cascaded Convolutional Networks (MTCNN)

Black-Box-Tuning - Black-Box Tuning for Language-Model-as-a-Service

CenterPoint 3D Object Detection and Tracking using center points in the bird-eye view.

Official Implementation of DE-CondDETR and DELA-CondDETR in "Towards Data-Efficient Detection Transformers"

An easier way to build neural search on the cloud

GluonMM is a library of transformer models for computer vision and multi-modality research

ThunderSVM: A Fast SVM Library on GPUs and CPUs

Intelligent Video Analytics toolkit based on different inference backends.

A toolset for creating Qualtrics-based IAT experiments

Convolutional 2D Knowledge Graph Embeddings resources

To build a regression model to predict the concrete compressive strength based on the different features in the training data.

This Deep Learning Model Predicts that from which disease you are suffering.

FCN (Fully Convolutional Network) is deep fully convolutional neural network architecture for semantic pixel-wise segmentation

Unofficial PyTorch Implementation for HifiFace (https://arxiv.org/abs/2106.09965)

Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch

How to train a CNN to 99% accuracy on MNIST in less than a second on a laptop

Scales, Chords, and Cadences: Practical Music Theory for MIR Researchers

UCSD Oasis platform

Code for the Convolutional Vision Transformer (ConViT)

Capstone-Project-2 - A game program written in the Python language