Parameter-ensemble-differential-evolution - Shows how to do parameter ensembling using differential evolution.

Last update: May 04, 2022

Overview

Ensembling parameters with differential evolution

This repository shows how to ensemble parameters of two trained neural networks using differential evolution. The steps followed are as follows:

Train two networks (architecturally same) on the same dataset (CIFAR-10 used here) but from two different random initializations.
Ensemble their weights using the following formulae:
```
w_t = w_o * ema + (1 - ema) * w_p
```
w_o and w_p represents the learned of a neural network.
Randomly initialize a network (same architecture as above) and populate its parameters w_t using the above formulae.

ema is usually chosen by the developer in an empirical manner. This project uses differential evolution to find it.

Below are the top-1 accuracies (on CIFAR-10 test set) of two individually trained two models along with their ensembled variant:

Model one: 63.23%
Model two: 63.42%
Ensembled: 63.35%

With the more conventional average prediction ensembling, I was able to get to 64.92%. This is way better than what I got by ensembling the parameters. Nevertheless, the purpose of this project was to just try out an idea.

Reproducing the results

Ensure the requirements.txt is satisfied. Then train two models with ensuring your working directory is at the root of this project:

$ git clone https://github.com/sayakpaul/parameter-ensemble-differential-evolution
$ cd parameter-ensemble-differential-evolution
$ pip install -qr requirements.txt
$ for i in `seq 1 2`; python train.py; done

Then just follow the ensemble-parameters.ipynb notebook. You can also use the networks I trained. Instructions are available inside the notebook.

Parameter-ensemble-differential-evolution - Shows how to do parameter ensembling using differential evolution.

Related tags

Overview

Ensembling parameters with differential evolution

Reproducing the results

References

You might also like...

Neural Ensemble Search for Performant and Calibrated Predictions

An Ensemble of CNN (Python 3.5.1 Tensorflow 1.3 numpy 1.13)

zeus is a Python implementation of the Ensemble Slice Sampling method.

Pytorch implementation of SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning

This Jupyter notebook shows one way to implement a simple first-order low-pass filter on sampled data in discrete time.

A fast Evolution Strategy implementation in Python

Code for the paper Task Agnostic Morphology Evolution.

Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Releases(v0.1.0)

v0.1.0(Jan 2, 2022)

Owner

Sayak Paul

Official implementation of the paper ``Unifying Nonlocal Blocks for Neural Networks'' (ICCV'21)

Multiple-criteria decision-making (MCDM) with Electre, Promethee, Weighted Sum and Pareto

Python scripts form performing stereo depth estimation using the HITNET model in ONNX.

KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control

Deep Inside Convolutional Networks - This is a caffe implementation to visualize the learnt model

Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks

⚖️🔁🔮🕵️‍♂️🦹🖼️ Code for Measuring the Contribution of Multiple Model Representations in Detecting Adversarial Instances paper.

Evaluation suite for large-scale language models.

Tensorflow python implementation of "Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos"

sense-py-AnishaBaishya created by GitHub Classroom

Fast Axiomatic Attribution for Neural Networks (NeurIPS*2021)

level1-image-classification-level1-recsys-09 created by GitHub Classroom

Byte-based multilingual transformer TTS for low-resource/few-shot language adaptation.

Keras Implementation of The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation by (Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, Yoshua Bengio)

Neural style transfer as a class in PyTorch

A CNN model to detect hand gestures.

A simple and extensible library to create Bayesian Neural Network layers on PyTorch.

Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021)

Implementation of "A Deep Learning Loss Function based on Auditory Power Compression for Speech Enhancement" by pytorch

HomoInterpGAN - Homomorphic Latent Space Interpolation for Unpaired Image-to-image Translation

Parameter-ensemble-differential-evolution - Shows how to do parameter ensembling using differential evolution.

Related tags

Overview

Ensembling parameters with differential evolution

Reproducing the results

References

You might also like...

Neural Ensemble Search for Performant and Calibrated Predictions

An Ensemble of CNN (Python 3.5.1 Tensorflow 1.3 numpy 1.13)

zeus is a Python implementation of the Ensemble Slice Sampling method.

Pytorch implementation of SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning

This Jupyter notebook shows one way to implement a simple first-order low-pass filter on sampled data in discrete time.

A fast Evolution Strategy implementation in Python

Code for the paper Task Agnostic Morphology Evolution.

Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Releases(v0.1.0)

v0.1.0(Jan 2, 2022)

Owner

Sayak Paul

Official implementation of the paper ``Unifying Nonlocal Blocks for Neural Networks'' (ICCV'21)

Multiple-criteria decision-making (MCDM) with Electre, Promethee, Weighted Sum and Pareto

Python scripts form performing stereo depth estimation using the HITNET model in ONNX.

KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control

Deep Inside Convolutional Networks - This is a caffe implementation to visualize the learnt model

Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks

⚖️🔁🔮🕵️‍♂️🦹🖼️ Code for *Measuring the Contribution of Multiple Model Representations in Detecting Adversarial Instances* paper.

Evaluation suite for large-scale language models.

Tensorflow python implementation of "Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos"

sense-py-AnishaBaishya created by GitHub Classroom

Fast Axiomatic Attribution for Neural Networks (NeurIPS*2021)

level1-image-classification-level1-recsys-09 created by GitHub Classroom

Byte-based multilingual transformer TTS for low-resource/few-shot language adaptation.

Keras Implementation of The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation by (Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, Yoshua Bengio)

Neural style transfer as a class in PyTorch

A CNN model to detect hand gestures.

A simple and extensible library to create Bayesian Neural Network layers on PyTorch.

Tracing Versus Freehand for Evaluating Computer-Generated Drawings (SIGGRAPH 2021)

Implementation of "A Deep Learning Loss Function based on Auditory Power Compression for Speech Enhancement" by pytorch

HomoInterpGAN - Homomorphic Latent Space Interpolation for Unpaired Image-to-image Translation

⚖️🔁🔮🕵️‍♂️🦹🖼️ Code for Measuring the Contribution of Multiple Model Representations in Detecting Adversarial Instances paper.