Integration of revolution and batch normalization

Theoretical calculation

At present CNN The basic composition unit of convolution layer is standard ：Conv + BN +ReLU Three sub modules . But in fact, in the reasoning stage of the network , Can be BN The operation of layer is integrated into Conv Layer , Reduce the amount of computation , Speed up reasoning . In essence, the parameters of convolution kernel are modified , Without adding Conv While calculating the amount of layers , Omit BN Calculation amount of layers . The formula is derived as follows .

conv Parameters of the layer

BN Parameters of the layer

Suppose the input is x, be x->Conv->BN The output of is ：

Make a simple formula deformation ：

Code implementation

In actual use , First of all, we need to position conv and bn The location of , Replace or delete according to the actual situation BN layer . In this embodiment , Split the model base with open source https://github.com/qubvel/segmentation_models.pytorch Conduct a fusion experiment for the case , Yes BN Layer has been replaced .

class Conv2dReLU(nn.Sequential):
    def __init__(
            self,
            in_channels,
            out_channels,
            kernel_size,
            padding=0,
            stride=1,
            use_batchnorm=True,
    ):

        if use_batchnorm == "inplace" and InPlaceABN is None:
            raise RuntimeError(
                "In order to use `use_batchnorm='inplace'` inplace_abn package must be installed. "
                + "To install see: https://github.com/mapillary/inplace_abn"
            )

        conv = nn.Conv2d(
            in_channels,
            out_channels,
            kernel_size,
            stride=stride,
            padding=padding,
            bias=not (use_batchnorm),
        )
        relu = nn.ReLU(inplace=True)

        if use_batchnorm == "inplace":
            bn = InPlaceABN(out_channels, activation="leaky_relu", activation_param=0.0)
            relu = nn.Identity()

        elif use_batchnorm and use_batchnorm != "inplace":
            bn = nn.BatchNorm2d(out_channels)

        else:
            bn = nn.Identity()

        super(Conv2dReLU, self).__init__(conv, bn, relu)

from turtle import forward
from torch.fx.experimental.optimization import fuse
import torch
import torch.nn as nn
import time

import segmentation_models_pytorch.base.modules as md

from utils.torchUtils import fuse_conv_and_bn


def fuseModel(model):  # fuse model Conv2d() + BatchNorm2d() layers
    for m in model.modules():
        if isinstance(m, (md.Conv2dReLU)) and isinstance(m[1], (nn.BatchNorm2d)):
            m[0] = fuse_conv_and_bn(m[0], m[1])  # update conv
            m[1] = nn.Identity()
            count += 1
    return model