One of yolox improvements: add CBAM, Se, ECA attention mechanism

front said ： The series released before has a pair of 2020 Published in YOLOv5 Improvement , Many friends consult YOLOX How to improve , This series focuses on YOLOX How to improve is introduced in detail , Basic heel YOLOv5 Agreement , There are subtle differences . Subsequent articles , Focus on YOLOX How to improve is introduced in detail , The purpose is to provide their own meager help and reference for those who need innovation in scientific research or friends who need to achieve better results in engineering projects .

You are welcome to pay attention to more procedural information and answer questions —— WeChat official account ： Artificial intelligence AI Algorithm engineer  

solve the problem ： This article is added with CBAM Take the dual channel attention mechanism as an example , It can make the network pay more attention to the target to be detected , Improve the detection effect , Solve the situation that it is easy to miss detection under the background of complex environment .

Add method ：

First step ： Determine where to add , As a plug and play attention module , Can be added to YOLOX Anywhere in the network . This article adds convolution Conv Module as an example .

The second step ：darknet.py structure CBAM modular .

class SE(nn.Module):
    def __init__(self, channel, ratio=16):
        super(SE, self).__init__()
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Sequential(
                nn.Linear(channel, channel // ratio, bias=False),
                nn.ReLU(inplace=True),
                nn.Linear(channel // ratio, channel, bias=False),
                nn.Sigmoid()
        )

    def forward(self, x):
        b, c, _, _ = x.size()
        y = self.avg_pool(x).view(b, c)
        y = self.fc(y).view(b, c, 1, 1)
        return x * y
class ECA(nn.Module):
    def __init__(self, channel, b=1, gamma=2):
        super(ECA, self).__init__()
        kernel_size = int(abs((math.log(channel, 2) + b) / gamma))
        kernel_size = kernel_size if kernel_size % 2 else kernel_size + 1
        
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.conv = nn.Conv1d(1, 1, kernel_size=kernel_size, padding=(kernel_size - 1) // 2, bias=False) 
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        y = self.avg_pool(x)
        y = self.conv(y.squeeze(-1).transpose(-1, -2)).transpose(-1, -2).unsqueeze(-1)
        y = self.sigmoid(y)
        return x * y.expand_as(x)
class ChannelAttention(nn.Module):
    def __init__(self, in_planes, ratio=8):
        super(ChannelAttention, self).__init__()
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.max_pool = nn.AdaptiveMaxPool2d(1)

        #  utilize 1x1 Convolution instead of full connection 
        self.fc1   = nn.Conv2d(in_planes, in_planes // ratio, 1, bias=False)
        self.relu1 = nn.ReLU()
        self.fc2   = nn.Conv2d(in_planes // ratio, in_planes, 1, bias=False)

        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        avg_out = self.fc2(self.relu1(self.fc1(self.avg_pool(x))))
        max_out = self.fc2(self.relu1(self.fc1(self.max_pool(x))))
        out = avg_out + max_out
        return self.sigmoid(out)

class SpatialAttention(nn.Module):
    def __init__(self, kernel_size=7):
        super(SpatialAttention, self).__init__()

        assert kernel_size in (3, 7), 'kernel size must be 3 or 7'
        padding = 3 if kernel_size == 7 else 1
        self.conv1 = nn.Conv2d(2, 1, kernel_size, padding=padding, bias=False)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        avg_out = torch.mean(x, dim=1, keepdim=True)
        max_out, _ = torch.max(x, dim=1, keepdim=True)
        x = torch.cat([avg_out, max_out], dim=1)
        x = self.conv1(x)
        return self.sigmoid(x)

# CBAM Attention mechanism 
class CBAM(nn.Module):
    def __init__(self, channel, ratio=8, kernel_size=7):
        super(CBAM, self).__init__()
        self.channelattention = ChannelAttention(channel, ratio=ratio)
        self.spatialattention = SpatialAttention(kernel_size=kernel_size)

    def forward(self, x):
        x = x*self.channelattention(x)
        x = x*self.spatialattention(x)
        return x

The third step ：yolo_pafpn.py Register our modification CBAM modular

self.cbam_1 = CBAM(int(in_channels[2] * width)) #  Corresponding dark5 Output 1024 Dimension channel 
self.cbam_2 = CBAM(int(in_channels[1] * width))   #  Corresponding dark4 Output 512 Dimension channel 
self.cbam_3 = CBAM(int(in_channels[0] * width))   #  Corresponding dark3 Output 256 Dimension channel  

def forward(self, input):
        """
        Args:
            inputs: input images.

        Returns:
            Tuple[Tensor]: FPN feature.
        """

        #  backbone
        out_features = self.backbone(input)
        features = [out_features[f] for f in self.in_features]
        [x2, x1, x0] = features

        # 3、 Use the attention mechanism directly on the input feature map 
        x0 = self.cbam_1(x0)
        x1 = self.cbam_2(x1)
        x2 = self.cbam_3(x2)

junction fruit ： I have done a lot of experiments on multiple data sets , For different data sets, the effect is different , There are also differences in the methods of adding locations to the same dataset , You need to experiment . Most cases are effective and improved .

You are welcome to pay attention to more procedural information and answer questions —— WeChat official account ： Artificial intelligence AI Algorithm engineer  

PS：CBAM And other attention mechanisms , Not only can it be added YOLOX, It can also be added to any other deep learning network , Whether it is classification, detection or segmentation , Mainly in the field of computer vision , May have different degrees of improvement effect .

Last , I hope I can powder each other , Be a friend , Learn and communicate together .