Mosaic data enhanced mosaic

————————————————
Copyright notice ： This paper is about CSDN Blogger 「 Brother Dalin 」 The original article of , follow CC 4.0 BY-SA Copyright agreement , For reprint, please attach the original source link and this statement .
Link to the original text ：https://blog.csdn.net/weixin_46142822/article/details/123805663

Reference resources

https://blog.csdn.net/weixin_44751294/article/details/124363703

Other data enhancements ;https://blog.51cto.com/u_11299290/5144799

effect

​（1） Expanded the number of datasets ​

​（2） Increased the number of small samples ​： The large sample is randomly reduced to a small sample , Therefore, the number of small samples is increased .

​（3） Small samples are smaller ：​ Due to the random zoom , After the merger , This results in smaller sample sizes .

Preface

stay Yolo-V4、Yolo-V5 in , Have a very important skill , Namely Mosaic Data to enhance , This data enhancement method is simply to put 4 A picture , By randomly scaling 、 Random reduction 、 Random layout of the way to splice .Mosaic It has the following advantages ：
（1） Rich data set ： Random use 4 A picture , Random scaling , And then random distribution for stitching , Greatly enriched the detection data set , In particular, random scaling adds a lot of small targets , Make the network more robust ;
（2） Reduce GPU memory ： Direct calculation 4 Picture data , bring Mini-batch The size does not need to be very large to achieve better results .

chart 1 mosaic effect

mosaic python Realization
Ideas ： Randomly select four pictures , Take part of it and put it into the picture , As shown in the figure below , The four colors represent four sample images , The excess will be discarded .

chart 2 mosaic Ideas

The specific methods are as follows ：

step1： newly build mosaic canvas , And in mosaic Randomly generate a point on the canvas

im_size = 640
mosaic_border = [-im_size // 2, -im_size // 2]
s_mosaic = im_size * 2

mosaic = np.full((s_mosaic, s_mosaic, 3), 114, dtype=np.uint8)
yc, xc = (int(random.uniform(-x, s_mosaic + x)) for x in mosaic_border)

chart 3 mosaic canvas

step2： Around random points (x_c, y_c) place 4 A jigsaw puzzle

（1） Top left

Canvas placement area ： (x1a, y1a, x2a, y2a)

case1： The picture does not extend beyond the canvas , The canvas placement area is (x_c - w , y_c - h , x_c, y_c)

case2： The picture goes beyond the canvas , The canvas placement area is (0 , 0 , x_c, y_c)

comprehensive case1 and case2, The canvas area is ：

 x1a, y1a, x2a, y2a = max(x_c - w, 0), max(y_c - h, 0), x_c, y_c

chart 4 mosaic Upper left puzzle

Picture area ： (x1b, y1b, x2b, y2b)

case1： The picture does not extend beyond the canvas , Pictures need not be cropped , The picture area is (0 , 0 , w , h)

case2： The picture goes beyond the canvas , The excess part of the picture needs to be cropped , The area is (w - x_c , h - y_c , w , h)

comprehensive case1 and case2, The picture area is ：

 x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h

（2） Top right

Canvas placement area ： (x1a, y1a, x2a, y2a)

case1： The picture does not extend beyond the canvas , The canvas area is (x_c , y_c - h , x_c + w , y_c)

case2： The picture goes beyond the canvas , The canvas area is (x_c , 0 , s_mosaic , y_c)

comprehensive case1 and case2, The canvas area is ：

 x1a, y1a, x2a, y2a = x_c, max(y_c - h, 0), min(x_c + w, s_mosaic), y_c

chart 5 mosaic Upper right puzzle

Picture area ： (x1b, y1b, x2b, y2b)

case1： The picture does not extend beyond the canvas , Pictures need not be cropped , The picture area is (0 , 0 , w , h)

case2： The picture goes beyond the canvas , The picture needs to be cropped , The picture area is (0 , h - (y2a - y1a) , x2a - x1a , h)

comprehensive case1 and case2, The picture area is ：

 x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h 

In the same way, the lower left and lower right jigsaw puzzles can be realized .

step3： to update bbox coordinate
4 Picture of bbox (n,4), among n by 4 In the picture bbox Number ,4 Represents four coordinate values (xmin,ymin,xmax,ymax) , Add the offset to get mosaic bbox coordinate ：

def xywhn2xyxy(x, padw=0, padh=0):
    # x: bbox coordinate  (xmin,ymin,xmax,ymax)
    x = np.stack(x)
    y = x.clone() if isinstance(x, torch.Tensor) else np.copy(x)
    y[:, 0] = x[:, 0] + padw  # top left x
    y[:, 1] = x[:, 1] + padh  # top left y
    y[:, 2] = x[:, 2] + padw  # bottom right x
    y[:, 3] = x[:, 3] + padh  # bottom right y
    return y

mosaic python The complete implementation code is as follows :

import cv2
import torch
import random
import os.path
import numpy as np
import matplotlib.pyplot as plt
from camvid import get_bbox, draw_box

def load_mosaic(im_files, name_color_dict):
    im_size = 640
    s_mosaic = im_size * 2
    mosaic_border = [-im_size // 2, -im_size // 2]
    labels4, segments4, colors = [], [], []
    # mosaic center x, y
    y_c, x_c = (int(random.uniform(-x, s_mosaic + x)) for x in mosaic_border)
    
    img4 = np.full((s_mosaic, s_mosaic, 3), 114, dtype=np.uint8)
    seg4 = np.full((s_mosaic, s_mosaic), 0, dtype=np.uint8)
    
    for i, im_file in enumerate(im_files):
        # Load image
        img = cv2.imread(im_file)
        seg_file = im_file.replace('images', 'labels')
        name = os.path.basename(seg_file).split('.')[0]
        seg_file = os.path.join(os.path.dirname(seg_file), name + '_L.png')
        seg, boxes, color = get_bbox(seg_file, names, name_color_dict)
        colors += color
        h, w, _ = np.shape(img)
        
        # place img in img4
        if i == 0:  # top left
            x1a, y1a, x2a, y2a = max(x_c - w, 0), max(y_c - h, 0), x_c, y_c
            x1b, y1b, x2b, y2b = w - (x2a - x1a), h - (y2a - y1a), w, h 
        elif i == 1:  # top right
            x1a, y1a, x2a, y2a = x_c, max(y_c - h, 0), min(x_c + w, s_mosaic), y_c
            x1b, y1b, x2b, y2b = 0, h - (y2a - y1a), min(w, x2a - x1a), h
        elif i == 2:  # bottom left
            x1a, y1a, x2a, y2a = max(x_c - w, 0), y_c, x_c, min(s_mosaic, y_c + h)
            x1b, y1b, x2b, y2b = w - (x2a - x1a), 0, w, min(y2a - y1a, h)
        elif i == 3:  # bottom right
            x1a, y1a, x2a, y2a = x_c, y_c, min(x_c + w, s_mosaic), min(s_mosaic, y_c + h)
            x1b, y1b, x2b, y2b = 0, 0, min(w, x2a - x1a), min(y2a - y1a, h)
        img4[y1a:y2a, x1a:x2a] = img[y1b:y2b, x1b:x2b]
        
        # place seg in seg4
        seg4[y1a:y2a, x1a:x2a] = seg[y1b:y2b, x1b:x2b]
        
        # update bbox
        padw = x1a - x1b
        padh = y1a - y1b
        boxes = xywhn2xyxy(boxes, padw=padw, padh=padh)
        labels4.append(boxes)
    labels4 = np.concatenate(labels4, 0)
    for x in labels4[:, 1:]:
        np.clip(x, 0, s_mosaic, out=x)  # clip coord
    
    # draw result
    draw_box(seg4, labels4, colors)
    
    return img4, labels4,seg4

if __name__ == '__main__':
    names = ['Pedestrian', 'Car', 'Truck_Bus']
    
    im_files = ['camvid/images/0016E5_01440.png',
                'camvid/images/0016E5_06600.png',
                'camvid/images/0006R0_f00930.png',
                'camvid/images/0006R0_f03390.png']

    load_mosaic(im_files, name_color_dict)
    ```

 chart 6 mosaic  Data enhancement results 
————————————————
 Copyright notice ： This paper is about CSDN Blogger 「 Brother Dalin 」 The original article of , follow CC 4.0 BY-SA Copyright agreement , For reprint, please attach the original source link and this statement .
 Link to the original text ：https://blog.csdn.net/weixin_46142822/article/details/123805663