mmdetection3D---(1)

mmdetection3d Description documentation ：

https://mmdetection3d.readthedocs.io/en/latest/getting_started.html

One 、config file

1、config File structure

config Files may have inheritance base Parent class under , So maybe just watch configs The file under is not complete , You can use the following script to print the complete config file .

python tools/misc/print_config.py /PATH/TO/CONFIG

stay config Of the network under the directory config Files will inherit from the same folder **_base_ The parent class under the directory **, be-all config The files are all in the same directory . The official recommendation is to use only one base parent , The maximum inheritance level does not exceed 3 layer .config Files are generally composed of four basic modules ：
dataset： Data set related configurations
model： Model structure configuration
schedule： Training optimization strategy
default_runtime： Training iteration settings

2.config File naming

{
    model}_[model setting]_{
    backbone}_{
    neck}_[norm setting]_[misc]_[batch_per_gpu x gpu]_{
    schedule}_{
    dataset}
#############################################################################################################
[model setting]: specific setting for some model.

{
    backbone}: backbone type like regnet-400mf, regnet-1.6gf.

{
    neck}: neck type like fpn, secfpn.

[norm_setting]: bn (Batch Normalization) is used unless specified, other norm layer type could be gn (Group Normalization), sbn (Synchronized Batch Normalization). gn-head/gn-neck indicates GN is applied in head/neck only, while gn-all means GN is applied in the entire model, e.g. backbone, neck, head.

[misc]: miscellaneous setting/plugins of model, e.g. strong-aug means using stronger augmentation strategies for training.

[batch_per_gpu x gpu]: samples per GPU and GPUs, 4x8 is used by default.

{
    schedule}: training schedule, options are 1x, 2x, 20e, etc. 1x and 2x means 12 epochs and 24 epochs respectively. 20e is adopted in cascade models, which denotes 20 epochs. For 1x/2x, initial learning rate decays by a factor of 10 at the 8/16th and 11/22th epochs. For 20e, initial learning rate decays by a factor of 10 at the 16th and 19th epochs.

3. differ mmdetection,train_cfg and test_cfg stay mmdetection3d Is put into the model definition .

# recommended
model = dict(
   type=...,
   ...
   train_cfg=dict(...),
   test_cfg=dict(...)
)

install open3d
https://stackoverflow.com/questions/49911550/how-to-upgrade-disutils-package-pyyaml

4. in the light of _base_ Modification and replacement of basic configuration in

Use _delete_=True keyword

for example ：_base_ Network defined in neck as follows ：

model = dict(
    type='MVXFasterRCNN',
    pts_voxel_layer=dict(...),
    pts_voxel_encoder=dict(...),
    pts_middle_encoder=dict(...),
    pts_backbone=dict(...),
    pts_neck=dict(
        type='FPN',
        norm_cfg=dict(type='naiveSyncBN2d', eps=1e-3, momentum=0.01),
        act_cfg=dict(type='ReLU'),
        in_channels=[64, 128, 256],
        out_channels=256,
        start_level=0,
        num_outs=3),
    pts_bbox_head=dict(...))

I want to replace it with another type neck, In a subclass config Inheritance in file _base_ after , Make the following definition changes ：

_base_ = '../_base_/models/hv_pointpillars_fpn_nus.py'
model = dict(
    pts_neck=dict(
        _delete_=True,
        type='SECONDFPN',
        norm_cfg=dict(type='naiveSyncBN2d', eps=1e-3, momentum=0.01),
        in_channels=[64, 128, 256],
        upsample_strides=[1, 2, 4],
        out_channels=[128, 128, 128]),
    pts_bbox_head=dict(...))

5. Intermediate variable train_pipeline and test_pipeline

train_pioeline and test_pipeline It is an intermediate variable for processing data , After setting , Need to transfer it to data in .

_base_ = './nus-3d.py'
train_pipeline = [
    dict(
        type='LoadPointsFromFile',
        load_dim=5,
        use_dim=5,
        file_client_args=file_client_args)
        ..........
]
test_pipeline = [
    dict(
        type='LoadPointsFromFile',
        load_dim=5,
        use_dim=5,
        file_client_args=file_client_args),
    ...................
]
data = dict(
    train=dict(pipeline=train_pipeline),
    val=dict(pipeline=test_pipeline),
    test=dict(pipeline=test_pipeline))

Two 、 Custom related

1. Custom datasets

2、 Custom data processing pipeline

Data processing involves dataset,dataloader, in addition , To store data of different sizes （ For example, pictures ,bbox）, Use a container class datacontainer As a data storage container .
pipeline Define every step of data processing , Each step is based on a Dictionary as input , Then output a dictionary For the next stage .dataset and pipiline It's two separate parts ,dataset It is mainly used to define and process data annotation label information ,pipeline Define all the steps of processing data into dictionary form .
Below is the pipeline Processing module flow , Green represents add New content , Orange represents the updated content .
The overall data processing process is divided into four stages ：
Data loading
Preprocessing
Formatting Data
Data to enhance
You can customize pipeline Processing steps of , And then use it in pipeline in .

3. Custom model

1.model The module of is in mmdetection3d Divided into 6 A type of ：
encoder：

 e.g., HardVFE and PointPillarsScatter.
  Mainly in the voxel-based Method used to encode features , Include voxel layer,voxel encoder and middle encoder Three small parts .

backbone：

e.g., ResNet, SECOND.
 It is usually a convolution network used to extract features 

neck：

e.g., FPN, SECONDFPN
 A bottom-up line , A top-down line , Transverse connection . Integrate low-level high-resolution information and high-level deep semantic information .

FPN come from detection Mission ;U-Net come from segmentation Mission . FPN Of “ Zoom in ” Part is directly interpolated and amplified , No, deconvolution Of filters Learning parameters ;U-Net“ Zoom in ” Part is Decoder, need deconvolution Of filters Learning parameters .FPN And most of its improvements are based on the original Feature Map and FPN Of Feature Map add ;U-Net And most of its improvements are based on the original Feature Map and Decoder Of Feature Map do Concatiantion, Do it again 1x1 Convolution .FPN For each fusion layer detection;U-Net Only on the last floor segmentation Of pixel forecast . author ： Meal link ：https://www.zhihu.com/question/351279839/answer/1002339902

head：

 e.g., bbox prediction and mask prediction.

roi extractor:

e.g., H3DRoIHead and PartAggregationROIHead.
 extract roi Characteristic map of the area 

loss:

e.g., FocalLoss, L1Loss, and GHMLoss.

2. Each module can be customized , The defined steps mainly include three steps
Named by definition second Of backbone For example ：
（1） Create custom backbone Class file
mmdet3d/models/backbones/second.py

import torch.nn as nn

from ..builder import BACKBONES


@BACKBONES.register_module()
class SECOND(nn.Module):

    def __init__(self, arg1, arg2):
        pass

    def forward(self, x):  # should return a tuple
        pass

    def init_weights(self, pretrained=None):
        pass

（2） stay init Import in file
mmdet3d/models/backbones/init.py

from .second import SECOND

（3） stay config Use in

model = dict(
    ...
    backbone=dict(
        type='SECOND',
        arg1=xxx,
        arg2=xxx),
    ...

Detailed Links

4. Customize the training time policy 、 Optimization strategy

Reference resources

Here is an interesting pytorch modular ：
hooks------- hook
When you train a network , Want to extract the parameters of the middle layer 、 Or feature map , Use hook It can be used .
Detailed introduction

3、 ... and 、 Tool related

1. Draw image curve ：
Loss curve 、 evaluation mAP curve
2. computing time
3. Visualization tools
Visualization results , Visual datasets
Dictionaries dic()
4. Model complexity statistics

Dictionary creation ： A colon separates key:value, Commas separate pairs of elements

dict = {
    'Name': 'Zara', 'Age': 7, 'Class': 'First'}

print ("dict['Name']: ", dict['Name'])
print ("dict['Age']: ", dict['Age'] )

# The output of the above example ：

#dict['Name']: Zara
#dict['Age']: 7

Error output ：

dict = {
    'Name': 'Zara', 'Age': 7, 'Class': 'First'}
print ("dict['Alice']: ", dict['Alice'] )

#  The output of the above example ：

#KeyError: 'Alice'

super().__init__() It's obvious that , Is to execute the constructor of the parent class , Enables us to call the properties of the parent class .

Try to use open3d==0.11