Based on butterfly species recognition

pick want

In view of the existing data sets used in butterfly recognition research, there are few butterfly species , It only contains photos of butterfly specimens , It does not include the problem of Butterfly Photos in the ecological environment , A butterfly image data set containing both specimen photos and ecological photos has been released , Among them, the specimen photos include all butterfly species of Chinese butterflies , total 4 270 A picture ,1 176 Kind of , Photos of butterfly ecological environment 1 425 Zhang ,111 Kind of . Propose deep learning technology Faster R-CNN Butterfly species automatic identification system , Including automatic detection of butterfly position in ecological photos and species identification . The experiment removes butterfly species that only contain a single ecological photo , Take the remaining butterfly ecological photos 5-5 Divide , structure 2 Different training data sets : Half ecological photos + All mode photos , Half ecological photos + Corresponding category mode photos ; Training 3 A butterfly automatic recognition system with different network structure , Average with average precision (mean average precision,mAP) For the evaluation index , Use up and down , Flip left and right , Rotate at different angles , Add noise , Varying degrees of blur , Contrast rise and fall 9 Two ways to expand the training set . Experimental results show that , be based on Faster R-CNN The butterfly automatic recognition system based on the deep learning framework can realize the automatic detection of butterfly position and species recognition in the butterfly photos in the ecological environment , Model mAP The lowest value is close 60%, It can detect multiple butterflies in birth photos and complete species identification at the same time .
key word Butterfly ; Automatic identification ; object detection ; Deep learning ; classification

introduction

Butterflies are one of the important environmental indicators of insects , The variety is complex . The identification of butterfly species is directly related to the crops eaten by humans and animals . at present , The widely used reliable butterfly recognition method is not effective . Manual identification of butterfly species is not only a heavy workload , And it requires long-term experience and knowledge accumulation . How to realize the automatic identification of insect species is one of the research hotspots in the field of computer vision .
Automatic identification of insect species requires recognition and classification of digital images , The effect of image classification is closely related to the quality of texture feature extraction .2004 year ,Gaston wait forsomeone [2] This paper introduces the application of artificial intelligence technology and digital image processing method in digital image recognition . thereafter , Many experts and scholars have done a lot of work in this regard [3,4]. In recent years , With the development of machine learning , Researchers have proposed many related butterfly detection application algorithms .2012 year ,Wang wait forsomeone [5] Use content-based image retrieval （CBIR） Extract the image features of butterflies , Such as color 、 shape .Bohan Liang wait forsomeone . For different texture features 、 Feature weight and similarity matching algorithm are compared , And put forward the corresponding classification method .2014 year ,Kaya Y wait forsomeone [6] Apply local binary mode （LBP）[7] And gray level co-occurrence matrix （GLCM）[8] Extract texture features of butterfly image , Then the single hidden layer neural network is used for classification , An automatic butterfly species recognition method based on extremum learning machine rule is proposed . Same year ,Kang S H wait forsomeone [9] Based on the branch length similarity （BLS） An effective identification scheme of entropy distribution , This scheme uses butterfly images observed from different angles as training data of neural network .2015 year , Li Fan [10] According to the morphological characteristics and texture distribution of butterflies , Using gray level co-occurrence matrix of image block （GLCM） The characteristics and K Neighbor classification algorithm , The corresponding feature extraction and classification decision-making methods are proposed .2016 year ,Zhou A M wait forsomeone [11] The feasibility of the deep learning model is proved , And it has strong generalization ability in the automatic recognition of butterfly specimens .
2018 year , Xiejuanying et al [12] Based on fast RCNN Butterfly detection and species identification methods [13], And in the natural ecological environment, butterfly photos are used to expand the data set . The algorithm has high positioning accuracy and recognition accuracy for butterfly photos in natural ecological environment . The traditional butterfly recognition algorithm has the following problems ：
1. In natural ecological photos , Butterflies often appear in the form of small targets （ The area of butterfly image is too small ）, Traditional butterfly recognition algorithms are often powerless .
2. The amount of data required for training is huge , But we can't find a high-quality annotated public dataset .
3. There are too few photos of some rare butterflies in their natural state , Cannot be used directly as a training set .
Joe Redmon[1] Proposed YOLO Model is a famous end-to-end learning model in the field of target detection . And RCNN[13] Compared with the two-step model of the sequence ,YOLO The execution speed of the model is fast , Avoid background error , But the positioning accuracy is poor , The single model of some classifications has a high false positive rate .
In order to improve the efficiency of butterfly recognition , This article will make full use of the data provided by China data mining competition and Baidu Encyclopedia , Create a butterfly data set containing a large number of butterfly ecological photos , Model training using ecological photos in the natural environment , And in YOLO V3 Based on the algorithm , An integrated algorithm for butterfly ecological photo location and recognition is proposed . The main structure of this paper is as follows ：
1. Data sets 、 Data annotation and data preprocessing . Using literature [14] Butterfly ecological photos provided and pictures in Baidu and other online photo databases , By self marking , Established a system that contains 2342 A butterfly dataset that accurately marks the natural environment . The method suitable for butterfly marking is selected through experiments , And the marking methods are summarized .
2. Integrated YOLO Algorithm . The algorithm inherits YOLO The recognition speed of the algorithm , To optimize the YOLO Algorithm , Improved YOLO The recognition ability of the algorithm to fine targets , Improved YOLO The learning ability and generalization ability of the algorithm for small samples . Based on atlas The follow-up target detection provides a good idea .
3. Experiment and analysis . Introduce and analyze YOLO The performance of the algorithm in different annotation and processing modes , And integration on butterfly test set YOLO Performance of the algorithm .

1 Data preprocessing

The butterfly datasets used in this paper are all photos of butterflies in the natural ecological environment , Hereinafter referred to as ecological photos . Part comes from literature [14] Data set provided , The other part comes from the pictures in Baidu and other search engines and picture libraries , Including butterflies 94 Kind of 11 Belong to . The following figure shows some samples of butterfly ecology .














Take photos 5695 Zhang , Butterfly Photos include specimen photos and ecological photos . The literature [12] Pointed out that , Because the shooting angle and background environment of the specimen are very different from the ecological photos , The training effect of using only ecological photos in the training set is significantly better than that of using specimens and ecological photos for butterfly detection and classification , The purpose of this study is to locate butterflies in the natural environment , Identify butterfly species , Therefore, this article only selects 1048 A picture of butterflies in the natural ecological environment .
Most of the butterfly samples contained in each photo in the data set are only one , The maximum quantity shall not exceed 20 individual . Each butterfly is composed of at least four samples with a typical heavy tail distribution .
The test set is based on literature [14] Standard test set provided , contain 678 Ecological photos , The rest is the training set .

1.1 Data collection

Because the posture of butterflies in ecological photos is more complex , There are even many butterflies overlapping , The literature [14] The data set provided is chaotic , There is no unified annotation standard . We have formulated a set of unified labeling standards , According to the standard, the positions and species of all butterfly samples in all photos are manually marked .
In the literature [14] Data set provided , There are two ways to mark the area where the butterfly is located ： First, it is bounded by the antennae and legs of butterflies , As shown in the figure ; The second is based on the butterfly's trunk and wings , As shown in the figure . We use two annotation methods to unify data sets .

1.2 Data to enhance

Because some butterflies have social attributes , Many butterflies often overlap in photos . The literature [14] The dataset provided in uses the method of marking multiple butterflies in the overlapping area as a single sample , As shown in the figure . We have also developed a standard for marking this situation ： Each butterfly in the overlapping area is marked independently , And the occluded part is ignored , As shown in the figure . This method not only increases the number of training samples , And it improves the recognition effect of the model on complex scenes .

Target detection algorithms based on deep learning often need a large amount of data as training sets . This paper uses rotation 、 Mirror image 、 Fuzzy 、 Contrast rise and fall 9 A transformation method is used to expand the training set , Combined with different pretreatment methods and their parameters （ Such as rotation angle 、 Exposure, etc ）, Get the optimal pretreatment method . The result will be in 4 Section shows .

2 Butterfly detection and identification methods

Through the above process , The problem of automatic detection and classification of butterflies in natural ecological environment has been transformed into a multi-objective detection and classification problem . Different from common target detection problems , There are three difficulties in butterfly automatic detection and classification ：1） There are many categories （94 A classification ）;2） The sample distribution is uneven . The butterfly samples of some rare species are significantly less than those of other species ;3） Different subclasses must be classified （ Different kinds of butterflies ） Fall into the same general category （ Butterfly ）, That is, fine-grained classification is required . therefore , In this paper, it is more difficult to study the automatic detection and classification of butterflies .

2.1 YOLO Model

Joe-Redmon Proposed YOLO Model [1] It is a famous end-to-end learning model in the field of target detection . Its characteristics are similar to RCNN[8] Compared with the two-step model of series ,YOLO The execution speed of the model is much faster , It performs well in fine-grained detection . Our task has been chosen YOLO The third generation of models YOLO v3.
YOLO V3 The structure of the model is shown in the figure . In order to detect different sizes in natural photos （ The area proportion is different ） Butterfly ,YOLO V3 In the feature extraction network （darknet-53） after , Using multi-scale feature map to detect targets of different sizes .
YOLO V3 Output three characteristic maps with different scales . after 79 After layer convolution network , The detection results of the first scale are obtained through the three-layer feature extraction network . Here, the feature map used for detection is the input image 32 Double down sampling . Due to the high down sampling rate , The perception field of feature map is relatively large , It is suitable for detecting objects with large area in the image .
The upsampling convolution is from 79 Finished behind the layer . The first 81 The layer characteristic diagram is the same as that of 61 Combination of layer characteristic diagram . After three layers of feature extraction network , The fine-grained characteristic graph of the 91st layer is obtained , That is, relative to the input image 16 Down sampling characteristic graph . It has a medium-sized perceptual field of vision , It is suitable for detecting targets with medium area proportion in the image .
Last , Again 91 Layer feature map for sampling , And with 36 Connect the layer characteristic diagram . After three layers of feature extraction network , Get low relative to the input image 8 Characteristic graph of multiple sampling . It has the smallest perceptual field , It is suitable for detecting targets with small area proportion in the image .
Each output contains 3d2 Division 5+N Dimension vector , Symbols D Represents the edge length of the output feature graph at this scale . Numbers 3 Centered on the number of prior frames in each grid cell , Symbol N It's the number of categories . The first four dimensions of each vector represent the position of the prediction box , The fifth dimension represents the probability of the target in the candidate box ,5+i Dimension indicates that the target in the candidate box belongs to i The probability of a class .
YOLO Use the mean square sum error as the loss function . It consists of four parts ： Prediction box error （ERRcenter）、 Error in predicting boundary width and height （ERRwh）、 Classification error （ERRclass） And prediction confidence error （ERRconf）.

here ,n Is the number of prediction frames in the grid cell ,（x,y） Is the central coordinate of the prediction frame ,w and h Respectively, the width and height of the prediction frame . The error caused by large prediction box is obviously larger than that caused by small prediction box ,YOLO The method of predicting the square root of width and height is adopted , Instead of directly predicting width and height . If the... In the first grid cell j A prediction box is responsible for the object , Then we get ιobj ij=1, conversely ,ιobj ij=0.

YOLO It is considered that each grid cell contains only one classification object . If c Is the correct category , that ˆpi（c）=1, conversely ,ˆpi（c）=0.

among ci Indicates the confidence of the object contained in the prediction box . If there are objects in the actual bounding box , be ˆci Is the actual bounding box and prediction box IoU value . contrary , Yes ˆCi=0. Introduce parameters in different weighted parts of the loss function λ, The robustness of the model is improved . In this paper , We have λcoord=5,λcoord=0.5.

2.2 comprehensive YOLO Algorithm

In order to get more accurate classification and detection results , Improve magic Generalization ability , This article further discusses several YOLO The results of the model are processed , Integrated YOLO Algorithm . The pseudo code of the algorithm is like the algorithm 1 Shown . Its core idea is to use multiple models with good training effect to predict the image respectively , And cluster the predicted frames . The clustering process is shown in the figure .

Each positioning box can be described as a four-dimensional vector bi=（x1i,x2i,y1i,y2i）, An integer and a real number pi. It said ： In the upper left corner is x1i、y1i、 The lower right corner is x2iand and y2i In the rectangle of , There are several species ciis pi The probability of butterflies . Set each cluster set as S1,S2,…,Sk. For each S i（i=1,2,…,k）, Satisfy ： about ∀i,j∈S,ci=cj. Set the “ general ” Defined as ：

among ,B（S） Is a collection S Of “ polymerization ” Positioning box ,P（S） Is a collection S Of “ polymerization ” probability ,C（S） Is a collection S Of “ polymerization ” classification . Each time a single prediction bounding box is classified , Select the set with the highest probability from all sets with the same classification as the detection box S, also IoU（B（S）,B）≥0.5. If there is no qualified , Then place the box in the new set Sk+1 in .

3 Experimental results and Analysis

In order to verify the validity of the convolution neural network model established in this paper , When designing the experiment, the comparative experiments are carried out from two aspects: data modeling method and data grouping test .

3.1 The evaluation index

This paper takes the joint intersection （IoU） As the evaluation index of butterfly positioning task , It is defined as the ratio of the area of two areas to the area of the merged area . The best proportion is 1, That is, complete overlap . In the experiment IoU=0.5 As a threshold , That is, the prediction box and the original label box are positioned ,IoU>0.5,IoU≤0.5 The positive positioning is wrong . In this paper, the average precision is （mAP） As the evaluation index of butterfly classification task . Maps come from accuracy （pre） And recall （recall）. The calculation formula is as follows

among ,TP（rue Positive number ）、FP（ False positive number ） and FN（ False negative number ） Respectively represents the number of positive samples predicted to be positive 、 The number of negative samples predicted to be positive and the number of positive samples predicted to be negative .
According to different confidence levels , You can get multiple （pre,recall） spot , And take recall as abscissa , The precision is the ordinate , Draw the pre recall curve . among , Average precision AP Is the area around the pre recall curve and recall axis , This is the integral of the pre recall curve , As shown below

in application , The sum of rectangular areas is usually used to approximate the integral . In this paper PASCAL-VOC Challenge stay 2010 Calculation method after years [15], Recall can be divided into n Block [0,1n,…（n−1）n,1], Then average accuracy （AP） It can be expressed as

3.2 YOLO V3 Model effect experiment

This paper tests two different butterfly marking methods ： One is to use the antennae of butterflies as the boundary of the frame , It is called full scale marking ; The other is to use the butterfly's trunk and wings as the boundary of the frame , It is called non full scale marking . Adopt different pretreatment methods , The best results of the two annotation methods are shown in the table 1 Shown .
so , The calculation result of non full scale method is obviously better than that of full scale method . Due to the large area around the antennae of butterflies in the full-scale method , This will reduce the proportion of the area around the antennae of the butterfly in the marked area , Thus, the background environment has a greater impact on classification . The incomplete marking method is more suitable for the automatic detection and recognition of butterflies .
because YOLO The different size of the input image in the model will lead to the different number of grid cells under different output scales , Therefore, this paper tests a single YOLO V3 Performance of the model in butterfly automatic positioning task .
Use the non full scale method , No other pretreatment is required , The best results of the three input sizes are shown in the table 2 Shown .
so , use 608px×608px The resolution input of has better accuracy .

In the task of butterfly automatic location and classification , Adopt different pretreatment methods to 608px×608px Input for incomplete marking , Tested a single YOLO V3 The performance of the model . surface 3 It shows the influence of two parameters that have a great influence on the classification results .
It can be seen that , By rotating the original image （0°、45°、90°、180°） And expose them separately 1.0、1.2、1.5 and 1.8 Time , The best classification results can be obtained .mAP The value of can reach 0.7766. In particular, the recognition rate of dark and protective butterflies is significantly higher than that of the first group .

4 Conclusion

at present , The field of target detection is mainly divided into two schools ： End to end detection and distributed detection . End to end detection is fast , But compared with the distributed detection scheme , There is a big gap in accuracy . Take advantage of the rapidity of the end-to-end model and the high similarity between butterfly populations , Based on different concentrations YOLO Comprehensive model of , The detection speed of the end-to-end model is maintained , The detection accuracy and positioning accuracy are improved .
The essence of integration model is under various conditions , Find a better solution according to the optimal solution of the model . This can improve the performance of the model on a specific training set , Improve the comprehensive generalization ability of the model . In the literature [14] Test set provided , The accuracy of the model in the ecological photo location task reaches 98.35%, The accuracy rate in the task of species location and identification has reached 0.7978, The accuracy rate in the positioning and recognition task of the subjects reached 0.8501.
Butterflies are highly similar among species , This shows that in the knowledge map , There is a strong relationship between butterflies . For a wider range of target detection tasks , Species can be accurately classified and located by using knowledge map , Through different model capabilities and different recognition focus, its recognition task can be further optimized .

ginseng Examination writing offer

[1] Based on improved YOLO v3 Vegetable recognition algorithm [J]. Wei Hongbin , Zhang Duanjin , Du Guangming , Xiao Wenfu . Journal of Zhengzhou University ( Engineering Edition ). 2020(02).
[2] Research on automatic identification of butterfly species [J]. Xie juanying , Hou Qi , Shi Yinghuan , Lu Peng , Jing Liping , Zhuang Fuzhen , Zhang Junping , Tanxiaoyang , Xu Shengquan . Computer research and development . 2018(08).
[3] World Butterfly catalogue [M]. Shaanxi science and Technology Press , Shoujianxin , Zhou Yao , Edited by Li Yufei , 2006.
[4] Investigations of butterfly species identification from images in natural environments[J] . Juanying Xie,Yinyuan Lu,Zhaozhong Wu,Shengquan Xu,Phil W. Grant. International Journal of Machine Learning and Cybernetics . 2021 (prep).
[5] Fine-Grained Butterfly Classification in Ecological Images Using Squeeze-And-Excitation and Spatial Attention Modules[J] . Dongjun Xin,Yen-Wei Chen,Jianjun Li. Applied Sciences . 2020 (5).
[6] Automatic identification of butterfly species based on HoMSC and GLCMoIB[J] . Fan Li,Yin Xiong. The Visual Computer . 2018 (11).
[7] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.[J] . Ren Shaoqing,He Kaiming,Girshick Ross,Sun Jian. IEEE transactions on pattern analysis and machine intelligence . 2017 (6).
[8] The identification of butterfly families using content-based image retrieval[J] . Jiangning Wang,Liqiang Ji,Aiping Liang,Decheng Yuan. Biosystems Engineering . 2011 (1).
Welcome to join me for wechat exchange and discussion （ Please note csdn Add ）