Shared by Merrill Lynch data technology expert team, smoking detection related practice based on Jetson nano

With big data 、 The continuous development of artificial intelligence technology , Video surveillance technology is more and more widely used in “ Smart city ”、“ Safe city ” And other public safety projects , Compared with cloud based video analysis services , People prefer to run models directly on edge devices close to the video source, such as smart cameras , To reduce network latency 、 Bandwidth usage 、 Deployment costs and privacy protection . This paper takes smoking detection as an example , To elaborate on Jetson Nano This computing resource is limited on the device , The practice of providing near end intelligent analysis services .
For the detection of smoking , It is a typical small target detection in computer vision scene , Such targets usually have low resolution 、 Less information 、 A lot of noise , It will bring great difficulties to the detection , First , Let's review some common methods of small target detection based on deep learning .
Common techniques for small target detection
There are several common techniques for small target detection ：
Method based on multi-scale sliding window
Through the original image to build different resolution image pyramid , The classifier slides in different layers to detect targets with different scales ;
Method based on data enhancement
The basic idea is to improve the effect of the model by increasing the number of small objects in the image , You can use the oversampling strategy or copy and paste the small target area ;
Feature based pyramid FPN Methods
It considers that the characteristic graphs of different layers of the network have different degrees of expression of information , The shallow receptive field is small , More suitable for small target detection ;
A puzzle based approach
Without changing the size of the original image , The image is segmented according to the network input resolution , Form a batch of data , Incoming network for detection ;
The method based on tangent graph
The basic idea is to select the target whose truth scale is close to the anchor frame for training , The representative methods are SNIP and SNIPER etc. ;
Method based on adaptive anchor frame
Tune the preset anchor box according to the specific task , If... Can be used KMeans Clustering algorithm is used to cluster the data set , Find the appropriate anchor frame size .
Specifically for smoking scenes ,“ smoke ” Its length width ratio is special , Without obvious characteristics , And there is often hand shielding , therefore , In practice , On the one hand, we adopt the method of adaptive anchor box to adapt to the proportion of smoke , At the same time, the method of data enhancement is used to increase the number of cigarettes , On the other hand , use “ Check the hand first , Retest smoke ” This two-stage detection method . Because we are running the model on edge devices , The model is too large, resulting in poor real-time performance , This two-stage approach , It can greatly reduce the difficulty of target detection , That is, it can be realized with a simple model , And it can eliminate the false detection .

Smoking detection process based on two stages

The smoking detection process based on two stages is shown in the figure above , The first stage is based on Yolov4-tiny Hand detection model , Locate the area of the shot from the whole picture , This can greatly reduce the calculation of subsequent smoking detection ; The second stage , Considering the discrimination information contained in the target itself , The semantic segmentation model is used to distinguish the pixels in the region where the smoke is located , In this way, the relevant information of surrounding pixels can be fully utilized . The specific application is based on U-Net Semantic segmentation model of smoke , Then judge whether there is smoke , Final , Complete the smoking judgment .
To speed up the model in Jetson Nano Running speed on , We are based on TensorRT Model optimization acceleration framework , Generate for Nano GPU Model file of hardware optimization , The specific process is as follows ：

Because the detection model adopted in the first stage is Yolov4-tiny, and TensorRT The operator of some of these layers is not supported in , To generate an optimization model , Need to combine DeepStream Deployment framework and TensorRT Relevant standards , Implement correspondence IPluginV2 and IPluginCreator The interface of ：

be based on Jetson Nano Smoking detection model deployment
Smoking detection is connected to real-time video stream , Deploy with DeepStream Framework for deployment ：

First, for the input video stream , use H264 Such as the decoder combined with hardware to decode , And then enter the model , The model part relies on DeepStream Of nvinfer Interface to load via TensorRT The optimized model , And combine OSD Show the detection box 、 Category etc. , At last EGL etc. Sink Make a desktop presentation . among , For the test results , With the help of GstPlugin Interface to integrate with business ：

In the smoking detection scene , Based on this, we achieve the acquisition of smoke categories in the smoke segmentation results 、 The acquisition of cigarettes in the area and the determination of smoking violations , And generate the corresponding detection element information , After message conversion and message Broker Realize the sending of test results .

Sum up , Using the method of small target detection based on deep learning , With the help of Nano GPU Hardware and DeepStream And other software frameworks , It can realize the real-time detection of smoking at the edge , Meet the real-time requirements of intelligent monitoring and analysis scenarios , And the bandwidth consumption is reduced 、 Protect data privacy , At the same time, it can be combined with cloud storage , Realize real-time alarm in the event 、 Ex post facto .