AI enhanced safety monitoring project [with detailed code]

" By using microwave sensors and OAK Cameras make it safer to enter the building ."

— Ralph Yamamoto

This article was first published in oakchina.cn

Things used in this project

Hardware components ：

• OAK-D-IoT-40
• Microwave Proximity Detector RCWL-0516
• M5Stack M5StickC ESP32-PICO Mini IoT Development Board
• M5Stack M5StickC 18650C
• Raspberry Pi 4 Model B

Online applications and services :

• Intel OpenVINO toolkit
• DepthAI Gen2 API
• Espressif ESP-IDF
• Arduino IDE
• Node-RED
• PlatformIO IDE

Hand tools and manufacturing machines :

• AnyCubic Mega S 3D The printer （ Used to print brackets ）

background

in order to tinyML Visual challenge , I'm working on a proof of concept project , To identify people close to my front door .OpenCV Of OAK I was deeply impressed by the performance of the camera , I want to try to use it ESP32 Version of the product , It integrates a system for host processing and communication ESP32. Use something like OAK High performance devices such as cameras need to carefully manage power consumption . Most of the power is consumed in the process of reasoning , So I decided to add an additional microwave sensor , The reasoning function is enabled only when the object enters the detection boundary .

I don't have time to implement a fully packaged and integrated solution , But I want to demonstrate how to implement the various project elements and how they communicate with each other .

Microwave detector

The microwave detector is RCWL-0516, The price is low , Easy to integrate . It USES 3 The Doppler frequency shift of GHz signal is used to detect moving objects . The microwave detector is RCWL-0516, It's cheap , Simple integration . It USES 3GHz The Doppler frequency shift of the signal is used to detect moving objects . The device has a large 4-28V Power range , Adjusted to 3.3V. It provides a number for anything in the detection range 3.3V Test output . The nominal detection range is 7 rice . The sensitivity is orthogonal to the front of the circuit board .

RCWL-0516 positive

RCWL-0516 Back

My request is at the entrance door 5 Meter probe . For prototyping and testing , I decided to make this device portable . I will RCWL-0516 Installed in a with 18650 Of rechargeable battery pack M5StickC On . When movement is detected , The user indicator will illuminate . Detection events and battery voltage are also passed MQTT Release . Attach the connection diagram and code .

M5StickC Upper RCWL-0516

This is a picture leading to the front door . The first step is to leave the door 5 rice . In the test , When I was before this step , This unit will trigger . Unfortunately , I can't expose it on my mobile camera to take a demo video ( You can't see the LED at this resolution ), Although it is clearly visible .

Front door entrance

This is a video taken indoors . I started from 6 A room meters away began . You need to look carefully , But when I started walking , The LED will turn on , As long as there is exercise , The LED will always be triggered . When I stop , It will close , When I step back , It will trigger again .

AI Enhanced safety monitoring project ：1 Microwave detector （ See the introduction for project details and code ）

Microwave detector MQTT instrument panel

This is the instrument panel that displays the status of the microwave detector module . Node red dashboard runs on raspberry pie 4 On , I use it as MQTT The server / Agent to handle message passing between devices . state Or Safe perhaps Move and Doppler Displays the history of the test .

Microwave detector MQTT instrument panel

OAK The camera

OAK-D-IoT-40 yes OAK-D Embedded in /IoT edition , As shown in the figure below , It has binocular depth OV9282 (1280x800) The camera , Global shutter with synchronization , In order to realize the 3D Artificial intelligence . Plus high resolution IMX378 (12MP,4056x3040) Color cameras . The main board is in SOM There is an Intel Movidius Myriad X VPU, There are heat sinks on the back of the motherboard .3 Cameras and VPU Used to implement DepthAI. Besides , One more ESP32, adopt SPI And VPU signal communication , And provide Bluetooth and WiFi Connect .

I am for OAK-D-IoT-40 3D Printed a tripod bracket , I can use it in the prototyping phase . It is a completely open framework , Allow access to the circuit board when needed , On the back of the SOM The radiator is completely exposed , To improve heat dissipation performance .

3D Print tripod bracket

below , The device is mounted on a short tripod , For program development and testing . Blue at the top microUSB Cables are used for programming and communication ESP32 signal communication . The red in the middle USB-C Cables are used for programming and communication VPU signal communication . The black power cord at the bottom provides 5V @ 3A( If there is USB-C The power supply can 3A, You don't need to ). To prove my concept , I may keep this configuration , Although I will try to get a separate configuration , among VPU Start and run the model . under these circumstances , Just the power cord .

Mounted on a short tripod

This is the back of the motherboard ,SOM The radiator is completely exposed . For initial testing , I expect the reasoning to continue for a long time . I noticed that the radiator felt hot when running reasoning . As part of this project , I plan to measure power and heat in different operation modes . In the final deployment , I will minimize the reasoning duty cycle , To reduce the overall power consumption . The worst case scenario is , I may need to add a fan to the chassis ( Or try a bigger radiator ).

SOM The radiator is completely exposed

OpenVino tool kit

The edge oriented Intel development cloud provides a sandbox , For prototyping and experimenting with AI inference workloads on Intel hardware dedicated to deep learning . You can use Intel OpenVINO Tools built into the distribution kit optimize your deep learning model , And then in CPU、GPU and vpu( Visual processing unit ) Test their performance on a combination of .

I was in... A long time ago NCS and NCS2 It has been used. OpenVino Software , I wish I could be in DevCloud Complete all model optimization and testing , So I don't have to upgrade or reinstall locally OpenVino 了 . Unfortunately , When I try to run the tutorial , I find VPU Resource not available . This seems to be a mistake in the tutorial , But the host pointed out a post to me ( Small YOLO V3 Object detection example ), It allows me to compare running on different hardware devices ( a central processor 、 Graphics processor 、VPU etc. ) The model on .

For my proof of concept , I don't think I have time to do any model development or play with model optimizer settings . The good news is , The Intel open model zoo repository has many pre training models for my use cases . hope DevCloud In the notebook MYRIAD The device can fix this error , I can try to test different model configurations later .

Here are OAK-D-IoT-40 Basic flow

For this project , I'll start with an existing model ( Maybe a few , Because the processing pipeline allows ). The key part of the process I need to figure out is how to transform the intermediate representation (IR) file (bin and .xml) Add to . stay MyriadX What is needed to deploy the model blob file .

stay “tinyML Visual challenge -Intel-Luxonis Visual platform webinar ” period , come from Luxonis Of Erik Shows how to add file conversion code to DevCloud Object detection tutorial at the end of the notebook . The process is simple —— install blobconverter python modular , Use IR Run the path of the file and the output file blobconverter, And specify the razor to use ( kernel ) Number . Then use the file link module to blob Download the file to the host .

This is the output of the notebook :

A very short clip , Used to verify blob Whether the file is generated and downloaded correctly . Use depthai_demo Script ,blob The file has been copied to depthai/resources/nn/mobilenet-ssd Folder .

python dep Thai _ demo . py-CNN mobile net-SSD

I later found out ,OAK It also provides an online blob converter , You can easily OpenVino The model is converted to different versions and shave frequency ：//blobconverter.luxonis.com/ .

AI Enhanced safety monitoring project ：2 Object detection demo（ See the introduction for project details and code ）

OAK-D-IoT-40 Communication between devices on

The reasoning is entirely in Myriad X Up operation . In prototyping and development ,Myriad X By the host processor USB-C Interface operation . However , In a stand-alone deployment ,Myriad X Run in a separate bootable configuration . under these circumstances , The communication of reasoning results is controlled by ESP32 Handle , The ESP32 adopt SPI And Myriad X signal communication . The next step is to verify the programming and operation of the interface .

OAK There is one running in ESP32 Sample program on , It passes through spi News and VPU Interface . I tried esp32-spi-message-demo  It is from Character tracker Pipes in VPU Up operation . Track applet provides information about tracking status (“ new ”、“ Tracked ”、“ Lost ”、“ Removed ”) And the of the detected bounding box X、Y Coordinate information . As shown in the figure below , I'm having some delays —— come from ESP32 The serial stream of lags behind the video output .

I also have some instability of periodic interface collapse .Luxonis Currently refactoring spi-api In order to improve the spi throughput , I hope this will also solve the problem of instability . I will postpone the work of this interface , Until the update is released . In principle, , In my final solution ,ESP32 Will process data from VPU Detection and identification data , And will MQTT Send the information to the broker . I will also use... From microwave detectors MQTT Information to trigger VPU infer .

Power management

One of the things I observed when running reasoning for a long time is ,VPU The radiator on the becomes very hot under passive cooling . If I can change when I run reasoning continuously for a long time , I observed one thing , Namely VPU The heat sink on the becomes very hot during passive cooling . If I can change the direction of the heat sink , Make it vertical , For better convective cooling , It will help . I may try different radiators later , Or install a fan for active cooling . In any case , I need to study the power used to quantify , And try to reduce it . This will enable me to use the backup battery in case of power failure , And may continue to use passive cooling .Luxonis Pointed out that ,VPU The maximum operating temperature of is 105℃.

DepthAI Of API Allow to record VPU Performance information for , Such as cpu And memory usage and chip temperature . To get a baseline , I use the mobilenet-ssd Example , And in 30 Drew it in minutes .

Use the following command to record the temperature to a file :

python dep Thai _ demo . py-CNN mobile net-SSD-report temp-report _ file mobile net _ templog 30min . CSV

mobilenet-ssd Temperature measurement test conditions :

Usually , I don't expect someone to / Something exceeds... In the detection area 5 minute , So in terms of chip temperature , I should have no problem （ Unless the ambient temperature is very high or I use high-power pipes , Otherwise, there is room for continuous work ）.

However , There is still a concern about total power . A quick way to reduce power by reducing the reasoning cycle is to reduce the frame rate of the camera （FPS）. The following is the face detection I plan to use in this project / Face recognition pipeline NN Fast test of power and frame rate .

I use 1FPS As my baseline , And draw when I increase the frame rate to 30FPS Power difference at . The baseline power is 2.1 watts .

The relationship between the increase of power and frame rate is from 5FPS It's pretty linear at first . stay 5FPS when , The reaction to exercise seems a little too slow , So I'm going to see 10FPS Is it a good compromise .

Project pipeline

DepthAI There is one in VPU The concept of a running processing pipeline . It consists of a set of nodes and links between them . This is a similar to what I used to do in OpenCV Used in Gstreamer The concept of pipes .

The following is a simplified pipeline diagram of my project . It uses two sets of neural network nodes . The first group will detect more than 2 Carry out personnel detection and tracking at the time of meter , The second group will be the object in 2 Face detection and recognition within meters （ The threshold may change ）. The stereo camera provides distance measurement from the spatial position calculator , And pass it to a script node , The node controls which neural network receives RGB Infer from the output of the camera .RGB Camera images and NN The inference result uses Xout Node passed to host .

DepthAI The Conduit

Below I create an operation that I want to implement to handle the pipeline " simulation ". This is a simulation , Because it operates the neural network sequentially on the input video , Instead of switching the input of the camera . As you can see from the video , Independent character detection and image recognition network pipelines are operating separately . I'm still trying to make the network switch work with the camera .

AI Enhanced safety monitoring project ：3 Human detection demo（ See the introduction for project details and code ）

Control the image flow in the pipeline

I struggled for a while to understand how to use script nodes in the pipeline to guide the image flow .

I finally created a simple example to test whether I was right . I started from "ImageManip Tiling " Example of , It divides the camera preview box horizontally into two halves , And display the two halves side by side . then , I added a spatial positioning calculator and a script node , So that the average depth of return on investment is greater than 2 Swap tile images in meters . You can see it working below . When I move in and out of the region of interest in the center of the picture （ROI） when , The measured depth has changed , This leads to the exchange of images in the tiles .

AI Enhanced safety monitoring project ：4 Use deep swap tiles（ See the introduction for project details and code ）

Now I can adjust the script node with the spatial positioning calculator , To guide NN The input of .

Switch the used in the pipeline NN Output

I made a short presentation , stay "mobilenet-ssd " and "face-detection-retail-0004 " The neural networks switch according to the object distance calculated based on spatial depth . I need to integrate this function into my terminal application , The more complex pipeline I'm using .

AI Enhanced safety monitoring project ：5 Based on close exchange NN（ See the introduction for project details and code ）

Project summary

I couldn't be in tinyML Complete a deployed independent project within the time frame of the visual challenge , But I've been able to show all the elements I'm integrating .

I want to thank Luxonis And Intel's great product for achieving visual AI at the edge . And special thanks Luxonis The team is in its Discord Timely support on the channel .

I look forward to using DepthAI and OpenVino Complete this project and many other projects . These tools actually make development very interesting , Because your ability to them is getting stronger and stronger .

Chart

Portable microwave sensor schematic link

Code

Download link

---

OAK China | track AI New developments in technology and products
official account | OAK Visual artificial intelligence development
spot 「 here 」 Add wechat friends （ Remarks ）
stamp 「+ Focus on 」 Get the latest information