The programmer's girlfriend gave me a fatigue driving test

This article is shared from Huawei cloud community 《

ModelBox  The development case  -  Fatigue driving test

》, author ：  Push ups grab a cake  .

This case will use the  Face Detect  And  Face Mesh  Two models , Realize a simple fatigue driving detection application and remind , The final effect is as follows ：

Resources required for the case （ Code 、 Model 、 Test data, etc ）  Can be obtained from obs Bucket download .

Development background

" Thousands of roads , Safety clause 1 ", Traffic safety is the primary concern of our travel , Fatigue driving has always been a major cause of traffic accidents .

stay  AI  Algorithms benefit the present of life , We want to use  ModelBox  Rapid implementation of development board 、 Features of convenient debugging and development , Implement a fatigue driving reminder application case  demo, Realize basic fatigue driving detection and auxiliary reminding effect . I hope you can get inspiration and reference from this case , Develop more rich 、 Interesting applications .

Algorithm principle

The main principle of the project is ： Locate the face area through the face detection module , Input it to the face key module to detect the face key points , And to the eyes 、 The mouth and other key points calculate its  EAR  Value and duration judge whether it is in a fatigue state and remind .

Detect the face area from the camera

Key points are detected from the face

Calculate by formula  EAR  value , And calculate the proportion according to the real-time sliding window , If the threshold value is exceeded, it is judged as fatigue and should be reminded

ModelBox  Introduce

ModelBox  It is a suitable for the end cloud scene  AI  Reasoning application development framework , Provides the basis for  Pipeline  Parallel execution process of , Can help  AI  Application developers develop highly efficient software quickly , High performance , And support software and hardware collaborative optimization  AI  application .

Easy to develop
AI Reasoning business visual choreography development , Function modularization , Rich component library ;c++,python Multilingual support .

Easy to integrate
Integrate components docked on the cloud , Cloud docking is easier .

High performance , Highly reliable
pipeline Run concurrently , Data computing intelligent scheduling , Refinement of resource management and scheduling , Business operation is more efficient .

Heterogeneous software and hardware
CPU,GPU,NPU Multi heterogeneous hardware support , More convenient and efficient use of resources .

The whole scene
video , voice , Text ,NLP The whole scene , Customized for service , Cloud integration is easier , Seamless exchange of end cloud data .

Easy to maintain
Visualization of service running status , application , Real time monitoring of component performance , Optimization is easier .

model training

We can use a one-stop shop for developers  AI  Development platform  ModelArts  Train the model ：

ModelArts  Includes data annotation , Training environment , Rich functions including preset algorithms , You can even subscribe to preset algorithms to achieve zero code model training . Of course, you can also train your own model locally . We assume that you now have a trained model , Next, we need to convert the trained model into a model that can run on the development board . This case can directly use the pre trained model to build the initial application .

Model transformation

We released a development board model transformation case , See RK3568 Model transformation validation cases  ：

In this case, we demonstrate the whole process sample code from environment adaptation to model transformation verification , Developers can use  “Run in ModelArts”  One click  Notebook  The case is  ModelArts  The console opens quickly 、 Operation and secondary development .

Development environment deployment

Use the development board to  ModelBox AI  There are two ways to develop applications , First, the development board connects the display with the keyboard and mouse , install  Ubuntu  desktop , Develop directly on the development board ; The second is to use remote connection tools （ Such as  VS Code  Medium  Remote-SSH） from  PC  Log in to the development board for development . Here we recommend the second way , because  PC  The end can be used with more functions 、 More friendly interface  IDE. Please refer to the development environment deployment of this case to build the environment first ：

application development

Next, we will take fatigue driving test  demo  For example , How to use  ModelBox  Development  AI  application .

Create a project

SDK  Provides project scripts  create.py, have access to ./create.py -h  View script help ：

[email protected]:~/yourpath/v1.0.8.21$ sudo ./create.py -h

Usage: Create ModelBox project and flowunit

NOTE : you must firstly use bellow cmd to create a project in workspace
 create.py -t server -n your_proj_name {option: -s name, create this project from a solution}, support hilens deployment
 or create.py -t project -n your_proj_name {option: -s name, create this project from a solution}, generally not use
AND : use bellow cmd to create [c++|python|infer] flowunit in this project
 create.py -t c++ -n your_flowunit_name -p your_proj_name
AND : call workspace/your_proj_name/build_project.sh to build your project, call bin/main.sh[bat] to run
FINAL: create.py -t rpm -n your_proj_name to package your project (the same folder with create.py) if upload to hilens

NOTE: create.py -t editor {option: -i ip or ip:port to start editor server in your config ip:port}
NOTE: create.py -t demo to create solutions to runnable demo

for ex: create.py -t server -n my_det -s car_det

-h or --help：show help
-t or --template [c++|python|infer|yolo|project|server|rpm|editor|demo] create a template or package to rpm ...
-n or --name [your template name]
-p or --project [your project name when create c++|python|infer|yolo]
-s or --solution [the solution name when create project] create a project from solution
-c or --cfg [flowunit configure json, it's used by UI, you can use it too, but too complicated]
-v or --version：show sdk version

ModelBox  Provides a visual diagram layout tool ：Editor, have access to ./create.py -t editor  Start the diagram editing service ： Service default  ip  That is to say  192.168.2.111, If you need to configure other  ip  Or port , Can pass  - i ip：port  Parameters to configure .

Type in the browser  ip：port/editor  You can enter the visual editing interface , We click Edit to enter the engineering development interface , If you learn more about  ModelBox  Related content , You can click Help in the upper right corner ：

Enter the editing interface , Click new project in the upper right corner ：

We are  workspace  Project path , Create project name  fatigue_detect, Click on the confirmation ：

You can see from the default template , We initially had a  http  Default diagram of transceiver unit ：

among , Area  1  by  SDK  Preset high-performance universal flow unit , Area  2  Choreograph the interface for visualization , Area  3  Configure the file contents for the corresponding graph . meanwhile ,VS Code  The corresponding directory also appears  fatigue_detect  project ：

Create reasoning function flow unit

Next , We go through  UI  Interface to create inference flow unit ：

Click the function unit in the upper right corner , Create cell

For face detection model , We name the flow unit face_detection, The model file name is the converted detection model name ：face_detect.rknn, This model is input as an image , Output as eigenvector , So we add  int  The input port of type is the same as  float  Type of output port . About the reasoning flow unit creation of the development board , When dealing with types, we choose  cuda, That is to say  npu  Reasoning , The inference engine can choose any one , Current development board  SDK  Automatic recognition and conversion . Finally, the grouping of functional units is modified to  inference, Click on the confirmation , You can see , On the left side  inference  The newly added inference flow unit appears under the tab

meanwhile , stay  VS Code  engineering  model  The created inference flow unit can be seen in the directory ：

Allied , We created  face_landmark_detection  Inference flow unit ：

Create post-processing function unit

In addition to inference flow units , Fatigue driving test  demo  Some common functional units are also required ： A post-processing unit for face detection 、 Region of interest （ Face detection area ） Extraction unit 、 Fatigue logic judgment unit , Let's build three  python  Functional units to meet the above requirements .

For the post-processing unit of face detection , We hope that through the original drawing and  face_detection  The face detection frame is decoded from the two output vectors of , So the unit should have three inputs . Besides , Whether there are two kinds of facial states detected in the picture , We hope that this function unit can handle the problem according to different situations , When a face is detected , The face detection result is sent to the region of interest extraction unit , When no face is detected , Go straight back to , Therefore, the function unit type is selected ：IF_ELSE. The new unit is as follows ：

alike , According to input and output and functional unit status , We created  extract_roi  and  fatigue_detect  Two functional units ：

Flow chart layout

Drag and drop to create all the required functional units , We can start to lay out the flow chart , Let's make a diagram of video processing , You can delete what you don't need  http  Transceiver unit ：

stay  Generic  The virtual input unit... Will be displayed under the list  input  And the three function units we just created are dragged into the canvas ：

stay  Image  List the preprocessing units required for model reasoning  resize  Drag into the canvas , Because both the face detection model and the face key point model  resize  Handle , Take two.  resize  Unit drag ：

It is worth noting that ,resize  The unit needs to adapt to the model parameters , You need to click this unit to configure ：

stay  Input  Drag the input parsing unit under the list  data_source_parser  As the input source of video stream

stay  Video  Drag the unit required for video processing under the list  video_demuxer、video_decoder、video_out：

Last , stay  Inference  Drag the two reasoning units we created under the list ：

layout

Connect the functional units according to the processing logic ：

Click the previous flow cell and drag the arrow to connect the next flow cell

Virtual input  input  Connection input resolution  data_source_parser, After parsing, it is sent to the video for unpacking  video_demuxer  And decoding unit  video_decoder：

After the decoded output is sent into the preprocessing, the inference can be carried out directly , The face detection output is connected to the post-processing module , Used to extract coordinates input to the region of interest , If no face is detected, skip face key point detection and fatigue driving judgment ：

Last , hold  ROI  Area map  resize  Then input to face key point detection , Finally, the key points of the face are input into the fatigue driving judgment module to give the results ：

such , We have finished the layout of the flow chart , You can see in the  GraphViz  There is also a complete graphical representation of the region ：

Save the project , Let's switch to  VS Code  The code implementation of each unit is adjusted and supplemented ：

Code completion

Visual orchestration , The reasoning unit created is located in the project  model  Under the table of contents , The common unit is located at  etc/flowunit  Under the table of contents , The flow chart is located at  graph  Under the table of contents , You can see that the created cells and diagrams have been synchronized ：

among , We need to modify it in the main diagram code  video_decoder  Type required ：

`video_decoder [ type=flowunit flowunit=video_decoder device=rknpu deviceid=&quot;0&quot; pix_fmt=bgr label=&quot;{{<in_video_packet> in_video_packet}|video_decoder|{<out_video_frame> out_video_frame}}&quot; ]`
** Reasoning unit **

First, improve the reasoning unit , For reasoning functional units , Only need to provide independent  toml  The configuration file , Specify the basic attributes of the reasoning function unit , The directory structure is ：

[flowunit-name]
 |---[flowunit-name].toml # Inference function unit configuration
 |---[model].rknn # Model file
 |---[infer-plugin].so # Reasoning custom plug-ins

ModelBox  When the framework is initialized , Will scan the directory for  toml  Postfix file , And read the relevant reasoning function unit information .[infer-plugin].so  Is the plug-in required for reasoning , The reasoning function unit supports loading custom plug-ins , Developers can implement custom operators .

Copy the model into the corresponding folder , With  face_detection  For example, let's take a look at the inference function unit configuration file ：

Because the input of inference unit is image , Note that the input type is modified to  uint8, There are some unit types in the configuration file 、 The model name 、 Basic configuration of input and output , It can be modified as appropriate .

General unit

Python  The common unit needs to provide independent  toml  The configuration file , Appoint  python  Basic attributes of functional units . General situation , The directory structure is ：

[FlowUnitName]
 |---[FlowUnitName].toml
 |---[FlowUnitName].py
 |---xxx.py

Compared with the reasoning unit , A generic unit has not only configuration files , It also needs to improve the specific function code , With  face_detect_post  For example , The first is the functional unit configuration file ：

Basic config  Some basic configurations such as unit names ,Flowunit Type  Is the type of functional unit ,face_detect_post  Is a conditional unit , So you can see  condition  by  true, There are also some expansions 、 Convergence and other properties , Can be in  AI Gallery ModelBox)  See more cases under the plate .

Besides , Input and output  type  Some modifications may be made according to the actual logic .

Next , We see the  face_detect_post.py, You can see that the basic interface has been generated when the unit is created ：

# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import _flowunit as modelbox

class face_detect_postFlowUnit(modelbox.FlowUnit):
 # Derived from modelbox.FlowUnit
 def __init__(self):
 super().__init__()

 def open(self, config):
 # Open the flowunit to obtain configuration information
 return modelbox.Status.StatusCode.STATUS_SUCCESS

 def process(self, data_context):
 # Process the data
 in_data = data_context.input("in_1")
 out_data = data_context.output("out_1")

 # yolox_post process code.
 # Remove the following code and add your own code here.
 for buffer in in_data:
 response = "Hello World " + buffer.as_object()
 result = response.encode('utf-8').strip()
 add_buffer = modelbox.Buffer(self.get_bind_device(), result)
 out_data.push_back(add_buffer)

 return modelbox.Status.StatusCode.STATUS_SUCCESS

 def close(self):
 # Close the flowunit
 return modelbox.Status()

 def data_pre(self, data_context):
 # Before streaming data starts
 return modelbox.Status()

 def data_post(self, data_context):
 # After streaming data ends
 return modelbox.Status()

 def data_group_pre(self, data_context):
 # Before all streaming data starts
 return modelbox.Status()

 def data_group_post(self, data_context):
 # After all streaming data ends
 return modelbox.Status()

If the working mode of the functional unit is  stream = false  when , The functional unit will call  open、process、close  Interface ; If the working mode of the functional unit is  stream = true  when , The functional unit will call  open、data_group_pre、data_pre、process、data_post、data_group_post、close  Interface ; Users can implement corresponding interfaces according to actual requirements .

According to the nature of the unit , We mainly need to improve  open、process  Interface ：

def process(self, data_context):
 # Process the data
 in_image = data_context.input("image")
 in_feat = data_context.input("in_feat")
 in_score = data_context.input("in_score")

 has_face = data_context.output("has_face")
 no_face = data_context.output("no_face")
 # face_detect_post process code.
 # Remove the following code and add your own code here.
 for buffer_img, buffer_feat, buffer_score in zip(in_image, in_feat, in_score):
 width = buffer_img.get('width')
 height = buffer_img.get('height')
 channel = buffer_img.get('channel')

 img_data = np.array(buffer_img.as_object(), copy=False)
 img_data = img_data.reshape((height, width, channel))

 feat_data = np.array(buffer_feat.as_object(), copy=False)
 feat_data = feat_data.reshape(896,16)

 feat_score = np.array(buffer_score.as_object(), copy=False)
 feat_score = feat_score.reshape(896,1)

 dets = process_cpu(feat_data, feat_score, self.anchors)

 if dets.shape[0] > 0:
 box = [int(dets[0]*width), int(dets[1]*height),int(dets[2]*width),int(dets[3]*height)]
 box = self.expend(box, height, width)
 buffer_img.set("bboxes", box)
 has_face.push_back(buffer_img)

 else:
 img_buffer = modelbox.Buffer(self.get_bind_device(), img_data)
 img_buffer.copy_meta(buffer_img)
 no_face.push_back(img_buffer)

 return modelbox.Status.StatusCode.STATUS_SUCCESS


You can see , stay  open  In, we get some parameters ,process  Do logical processing , Input and output can be through  data_context  To get , It is worth noting that when we output, we return a graph , When the hand is detected, the detection frame information is attached to the graph , This information can be obtained by the next unit .

alike , Improve other general functional units , For details, please refer to the code provided by us .

Application and operation

We need to prepare a  mp4  File copy to  data  Under the folder , We provide a test video  fatigue.mp4, Then open the project directory  bin/mock_task.toml  file , Modify the task input and task output configuration as follows ：

#  For local mock File reading task , The script has been configured IVA_SVC_CONFIG environment variable ,  Added this file path
###########  Please make sure to use linux Path type , For example windows Use  D:/xxx/xxx  Out-of-service D:\xxx\xxx ###########
#  The task parameter is a compressed and escaped json character string
#  Writing directly requires escaping double quotation marks ,  It can also be used.  content_file  Add one json file , If content and content_file All exist content Will be covered
# content_file Support absolute path or relative path , Parsing environment variables... Is not supported ( Include ${HILENS_APP_ROOT}、${HILENS_DATA_DIR} etc. )
[common]
content = "{\"param_str\":\"string param\",\"param_int\":10,\"param_float\":10.5}"

#  Task input ,mock Simulation currently only supports one way rtsp Or local url
# rtsp camera ,type = "rtsp", url Write in rtsp Address
#  Other uses "url", For example, it can be a local file address ,  perhaps httpserver The address of ,( camera  url = "0")
[input]
type = "url"
url = "../data/fatigue.mp4"

#  Task output , Currently only supported "webhook",  And local output "local"( Output to the screen ,url="0",  Output to rtsp, Fill in rtsp Address )
# (local  You can also export to a local file , Pay attention to , Files can be relative paths , Is relative to this mock_task.toml The document itself )
[output]
type = "local"
url = "../hilens_data_dir/fatigue.mp4"

After configuration, execute in the project path  build_project.sh  Carry out engineering construction ：

[email protected]:~/yourpath/v1.0.8.21/workspace/fatigue_detect$ sudo ./build_project.sh
./build_project.sh: line 26: dos2unix: command not found
./build_project.sh: line 26: dos2unix: command not found
./build_project.sh: line 26: dos2unix: command not found
./build_project.sh: line 26: dos2unix: command not found
./build_project.sh: line 26: dos2unix: command not found
./build_project.sh: line 26: dos2unix: command not found
./build_project.sh: line 26: dos2unix: command not found
./build_project.sh: line 26: dos2unix: command not found

build success: you can run main.sh in ./bin folder Copy

Run the project after the build is complete ：

[email protected]:~/yourpath/v1.0.8.21/workspace/fatigue_detect$ sudo ./bin/main.sh Copy

Wait a while and you will be able to  hilens_data_dir  See the running results under the folder ：

open  mp4  You can also see the video with the reminder effect superimposed on the file

Packaged deployment

After debugging ,create.py  The script packages and publishes the application , Please refer to

Packaging deployment chapter

Click to follow , The first time to learn about Huawei's new cloud technology ～