Use openvinotm preprocessing API to further improve the reasoning performance of yolov5

summary

stay 《 be based on OpenVINOTM 2022.1 Realization YOLOv5 Reasoning program 》 Detailed in ：

1. YOLOv5 How to install and export the framework YOLOv5.onnx Model
2. OpenVINOTM 2022.1 And how to write YOLOv5 Model inference program

This article will show you how to use OpenVINOTM 2022.1 Preprocessing API, Further improve YOLOv5 The reasoning and computing performance of the model

What is pretreatment API function ？

OpenVINO 2022.1 Previous versions did not provide OpenVINOTM Runtime Native for data preprocessing API function [1], Pictured 1-1 Shown , Developers must use third-party libraries ( for example ：OpenCV) To achieve data preprocessing .

chart 1-1  OpenVINOTM Runtime Preprocessing API

Assume no pretreatment API, Then the preprocessing operation of input data can only be placed in CPU Implemented on ,CPU After data preprocessing , And then transmit the preprocessed data to iGPU、VPU etc. AI Accelerate computing devices for reasoning .

With pretreatment API after , The preprocessing operations can be integrated into the model execution diagram , such iGPU、VPU Or upcoming Intel Independent graphics cards can perform data preprocessing , Don't need to rely on CPU, Improved execution efficiency , Pictured 1-2 Shown .

 

chart 1-2  Preprocessing OpenCV vs OpenVINO

​​​​​​​ Typical operations of data preprocessing

Due to the Shape、Precision Other characteristics , Inconsistent with the requirements of model input tensor , Therefore, pretreatment is required , Transform the input data according to the requirements of the model input tensor , Such as chart 1-3 Shown .

 

 

chart 1-3  input data vs Model input tensor

From the picture 1-3 You can see , Typical operations of data preprocessing are ：

1. Change the shape of the input data ：[720, 1280,3] → [1, 3, 640, 640]
2. Change the accuracy of the input data ：U8 → f32
3. Change the color channel order of input data ：BGR → RGB
4. Change the layout of the input data (layout)：HWC → NCHW
5. Normalized data ： Subtract the mean (mean), Divided by standard deviation (std)
   1. Data preprocessing API How to use

Typical operations corresponding to data preprocessing ,OpenVINOTM Preprocessing API Corresponding classes are provided , Easy for developers to use quickly , The main processes are 6 Step [2], Pictured 1-4 Shown , In turn, is ：

1. Instantiation PrePostProcessor object ;
2. Declare the information of the input data
3. Specify the data layout of the model (layout)
4. Set the information of the model output tensor
5. Define the specific steps of preprocessing
6. Integrate preprocessing steps into the model

 

chart 1-4  Use pretreatment API The process of

This article will introduce... In the above order .

Instantiation PrePostProcessor object

Instantiation PrePostProcessor Object's Python Code , Such as code list 1-1 Shown .

Code list 1-1  Instantiation PrePostProcessor object

from openvino.runtime import Core, Type, Layout

from openvino.preprocess import PrePostProcessor, ColorFormat

# Please modify the model path

model_path ="yolov5s.onnx"

model = core.read_model(model_path)

# Step1: Instance PrePostProcessor object

ppp = PrePostProcessor(model)

​​​​​​​ Declare the information of the input data

Declare the information of the input data Python Code , Such as code list 1-2 Shown .

Code list 1-2  Declare the information of the input data

# Step2: Declare input data information:

ppp.input().tensor() \

    .set_color_format(ColorFormat.BGR) \

    .set_element_type(Type.u8) \

    .set_layout(Layout('NHWC'))  

​​​​​​​ Specify the data layout of the model (layout)

Specify the data layout of the model (layout) Of Python Code , Such as code list 1-3 Shown .

Code list 1-3  Specify the data layout of the model (layout)

# Step3: Specify actual model layout

ppp.input().model().set_layout(Layout('NCHW'))

​​​​​​​ Set the information of the model output tensor

Set the information of the model output tensor Python Code , Such as code list 1-4 Shown .

Code list 1-4  Set the information of the model output tensor

# Step4: Set output tensor information:

# - precision of tensor is supposed to be 'f32'

ppp.output().tensor().set_element_type(Type.f32)

​​​​​​​ Define the specific steps of preprocessing

Define the specific steps of preprocessing Python Code , Such as code list 1-5 Shown .

 

Code list 1-5  Define the specific steps of preprocessing

# Step5: Apply preprocessing modifing the original 'model'

# - Precision from u8 to f32

# - color plane from BGR to RGB

# - subtract mean

# - divide by scale factor

# - Layout conversion will be done automatically as last step

ppp.input().preprocess() \

    .convert_element_type(Type.f32) \

    .convert_color(ColorFormat.RGB) \

    .mean([0.0, 0.0, 0.0]) \

    .scale([255.0, 255.0, 255.0])

​​​​​​​ Integrate preprocessing steps into the model

Integrate preprocessing steps into the model's Python Code , Such as code list 1-6 Shown .

Pretreatment method	CPU/iGPU model	Reasoning equipment	The number of iterations	The elapsed time
OpenVINOTM Preprocessing API	i7-1165G7 IntelIrisXe	device=”GPU”	500	12.68s
OpenCV	i7-1165G7 IntelIrisXe	device=”GPU”	500	13.43s

Code list 1-6  Integrate preprocessing steps into the model

# Step6: Integrate preprocessing steps into model

print(f'Build preprocessor: { ppp}')

model = ppp.build()

​​​​​​​ Export the model with integrated preprocessing steps

Use serialize() function , Models with integrated preprocessing steps can be exported , Convenient for subsequent calls , Such as code list 1-7 Shown .

Code list 1-7  Export the model with preprocessing steps

# Save the Model with preprocess

from openvino.offline_transformations import serialize

serialize(model, 'yolov5s.xml', 'yolov5s.bin')

Use Netron Open the export model , You can see that the preprocessing steps have been integrated into the execution diagram , Pictured 1-5 Shown .

chart 1-5  Preprocessing integrated into execution diagram

Export the complete source code of the integration preprocessing model ：https://gitee.com/ppov-nuc/yolov5_infer/blob/main/preprocessing_with_saving_to_IR.py

Complete sample code and test results

Used with this document OpenVINOTM 2022.1 Preprocessing API Realization YOLOv5s The complete source code of the inference program , See ：infer_with_openvino_preprocess.py · PPOV_NUC/yolov5_infer - Gitee.com.

surface 1-1 Use OpenVINOTM 2022.1 Preprocessing API And use OpenCV Performance comparison of pretreatment

operating system ：Windows10;Python edition ：3.8;OpenVINO edition ：2022.1

Model ：yolov5s.onnx

Pretreatment method	CPU/iGPU model	Reasoning equipment	The number of iterations	The elapsed time
OpenVINOTM Preprocessing API	i7-1165G7 IntelIrisXe	device=”GPU”	500	12.68s
OpenCV	i7-1165G7 IntelIrisXe	device=”GPU”	500	13.43s

Carry out orders , take yolov5s.onnx Convert to FP16 Accurate yolov5s.xml after

mo --input_model yolov5s.onnx --data_type FP16

operating system ：Windows10;Python edition ：3.8;OpenVINO edition ：2022.1

Model ：yolov5s.xml @ FP16

Pretreatment method	CPU/iGPU model	Reasoning equipment	The number of iterations	The elapsed time
OpenVINOTM Preprocessing API	i7-1165G7 IntelIrisXe	device=”GPU”	500	7.41s
OpenCV	i7-1165G7 IntelIrisXe	device=”GPU”	500	8.24s

​​​​​​​ summary

This article gives a complete introduction to what is OpenVINO  Preprocessing API And why pretreatment is recommended API Integrate preprocessing operations into the model execution diagram , Then it introduces the use steps in detail and provides the complete sample source code .

By running the source code , You can see , Used OpenVINO Preprocessing API, Make the input data preprocessing operation no longer depend on CPU, This can be done by reasoning devices ( Such as GPU/VPU) complete , It improves the efficiency of reasoning calculation , Reduced running time .

reference ：

[1] openvino.preprocess — OpenVINO documentation

[2] Preprocessing API - details — OpenVINO documentation

[3] openvino_notebooks/002-openvino-api.ipynb at main · openvinotoolkit/openvino_notebooks · GitHub