Using OpenVINO to implement the flying paddle version of the PGNet inference program

目录

第1章 使用OpenVINOTMRealize the flying paddle versionPGNet推理程序

1.1OpenVINOTM 简介

1.2 PGNet简介

1.3准备PGNet的OpenVINO推理程序开发环境

1.4下载PGNet预训练模型

1.5用OpenVINOTM读取PGNet预训练模型

 1.6使用OpenVINOTM开发PGNet的推理程序

    1.6.1升级PaddleOCRComes with a predictor,支持OpenVINO推理

  1.6.2借鉴PaddleOCRpreprocessing and postprocessing code,实现OpenVINO推理程序

 1.7总结：

---

 使用OpenVINOTMRealize the flying paddle versionPGNet推理程序

作者：杨雪峰  英特尔物联网行业创新大使

1.1OpenVINOTM 简介

OpenVINO 工具包2022.1版于2022年3月22日正式发布,根据官宣《OpenVINO 迎来迄今为止最重大更新,2022.1新特性抢先看》,OpenVINO 2022.1将是迄今为止最大变化的版本,And can directly support reading the propeller model.

1.2 PGNet简介

Recognition of free-form text(例如,弯曲文本)received more and more research attention,And the application is more and more extensive(例如,Billboard identification、Seal recognition, etc),如下图所示.

AAAI 2021会议上发表的百度自研的端到端场景文本识别PGNet[1]算法,Effectively meet the above requirements,其典型特点有：

• PGNet是一种新颖的、端到端的、Arbitrary shape text detection recognizer framework
• 不需要字符级别的标注,NMS操作以及ROI操作[2]
• A prediction module of reading order in text lines and a graph-based correction module are proposed to improve the text recognition effect
• The recognition accuracy and running speed are excellentSOTA

*上图引用自PNGet论文[1]

另外,paddle versionPGNet的工程化和文档化做的极好,没有任何基础的人,都可以在不到半天的时间内完成PGNet的开发环境搭建、模型训练、ONNX模型导出.

PGNet的文档链接：https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.5/doc/doc_ch/algorithm_e2e_pgnet.md

1.3准备PGNet的OpenVINO推理程序开发环境

要完成PGNet的OpenVINO推理程序开发,需要安装：

• PaddleOCR运行环境,参考：
  1. 安装PaddleOCR运行环境
  2. 克隆PaddleOCR到本地,and install third-party libraries
• OpenVINOTM开发工具

pip install openvino-dev[onnx]

1.4下载PGNet预训练模型

               PaddleOCR已提供PGNet预训练模型,Please download and unzip it by yourself,如下图所示.

PGNet预训练模型下载链接：https://paddleocr.bj.bcebos.com/dygraph_v2.0/pgnet/e2e_server_pgnetA_infer.tar

下载完毕后,运行PaddleOCRBuilt-in forecasting program tools\infer\predict_e2e.py

# 如果想使用CPU进行预测,需设置use_gpu参数为False python3 tools/infer/predict_e2e.py --e2e_algorithm="PGNet" --image_dir="./doc/imgs_en/img623.jpg" --e2e_model_dir="./inference/e2e_server_pgnetA_infer/" --e2e_pgnet_valid_set="totaltext" --use_gpu=False

运行结果,如下图所示,说明PaddleOCR和PGNetThe pretrained models are all ready.

 从图中可见,The average time to run is 5.09s.

1.5用OpenVINOTM读取PGNet预训练模型

     如前所述,OpenVINOTM支持直接读入飞桨模型.使用下面的代码,可以快速测试OpenVINOTM读入PGNet模型的效果.

from openvino.runtime import Core

# 指定PGNet模型路径

pgnet_path = ".\e2e_server_pgnetA_infer\inference.pdmodel"

# 创建Core对象

core = Core()

# Load and compilePGNet模型

sess = core.compile_model(model=pgnet_path, device_name="CPU")

# 输出PGNet模型输入&输出信息

print(sess.input)

运行效果如下图所示,说明OpenVINOTMRead the paddle version directlyPGNet模型成功.

 1.6使用OpenVINOTM开发PGNet的推理程序

        基于上述代码,结合PGNetdata preprocessing and postprocessing code,A complete one can be developedPGNet推理程序.

    1.6.1升级PaddleOCRComes with a predictor,支持OpenVINO推理

        由于PaddleOCRBuilt-in forecasting program tools\infer\predict_e2e.py,已经实现了PGNetdata preprocessing and postprocessing code,所以,只需要稍微修改：

• tools\infer\predict_e2e.py
• tools\infer\utility.py

   添加OpenVINO的推理代码,即可升级PaddleOCR对OpenVINO的支持.

  上述代码,可以从PGNet OpenVINO Inference: Do the PGNet inference based on OpenVINO 下载.下载后,Please and replace the file with the same name.

    OpenVINOThe supporting code is shown below,All modifications have been added“# OpenVINO Support Here”：

        # OpenVINO Support Here

        elif self.use_openvino:

            outputs = self.predictor([img])

            out_layers = self.predictor.output

            preds = {}

            preds['f_border'] = outputs[out_layers(0)]

            preds['f_char'] = outputs[out_layers(1)]

            preds['f_direction'] = outputs[out_layers(2)]

            preds['f_score'] = outputs[out_layers(3)]

After replacing the file with the same name,使用命令：

python tools\infer\predict_e2e.py --e2e_algorithm="PGNet" --image_dir=.\doc\imgs_en --e2e_model_dir=ov_infer\e2e_server_pgnetA_infer\inference.pdmodel --e2e_pgnet_valid_set="totaltext" --use_gpu=False --use_openvino=True

运行效果如下所示：

从运行结果可以看到,使用了OpenVINO的推理程序,The average time to run is 1.33s;没有使用OpenVINO的推理程序,The average time to run is 5.09s.OpenVINOGreatly improved the modelCPUon the computational efficiency of inference.

  1.6.2借鉴PaddleOCRpreprocessing and postprocessing code,实现OpenVINO推理程序

        直接使用ppocrThe pre-processing and post-processing code that comes with it,可以轻松实现PGNet的OpenVINO推理程序：pgnet_ov_infer.py · PPOV_NUC/PGNet OpenVINO Inference - Gitee.com

from openvino.runtime import Core

from ppocr.data import create_operators, transform

from ppocr.postprocess import build_post_process

import cv2

import numpy as np

import time



# 指定PGNet模型路径

pgnet_path = "./e2e_server_pgnetA_infer/inference.pdmodel"



# 创建Core对象

core = Core()

# Load and compilePGNet模型

pgnet = core.compile_model(model=pgnet_path, device_name="CPU")



# 创建preprocess_op

pre_process_list = [{

            'E2EResizeForTest': {

        'max_side_len': 768,

        'valid_set': 'totaltext'}

        }, {

            'NormalizeImage': {

                'std': [0.229, 0.224, 0.225],

                'mean': [0.485, 0.456, 0.406],

                'scale': '1./255.',

                'order': 'hwc'

            }

        }, {

            'ToCHWImage': None

        }, {

            'KeepKeys': {

                'keep_keys': ['image', 'shape']

            }

        }]

preprocess_op = create_operators(pre_process_list)



# 创建postprocess_op

postprocess_params = {}

postprocess_params['name'] = 'PGPostProcess'

postprocess_params["score_thresh"] = 0.5

postprocess_params["character_dict_path"] = "./ic15_dict.txt"

postprocess_params["valid_set"] = 'totaltext'

postprocess_params["mode"] = 'fast'

postprocess_op = build_post_process(postprocess_params)



def clip_det_res(points, img_height, img_width):

    for pno in range(points.shape[0]):

        points[pno, 0] = int(min(max(points[pno, 0], 0), img_width - 1))

        points[pno, 1] = int(min(max(points[pno, 1], 0), img_height - 1))

    return points

# 定义filter_tag_det_res_only_clip函数

def filter_tag_det_res_only_clip(dt_boxes, image_shape):

        img_height, img_width = image_shape[0:2]

        dt_boxes_new = []

        for box in dt_boxes:

            box = clip_det_res(box, img_height, img_width)

            dt_boxes_new.append(box)

        dt_boxes = np.array(dt_boxes_new)

        return dt_boxes



# Load image data and implement preprocessing

image_path = 'img623.jpg'

image = cv2.imread(image_path)

data = {'image': image}

data = transform(data, preprocess_op)

img, shape_list = data

img = np.expand_dims(img, axis=0)

shape_list = np.expand_dims(shape_list, axis=0)

starttime = time.time()



# Do the inference by OpenVINO

outputs = pgnet([img])

out_layers = pgnet.output

preds = {}

preds['f_border'] = outputs[out_layers(0)]

preds['f_char'] = outputs[out_layers(1)]

preds['f_direction'] = outputs[out_layers(2)]

preds['f_score'] = outputs[out_layers(3)]



post_result = postprocess_op(preds, shape_list)

points, strs = post_result['points'], post_result['texts']

dt_boxes = filter_tag_det_res_only_clip(points, image.shape)

elapse = time.time() - starttime

print(f"Predict time: {elapse}s")

import utility

src_im = utility.draw_e2e_res(points, strs, image_path)

cv2.imshow("PGNet infer by OpenVINO", src_im)

cv2.waitKey(0)

cv2.destroyAllWindows()

运行结果,如下图所示：

 1.7总结：

        Paddle versionPGNetis an easy to learn and easy to use、Documentation and engineering are very well done[3]、端到端的、Models that can detect curved text.

        由于从OpenVINO 2022.1版本开始,OpenVINO RuntimeDirect reading of the propeller model has been supported,且OpenVINO的API简单易用,所以,Can be easily upgradedPaddleOCRBuilt-in forecasting program tools\infer\predict_e2e.py支持OpenVINO推理计算,Or develop from scratchOpenVINO推理程序.

        经过OpenVINO优化后,PGNet的CPUThe inference calculation efficiency is greatly improved,Average runtime from 5.09s,提升到1.33s,效率提升非常明显.

        注意：运行时间,仅供参考,在不同CPU上,The running time will vary.

参考资料

[1] PGNet: Real-time Arbitrarily-Shaped Text Spotting with Point Gathering Network, https://www.aaai.org/AAAI21Papers/AAAI-2885.WangP.pdf

[2] PaddleOCR FAQ: PaddleOCR/FAQ.md at release/2.5 · PaddlePaddle/PaddleOCR · GitHub

[3] PGNet repo: https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.5/doc/doc_ch/algorithm_e2e_pgnet.md