Inference optimization implementation of tensorrt model

Express parcel segmentation module is a decisive link in the single piece separation system , The accuracy and reasoning speed of express parcel segmentation algorithm affect the overall performance of the system . Want to TensorRT The optimized express parcel segmentation algorithm needs to go through three stages to deploy to the embedded development board , They are the training stages of the network model 、 Network model format conversion stage and TensorRT The optimization stage of the model structure by the inference optimizer .

1  Network model storage format conversion  

      In the use of TensorRT Before the inference optimizer optimizes the model structure , You need to import the network model structure and weight parameters into the inference optimizer first . At present ,TensorRT contain nvcaffeparse、nvuffparse and nvonnxparse Three file parsers , Can support right .caffe file 、.uff Document and .onnx File analysis . Express parcel segmentation algorithm is based on PyTorch Neural network model of deep learning framework training , therefore , The trained network model structure and weight parameters can be expressed as ONNX Save as .onnx file . 

      at present ,PyTorch The deep learning framework can use torch.onnx.export Function saves the trained network model structure and weight parameters as .onnx file . however , Before using the above functions to derive the network model structure and weight parameters , You need to consider ONNX Whether all operations in the network structure are supported and whether post-processing operations in the network model should also be converted to .onnx In file . about ONNX Some operations that are not supported , There are usually two solutions ： The first method is to use other approximate operations to replace ; The second way is to put PyTorch Replace the defined operation with the operation implemented by yourself . For using custom actions , You need to consider TesnorRT Whether the framework supports and implements efficiently , therefore , In the process of network model format conversion, try to use approximate operations to replace . For the post-processing process of the network model , There are usually operations such as non maximum suppression , There are some logical judgments in this operation （ such as if sentence ）,ONNX The network model file saved in this format is powerless for these logical operations . meanwhile , For the implementation of post-processing , Everyone may operate differently , This is useful for TesnorRT The inference optimizer increases the difficulty of optimizing the network model structure . All in all , In order to reduce the possible risks in the process of network model structure transformation , The post-processing operation is best implemented by the developer . 

2 TensorRT Optimize model structure

      Use TensorRT The reasoning framework deploys the network model , Must be used first TensorRT The inference optimizer pairs ONNX The model structure is optimized and generated TensorRT After the runtime engine , To input data to the network model , Then the inference operation is performed to obtain the final output result . however ,TensorRT The process of optimizing the network model structure by the inference optimizer is very time-consuming , therefore , In the actual application process, it is generated separately first TensorRT Inference engine , Then it is serialized into binary files and saved locally . When applied , You just need to deserialize the binary file stored locally , You can quickly generate an inference engine , Saved a lot of time . 

TensorRT The construction process of the inference engine is shown in the following figure , The specific steps are as follows ： 

1.  Use nvinfer1::createInferBuilder Function to create a nvinfer1::IBuilder object builder; 

2.  stay builder Yes like On transfer use createNetworkV2 Letter Count gen build One individual nvinfer1::INetworkDefinition object network; 

3.   By calling nvonnxparse::createParser Function and pass in network Parameter to create a nvonnxparser::IParser object parser To resolve the imported ONNX Network model file ; 

4.   stay builder Yes like On transfer use createBuilderConfig Letter Count gen build One individual nvinfer1::IBuilderConfig object config, Through this object, you can set some parameters used in the process of model reasoning and optimization , for example , The workspace size of the network model 、 The number of input pictures and the type of data used in network model reasoning each time ; 

5.   stay builder Object builderEngineWithConfig Function and pass in network and config Parameters , You can do that. ONNX The network model structure and weight parameters are optimized for reasoning and data type calibration , The last generation nvinfer1::ICudaEngine object mEngine, That is to say TensorRT The inference engine generated after the inference optimizer processes ; 

6.   Put the generated reasoning engine mEngine Serialized into binary files and stored on disk , It is convenient to call directly in the process of subsequent model deployment .