当前位置:网站首页>[deploy pytoch project through onnx using tensorrt]
[deploy pytoch project through onnx using tensorrt]
2022-07-05 11:42:00 【Network starry sky (LUOC)】
List of articles
1 Model scheme
TensorRT The installation of can be described according to the blog TensorRT Installation and usage guide and windows install tensorrt understand .
quote 【1】 title :TensorRT Installation and usage guide link :
https://blog.csdn.net/zong596568821xp/article/details/86077553
author :ZONG_XP
quote 【2】 title : windows install tensorrt link :
https://zhuanlan.zhihu.com/p/339753895
author : Know user 15Z1y4
1.1 Deployment process
be based on ONNX Route , call C++、Python Interface and hand it over to Builder, Finally, build the engine .
1.2 Export correctly onnx
I wrote a simple example :
import torch
import torch.nn as nn
class Model(nn.Module):
def __init__(self):
super().__init__()
self.conv = nn.Conv2d(in_channels=1,out_channels=1,kernel_size=3,stride=1,padding=1,bias=True)
self.conv.weight.data.fill_(1)
self.conv.bias.data.fill_(1)
def forward(self,x):
x = self.conv(x)
return x.view(-1,int(x.numel()//x.size(0)))
model = Model().eval()
x = torch.full((1,1,3,3),1.0)
y = model(x)
torch.onnx.export(
model,(x,),"test.onnx",verbose=True
)
use netron Import the generated onnx file , The network structure and parameters can be viewed online :
1.3 stay C++ Use in
First of all, will tensorRT The next path include Copy files in to cuda Corresponding include Under the folder ,lib In the folder lib Document and dll The files are copied to cuda Corresponding lib/x64 Under the folder and bin Under the folder .
To configure VS Environmental Science , Here is the list of required configurations :
├── VC++ Catalog
│├── Contains the directory
%OPENCV_PATH%\opencv\build\include
%OPENCV_PATH%\opencv\build\include\opencv2
│├── The library catalog
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\lib\x64
%OPENCV_PATH%\opencv\build\x64\vc15\lib
├──C/C++
│├── routine
││├── Attach include directory
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\include
├── The linker
│├── routine
││├── Additional Library Directory
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\lib\x64
│├── Input
││├── Additional dependency
c10.lib
libprotobuf.lib
mkldnn.lib
torch.lib
torch_cpu.lib
nvinfer.lib
nvinfer_plugin.lib
nvonnxparser.lib
nvparsers.lib
cudnn.lib
cublas.lib
cudart.lib
nvrtc.lib
opencv_world3416.lib
according to TensorRT In the official developer's Guide C++ API example , The development process is as follows :
1.3.1 Construction phase
First create builder
, First instantiation ILogger
Interface to catch exceptions :
class Logger : public ILogger
{
void log(Severity severity, const char* msg) override
{
// suppress info-level messages
if (severity <= Severity::kWARNING)
std::cout << msg << std::endl;
}
} logger;
But writing according to official documents will report an error : The restrictive exception specification of overriding virtual functions is better than that of base class virtual member functions
I saw the official sample,logging.h In the writing
So this is changed to :
class Logger : public ILogger
{
//void log(Severity severity, const char* msg) override
void log(Severity severity, const char* msg) noexcept override
{
// suppress info-level messages
if (severity <= Severity::kWARNING)
std::cout << msg << std::endl;
}
} logger;
Then instantiate builder
:
IBuilder* builder = createInferBuilder(logger);
Instantiation builder
after , The first step in optimizing the model is to create a network definition :
uint32_t flag = 1U <<static_cast<uint32_t>(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH)
INetworkDefinition* network = builder->createNetworkV2(flag);
After defining the network , You can create ONNX Parser to populate the network :
IParser* parser = createParser(*network, logger);
then , Read the model file and handle any errors :
parser->parseFromFile(modelFile, ILogger::Severity::kWARNING);
for (int32_t i = 0; i < parser.getNbErrors(); ++i)
{
std::cout << parser->getError(i)->desc() << std::endl;
}
Then build engine, Create a build configuration , Appoint TensorRT How to optimize the model .
unsigned int maxBatchSize = 1;
builder->setMaxBatchSize(maxBatchSize);
IBuilderConfig* config = builder->createBuilderConfig();
This interface has many properties , You can set these properties to control TensorRT How to optimize the network . An important attribute is the maximum workspace size . Layer implementations usually require a temporary workspace , And this parameter limits the maximum size that can be used by any layer in the network . If the workspace provided is insufficient ,TensorRT The implementation of the layer may not be found .
config->setMaxWorkspaceSize(1U << 20);
Specify the configuration to build the engine , Serialization model , Namely the engine Convert to a storable format for later use . Inference time , Simply deserialize this engine It can be directly used for reasoning . Usually create a engine It takes a lot of time , You can use this serialization method to avoid every time you re create engine:
IHostMemory* serializedModel = builder->buildSerializedNetwork(*network, *config);
The serialization engine contains the necessary weight copies , Parser 、 Network definition 、 The builder configuration and builder are no longer required , Can be safely removed :
delete parser;
delete network;
delete config;
delete builder;
1.3.2 Deserialization model
Assume that you have serialized an optimization model and want to perform reasoning :
IRuntime* runtime = createInferRuntime(logger);
ICudaEngine* engine = runtime->deserializeCudaEngine(modelData, modelSize);
1.3.3 To carry out reasoning
IExecutionContext *context = engine->createExecutionContext();
int32_t inputIndex = engine->getBindingIndex(INPUT_NAME);
int32_t outputIndex = engine->getBindingIndex(OUTPUT_NAME);
void* buffers[2];
buffers[inputIndex] = inputBuffer;
buffers[outputIndex] = outputBuffer;
context->enqueueV2(buffers, stream, nullptr);
This is for asynchronous reasoning enqueueV2
, If you want synchronous reasoning , have access to excuteV2
2 Use TensorRT Deploy YOLOv5
2.1 download YOLOv5 Source code
download https://github.com/ultralytics/yolov5master
Branch .
2.2 export YOLOv5 onnx Model
open export.py
Files are generated after they are run directly yolov5s.onnx
file , Open it as shown in the figure :
2.3 stay C++ Use in
Load exported yolov5s.onnx
And reason ( The complete code of the project is in TensorRT Deploy yoloV5 Source code ):
#include <iostream>
#include "NvOnnxParser.h"
#include "NvInfer.h"
#include "opencv2/opencv.hpp"
#include <cuda_runtime_api.h>
using namespace cv;
using namespace nvinfer1;
using namespace nvonnxparser;
using namespace std;
class Logger : public ILogger
{
//void log(Severity severity, const char* msg) override
void log(Severity severity, nvinfer1::AsciiChar const* msg) noexcept override
{
// suppress info-level messages
if (severity <= Severity::kWARNING)
std::cout << msg << std::endl;
}
} gLogger;
int main() {
// Instantiation builder
IBuilder* builder = createInferBuilder(gLogger);
nvinfer1::INetworkDefinition* network = builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH));
// load onnx file
nvonnxparser::IParser* parser = nvonnxparser::createParser(*network, gLogger);
const char* onnx_filename = "yolov5s.onnx";
parser->parseFromFile(onnx_filename, static_cast<int>(Logger::Severity::kWARNING));
for (int i = 0; i < parser->getNbErrors(); ++i)
{
std::cout << parser->getError(i)->desc() << std::endl;
}
std::cout << "successfully load the onnx model" << std::endl;
// Create the engine
unsigned int maxBatchSize = 1;
builder->setMaxBatchSize(maxBatchSize);
IBuilderConfig* config = builder->createBuilderConfig();
config->setMaxWorkspaceSize(1 << 20);
ICudaEngine* engine = builder->buildEngineWithConfig(*network, *config);
// serialize
IHostMemory* serializedModel = engine->serialize();
std::ofstream serialize_output_stream("./build/yolov5_engine_output.trt", std::ios_base::out | std::ios_base::binary);;
serialize_output_stream.write((char*)serializedModel->data(), serializedModel->size());
serialize_output_stream.close();
delete parser;
delete network;
delete config;
delete builder;
// Deserialization
IRuntime* runtime = createInferRuntime(gLogger);
std::string cached_path = "./build/yolov5_engine_output.trt";
std::ifstream trtModelFile(cached_path, std::ios_base::in | std::ios_base::binary);
trtModelFile.seekg(0, ios::end);
int size = trtModelFile.tellg();
trtModelFile.seekg(0, ios::beg);
char* buff = new char[size];
trtModelFile.read(buff, size);
trtModelFile.close();
ICudaEngine* re_engine = runtime->deserializeCudaEngine((void*)buff, size, NULL);
delete buff;
// establish context
// establish context
IExecutionContext* context = re_engine->createExecutionContext();
// The image processing
string img_path = "bus.jpg";
cv::Mat img = imread(img_path);
int h = img.rows;// Get the length of the picture
int w = img.cols;// Get the width of the picture
cvtColor(img, img, COLOR_BGR2YCrCb);// Convert the picture to YCrCb
vector<Mat> over;
split(img, over);
over[0].convertTo(over[0], CV_32F, 1 / 255.0); // take Y Channel normalization
// establish buffers Point to input / output stream
void* buffers[2];
int inputIndex;
int outputIndex;
for (int bi = 0; bi < re_engine->getNbBindings(); bi++)
{
if (re_engine->bindingIsInput(bi) == true)
inputIndex = bi;
else
outputIndex = bi;
}
// Distribute buffers Space
cudaMalloc(&buffers[inputIndex], h * w * sizeof(float));
cudaMalloc(&buffers[outputIndex], h * 2 * w * 2 * sizeof(float));// The size of the superscript
// establish cuda flow
cudaStream_t stream;
cudaStreamCreate(&stream);
// Copy picture data to GPU
cudaMemcpyAsync(&buffers[inputIndex], &over[0], h * w * sizeof(float), cudaMemcpyHostToDevice, stream);
// To carry out reasoning
context->enqueue(1, buffers, stream, nullptr);
// take 3 The size of each channel after being converted into a super division
for (int i = 0; i < 3; i++)
resize(over[i], over[i], Size(w * 2, h * 2), 0, 0, INTER_CUBIC);
// take GPU Copy data back to CPU
cudaMemcpy(&over[0], &buffers[outputIndex], h * 2 * w * 2 * sizeof(float), cudaMemcpyDeviceToHost);
// Anti normalization , Convert data type
over[0].convertTo(over[0], CV_8U, 1 * 255);
// Merge 3 passageway , write file
merge(over, img);
// take YCrCb Turn it back RGB(opencv Medium is BGR), write file ;
cvtColor(img, img, COLOR_YCrCb2BGR);
imwrite("bus_infer.jpg", img);
// Release resources
cudaStreamDestroy(stream);
context->destroy();
re_engine->destroy();
runtime->destroy();
cudaFree(buffers[inputIndex]);
return 0;
}
Running results :
If you make a mistake C4996:
‘XXXX’: This function or variable may be unsafe. Consider using localtime_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
Add... To the error file :
#pragma warning(disable:4996)
If you make a mistake C2664:
“HMODULE LoadLibraryW(LPCWSTR)”: Unable to set parameter 1 from “const _Elem *” Convert to “LPCWSTR”
Right click solution -> attribute -> Configuration properties -> senior -> Character set
take " Use Unicode Character set " Change to " Using multibyte character sets "
TensorRT Report errors :
TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.5.1
By checking the blog https://blog.csdn.net/qq_41151162/article/details/118735414 Found to be cuda Version of the problem .
TensorRT Report errors :
Parameter check failed at: ...
The reason for this is that the pre training model does not match the network structure , Regenerate the corresponding pre training model .
边栏推荐
- 以交互方式安装ESXi 6.0
- Harbor image warehouse construction
- comsol--三维图形随便画----回转
- Error assembling WAR: webxml attribute is required (or pre-existing WEB-INF/web.xml if executing in
- 【主流Nivida显卡深度学习/强化学习/AI算力汇总】
- Solve readobjectstart: expect {or N, but found n, error found in 1 byte of
- go语言学习笔记-初识Go语言
- 【无标题】
- Dynamic SQL of ibatis
- Proof of the thinking of Hanoi Tower problem
猜你喜欢
[crawler] bugs encountered by wasm
yolov5目标检测神经网络——损失函数计算原理
网络五连鞭
iTOP-3568开发板NPU使用安装RKNN Toolkit Lite2
COMSOL -- establishment of 3D graphics
Go language learning notes - analyze the first program
【上采样方式-OpenCV插值】
OneForAll安装使用
【 YOLOv3中Loss部分计算】
COMSOL -- three-dimensional graphics random drawing -- rotation
随机推荐
[office] eight usages of if function in Excel
【SingleShotMultiBoxDetector(SSD,单步多框目标检测)】
Solve readobjectstart: expect {or N, but found n, error found in 1 byte of
解决grpc连接问题Dial成功状态为TransientFailure
1.php的laravel创建项目
11.(地图数据篇)OSM数据如何下载使用
go语言学习笔记-分析第一个程序
comsol--三维图形随便画----回转
Empêcher le navigateur de reculer
【TFLite, ONNX, CoreML, TensorRT Export】
How to get a token from tokenstream based on Lucene 3.5.0
COMSOL--建立几何模型---二维图形的建立
15 methods in "understand series after reading" teach you to play with strings
IPv6与IPv4的区别 网信办等三部推进IPv6规模部署
redis 集群模式原理
解决readObjectStart: expect { or n, but found N, error found in #1 byte of ...||..., bigger context ..
跨境电商是啥意思?主要是做什么的?业务模式有哪些?
Mysql统计技巧:ON DUPLICATE KEY UPDATE用法
CDGA|数据治理不得不坚持的六个原则
Redis如何实现多可用区?