Tis tutorial 03 export

brief introduction

as everyone knows , The model deployment framework deploys a deep learning model , So we hope TIS Only the reasoning of the model is processed , Therefore, it is necessary to provide it with an appropriate model . And for PyTorch In terms of the model of , There are two main deployment level models , One is universal ONNX Model In order to support many frameworks , The other is PyTorch Self contained TorchScript Model . This paper and This series All other articles in have adopted TorchScript Model for deployment .

TorchScript

First of all, let's introduce TorchScript What is it ？

TorchScript The main application scenario is to put Python/PyTorch Convert the code to an equivalent C++ Code to improve the operational efficiency of the online production environment deployment of the deep learning model .Python The code will be compiled into TorchScript A format that the compiler can understand （ScriptModule）,C++ The production environment can load files in this format and use the built-in JIT To execute the corresponding code .

Landing on the model level , It's really just the PyTorch Model from Python process （ It generally refers to the development environment , For model design and training ） Save in and load into no Python rely on （ Generally refers to the deployment environment , For online deployment of models ） In the process of . In the process , pure Python The model is transformed to be independent of Python Of TorchScript Program , Such as C++ Environment , It makes the deployment of production environment possible , In this environment ,Python The program may not be applicable due to performance and multithreading .

TorchScript from PyTorch1.x Start to improve , yes PyTorch and TensorFlow The competitive industrial deployment is a significant contribution to the market .

Model export

actually ,TorchScript Support is Python To C++ This kind of efficient locale transformation , One of the most typical is the use of C++ Compiling LibTorch library . however ,TorchScript Become an ecological , There is also a lower level of grammar to learn , In the following, I will just introduce the common strategies for model export , as for LibTorch The details of using Triton Inference Server There is no need to pay too much attention , We just need to focus on the model in TIS The reasoning operation structure in the .

To preserve TorchScript Model , You need to build this model first and track it once , You can serialize the model to a local file .

import torch

x = torch.randn(1, 3, 640, 640)
model = create_model('yolov5s')
ts_model = torch.jit.trace(model, x, strict=False)
ts_model.save('yolov5.ts.pt')

The top one is to use TorchScript take PyTorch Example code of model export , I'm here to yolov5s export . As for the details of model loading , By TIS To be responsible for , in fact TIS Support includes TensorRT、TorchScript（LibTorch）、TensorFlow And a variety of back ends , We just need to improve the input and output configuration details .

summary

This article simply introduces TorchScript Simple operation of model export , If you want to know more, please study it in detail Official documents .