Openvino integrates tensorflow to accelerate reasoning

author ： Arindam, Yamini, Mustafa, Ritesh, Priya, Chandrakant, Surya, Amar, Sesh

compile ： Li Yiwei

 

technology spread Adoption is usually triggered by a leap in user experience . for example ,iPhone Promote smart phones and “ The app store ” The rapid spread of . lately ,TensorFlow The ease of use of started the massive growth of artificial intelligence , It touches almost every aspect of our daily life today .

OpenVINO tool kit Redefined... On devices using Intel technology AI Reasoning Ability , And got it. vast Developers use . Now , Thousands of developers use OpenVINO Toolkits to speed up almost everything imaginable To the AI Reasoning Application scenarios , From human visual simulation , Automatic speech recognition , natural language processing , Recommendation system, etc . The toolkit is based on the latest generation of artificial neural networks , Including convolutional neural networks （CNN）、 Network based on circulation and attention , Extend computer vision and non vision workloads , Can span Intel hardware （ Intel CPU、 Intel integrated graphics card 、 Intel neural computing stick 2 And Intel visual accelerator design with Intel Movidius VPU） To maximize performance . It is achieved through high-performance deployment from the edge to the cloud 、AI And deep learning reasoning to accelerate application .

We are honored to be able to cooperate with our customers / developer cooperation , Contribute to their success . through Constantly listen and innovate , To meet their changing needs , At the same time, it is committed to providing world-class user experience . therefore , Based on customer feedback , stay OpenVINO Based on the success of the toolkit , We will OpenVINO And TensorFlow* Integrate .

There is something involved AI All of you in edge operation are interested in OpenVINO Should have a basic understanding ： Different frameworks （ Such as TensorFlow、PyTorch etc. ） The model file after training is passing OpenVINO After conversion, reasoning acceleration can be performed on different edge computing devices .

If I tell you , Now it can be directly in... Without model transformation TensorFlow Complete when reasoning OpenVINO What about acceleration ？

Yes, you are right ！ Intel stay 2021 Launched in the second half of the year OpenVINO integration with TensorFlow（ hereinafter referred to as OVTF） Can be realized in TensorFlow Intermediary connection OpenVINO Executive reasoning accelerates .

OpenVINO x TensorFlow Happiness comes too suddenly

Yes TensorFlow Developer benefits ： There is no need to convert , just Add 2 Lines of code can speed up its TensorFlow Model reasoning Speed .

OpenVINO And TensorFlow* Integration of Provides enhance TensorFlow Required for compatibility OpenVINO tool kit Inline optimization and run time. It is designed for use OpenVINO Toolkit developers - Help improve the performance of their reasoning applications - With minimal code changes . It can speed up all kinds of Intel chips many AI Model The reasoning of , for example ：

• Intel CPU
• Intel integrated graphics card
• Intel Movidius Visual processing unit – also call VPU
• use 8 Intel Movidius MyriadX VPC Intel visual accelerator design - be called VAD-M or HDDL

Developers who take advantage of this integration can expect the following benefits ：

• Performance acceleration  - And Original TensorFlow comparison ( Depends on the underlying hardware configuration ) 
• precision  – Maintain almost the same accuracy as the original model .
• simplicity  – Continue to use TensorFlow API Reasoning . No need to refactor code . Just import , Enable and set up the device .
• Robustness,  – Designed to support a variety of operating systems /Python Various in the environment TensorFlow Models and operators .
• seamless Speed up  - Inline model transformation – No model transformation required .
• Lightweight footprint  – The incremental memory and disk footprint required is minimal .
• Support a wide range of Intel product  – CPU、iGPU、VPU （Myriad-X）.

Be careful ： For best performance 、 efficiency 、 Tool customization and hardware control , We suggest using this machine OpenVINO API And Its run time function .

how Realization ？

Developers can add the following two lines of code to their Python Code or Jupyter Notebooks China has greatly accelerated TensorFlow Model reasoning .

1. import openvino_tensorflow
2. openvino_tensorflow.set_backend（'<backend_name>'）

Back end of support <backend_name> Include “CPU”,“GPU”,“MYRIAD” and “VAD-M”. See chart 1.

The first line above is not strictly an instruction , Only OpenVINO Integrate TensorFlow Kit . And the second line calls openvino_tensorflow Set the instructions of the back-end computing hardware , The parameters brought in can be set as CPU（Intel processor ）、GPU（Intel Integrated graphics card in processor ）、MYRIAD（AI To speed up the chip VPU） etc. .  In this way, it is finished TensorFlow Reasoning accelerates . 

Sample code ：

Here are OpenVINO And TensorFlow* Examples of integration ：

 

How it is a Of ？

And its particularity can be seen from the architecture diagram in the original TenorFlow And OpenVINO toolkit Between more Operator Capability Manager （OCM）、Graph Partitioner、 TensorFlow Importer And Backend Manager, Let the above two be naturally combined . To put it simply, we will interpret all operations of neural network when executing reasoning , Whether it can penetrate OpenVINO Accelerate , And let it correspond to OpenVINO The corresponding operator of , Finally, it is allocated to the specified back-end hardware for calculation , On the contrary, if the operation cannot be accelerated, let it return to  TensorFlow In dealing with .

The details of each function can be found in github repo And Documentation Explore in depth . It doesn't matter if you don't understand these technical details , Reference resources Model supporting documents You can see that each TensorFlow Model （ contain TF-Slim Classification、Object Detecion、 TF- Hub And many other sources ） Degree of support , Or follow our next steps to experience ！  

 

chart 2：OpenVINO integration with TensorFlow Architecture Graph （openvino_tensorflow/ARCHITECTURE.md at master · openvinotoolkit/openvino_tensorflow · GitHub）

OpenVINO And TensorFlow* Integration of By way of TensorFlow Graph is effectively divided into multiple subgraphs to provide accelerated TensorFlow performance , These subgraphs are then scheduled to TensorFlow Run or OpenVINO Run time for optimal accelerated reasoning . Final combination Out The final reasoning result .

 

chart 3： End to end overview of workflow

Deploy at the edge and in the cloud

OpenVINO And TensorFlow The integration of is applicable to various environments from cloud to edge , As long as the underlying hardware is an Intel platform . Applicable to the following cloud platforms ：

• Edge oriented Intel DevCloud
• AWS Deep Learning AMI Ubuntu 18 and Ubuntu 20 on EC2 C5 example , Optimize for reasoning
• Azure ML
• Google Labs

Support any AI Edge devices .

The sample in gitrepo  An example of / The catalog provides .

This is different from using native OpenVINO What is the difference between toolkits ：

OpenVINO And TensorFlow* Integration of TensorFlow Developers can accelerate their... In a very quick and easy way TensorFlow Model reasoning - just 2 Line code .OpenVINO The model optimizer can speed up reasoning performance , And rich integration developer tools and advanced functions , But as mentioned before , For optimal performance , efficiency , Tool customization and hardware control , We recommend using this machine OpenVINO API And run time function .

Case study

following The customer is putting OpenVINO Integrated for TensorFlow For various use cases . Here are some examples

1. Extreme Vision（ Polar Perspective ）： Polar Perspective   Of CV MART Special purpose AI cloud   It can help hundreds of thousands of developers to provide rich services 、 Model and framework catalog , Thus, on various Intel platforms （ Such as CPU and iGPU） Further optimize its AI The workload . And AI frame （ Such as OpenVINO And TensorFlow* Integration of ） A properly integrated, easy-to-use developer toolkit accelerates the model , So as to provide the best of both worlds - Improve reasoning speed and reuse created with minimal change AI The ability to reason about code .Extreme Vision The team is testing OpenVINO And TensorFlow* Integration of , The goal is to Extreme Vision On the platform TensorFlow Developers provide support .
2. The genome analysis toolkit developed by the Bode Institute （GATK） It is one of the most widely used variant call open source toolkits in the world .Terra Is a safer , Scalable open source platform , For biomedical researchers to access data , Run analysis tools and collaboration . The cloud based platform was developed by MIT bode Institute and Harvard University , Microsoft and Verily Jointly develop .Terra The platform includes GATK Tools and piping , For the research community to run its analysis .CNNScoreVariants yes GATK One of the deep learning tools included in , It applies convolution neural network to filter annotated variants . In an article Blog in ,Broad Institute Shows how to use OpenVINO And TensorFlow* Integration to further accelerate CNNScoreVariants Reasoning performance .

Conclusion

Now? , You have learned its advantages 、 working principle 、 Deployment environment and OpenVINO And TensorFlow Integration and use of native OpenVINO API The difference , I believe you have I can't wait to try it myself OpenVINO And TensorFlow Integrate , And experience on Intel platform AI The reasoning performance of the model is improved . As usual , We would love to hear your feedback on this integration , please adopt [email protected]  Contact us Or in the gitrepo Ask questions in . thank you ！

resources

The following resources can help you learn more ：

• Github repository
• Solution brief
• Dynamic feedback cartoon
• frequently asked questions
• Intel DevCloud for the Edge Example above
• Google Colab Notebook example