Tensorflow serving high performance machine learning model service system

framework
TensorFlow Serving It's a flexible one 、 High performance Machine learning model service system , Designed for the production environment . TensorFlow Serving New algorithms and experiments can be easily deployed , Keep the same server architecture and API. TensorFlow Serving Provide with TensorFlow Out of the box integration of models , But it can be easily extended to serve other types of models .

Key concepts
To understand TensorFlow Serving The architecture of , You need to understand the following key concepts ：

Serviceable Servables
Servables yes TensorFlow Serving The core abstraction in . Servables yes The client is used to perform calculations （ for example , Find or reason ） Of Underlying object .

Servable The size and granularity of are flexible . Single Servable May include from Lookup table Single slice to Single model Until then Inference model tuple Any content of . Servables It can be Any type and interface , To achieve flexibility and future improvements , for example ：

Streaming results
experimental API
Asynchronous operation mode
Servables Don't manage your life cycle .

Typical services include the following ：

One TensorFlow SavedModelBundle (tensorflow::Session)
Lookup table for embedding or vocabulary lookup

Serviceable version
TensorFlow Serving One or more versions of can be processed during the life cycle of a single server instance servable. This allows new algorithm configurations to be loaded over time 、 Weights and other data . Version can load multiple versions at the same time servable, Support the gradual introduction and testing . When serving , The client can request the latest version of a specific model or a specific version ID.

Serviceable flow
Serviceable flow Is a serviceable version sequence , Sort by version number .

model
TensorFlow Serving Represent the model as one or more serviceable objects . Machine learning models may include one or more algorithms （ Including the weight of learning ） And finding or embedding tables .

You can express the composite model as any of the following ：

Multiple independent serviceable objects
A single composite can serve （single composite servable）

A serviceable object may also correspond to a part of the model . for example , A large lookup table can span multiple TensorFlow Serving The examples are divided .

Loaders
Loader management servable Life cycle of . Loader API Support independent of the specific learning algorithm involved 、 Common infrastructure for data or product use cases . say concretely ,Loaders Standardized for loading and unloading servable Of API.