[micro service series] protocol buffer dynamic analysis

In recent work grpc. The previous work was based on thrift The microservice framework of , Yes grpc Not very familiar with , Only know grpc It's based on http2 and protobuf Of rpc frame . But the methods of use are similar , be based on idl Generate the corresponding file , On the server side, you can achieve specific service And provide services to the outside world , For the client, you need to introduce client Package initiation rpc call .

Here's the problem , To invoke downstream services, you need to introduce downstream services client, If downstream Services idl If something changes, we need to update it in the code client package , Recompile deployment release . In normal business development, there is no problem with this mode , The rhythm of release and deployment follows the rhythm of business iteration . But in some scenarios, there may be big problems , For example, a gateway service , Many services are registered on the gateway , Gateway through rpc To invoke business services . In this case, if every business change idl The gateway needs to be deployed with the release, which is obviously a big problem .

The above problems were investigated , It is found that dynamic parsing can be used idl Document the way to solve the problem . have access to github.com/jhump/protoreflect To achieve pb and grpc Reflection of . Originally mainly introduced pb Partial content .

Code example

Establish a simple proto The documents are as follows , It defines a simple structure HelloReq and HelloResp.

syntax = "proto3";
package test;

option go_package = "data/";

message HelloReq {
    
  int64 ID = 1;
  string Name = 2;
}

message HelloResp {
    
  int64 Code = 1;
  string Message = 2;
}

According to the general practice , At this time, you should execute protoc --go_out=. ./test.proto Command to generate test.pb.go, Then use the structure to do the corresponding operation .

Let's use the above mentioned protoreflect Package for runtime parsing , The code is as follows .

package main

import (
   "encoding/json"
   "github.com/jhump/protoreflect/desc/protoparse"
   "github.com/jhump/protoreflect/dynamic"
   "grpc_practice/data"
   "log"
)

func main() {
    
   //  establish parser object 
   p := protoparse.Parser{
    }
   //  Use path Some columns of file description objects are obtained by parsing , There is only one file description object 
   fileDescs, err := p.ParseFiles("./test.proto")
   if err != nil {
    
      log.Printf("parse proto file failed, err = %s", err.Error())
      return
   }
   
   //  Get the message contempt object from the file description object according to the message name 
   helloReqDesc := fileDescs[0].FindMessage("test.HelloReq")
   if helloReqDesc == nil {
    
      log.Printf("no message matched")
      return
   }
   
   //  Generate dynamic messages according to the message description object 
   dMsg := dynamic.NewMessage(helloReqDesc)
   
   //  Use this dynamic object to start from GetHelloReq Simulated json Object decoding 
   _ = dMsg.UnmarshalJSON(GetHelloReq())
   log.Printf("req ID = %d, req Name = %s", dMsg.GetFieldByNumber(1), dMsg.GetFieldByNumber(2))

}

func GetHelloReq() []byte {
    
   req := data.HelloReq{
    }
   req.ID = 1
   req.Name = "xiaoshuai"
   d, _ := json.Marshal(&req)
   return d
}

Principle analysis

above demo It's a little bit easier , Looking at the code above, it is not difficult to find that there are two core objects , One is descriptor The descriptor , It corresponds to proto Objects at different levels in the file ; Two is dynamicMessage Dynamic news , It replaces pb.go Message body in file .

In a nutshell ,descriptor It's parsing proto file , And organize and hold in the form of data structure proto Information in the file .dynamicMessage hold MessageDescriptor object , It is equivalent to knowing all the information of the structure .

Descriptor

The relationships of descriptors at different levels are as follows , There is nothing to go into detail .

dynamicMessage

// Message is a dynamic protobuf message. Instead of a generated struct,
// like most protobuf messages, this is a map of field number to values and
// a message descriptor, which is used to validate the field values and
// also to de-serialize messages (from the standard binary format, as well
// as from the text format and from JSON).
type Message struct {
    
   md            *desc.MessageDescriptor
   er            *ExtensionRegistry
   mf            *MessageFactory
   extraFields   map[int32]*desc.FieldDescriptor
   values        map[int32]interface{
    }
   unknownFields map[int32][]UnknownField
}

dynamic message The source code is as follows , The main object to see is message descriptor, hold md object dynamic message You have all the information in one message , For example, field name 、 Serial number 、 Type, etc . Then its value is stored in values This map Field , Its value is expressed in interface{} Deposit , So there will be a lot of reflection when using , I will add some later benchmark, Observe its performance .