Go microservice (II) - detailed introduction to protobuf

The length may be longer , You can collect it first , Convenient for follow-up viewing .

Protobuf introduction

1. Why choose Protobuf

Protobuf yes Protocol Buffers For short , It is Google company-developed ⼀ Data description ⾔, And in 2008 Opening up in Source .Protobuf When it was just open source, its positioning was similar to XML、JSON Data description ⾔, By attaching ⼯ have ⽣ Code and implement will knot The function of structuring data serialization . But we are more concerned about Protobuf As a connection ⼝ Normative descriptors ⾔, It can be used as a safe design Cross language ⾔PRC Pick up ⼝ The basis of ⼯ have .

1. What is? Protobuf?

   • Protobuf yes Protocol Buffers For short .
   • Google 2008 An open source data description language .
   • Description language of interface specification
   • Attached tools generate code And implement Integrate structured data The function of serialization .
   • Design safe cross language RPC Basic tools of interface

2. Protobuf The advantages of ：

   • Encoding and decoding efficiency
   • High compression ratio
   • Multilingual support

   

2. Go Language Protobuf Development environment construction

Protobuf compiler ：

Protobuf Our compiler is called ：protoc（protobuf compiler）

Golang Install and use Protobuf：

1. download Protobuf The compiler

1. download protobuf The compiler , Click to go to ：

   

2. decompression , Decompression location depends on yourself , After decompression /bin/protoc.exe Add to environment variables

   

   

3. test ：

   cmd Input protoc --version

   

2. install go protocol buffers Plug in for protoc-gen-go

Protobuf nucleus ⼼ Of ⼯ With set yes C++ language ⾔ Developed , In office ⽅ Of protoc Compiler does not ⽀ a Go language ⾔. To be based on .proto⽂ Pieces of ⽣ Become corresponding Go Code , You need to install the plug-in .

cmd Input ：

go install google.golang.org/protobuf/cmd/[email protected]

This plug-in will be automatically downloaded to your go path Of bin Under the table of contents .（ The compiler will automatically find this plug-in here ）

3. Actual test ：

establish myProto.proto file

syntax = "proto3";
package main;

option go_package = "./";

message String {
  string name = 1;
  int64  age  = 2;
}

Be careful ： If it's up there protoc --version success , The prompt here is unrecognized protoc If you order , Restart it goland Just fine .

3. Protobuf Basic usage

First take a look at the following proto file , The one behind us proto The basic usage is based on this proto Explain

syntax = "proto3";
package pkgName;

option go_package = "./";

message mmData {
	optional int32 num = 1;
	optional int32 def_num = 2 [default=10];
	required string str = 3;
	repeated string rep_str = 4;
}

0. Use protoc Generate go file

（ But after reading ·3. Protobuf The basic usage of ·, Look at this part ）

---my_project
   |---06-protocol_buffers
       |---pbrpc
            |---service
                   |---service.proto
           

pbrpc/service/service.proto:

syntax = "proto3";
package hello;

// go module = MicroServiceStudy01
option go_package = "MicroServiceStudy01/06-protocol_buffers/pbrpc/service";

message Request{
  string value = 1;
}

Use protoc：

cd 06-protocol_buffers/pbrpc/service

protoc -I . --go_out=. hello.proto

If it's done this way , His result is in you go_out Catalog （ Here is the current directory ） Deposit , And according to your definition go_package The name of , In you go_out Create a directory structure under the directory ：

If you don't want him to help you generate a go_package Directory structure of , Then you need to specify a prefix ：

protoc -I . --go_out=. --go_opt=module="MicroServiceStudy01/06-protocol_buffers/pbrpc/service" hello.proto

In this way, there is no basis go_pacakage Generate directory structure , It is stored directly in go_out Catalog ：

I don't understand , If the purpose is to store in the current directory , Why not take go_package="./", If you want to store it in a subdirectory under the current directory , Just go_package=“./subpkg “, The above method , I can't understand , For the time being, let's learn how to use parameters , If you know something, you can leave a message .

• --go_out=./：proto-gen-go The storage directory of plug-in compilation products , Here is the current directory , Pay attention to the generation of Of .pb.go The final location of the file is yours --go_out=? Location +go_package=? Location , The latter is in --go_out After the position , Further specify the generated .pb.go File storage path .

• -I ../：--proto_path=PATH Abbreviation

  Indicates the directory path of the imported file , Here you are pit .（ If you don't understand here , See below import Will understand the ）

  -I Parameters are simply , If there are more proto There are interdependencies between files , Generate a proto When you file , need import Others proto file , It's time to use -I To specify the search directory . If not specified -I Parameters , Search in the current directory .（ The example command here is ）

  Every -I Parameters are introduced into a directory ,proto Several external proto Theoretically speaking, how many documents are needed -I（ The same directory can be imported at one time ）, Plus the to be compiled proto We also need to introduce , So two are used here -I To import directory files .

• --go_opt=moudle=....：protoc—gen-go The plug-in opt Parameters , use go moudle Pattern .

• hello.proto：proto File path .

1. syntax

Indicate the use of proto3 grammar ; If you don't specify this , The compiler will use proto2 grammar ; The specified syntax line must be the first line of the file that is not empty or comment

2. package （Package）

proto The file uses keywords package Specify the current package name , Similar to modules , Definition proto Package name , It can be for .proto Add an optional... To the file package Declarator as generation language namespace, be used for Prevent naming conflicts between different message types .

3. Options （Options）

In defining .proto The file can be marked with a series of options.Options Does not change the meaning of the entire document statement , But it can affect the treatment in a specific environment . Complete options are available in google/protobuf/descriptor.proto find .

Before message definition , Can pass option To configure , frequently-used option：

option go_package = "path;name";

• path Represents generated go File storage address , The directory will be automatically generated .
• name Represents generated go Package name to which the file belongs

4. Message type （message）

Protobuf A message type is defined through the keyword message Field specified , This keyword can be understood as Go Linguistic stuct keyword , use protobuf Compiler will proto Translate it into Go After code , Every message Will generate a name corresponding to stuct Structure .

As above , Will generate a name of mmData The structure of the body .

Variable （ Field ） The definition format of is ：

[ Modifier （ Optional ）][ data type ][ Variable name （ Field name ）] = [ Unique identifier ] ;

The unique identifier is used to identify the field , The same message The identifiers of fields in cannot be the same .

1. Rules of the field （ Field modifier ）

message There are three field rules in .

• required: Field attribute is required . If we do not set up , Will cause encoding and decoding exceptions , Causes the message to be discarded .

• optional : The field attribute is an optional field . The sender can optionally set as needed ;

  about optional Fields for properties , Can pass default Keyword sets the default value for the field , That is, when the sender does not set this field , The default value... Will be used .

  If no default value is set for the field , The field will be given a specific default value according to the specific type .

  about bool type , The default value is false; about string type , The default value is an empty string ; For numeric types , The default value is 0; For enumeration types , The default value is the first value in the enumeration type .

• repeated : The field attribute is a repeatable field , This field can contain [0,n] Elements , The order of elements in the field is preserved . Be similar to go The section of .

Be careful :

1. stay proto3 In the version , Removed from the field rule required, And put optional The field is renamed singular. All fields without specified field rules default to optional, For why deleted require The rules , Reference resources : Why? proto3 Removed required and optional？
2. stay proto2 In the version , Under default configuration , One optional The setting that is not set or displayed is the default , When serializing binary formats , This field will be removed , After deserialization , It is impossible to distinguish whether the default value is set or not , Even using hasXXX() Method , For fields with default values set , Also return false. resolvent ： distinguish Protobuf Missing and default values in

2. Identification number （ Unique identifier ）

In the definition of the message body , Each field must have a Unique identification number .

These identification numbers are used to identify each field in the binary format of the message , Once used, it cannot be changed , Otherwise, the encoding and decoding of the original message will be abnormal .

The identification number is [0,2^29 - 1] An integer in the range , among **[19000,19999) The identification number between protobuf The implementation of the protocol is reserved **, So be careful not to use the identification number in this range in the close-up , If it is used for compilation, it will also alarm ：

Field numbers 19000 through 19999 are reserved for the protocol buffer library implementation.

Be careful ：

[1,15] The identification number in takes up one byte when encoding ,[16,2047] The identifier within takes two bytes , So try to allocate for frequently used fields [1,15] Identification number in , in addition Reserve a part for the fields that may be used frequently in the future .

3. data type

3.1 Basic data type

About the default value of the field ：
string Variable of type , The default is an empty string
bytes Variable of type , The default value is null byte Array
bool Variable of type , The default value is false
Variable of numeric type , The default value is 0
Enumerating variables of type , The default value is the first enumeration value , And the numerical value of the first enumerated value must be 0

3.2 Enumeration type

Field types in addition to the above basic field types , It can also be an enumeration type .

syntax = "proto3";
package main;

option go_package = "./";

//  Define enumeration types 
enum DayName {
  Sun  = 0;
  Mon  = 1;
  Tues = 2;
  Wed  = 3;
  Thur = 4;
  Fri  = 5;
  Sat  = 6;
}

message workDay {
  //  Message types use enumeration types 
  optional DayName day = 1;
}

protoc --go_out=./ hello.proto Generated go In the document, it corresponds to const：

//  Define enumeration types 
type DayName int32

const (
	DayName_Sun  DayName = 0
	DayName_Mon  DayName = 1
	DayName_Tues DayName = 2
	DayName_Wed  DayName = 3
	DayName_Thur DayName = 4
	DayName_Fri  DayName = 5
	DayName_Sat  DayName = 6
)
....
type WorkDay struct {
    
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	//  Message types use enumeration types 
	Day *DayName `protobuf:"varint,1,opt,name=day,proto3,enum=main.DayName,oneof" json:"day,omitempty"`
}

The value of the enumeration constant must be 32 Bit integer range , because enum Values are stored in an encoded way , Storing negative numbers is not efficient , Therefore, it is not recommended to enum Use negative numbers in .

Enumeration types can be defined in message Inside , It can also be defined in message Outside , If defined in message Inside , other message To use, you need to pass messageType.enumType To quote .

By default , Field values in enumeration types cannot be repeated , But by right enum add to option allow_alias = true; To achieve the purpose of aliasing the same enumeration value , If you don't add allow_alise And there are repeated enumeration values, and errors will be reported when compiling .

syntax = "proto3";
package pkgName;

option go_package = "./";

//  Define enumeration types 
enum DayName {
  //  If you do not add this option, Will report a mistake :
  // "pkgName.Test" uses the same enum value as "pkgName.Sat".
  // If this is intended, set 'option allow_alias = true;' to the enum definition.
  option allow_alias = true;
  Sun                = 0;
  Mon                = 1;
  Tues               = 2;
  Wed                = 3;
  Thur               = 4;
  Fri                = 5;
  Sat                = 6;
  Test               = 6;  // Test And Sat The field value has the same name 
}

3.3 map data type

In addition to the above types ,message And support map<Type,Type> type .

syntax = "proto3";
package pkgName;

option go_package = "./";

message TData {
  map<int32, string> data = 1;
}

In the generated go File corresponding to the map type ：

type TData struct {
    
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Data map[int32]string `protobuf:"bytes,1,rep,name=data,proto3" json:"data,omitempty" protobuf_key:"varint,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
}

Be careful :

1. protobuf Medium map In essence, it is disordered
2. proto in map Type cannot be used optional/required/repeated Any type of decoration .

3.4 message type

protobuf Allow other message types to be used as field types .

As below userData One in workDay Data of type ：

syntax = "proto3";
package pkgName;

option go_package = "./";
message workDay {
        int day = 1;
}

message userData {
        workDay userDays = 1;
}

3.5 Nested message types

message You can nest... Infinitely

syntax = "proto3";
package pkgName;
option go_package = "./";

message OuterData1 {
  //  Nested message definitions 
  message TData {
    int32 a = 1;
  }

  //  Reference nested messages 
  TData            data1 = 1;
  OuterData2.TData data2 = 2;
}

message OuterData2 {
  //  Nested message definitions 
  message TData {
    int32 a = 1;
  }
}

In the generated hello.pb.go In the corresponding ：

type OuterData1 struct {
    
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	//  Reference nested messages 
	Data1 *OuterData1_TData `protobuf:"bytes,1,opt,name=data1,proto3" json:"data1,omitempty"`
	Data2 *OuterData2_TData `protobuf:"bytes,2,opt,name=data2,proto3" json:"data2,omitempty"`
}

....

type OuterData2 struct {
    
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields
}

3.6 Any Field ( Do not understand )

syntax = "proto3";

import "google/protobuf/any.proto";

package pkgName;

option go_package = "./";

message ErrorStatus {
  string                       message = 1;
  repeated google.protobuf.Any details = 21;
}

3.7 oneof Field

If your message Contains many optional fields , At most one field can be set at the same time , You can use oneof The function enforces this behavior and saves memory .

Oneof All fields in shared memory , At most one field can be set at the same time . Set up oneof Any member of will automatically clear all other members . You can use special case() or WhichOneof() Methods to check oneof Which value is currently in the field （ If there is ） Set up , The specific method depends on the language you choose .

Use cases ：

To be in .proto In the definition of oneof, Please use oneof keyword , Follow your oneof name , In this case test_oneof：

syntax = "proto3";

import "google/protobuf/any.proto";

package pkgName;

option go_package = "./";

message SampleMessage {
  oneof test_oneof {
    string name      = 1;
    string nike_name = 2;
  }
}

then , take oneof Field added to test_oneof The definition of .

You can test_oneof Add fields of any type , But you can't use required,optional or repeated keyword . If required oneof Add duplicate fields , You can use message.

In the generated code ,oneof Fields and general optional Methods have the same getter and setter. You can also use special methods to check oneof The value in （ If there is ）.

In the generated hello.pb.go In Chinese, it means ：

type SampleMessage struct {
    
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	// Types that are assignable to TestOneof:
	// *SampleMessage_Name
	// *SampleMessage_NikeName
	TestOneof isSampleMessage_TestOneof `protobuf_oneof:"test_oneof"`
}
....
type SampleMessage_Name struct {
    
	Name string `protobuf:"bytes,1,opt,name=name,proto3,oneof"`
}

type SampleMessage_NikeName struct {
    
	NikeName string `protobuf:"bytes,2,opt,name=nike_name,json=nikeName,proto3,oneof"`
}
....

5. Defining services （service）

If you want to message The type and RPC（ Remote procedure call ） Use the system together , Can be in .proto The document defines RPC Service interface ,protocol buffer The compiler will generate service interfaces and stub（ pile ）.

therefore , for example , If you want to define a RPC service , It contains a basis SearchRequest return SearchResponse Methods , Can be in .proto It's defined in the file , As shown below ：

syntax = "proto3";
package pkgName;

option go_package = "./";
message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 result_per_page = 3;
}

message SearchResponse {
  string result = 1;
}

service SearchService {
  rpc Search (SearchRequest) returns (SearchResponse);
}

And ProtoBuf Used directly with RPC System is gRPC : One Google Development platform independent, language independent, open source RPC System .gRPC and ProtoBuf Can cooperate perfectly , You can use special ProtoBuf Compile plug-ins directly from .proto File generation related RPC Code .

6. import Import other proto file

import

We can go through import Import other proto file , And use the proto Message types defined in the file .

---my_project
   |---protocol
       |---aaa
       |   |---aaa.proto
       |---bbb
           |---bbb.proto

aaa/aaa.proto

syntax = "proto3";
package aaa;
option go_package = "./";

message Something {
  string msg = 1;
}

bbb/bbb.proto

syntax = "proto3";
package bbb;
option go_package = "./";
import "aaa/aaa.proto";

message Something2 {
  aaa.Something something = 1;
}

Although it will be red, don't worry , Generate pb.go When , Let's say it's in my_project/protocol/bbb Under the table of contents , execute ：

protoc -I ../ -I ./ --go_out=./ bbb.proto
# -I ../ :  Look for the imported proto file 
# -I ./ :  Find the file to be compiled in this layer proto file （ Order doesn't matter ）

protoc There is a parameter -I, Indicates the directory path of the imported file , Here you are pit .

-I Parameters are simply , If there are more proto There are interdependencies between files , Generate a proto When you file , need import Others proto file , It's time to use -I To specify the search directory . If not specified -I Parameters , Search in the current directory .

Every -I Parameters are introduced into a directory ,proto Several external proto Theoretically speaking, how many documents are needed -I（ The same directory can be imported at one time ）, Plus the to be compiled proto We also need to introduce , So two are used here -I To import directory files .

such protoc Can be in -I path + import path => “./…/aaa/aaa.proto” Found under path aaa.proto This file .

#  Of course  import “aaa.proto”,-I=./…/aaa, Can also execute successfully .
protoc -I ../aaa -I ./ --go_out=./ bbb.proto

import public

By default ,proto Only direct references are allowed import Data types defined in the file .

Such as b.proto In the import a.proto,c.proto In the import b.proto; By default ,c.proto Only... Can be referenced in b.proto Data types defined in , But cannot quote a.proto Data types in . if c.proto To use a.proto Data types defined in , be b.proto quote a.proto Use it when import public.

---my_project
   |---protocol
       |---aaa
       |   |---aaa.proto
       |---bbb
           |---bbb.proto
       |---ccc
       	   |---ccc.proto

aaa/aaa.proto

syntax = "proto3";
package aaa;
option go_package = "./";

message Something {
  string msg = 1;
}

bbb/bbb.proto

syntax = "proto3";
package bbb;
option go_package = "./";

// import "aaa/aaa.proto";  No mistake will be reported 
import public "aaa/aaa.proto";

message Something2 {
  aaa.Something something = 1;
}

ccc/ccc.proto

syntax = "proto3";
package ccc;
option go_package = "./";

import "bbb/bbb.proto";

message Something3 {
  aaa.Something something = 1;
}

perform ：

protoc -I ../ -I ../ -I ./ --go_out=./ ccc.proto

This usage is migrating proto Files are very useful when they are in a new location , Such as Message Class from old.proto Migrate to new.proto In file , At this time, if you want to modify the right old.proto In the case of documents , Direct will Message Move to new.proto in , And then in old.proto in import public new.proto that will do .

7. to update Message Message type principle

In order to achieve the purpose of compatible message types , Expand Message When it comes to message types, you need to pay attention to several points ：

1. Do not change the numeric ID of any existing fields .
2. The added field attribute must be optional perhaps repeated type , If extended required type , Will cause old message parsing exceptions
3. Not required Fields can be removed . Make sure that their tags are no longer used in new message types
4. A non required The field can be converted into an extension , vice versa —— As long as its type and identification number remain unchanged .
5. int32, uint32, int64, uint64, and bool Are all compatible , This means that you can convert one of these types to another , Without breaking forward 、 Backward compatibility . If the parsed number does not match the corresponding type , Then the result is like C++ Cast it in the same way （ for example , If you put one 64 Digit as int32 Come on Read , Then it will be truncated to 32 Digit number ）.
6. sint32 and sint64 It's compatible with each other , However, they are not compatible with other integer types .
7. string and bytes Is compatible —— as long as bytes It works UTF-8 code .
8. Nested messages and bytes Is compatible —— as long as bytes Contains an encoded version of the message .
9. fixed32 And sfixed32 Is compatible ,fixed64 And sfixed64 Is compatible .

4. be based on Protobuf Of RPC( You can skip this part )

For no ⽤ too Protobuf Readers of , It is suggested to start from the official ⽹ Understand the basic ⽤ Law . this ⾥ We try to Protobuf and RPC Combined in ⼀ Start to make ⽤, adopt Protobuf To finally guarantee RPC Of ⼝ Specification and safety .Protobuf The most basic unit of data in is message, Is similar to Go language ⾔ The existence of structures in . stay message You can nest message Or other basic data types member .

Definition RPC data structure ：

07-pbrpc/service/service.proto

syntax = "proto3";

package hello;

// go module = MicroServiceStudy01
option go_package = "MicroServiceStudy01/07-pbrpc/service";

message Request{
  string value = 1;
}

message Response{
  string value = 1;
}

Generate go Language structure ：

$ cd 07-pbrpc

$ protoc -I ./service --go_out=./service --go_opt=module="MicroServiceStudy01/07-pbrpc/service" service/service.prot
o

Definition RPC Interface ：

be based on Generated data structure , Defining interfaces ：

07-pbrpc/service/interface.go

package service

const HelloServiceName = "HelloService"

type HelloService interface {
    
	// Hello
	//  there  Request  and  Response  Is based on protobuf Generated service.pb.go The structure in 
	Hello(request *Request, response *Response) error
}

This interface is used to constrain parameters , See 2. Safer RPC Interface

There is no union before protobuf When you use it , The parameter type of our interface method here is the structure type written by ourselves , And used protobuf after , The parameter types here need to reference us through protobuf Generated .pb.go The structure type in the file .

The interface we define should be placed in a Separate files are similar to the current service package , He is equivalent to a contract package , be used for Constrain the server server（ Provide RPC service ） And our client client（ call RPC service ）.

Define the server ：

07-pbrpc/server/server.go

type HelloService struct{
    }

func (hs *HelloService) Hello(req *service.Request, resp *service.Response) error {
    
	resp.Value = "hello:" + req.Value
	return nil
}

//  Through interface constraints  Server  End 
var _ service.HelloService = (*HelloService)(nil)

func main() {
    
	rpc.RegisterName(service.HelloServiceName, new(HelloService))
	listen, err := net.Listen("tcp", ":1234")
	if err != nil {
    
		log.Fatal("Listen TCP err:", err)
	}
	for {
    
		conn, err := listen.Accept()
		if err != nil {
    
			log.Fatal("Accept err:", err)
		}
        //  It's still used here json, Ignore first   To look down 
		go rpc.ServeCodec(jsonrpc.NewServerCodec(conn))
	}
}

Define the client ：

07-pbrpc/client/client.go

type HelloServiceClient struct {
    
	*rpc.Client
}

func (hsc HelloServiceClient) Hello(req *service.Request, resp *service.Response) error {
    
	return hsc.Client.Call(service.HelloServiceName+".Hello", req, resp)
}

//  Through interface constraints  Client  End 
var _ service.HelloService = (*HelloServiceClient)(nil)

func DialHelloService(network, address string) (*HelloServiceClient, error) {
    
	conn, err := net.Dial(network, address)
	if err != nil {
    
		log.Fatal("net.Dail err: ", err)
	}
	client := rpc.NewClientWithCodec(jsonrpc.NewClientCodec(conn))
	return &HelloServiceClient{
    client}, nil
}

func main() {
    
	client, err := DialHelloService("tcp", "localhost:1234")
	if err != nil {
    
		log.Fatal("Dial err: ", err)
	}
	resp := &service.Response{
    }
	err = client.Hello(&service.Request{
    Value: "world"}, resp)
	if err != nil {
    
		log.Fatal(err)
	}
	fmt.Println(resp)
}

At this time, we just Hello Method parameters use protobuf Generated service.pb.go Structural body in , But other logic has not changed , Using or json-rpc, So here you will find , Although we have defined relevant protobuf, But we and protobuf It doesn't matter half a dime yet , It just uses the structure he generated for us ;

So how do we json Replace the code with protobuf What about encoding? ？

take 07-pbrpc/server/server.go Inside go rpc.ServeCodec(jsonrpc.NewServerCodec(conn)) Modified into go rpc.ServeCodec(server.NewServerCodec(conn)) that will do .

This is us. 4. be based on Protobuf Of RPC Key points of , Official net/rpc There is nothing in the bag protoc Plug in for

The publisher of the video I watched imitated net/rpc/jsonrpc I wrote a story about Proto Codec Codec package , But it's not shown in the video , And here it is NewServerCodec Just use the method in that bag , You don't have to go deep , Logic is such a logic , Focus on understanding .