[protobuf] some pits brought by protobuf upgrade

---

Some time ago, the company relied on github.com/golang/protobuf Version from v1.3.3 Upgrade to v1.4.2, This article records some problems encountered during the upgrade process .

Google Yes Go Of protobuf The bottom layer of the library has been greatly improved , The package path of the new version is transferred to google.golang.org/protobuf.
meanwhile , These improvements have also been brought into github.com/golang/protobuf： from v1.4 Since version ,github.com/golang/protobuf Will be in google.golang.org/protobuf On the basis of , But it will ensure that the interface is compatible , This also indicates the current dependency github.com/golang/protobuf The project can be upgraded directly , There is no need to change the upper code .

However , New version of the protobuf-gen-go Used google.golang.org/protobuf/protoreflect, Compile the message The structure is completely different , This has brought some trouble to our upgrade work .

1. Right in code XXX_Unmarshal Direct call to

The original protoc-gen-go Will expose a XXX_Unmarshal Interface , Used in proto.Unmarshal Called when , So some colleagues choose to call directly message.XXX_Unmarshal Method . New version of the proto adopt ProtoReflect Interface exposed message Internal information , compile pb.go Time is gone XXX_Unmarshal Method , So it will cause compilation errors message.XXX_Unmarshal undefined.

The solution is simple ： change to the use of sth. proto.Unmarshal that will do .

2. Changes in the internal structure of the structure lead to test errors

For the same message, The structure of the compiled structure of the old version is as follows :

type HealthCheckResponse struct {
    
    Status               HealthCheckResponse_ServingStatus `protobuf:"varint,1,opt,name=status,proto3,enum=liulishuo.common.health.v1.HealthCheckResponse_ServingStatus" json:"status,omitempty"`
    XXX_NoUnkeyedLiteral struct{
    }                          `json:"-"`
    XXX_unrecognized     []byte                            `json:"-"`
    XXX_sizecache        int32                             `json:"-"`
}

The compiled structure of the new version is as follows ：

type HealthCheckResponse struct {
    
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    Status HealthCheckResponse_ServingStatus `protobuf:"varint,1,opt,name=status,proto3,enum=liulishuo.common.health.v1.HealthCheckResponse_ServingStatus" json:"status,omitempty"`
}

You can see , Three non exported fields have been added in the new version , These three fields bring some trouble to our test code .

1. cmp.Equal when panic
   Our tests used github.com/google/go-cmp/cmp.Equal Come on proto Compare the structure , The non exported fields in the structure will make cmp.Equal and cmp.Diff panic:

panic: cannot handle unexported field at {
    *pkg.SomeRequest}.state:
    ".../services_go_proto".SomeRequest
consider using a custom Comparer; if you control the implementation of type, you can also consider using an Exporter, AllowUnexported, or cmpopts.IgnoreUnexported [recovered]

go-cmp The recommended way is to use IgnoreUnexported, But this method needs to pass all types that need to be ignored , For those with multiple levels of nesting message Very unfriendly .
After a search , Find out protocmp.Transform I can take all of them protobuf message Convert to custom map[string]interface{} Compare , So we can use Transform() To solve the problem ：

import "google.golang.org/protobuf/testing/protocmp"

// ...
opt := protocomp.Trnasform()
if !cmp.Equal(exp, got, opt) {
    
    t.Error(exp, got, opt)
}

2. assert Stuck and full of memory
   Compared with the above problem , The following question is even more strange ： Use github.com/stretchr/testify/assert.Equal Compare some special proto message It will get stuck when , At the same time, memory usage will skyrocket .

Try to use pprof sampling , Extracted CPU And the heap sample graph looks like this ：

You can see spew.Dump There is infinite recursion in , This leads to program sticking and persistent memory allocation .

Then we found it testify Of issue, Several bypass schemes have been proposed in relevant comments , However, this problem has not been solved .

There are two personally recommended solutions ：

1. Use marshalTextString() take message convert to proto text, Then compare ;
2. Use cmp.Equal, combination protocmp.Transform.

3. lint Report errors copylocks

After processing the business code processing test , After processing the test code, there are lint To deal with .

After upgrading our project ,go vet Will be submitted to the copylocks error ：assignment copies lock value to got: .../message_go_protos.Message contains google.golang.org/protobuf/internal/impl.MessageState contains sync.Mutex

The solution is simple ： Will all proto message Change to pointer passing .