Full resolution of the use of go native plug-ins

writing ｜ Ding Fei （ flower ： Luther  )

Senior engineer of ant group

Deep in  SOFAMesh  Commercialization of products

The main direction is the design and implementation of system architecture upgrading scheme based on Service Grid Technology

this paper  4394  word   read  10  minute

｜ Preface ｜

MOSN  As an ant group  ServiceMesh  Data side components in the solution , From the beginning of the design, the expansion development needs of the third party have been taken into account . at present ,MOSN  Supported by  gRPC、WASM、 as well as  Go  Three mechanisms of native plug-ins extend it .

I am leading the design and landing based on  Go  There are many problems in the extension ability of the native plug-in mechanism , In view of the lack of relevant information in this regard , So I have this idea to make a very shallow summary , I hope it can help you .

notes ： This article only talks about problems and solutions , Don't read the code , The core source code will be given at the end of the article  checklist.

PART. 1-- Article technical background

One 、 Runtime

generally , In the field of computer programming languages ,“ Runtime ” The concept and some need to be used  VM  Language related . The operation of the program consists of two parts ： Object code and “ virtual machine ”. For example, the most typical  JAVA, namely  Java Class + JRE.

For some seemingly unnecessary “ virtual machine ” Programming language , There will not be “ Runtime ” The concept of , The program only needs one part to run , I.e. target code . But in fact , Even if it's  C/C++, Also have “ Runtime ”, That is, the platform it runs  OS/Lib.

Go  Is the same , Because running  Go  The program does not need pre deployment, similar to  JRE  Of “ Runtime ”, So it looks like “ virtual machine ” perhaps “ Runtime ” Do not have what relation . But in fact ,Go  Linguistic “ Runtime ” It is compiled into part of the binary object code by the compiler .

chart  1-1. Java  Program 、runtime  and  OS  Relationship

chart  1-2. C/C++  Program 、runtime  and  OS  Relationship

chart  1-3. Go  Program 、runtime  and  OS  Relationship

Two 、Go  Native plug-in mechanism

As one looks closer  C/C++  Technology stack  Go  language , Supporting the expansion of similar dynamic link libraries has always been a strong demand in the community .

Pictured  1-5,Go  In the standard library, there is a  plugin  package , Language level programming interface as a plug-in ,src/plugin  The essence of a package is

Use  cgo  Mechanism call  unix  Standard interface for ：dlopen()  and  dlsym()

. therefore , It gives  C/C++  A programmer with a background “ I will answer this question ” Illusion .

chart  1-4. C/C++  Program loading dynamic link library

chart  1-5. Go  Program loading dynamic link library

PART. 2-- Typical problem solving

unfortunately , And  C/C++  Compared with the technology stack ,Go  Although the output of the plug-in is also a dynamic link library file , But it is for plug-in development 、 There are a series of complex built-in constraints . What's bigger is ,Go  The language not only does not systematically introduce these constraints , Even wrote some poor design and implementation , The troubleshooting that leads to plug-in related problems is very anti-human .

This chapter focuses on following with you , Developing 、 Use  Go  plug-in unit , Mainly compiling 、 When loading plug-ins , Most common 、 But it must be located  Go  Standard library

（ Mainly including compiler 、 The linker 、 Packer and runtime part ）

Several problems that can be fully understood by the source code , And corresponding solutions .

In short ,Go  The main program of is loading  plugin  when , Will be in “runtime” A bunch of constraint checks are carried out for both , Including but not limited to ：

-

 go version  Agreement

- 

go path  Agreement

- 

go dependency  The intersection of is consistent

• The code is consistent

• path  Agreement

- 

go build  some  flag  Agreement

One 、 Inconsistent standard library versions

Error occurs when the main program loads the plug-in ：

plugin was built with a different version of package runtime/internal/sys

From the text of this error report, we can know , The library with specific problems is  runtime/internal/sys , Obviously this is a  go  Built in standard library . See here , You may have great doubts ：

I obviously use the same local environment to compile the main program and plug-in , Why is the standard library not a version ？

The answer is ,

Go  Of  error  The log description is inaccurate

. The root cause of this error can be attributed to ：

Some key compilation of main programs and plug-ins  flag  atypism

, Follow “ edition ” Do not have what relation .

such as , You use the following command to compile the plug-in ：

GO111MODULE=on go build --buildmode=plugin -mod readonly -o ./codec.so ./codec.go

But you use  goland  Of  debug  Mode debugging main program , here ,goland  Will help you with  go build  The command is assembled according to the following example ：

Be careful ,goland  The compilation command of the assembly contains key  -gcflags all=-N -l  Parameters , But there are no commands in the plug-in compilation . here , When you try to pull up the plug-in, you will get a related  runtime/internal/sys  The error of .

chart  2-1.  compile  flag  Loading failure caused by inconsistency

The solution to this kind of standard library version inconsistency is relatively simple ：

Try to align the main program with the compiled plug-in  flag

. in fact , Somewhat  flag  It does not affect the loading of plug-ins , You can explore slowly in specific practice .

Two 、 Inconsistent third-party library versions

If you use  vendor  To manage  Go  The dependent libraries , Then, after solving the problems in the previous section , you  100%  You will immediately encounter the following error report ：plugin was built with a different version of package xxxxxxxx

among ,xxxxxxxx  It refers to a specific third-party library , such as  github.com/stretchr/testify. There are several typical reasons for this error , If there is no relevant troubleshooting experience , Several of them may burn developers' time .

Case 1.  Version inconsistency

As shown in the error report , It seems that the reason is very clear , namely

The version of a third-party library on which the main program and plug-in depend is inconsistent

, The error report will clearly tell you which database has problems . here , You can compare and check the main program and plug-in  go.mod  file , Find the version of the problem base respectively , See if they agree . If at this time you find that the main program and plug-ins do have  commitid  or  tag  Inconsistencies in , The solution is also very simple ：

Align them

.

But in many scenarios , You will only use part of the tripartite Library ： Like a  package, Or just some  interface. This part of the code may not have changed at all in different versions , But other unused code changes , It will also lead to changes in the version of the entire third-party library , Which leads you to become that “ Version inconsistency ” Innocent victims of .

and , At this point, you may immediately encounter another problem ： Align with who ？ The main program ？ Or plug-ins ？

In common sense , Taking the main program as the baseline for alignment is a better strategy , After all, plug-ins are new “ Accessories ”, And the main program and plug-in are usually “ One to many ” The relationship between . however , What if the third-party library dependency of the plug-in cannot be aligned with the main program for any reason ？ After trying for a long time , I haven't found a perfect solution to this problem for the time being .

If the version cannot be aligned , You can only fundamentally give up taking the plug-in path .

Go  Language of this kind of tripartite Library 、 Almost mindless strong consistency constraint , On the one hand , Avoid potential problems caused by inconsistent versions at runtime ; On the other hand , This design deliberately does not give programmers flexibility , Plug in 、 Customized 、 Extended development is very unfriendly .

chart  2-2.  Loading failure caused by inconsistent versions of the three party libraries of common dependencies

Case 2.  The version number is the same , Code inconsistency

When you follow  case 1  The way of thinking  go.mod  file , But I was surprised to find that the wrong library version is the same , It's going to get complicated . You may come up with the most advanced text checking tool in the world , And spend the whole morning going  diff  Three party Library  commitid, But they are exactly the same , It seems to fall into the Schrodinger version .

One possible reason for this problem is ： Someone directly modified  vendor  Directory code ,Go  The plug-in mechanism will verify the consistency of code content .

This is really a very heady , And the reasons that are difficult to check . Except for the person who modified the code , And already in other  case  Middle quilt “ pit ” Those who have , No one will know about it . If modified  vendor  The code appears in the main program , You have almost no reliable way to make them work properly .

Don't go straight to  vendor  Change the code inside ！！！

Don't go straight to  vendor  Change the code inside ！！！

Don't go straight to  vendor  Change the code inside ！！！

Give back to the open source community , perhaps  fork-replace！！！

The good news is , You don't need to solve this problem . Because even if it is solved , There will also be bigger problems waiting for you .

chart  2-3.  The loading failure caused by the local modification of the code of the jointly dependent third-party library

Case 3.  The path is inconsistent

When according to  case 1  and  case 2  The idea of troubleshooting 、 Solved , But it still reports  different version of package  When , Maybe you will start to be right  Go  The plug-in mechanism of has lost patience ：

Version really “ 10 Fen is the same ”, The code really doesn't move a line , Why do you report different versions ？？？

as a result of ：

The plug-in mechanism will verify the source code of the dependent Library 「 route 」

, So it can't be used  vendor  Management dependence .

for instance ： Your main program source code is placed in  /path/to/main  Under the table of contents , therefore , The directory on which one of your third-party libraries depends should be ：

 /path/to/main/vendor/some/thrid/part/lib;

Empathy , Your plug-in source code is placed in  /path/to/plugin  Under the table of contents , therefore , The directory that the same third-party library depends on should be ：

 /path/to/plugin/vendor/some/thrid/part/lib.

these 「

File path

」 The data will be packaged into binary executable files and used for verification , When the main program loads the plug-in ,Go  Of “ Runtime ”“ smart ” adopt 「

File path

」 The difference is that it is not the same code as the plug-in , And then I reported  different version of package.

chart  2-4.  Use  vendor  Loading failure caused by mechanism management of third-party libraries

The same problem may also occur when using different machines / user , Compile the main program separately 、 Plug in scenario ： Different user names ,go  The path of the code should also be different .

The way to solve this kind of problem is very violent and direct ：

Delete the main program and plug-in  vendor  Catalog , Or use

 -mod=readonly 

compile  flag

.

Come here , If you are using the same machine to compile the main program and plug-in , Then the common problems should be basically solved , The plug-in mechanism should work properly . On the other hand , Because it is no longer used  vendor  Management dependence , therefore  case 2  The problem of will also be forcibly solved here ： Or mention  PR  To the community , or  fork-replace.

chart  2-5.  Successfully loaded

3、 ... and 、 inconsistent  Go  edition

fatal error: runtime: no plugin module data

In addition to the above problems , There is also one that compiles the main program separately on multiple machines / Common error reports in plug-in scenarios . One possible reason for this error is

 Go  Version inconsistency

, Just align them .（ What if the alignment is not possible at the machine level ？……）

chart  2-6. Go  Loading failure caused by version inconsistency

PART. 3-- Unified solution

From the second  Part  in , We have seen some that are difficult to check , It's not a very easy problem to deal with . besides , In fact, there are still some problems that have not been highlighted . As an extension mechanism officially supported by a programming language , It's really unexpected to do so user unfriendly .

because 「

Proprietary cloud  MOSN

」 Focus on  Go  The plug-in mechanism of is set to open , Therefore, we must come up with a systematic plan to solve all these problems . Trying to modify directly  Go  After the source code is fruitless

（ Make complaints ：Go  The source code of the plug-in mechanism is slightly regrettable ）

, We focus on “ Product layer ” And peripheral infrastructure ：

- 

Unified compilation environment ：

• Provide a standard  docker image  Used to compile main programs and plug-ins , Avoid any  go  edition 、gopath  route 、 Problems caused by inconsistent user names ;

• Prefabrication  go/pkg/mod, Reduce as much as possible because it is not used  vendor  Pattern causes the problem of downloading dependencies again every time you compile .

- 

Unified  Makefile：

• Provide a set of compilation of main programs and plug-ins  Makefile, Avoid anything because  go build  Problems with orders .

- 

Unified plug-in development scaffolding ：

• By scaffolding , Instead of developers pulling together the dependent versions of plug-ins and main programs . And the scaffold solves other related problems .

- 

Pipelining ：

• Pipelining compilation and deployment , Further avoid mistakes .
• 
  

chart  3-1.  Unified solution

PART. 4-- Key source location

If you really want to fundamentally understand the mechanism of plug-in verification , Here are some entries for you to quickly enter the source code reading state . I use  Go  The source code is  1.15.2  edition . relevant  Go  Source location ：

- compiler

：go/src/cmd/compile/*

- linker

：go/src/cmd/link/internal/ld/*

- pkg loader

：go/src/cmd/go/internal/load/*

- runtime

：go/src/runtime/*

One 、go build  What are you doing

You can  go build  Add  -x  Parameters , Print out explicitly  Go  Program compilation 、 link 、 The whole process of packaging , for example ：

go build -x -buildmode=plugin -o ../calc_plugin.so calc_plugin.go

Two 、 Target code generation

go/src/cmd/compile/internal/gc/obj.go:55 ： Pay attention to the  67  And the  72  That's ok , Here are two entrances ;

go/src/cmd/compile/internal/gc/iexport.go:244 ： Be careful  280  That's ok , It will be recorded here  path  related data .

3、 ... and 、 Library hash generation algorithm

go/src/cmd/link/internal/ld/lib.go:967： Pay attention to the  995-1025  That's ok , Here's the calculation  pkg  Of  hash.

Four 、 Library hash check

go/src/runtime/symtab.go:392 ： Key data structure ;

go/src/runtime/plugin.go:52 ： Link period  hash  And runtime  hash  Value check point ;

go/src/cmd/link/internal/ld/symtab.go:621 ： Link period  hash  Assignment point ;

go/src/cmd/link/internal/ld/symtab.go:521 ： Runtime  hash  Assignment point .

PART. 5-- summary

You can see , Even if  Go  The native plug-in mechanism of has various headache problems ,SOFAStack  The team still upholds “ Open source 、 to open up 、 Scalable ” The purpose of , Solve problems by various means , And finally make this capability available for production .

at present , Proprietary cloud  MOSN  Protocol codec and  logger  The customized development of has achieved a comprehensive plug-in . Next , We will continue to  MOSN  Architecture upgrade , The target pair includes routing logic 、LB  Logic 、 Registry Center / Plug in support of various capabilities including configuration center docking .

  Learn more about ...

MOSN Star  once ：

https://github.com/mosn/mosn

Let's build together 🧸