golang with script

background

go It's a static language , To run, you need to compile . In fact, in the process of using , Sometimes I hope go Be able to execute some dynamic commands like footstep language , This feature has at least the following benefits :

• Learn to use , As a kind of repl Tool validation syntax
• Quickly verify the functionality of a package , Or some remote resource content
• As the embedded code of the project , Provide flexibility , For example, as a crawler project , Capturing script changes certainly does not want the entire project to be recompiled
• As a test script , Script language development is fast , Fast verification , It is suitable for developing some testing tools

Open source project

Tools	grammar	As repl Usability	Availability as an embedded script	principle	remarks
direct go run	golang	Very low	in	go run	Can be in go Add... To the language file env identification , meanwhile Yes go run Encapsulate the , such as such , You can get packages dynamically
gop	golang like	high (v0.7)	high	Two modes ,bytecode Mode use VM Realization , Another pattern uses code generation + go run	v0.7 Support repl,master Has deleted
tengo	golang like	high	high	VM	official bench Indicates that the performance is very high in the same category
go-lua	lua	high	high	VM	Shopify Realization
gopher-lua	lua	high	Very high	VM	The document is rich , Easy to expand
goja	ECMAScript/JavaScript	high	in	VM	-
starlark-go	starlark(python A subset of )	high	high	VM	Google Produce , Grammar is python A subset of , Low learning cost and comfortable use
gpython	python	in	in	Interpreter	Low maturity
yaegi	golang	high	high	Interpreter	traefik Produce
go-pry	golang	in	Very low	Code generation + Interpreter	It can only be used as a grammar validation tool , The interaction is friendly , Cannot be used for inline
otto	JavaScript	in	high	Interpreter	The project is quite active
anko	golang like	in	in	Interpreter	-
gore	golang like	high	low	Code generation + go run	As repl The tools are OK , Support code prompt
gomacro	golang like	Very high	in	Code generation + Interpreter	As repl The best choice of tools , Support import Third party package , Very functional , This go jupyter kernel Is based on gomacro

explain

1. The tools are arranged in 202110
2. As an embedded script , Should consider whether to use go The language comes with its own package or develops a third-party library to enrich its functions , this gopher-lua It's more convenient
3. As an embedded script , To reduce the use cost , Use what already exists 、 The advantages of simple language are obvious , such as lua、starlark
4. Whether it can be dynamic import The third-party library is for repl Use is a very important function , This only gomacro Support
5. The availability score is Very low / low / in / high / Very high
6. There are two main implementations ,VM perhaps Interpreter, Generally speaking VM The performance will be much higher 【 be based on vm There are some optimization strategies 】, But not absolutely . such as starlark-go It is thought in the document that go Realization vm Efficiency has no advantage , But at the moment starlark-go And the realization of vm （docs It's old , The code is already vm Realized ）

Expand

Use any of the kernel implementations listed above , An important problem to face is , How to extend efficiently . For example, use gopher-lua, Just to support lua Language is certainly not enough , Need to use some lua library . gopher-lua Provides a lua call go How to function , Make use of go To write gopher-lua Available third-party libraries become possible . in fact gopher-lua Of Introduce There are links to some third-party libraries , such as http, json, yaml And other common third-party libraries .

But it's not enough ,go Language has a very rich ecology , If you use it directly in a very simple way go The library of languages , That would be convenient . The solution to this problem is a library ：https://github.com/layeh/gopher-luar The idea of this library is through reflect The way to encapsulate go The library of languages , Method , And type , stay go Language and lua Automatic mapping between languages , Then do lua A third-party library of is very convenient .

Examples are as follows , Example as my gopher-lua The third party package has been submitted to github, Currently supported http, strings, regexp, ioutil, exec, crypto, json/yaml, colly, resty Wait for the third party package , And the whole implementation also has about 100 lines of code .

func NewLuaStateWithLib(opts ...lua.Options) *lua.LState {
	L := lua.NewState(opts...)
	InstallAll(L)
	return L
}

func InstallAll(L *lua.LState) {
	L.SetGlobal("http", luar.New(L, &Http{}))
	L.SetGlobal("regexp", luar.New(L, &Regexp{}))
	L.SetGlobal("ioutil", luar.New(L, &IoUtil{}))
	L.SetGlobal("exec", luar.New(L, &Exec{}))
	L.SetGlobal("time", luar.New(L, &Time{}))
	L.SetGlobal("resty", luar.New(L, &Resty{}))
	L.SetGlobal("colly", luar.New(L, &Colly{}))

	// more... pls refer to: github.com/vadv/gopher-lua-libs
}

// Http ----------------------------------------------------------------------------------------------------------------
type Http struct{}

func (h Http) DefaultClient() *http.Client { return http.DefaultClient }

// Regex ---------------------------------------------------------------------------------------------------------------
type Regexp struct{}

func (r Regexp) Compile(a string) (*regexp.Regexp, error)     { return regexp.Compile(a) }
func (r Regexp) Match(a string, b []byte) (bool, error)       { return regexp.Match(a, b) }
func (r Regexp) MatchString(a string, b string) (bool, error) { return regexp.MatchString(a, b) }

// IoUtils -------------------------------------------------------------------------------------------------------------
type IoUtil struct{}

func (i IoUtil) ReadAll(a io.Reader) ([]byte, error)               { return ioutil.ReadAll(a) }
func (i IoUtil) ReadFile(filename string) ([]byte, error)          { return os.ReadFile(filename) }
func (i IoUtil) WriteFile(a string, b []byte, c fs.FileMode) error { return os.WriteFile(a, b, c) }
func (i IoUtil) ReadDir(dirname string) ([]fs.FileInfo, error)     { return ioutil.ReadDir(dirname) }

type Url struct{}

func (u Url) Parse(a string) (*url.URL, error) { return url.Parse(a) }

// Cmd -----------------------------------------------------------------------------------------------------------------
type Exec struct{}

func (c Exec) Cmd(a ...string) *exec.Cmd { return exec.Command(a[0], a[1:]...) }
func (c Exec) Run(a ...string) ([]byte, []byte, error) {
	out, err := bytes.NewBuffer(nil), bytes.NewBuffer(nil)
	cmd := exec.Command(a[0], a[1:]...)
	cmd.Stdout, cmd.Stderr = out, err
	err1 := cmd.Run()
	return out.Bytes(), err.Bytes(), err1
}

type Time struct{}

func (t Time) Parse(a, b string) (time.Time, error) { return time.Parse(a, b) }
func (t Time) Now() time.Time                       { return time.Now() }

// Colly ---------------------------------------------------------------------------------------------------------------
type Colly struct{}

func (c Colly) New() *colly.Collector { return colly.NewCollector() }

// Resty ---------------------------------------------------------------------------------------------------------------
type Resty struct{}

func (c Resty) New() *resty.Client { r := resty.New(); return r }
func (c Resty) NewRequestWithResult(a M) *resty.Request {
	r := resty.New().NewRequest().SetResult(&a)
	return r
}

gopher-luar How to realize , I further expanded to starlark-go, To form the starlark-go-lib, In this bag , I offer something like gopher-luar Infrastructure of , Cause to give starlark-go Making a third-party package is extremely easy , Take the following example , Use very simple code to starlark-go Provides http And many third-party packages :

func InstallAllExampleModule(d starlark.StringDict) {
	for _, v := range exampleModules {
		d[v.Name] = v
	}
}

var exampleModules = []*starlarkstruct.Module{
	GreetModule,
	{
		Name: "modules",
		Members: starlark.StringDict{
			"all": ToValue(func() (ret []string) {
				for _, v := range starlark.Universe {
					if m, ok := v.(*starlarkstruct.Module); ok {
						ret = append(ret, m.Name)
					}
				}
				return
			}),
			"inspect": ToValue(func(a string) (ret []string) {
				if v, ok := starlark.Universe[a]; ok {
					if m, ok := v.(*starlarkstruct.Module); ok {
						for x, y := range m.Members {
							ret = append(ret, fmt.Sprintf("%s: [%s, %s]", x, y.Type(), y.String()))
						}
					}
				}
				return
			}),
		},
	},
	{
		Name: "http",
		Members: starlark.StringDict{
			"get":           ToValue(http.Get),
			"pos":           ToValue(http.Post),
			"defaultClient": ToValue(http.DefaultClient),
		},
	},
	{
		Name: "ioutil",
		Members: starlark.StringDict{
			"read_all":   ToValue(ioutil.ReadAll),
			"read_file":  ToValue(os.ReadFile),
			"write_file": ToValue(os.WriteFile),
			"read_dir":   ToValue(os.ReadDir),
		},
	},
	{
		Name: "strings",
		Members: starlark.StringDict{
			"contains": ToValue(strings.Contains),
			"split":    ToValue(strings.Split),
		},
	},
	{
		Name: "url",
		Members: starlark.StringDict{
			"parse": ToValue(url.Parse),
		},
	},
	{
		Name: "resty",
		Members: starlark.StringDict{
			"new": ToValue(resty.New),
		},
	},
}

demonstration

In order to use these packages more conveniently , I made one go Realized jupyter kernel The template of u2takey/gopyter, This bag is right gophernotes The implementation of , Use this template to implement your own kernel It will also be very simple , Only the following interface Can

type KernelInterface interface {
	Init() error
	Eval(outerr OutErr, code string) (val []interface{}, err error)
	Name() string
}

This package contains a large number of third-party libraries lua-go, stark-go Of kernel Realization . The final use example is as follows :

more

gopher-luar and starlark-go-lib Have put in gopher-lua and starlark-go Use in go Third party packages for languages become extremely easy . Of course you can do better , We can support similar import The sentence of , Automatically download third-party packages , And make grammatical analysis , Generate to gopher-lua and starlark-go As a third-party package , It is not logically difficult . But is it really necessary to do this , Because as repl Tools ,gomacro It's done well enough , Let's stop here .