The design idea of ​​DMicro, the Go microservice development framework

Go 微服务开发框架 DMicro 的设计思路

DMicro 源码地址:

• Gitee:
• dmicro: dmicro是一个高效、Extensible and easy to use micro service framework.包含drpc,dserver等

背景

DMicro 诞生的背景,Because I wrote 10 来年的 PHP,Want to be in company internal promotion Go, Internal components and rpc 协议都是基于 swoole 定制化开发的.The research on the market all kinds of framework,包括 beego,goframe,gin,go-micro,go-zero,erpc 等等,May be I was technical ability is limited,Doesn't make these frameworks good fit our business.

We have several business development pain points,在当时 golang A set of solution cannot be found in the ecological.

• Micro service application and monomer application development at the same time.
• 高性能,The high availability of network communication.
• Need a custom application layer protocol (重点).
• Need to be flexible plug-in extension mechanism,Convenient adaptation of existing system (重点).
• Concept of server and client,Can use the same each other api 调用对方.
• 支持 Push 消息.
• 连接 / 会话管理.
• 高效率的开发,支持通过 proto 生成代码.
• 支持多种网络协议,tcp,websocket,quic,unixsocket.
• 兼容 http 协议.
• Can be more rapid positioning problem.
• 更便捷的增加新特性.

In the commonly used open source frameworks do the simple research later,Discovery is not an appropriate framework can meet all my needs.After serious thinking,发现 erpc 和 goframe The combination of the two frameworks fitness to satisfy my needs,So was born the r DMicro.

概述

DMicro 中的 drpc Thought is the reference component erpc 实现,Even it's successor.

drpc 组件是 DMicro 框架的一部分,为了适配 DMicro 框架,在 erpc On the basis of a deeply extension development.

整个 DMicro 大量使用 goframe 中的组件,如果业务使用 goframe 框架,可以无缝接入.

DRpc 特性列表:

• 对等通信 , 对等Api
• 高性能 , 非阻塞异步IO
• 自定义Proto,, 兼容http协议 , 自定义Codec
• Hook点 , 插件系统 ,
• Push消息 ,session管理,Socket抽象 ,
• 断线重连 , 过载保护 , 负载均衡 , 心跳机制 ,
• 平滑重启 ...

DServer 特性列表：

• 快速构建 , 平滑重启 , 多进程支持 , 单/Multiple processes consistent
• Predefined command line ,ctrl命令管理服务
• 可观测 , 可控制 , 应用沙盒

DMicro Have built-in component：

• [x] Registry 服务注册
• [x] Selector 服务发现
• [x] Eventbus 事件总线
• [x] Supervisor 进程管理
• [ ] Code gen 代码生成
• [ ] Tracing 链路追踪
• [ ] Metrics Statistics the alarm
• [ ] Broker 限流熔断
• [ ] OpenAPI 文档自动生成

架构

设计理念

对 DMicro 框架的设计,From the beginning of the design is in the pursuit of flexibility,适应性.Under the premise that ensure the stability of the micro service,Pursuing the development of the project efficiency.

• 面向接口设计,保证代码稳定,Provide flexible customization.
• Abstract interface components,高内聚,低耦合.
• 分层设计,Step by step a top-down assembly,Conducive to stability and maintain.
• 高性能,高可用,低消耗.
• 对开发友好,封装复杂度.
• Provide a rich component and function,让开发专注业务.

Countless write DMicro Day and night,I remember that developed three principles:

• Clarity(清晰)
• Simplicity(简单)
• Productivity(生产力)

无论工作,Open source projects or do,Should keep the three principles,养成良好的习惯.

面向接口设计

DMicro The principle of taking everything interface,Provide the framework unparalleled extensibility.

Below is a message sent by the circulation flow,可以看到,All function points are abstracted to interface,The realization of each function point provides different.

会话 Session

大多数的 Rpc Framework does not emphasize the session (session) 的概念,Because of its application scenario does not need to use the session (session). 那么 drpc Why need to abstract the session (session) 呢？

• Endpoint 融合了 Client 和 Server, 需要提供相同的 Api.
• 服务端Need to客户端发送消息,And access to the client's response.
• 服务端支持对多个客户端批量发送消息.
• Asynchronous active disconnect一个或多个会话.
• Access to the session at the bottom of the文件描述符 , For performance tuning.
• For each session can bind special数据/属性.

Session Abstract the whole drpc Framework of the session,把 Socket,Message,Context All mixed together.开发者只需要对 session 进行操作,Can achieve most demand.

• 获取连接信息
• The life cycle of control connection (超时时间)
• Control of a single request life cycle (超时时间)
• 接收消息
• 发送消息
• Create a message context
• Binding session information (如用户信息)
• 断线重连
• Take the initiative to disconnect session.
• 健康检查
• Get connection is closed event
• As the session set separate id

Session Interface can be subdivided into 4 个 interface{}, 分别是 EarlySession,BaseSession,CtxSession,Session. Corresponding to the application of different life stages session (Session) Have different properties.

• EarlySession Said just generate the session,尚未启动 goroutine Read the data of phase.
• BaseSession Only the most basic way,Used to close the connection when the plug-in parameters.
• CtxSession In the handler context transfer session object.
• Session Full-featured session object.

正常情况下,Developers are used Session,CtxSession 这两个接口,其他 2 An interface is in the plug-in to use.

消息 Message

消息 Message 包含消息头 Header, 消息体 Body, Is the client and the server communication between entities.

Message interface{} Abstract to the operation of the communication entity.

• Size 消息的长度
• Transfer-Filter-Pipeline Packet filtering data processing pipeline
• Seq 序列号
• MType 消息类型
• ServiceMethod 资源标识符
• Meta 消息的元数据
• BodyCodec The body of the message encoding format
• Body 消息体

协议 Proto

协议是对消息Message Object serialization and reverse serialization,框架提供 Proto 接口.只需要实现该接口,Developers can customize in accordance with business requirements custom agreement,Thus promotes the flexibility of the frame.

接口的定义如下:

type Proto interface {
	Version() (byte, string)
	Pack(Message) error
	Unpack(Message) error
}

• Version() Return the agreement id 和名字,Two of the only version number.
• Pack 对消息 Message 对象进行序列化.
• Unpack The byte stream deserialization,生成一个消息 Message 对象.

At present, the framework has support Http,Json,Raw,Protobuf,JsonRpc 这 5 个协议.

RAW Agreement of the following:

Other agreements can reference code.

编码 Codec

As a general framework of,The supported protocols can have a variety of,The body of the codec can also have many kinds of. drpc 使用 Codec Interface to the body Body 进行编解码.

接口的定义如下:

type Codec interface {
	ID() byte
	Name() string
	Marshal(interface{}) ([]byte, error)
	Unmarshal([]byte, interface{}) error
}

• ID Return to weave Codec 的 id
• Name Return to weave Codec 的名字,Name is for developers to more easily identify.
• Marshal To encode the message content
• Unmarshal To decrypt the message content

At present, the framework has support Form,Json,plain,Protobuf,XML 这 5 A codec.

连接 Socket

Socket 扩展了 net.Conn, And abstract the interface,Convenient framework to the underlying network protocol integration.

Socket The interface part of Session 接口的功能,Session Some methods of interface call,Is actually a forwarding call Socket 中的方法.

The layered realization,让 Socket Have the ability to integrate other agreement.

• TCP V4,TCP V6
• Unix Socket
• KCP
• QUIC

Support performance tuning of the connection in the.

• SetKeepAlive Open links to keep alive
• SetKeepAlivePeriod Link to keep alive interval
• SetReadBuffer Set link to read buffer size
• SetWriteBuffer For a link to write buffer size
• SetNoDelay 开启关闭 no delay 算法
• ControlFD Support operation links of the original handle

有机的组合

前面讲到,DMicro Framework and all the interface,分层 + 接口的设计,让 DMicro Of a flexible efficient and accord with the actual situation of the business ability.

Next we want to talk about the basis of these skills.插件系统.

插件 Plugin

Plug-in system brought great extensibility and flexibility to framework,Is a soul of the framework module,有了它,The framework has infinite possibility.

What kind of plug-in system to be elegant？我能想到的有以下几点：

• Reasonable and rich hook 位置,Can cover the whole framework of life cycle,Through the communication of each link.
• 每个 hook Location into and out of the refs are carefully designed.
• Each plug-in can use multiple hook 位置,每个 hook Position can be used by multiple plug-ins.
• Design concise enough,优雅.Can be convenient for secondary development custom.

在 drpc 中,Throughout the hook with the Endpoint 的生命周期,Is it one of the most important aspects of the integral.

 通过这些钩子 Hook 点,Given the plug-in infinite possibility.

组件

有了插件,Can through the combination of plug-ins,Write the comprehensive function of the component,The framework provides some of the built-in components,

• 服务端 Rpc Server
• 客户端 Rpc Client
• 服务注册 Registry
• 服务发现 Selector
• 事件总线 EventBus
• 进程管理 Supervisor

即将提供:

• 链路追踪 Tracing
• Statistics the alarm Metrics
• 限流熔断 Broker.

限于篇幅的原因,The realization of the specific components,这里就不深入讲解,请关注后续的文章.

未来展望

如果把 DMicro Compared to life,Now the stage of growth is still at the teenager,Only completed the basic part of architecture design and components development.

The next direction is mainly to develop in the direction of ease of use and reliability.

易用性:

• Project performance tools dmctl 工具的开发,包括代码生成,The project structure generated,打包,Compile, etc.
• 符合 openapi The development of document components of the definition of.
• More complete documentation and usage examples.

可靠性:

• 可观测性
  • 链路追踪
  • 指标信息
  • 日志流
• 生产可用
  • 测试用例的完善
  • 代码覆盖率
  • 性能调优

希望 DMicro Can growing under everybody's care and spur growth.