Ekuiper newsletter 2022-05 protobuf codec support, visual drag and drop writing rules

This may ,eKuiper be in 1.6.0 The early stage of the development of new functions in the version , Product function development mainly includes protobuf Codec support . It is aimed at the user's requirement of writing data processing rules by dragging , We are also doing prototype verification .v1.5.1 Of bug The repair is also in progress , Is due to 6 Early release .

Besides , As an edge stream data processing software ,eKuiper Also participated in the 《 Technical capability requirements for edge stream data processing 》 Standard assessment , After the adoption of eKuiper It will be the first product to pass this standard .

protobuf Codec support

at present source and sink About the configuration properties of codec format The default value is JSON, Users can use the new protobuf Options for protobuf Coding and decoding capability of . Compared with JSON,protobuf The amount of encoded data is smaller , It helps to save the transmission bandwidth between cloud edges . This function has been developed but not released yet , Users can use the edge source code v1.6.0 Branch for trial .

Compared to no mode （schema） Of JSON Format ,protobuf Need to define proto File as codec schema. In the use of protobuf Before the format , Users need to register first schema, And through the new schemaId attribute , Specify the encoding and decoding options schema.

Schema management

Use REST API register ,schema The content can be provided through file path or text .

// POST /schemas
{
  "id":"fileName",
  "file":"http://myhost/files/abc.proto"
}

Or configure through text content ：

// POST /schemas
{
  "id":"schemaName",
  "content":"message Person {
    required string name = 1;
    required int32 id = 2;
    optional string email = 3;
  }"
}

No matter how , Content will be stored in etc/schemas/protobuf.

Sink Use in protobuf

• format, Used to specify the encoding used
• schemaId, Used to designate the use of schema

{
  "mqtt":{
    "server":"tcp://127.0.0.1:1883",
    "topic": "result",
    "format":"protobuf",
    "schemaId":"schemaName.Person"    // protobuf Of  ID In two parts , The front is the file name , The back is message name 
  }
}

Source Use in protobuf

FORMAT, Support protobuf

SCHEMA_ID , Used to designate the use of schema

CREATE STREAM demo() WITH (TYPE="NEURON", FORMAT="protobuf", SCHEMA_ID="schemaName.Person")

Visual drag and drop editing capability

Before eKuiper Only... Is supported for streaming data SQL Form to write data processing rules , There is a certain threshold , It is not convenient for business personnel to directly participate in rule writing . In order to further reduce the use threshold ,eKuiper The preparation supports the visual dragging and dropping of data processing units and simple editing and configuration , Finally, multiple data processing units are connected to form data processing rules automatically , It is convenient for more people to use it directly eKuiper Conduct business processing . At present, this scheme is in the prototype design stage .

Coming soon

Next month we will improve visual drag and drop editing , And merge into the main branch . in addition ,eKuiper Will optimize sink Cache mechanism after error , Implement memory + Offline storage of disk , And resend in sequence after error recovery , To support stronger recovery capability in case of network disconnection , Cache data for a longer time .

Copyright notice ： This paper is about EMQ original , Reprint please indicate the source .

Link to the original text ：https://www.emqx.com/zh/blog/ekuiper-newsletter-202205