The writing speed is increased by dozens of times, and the application of tdengine in tostar intelligent factory solution

Small T Reading guide ： In tostar's intelligent factory overall solution project , The traditional relational database has been unable to process time series data efficiently , In the load 、 Storage, query and many other aspects have encountered challenges , Finally they chose TDengine To match the application analysis scenarios of industrial sensor data . This article will describe their application TDengine The concrete practice of .

Company profile

Guangdong tostar Technology Co., Ltd （ abbreviation ： Dostar , Stock code ：300607） It is the first robot backbone enterprise landing on the gem in Guangdong Province . Tostar insists “ Make industrial manufacturing better ” Enterprise mission of , By using industrial robots 、 Injection molding machine 、CNC As the core intelligent equipment , And control 、 servo 、 Three core technologies of vision , Build an intelligent hardware platform driven by core technology , Provide manufacturing enterprises with overall solutions for intelligent factories .

Project introduction

In the field of industry , production 、 test 、 A large amount of sensor data with time stamp may be generated in the operation stage , These are typical time series data . Time series data are mainly monitored by various types of real-time data 、 Collected or generated by inspection and analysis equipment , Involving manufacturing 、 Electric power 、 chemical 、 Engineering operation and other industries , Ability to write more and read less 、 Typical characteristics such as very large quantity .

In our business , Traditional relational database （Relational Database） Time series data cannot be processed efficiently , In the load 、 Storage, query and many other aspects have encountered challenges . The main problems can be summarized as follows ：

• Low write throughput ： Single machine write throughput is low , It is difficult to meet the write pressure of tens of millions of sequential data ;
• Storage costs are high ： Poor performance when compressing time series data , It takes up a lot of machine resources ;
• Maintenance costs are high ： Stand alone system , It is necessary to manually divide the warehouse and table on the upper layer , Maintenance costs are high ;
• Poor query performance ： The aggregation and analysis performance of massive real-time data is poor .

In order to better meet the processing requirements of time series data , We decided to conduct a database selection survey , Finally, the time series database is selected （Time-Series Database）TDengine. The fact proved that ,TDengine Write to timing data 、 Storage 、 Indexes 、 Query and other aspects have been specifically optimized , Thus, better data loading is realized 、 data compression 、 Query write performance , It is very suitable for the application analysis scenario of industrial sensor data .

choice TDengine The reason of

Compared with general database ,TDengine Amazing compression , The core reason is that it uses columnar storage , And a two-stage compression strategy is adopted , Different compression algorithms are adopted for different data types , This compression mechanism makes its compression rate far higher than that of other databases .

Besides TDengine It also has extremely high read and write performance , And the speed of reading and writing is little affected by the size of data storage , You should know that once the data volume of the general database exceeds one million , The reading and writing speed will decrease obviously , We did it once before MySQL Test of batch inserting data , The performance gap is obvious , This is also what we will choose in the case of large amount of data storage TDengine One of the important reasons . say concretely ,TDengine The advantages are as follows ：

• The data can be read and written quickly with time stamp , Use SQL Perform database operations , Easy to learn , Support complex queries
• High data compression rate , A large amount of data will not occupy too much storage space , Data can be exported for backup
• Have an exchange community and exchange group , Meet problems and TDengine Other users of , And the official students can also provide timely help

Of course , There is no perfect database , After the application, we summarized two areas for improvement ：

• Cannot use visualization software such as Navicat And so on （TDengine GUI）
• Not yet Windows Server of version , Like our last customer , Only locally Windows Use the program on , Without installing the virtual machine and deploying it to the server , You can't deploy TDengine

But each product is gradually progressing in the step of finding and improving problems , And for our business implementation ,TDengine There is no obvious short board . There is no optimal database , In case of scene matching , We finally adopted TDengine.

Landing practice

• The platform architecture

We push the data collected by the gateway to MQTT,Java After listening, the backend will write TDengine, The data is returned to the front end after the back end queries and processes according to the requirements .

say concretely , The gateway will first read the uplink rules published in the background , After collecting device data , Use the uplink rules to process and calculate the data, and then return the results to the downlink rules module , After background monitoring , Will connect TDengine Create and modify database tables and write data . We have used the cloud platform before Kafka Publish and subscribe data , Now all environments are changed to MQTT 了 .

• Super table and modeling ideas

In the application TDengine when , We set up a pipeline database “iot_platform” It is used to store the data from the gateway , It is convenient for future query . We followed “ A collection point, a table , A kind of data is a super table ” The idea of building tables , Two super tables are designed in practice , One is for storing log Of instruction content “logs” surface , The other one is for storing the contents of other instructions “datas” surface , The data type is basically current and voltage 、 Equipment status, etc . When storing data, we will first judge the data , Then decide which table to store the data in .

Landing effect

After running for a while ,TDengine Query for 、 The writing speed can fully meet our current customer needs , The slowest minute , The fastest way to reach 1 One second ; At most one device a day can Write nearly 100000 pieces of data , Nearly a thousand devices write at the same time also No problem at all , Compared with before , The write speed is increased by tens of times . Query data in months by The time range of units is not too obvious Of Delay , The overall data compression ratio is about 1/10, The amount of data currently generated per day stay Count G about .

• Pipeline data query

Query the flow data in a certain time period , Using query statements ：

SELECT datagettime as ts , ${dataName} as data FROM ${tableName}
 <where>
    uuid = ${uuid}
    <if test="startTime != null ">
        and datagettime > #{startTime}
    </if>
    <if test="endTime != null ">
        and #{endTime} > datagettime
    </if>
    and ${dataName} is not null
 </where>
 limit 0, ${countLimit}

•   The aggregate function calculates the data for a day

Use TDengine Calculate the daily electricity consumption as a function of , Then calculate the monthly and annual data every day , The query statement is ： 

select diff(${dataName}) as data
 from ${tableName}
 where ${dataName} > 0
  and datagettime > #{startTime}
   and #{endTime} >= datagettime

• Calculate the data in a certain period of time

select ${method}(${dataName}) as data
 from ${tableName}
 where uuid = ${uuid}
  and datagettime > #{startTime}
  and #{endTime} > datagettime;

At the end

Under the background of the rapid development of industrial Internet , A large number of equipment sensors and monitoring systems have been put into the industrial production site , The real-time data provided by the two can reflect the equipment status and production progress , Most of the data are real-time data formed in chronological order , These massive real-time data have diverse analysis needs and important reference value .

Hope for the future TDengine It can provide more complex flow computing 、 Query, analysis, monitoring and early warning , It can be used for visual operation and maintenance of products 、 Predictive maintenance 、 Provide data basis for remote intelligent management , Thus reducing personnel 、 Time and other costs , Accelerate the deep integration of industrialization and informatization , Promote the transformation and upgrading of complex heavy equipment manufacturing industry , Produce social and economic benefits .

---

Want to know more TDengine Database Specific details of , Welcome to GitHub View the relevant source code on .