当前位置:网站首页>The writing speed is increased by dozens of times, and the application of tdengine in tostar intelligent factory solution
The writing speed is increased by dozens of times, and the application of tdengine in tostar intelligent factory solution
2022-07-05 09:43:00 【Tdengine】
Small T Reading guide : In tostar's intelligent factory overall solution project , The traditional relational database has been unable to process time series data efficiently , In the load 、 Storage, query and many other aspects have encountered challenges , Finally they chose TDengine To match the application analysis scenarios of industrial sensor data . This article will describe their application TDengine The concrete practice of .
Company profile
Guangdong tostar Technology Co., Ltd ( abbreviation : Dostar , Stock code :300607) It is the first robot backbone enterprise landing on the gem in Guangdong Province . Tostar insists “ Make industrial manufacturing better ” Enterprise mission of , By using industrial robots 、 Injection molding machine 、CNC As the core intelligent equipment , And control 、 servo 、 Three core technologies of vision , Build an intelligent hardware platform driven by core technology , Provide manufacturing enterprises with overall solutions for intelligent factories .
Project introduction
In the field of industry , production 、 test 、 A large amount of sensor data with time stamp may be generated in the operation stage , These are typical time series data . Time series data are mainly monitored by various types of real-time data 、 Collected or generated by inspection and analysis equipment , Involving manufacturing 、 Electric power 、 chemical 、 Engineering operation and other industries , Ability to write more and read less 、 Typical characteristics such as very large quantity .
In our business , Traditional relational database (Relational Database) Time series data cannot be processed efficiently , In the load 、 Storage, query and many other aspects have encountered challenges . The main problems can be summarized as follows :
- Low write throughput : Single machine write throughput is low , It is difficult to meet the write pressure of tens of millions of sequential data ;
- Storage costs are high : Poor performance when compressing time series data , It takes up a lot of machine resources ;
- Maintenance costs are high : Stand alone system , It is necessary to manually divide the warehouse and table on the upper layer , Maintenance costs are high ;
- Poor query performance : The aggregation and analysis performance of massive real-time data is poor .
In order to better meet the processing requirements of time series data , We decided to conduct a database selection survey , Finally, the time series database is selected (Time-Series Database)TDengine. The fact proved that ,TDengine Write to timing data 、 Storage 、 Indexes 、 Query and other aspects have been specifically optimized , Thus, better data loading is realized 、 data compression 、 Query write performance , It is very suitable for the application analysis scenario of industrial sensor data .
choice TDengine The reason of
Compared with general database ,TDengine Amazing compression , The core reason is that it uses columnar storage , And a two-stage compression strategy is adopted , Different compression algorithms are adopted for different data types , This compression mechanism makes its compression rate far higher than that of other databases .
Besides TDengine It also has extremely high read and write performance , And the speed of reading and writing is little affected by the size of data storage , You should know that once the data volume of the general database exceeds one million , The reading and writing speed will decrease obviously , We did it once before MySQL Test of batch inserting data , The performance gap is obvious , This is also what we will choose in the case of large amount of data storage TDengine One of the important reasons . say concretely ,TDengine The advantages are as follows :
- The data can be read and written quickly with time stamp , Use SQL Perform database operations , Easy to learn , Support complex queries
- High data compression rate , A large amount of data will not occupy too much storage space , Data can be exported for backup
- Have an exchange community and exchange group , Meet problems and TDengine Other users of , And the official students can also provide timely help
Of course , There is no perfect database , After the application, we summarized two areas for improvement :
- Cannot use visualization software such as Navicat And so on (TDengine GUI)
- Not yet Windows Server of version , Like our last customer , Only locally Windows Use the program on , Without installing the virtual machine and deploying it to the server , You can't deploy TDengine
But each product is gradually progressing in the step of finding and improving problems , And for our business implementation ,TDengine There is no obvious short board . There is no optimal database , In case of scene matching , We finally adopted TDengine.
Landing practice
- The platform architecture
We push the data collected by the gateway to MQTT,Java After listening, the backend will write TDengine, The data is returned to the front end after the back end queries and processes according to the requirements .
say concretely , The gateway will first read the uplink rules published in the background , After collecting device data , Use the uplink rules to process and calculate the data, and then return the results to the downlink rules module , After background monitoring , Will connect TDengine Create and modify database tables and write data . We have used the cloud platform before Kafka Publish and subscribe data , Now all environments are changed to MQTT 了 .
- Super table and modeling ideas
In the application TDengine when , We set up a pipeline database “iot_platform” It is used to store the data from the gateway , It is convenient for future query . We followed “ A collection point, a table , A kind of data is a super table ” The idea of building tables , Two super tables are designed in practice , One is for storing log Of instruction content “logs” surface , The other one is for storing the contents of other instructions “datas” surface , The data type is basically current and voltage 、 Equipment status, etc . When storing data, we will first judge the data , Then decide which table to store the data in .
Landing effect
After running for a while ,TDengine Query for 、 The writing speed can fully meet our current customer needs , The slowest minute , The fastest way to reach 1 One second ; At most one device a day can Write nearly 100000 pieces of data , Nearly a thousand devices write at the same time also No problem at all , Compared with before , The write speed is increased by tens of times . Query data in months by The time range of units is not too obvious Of Delay , The overall data compression ratio is about 1/10, The amount of data currently generated per day stay Count G about .
- Pipeline data query
Query the flow data in a certain time period , Using query statements :
SELECT datagettime as ts , ${dataName} as data FROM ${tableName}
<where>
uuid = ${uuid}
<if test="startTime != null ">
and datagettime > #{startTime}
</if>
<if test="endTime != null ">
and #{endTime} > datagettime
</if>
and ${dataName} is not null
</where>
limit 0, ${countLimit}
- The aggregate function calculates the data for a day
Use TDengine Calculate the daily electricity consumption as a function of , Then calculate the monthly and annual data every day , The query statement is :
select diff(${dataName}) as data
from ${tableName}
where ${dataName} > 0
and datagettime > #{startTime}
and #{endTime} >= datagettime
- Calculate the data in a certain period of time
select ${method}(${dataName}) as data
from ${tableName}
where uuid = ${uuid}
and datagettime > #{startTime}
and #{endTime} > datagettime;
At the end
Under the background of the rapid development of industrial Internet , A large number of equipment sensors and monitoring systems have been put into the industrial production site , The real-time data provided by the two can reflect the equipment status and production progress , Most of the data are real-time data formed in chronological order , These massive real-time data have diverse analysis needs and important reference value .
Hope for the future TDengine It can provide more complex flow computing 、 Query, analysis, monitoring and early warning , It can be used for visual operation and maintenance of products 、 Predictive maintenance 、 Provide data basis for remote intelligent management , Thus reducing personnel 、 Time and other costs , Accelerate the deep integration of industrialization and informatization , Promote the transformation and upgrading of complex heavy equipment manufacturing industry , Produce social and economic benefits .
Want to know more TDengine Database Specific details of , Welcome to GitHub View the relevant source code on .
边栏推荐
- OpenGL - Coordinate Systems
- OpenGL - Model Loading
- Why do offline stores need cashier software?
- [ManageEngine] how to make good use of the report function of OpManager
- Kotlin introductory notes (I) kotlin variables and non variables
- About getfragmentmanager () and getchildfragmentmanager ()
- VS Code问题:长行的长度可通过 “editor.maxTokenizationLineLength“ 进行配置
- 从“化学家”到开发者,从甲骨文到TDengine,我人生的两次重要抉择
- Alibaba's ten-year test brings you into the world of APP testing
- LeetCode 31. Next spread
猜你喜欢
Understanding of smt32h7 series DMA and DMAMUX
Applet data attribute method
Figure neural network + comparative learning, where to go next?
干货整理!ERP在制造业的发展趋势如何,看这一篇就够了
基于宽表的数据建模应用
【组队 PK 赛】本周任务已开启 | 答题挑战,夯实商品详情知识
Why does everyone want to do e-commerce? How much do you know about the advantages of online shopping malls?
Kotlin introductory notes (VIII) collection and traversal
LeetCode 31. 下一个排列
LeetCode 503. 下一个更大元素 II
随机推荐
Wxml template syntax
Principle and performance analysis of lepton lossless compression
[listening for an attribute in the array]
[team PK competition] the task of this week has been opened | question answering challenge to consolidate the knowledge of commodity details
TDengine 离线升级流程
The popularity of B2B2C continues to rise. What are the benefits of enterprises doing multi-user mall system?
MySQL installation configuration and creation of databases and tables
c语言指针深入理解
LeetCode 503. 下一个更大元素 II
观测云与 TDengine 达成深度合作,优化企业上云体验
揭秘百度智能测试在测试自动执行领域实践
Thermometer based on STM32 single chip microcomputer (with face detection)
百度APP 基于Pipeline as Code的持续集成实践
TDengine可通过数据同步工具 DataX读写
【对象数组a与对象数组b取出id不同元素赋值给新的数组】
【阅读笔记】图对比学习 GNN+CL
一文详解图对比学习(GNN+CL)的一般流程和最新研究趋势
What are the advantages of the live teaching system to improve learning quickly?
Go 语言使用 MySQL 的常见故障分析和应对方法
LeetCode 31. 下一个排列