author ： Xu Haijun
Small T Reading guide ： Gechuang Dongzhi Technology Co., Ltd. was established in 2018 year , Incubated in China 500 Strong enterprise TCL, It is a well-known industrial Internet platform service provider in China . The company relies on TCL The group 40 Annual industrial scene and manufacturing gene precipitation , be based on “ Facing the industrial site ” R & D direction and “ Connect 、 synergy 、 share ” The concept of development , Deep integration of artificial intelligence 、 big data 、 Cloud computing 、 Internet of things and other cutting-edge technologies , Digital energy management solutions for smart factories and manufacturing parks , It can be widely used in many vertical industries , Operation monitoring 、 Energy efficiency management 、 Intelligent analysis 、 Mocha ITOM 、 Energy procurement 、 Dozens of functional modules such as carbon emission management , Realize the closed loop of perfect energy management value , Establish digital energy & Carbon measurement system , Intelligent energy use and carbon management system , Build a carbon tracking enterprise 、 Carbon measurement 、 Carbon Management 、 One stop solution for carbon neutralization .
TCL Air conditioning energy management platform for factory electricity 、 water 、 Natural gas 、 Oil and other indicators are collected in real time 、 Dynamic monitoring , And in the dimension of time 、 manufacturer 、 workshop 、 Type of production line 、 The production line 、 Equipment and other dimensions 、 Energy saving measurement 、 charging 、 Costing 、 Conduct industry benchmarking 、 Generate analysis report , Realize the refined management of enterprise energy , Promote energy conservation and consumption reduction ; Real time monitoring of users' energy safety data , Send alarm information to safety management personnel in time , Guide them to carry out hidden danger treatment , Provide information services to help enterprises implement energy safety and energy management .
G-Things It is the Internet of things platform of our application intelligent platform product family , Let's take a look at the data flow on this platform .
Industrial equipment will report data to the platform data access gateway , Then the access gateway is responsible for parsing the message , And filter out illegal data messages , Then distribute the data to Apache Kafka Message middleware , Real time processing by the platform 、 Consumption of persistent Services , The persistence service will write the latest data Redis, And persist the data to the temporal database . For platform storage architecture , Design support OpenTSDB、ClickHouse、TDengine Wait for the timing database to switch , We should choose the type according to the actual situation of the project .
One 、 Storage scheme selection
Let's take a look at the characteristics of energy industry data ：
- Timing of data ： Devices constantly generate data , These data will be reported to the platform with a timestamp
- The data flow is stable ： The reporting frequency is relatively stable , The acquisition frequency is 30 Seconds at a time
- Data is of numeric type ： It's some cumulative usage 、 electric current 、 electric current 、 Data like pressure
- There is no change in the data ： Data is the data recorded in the acquisition table at a certain time , There is no need to update or delete the report ;
- Data aggregation and analysis are based on time dimension 、 Spatial dimension ： The time dimension has years 、 month 、 Zhou 、 Japan 、 when , The shortest 15 Every minute , There are manufacturers in the space dimension 、 workshop 、 Type of production line 、 The production line 、 Equipment etc.
- Large amount of data ： Press a factory 4 Ten thousand meter calculation , Every time 30 One second of data , The data collected in one day will exceed 1 Billion bars
Based on the data characteristics of the energy industry , We need to support on the platform OpenTSDB、ClickHouse and TDengine this 3 Choose among the three temporal database storage engines . Here is a comparison ：
- OpenTSDB： rely on HBase、HDFS and ZooKeeper And so on , High requirements for hardware resources 、 The high cost , When the query time span is large , Performance plummets , In addition, the support for aggregation analysis and query is poor .
- ClickHouse： In data storage 、 Cross time period query and data aggregation analysis query , Meet all our selection requirements , But the cost of operation and maintenance is too high , Extension is too complex , More resources are used .
- TDengine： In data storage 、 Data analysis and query meet our needs , And the cluster version is also open source , Support horizontal scaling , Less occupied resources , Under the condition of limited resources of customers , Is the best choice for storage engine .
After the above comparison , We chose to TDengine As your own storage engine .
Two 、TDengine Database modeling
TDengine There are two unique innovations , One is “ A data collection point, a table ”, One is “ Supertable ”. So when designing the data model , It is necessary to consider how the business model maps to super tables and specific tables .
First, let's look at the super table .
according to TDengine The characteristics of the database , We put the meter 、 Water meter 、 LPG meter 、 Oxygen meter , Create a super table for each type of data collection point . Take creating an electricity meter as an example ：
Let's look at ordinary watches .
Each data collection point needs to create a table independently . As with standard relational databases , A table has a name ,Schema, But beyond that , You can also have one or more labels .
3、 ... and 、 Summary of practical application
Half a year since the project was launched , Keep running smoothly .
stay TCL Air conditioning energy management project , The hardware resources we use are significantly reduced , Same as ClickHouse Cluster as storage TCL Comparison of Internet of things platforms in electronic industry , The data scale of the two projects is similar ,TCL The database server of air conditioning energy management project has been reduced by half .
We realize the visual monitoring and real-time query of the operation parameters of each system in the form of configuration view , A complete digital energy management solution has been realized , It improves the efficiency of operation management decision . By electricity 、 water 、 oil 、 Natural gas usage analysis , Optimize the energy structure of the factory and save energy and reduce consumption , Reduce energy consumption 5% about , It is estimated to be TCL The cost of air conditioning has been reduced by tens of millions of Yuan .
In the use of TDengine In the process of , We have also summed up some experience .
such as ,TDengine Provide a cache of the last record , adopt last_row Function can quickly obtain real-time data . The platform is in use TDengine Before , use Redis To cache the latest data , And a lot of writing Redis It will have a certain impact on the performance , Now use TDengine cache Instead of redis cache , Can solve this problem perfectly .
in addition , In use ,TDengine Our technical experts also provide a lot of support .TDengine We have our own wechat group for technical exchange , Very active , Many technical questions have been answered by Taosi data experts in person , Very fast response .
Author's brief introduction ：
Xu Haijun , He is currently responsible TCL AIoT Development of the overall technical architecture of the platform , Mainly engaged in industrial Internet 、 Big data system development , Once participated in the construction of Guangdong model industrial Internet platform .
Want to know more TDengine Specific details of , Welcome to GitHub View the relevant source code on .
