One stop monitoring of the software and hardware infrastructure of the whole university, and Suzhou University replaces PostgreSQL with time series database

Small  T  Reading guide ： In addition to data acquisition and online detection of thousands of cameras , Suzhou University and 1500 Multiple switches and 4000 Multiple servers , In the choice of database , It needs to carry so much equipment 24 * 7 While writing high frequency for a long time , Also ensure excellent query efficiency . from PostgreSQL To TDengine, This article shares the business difficulties faced by Jiangsu Zongmu , In the choice of database 、 Experience in application and effectiveness .

Company profile

Suzhou University （Soochow University）, abbreviation “ Su Da ”, Located in the beautiful city of Suzhou, Jiangsu Province , It is a University jointly built by the Ministry of education and the people's Government of Jiangsu Province , Country “ Double top ” Building colleges and Universities , Country “211 engineering ”、“2011 plan ” The first batch of selected universities , Formerly known as 1900 Soochow University founded in , It is the first university in China to hold a modern university discipline system .

Project introduction

As a key university , Suzhou University has a large scale IT Infrastructure and application system , There's a lot of pressure on resources , Network failure affects a large number of users . To ensure the whole school system and network 、 Server hardware 、 Availability of the operating system 、 Reliability and safety , The school must establish a standardized and comprehensive operation and maintenance management system . In this context , We Jiangsu Zongmu And Suzhou University carries out technical cooperation with , Created the smart operation and maintenance management platform of Suzhou University .

The business scenario faces the following difficulties ：

• Resource device type 、 brand 、 Miscellaneous versions , The agreements of various manufacturers differ greatly , Personnel experience cannot be fully covered , The original factory is difficult to respond in time .
• Business applications 、 The relationship between services and resources is complex , Problem location time is much longer than problem solving time .
• Lack of effective tools for ex ante operation and maintenance , Abnormal system network will affect all departments / The teaching work of teachers and students in Colleges and departments , The burden of passive processing is heavy , Cause complaints from the business department .

The storage and query of massive device data bear the brunt

For such a large-scale IT Infrastructure and application systems , Solving the problem itself is not difficult , The difficulty is how to deal with small problems efficiently 、 Prevent big problems scientifically 、 Quickly locate the root cause of the problem . From the perspective of monitoring data , If you want to 24 * 7 Continuous monitoring , So massive device data storage and query are important for Database The pressure will be very great .

Suzhou University has thousands of cameras , The amount of data collected by the optical camera and online detection is already very large , Not to mention 1500 Multiple switches 、4000 Multiple servers , In the choice of database , It needs to carry so much equipment 24 * 7 While writing high frequency for a long time , Also ensure excellent query efficiency .

before , We use PostgreSQL Stand alone version of database , Because it is a relational database （Relational Database）, In the scenario of time series data, when the number reaches 100 million , The query analysis delay will reach several seconds , The compression ratio is not ideal （ There will be a practical comparison later ）, Unable to support the continuous operation of a global integrated operation and maintenance monitoring platform .

Data storage, reading and writing are the foundation of all businesses , Therefore, database selection is particularly important . As early as before , We are aiming at this kind of business to time series database （Time Series Database） We have done sufficient research and actual measurement .

among TDengine As one designed for the Internet of things 、 Car networking 、 Industrial Internet 、 Timing database designed for optimization of operation and maintenance monitoring , It fits the scene very well . Finally, we choose to TDengine Integrated into our self-developed and dedicated to monitoring timing data Argus In the platform https://zmops.com.

Practical application and effect analysis

Actually , from 2020 Year begins , We began to pay attention and contact TDengine 了 , I'm very happy that I finally achieved the right result , In the use of TDengine Yes Argus After a comprehensive upgrade of the platform , Whether it's query efficiency 、 Analyze performance or disk occupancy , All of them have been improved qualitatively .

Will be TDengine After being permanently stored as platform timing data , All functions meet or even exceed our expectations ：

Put it into practice , We are on a 4C 16G The project is implemented on the server with mechanical hard disk specification , Use a single column model to create a table , Create a sub table for each data type indicator , And use a super table to manage . At present , The number of sub tables has reached more than 400000 , It is easy to achieve real-time monitoring of hundreds of thousands of indicators .

At the writing level , Because the acquisition frequency of each device is different , Write about every second 6000 Multiple lines , This is for TDengine There's no pressure , We passed the official test tool taosBenchmark, In their own virtual machine You can write scores of millions of test points per second .

at present , Our data storage cycle （keep 7 God ） For a week ,TDengine The amount of data contained is as follows ： 

The following is for comparison PostgreSQL The amount of data in .

You can see ,TDengine Stored roughly 2 Billion lines of data , Actually occupied storage space, but 2GB.（ notes ：Vnode2 yes log The space occupied by the Library , namely TDengine The built-in database for internal monitoring ）, Compared with PostgreSQL The amount of exceed  200GB Space , Almost negligible .

The same is true for queries , Index query for the performance details page ,PostgreSQL Many queries require a few seconds to return results , and TDengine It's all in milliseconds .

At the end

The present , Because our key business is real-time monitoring , Therefore, there is no such high security priority for historical data , But the follow-up business will involve the resumption of the previous monitoring , We will upgrade to TDengine Cluster version to ensure high availability of data . To make a long story short , Judging from the current application ,TDengine The adaptation is very smooth , It provides a great help for our system .

---

＼＼\\٩( 'ω' )و //／／

First 「TDengine Developer conference 」 Come on. ！！

️ Has been expected by everyone  TDengine 3.0  I will officially meet you  

️ Well known in the field of basic software 、 Live speech by well-known investors  

️ All walks of life TDengine Typical users speak from their own experience  

️ The scene is full of surprises , Exquisite gifts get soft  

️ You leave 「TDengine Developer conference 」 Only one short SQL sentence ！

One step at a time get Admission to the Conference

8 month 13 Japan Hello TDengine！Hello World！

---

  Click on Read the original , Understand the experience TDengine！

This article is from WeChat official account. - TDengine（taosdata_news）.
If there is any infringement , Please contact the [email protected] Delete .
Participation of this paper “OSC Source creation plan ”, You are welcome to join us , share .