With a total data volume of more than trillions of lines, Yuxi cigarette factory can easily deal with it by correctly selecting the timing database

Company profile

The predecessor of Yuxi cigarette factory was founded in 1956 Yuxi Tobacco Redrying Factory in .1991 year , Yuxi cigarette factory has become the only cigarette factory in the national tobacco industry “ National first class enterprise ”,“ hongta ” Won the national gold medal .

After decades of development , Yuxi cigarette factory has become the main cigarette manufacturer of Hongta Group .

By 2021 end of the year , Yuxi cigarette factory has a complete threshing and redrying line 、 Silk production line 、 Stem wire production line 、 Coiling unit 、 Packaging unit 、 Filter rod forming unit , Annual cigarette production capacity 220  More than 10000 cases , The plant covers an area of 134.04 Thousands of square meters , Thousands of on-the-job employees . factory 22 Benchmarking indicators increased year on year 18 term , among 4 Item ranks first in the industry 、13 Items rank among the top ten in the industry , The number of the top ten indicators ranks second among cigarette factories in the industry 、 It ranks first among the cigarette factories of Yunnan China Tobacco .

Business background

The main production business of Yuxi cigarette factory involves redrying 、 There are three technological processes of silk making and wrapping . The tobacco redrying link of cigarette raw materials is the first technological process of cigarette production , The tobacco leaves are picked from the tobacco field , Transfer to the tobacco farmer for initial roasting , Send to the redrying workshop according to the purchase of the commercial company , According to the preset formula , Process and store tobacco leaves . Cigarette silk making technology is the main processing technology of cigarette production , Silk processing is based on the physical and chemical characteristics of raw tobacco , According to certain processing procedures , The process of making tobacco into qualified cut tobacco through a variety of processing procedures . When cut tobacco 、 Auxiliary materials and wrapping equipment are ready , Production will enter the winding process , It mainly includes wind wire feeding 、 Filter rod conveying 、 Auxiliary materials in and out of warehouse 、 Crimping 、 packing 、 And finished products in and out of the warehouse . Corresponding to the above process , The production business involves the redrying workshop 、 There are three workshops in the silk making workshop and the rolling car room . The redrying workshop mainly separates the raw tobacco into leaves and stems , Then dry them separately ; The silk workshop mainly feeds, perfumes and cuts Chunhua's tobacco leaves ; The cigarette rolling workshop is mainly used to connect tobacco rolls into sticks , Then pack them separately 、 Bars and pieces .

The silk making process directly determines the sensory quality and cut tobacco quality of finished cigarettes , It is an important part of cigarette processing , Improving the quality assurance ability of the silk making process directly determines the internal quality of cigarettes , It further affects the recognition of the brand in the market . therefore , Around process manufacturing capabilities , Improve ability around process quality , Focus on digital management capabilities , Build stability 、 Powerful 、 Reliable silk making data , Effectively ensure production operation and process improvement , It's digital transformation 、 The foundation of realizing intelligent production line .

The silk workshop itself has many production equipment and complex processes , A silk thread has 8 Process sections ,260 Multi class equipment , Every device 11 A classification , total 3 ten thousand 6 More than a thousand data acquisition points . Mainly related to temperature 、 humidity 、 moisture 、 weight 、 Air volume 、 Traffic 、 level 、 Frequency and other data , These data are generated with high frequency , And it's real-time , Each monitoring point generates data every second . The amount of real-time data generated is also conceivable , The consumption of disk storage space is also very serious . Besides , Most of these data are generated by equipment , At the same time, it is also collected and recorded in real time according to the timestamp , Therefore, there will be many insertion operations , For real-time 、 High frequency 、 Massive writing and storage are required . From these two points , Time series database （Time Series Database） It's an ideal choice .

Our project this time is 「 Intelligent manufacturing pilot of high-quality line 」, The database stores the timing data generated on the silk production line . Previously, we used the foreign traditional time series database , Although it supports OPC Direct data collection , However, there are some deficiencies in stability and security , Moreover, the query speed is slow 、 High operation and maintenance cost .

Switch back to TDengine after , With the super table mechanism designed from the characteristics of time series data , And the storage engine divided by time , Even the total data volume of a single meter at the level of 100 billion ,TDengine It can also be divided physically and logically , Directly operate the data file of the corresponding time period , Discard invalid search consumption , This solves our biggest problem . In addition, based on efficient column compression design , The storage cost of data has also been significantly reduced .

practical experience

To promote the construction of high-end and high-end boutique lines , Improve the internal quality of high-end cigarette processing , The pilot project of intelligent manufacturing of high-quality product line is to improve the manufacturing capacity of the production process of high-quality product line 、 Aim at the technological level , Make full use of new technology 、 Cutting edge information means , Create a whole process perception 、 The whole industry chain is coordinated 、 Real time decision-making process 、IOT Integrated cutting-edge production and manufacturing workshops , Ensure the stability and consistency of quality within and between batches in the whole production process . The whole process perception architecture is shown in the figure below ：

Among them, the data link of high-quality silk thread is like this ：PLC Equipment passing OPC Server With OPC UA Protocol collection data , Then send the data to Kafka/Flink colony , among Flink Play the role of message buffer .Flink Of job Running on big data platform Yarn On the computing platform , It contains Kafka client、 Time stamp 、 window 、 Data filtering and aggregation operators , The architecture is shown in the following figure .

Last , Data is passed through TAOSC The module writes data into our deployed TDengine Enterprise Edition 6 In the node cluster .

We have 8 Process sections 、 Each process section 30  Multiple devices 、 Each device has 1000  Multiple data acquisition point types , In conclusion , The equipment collection points share 36526 individual , Data types include floating point numbers 、 Boolean type 、 Integers and strings . come very naturally , We need to build databases and tables according to this system .

Considering that good modeling is used smoothly in the future TDengine The foundation of , So we have also put a lot of thought on this . At that time, there were three schemes ：

• Create a super table to store the data of all collection points , One collection point, one sub table . Super table adopts single column model , The type of collected value is string .

• advantage ： Getting a MES Business computing needs n When measuring the full amount of data at each measuring point , You can use one SQL Find out , No need join

• shortcoming ： Boolean 、 Floating point numbers are also stored as strings , The compression ratio will be affected , And pre calculation cannot be used , You may also need to perform column to row operations on the result set

• Create multiple super tables , Single column mode , Create a super table for each data type , One collection point, one sub table , The collection point uses the corresponding type super table according to the corresponding data type .

• advantage ： The increase or decrease of acquisition points can be flexibly matched , And the numerical collected values have pre aggregation and optimization

• shortcoming ： Get a MES Business computing needs n When measuring the full amount of data at each measuring point , You need to query from three super tables , Consider using join, But there will still be the work of converting the result set column to row

• For each  MES Business computing tasks （ For example, product quality calculation ） The involved collection points build a super table （ It may also be an ordinary watch ）, Adopt multi column mode .

• advantage ：MES The queries involved in the business can be found directly from the corresponding table by time period , And the data types of different measuring points can be preserved , Precomputation optimization will also be preserved

• shortcoming ： The table structure is fixed , Different MES Super tables between business systems may use the same indicators , The data of these indicators will be stored repeatedly . If MES Business computing needs change , You need to modify the table structure , When writing, all acquisition point fields are required to be written at the same time

Final , We decided to take option two 、 Combination of scheme 3 . Data sub database storage , Full raw data of an inventory single column mode , A library will MES The collection point of business calculation is stored separately , Another high-frequency data of inventory storage vibration .

The main benefits of this design are as follows ：

    1. Single column mode can ensure the flexibility of other businesses to use data , and MES The combination of indicators required by the business is almost unchanged , Targeted table building can ensure the convenience of calculation ;

    2.  Super tables can be labeled with data , The single column mode is convenient for the management of logarithmic sampling points , Capable of collecting the number of silk making points  BOM  The structure extends to  TDengine  in , Thus, it can realize fast query and filtering based on tags ;

    3.  Thanks to the  TDengine Powerful compression capability , The extra data will not bring a lot of costs ;

    4.  According to the importance of the data and the acquisition frequency, it is stored in sub databases , Separate maintenance of each acquisition and storage program , Mutual interference , Effectively ensure storage stability , The storage failure of one library will not affect other libraries .

Landing process & Application display

Final , According to the above scheme, we divide the database and build the table .

With yjpzsa library （ Silk production volume database ） As an example , We put 36526 A collection point , It is designed as 7 A super watch . Follow a data type, a super table , One data collection point, one sub table , The table name uses the data point code , And the Super Table label inherits the data collection point table BOM The principle of structure , Build up “ One yard to the end ” Data acquisition system .

Respectively yjpzsa_bigint,yjpzsa_float,yjpzsa_nchar,yjpzsa_bool,yjpzsa_binary,yjpzsa_int,yjpzsa_timestamp.

It is convenient to query a process section according to the data label 、 Data acquisition point of a certain equipment , The figure below shows the data of data acquisition points under the actual measurement classification of the dryer in the silk drying process section .

storage

There is a single field 16 KB The big table of , There are also tables with only numeric data in the deposit receipt column ：

Our main super tables have stored hundreds of billions of rows of data , The total data volume easily exceeds trillion lines . Even in three copies （ Three copies of data are stored to ensure high availability ） Under the configuration of , At present, the total space occupied is only  1TB about .

We collect the high-frequency vibration signal of the motor , The signal frequency is 10 Khz,TDengine It supports us to store 10000 pieces of vibration acceleration data per second into the database , For time domain and frequency domain analysis , As shown in the figure below .

In terms of inquiry

1. Yes, Super Table voltage_prop Table execution period query .

select * from yjpzsa_phm_opc.voltage_prop where ts >'2022-06-22 10:00:000' and ts <'2022-06-22 15:10:000';

2. Query of single equipment , You can find sub tables through Super Table filtering , You can also find it directly through the sub table .

The above is what we use TDengine The landing process and application display of .

At the end

From model selection test to ground use to now , We and TDengine The team has always maintained full cooperation and communication . Helping TDengine While perfecting oneself , We also get better and better products and services , Finally, the current use effect is achieved , Both of us are satisfied with this . I also wish TDengine More and more good .

---

＼＼\\٩( 'ω' )و //／／

First 「TDengine Developer conference 」 Come on. ！！

️ Has been expected by everyone  TDengine 3.0  I will officially meet you  

️ Well known in the field of basic software 、 Live speech by well-known investors  

️ All walks of life TDengine Typical users speak from their own experience  

️ The scene is full of surprises , Exquisite gifts get soft  

️ Click the picture above to learn more about the Conference , There are surprises at the end of reading , Quick man gains in one step ️ Admission to the Conference  

You leave 「TDengine Developer conference 」 Only one short SQL sentence ！

8 month 13 Japan Hello TDengine！Hello World！

---

  Click on Read the original , Understand the experience TDengine！

This article is from WeChat official account. - TDengine（taosdata_news）.
If there is any infringement , Please contact the [email protected] Delete .
Participation of this paper “OSC Source creation plan ”, You are welcome to join us , share .