The integrated real-time HTAP database stonedb, how to replace MySQL and achieve nearly 100 times the improvement of analysis performance

as everyone knows ,MySQL It's the most popular in the world OLTP One of the databases , By 2022 In, its market share in the entire database industry reached 43.04%（ Data sources ：Slintel Website ）. Many enterprises apply various business systems to MySQL On . However , With the continuous increase of enterprise data , Besides reading and writing online business logic , Databases also face increasingly complex analytical business needs , such as BI report form 、 visualization 、 Big data applications, etc . and MySQL Native architecture （ Based on the flow iterator model Volcano Iterator Execution engine for , No use of modern multicore CPU Parallel processing power , Storage engine for row storage ） stay AP There are natural defects in the scene . In this case , To make up for MySQL Of AP Ability defect , The industry revolves around MySQL Many solutions have been made . Mainly around MySQL Built heterogeneous HTAP Database system .

What is? HTAP ？ stay 2014 year ,Gartner given HTAP The strict definition of ： Its purpose is to break , Between transactional and analytical loads “ barrier ”, Enable the system to support more “ data ” Flow between two systems , And based on these data “ Real time service ” The decision .

In the form of traditional architecture , In order to solve the problem of simultaneous processing TP Load and AP Load problem , Usually one set TP System plus a set AP System approach ,TP and AP Through between ETL To meet the real-time requirements of the business , This is also the current industry structure HTAP The mainstream solution of .

01

The industry revolves around MySQL build HTAP Mainstream programs

Let's take a look at the mainstream of the industry based on MySQL Of HTAP Solution .

1. MySQL + Hadoop

With the help of Hadoop system , take MySQL Business data , adopt ETL The tool is synchronized to the open source big data system （ Such as Hive,Hadoop,Spark etc. ） Build a data warehouse , Then do data analysis based on the data warehouse .

2. MySQL +   Data Lake

With the help of data Lake platform , adopt ETL Tools will MySQL Data synchronization to data Lake , Then based on the data Lake data 、 report form 、BI Wait for analysis .

3. MySQL + ClickHouse/Greenplum

adopt ETL And other data migration tools will MySQL Data migration to ClickHouse/Greenplum Do analysis .

ClickHouse The official in the 20 The community version was released in the second half of the year MaterializeMySQL engine , Can be ClickHouse As MySQL A slave Library of synchronizes master node data , except ETL Tools , There are also direct in the industry ClickHouse As a MySQL The scheme of directly mounting from the Library .

4. Based on multiple copies of Divergent Design

For example, compatibility MySQL Agreed TiDB, In a Raft Group On one of the copies , Through self-developed in-line storage (TiFlash) To respond to complexity AP Inquire about , And pass TiDB Intelligent routing function to automatically select data sources , Implement a set of distributed systems HTAP Database system , What we do in the distributed field is better .

The picture is from TiDB Official website

The problems of the above scheme

Some of the above HTAP Solution , Although it is the mainstream in the industry , But there are still some problems , Include ：

1. The system architecture is too heavy , The complexity of operation and maintenance is high ;
2. TP Data is passed through ETL Mode to AP In the system , The data delay is large , It is difficult to meet the real-time requirements of services for analysis ;
3. Heterogeneous database combination , Technically, two sets of database systems need to be maintained , It involves many technology stacks , Higher requirements for technical personnel ;
4. NewSQL System , Various compatibility adaptations are required , Adaptation work will be more complex , The requirements for technicians are also relatively high .
5. So , We brought in HTAP The solution to the problem ：StoneDB, An open source integrated real-time HTAP database .

02

StoneDB： Fully compatible with MySQL Ecological integrated row and column hybrid storage HTAP database

StoneDB Is a new open source based on native MySQL Integrated real-time HTAP database , Using the domestic first integrated row and column hybrid storage architecture , Achieve high-performance real-time at a very low cost HTAP.

StoneDB Adopt integrated row column hybrid storage , With distributed multiple copies Divergent Design Different approaches , It is a scheme of using row column mixed storage in the same database instance , Highly integrated , The complexity of operation and maintenance is low , Better user experience . The original intention of this architecture is to use a set of database , Simultaneous solution TP and AP The problem of , A lightweight , More elegant , More convenient . At present, foreign manufacturers such as Oracle / SQL Server / DB2 And others have adopted similar schemes , But they are not open source .

StoneDB Overview of integrated architecture （v1.0）

StoneDB Plug in access to MySQL, By inquiring / Write interface and MySQL server Layers interact , The main features of the current integrated architecture are ：

• Organize data by column storage , Combined with efficient compression algorithm , bring StoneDB While obtaining high performance, it also has the advantage of storage cost .
• Based on knowledge grid （Knowledge Grid） Approximate query and parallel processing mechanism , bring StoneDB When dealing with massive data and complex queries , It can minimize irrelevant data IO.
• Using histogram , Data block bitmap and many other statistical information to further accelerate the speed of query processing .
• Adopt Column-at-a-time An execution engine for columnar storage , And further improve the efficiency of the execution engine .
• Provide high-speed data loading capability .

Let's take a look StoneDB Architecture design ：

Architecture design ： Data organization form

stay StoneDB in , Data is organized in columns . This form of data organization , Friendly to all kinds of compression algorithms , According to the type of each column 、 Data and other factors, choose the appropriate efficient compression algorithm , To save IO and Memory Purpose of resources . It also has the following advantages ：

• Cache Line friendly .
• During query , The operations for each column are performed concurrently , Finally, aggregate the complete recordset in memory .
• When querying ad hoc , Just scan specific columns , No need to consume IO Resources to read the values of other columns .
• No need to maintain index , Support ad hoc query of any column combination .
• It can provide knowledge-based grid capabilities ,   Improve data search efficiency .

Architecture design ： Column based data compression

As mentioned above , Data is organized in columns , All records in the column have the same type , You can choose the corresponding efficient compression algorithm according to the data type , because ：

• The probability of duplicate values in the column is high , The compression effect is obvious .
• The data node size is fixed , Maximize compression performance and efficiency .
• Compress according to a specific numeric type （int,float,date/time,string etc. ）.

StoneDB It can support up to 20+ An adaptive compression algorithm , At present, it mainly uses ：

• PPM
• LZ4
• B2
• Delta wait

Architecture design ： Data organization structure and knowledge grid

StoneDB The query processing part of is shown in the above figure . Query processing is the brain of the whole database , Query optimization algorithm is good or bad , It directly affects the query efficiency .

Let's talk about data organization structure and knowledge grid . When I introduced the architecture , We also mentioned the column organization of data , And in each column , The data is divided into finer grained data blocks . The advantages of this method are ：

1. Physical data is divided into fixed data blocks , For storage , Usually called ：Data Node, Usually it is ：128KB, The system is convenient for IO Optimization of efficiency . meanwhile , It can also provide block based （Block） Efficient compression / encryption algorithm .
2. Knowledge grids can be query optimizers , Implementation and compression algorithm support . for example ： Query based on knowledge grid , The optimizer will use the knowledge grid to decide what to grab Data Node To perform data operations .

Let's explain the related concepts , The following data nodes 、 Metadata nodes are all logical concepts ：

• Data nodes （Data Node,DN）： The data block size is fixed （ Typical values 128KB）, Optimize IO efficiency , Provide block based （Block） Efficient compression / encryption algorithm .
• Knowledge grid （Knowledge Grid,KG）： For metadata storage .
• Metadata node （Metadata Node,MDN）： Metadata information describing the data node . By the knowledge node （Knowledge Node,KN） form , For the query optimizer , Support plan execution and compression algorithm .

Architecture design - Inquire about ： Knowledge grid （ Knowlegde Grid ） overview

Architecture design - Inquire about ： be based on Knowlegde Grid The optimizer for

As shown in the figure above ： First, the query optimizer optimizes based on knowledge grid , Prune the data it needs to process , Its strategy is ： For data nodes that meet query conditions , That is, related data nodes , Adopt the strategy of directly reading and returning ; For uncertain data nodes , Decompress first , Then we are processing based on query conditions , Finally, the processing result is returned ; For data nodes that are completely unrelated to query conditions , Then ignore it directly .

Then rough set is carried out based on the information in the knowledge grid （Rough Set） structure , And determine the data nodes required for this request . be based on KN and MD , Determine what the query involves DN Node collection , And will DN Node classification . When executing a plan build , Will completely avoid non relevance DN, Only read and decompress associations DN, Decide whether to read uncertain according to specific conditions DN. If the result of the query request can be directly from the metadata node （MDN） Produced in （ for example count,max,min Wait for the operation ）, Then directly return the data in the metadata node , No need to access physical data files .

Architecture design - Inquire about ： Processing flow

For example, for a query request , adopt KG（ Knowledge grid ） Can be determined 3 Associations DN and 1 An uncertainty DN. If , This request contains aggregate functions . At this time, we only need to decompress the uncertainty DN, And calculate the aggregate value , combining 3 Associations DN in MDN The final result can be obtained by the statistical value on . If , This request needs to return specific data , So regardless of relevance DN Or uncertainty DN, You need to read data blocks and decompress them in parallel , To get the final result set .

such as , To perform a select * from xx where seller = 86, The internal execution process is as follows ：

1. Implementation plan optimization and implementation ：

• Based on knowledge grid Cost-based Optimize
• IO Thread pool maintenance
• Memory allocation and management

2. SMP Support （ Concurrent query ）
3. Vectorization execution

03

Fully compatible with MySQL Ecological StoneDB Integrated HTAP Advantages of the system

Fully compatible with MySQL Of StoneDB Integrated HTAP database . It has the following characteristics ：

• Fully compatible with MySQL. Whether it's grammar or ecology MySQL Users can seamlessly switch to StoneDB.
• Business 、 Integration of analysis . There is no need to ETL, Transactional data is synchronized to the analysis engine in real time . It enables users to obtain real-time business analysis results .
• Fully open source .
• Compare with MySQL Provide 10-100 Times AP Ability . Hundred million level multi meter correlation rapid response , There is no need to wait for the result of the decision .
• 10 Times the import speed . because AP scenario , The amount of analysis data is huge , Efficient import speed , It can bring you a good user experience .
• 1/10 Of TCO cost ,StoneDB Have efficient compression algorithm , Seamless business migration capability , And its simple architecture , Can bring to users TCO Reduction .

04

StoneDB 2.0 Will bring a new architecture

What is introduced above is StoneDB Stand alone version of 1.0 framework . although StoneDB The disk based inventory engine is AP The performance under the scene has been excellent , But after all, it is a disk based solution . We know ,IO And memory are extremely valuable resources in the field of database , Think to further improve StoneDB Performance of , At the same time, in order to reduce AP The load is for TP Effect of load . In the future, we will 2.0 A version similar to HEATWAVE A new architecture of column storage engine based on memory computing . This version will be based on MySQL 8.0 structure , Based on this engine, we will realize AP Full memory calculation of load .

About 2.0 More information is welcome StoneDB Official website ：https://stonedb.io

meanwhile ,StoneDB stay 6 month 29 The official open source has been announced on the th . If you are interested , You can view it through the link below StoneDB Source code 、 Read the document , I look forward to your contribution ！

StoneDB Open source warehouse

https://github.com/stoneatom/stonedb

‍

‍ Li Hao ‍

‍

StoneDB chief architect 、StoneDB PMC

I worked in Huawei 、 Iqiyi 、 Peking University is engaged in the design of the core architecture of the database kernel . exceed 10 Experience in database kernel development , Good at query engine , Execution engine , Large scale parallel processing and other technologies . It has dozens of database invention patents , The author of 《PostgreSQL Inquiry engine source code technology analysis 》.

Gao riyao

StoneDB PMC、HTAP Kernel architect

Graduated from Huazhong University of science and technology , Like to study the mainstream database architecture and source code .8 Years of experience in database kernel development , Once engaged in distributed database CirroData 、RadonDB and TDengine Kernel Research and development , I am now in the position of StoneDB Kernel architects and StoneDB project PMC.

END

StoneDB The complete video of the open source press conference has been uploaded B standing

https://www.bilibili.com/video/BV1U3411F76U

https://www.bilibili.com/video/BV1gS4y1H7NK

https://www.bilibili.com/video/BV19f4y1Z7JB

Scan to add StoneDB Open source community

What's new in the community

Technical experts answer questions

Participate in community building

This article is from WeChat official account. - StoneDB（StoneDB2021）.
If there is any infringement , Please contact the [email protected] Delete .
Participation of this paper “OSC Source creation plan ”, You are welcome to join us , share .