Here comes a white paper to uncover the technology behind Clickhouse, a node with 10000 bytes!

ClickHouse Open source from 2016 year , With outstanding advantages in performance , In the field of analytical database, the development can be described as a rising tide . at present , Many large head manufacturers at home and abroad are deeply using ClickHouse technology .

In terms of performance ,ClickHouse stay OLAP The performance under the scenario exceeds that of similar products several times , It allows the system to start with a sub second delay PB Level raw data generation report , The server throughput is up to hundreds of millions of lines per second .

But will ClickHouse Introduction into enterprise production environment , There are still problems . About the pit of landing practice , Not all teams in the industry need to step on it by themselves , Not all teams can afford such costs , What we need to do is learn enough experience , And choose self-study 、 Purchase and other more practical solutions .

At this point , ByteDance is undoubtedly a very representative domestic enterprise ： Byte beat from 2017 In, it was put into use on a large scale ClickHouse; As its deep user , It has the largest in China ClickHouse colony .

at present , Byte beating internal ClickHouse The total number of nodes exceeds 1.8 m , The total amount of data managed exceeds 700PB, The largest single cluster deployment scale is about 2400 More than nodes .

At present , ByteDance has been customized for five years ClickHouse, Precipitate into ByteHouse, Officially provide services through volcanic engine .

From adopting and transforming open source products , Go to the online commercial version for external service , This is a very difficult road , At the same time, it also makes the practical thinking and experience more valuable .

lately , Volcano engine ByteHouse union InfoQ Publishing white papers 《 from ClickHouse To ByteHouse》, In depth introduction of ByteDance 10000 nodes ClickHouse Technical implementation behind , This volume of white paper is roughly divided into four chapters ：

1. ClickHouse Introduction to ;
2. ClickHouse Typical scenario ;
3. For ClickHouse,ByteHouse Technology optimization thinking ;
4. ByteHouse Design and evolution ideas .

among ,《 from ClickHouse To ByteHouse》 From chapter three , Emphasis on the ByteHouse The optimization idea of .

at present ,ByteHouse Yes ClickHouse Many upgrades and optimizations have been made , This time I chose ByteHouse Yes ClickHouse Three very important aspects of optimization and upgrading are expanded in detail ：

1. Self research table engine ;
2. Query optimizer ;
3. Elastic and expandable .

In the self research table engine module , Even though ClickHouse Provide MergeTree Family、Memory、File、Interface And dozens of different table engines , But in the actual use of bytes , It is obvious that some table engines are not enough to meet the business needs , So the corresponding optimization is carried out .

among , Emphasis on the 了 HaMergeTree 、HaUniqueMergeTree、HaKafka Three table engines .

Excerpts from the white paper ：HaMergeTree Replica collaboration principle

In the query optimizer module ,ByteHouse Yes Optimizer For more than one year , Comprehensively upgrade product capabilities , The white paper details ByteHouse Transformation and optimization function on query optimizer .

In pursuit of ultimate performance ,ClickHouse It adopts a strong coupling architecture between computing and storage nodes , The capacity cannot be expanded separately according to their actual needs , And the problem that the data cannot be automatically redistributed after the node is expanded ClickHouse Expansion brings a lot of trouble in operation and maintenance .

ByteHouse In improving and optimizing ClickHouse In the process of , It also focuses on the adjustment based on this architecture , Such as ByteHouse Decoupling in storage and computation , Realize flexible and scalable technology optimization scheme .

Excerpts from the white paper ： Computing storage separation architecture

besides ,《 from ClickHouse To ByteHouse》 Give out advertisements 、 Finance 、 Practice cases of the three major industries of industrial Internet , These belong to OLAP A typical application industry , And from the perspective of technology and enterprise landing, it gives the current situation of enterprises in OLAP Three core concerns of data engine selection .

Click to read the original text to download the white paper