When tidb and Flink are combined: efficient and easy to use real-time data warehouse

With the rapid development of Internet , There will be more and more kinds of business , The volume of business data will grow , When it reaches a certain scale , The traditional data storage structure can not meet the needs of enterprises , Real time data warehouse becomes a necessary basic service . In terms of dimension Join For example , Data is stored in a business data source in the form of a normal form table , A lot of Join operation , Reduce performance . If it can be completed in the process of data cleaning and importing Join, Then there is no need to analyze again Join, To improve query performance .

Using real-time data warehouse , Enterprises can achieve real-time OLAP analysis 、 Real time data Kanban 、 Real time business monitoring 、 Real time data interface service, etc . But think of real-time data warehouse , Many people's first impression is that the architecture is complex , Difficult to operate and maintain . And thanks to the new version Flink Yes SQL Support for , as well as TiDB HTAP Characteristics of , We explored an efficient 、 Easy-to-use Flink+TiDB Real time data warehouse solution .

This article will first introduce the concept of real-time data warehouse , Then introduce Flink+TiDB The architecture and advantages of real-time data warehouse , Then we give some user scenarios that are already in use , Finally, it is given in docker-compose In the environment Demo, For readers to try .

The concept of real-time data warehouse

The concept of data warehouse is in 90 Age from Bill Inmon Put forward , It refers to a topic oriented 、 Integrated 、 Relatively stable 、 A collection of historical changes , Used to support management decisions . The data warehouse at that time collected data from data sources through message queues , By calculating daily or weekly for use in reports , Also known as offline data warehouse .

Link to the original text :【https://www.infoq.cn/article/IoD228mbbr7wylDEQKkh】. Without the permission of the author , Prohibited reproduced .