Basic architecture of data Lake

Data lake can be regarded as a new generation of big data infrastructure . In order to better understand the basic architecture of the data Lake , Let's take a look at the evolution of big data infrastructure .

1）  The first stage ： With Hadoop Represents the offline data processing infrastructure . As shown in the figure below ,Hadoop In order to HDFS For core storage , With MapReduce（ abbreviation MR） Batch data processing infrastructure for basic computing models . around HDFS and MR, Produced a series of components , Continuously improve the data processing capacity of the whole big data platform , For example, for online KV Operation of the HBase、 oriented SQL Of HIVE、 Workflow oriented PIG etc. . meanwhile , With the increasing demand for batch processing performance , New computational models have been proposed , Produced Tez、Spark、Presto Wait for the computing engine ,MR The model gradually evolved into DAG Model .

DAG On the one hand, the model , Increase the abstract concurrency of the computing model ： Decompose each calculation process , The tasks are logically segmented according to the aggregation operation points in the calculation process , Tasks are divided into one by one stage, Every stage Can have one or more Task form ,Task It can be executed concurrently , So as to improve the parallel ability of the whole computing process ; On the other hand , In order to reduce the intermediate result of data processing, write file operation ,Spark、Presto And other computing engines try to use the memory of the computing node to cache the data , So as to improve the efficiency of the whole data process and system throughput .

2）  The second stage ：lambda framework . With the continuous change of data processing capacity and processing requirements , More and more users are finding , Batch mode improves performance anyway , It can not meet some processing scenarios with high real-time requirements , Streaming computing engine came into being , for example Storm、Spark Streaming、Flink etc. .

However , With more and more applications online , We found , In fact, batch processing and stream computing are used together , To meet most application needs ; For users , In fact, they don't care what the underlying computing model is , Users want either batch or stream computing , Can return processing results based on a unified data model , therefore Lambda Architecture proposed , As shown in the figure below .

Lambda The core idea of architecture is “ Stream batch integration ”, As shown in the figure above , The entire data flow flows from left to right into the platform . After entering the platform, one is divided into two , Some of them go to batch processing mode , Part of the flow calculation mode . No matter which calculation mode , The final processing results are provided to the application through the service layer , Ensure consistency of access .3）  The third stage ：Kappa framework .Lambda The architecture solves the consistency problem of application reading data , however “ Flow batch separation ” The processing link increases the complexity of research and development . therefore , Some people ask whether we can use a system to solve all problems . At present, the more popular method is based on flow computing . Stream computing is a natural distributed feature , Destined to be more scalable . By increasing the concurrency of flow computing , Increase the of streaming data “ Time window ”, To unify batch processing and stream processing .

Sum up , From traditional hadoop The architecture goes to lambda framework , from lambda The architecture goes to Kappa Evolution of Architecture , The evolution of big data platform infrastructure gradually includes all kinds of data processing capabilities required by applications , The big data platform has gradually evolved into an enterprise / The organization's full data processing platform . In the current enterprise practice , In addition to relational databases, they rely on independent business systems ; The rest of the data , Almost all of them are considered to be incorporated into the big data platform for unified processing . However , Current big data platform infrastructure , Both lock the perspective on storage and Computing , And ignore the asset management of data , This is precisely one of the focuses of data Lake as a new generation of big data infrastructure .

I once read a very interesting article , The following questions have been raised ： Why is a data lake called a data Lake instead of a data river or data sea ？ An interesting answer is ：

• “ The river ” The emphasis is on liquidity ,“ All rivers run into sea ”, After all, the river will flow into the sea , Enterprise level data needs long-term precipitation , So it's called “ lake ” It's called “ The river ” Be appropriate ; meanwhile , The lake water is naturally stratified , Meet different ecosystem requirements , This is related to the construction of a unified data center , The requirements for storing management data are consistent ,“ heat ” The data is on top , Easy to use at any time ; Temperature data 、 Cold data is located in different storage media in the data center , Achieve a balance between data storage capacity and cost .

• Don't cry “ The sea ” The reason is , The sea is boundless , and “ lake ” There are boundaries , This boundary is the enterprise / The business boundaries of the organization ; Therefore, the data Lake needs more data management and permission management capabilities .

• It's called “ lake ” Another important reason is that the data Lake needs fine management , A lack of control 、 Lack of governance data, the lake will eventually degenerate into “ Data swamp ”, Thus, the application cannot effectively access the data , Make the data stored in it worthless .

Evolution of big data infrastructure , In fact, it reflects a little ： In Enterprises / Inside the organization , It has become a consensus that data is an important asset ; In order to make better use of data , Enterprises / Organizations need to be aware of data assets ：

• For long-term as is storage

• Effective management and centralized governance

• Provide multi-mode computing power to meet processing requirements

• And business oriented , Provide a unified data view 、 Data model and data processing results

Data lake is produced under this background , In addition to the various basic capabilities possessed by the big data platform , The data Lake emphasizes the management of data 、 Governance and capitalization capabilities . Fall on the specific implementation , The data Lake needs to include a series of data management components , Include ：

• Data access

• Data relocation

• Data governance

• Quality management

• Asset catalog

• Access control

• task management

• Task arrangement

• Metadata management, etc

As shown in the figure below , A reference architecture of data lake system is given . For a typical data Lake , What it has in common with the big data platform is that it also has the storage and computing power required to process large-scale data , It can provide multi-mode data processing capability ; The enhancement point is that the data Lake provides a more perfect data management capability , It is embodied in ：

• More powerful data access capability . The data access capability is reflected in the definition and management capability of various external heterogeneous data sources , And the ability to extract and migrate relevant data from external data sources , Extract the migrated data, including the metadata of external data sources and the actually stored data .

• More powerful data management capabilities . Management ability can be divided into basic management ability and extended management ability . Basic management capabilities include the management of various metadata 、 Data access control 、 Data asset management , It is necessary for a data lake system , We'll be back “ Data Lake solutions from various manufacturers ” In this section, I believe to discuss the support methods of various manufacturers for basic management capabilities . Expand management capabilities, including task management 、 Process layout and data quality 、 Capabilities related to data governance . Task management and process arrangement are mainly used to manage 、 layout 、 Dispatch 、 Monitor various tasks of processing data in the data lake system , Usually , The data Lake builder will buy / Develop customized data integration or data development subsystem / Modules to provide such capabilities , Customized system / The module can read the relevant metadata of the data lake , To realize the integration with the data lake system . Data quality and data governance are more complex issues , In general , The data lake system will not directly provide relevant functions , But it will open all kinds of interfaces or metadata , For competent enterprises / The organization integrates with existing data governance software or makes customized development .

• Sharable metadata . Various computing engines in the data lake will be deeply integrated with the data in the data lake , The integration is based on the metadata of the data lake . Good data lake system , When the computing engine processes data , The data storage location can be obtained directly from the metadata 、 data format 、 Data patterns 、 Data distribution and other information , Then directly process the data , Without manual / Programming intervention . Further more , A good data lake system can also access and control the data in the data lake , The strength of control can do “ Library table column row ” And so on .

Another point that should be pointed out is , The image above “ Centralized storage ” More is the concentration of business concepts , In essence, I want an enterprise / The data within the organization can be precipitated in a clear and unified place . in fact , The storage of data lake should be a kind of distributed file system that can be expanded on demand , Most data lakes are also recommended in practice S3/OSS/OBS/HDFS The equally distributed system is used as the unified storage of data lake .

We can switch to the data dimension , Look at the data processing method of the data lake from the perspective of the data life cycle , The whole life cycle of data in the data lake is shown in the figure 6 Shown . Theoretically , The data in a well managed data lake will permanently retain the original data , At the same time, the process data will be continuously improved 、 Evolution , To meet the needs of the business .