Data warehouse data processing and data flow

One 、 Data flow

Data operation layer （ODS） Data collection

ODS：Operation Data Store Data preparation area , Also known as paste source layer

The data table of the source system of the data warehouse is usually stored intact , This is called ODS layer , It is the source of subsequent data warehouse processing data .

ODS The meaning of , Is to keep the most complete data on site , It is convenient for troubleshooting in some special scenarios .

Click on 、 show 、 Order 、 viewing

Data warehouse （DW）

Data warehouse（ It can be abbreviated as DW perhaps DWH） Data warehouse , It's when there are a lot of databases , In order to further Mining data resources 、 In order to make a decision And the , It is The whole set includes etl、 Dispatch 、 A complete theoretical system including modeling .

Data warehouse , It's a decision-making process for all levels of the enterprise , A strategic set that provides support for all types of data . It's a single data store , Created for analytical reporting and decision support purposes .

DW Data tiering , From bottom to top for DWD,DWB,DWS.

DWD：data warehouse details Format preprocessing

DWD：data warehouse details Detail data layer , It is the isolation layer between business layer and data warehouse .

Mainly for ODS The data layer does some data cleaning and standardized operations .

Preprocessing refers to turning data into semi formatted or formatted data , For example, being HDFS Standard format on , We use the format of string to store .

Deal with some dirty data , For example, the field is missing 、 Format error 、 The statement 、 Null value , wait

• Data cleaning ： Remove null value 、 Dirty data 、 Beyond the limit range

Unified pretreatment

DWB：data warehouse base

DWB：data warehouse base Data base layer , It's storing objective data , Generally used as an intermediate layer , It can be considered as the data layer of a large number of indicators .

DWS：data warehouse service

DWS：data warehouse service Data service layer , be based on DWB Basic data on , Integrate and analyze the service data layer of a certain subject domain , It's usually a wide watch . Used to provide subsequent business queries ,OLAP analysis , Data distribution, etc .

• User behavior , Mild polymerization
• Mainly for ODS/DWD Do some light summary of layer data .

Data service layer / application layer （ADS）

• ADS：applicationData Service Application data services , This layer mainly provides data products and data used for data analysis , Usually stored in ES、mysql And other systems for online systems .

Ensure data consistency , Ensure the speed of demand response

Two 、 Data warehouse data processing flow

Data collection

Log file

Data collection process , ZTE cdn Take the TV bill collection process as an example

DB Data source data collection

DB Data source acquisition , Common are Mysql、mongodb, The main use of datax As a data extraction tool , With CMS Take content data extraction as an example , Introduce DB Data collection process ：

Data warehousing

Data warehousing process , Data collection node Flume After collecting the data , The data warehouse node Flume send out , Data warehouse node flume Write the data to the file system after receiving it , Write after the file is written hdfs, The accounting period data arrives hdfs The dispatching system will be notified , The dispatching system receives signaling to know that certain accounting period data has been warehoused .

Data cleaning ODS

Data cleaning operations , take hdfs File loading to ods The layer data , Complete the original data ID Unification 、 Outliers cleaning 、 Unified data format 、 Associate dimension tables to supplement dimension attributes .

Data is slightly aggregated DWS

DWS Layer data is slightly aggregated , Divide the subject area , Redundant dimension information as much as possible , For improving the speed of downstream calculation 、 Reduce the amount of operation data 、 Simplify business logic 、 Merging computing units and so on has great advantages . During polymerization , Fact tables are often associated with one or more dimension tables , The calculation process generates a lot of intermediate data , Clean up intermediate process data after calculation , Write the result data to the target table .