About the two architectures of ETL (ETL architecture and ELT architecture)

ETL, the abbreviation of English Extract-Transform-Load, is used to describe the process of extracting, transforming, and loading data from the source to the destination.The term ETL is more commonly used in data warehouses, but its objects are not limited to data warehouses.

ETL is an important part of building a data warehouse. The user extracts the required data from the data source, and after data cleaning, finally loads the data into the data warehouse according to the pre-defined data warehouse model.

ETL is mainly reflected in the following aspects in the process of conversion:

1. Null value processing: It can capture the null value of the field, load it or replace it with other meaning data, and load it to different target libraries according to the null value of the field.
2. Normalized data format: It can realize the definition of field format constraints. For data such as time, value, and character in the data source, the loading format can be customized.
3. Split data: Fields can be decomposed according to business requirements.For example, the calling number is 861082585313-8148, the area code and telephone number can be decomposed.
4. Verify data correctness: Lookup and split functions can be used to verify data.For example, the calling number is 861082585313-8148. After the area code and the phone number are decomposed, Lookup can be used to return the calling area recorded by the calling gateway or switch for data verification.
5. Data replacement: For business factors, invalid data and missing data can be replaced.
6. Lookup: Find missing data Lookup implements sub-queries and returns missing fields obtained by other means to ensure field integrity.
7. Establish the primary and foreign key constraints of the ETL process: illegal data without dependencies can be replaced or exported to the wrong data file to ensure the loading of the unique records of the primary key.

The advantages of ETL architecture:

1. ETL can share the load of the database system (using a separate hardware server)
2. Compared with EL-T architecture, ETL can implement more complex data transformation logic
3. ETL uses a separate hardware server..
4. ETL has nothing to do with the underlying database data store.

ELT

In the ELT architecture, ELT is only responsible for providing a graphical interface to design business rules. The entire process of data processing flows between target and source databases. ELT coordinates related database systems to execute related applications and data.The processing process can be executed either on the source database side or on the target data warehouse side (mainly depending on the architecture design and data attributes of the system).When the ETL process needs to improve efficiency, it can be achieved by tuning the relevant database or changing the server that performs processing.General database vendors will strongly promote this kind of architecture, such as Oracle and Teradata are strongly promoting the ELT architecture.

Advantages of ELT architecture:

1. ELT mainly realizes the scalability of the system through the database engine (especially when the data processing process is at night, the resources of the database engine can be fully utilized)
2. ELT can keep all data in the database at all times, avoid data loading and exporting, thus ensuring efficiency and improving system monitorability.
3. ELT can optimize parallel processing according to the distribution of data, and can optimize disk I/O by utilizing the inherent capabilities of the database.
4. The scalability of ELT depends on the scalability of the database engine and its hardware server.
5. It is generally not particularly difficult to obtain a 3 to 4 times efficiency improvement in the ETL process through performance tuning of the relevant database.