当前位置:网站首页>About the two architectures of ETL (ETL architecture and ELT architecture)
About the two architectures of ETL (ETL architecture and ELT architecture)
2022-08-04 17:32:00 【Microservice mall technology sharing】
ETL, the abbreviation of English Extract-Transform-Load, is used to describe the process of extracting, transforming, and loading data from the source to the destination.The term ETL is more commonly used in data warehouses, but its objects are not limited to data warehouses.
ETL is an important part of building a data warehouse. The user extracts the required data from the data source, and after data cleaning, finally loads the data into the data warehouse according to the pre-defined data warehouse model.
ETL is mainly reflected in the following aspects in the process of conversion:
- Null value processing: It can capture the null value of the field, load it or replace it with other meaning data, and load it to different target libraries according to the null value of the field.
- Normalized data format: It can realize the definition of field format constraints. For data such as time, value, and character in the data source, the loading format can be customized.
- Split data: Fields can be decomposed according to business requirements.For example, the calling number is 861082585313-8148, the area code and telephone number can be decomposed.
- Verify data correctness: Lookup and split functions can be used to verify data.For example, the calling number is 861082585313-8148. After the area code and the phone number are decomposed, Lookup can be used to return the calling area recorded by the calling gateway or switch for data verification.
- Data replacement: For business factors, invalid data and missing data can be replaced.
- Lookup: Find missing data Lookup implements sub-queries and returns missing fields obtained by other means to ensure field integrity.
- Establish the primary and foreign key constraints of the ETL process: illegal data without dependencies can be replaced or exported to the wrong data file to ensure the loading of the unique records of the primary key.
The advantages of ETL architecture:
- ETL can share the load of the database system (using a separate hardware server)
- Compared with EL-T architecture, ETL can implement more complex data transformation logic
- ETL uses a separate hardware server..
- ETL has nothing to do with the underlying database data store.
ELT
In the ELT architecture, ELT is only responsible for providing a graphical interface to design business rules. The entire process of data processing flows between target and source databases. ELT coordinates related database systems to execute related applications and data.The processing process can be executed either on the source database side or on the target data warehouse side (mainly depending on the architecture design and data attributes of the system).When the ETL process needs to improve efficiency, it can be achieved by tuning the relevant database or changing the server that performs processing.General database vendors will strongly promote this kind of architecture, such as Oracle and Teradata are strongly promoting the ELT architecture.
Advantages of ELT architecture:
- ELT mainly realizes the scalability of the system through the database engine (especially when the data processing process is at night, the resources of the database engine can be fully utilized)
- ELT can keep all data in the database at all times, avoid data loading and exporting, thus ensuring efficiency and improving system monitorability.
- ELT can optimize parallel processing according to the distribution of data, and can optimize disk I/O by utilizing the inherent capabilities of the database.
- The scalability of ELT depends on the scalability of the database engine and its hardware server.
- It is generally not particularly difficult to obtain a 3 to 4 times efficiency improvement in the ETL process through performance tuning of the relevant database.
边栏推荐
- 面试官:可以谈谈乐观锁和悲观锁吗
- 小程序经典案例
- Cholesterol-PEG-Maleimide,CLS-PEG-MAL,胆固醇-聚乙二醇-马来酰亚胺一种修饰性PEG
- 【论文阅读】Decision Transformer: Reinforcement Learning via Sequence Modeling
- shell函数内如何调用另一个函数
- Clearance sword refers to Offer——The sword refers to Offer II 010. and the sub-array of k
- 在VMD上可视化hdf5格式的分子轨迹文件
- 【技术笔记】let 和 var和const的异同
- 【无标题】
- Json的FastJson与Jackson
猜你喜欢
小满nestjs(第一章 介绍nestjs)
Clearance sword refers to Offer——The sword refers to Offer II 010. and the sub-array of k
启动项目(瑞吉外卖)
codeforces每日5题(均1600)-第二十八天
《机器学习理论到应用》电子书免费下载
《中国综合算力指数》《中国算力白皮书》《中国存力白皮书》《中国运力白皮书》在首届算力大会上重磅发出
【技术积累】JS事件循环,Promise,async/await的运行顺序
CF86D Powerful array
NLP未来,路在何方?从学术前沿和业界热点谈起
学习探索-给字体设置前景色
随机推荐
shell函数内如何调用另一个函数
微信jsApi调用失效的相关问题
Digital-intelligent supply chain management system for chemical manufacturing industry: build a smart supply system and empower enterprises to improve production efficiency
【日记】mysql数据库连接池
《机器学习的随机矩阵方法》
租房小程序登顶码云热门
我的大一.
R语言ggpubr包的ggtexttable函数可视化表格数据(直接绘制表格图或者在图像中添加表格数据)、使用ggarrange函数将表格数据和可视化图像组合起来(表格数据在可视化图像下方)
Flutter实战-请求封装(四)之gzip报文压缩
js函数传参是按值传递还是按引用传递?
要有遥不可及的梦想,也要有脚踏实地的本事
JWT主动校验Token是否过期
el-date-picker 设置时间范围
餐饮供应链管理系统
Cron表达式
小满nestjs(第一章 介绍nestjs)
Cholesterol-PEG-Maleimide,CLS-PEG-MAL,胆固醇-聚乙二醇-马来酰亚胺一种修饰性PEG
公司自用的国产API管理神器
树莓派通过API向企业微信推送图文
To eliminate asynchronous callbacks, it has to be async-await