[azure data platform] ETL tool (1) -- Introduction to azure data factory

This paper belongs to 【Azure Data Platform】 series .

Due to the need of work , Record daily learning practice into a series of articles , Hope to be helpful to readers in need .

About the use of data , In addition to storage （ database ）, And collection （ETL/ELT etc. ） And analysis . This series focuses on Azure Of ETL Tools Azure Data Factory（ADF） in .

Preface

As a data platform solution , Data collection is an essential part of it . When selecting a data collection tool , There are several points to consider （ Suppose your environment is cloud based , there Azure）：

1. Whether to use the... Provided by the platform ？ Many vendors provide local and cloud based tools . How to choose needs many considerations , Especially the established strategy of the enterprise .
2. timeliness ： Tradition ETL Tools are usually T+1, The delay is serious , For today's environment , A little inappropriate . However, the details still need to be determined according to the project .
3. cost ： Of course , Free tools are not recommended for commercial use .

ADF brief introduction

According to the strategy of the enterprise , My project is based on ADF by ETL Tools , Then the next series of articles will mainly introduce and demonstrate it . Then we must first make clear ADF This ETL What tools can do ？

The official introduction will not be posted , Here are the most important highlights ： Replicate and transfer data in local and cloud environments , This is one of the pain points in the process of data synchronization in many hybrid cloud environments .

SQL Server Users should have heard SSIS（SQL Server Integration Services）, ADF Not a cloud version SSIS, It is not very strongly bound to SQL Server The characteristics of , It supports a wider range of data transmission .ADF Although it can be executed on the cloud SSIS package , But it's just using ADF Scalability and SSIS Advanced ETL It's just a function .

ADF And others ETL The difference between the tools

Based on the following points , In the use of Azure Platform time ,ADF Is a relatively good candidate ETL One of the tools ：

• Can run SSIS package , This has been used maturely for the local environment SSIS In terms of projects , Can reduce migration costs .
• Automatic scaling based on load , And completely by PaaS Product hosting .
• The running interval can be reduced to every minute .
• Through a gateway （gateway） Seamlessly connect your local environment to Azure Cloud platform .
• Ability to handle large-scale data .
• It can be connected with other computing services HDI Connect and collaborate on real big data .

Of course , If it is just an ordinary ETL Tools , It is difficult to compete with other majors in the market ETL Manufacturers compete . therefore ADF There must be some other advantages . In addition to support for mainstream data sources , Customizable code logic , Monitoring and other common functions ,ADF Also integrates Azure DevOps and GitHub Of CI/CD function .
In Microsoft's official parlance ,ADF It simplifies mixed data integration ： Azure Data Factory

Let's briefly introduce ADF The components of , The content mainly comes from ADF file ：

• Pipeline： The Conduit , Its function is mainly to encapsulate functions in the form of collections （ Mission ）. And pass the parameter value to the next task .
• Mapping Data Flow： Mapping data streams , Visual transfer logic （ETL Medium T）. Run on fully managed Spark On the cluster , And meet the load demand in the form of horizontal expansion .
• Activity： Activities , A single execution step in a pipeline .ADF Currently, three types of activities are supported ： Data mobility activities 、 Data conversion activities and control activities .
• Dataset： Data sets , The data source involved in the pipeline output input .
• Linked Service： Link services , Similar to the connection string , Connecting data and computing resources （ such as HDI）.
• Trigger： trigger , A processing unit that defines when a pipeline will be executed .
• Control flow： control flow , Control the business process of the pipeline , Set up branches , Cycle, etc .

More introductions will be provided in the follow-up demonstration and use . Let's prepare the environment first .

Demonstration of practical operation

open Azure Portal, Search service “ Data factory ” Or the English version “Data Factory”,

“ basic ” Options page ： Fill in the necessary information

It is suggested to choose V2, because V1 Will be discarded .

“Git To configure ” Options page ： This step does not need , So choose to configure later Git：

“ The Internet ” Options page ： If you haven't figured out what the options are for , Leave the default options and click next ：

“ senior ” Options page , As an introduction , Do not select key encryption ：

“ Mark ” It is more suitable for the management of a large number of resources in enterprise applications , The default is also kept here ：

Notice here the... Below “ Download automation templates ”, If in the enterprise , Use a defined template , It can save configuration time and the risk of omission caused by manual operation .

Click on “ establish ” After the button , Probably 1 It will be deployed in about minutes ：

Now we have one ADF service . The next article will demonstrate the simplest ADF Use .
【Azure Data Platform】ETL Tools (2)——Azure Data Factory “ Copy the data ” Tools （ Copy in the cloud ）