[azure data platform] ETL tool (5) -- use azure data factory data stream to convert data

This paper belongs to 【Azure Data Platform】 series .
Continued above ：【Azure Data Platform】ETL Tools (4)——Azure Data Factory Debug The Conduit
This article describes how to ADF Data stream transformation data .

The previous articles mainly focused on “ Copy the data ” Use of tools .

Data conversion

This demonstration Azure SQL DB After calculating the value of a table in , Write to a new table . So , First create a... In the pipe data flow：

Create a new data stream ：

Add source ：

Here, create a new source to distinguish it from the previous one ：

Create a new link service to ADF Be able to access Azure SQL DB：

Open source , Configure specific source options ：

Select the table to be processed , It can be downloaded from “ Preview data ” Check whether it is connected ：

In the source options , You can define full table operations , Or use specific SQL/ Stored procedures to process data , You can also define the batch size and isolation level .

stay 【 Optimize 】 In the options , You can also perform additional processing on partitioned tables .

And then click 【 Check 】 Options , View the table definition of the source table ：

Then click... In the lower right corner of the graphical interface 【+】, As shown in the figure , Configure a data processing operation , Select aggregation here .

Select the columns to be processed in the following order , In the last step 【 Open the expression builder 】 In, we select the columns and operations to be processed ：

choice ListPrice Column , And then use countAll Function to calculate ：

Next, add another process 【 Receiver 】, Means the target source , For the convenience of the period , Choose the same one here Azure SQL DB The library of , Just use different tables ：

For the convenience of , First, create the target table ：

Enter this table in the destination and view the data , It can be seen that the current table is empty .

When everything is configured , Publish this pipeline ：

After publishing , Use the debugging function , The execution of the pipeline can be triggered manually . You can see from the figure below that the calculation results have been obtained ：

But the previous ones are still in debug mode , To actually run , Need to jump back Data Flow The pipe in which it is located ：

Then click debug , As mentioned above , The debug button actually executes the pipeline , So the data is actually calculated and written to the target ：

In the data table, you can see that it has actually run ：

summary

So far, , We have realized data conversion with data stream . During this presentation , I learned one “ New skills ”, Is simply in Data flow Debugging , The data will not be written to the target , It needs to be executed on the same pipeline .