[azure data platform] ETL tool (4) - azure data factory debug pipeline

This paper belongs to 【Azure Data Platform】 series .
Continued above ：【Azure Data Platform】ETL Tools (3)——Azure Data Factory Copy from local data source to Azure
This article describes how to Debug ADF The pipe .

reason

In the use of ADF In the process of , Encountered a lot of errors , So I feel like taking it easy first , How to debug Put it in front , So as not to waste too much time looking for problems . Ensure that each function can meet the expected results and performance , It can save a lot of time in large projects .

Debug Will actually perform the operation . Such as copying data , stay debug The data will still be copied , It's just that the pipes are debug The execution log in mode will be output .

Practical pipeline Debug

Start with a simple copy operation , Specific reference ：[【Azure Data Platform】ETL Tools (2)——Azure Data Factory “ Copy the data ” Tools （ Copy in the cloud ）(https://blog.csdn.net/DBA_Huangzj/article/details/122279572)

Click on 【 debugging 】 after , You can see the results in the figure below ：

Click on the bottom right corner 【 View debug run consumption 】：

Move the mouse to 【 Output 】 At the name of , You can see three icons appear ：

Let's open it one by one ：
They are input , Output and details , See the picture below ：

Through the top 4 Messages , You can get a general idea of the operation of the pipeline .

Of course, our goal is to find the problem and fix it , So if the pipeline operation reports an error , What will it be ？ Let's just change the pipeline to report an error . You can see that the icon below has changed , One more. “ error ”：

Click on the error message ：

Check the questions to see ：
ErrorCode=UserErrorSourceBlobNotExist,‘Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The required Blob is missing. ContainerName: https://salab4adf.blob.core.windows.net/adflab, path: ADFTest1.txt.,Source=Microsoft.DataTransfer.ClientLibrary,’‘Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The remote server returned an error: (404) Not Found.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=RequestId:ec89085f-101e-0073-292b-08cbb2000000 Time:Thu, 13 Jan 2022 03:15:46 GMT,’‘Type=System.Net.WebException,Message=The remote server returned an error: (404) Not Found.,Source=Microsoft.WindowsAzure.Storage,’

In fact, I did not specify the file on the source , So it is normal to report that there is no such thing . In addition, we can see the refresh strategy in the following figure ：

If we come to 【 The monitor 】 View in , There is no pipeline information in this part , Because it runs on Debug In mode ：

besides , Debug mode also supports breakpoints , This is the debugging deadline . As shown in the red circle below , When there are multiple steps in a pipeline , You can click on the red circle （ When clicked, it will turn into a red circle ） Pause the next step .

Click again to resume subsequent debugging .

Summary

This article provides a quick demonstration of the common debug, But there is also a common tool “ Data flow ” And its debug, In the next article, we will demonstrate how to use data flow ..