当前位置:网站首页>[azure data platform] ETL tool (4) - azure data factory debug pipeline

[azure data platform] ETL tool (4) - azure data factory debug pipeline

2022-06-13 03:23:00 Hair dung coating wall

This paper belongs to 【Azure Data Platform】 series .
Continued above :【Azure Data Platform】ETL Tools (3)——Azure Data Factory Copy from local data source to Azure
This article describes how to Debug ADF The pipe .

reason

In the use of ADF In the process of , Encountered a lot of errors , So I feel like taking it easy first , How to debug Put it in front , So as not to waste too much time looking for problems . Ensure that each function can meet the expected results and performance , It can save a lot of time in large projects .

Debug Will actually perform the operation . Such as copying data , stay debug The data will still be copied , It's just that the pipes are debug The execution log in mode will be output .

Practical pipeline Debug

Start with a simple copy operation , Specific reference :[【Azure Data Platform】ETL Tools (2)——Azure Data Factory “ Copy the data ” Tools ( Copy in the cloud )(https://blog.csdn.net/DBA_Huangzj/article/details/122279572)
 Insert picture description here
Click on 【 debugging 】 after , You can see the results in the figure below :
 Insert picture description here
Click on the bottom right corner 【 View debug run consumption 】:
 Insert picture description here

Move the mouse to 【 Output 】 At the name of , You can see three icons appear :
 Insert picture description here
Let's open it one by one :
They are input , Output and details , See the picture below :

 Insert picture description here
 Insert picture description here
 Insert picture description here
Through the top 4 Messages , You can get a general idea of the operation of the pipeline .

Of course, our goal is to find the problem and fix it , So if the pipeline operation reports an error , What will it be ? Let's just change the pipeline to report an error . You can see that the icon below has changed , One more. “ error ”:

 Insert picture description here
 Insert picture description here
Click on the error message :

 Insert picture description here
Check the questions to see :
ErrorCode=UserErrorSourceBlobNotExist,‘Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The required Blob is missing. ContainerName: https://salab4adf.blob.core.windows.net/adflab, path: ADFTest1.txt.,Source=Microsoft.DataTransfer.ClientLibrary,’‘Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The remote server returned an error: (404) Not Found.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=RequestId:ec89085f-101e-0073-292b-08cbb2000000 Time:Thu, 13 Jan 2022 03:15:46 GMT,’‘Type=System.Net.WebException,Message=The remote server returned an error: (404) Not Found.,Source=Microsoft.WindowsAzure.Storage,’

In fact, I did not specify the file on the source , So it is normal to report that there is no such thing . In addition, we can see the refresh strategy in the following figure :

 Insert picture description here
If we come to 【 The monitor 】 View in , There is no pipeline information in this part , Because it runs on Debug In mode :

 Insert picture description here
 Insert picture description here
besides , Debug mode also supports breakpoints , This is the debugging deadline . As shown in the red circle below , When there are multiple steps in a pipeline , You can click on the red circle ( When clicked, it will turn into a red circle ) Pause the next step .

 Insert picture description here
Click again to resume subsequent debugging .

Summary

This article provides a quick demonstration of the common debug, But there is also a common tool “ Data flow ” And its debug, In the next article, we will demonstrate how to use data flow ..

原网站

版权声明
本文为[Hair dung coating wall]所创,转载请带上原文链接,感谢
https://yzsam.com/2022/02/202202280531217473.html