当前位置:网站首页>[azure data platform] ETL tool (4) - azure data factory debug pipeline
[azure data platform] ETL tool (4) - azure data factory debug pipeline
2022-06-13 03:23:00 【Hair dung coating wall】
This paper belongs to 【Azure Data Platform】 series .
Continued above :【Azure Data Platform】ETL Tools (3)——Azure Data Factory Copy from local data source to Azure
This article describes how to Debug ADF The pipe .
reason
In the use of ADF In the process of , Encountered a lot of errors , So I feel like taking it easy first , How to debug Put it in front , So as not to waste too much time looking for problems . Ensure that each function can meet the expected results and performance , It can save a lot of time in large projects .
Debug Will actually perform the operation . Such as copying data , stay debug The data will still be copied , It's just that the pipes are debug The execution log in mode will be output .
Practical pipeline Debug
Start with a simple copy operation , Specific reference :[【Azure Data Platform】ETL Tools (2)——Azure Data Factory “ Copy the data ” Tools ( Copy in the cloud )(https://blog.csdn.net/DBA_Huangzj/article/details/122279572)
Click on 【 debugging 】 after , You can see the results in the figure below :
Click on the bottom right corner 【 View debug run consumption 】:
Move the mouse to 【 Output 】 At the name of , You can see three icons appear :
Let's open it one by one :
They are input , Output and details , See the picture below :
Through the top 4 Messages , You can get a general idea of the operation of the pipeline .
Of course, our goal is to find the problem and fix it , So if the pipeline operation reports an error , What will it be ? Let's just change the pipeline to report an error . You can see that the icon below has changed , One more. “ error ”:
Click on the error message :
Check the questions to see :
ErrorCode=UserErrorSourceBlobNotExist,‘Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The required Blob is missing. ContainerName: https://salab4adf.blob.core.windows.net/adflab, path: ADFTest1.txt.,Source=Microsoft.DataTransfer.ClientLibrary,’‘Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The remote server returned an error: (404) Not Found.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=RequestId:ec89085f-101e-0073-292b-08cbb2000000 Time:Thu, 13 Jan 2022 03:15:46 GMT,’‘Type=System.Net.WebException,Message=The remote server returned an error: (404) Not Found.,Source=Microsoft.WindowsAzure.Storage,’
In fact, I did not specify the file on the source , So it is normal to report that there is no such thing . In addition, we can see the refresh strategy in the following figure :
If we come to 【 The monitor 】 View in , There is no pipeline information in this part , Because it runs on Debug In mode :
besides , Debug mode also supports breakpoints , This is the debugging deadline . As shown in the red circle below , When there are multiple steps in a pipeline , You can click on the red circle ( When clicked, it will turn into a red circle ) Pause the next step .
Click again to resume subsequent debugging .
Summary
This article provides a quick demonstration of the common debug, But there is also a common tool “ Data flow ” And its debug, In the next article, we will demonstrate how to use data flow ..
边栏推荐
- Application framework / capability blueprint
- Using linked list to find set union
- Transaction processing in PDO
- Alibaba cloud OSS access notes
- [JVM series 8] overview of JVM knowledge points
- How to write product requirements documents
- 2022.05.29
- MySQL create user authorization remote access
- Brew tool - "fatal: could not resolve head to a revision" error resolution
- How to become a technological bull -- from the bull man
猜你喜欢
Technology blog, a treasure trove of experience sharing
Open source - campus forum and resource sharing applet
Available types in C #_ Unavailable type_ C double question mark_ C question mark point_ C null is not equal to
This article takes you to learn DDD, basic introduction
Parallel one degree relation query
Vscode liveserver use_ Liveserver startup debugging
Data Governance Series 1: data governance framework [interpretation and analysis]
English grammar_ Mode adverb position
Quickly obtain the attributes of the sub graph root node
C simple understanding - generics
随机推荐
Nuggets new oil: financial knowledge map data modeling and actual sharing
Applet image component long press to identify supported codes
Age anxiety? How to view the 35 year old programmer career crisis?
English语法_方式副词-位置
Es and kibana deployment and setup
Figure data * reconstruction subgraph
2021-08-30 distributed cluster
Querywrapper constructor method
Filters in PHP
Implode and explode in golang
MASA Auth - SSO与Identity设计
Review notes of RS data communication foundation STP
Golang picks up: why do we need generics
Beginner development process_ Project development FAQs
【同步功能】2.0.16-19 版本都有同步功能修复的更新,但未解决问题
最近最少使用缓存(来源力扣)
English grammar_ Frequency adverb
Spoon database insert table operation
2-year experience summary to tell you how to do a good job in project management
On the limit problem of compound function