当前位置:网站首页>Azure data factory (3) integrate azure Devops to realize CI / CD
Azure data factory (3) integrate azure Devops to realize CI / CD
2020-11-06 01:28:00 【itread01】
One , introduction
Due to the length of the previous section , Can't share Azure Data Factory Continuous integration of , Continue to release . Today we will focus on the use of Azure DevOps Pipeline waive , Automatic continuous integration , And has automatically released , Will Azure Data Factory Deploy to multiple environments .
In fact, we need not be surprised , The deployment here is not so mysterious , We're in ADF Medium master After the branch is released , In fact, it will ADF Package all configuration information in , Compile to adf_master Below the branch , If you look at the released code carefully , It's easy to find out , It's all some ARM Template resources . With what I'm building right now demo For example
This is what someone is asking , What is ARM Templates ? Here's your overview :
For Azure Deployment of some infrastructure resources on , You can do this by using infrastructure as code , But to automate deployment , Define the infrastructure resources to be deployed in your code . Make these infrastructure source codes part of the project , Like application code . Through code management tools , Manage it , Easy to deploy , Create by code 、 Delete resource . and ARM Templates are a form of infrastructure or code ( The other is Terraform), The template is a JavaScript Object representation (JSON) Archives . The template uses declarative syntax , Allows you to specify what to deploy , Instead of writing a series of programming commands to create content . In this template , Specify the resources to deploy and the properties of those resources .
Talk about people , Namely ARM The template describes the cloud resources that we need to deploy and the arguments required by its resources , For example, by using ARM It's not a VM, So ARM It is described in the template VM Resources and their establishment VM The necessary arguments required .
go back to ADF in , That is to say, we finally passed master Branch out to adf_master Branch code , It's just a bunch of descriptions ADF Resources and ADF Configured properties and arguments . What we're talking about today is through ARM To achieve ADF Of UAT,PRO Deployment of the environment .
-------------------- I am the dividing line --------------------
1,Azure Data Factory( One ) Introduction
2,Azure Data Factory( Two ) Copy data
3,Azure Data Factory( 3、 ... and ) Integrate Azure Devops Realize CI/CD
Two , Text
1, establish ADF Of UAT The environment
We use resource names +UAT To simulate the test environment , Here's a new one called "ADF-CnBateBlogWeb-UAT" Of Azure Data Factory and Two UAT Environmental Blob Storage
”cnbateblogwebaccount1dev“ As UAT Sources of environmental information , We are also for UAT Environment building is called “resformfolder” The container of
“cnbateblogwebaccount2dev” As UAT The target source of the environment , We are also for UAT Environment building is called “restofolder” The container of
2, Initialize pre and post deployment scripts
In fact , We're deploying ADF Even other project code , It's all about putting AFD The trigger stops , When deployment is complete , We have to restart ADF Triggers configured in . Microsoft has helped us provide scripts that we can use before and after deployment
1 param 2 ( 3 [parameter(Mandatory = $false)] [String] $armTemplate, 4 [parameter(Mandatory = $false)] [String] $ResourceGroupName, 5 [parameter(Mandatory = $false)] [String] $DataFactoryName, 6 [parameter(Mandatory = $false)] [Bool] $predeployment=$true, 7 [parameter(Mandatory = $false)] [Bool] $deleteDeployment=$false 8 ) 9 10 function getPipelineDependencies { 11 param([System.Object] $activity) 12 if ($activity.Pipeline) { 13 return @($activity.Pipeline.ReferenceName) 14 } elseif ($activity.Activities) { 15 $result = @() 16 $activity.Activities | ForEach-Object{ $result += getPipelineDependencies -activity $_ } 17 return $result 18 } elseif ($activity.ifFalseActivities -or $activity.ifTrueActivities) { 19 $result = @() 20 $activity.ifFalseActivities | Where-Object {$_ -ne $null} | ForEach-Object{ $result += getPipelineDependencies -activity $_ } 21 $activity.ifTrueActivities | Where-Object {$_ -ne $null} | ForEach-Object{ $result += getPipelineDependencies -activity $_ } 22 return $result 23 } elseif ($activity.defaultActivities) { 24 $result = @() 25 $activity.defaultActivities | ForEach-Object{ $result += getPipelineDependencies -activity $_ } 26 if ($activity.cases) { 27 $activity.cases | ForEach-Object{ $_.activities } | ForEach-Object{$result += getPipelineDependencies -activity $_ } 28 } 29 return $result 30 } else { 31 return @() 32 } 33 } 34 35 function pipelineSortUtil { 36 param([Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline]$pipeline, 37 [Hashtable] $pipelineNameResourceDict, 38 [Hashtable] $visited, 39 [System.Collections.Stack] $sortedList) 40 if ($visited[$pipeline.Name] -eq $true) { 41 return; 42 } 43 $visited[$pipeline.Name] = $true; 44 $pipeline.Activities | ForEach-Object{ getPipelineDependencies -activity $_ -pipelineNameResourceDict $pipelineNameResourceDict} | ForEach-Object{ 45 pipelineSortUtil -pipeline $pipelineNameResourceDict[$_] -pipelineNameResourceDict $pipelineNameResourceDict -visited $visited -sortedList $sortedList 46 } 47 $sortedList.Push($pipeline) 48 49 } 50 51 function Get-SortedPipelines { 52 param( 53 [string] $DataFactoryName, 54 [string] $ResourceGroupName 55 ) 56 $pipelines = Get-AzDataFactoryV2Pipeline -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 57 $ppDict = @{} 58 $visited = @{} 59 $stack = new-object System.Collections.Stack 60 $pipelines | ForEach-Object{ $ppDict[$_.Name] = $_ } 61 $pipelines | ForEach-Object{ pipelineSortUtil -pipeline $_ -pipelineNameResourceDict $ppDict -visited $visited -sortedList $stack } 62 $sortedList = new-object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline] 63 64 while ($stack.Count -gt 0) { 65 $sortedList.Add($stack.Pop()) 66 } 67 $sortedList 68 } 69 70 function triggerSortUtil { 71 param([Microsoft.Azure.Commands.DataFactoryV2.Models.PSTrigger]$trigger, 72 [Hashtable] $triggerNameResourceDict, 73 [Hashtable] $visited, 74 [System.Collections.Stack] $sortedList) 75 if ($visited[$trigger.Name] -eq $true) { 76 return; 77 } 78 $visited[$trigger.Name] = $true; 79 if ($trigger.Properties.DependsOn) { 80 $trigger.Properties.DependsOn | Where-Object {$_ -and $_.ReferenceTrigger} | ForEach-Object{ 81 triggerSortUtil -trigger $triggerNameResourceDict[$_.ReferenceTrigger.ReferenceName] -triggerNameResourceDict $triggerNameResourceDict -visited $visited -sortedList $sortedList 82 } 83 } 84 $sortedList.Push($trigger) 85 } 86 87 function Get-SortedTriggers { 88 param( 89 [string] $DataFactoryName, 90 [string] $ResourceGroupName 91 ) 92 $triggers = Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName 93 $triggerDict = @{} 94 $visited = @{} 95 $stack = new-object System.Collections.Stack 96 $triggers | ForEach-Object{ $triggerDict[$_.Name] = $_ } 97 $triggers | ForEach-Object{ triggerSortUtil -trigger $_ -triggerNameResourceDict $triggerDict -visited $visited -sortedList $stack } 98 $sortedList = new-object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSTrigger] 99 100 while ($stack.Count -gt 0) { 101 $sortedList.Add($stack.Pop()) 102 } 103 $sortedList 104 } 105 106 function Get-SortedLinkedServices { 107 param( 108 [string] $DataFactoryName, 109 [string] $ResourceGroupName 110 ) 111 $linkedServices = Get-AzDataFactoryV2LinkedService -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName 112 $LinkedServiceHasDependencies = @('HDInsightLinkedService', 'HDInsightOnDemandLinkedService', 'AzureBatchLinkedService') 113 $Akv = 'AzureKeyVaultLinkedService' 114 $HighOrderList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService] 115 $RegularList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService] 116 $AkvList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService] 117 118 $linkedServices | ForEach-Object { 119 if ($_.Properties.GetType().Name -in $LinkedServiceHasDependencies) { 120 $HighOrderList.Add($_) 121 } 122 elseif ($_.Properties.GetType().Name -eq $Akv) { 123 $AkvList.Add($_) 124 } 125 else { 126 $RegularList.Add($_) 127 } 128 } 129 130 $SortedList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]($HighOrderList.Count + $RegularList.Count + $AkvList.Count) 131 $SortedList.AddRange($HighOrderList) 132 $SortedList.AddRange($RegularList) 133 $SortedList.AddRange($AkvList) 134 $SortedList 135 } 136 137 $templateJson = Get-Content $armTemplate | ConvertFrom-Json 138 $resources = $templateJson.resources 139 140 #Triggers 141 Write-Host "Getting triggers" 142 $triggersInTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/triggers" } 143 $triggerNamesInTemplate = $triggersInTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)} 144 145 $triggersDeployed = Get-SortedTriggers -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 146 147 $triggersToStop = $triggersDeployed | Where-Object { $triggerNamesInTemplate -contains $_.Name } | ForEach-Object { 148 New-Object PSObject -Property @{ 149 Name = $_.Name 150 TriggerType = $_.Properties.GetType().Name 151 } 152 } 153 $triggersToDelete = $triggersDeployed | Where-Object { $triggerNamesInTemplate -notcontains $_.Name } | ForEach-Object { 154 New-Object PSObject -Property @{ 155 Name = $_.Name 156 TriggerType = $_.Properties.GetType().Name 157 } 158 } 159 $triggersToStart = $triggersInTemplate | Where-Object { $_.properties.runtimeState -eq "Started" -and ($_.properties.pipelines.Count -gt 0 -or $_.properties.pipeline.pipelineReference -ne $null)} | ForEach-Object { 160 New-Object PSObject -Property @{ 161 Name = $_.name.Substring(37, $_.name.Length-40) 162 TriggerType = $_.Properties.type 163 } 164 } 165 166 if ($predeployment -eq $true) { 167 #Stop all triggers 168 Write-Host "Stopping deployed triggers`n" 169 $triggersToStop | ForEach-Object { 170 if ($_.TriggerType -eq "BlobEventsTrigger") { 171 Write-Host "Unsubscribing" $_.Name "from events" 172 $status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 173 while ($status.Status -ne "Disabled"){ 174 Start-Sleep -s 15 175 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 176 } 177 } 178 Write-Host "Stopping trigger" $_.Name 179 Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force 180 } 181 } 182 else { 183 #Deleted resources 184 #pipelines 185 Write-Host "Getting pipelines" 186 $pipelinesADF = Get-SortedPipelines -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 187 $pipelinesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/pipelines" } 188 $pipelinesNames = $pipelinesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)} 189 $deletedpipelines = $pipelinesADF | Where-Object { $pipelinesNames -notcontains $_.Name } 190 #dataflows 191 $dataflowsADF = Get-AzDataFactoryV2DataFlow -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 192 $dataflowsTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/dataflows" } 193 $dataflowsNames = $dataflowsTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40) } 194 $deleteddataflow = $dataflowsADF | Where-Object { $dataflowsNames -notcontains $_.Name } 195 #datasets 196 Write-Host "Getting datasets" 197 $datasetsADF = Get-AzDataFactoryV2Dataset -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 198 $datasetsTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/datasets" } 199 $datasetsNames = $datasetsTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40) } 200 $deleteddataset = $datasetsADF | Where-Object { $datasetsNames -notcontains $_.Name } 201 #linkedservices 202 Write-Host "Getting linked services" 203 $linkedservicesADF = Get-SortedLinkedServices -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 204 $linkedservicesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/linkedservices" } 205 $linkedservicesNames = $linkedservicesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)} 206 $deletedlinkedservices = $linkedservicesADF | Where-Object { $linkedservicesNames -notcontains $_.Name } 207 #Integrationruntimes 208 Write-Host "Getting integration runtimes" 209 $integrationruntimesADF = Get-AzDataFactoryV2IntegrationRuntime -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 210 $integrationruntimesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/integrationruntimes" } 211 $integrationruntimesNames = $integrationruntimesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)} 212 $deletedintegrationruntimes = $integrationruntimesADF | Where-Object { $integrationruntimesNames -notcontains $_.Name } 213 214 #Delete resources 215 Write-Host "Deleting triggers" 216 $triggersToDelete | ForEach-Object { 217 Write-Host "Deleting trigger " $_.Name 218 $trig = Get-AzDataFactoryV2Trigger -name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName 219 if ($trig.RuntimeState -eq "Started") { 220 if ($_.TriggerType -eq "BlobEventsTrigger") { 221 Write-Host "Unsubscribing trigger" $_.Name "from events" 222 $status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 223 while ($status.Status -ne "Disabled"){ 224 Start-Sleep -s 15 225 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 226 } 227 } 228 Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force 229 } 230 Remove-AzDataFactoryV2Trigger -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 231 } 232 Write-Host "Deleting pipelines" 233 $deletedpipelines | ForEach-Object { 234 Write-Host "Deleting pipeline " $_.Name 235 Remove-AzDataFactoryV2Pipeline -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 236 } 237 Write-Host "Deleting dataflows" 238 $deleteddataflow | ForEach-Object { 239 Write-Host "Deleting dataflow " $_.Name 240 Remove-AzDataFactoryV2DataFlow -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 241 } 242 Write-Host "Deleting datasets" 243 $deleteddataset | ForEach-Object { 244 Write-Host "Deleting dataset " $_.Name 245 Remove-AzDataFactoryV2Dataset -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 246 } 247 Write-Host "Deleting linked services" 248 $deletedlinkedservices | ForEach-Object { 249 Write-Host "Deleting Linked Service " $_.Name 250 Remove-AzDataFactoryV2LinkedService -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 251 } 252 Write-Host "Deleting integration runtimes" 253 $deletedintegrationruntimes | ForEach-Object { 254 Write-Host "Deleting integration runtime " $_.Name 255 Remove-AzDataFactoryV2IntegrationRuntime -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 256 } 257 258 if ($deleteDeployment -eq $true) { 259 Write-Host "Deleting ARM deployment ... under resource group: " $ResourceGroupName 260 $deployments = Get-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName 261 $deploymentsToConsider = $deployments | Where { $_.DeploymentName -like "ArmTemplate_master*" -or $_.DeploymentName -like "ArmTemplateForFactory*" } | Sort-Object -Property Timestamp -Descending 262 $deploymentName = $deploymentsToConsider[0].DeploymentName 263 264 Write-Host "Deployment to be deleted: " $deploymentName 265 $deploymentOperations = Get-AzResourceGroupDeploymentOperation -DeploymentName $deploymentName -ResourceGroupName $ResourceGroupName 266 $deploymentsToDelete = $deploymentOperations | Where { $_.properties.targetResource.id -like "*Microsoft.Resources/deployments*" } 267 268 $deploymentsToDelete | ForEach-Object { 269 Write-host "Deleting inner deployment: " $_.properties.targetResource.id 270 Remove-AzResourceGroupDeployment -Id $_.properties.targetResource.id 271 } 272 Write-Host "Deleting deployment: " $deploymentName 273 Remove-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName -Name $deploymentName 274 } 275 276 #Start active triggers - after cleanup efforts 277 Write-Host "Starting active triggers" 278 $triggersToStart | ForEach-Object { 279 if ($_.TriggerType -eq "BlobEventsTrigger") { 280 Write-Host "Subscribing" $_.Name "to events" 281 $status = Add-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 282 while ($status.Status -ne "Enabled"){ 283 Start-Sleep -s 15 284 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 285 } 286 } 287 Write-Host "Starting trigger" $_.Name 288 Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force 289 } 290 }
We will now powershell Script added to code . The file is named “adf_ci_cd.sh”
Will , The contents of the script file above are pasted in , Click on “Commit”, And submit and save to “master” In the branch
3, To configure Azure DevOps Pipeline The environment
go back to Azure DevOps in , Choice “Pipelines=》Releases”, Click on “New pipeline”
At this point we need to select the template , Let's click first “Empty job” Create an empty job
Change the name of the subsidy to “UAT”
Next, select Add “artifact”, Add a new one first afd_master Explain the source of products
Source Type Choice :”Azure Repos Git“
Project Choice :”CnBateBlogWeb_Proj“
Source(reposity) Choice :”CnBateBlogWeb_Proj“
Default branch:”adf_publish“
Source alias( Source alias ):”_CnBateBlogWeb_Proj_Publish“
Click on ”Add“ Add new operation .
Add new “artifact”, Add a new one first ADF Of ”master“ Distribution as a source of product release , Notice the arguments in it
Default branch:”master“
Source alias:”_CnBateBlogWeb_Proj_Master“
Click on ”OK“ Add new operation .
Finally , Let's give ”UAT“ stages newly added task, Click on the circled parts of a graph
Click on ”+“, Add new ”Azure PowerShell“ Task, At the same time PowerShell Instruction code to stop all currently executing Triggers
For the current stop Trigger Of Task, There are some attribute configurations that we need to pay attention to
1,Display name It can be changed to ”Stop Triggers“( On the surface Task What did you do )
2, Choice ”UAT“ Where the environment is Azure Subscribe to
3,Script Path( Script path )
4,Script Arguments( Script arguments )
The script file mentioned above explains , We were just in ”master“ Branch submitted adf_ci_cd.sh This file , We can choose
Before deployment , Need to stop all Trigger, About script arguments ( stop it Trigger)
-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $true -deleteDeployment $false
When deployment is complete , Need to activate all Trigger, About script arguments ( Start Trigger)
-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $false -deleteDeployment $true
We will paste the corresponding script arguments above with their own actual UAT The resource group where the environment is located , Resource name , And the path of the template
Here are my current template arguments ; You can refer to :
-armTemplate "$(System.DefaultWorkingDirectory)/_CnBateBlogWeb_Proj_Publish/ADF-CnBateBlogWeb-Dev/ARMTemplateForFactory.json" -ResourceGroupName "Web_Test_DF_RG_UAT" -DataFactoryName "ADF-CnBateBlogWeb-AUT" -predeployment $true -deleteDeployment $false
The next step is to add ARM The template is deployed , Search for ”ARM template deployment“, find ARM Template deployment , Click on ”Add“ Add new
Modify the current Task Properties
Display name:”ARM Template deployment:ADF Deploy“
Subscription Choice : At present, we need UAT The subscription where the environment is located
Resource group Choice :”Web_Test_DF_RG_UAT“( That is, at present UAT The environment ADF The resource group that you belong to )
Loaction Choice :”East Asia“
Template as well as Template parameters Choose... Separately ”_CnBateBlog_Proj_Publish“ Under the ”ARMTemplateForFactory.json“,"ARMTemplateParametersForFactory.json"
and Override template parameters( Replace template arguments ) We need to replace some arguments in the template argument file , If you need to deploy ADF The name of , Source 、 Target source blob Storage The link string of
as follows , The example of the overlay template argument I demonstrated ( You need to put UAT Environmental ADF Change the name to your own name , And two linked character argument names. You need to find their corresponding names in the template argument file , Link string copy and paste yourself UAT Two of the environment Blob Storage The link string of ):
-factoryName "ADF-CnBateBlogWeb-AUT" -CnBateBlogFromBlobStorage_connectionString "DefaultEndpointsProtocol=https;AccountName=cnbateblogwebaccount1uat;AccountKey=XF7XkFsnEdIGZT8CCUJs7At1E/Ni9k/XJBtwWaKvwAuBnp3cCN6e2CYiV2uzsq2iI0vI2eZVHS8Kh9CvcuoNcg==;EndpointSuffix=core.windows.net" -CnBateBlobToBlobStorage_connectionString "DefaultEndpointsProtocol=https;AccountName=cnbateblogwebaccount2uat;AccountKey=E5z+q7XL7+8xTlqHsyaIGr0Eje/0DhT9+/E+oro4D57tsSuEchmnjLiK8zftTtyvQKLQUvTGJEsAOFCBGdioHw==;EndpointSuffix=core.windows.net"
Recently, it's newly added to restart all Trigger Of PowerShell Of Task, The specific arguments are shown in the figure below
Script Path Choice :”_CnBateBlogWeb_Proj_Master“ Under the file “adf_ci_cd.sh” Archives
Script Path ( Resources need to be cleaned up 、 Restart all Trigger The script arguments of ):
-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $false -deleteDeployment $true
We will paste the corresponding script arguments above with their own actual UAT The resource group where the environment is located , Resource name , And the path of the template
Here are my current template arguments ; You can refer to :
-armTemplate "$(System.DefaultWorkingDirectory)/_CnBateBlogWeb_Proj_Publish/ADF-CnBateBlogWeb-Dev/ARMTemplateForFactory.json" -ResourceGroupName "Web_Test_DF_RG_UAT" -DataFactoryName "ADF-CnBateBlogWeb-AUT" -predeployment $true -deleteDeployment $true
Modify the current pipeline The name of , Click on "Save” To store .
Next , It's about setting pipeline Trigger conditions for
Enable continuous deployment trigger , Every time in the selected Repository Git Trigger on push pipeline, Next, add branch filter conditions
Type:Include,Branch:“adf_master”, That is to say, whenever “adf_publish” It happened git When pushing , Trigger this pipeline
After setting up , Click on “Save” To store
We can manually “ establish Release”, Test pipeline The status display is normal , success
go back to Azure UAT In the environment , We can find that UAT Environmental ADF Of pipeline dataset Etc. has been deployed and configured successfully
4, Test ADF Automated Deployment
Let's go back to ADF Of Dev The environment , We try to be in ADF Dev The environment creates a new “Feature Branch” Try to change pipeline The name of the copy in
We'll take the picture above of Copy Data The name of is given by “Copy data1” Change it to “Copy data2”, And store it , Verify the operation .
After verifying that there is no problem , We can put the current “allen/Feature_1” Changes on the branch are merged into “master” On the branch
By submitting “Create pull request”
I'm not going to show you any more , There is something you don't understand , You can read the last article . Wait for the merge to complete , Let's go back to Dev Environmental ADF You can do it in “master” Branch to release . When the release is complete , Will push the latest changes to "adf_master" On the branch ARM Templates and arguments . This time it will trigger the one set before Azure DevOps pipeline 了 , You can automatically deploy these changes to UA The environment .
3、 ... and , End
Today's content focuses on Azure Data Factory And Azure Pipeline Integration and automated deployment of , But there is also a problem left behind , It's deploying ARM The template , We will directly Blob Storage The link string is pasted directly into the override template argument column , If ADF Add a lot of sources of information to , It's not appropriate to use direct , How to solve this problem , In the next section, we will continue with .**,°*:.*( ̄▽ ̄)/$:*.°** .
author :Allen
Copyright : Please indicate the author and source in the obvious position of the article . If a mistake is found , Welcome criticism and correction .
版权声明
本文为[itread01]所创,转载请带上原文链接,感谢
边栏推荐
- use Asponse.Words Working with word templates
- 一篇文章带你了解CSS 分页实例
- 中小微企业选择共享办公室怎么样?
- 5.4 static resource mapping
- Tool class under JUC package, its name is locksupport! Did you make it?
- Mac installation hanlp, and win installation and use
- Analysis of etcd core mechanism
- 每个前端工程师都应该懂的前端性能优化总结:
- 6.2 handleradapter adapter processor (in-depth analysis of SSM and project practice)
- axios学习笔记(二):轻松弄懂XHR的使用及如何封装简易axios
猜你喜欢
前端基础牢记的一些操作-Github仓库管理
一篇文章带你了解HTML表格及其主要属性介绍
A course on word embedding
PN8162 20W PD快充芯片,PD快充充电器方案
一篇文章带你了解CSS对齐方式
Not long after graduation, he earned 20000 yuan from private work!
How long does it take you to work out an object-oriented programming interview question from Ali school?
Just now, I popularized two unique skills of login to Xuemei
Building and visualizing decision tree with Python
Tool class under JUC package, its name is locksupport! Did you make it?
随机推荐
合约交易系统开发|智能合约交易平台搭建
6.1.1 handlermapping mapping processor (1) (in-depth analysis of SSM and project practice)
PHP应用对接Justswap专用开发包【JustSwap.PHP】
助力金融科技创新发展,ATFX走在行业最前列
[C / C + + 1] clion configuration and running C language
With the advent of tensorflow 2.0, can pytoch still shake the status of big brother?
NLP model Bert: from introduction to mastery (1)
Windows 10 tensorflow (2) regression analysis of principles, deep learning framework (gradient descent method to solve regression parameters)
一篇文章带你了解HTML表格及其主要属性介绍
[JMeter] two ways to realize interface Association: regular representation extractor and JSON extractor
Mac installation hanlp, and win installation and use
Brief introduction of TF flags
百万年薪,国内工作6年的前辈想和你分享这四点
html
一篇文章带你了解CSS 分页实例
A brief history of neural networks
Vuejs development specification
Did you blog today?
Lane change detection
Summary of common algorithms of linked list