当前位置:网站首页>Azure data factory (3) integrate azure Devops to realize CI / CD
Azure data factory (3) integrate azure Devops to realize CI / CD
2020-11-06 01:28:00 【itread01】
One , introduction
Due to the length of the previous section , Can't share Azure Data Factory Continuous integration of , Continue to release . Today we will focus on the use of Azure DevOps Pipeline waive , Automatic continuous integration , And has automatically released , Will Azure Data Factory Deploy to multiple environments .
In fact, we need not be surprised , The deployment here is not so mysterious , We're in ADF Medium master After the branch is released , In fact, it will ADF Package all configuration information in , Compile to adf_master Below the branch , If you look at the released code carefully , It's easy to find out , It's all some ARM Template resources . With what I'm building right now demo For example
This is what someone is asking , What is ARM Templates ? Here's your overview :
For Azure Deployment of some infrastructure resources on , You can do this by using infrastructure as code , But to automate deployment , Define the infrastructure resources to be deployed in your code . Make these infrastructure source codes part of the project , Like application code . Through code management tools , Manage it , Easy to deploy , Create by code 、 Delete resource . and ARM Templates are a form of infrastructure or code ( The other is Terraform), The template is a JavaScript Object representation (JSON) Archives . The template uses declarative syntax , Allows you to specify what to deploy , Instead of writing a series of programming commands to create content . In this template , Specify the resources to deploy and the properties of those resources .
Talk about people , Namely ARM The template describes the cloud resources that we need to deploy and the arguments required by its resources , For example, by using ARM It's not a VM, So ARM It is described in the template VM Resources and their establishment VM The necessary arguments required .
go back to ADF in , That is to say, we finally passed master Branch out to adf_master Branch code , It's just a bunch of descriptions ADF Resources and ADF Configured properties and arguments . What we're talking about today is through ARM To achieve ADF Of UAT,PRO Deployment of the environment .
-------------------- I am the dividing line --------------------
1,Azure Data Factory( One ) Introduction
2,Azure Data Factory( Two ) Copy data
3,Azure Data Factory( 3、 ... and ) Integrate Azure Devops Realize CI/CD
Two , Text
1, establish ADF Of UAT The environment
We use resource names +UAT To simulate the test environment , Here's a new one called "ADF-CnBateBlogWeb-UAT" Of Azure Data Factory and Two UAT Environmental Blob Storage
”cnbateblogwebaccount1dev“ As UAT Sources of environmental information , We are also for UAT Environment building is called “resformfolder” The container of
“cnbateblogwebaccount2dev” As UAT The target source of the environment , We are also for UAT Environment building is called “restofolder” The container of
2, Initialize pre and post deployment scripts
In fact , We're deploying ADF Even other project code , It's all about putting AFD The trigger stops , When deployment is complete , We have to restart ADF Triggers configured in . Microsoft has helped us provide scripts that we can use before and after deployment
1 param 2 ( 3 [parameter(Mandatory = $false)] [String] $armTemplate, 4 [parameter(Mandatory = $false)] [String] $ResourceGroupName, 5 [parameter(Mandatory = $false)] [String] $DataFactoryName, 6 [parameter(Mandatory = $false)] [Bool] $predeployment=$true, 7 [parameter(Mandatory = $false)] [Bool] $deleteDeployment=$false 8 ) 9 10 function getPipelineDependencies { 11 param([System.Object] $activity) 12 if ($activity.Pipeline) { 13 return @($activity.Pipeline.ReferenceName) 14 } elseif ($activity.Activities) { 15 $result = @() 16 $activity.Activities | ForEach-Object{ $result += getPipelineDependencies -activity $_ } 17 return $result 18 } elseif ($activity.ifFalseActivities -or $activity.ifTrueActivities) { 19 $result = @() 20 $activity.ifFalseActivities | Where-Object {$_ -ne $null} | ForEach-Object{ $result += getPipelineDependencies -activity $_ } 21 $activity.ifTrueActivities | Where-Object {$_ -ne $null} | ForEach-Object{ $result += getPipelineDependencies -activity $_ } 22 return $result 23 } elseif ($activity.defaultActivities) { 24 $result = @() 25 $activity.defaultActivities | ForEach-Object{ $result += getPipelineDependencies -activity $_ } 26 if ($activity.cases) { 27 $activity.cases | ForEach-Object{ $_.activities } | ForEach-Object{$result += getPipelineDependencies -activity $_ } 28 } 29 return $result 30 } else { 31 return @() 32 } 33 } 34 35 function pipelineSortUtil { 36 param([Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline]$pipeline, 37 [Hashtable] $pipelineNameResourceDict, 38 [Hashtable] $visited, 39 [System.Collections.Stack] $sortedList) 40 if ($visited[$pipeline.Name] -eq $true) { 41 return; 42 } 43 $visited[$pipeline.Name] = $true; 44 $pipeline.Activities | ForEach-Object{ getPipelineDependencies -activity $_ -pipelineNameResourceDict $pipelineNameResourceDict} | ForEach-Object{ 45 pipelineSortUtil -pipeline $pipelineNameResourceDict[$_] -pipelineNameResourceDict $pipelineNameResourceDict -visited $visited -sortedList $sortedList 46 } 47 $sortedList.Push($pipeline) 48 49 } 50 51 function Get-SortedPipelines { 52 param( 53 [string] $DataFactoryName, 54 [string] $ResourceGroupName 55 ) 56 $pipelines = Get-AzDataFactoryV2Pipeline -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 57 $ppDict = @{} 58 $visited = @{} 59 $stack = new-object System.Collections.Stack 60 $pipelines | ForEach-Object{ $ppDict[$_.Name] = $_ } 61 $pipelines | ForEach-Object{ pipelineSortUtil -pipeline $_ -pipelineNameResourceDict $ppDict -visited $visited -sortedList $stack } 62 $sortedList = new-object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline] 63 64 while ($stack.Count -gt 0) { 65 $sortedList.Add($stack.Pop()) 66 } 67 $sortedList 68 } 69 70 function triggerSortUtil { 71 param([Microsoft.Azure.Commands.DataFactoryV2.Models.PSTrigger]$trigger, 72 [Hashtable] $triggerNameResourceDict, 73 [Hashtable] $visited, 74 [System.Collections.Stack] $sortedList) 75 if ($visited[$trigger.Name] -eq $true) { 76 return; 77 } 78 $visited[$trigger.Name] = $true; 79 if ($trigger.Properties.DependsOn) { 80 $trigger.Properties.DependsOn | Where-Object {$_ -and $_.ReferenceTrigger} | ForEach-Object{ 81 triggerSortUtil -trigger $triggerNameResourceDict[$_.ReferenceTrigger.ReferenceName] -triggerNameResourceDict $triggerNameResourceDict -visited $visited -sortedList $sortedList 82 } 83 } 84 $sortedList.Push($trigger) 85 } 86 87 function Get-SortedTriggers { 88 param( 89 [string] $DataFactoryName, 90 [string] $ResourceGroupName 91 ) 92 $triggers = Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName 93 $triggerDict = @{} 94 $visited = @{} 95 $stack = new-object System.Collections.Stack 96 $triggers | ForEach-Object{ $triggerDict[$_.Name] = $_ } 97 $triggers | ForEach-Object{ triggerSortUtil -trigger $_ -triggerNameResourceDict $triggerDict -visited $visited -sortedList $stack } 98 $sortedList = new-object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSTrigger] 99 100 while ($stack.Count -gt 0) { 101 $sortedList.Add($stack.Pop()) 102 } 103 $sortedList 104 } 105 106 function Get-SortedLinkedServices { 107 param( 108 [string] $DataFactoryName, 109 [string] $ResourceGroupName 110 ) 111 $linkedServices = Get-AzDataFactoryV2LinkedService -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName 112 $LinkedServiceHasDependencies = @('HDInsightLinkedService', 'HDInsightOnDemandLinkedService', 'AzureBatchLinkedService') 113 $Akv = 'AzureKeyVaultLinkedService' 114 $HighOrderList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService] 115 $RegularList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService] 116 $AkvList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService] 117 118 $linkedServices | ForEach-Object { 119 if ($_.Properties.GetType().Name -in $LinkedServiceHasDependencies) { 120 $HighOrderList.Add($_) 121 } 122 elseif ($_.Properties.GetType().Name -eq $Akv) { 123 $AkvList.Add($_) 124 } 125 else { 126 $RegularList.Add($_) 127 } 128 } 129 130 $SortedList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]($HighOrderList.Count + $RegularList.Count + $AkvList.Count) 131 $SortedList.AddRange($HighOrderList) 132 $SortedList.AddRange($RegularList) 133 $SortedList.AddRange($AkvList) 134 $SortedList 135 } 136 137 $templateJson = Get-Content $armTemplate | ConvertFrom-Json 138 $resources = $templateJson.resources 139 140 #Triggers 141 Write-Host "Getting triggers" 142 $triggersInTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/triggers" } 143 $triggerNamesInTemplate = $triggersInTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)} 144 145 $triggersDeployed = Get-SortedTriggers -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 146 147 $triggersToStop = $triggersDeployed | Where-Object { $triggerNamesInTemplate -contains $_.Name } | ForEach-Object { 148 New-Object PSObject -Property @{ 149 Name = $_.Name 150 TriggerType = $_.Properties.GetType().Name 151 } 152 } 153 $triggersToDelete = $triggersDeployed | Where-Object { $triggerNamesInTemplate -notcontains $_.Name } | ForEach-Object { 154 New-Object PSObject -Property @{ 155 Name = $_.Name 156 TriggerType = $_.Properties.GetType().Name 157 } 158 } 159 $triggersToStart = $triggersInTemplate | Where-Object { $_.properties.runtimeState -eq "Started" -and ($_.properties.pipelines.Count -gt 0 -or $_.properties.pipeline.pipelineReference -ne $null)} | ForEach-Object { 160 New-Object PSObject -Property @{ 161 Name = $_.name.Substring(37, $_.name.Length-40) 162 TriggerType = $_.Properties.type 163 } 164 } 165 166 if ($predeployment -eq $true) { 167 #Stop all triggers 168 Write-Host "Stopping deployed triggers`n" 169 $triggersToStop | ForEach-Object { 170 if ($_.TriggerType -eq "BlobEventsTrigger") { 171 Write-Host "Unsubscribing" $_.Name "from events" 172 $status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 173 while ($status.Status -ne "Disabled"){ 174 Start-Sleep -s 15 175 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 176 } 177 } 178 Write-Host "Stopping trigger" $_.Name 179 Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force 180 } 181 } 182 else { 183 #Deleted resources 184 #pipelines 185 Write-Host "Getting pipelines" 186 $pipelinesADF = Get-SortedPipelines -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 187 $pipelinesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/pipelines" } 188 $pipelinesNames = $pipelinesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)} 189 $deletedpipelines = $pipelinesADF | Where-Object { $pipelinesNames -notcontains $_.Name } 190 #dataflows 191 $dataflowsADF = Get-AzDataFactoryV2DataFlow -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 192 $dataflowsTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/dataflows" } 193 $dataflowsNames = $dataflowsTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40) } 194 $deleteddataflow = $dataflowsADF | Where-Object { $dataflowsNames -notcontains $_.Name } 195 #datasets 196 Write-Host "Getting datasets" 197 $datasetsADF = Get-AzDataFactoryV2Dataset -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 198 $datasetsTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/datasets" } 199 $datasetsNames = $datasetsTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40) } 200 $deleteddataset = $datasetsADF | Where-Object { $datasetsNames -notcontains $_.Name } 201 #linkedservices 202 Write-Host "Getting linked services" 203 $linkedservicesADF = Get-SortedLinkedServices -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 204 $linkedservicesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/linkedservices" } 205 $linkedservicesNames = $linkedservicesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)} 206 $deletedlinkedservices = $linkedservicesADF | Where-Object { $linkedservicesNames -notcontains $_.Name } 207 #Integrationruntimes 208 Write-Host "Getting integration runtimes" 209 $integrationruntimesADF = Get-AzDataFactoryV2IntegrationRuntime -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName 210 $integrationruntimesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/integrationruntimes" } 211 $integrationruntimesNames = $integrationruntimesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)} 212 $deletedintegrationruntimes = $integrationruntimesADF | Where-Object { $integrationruntimesNames -notcontains $_.Name } 213 214 #Delete resources 215 Write-Host "Deleting triggers" 216 $triggersToDelete | ForEach-Object { 217 Write-Host "Deleting trigger " $_.Name 218 $trig = Get-AzDataFactoryV2Trigger -name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName 219 if ($trig.RuntimeState -eq "Started") { 220 if ($_.TriggerType -eq "BlobEventsTrigger") { 221 Write-Host "Unsubscribing trigger" $_.Name "from events" 222 $status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 223 while ($status.Status -ne "Disabled"){ 224 Start-Sleep -s 15 225 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 226 } 227 } 228 Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force 229 } 230 Remove-AzDataFactoryV2Trigger -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 231 } 232 Write-Host "Deleting pipelines" 233 $deletedpipelines | ForEach-Object { 234 Write-Host "Deleting pipeline " $_.Name 235 Remove-AzDataFactoryV2Pipeline -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 236 } 237 Write-Host "Deleting dataflows" 238 $deleteddataflow | ForEach-Object { 239 Write-Host "Deleting dataflow " $_.Name 240 Remove-AzDataFactoryV2DataFlow -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 241 } 242 Write-Host "Deleting datasets" 243 $deleteddataset | ForEach-Object { 244 Write-Host "Deleting dataset " $_.Name 245 Remove-AzDataFactoryV2Dataset -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 246 } 247 Write-Host "Deleting linked services" 248 $deletedlinkedservices | ForEach-Object { 249 Write-Host "Deleting Linked Service " $_.Name 250 Remove-AzDataFactoryV2LinkedService -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 251 } 252 Write-Host "Deleting integration runtimes" 253 $deletedintegrationruntimes | ForEach-Object { 254 Write-Host "Deleting integration runtime " $_.Name 255 Remove-AzDataFactoryV2IntegrationRuntime -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 256 } 257 258 if ($deleteDeployment -eq $true) { 259 Write-Host "Deleting ARM deployment ... under resource group: " $ResourceGroupName 260 $deployments = Get-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName 261 $deploymentsToConsider = $deployments | Where { $_.DeploymentName -like "ArmTemplate_master*" -or $_.DeploymentName -like "ArmTemplateForFactory*" } | Sort-Object -Property Timestamp -Descending 262 $deploymentName = $deploymentsToConsider[0].DeploymentName 263 264 Write-Host "Deployment to be deleted: " $deploymentName 265 $deploymentOperations = Get-AzResourceGroupDeploymentOperation -DeploymentName $deploymentName -ResourceGroupName $ResourceGroupName 266 $deploymentsToDelete = $deploymentOperations | Where { $_.properties.targetResource.id -like "*Microsoft.Resources/deployments*" } 267 268 $deploymentsToDelete | ForEach-Object { 269 Write-host "Deleting inner deployment: " $_.properties.targetResource.id 270 Remove-AzResourceGroupDeployment -Id $_.properties.targetResource.id 271 } 272 Write-Host "Deleting deployment: " $deploymentName 273 Remove-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName -Name $deploymentName 274 } 275 276 #Start active triggers - after cleanup efforts 277 Write-Host "Starting active triggers" 278 $triggersToStart | ForEach-Object { 279 if ($_.TriggerType -eq "BlobEventsTrigger") { 280 Write-Host "Subscribing" $_.Name "to events" 281 $status = Add-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 282 while ($status.Status -ne "Enabled"){ 283 Start-Sleep -s 15 284 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name 285 } 286 } 287 Write-Host "Starting trigger" $_.Name 288 Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force 289 } 290 }
We will now powershell Script added to code . The file is named “adf_ci_cd.sh”
Will , The contents of the script file above are pasted in , Click on “Commit”, And submit and save to “master” In the branch
3, To configure Azure DevOps Pipeline The environment
go back to Azure DevOps in , Choice “Pipelines=》Releases”, Click on “New pipeline”
At this point we need to select the template , Let's click first “Empty job” Create an empty job
Change the name of the subsidy to “UAT”
Next, select Add “artifact”, Add a new one first afd_master Explain the source of products
Source Type Choice :”Azure Repos Git“
Project Choice :”CnBateBlogWeb_Proj“
Source(reposity) Choice :”CnBateBlogWeb_Proj“
Default branch:”adf_publish“
Source alias( Source alias ):”_CnBateBlogWeb_Proj_Publish“
Click on ”Add“ Add new operation .
Add new “artifact”, Add a new one first ADF Of ”master“ Distribution as a source of product release , Notice the arguments in it
Default branch:”master“
Source alias:”_CnBateBlogWeb_Proj_Master“
Click on ”OK“ Add new operation .
Finally , Let's give ”UAT“ stages newly added task, Click on the circled parts of a graph
Click on ”+“, Add new ”Azure PowerShell“ Task, At the same time PowerShell Instruction code to stop all currently executing Triggers
For the current stop Trigger Of Task, There are some attribute configurations that we need to pay attention to
1,Display name It can be changed to ”Stop Triggers“( On the surface Task What did you do )
2, Choice ”UAT“ Where the environment is Azure Subscribe to
3,Script Path( Script path )
4,Script Arguments( Script arguments )
The script file mentioned above explains , We were just in ”master“ Branch submitted adf_ci_cd.sh This file , We can choose
Before deployment , Need to stop all Trigger, About script arguments ( stop it Trigger)
-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $true -deleteDeployment $false
When deployment is complete , Need to activate all Trigger, About script arguments ( Start Trigger)
-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $false -deleteDeployment $true
We will paste the corresponding script arguments above with their own actual UAT The resource group where the environment is located , Resource name , And the path of the template
Here are my current template arguments ; You can refer to :
-armTemplate "$(System.DefaultWorkingDirectory)/_CnBateBlogWeb_Proj_Publish/ADF-CnBateBlogWeb-Dev/ARMTemplateForFactory.json" -ResourceGroupName "Web_Test_DF_RG_UAT" -DataFactoryName "ADF-CnBateBlogWeb-AUT" -predeployment $true -deleteDeployment $false
The next step is to add ARM The template is deployed , Search for ”ARM template deployment“, find ARM Template deployment , Click on ”Add“ Add new
Modify the current Task Properties
Display name:”ARM Template deployment:ADF Deploy“
Subscription Choice : At present, we need UAT The subscription where the environment is located
Resource group Choice :”Web_Test_DF_RG_UAT“( That is, at present UAT The environment ADF The resource group that you belong to )
Loaction Choice :”East Asia“
Template as well as Template parameters Choose... Separately ”_CnBateBlog_Proj_Publish“ Under the ”ARMTemplateForFactory.json“,"ARMTemplateParametersForFactory.json"
and Override template parameters( Replace template arguments ) We need to replace some arguments in the template argument file , If you need to deploy ADF The name of , Source 、 Target source blob Storage The link string of
as follows , The example of the overlay template argument I demonstrated ( You need to put UAT Environmental ADF Change the name to your own name , And two linked character argument names. You need to find their corresponding names in the template argument file , Link string copy and paste yourself UAT Two of the environment Blob Storage The link string of ):
-factoryName "ADF-CnBateBlogWeb-AUT" -CnBateBlogFromBlobStorage_connectionString "DefaultEndpointsProtocol=https;AccountName=cnbateblogwebaccount1uat;AccountKey=XF7XkFsnEdIGZT8CCUJs7At1E/Ni9k/XJBtwWaKvwAuBnp3cCN6e2CYiV2uzsq2iI0vI2eZVHS8Kh9CvcuoNcg==;EndpointSuffix=core.windows.net" -CnBateBlobToBlobStorage_connectionString "DefaultEndpointsProtocol=https;AccountName=cnbateblogwebaccount2uat;AccountKey=E5z+q7XL7+8xTlqHsyaIGr0Eje/0DhT9+/E+oro4D57tsSuEchmnjLiK8zftTtyvQKLQUvTGJEsAOFCBGdioHw==;EndpointSuffix=core.windows.net"
Recently, it's newly added to restart all Trigger Of PowerShell Of Task, The specific arguments are shown in the figure below
Script Path Choice :”_CnBateBlogWeb_Proj_Master“ Under the file “adf_ci_cd.sh” Archives
Script Path ( Resources need to be cleaned up 、 Restart all Trigger The script arguments of ):
-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $false -deleteDeployment $true
We will paste the corresponding script arguments above with their own actual UAT The resource group where the environment is located , Resource name , And the path of the template
Here are my current template arguments ; You can refer to :
-armTemplate "$(System.DefaultWorkingDirectory)/_CnBateBlogWeb_Proj_Publish/ADF-CnBateBlogWeb-Dev/ARMTemplateForFactory.json" -ResourceGroupName "Web_Test_DF_RG_UAT" -DataFactoryName "ADF-CnBateBlogWeb-AUT" -predeployment $true -deleteDeployment $true
Modify the current pipeline The name of , Click on "Save” To store .
Next , It's about setting pipeline Trigger conditions for
Enable continuous deployment trigger , Every time in the selected Repository Git Trigger on push pipeline, Next, add branch filter conditions
Type:Include,Branch:“adf_master”, That is to say, whenever “adf_publish” It happened git When pushing , Trigger this pipeline
After setting up , Click on “Save” To store
We can manually “ establish Release”, Test pipeline The status display is normal , success
go back to Azure UAT In the environment , We can find that UAT Environmental ADF Of pipeline dataset Etc. has been deployed and configured successfully
4, Test ADF Automated Deployment
Let's go back to ADF Of Dev The environment , We try to be in ADF Dev The environment creates a new “Feature Branch” Try to change pipeline The name of the copy in
We'll take the picture above of Copy Data The name of is given by “Copy data1” Change it to “Copy data2”, And store it , Verify the operation .
After verifying that there is no problem , We can put the current “allen/Feature_1” Changes on the branch are merged into “master” On the branch
By submitting “Create pull request”
I'm not going to show you any more , There is something you don't understand , You can read the last article . Wait for the merge to complete , Let's go back to Dev Environmental ADF You can do it in “master” Branch to release . When the release is complete , Will push the latest changes to "adf_master" On the branch ARM Templates and arguments . This time it will trigger the one set before Azure DevOps pipeline 了 , You can automatically deploy these changes to UA The environment .
3、 ... and , End
Today's content focuses on Azure Data Factory And Azure Pipeline Integration and automated deployment of , But there is also a problem left behind , It's deploying ARM The template , We will directly Blob Storage The link string is pasted directly into the override template argument column , If ADF Add a lot of sources of information to , It's not appropriate to use direct , How to solve this problem , In the next section, we will continue with .**,°*:.*( ̄▽ ̄)/$:*.°** .
author :Allen
Copyright : Please indicate the author and source in the obvious position of the article . If a mistake is found , Welcome criticism and correction .
版权声明
本文为[itread01]所创,转载请带上原文链接,感谢
边栏推荐
- 加速「全民直播」洪流,如何攻克延时、卡顿、高并发难题?
- 全球疫情加速互联网企业转型,区块链会是解药吗?
- Group count - word length
- 用一个例子理解JS函数的底层处理机制
- A course on word embedding
- Do not understand UML class diagram? Take a look at this edition of rural love class diagram, a learn!
- Analysis of partial source codes of qthread
- Solve the problem of database insert data garbled in PL / SQL developer
- Advanced Vue component pattern (3)
- The difference between Es5 class and ES6 class
猜你喜欢
一篇文章带你了解CSS 渐变知识
前端基础牢记的一些操作-Github仓库管理
有了这个神器,快速告别垃圾短信邮件
TensorFlow中的Tensor是什么?
小程序入门到精通(二):了解小程序开发4个重要文件
[JMeter] two ways to realize interface Association: regular representation extractor and JSON extractor
零基础打造一款属于自己的网页搜索引擎
Recommendation system based on deep learning
助力金融科技创新发展,ATFX走在行业最前列
How to select the evaluation index of classification model
随机推荐
6.2 handleradapter adapter processor (in-depth analysis of SSM and project practice)
Tool class under JUC package, its name is locksupport! Did you make it?
Synchronous configuration from git to consult with git 2consul
How to use Python 2.7 after installing anaconda3?
至联云解析:IPFS/Filecoin挖矿为什么这么难?
Python crawler actual combat details: crawling home of pictures
With the advent of tensorflow 2.0, can pytoch still shake the status of big brother?
Brief introduction of TF flags
ES6学习笔记(四):教你轻松搞懂ES6的新增语法
Python Jieba segmentation (stuttering segmentation), extracting words, loading words, modifying word frequency, defining thesaurus
Python download module to accelerate the implementation of recording
How to customize sorting for pandas dataframe
The difference between gbdt and XGB, and the mathematical derivation of gradient descent method and Newton method
I'm afraid that the spread sequence calculation of arbitrage strategy is not as simple as you think
Wiremock: a powerful tool for API testing
What is the side effect free method? How to name it? - Mario
Save the file directly to Google drive and download it back ten times faster
一篇文章教会你使用Python网络爬虫下载酷狗音乐
5.5 controlleradvice notes - SSM in depth analysis and project practice
Installing the consult cluster