Azure data factory (3) integrate azure Devops to realize CI / CD

One , introduction

　　 Due to the length of the previous section , Can't share Azure Data Factory Continuous integration of , Continue to release . Today we will focus on the use of Azure DevOps Pipeline waive , Automatic continuous integration , And has automatically released , Will Azure Data Factory Deploy to multiple environments .

　　 In fact, we need not be surprised , The deployment here is not so mysterious , We're in ADF Medium master After the branch is released , In fact, it will ADF Package all configuration information in , Compile to adf_master Below the branch , If you look at the released code carefully , It's easy to find out , It's all some ARM Template resources . With what I'm building right now demo For example

This is what someone is asking , What is ARM Templates ？ Here's your overview ：

　　 For Azure Deployment of some infrastructure resources on , You can do this by using infrastructure as code , But to automate deployment , Define the infrastructure resources to be deployed in your code . Make these infrastructure source codes part of the project , Like application code . Through code management tools , Manage it , Easy to deploy , Create by code 、 Delete resource . and ARM Templates are a form of infrastructure or code （ The other is Terraform）, The template is a JavaScript Object representation (JSON) Archives .  The template uses declarative syntax , Allows you to specify what to deploy , Instead of writing a series of programming commands to create content .  In this template , Specify the resources to deploy and the properties of those resources .

Talk about people , Namely ARM The template describes the cloud resources that we need to deploy and the arguments required by its resources , For example, by using ARM It's not a VM, So ARM It is described in the template VM Resources and their establishment VM The necessary arguments required .

go back to ADF in , That is to say, we finally passed master Branch out to adf_master Branch code , It's just a bunch of descriptions ADF Resources and ADF Configured properties and arguments . What we're talking about today is through ARM To achieve ADF Of UAT,PRO Deployment of the environment .

-------------------- I am the dividing line --------------------

1,Azure Data Factory（ One ） Introduction

2,Azure Data Factory（ Two ） Copy data

3,Azure Data Factory（ 3、 ... and ） Integrate Azure Devops Realize CI/CD

Two , Text

1, establish ADF Of UAT The environment

We use resource names +UAT To simulate the test environment , Here's a new one called "ADF-CnBateBlogWeb-UAT" Of Azure Data Factory and Two UAT Environmental Blob Storage

”cnbateblogwebaccount1dev“  As UAT Sources of environmental information , We are also for UAT Environment building is called “resformfolder” The container of

“cnbateblogwebaccount2dev”  As UAT The target source of the environment , We are also for UAT Environment building is called “restofolder” The container of

2, Initialize pre and post deployment scripts

In fact , We're deploying ADF Even other project code , It's all about putting AFD The trigger stops , When deployment is complete , We have to restart ADF Triggers configured in . Microsoft has helped us provide scripts that we can use before and after deployment

  1 param
  2 (
  3     [parameter(Mandatory = $false)] [String] $armTemplate,
  4     [parameter(Mandatory = $false)] [String] $ResourceGroupName,
  5     [parameter(Mandatory = $false)] [String] $DataFactoryName,
  6     [parameter(Mandatory = $false)] [Bool] $predeployment=$true,
  7     [parameter(Mandatory = $false)] [Bool] $deleteDeployment=$false
  8 )
  9 
 10 function getPipelineDependencies {
 11     param([System.Object] $activity)
 12     if ($activity.Pipeline) {
 13         return @($activity.Pipeline.ReferenceName)
 14     } elseif ($activity.Activities) {
 15         $result = @()
 16         $activity.Activities | ForEach-Object{ $result += getPipelineDependencies -activity $_ }
 17         return $result
 18     } elseif ($activity.ifFalseActivities -or $activity.ifTrueActivities) {
 19         $result = @()
 20         $activity.ifFalseActivities | Where-Object {$_ -ne $null} | ForEach-Object{ $result += getPipelineDependencies -activity $_ }
 21         $activity.ifTrueActivities | Where-Object {$_ -ne $null} | ForEach-Object{ $result += getPipelineDependencies -activity $_ }
 22         return $result
 23     } elseif ($activity.defaultActivities) {
 24         $result = @()
 25         $activity.defaultActivities | ForEach-Object{ $result += getPipelineDependencies -activity $_ }
 26         if ($activity.cases) {
 27             $activity.cases | ForEach-Object{ $_.activities } | ForEach-Object{$result += getPipelineDependencies -activity $_ }
 28         }
 29         return $result
 30     } else {
 31         return @()
 32     }
 33 }
 34 
 35 function pipelineSortUtil {
 36     param([Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline]$pipeline,
 37     [Hashtable] $pipelineNameResourceDict,
 38     [Hashtable] $visited,
 39     [System.Collections.Stack] $sortedList)
 40     if ($visited[$pipeline.Name] -eq $true) {
 41         return;
 42     }
 43     $visited[$pipeline.Name] = $true;
 44     $pipeline.Activities | ForEach-Object{ getPipelineDependencies -activity $_ -pipelineNameResourceDict $pipelineNameResourceDict}  | ForEach-Object{
 45         pipelineSortUtil -pipeline $pipelineNameResourceDict[$_] -pipelineNameResourceDict $pipelineNameResourceDict -visited $visited -sortedList $sortedList
 46     }
 47     $sortedList.Push($pipeline)
 48 
 49 }
 50 
 51 function Get-SortedPipelines {
 52     param(
 53         [string] $DataFactoryName,
 54         [string] $ResourceGroupName
 55     )
 56     $pipelines = Get-AzDataFactoryV2Pipeline -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
 57     $ppDict = @{}
 58     $visited = @{}
 59     $stack = new-object System.Collections.Stack
 60     $pipelines | ForEach-Object{ $ppDict[$_.Name] = $_ }
 61     $pipelines | ForEach-Object{ pipelineSortUtil -pipeline $_ -pipelineNameResourceDict $ppDict -visited $visited -sortedList $stack }
 62     $sortedList = new-object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline]
 63     
 64     while ($stack.Count -gt 0) {
 65         $sortedList.Add($stack.Pop())
 66     }
 67     $sortedList
 68 }
 69 
 70 function triggerSortUtil {
 71     param([Microsoft.Azure.Commands.DataFactoryV2.Models.PSTrigger]$trigger,
 72     [Hashtable] $triggerNameResourceDict,
 73     [Hashtable] $visited,
 74     [System.Collections.Stack] $sortedList)
 75     if ($visited[$trigger.Name] -eq $true) {
 76         return;
 77     }
 78     $visited[$trigger.Name] = $true;
 79     if ($trigger.Properties.DependsOn) {
 80         $trigger.Properties.DependsOn | Where-Object {$_ -and $_.ReferenceTrigger} | ForEach-Object{
 81             triggerSortUtil -trigger $triggerNameResourceDict[$_.ReferenceTrigger.ReferenceName] -triggerNameResourceDict $triggerNameResourceDict -visited $visited -sortedList $sortedList
 82         }
 83     }
 84     $sortedList.Push($trigger)
 85 }
 86 
 87 function Get-SortedTriggers {
 88     param(
 89         [string] $DataFactoryName,
 90         [string] $ResourceGroupName
 91     )
 92     $triggers = Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName
 93     $triggerDict = @{}
 94     $visited = @{}
 95     $stack = new-object System.Collections.Stack
 96     $triggers | ForEach-Object{ $triggerDict[$_.Name] = $_ }
 97     $triggers | ForEach-Object{ triggerSortUtil -trigger $_ -triggerNameResourceDict $triggerDict -visited $visited -sortedList $stack }
 98     $sortedList = new-object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSTrigger]
 99     
100     while ($stack.Count -gt 0) {
101         $sortedList.Add($stack.Pop())
102     }
103     $sortedList
104 }
105 
106 function Get-SortedLinkedServices {
107     param(
108         [string] $DataFactoryName,
109         [string] $ResourceGroupName
110     )
111     $linkedServices = Get-AzDataFactoryV2LinkedService -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName
112     $LinkedServiceHasDependencies = @('HDInsightLinkedService', 'HDInsightOnDemandLinkedService', 'AzureBatchLinkedService')
113     $Akv = 'AzureKeyVaultLinkedService'
114     $HighOrderList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]
115     $RegularList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]
116     $AkvList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]
117 
118     $linkedServices | ForEach-Object {
119         if ($_.Properties.GetType().Name -in $LinkedServiceHasDependencies) {
120             $HighOrderList.Add($_)
121         }
122         elseif ($_.Properties.GetType().Name -eq $Akv) {
123             $AkvList.Add($_)
124         }
125         else {
126             $RegularList.Add($_)
127         }
128     }
129 
130     $SortedList = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataFactoryV2.Models.PSLinkedService]($HighOrderList.Count + $RegularList.Count + $AkvList.Count)
131     $SortedList.AddRange($HighOrderList)
132     $SortedList.AddRange($RegularList)
133     $SortedList.AddRange($AkvList)
134     $SortedList
135 }
136 
137 $templateJson = Get-Content $armTemplate | ConvertFrom-Json
138 $resources = $templateJson.resources
139 
140 #Triggers 
141 Write-Host "Getting triggers"
142 $triggersInTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/triggers" }
143 $triggerNamesInTemplate = $triggersInTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)}
144 
145 $triggersDeployed = Get-SortedTriggers -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
146 
147 $triggersToStop = $triggersDeployed | Where-Object { $triggerNamesInTemplate -contains $_.Name } | ForEach-Object { 
148     New-Object PSObject -Property @{
149         Name = $_.Name
150         TriggerType = $_.Properties.GetType().Name 
151     }
152 }
153 $triggersToDelete = $triggersDeployed | Where-Object { $triggerNamesInTemplate -notcontains $_.Name } | ForEach-Object { 
154     New-Object PSObject -Property @{
155         Name = $_.Name
156         TriggerType = $_.Properties.GetType().Name 
157     }
158 }
159 $triggersToStart = $triggersInTemplate | Where-Object { $_.properties.runtimeState -eq "Started" -and ($_.properties.pipelines.Count -gt 0 -or $_.properties.pipeline.pipelineReference -ne $null)} | ForEach-Object { 
160     New-Object PSObject -Property @{
161         Name = $_.name.Substring(37, $_.name.Length-40)
162         TriggerType = $_.Properties.type
163     }
164 }
165 
166 if ($predeployment -eq $true) {
167     #Stop all triggers
168     Write-Host "Stopping deployed triggers`n"
169     $triggersToStop | ForEach-Object {
170         if ($_.TriggerType -eq "BlobEventsTrigger") {
171             Write-Host "Unsubscribing" $_.Name "from events"
172             $status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
173             while ($status.Status -ne "Disabled"){
174                 Start-Sleep -s 15
175                 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
176             }
177         }
178         Write-Host "Stopping trigger" $_.Name
179         Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force
180     }
181 }
182 else {
183     #Deleted resources
184     #pipelines
185     Write-Host "Getting pipelines"
186     $pipelinesADF = Get-SortedPipelines -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
187     $pipelinesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/pipelines" }
188     $pipelinesNames = $pipelinesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)}
189     $deletedpipelines = $pipelinesADF | Where-Object { $pipelinesNames -notcontains $_.Name }
190     #dataflows
191     $dataflowsADF = Get-AzDataFactoryV2DataFlow -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
192     $dataflowsTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/dataflows" }
193     $dataflowsNames = $dataflowsTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40) }
194     $deleteddataflow = $dataflowsADF | Where-Object { $dataflowsNames -notcontains $_.Name }
195     #datasets
196     Write-Host "Getting datasets"
197     $datasetsADF = Get-AzDataFactoryV2Dataset -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
198     $datasetsTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/datasets" }
199     $datasetsNames = $datasetsTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40) }
200     $deleteddataset = $datasetsADF | Where-Object { $datasetsNames -notcontains $_.Name }
201     #linkedservices
202     Write-Host "Getting linked services"
203     $linkedservicesADF = Get-SortedLinkedServices -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
204     $linkedservicesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/linkedservices" }
205     $linkedservicesNames = $linkedservicesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)}
206     $deletedlinkedservices = $linkedservicesADF | Where-Object { $linkedservicesNames -notcontains $_.Name }
207     #Integrationruntimes
208     Write-Host "Getting integration runtimes"
209     $integrationruntimesADF = Get-AzDataFactoryV2IntegrationRuntime -DataFactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName
210     $integrationruntimesTemplate = $resources | Where-Object { $_.type -eq "Microsoft.DataFactory/factories/integrationruntimes" }
211     $integrationruntimesNames = $integrationruntimesTemplate | ForEach-Object {$_.name.Substring(37, $_.name.Length-40)}
212     $deletedintegrationruntimes = $integrationruntimesADF | Where-Object { $integrationruntimesNames -notcontains $_.Name }
213 
214     #Delete resources
215     Write-Host "Deleting triggers"
216     $triggersToDelete | ForEach-Object { 
217         Write-Host "Deleting trigger "  $_.Name
218         $trig = Get-AzDataFactoryV2Trigger -name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName
219         if ($trig.RuntimeState -eq "Started") {
220             if ($_.TriggerType -eq "BlobEventsTrigger") {
221                 Write-Host "Unsubscribing trigger" $_.Name "from events"
222                 $status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
223                 while ($status.Status -ne "Disabled"){
224                     Start-Sleep -s 15
225                     $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
226                 }
227             }
228             Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force 
229         }
230         Remove-AzDataFactoryV2Trigger -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
231     }
232     Write-Host "Deleting pipelines"
233     $deletedpipelines | ForEach-Object { 
234         Write-Host "Deleting pipeline " $_.Name
235         Remove-AzDataFactoryV2Pipeline -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
236     }
237     Write-Host "Deleting dataflows"
238     $deleteddataflow | ForEach-Object { 
239         Write-Host "Deleting dataflow " $_.Name
240         Remove-AzDataFactoryV2DataFlow -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
241     }
242     Write-Host "Deleting datasets"
243     $deleteddataset | ForEach-Object { 
244         Write-Host "Deleting dataset " $_.Name
245         Remove-AzDataFactoryV2Dataset -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
246     }
247     Write-Host "Deleting linked services"
248     $deletedlinkedservices | ForEach-Object { 
249         Write-Host "Deleting Linked Service " $_.Name
250         Remove-AzDataFactoryV2LinkedService -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
251     }
252     Write-Host "Deleting integration runtimes"
253     $deletedintegrationruntimes | ForEach-Object { 
254         Write-Host "Deleting integration runtime " $_.Name
255         Remove-AzDataFactoryV2IntegrationRuntime -Name $_.Name -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Force 
256     }
257 
258     if ($deleteDeployment -eq $true) {
259         Write-Host "Deleting ARM deployment ... under resource group: " $ResourceGroupName
260         $deployments = Get-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName
261         $deploymentsToConsider = $deployments | Where { $_.DeploymentName -like "ArmTemplate_master*" -or $_.DeploymentName -like "ArmTemplateForFactory*" } | Sort-Object -Property Timestamp -Descending
262         $deploymentName = $deploymentsToConsider[0].DeploymentName
263 
264        Write-Host "Deployment to be deleted: " $deploymentName
265         $deploymentOperations = Get-AzResourceGroupDeploymentOperation -DeploymentName $deploymentName -ResourceGroupName $ResourceGroupName
266         $deploymentsToDelete = $deploymentOperations | Where { $_.properties.targetResource.id -like "*Microsoft.Resources/deployments*" }
267 
268         $deploymentsToDelete | ForEach-Object { 
269             Write-host "Deleting inner deployment: " $_.properties.targetResource.id
270             Remove-AzResourceGroupDeployment -Id $_.properties.targetResource.id
271         }
272         Write-Host "Deleting deployment: " $deploymentName
273         Remove-AzResourceGroupDeployment -ResourceGroupName $ResourceGroupName -Name $deploymentName
274     }
275 
276     #Start active triggers - after cleanup efforts
277     Write-Host "Starting active triggers"
278     $triggersToStart | ForEach-Object { 
279         if ($_.TriggerType -eq "BlobEventsTrigger") {
280             Write-Host "Subscribing" $_.Name "to events"
281             $status = Add-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
282             while ($status.Status -ne "Enabled"){
283                 Start-Sleep -s 15
284                 $status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
285             }
286         }
287         Write-Host "Starting trigger" $_.Name
288         Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force
289     }
290 }

adf_ci_cd.sh

We will now powershell Script added to code . The file is named “adf_ci_cd.sh”

Will , The contents of the script file above are pasted in , Click on “Commit”, And submit and save to “master” In the branch

3, To configure Azure DevOps Pipeline The environment

go back to Azure DevOps in , Choice “Pipelines=》Releases”, Click on “New pipeline”

At this point we need to select the template , Let's click first “Empty job” Create an empty job

Change the name of the subsidy to “UAT”

Next, select Add “artifact”, Add a new one first afd_master Explain the source of products

Source Type Choice ：”Azure Repos Git“

Project Choice ：”CnBateBlogWeb_Proj“

Source(reposity) Choice ：”CnBateBlogWeb_Proj“

Default branch：”adf_publish“

Source alias（ Source alias ）：”_CnBateBlogWeb_Proj_Publish“

Click on ”Add“ Add new operation .

Add new “artifact”, Add a new one first ADF Of ”master“ Distribution as a source of product release , Notice the arguments in it

Default branch：”master“

Source alias：”_CnBateBlogWeb_Proj_Master“

Click on ”OK“ Add new operation .

Finally , Let's give ”UAT“ stages newly added task, Click on the circled parts of a graph

Click on ”+“, Add new ”Azure PowerShell“ Task, At the same time PowerShell Instruction code to stop all currently executing Triggers 

For the current stop Trigger Of Task, There are some attribute configurations that we need to pay attention to

1,Display name It can be changed to ”Stop Triggers“（ On the surface Task What did you do ）

2, Choice ”UAT“ Where the environment is Azure Subscribe to

3,Script Path（ Script path ）

4,Script Arguments（ Script arguments ）

The script file mentioned above explains , We were just in ”master“ Branch submitted adf_ci_cd.sh This file , We can choose

Before deployment , Need to stop all Trigger, About script arguments （ stop it Trigger）

-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $true -deleteDeployment $false

  When deployment is complete , Need to activate all Trigger, About script arguments （ Start Trigger）

-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $false -deleteDeployment $true

We will paste the corresponding script arguments above with their own actual UAT The resource group where the environment is located , Resource name , And the path of the template

Here are my current template arguments ; You can refer to ：

-armTemplate "$(System.DefaultWorkingDirectory)/_CnBateBlogWeb_Proj_Publish/ADF-CnBateBlogWeb-Dev/ARMTemplateForFactory.json" -ResourceGroupName "Web_Test_DF_RG_UAT" -DataFactoryName "ADF-CnBateBlogWeb-AUT" -predeployment $true -deleteDeployment $false

The next step is to add ARM The template is deployed , Search for ”ARM template deployment“, find ARM Template deployment , Click on ”Add“ Add new

Modify the current Task Properties

Display name：”ARM Template deployment：ADF Deploy“

Subscription Choice ： At present, we need UAT The subscription where the environment is located

Resource group Choice ：”Web_Test_DF_RG_UAT“（ That is, at present UAT The environment ADF The resource group that you belong to ）

Loaction Choice ：”East Asia“

Template as well as Template parameters Choose... Separately ”_CnBateBlog_Proj_Publish“ Under the ”ARMTemplateForFactory.json“,"ARMTemplateParametersForFactory.json"

and Override template parameters（ Replace template arguments ） We need to replace some arguments in the template argument file , If you need to deploy ADF The name of , Source 、 Target source blob Storage The link string of

as follows , The example of the overlay template argument I demonstrated （ You need to put UAT Environmental ADF Change the name to your own name , And two linked character argument names. You need to find their corresponding names in the template argument file , Link string copy and paste yourself UAT Two of the environment Blob Storage The link string of ）：

-factoryName "ADF-CnBateBlogWeb-AUT" -CnBateBlogFromBlobStorage_connectionString "DefaultEndpointsProtocol=https;AccountName=cnbateblogwebaccount1uat;AccountKey=XF7XkFsnEdIGZT8CCUJs7At1E/Ni9k/XJBtwWaKvwAuBnp3cCN6e2CYiV2uzsq2iI0vI2eZVHS8Kh9CvcuoNcg==;EndpointSuffix=core.windows.net" -CnBateBlobToBlobStorage_connectionString "DefaultEndpointsProtocol=https;AccountName=cnbateblogwebaccount2uat;AccountKey=E5z+q7XL7+8xTlqHsyaIGr0Eje/0DhT9+/E+oro4D57tsSuEchmnjLiK8zftTtyvQKLQUvTGJEsAOFCBGdioHw==;EndpointSuffix=core.windows.net"

Recently, it's newly added to restart all Trigger Of PowerShell Of Task, The specific arguments are shown in the figure below

Script Path Choice ：”_CnBateBlogWeb_Proj_Master“ Under the file “adf_ci_cd.sh” Archives

Script Path （ Resources need to be cleaned up 、 Restart all Trigger The script arguments of ）：

-armTemplate "$(System.DefaultWorkingDirectory)/<your-arm-template-location>" -ResourceGroupName <your-resource-group-name> -DataFactoryName <your-data-factory-name> -predeployment $false -deleteDeployment $true

We will paste the corresponding script arguments above with their own actual UAT The resource group where the environment is located , Resource name , And the path of the template

Here are my current template arguments ; You can refer to ：

-armTemplate "$(System.DefaultWorkingDirectory)/_CnBateBlogWeb_Proj_Publish/ADF-CnBateBlogWeb-Dev/ARMTemplateForFactory.json" -ResourceGroupName "Web_Test_DF_RG_UAT" -DataFactoryName "ADF-CnBateBlogWeb-AUT" -predeployment $true -deleteDeployment $true

Modify the current pipeline The name of , Click on "Save” To store .

Next , It's about setting pipeline Trigger conditions for

Enable continuous deployment trigger , Every time in the selected Repository Git Trigger on push pipeline, Next, add branch filter conditions

Type：Include,Branch：“adf_master”, That is to say, whenever “adf_publish” It happened git When pushing , Trigger this pipeline

After setting up , Click on “Save” To store

We can manually “ establish Release”, Test pipeline The status display is normal , success

go back to Azure UAT In the environment , We can find that UAT Environmental ADF Of pipeline dataset Etc. has been deployed and configured successfully

4, Test ADF Automated Deployment

Let's go back to ADF Of Dev The environment , We try to be in ADF Dev The environment creates a new “Feature Branch” Try to change pipeline The name of the copy in

We'll take the picture above of Copy Data The name of is given by “Copy data1” Change it to “Copy data2”, And store it , Verify the operation .

After verifying that there is no problem , We can put the current “allen/Feature_1” Changes on the branch are merged into “master” On the branch

By submitting “Create pull request”

I'm not going to show you any more , There is something you don't understand , You can read the last article . Wait for the merge to complete , Let's go back to Dev Environmental ADF You can do it in “master” Branch to release . When the release is complete , Will push the latest changes to "adf_master" On the branch ARM Templates and arguments . This time it will trigger the one set before Azure DevOps pipeline 了 , You can automatically deploy these changes to UA The environment .

3、 ... and , End

 　　 Today's content focuses on Azure Data Factory And Azure Pipeline Integration and automated deployment of , But there is also a problem left behind , It's deploying ARM The template , We will directly Blob Storage The link string is pasted directly into the override template argument column , If ADF Add a lot of sources of information to , It's not appropriate to use direct , How to solve this problem , In the next section, we will continue with .**,°*:.*(￣▽￣)/$:*.°** .

author ：Allen 

Copyright ： Please indicate the author and source in the obvious position of the article . If a mistake is found , Welcome criticism and correction .