Azure Data Factory – Automated deployments (CI/CD) using Azure DevOps

In the Azure Data Factory – Promoting ADF Components manually to higher Environments article, we have learned how to promote ADF components manually to the next higher environments. Manual process works well but its always prone to error. So, fortunately, Azure Data Factory support automating the process of deploying the changes to the other environments with the help of Azure DevOps. In this article, we will learn how to implement automated deployments of ADF components.

Prerequisites:

In order to automate the deployment of ADF pipeline and it’s components, you need to integrate the ADF service with Azure DevOps. I would strongly recommend you to first go through the below articles.

In the Azure Data Factory – Promoting ADF Components manually to higher Environments article, we have learnt how to promote manually from Development Environment (adf-dev-eus-rg) to Staging Environment (adf-stg-eus-rg). In this article, we will learn how to promote the ADF pipeline and components automatically to the Staging Environment (adf-stg-eus-rg) using the Azure DevOps.

And, you also need to understand about adf_publish branch at Azure Data Factory – All about publish branch adf_publish

Configure CI / CD of the Azure Data Factory pipeline

Below are the steps that we will be configuring in this article.

  1. Create a Release pipeline
  2. Link the Artifacts with Release Pipeline
  3. Create Release Variables
  4. Configure the Staging Environment
  5. Manually run the Release pipeline for Staging Environment
  6. Automated Deployment to Staging

Create a Release pipeline

As shown in the below video, create a new Release pipeline and name the stage as Staging.

Link the Artifacts with Release Pipeline

Once the Release pipeline is created, we need to link it with the Artifacts. Go through the below video and linked the Artifacts from the Azure Devops Repository. And, once you configure click on Save to save the changes.

Create Release Variables for Staging Environment.

In this section, we need to create release variables that contain the Connection Strings of Storage And Database. As shown in the below screenshot, navigate to the Variables tab and then create the below two variables.

  1. Ensure that you choose Staging as the Scope
  2. Copy the connection strings of the Storage and Database of the Staging Environment (adf-stg-eus-adf).
Azure Data Factory - Automated deployments CI CD using Azure DevOps - Release Variables
Azure Data Factory – Automated deployments CI CD using Azure DevOps – Release Variables

Configure the Staging Environment

Now, Let’s configure the Tasks in the Stage Environment by following the steps provided in the below video.

In the below video, you will be referring to the variables created in the previous step.

Manually run the Release pipeline for Staging Environment

Let’s test the configurations by running the Release pipeline manually. In order to do that, navigate to the Release pipelines (Icon 1 below) and then click on the Release pipeline (Icon 2 below) and then click on Create Release button (Icon 3 below).

Azure Data Factory - Automated deployments CI CD using Azure DevOps - Run Release Manually

As you might have noticed in the above screen shot, the Release got failed multiple times. I got an error initially and I have fixed it. You can learn more about it Format of the initialization string does not conform to specification starting at index 0 here in case if you also get the same.

Now, Below is the screenshot of the Release Variables after the fix. Basically, I was referring to in incorrect variable. As shown in the below screenshot, I have updated the name of the SQL Connection variable.

Azure Data Factory - Automated deployments CI CD using Azure DevOps - Release Variables - After Fix

Now, I ran the pipeline and it was successful and below is the screenshot of the ADF service of the Staging Environment

Azure Data Factory - Automated deployments CI CD using Azure DevOps - Stg ADF - After Manual Release
Azure Data Factory – Automated deployments CI CD using Azure DevOps – Stg ADF – After Manual Release

Automated Deployment to Staging

In this section, let’s learn how to release the ADF Pipeline changes to Staging environment immediately after the publish is complete.

Navigate to the Release pipeline, click on the Continuous Deployment Trigger button as shown below.

Azure Data Factory - Automated deployments CI CD using Azure DevOps - Continuous Deployment Trigger
Azure Data Factory – Automated deployments CI CD using Azure DevOps – Continuous Deployment Trigger

It opens up another popup as shown below where you enabled the Continuous Deployment Trigger toggle button.

Azure Data Factory - Automated deployments CI CD using Azure DevOps - Enable Continuous Deployment Trigger
Azure Data Factory – Automated deployments CI CD using Azure DevOps – Enable Continuous Deployment Trigger

Once you enable the above feature, click on Save button to save the changes.

Let’s now make some changes to the Azure Data Factory (adf-dev-eus-adf) of Development Environment (adf-dev-eus-rg) and click on Publish button as shown below.

Azure Data Factory - Automated deployments CI CD using Azure DevOps - Dev ADF - Make Changes

The deployment trigger gets triggered Immediately and if you navigate to the Release pipelines, you will see a new release that got triggered automatically as shown below.

Azure Data Factory - Automated deployments CI CD using Azure DevOps - Dev ADF - Automated Release
Azure Data Factory – Automated deployments CI CD using Azure DevOps – Dev ADF – Automated Release

And, you can see the new changes in the Staging Environment now as shown below.

Azure Data Factory - Automated deployments CI CD using Azure DevOps - Staging ADF - After Automated Release
Azure Data Factory – Automated deployments CI CD using Azure DevOps – Staging ADF – After Automated Release

That’s It. We have learned how to automatically promote the changes to the higher environments using the Azure Devops.

 

 

One comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s