Azure Data Factory – Automated deployments (CI/CD) using Azure DevOps

In the Azure Data Factory – Promoting ADF Components manually to higher Environments article, we have learned how to promote ADF components manually to the next higher environments. Manual process works well but its always prone to error. Fortunately, Azure Data Factory supports automating the process of deploying the changes to the other environments with the help of Azure DevOps.

In this article, we will learn how to implement automated deployments of ADF components.

Prerequisites:

In order to automate the deployment of ADF pipeline and it’s components, you need to integrate the ADF service with Azure DevOps. I would strongly recommend you to first go through the below articles.

• Azure Data Factory – 3 ways to Integrate ADF Pipeline with Azure DevOps – Git
• Azure Data Factory – Collaborative development of ADF pipelines using Azure DevOps – Git

In the Azure Data Factory – Promoting ADF Components manually to higher Environments article, we have learnt how to promote manually from Development Environment (adf-dev-eus-rg) to Staging Environment (adf-stg-eus-rg). In this article, we will learn how to promote the ADF pipeline and components automatically to the Staging Environment (adf-stg-eus-rg) using the Azure DevOps.

And, you also need to understand about adf_publish branch at Azure Data Factory – All about publish branch adf_publish

Configure CI / CD of the Azure Data Factory pipeline

Below are the steps that we will be configuring in this article.

1. Create a Release pipeline
2. Link the Artifacts with Release Pipeline
3. Create Release Variables
4. Configure the Staging Environment
5. Manually run the Release pipeline for Staging Environment
6. Automated Deployment to Staging

Create a Release pipeline

As shown in the below video, create a new Release pipeline and name the stage as Staging.

Link the Artifacts with Release Pipeline

Once the Release pipeline is created, we need to link it with the Artifacts. Go through the below video and link the Artifacts from the Azure Devops Repository. And, once you configure click on Save to save the changes.

Create Release Variables for Staging Environment.

In this section, we need to create release variables that contain the Connection Strings of Storage And Database. As shown in the below screenshot, navigate to the Variables tab and then create the below two variables.

1. Ensure that you choose Staging as the Scope
2. Copy the connection strings of the Storage and Database of the Staging Environment (adf-stg-eus-adf).

Configure the Staging Environment

Now, Let’s configure the Tasks in the Stage Environment by following the steps provided in the below video.

In the below video, you will be referring to the variables created in the previous step.

Manually run the Release pipeline for Staging Environment

Let’s test the configurations by running the Release pipeline manually. In order to do that, navigate to the Release pipelines (Icon 1 below) and then click on the Release pipeline (Icon 2 below) and then click on Create Release button (Icon 3 below).

As you might have noticed in the above screen shot, the Release got failed multiple times. I got an error initially and I have fixed it. You can learn more about it Format of the initialization string does not conform to specification starting at index 0 here in case if you also get the same.

Now, Below is the screenshot of the Release Variables after the fix. Basically, I was referring to in incorrect variable. As shown in the below screenshot, I have updated the name of the SQL Connection variable.

Now, I ran the pipeline and it was successful and below is the screenshot of the ADF service of the Staging Environment

Automated Deployment to Staging

In this section, let’s learn how to release the ADF Pipeline changes to Staging environment immediately after the publish is complete.

Navigate to the Release pipeline, click on the Continuous Deployment Trigger button as shown below.

It opens up another popup as shown below where you enabled the Continuous Deployment Trigger toggle button.

Once you enable the above feature, click on Save button to save the changes.

Let’s now make some changes to the Azure Data Factory (adf-dev-eus-adf) of Development Environment (adf-dev-eus-rg) and click on Publish button as shown below.

The deployment trigger gets triggered Immediately and if you navigate to the Release pipelines, you will see a new release that got triggered automatically as shown below.

And, you can see the new changes in the Staging Environment now as shown below.

That’s It. We have learned how to automatically promote the changes to the higher environments using the Azure Devops.

Do you like this article? If you want to get more updates about these kind of articles, you can join my Learning Groups

WhatsApp

Telegram