Azure Data Factory – Promoting ADF Components manually to higher Environments

Today, we will learn how to promote a simple Azure Data Factory pipeline along with other components (Linked Services, Data Sets etc.) from a development environment to higher environments like Staging & Production.

In this article, we will learn how to implement this manually. In the next article Azure Data Factory – Automated deployments (CI/CD) using Azure DevOps we will learn how to automate the process using Azure DevOps.

Let me start by listing down the prerequisites. Create the following Azure services that the ADF pipelines would be using.

Resource Groups Azure Data Factory (Logical) SQL Server Azure SQL Database Storage
adf—rg adf—adf adf–dbserver adf—db adf—strg

Below is the list of all Azure Resources that I have created using Azure Management Portal.

Resource Groups

1.Resource groups - Microsoft Azure

Azure Data Factory Instances

2.Data factories - Microsoft Azure

(Logical) SQL Servers

3.SQL servers - Microsoft Azure

Azure SQL Database

4.SQL databases - Microsoft Azure

Azure Storage Accounts

5.Storage accounts - Microsoft Azure

Author Azure Data Factory Pipeline in the Dev Environment.

  1. Navigate to the Azure Data Factory named adf-dev-eus-adf (in your case name would be different) of Dev Environment.
  2. Create a new ADF Pipeline
  3. Create two Linked Services
    1. A Linked Service for Storage Account
    2. A Linked Service for Azure SQL Database
  4. Create two Datasets
    1. A Dataset for the Source Blob
    2. A Dataset for SQL Database table.
  5. Create a Copy Data activity and configure the Source and Sink Linked Services and Datasets for copying the data from the Blob to the Table.

As explained in the below video, create a pipeline with the above steps.

 

The general practice that any developer would follow is to develop /author the ADF pipeline in the Development Environment. Once the functionality is validated, we need to promote the ADF pipeline and its components (Linked Services, Datasets etc.) to the next higher environments without authoring again.

Note: In this article, we will learn how to promote ONLY the ADF pipelines and its components but not the Azure Services like Azure Data Factory, Storage and Azure SQL Database

If you would like to automate the process of promoting the pipelines, refer Azure Data Factory – Automated deployments ( CI/CD ) using Azure DevOps (Coming Soon. Stay tuned)

Once you complete the authoring of the ADF pipeline, it should like something as shown below.

6.Completed Azure Data Factory Pipeline

We are now done with the authoring in the Development environment. Let’s start the process of promoting the ADF pipeline and its components by following the below steps.

Export the ARM Template

From Azure Data Factory V2 (version2) onwards, ARM Templates are supported for creating the ADF pipelines and its related components. Let’s export the ARM template of our ADF pipeline by clicking on the ARM Template and then on Export ARM template as shown below.

7.ExportARMTemplate

If you are using Chrome browser then you should have the .zip file downloaded as shown below.

8.ARM Template - Zip file

Go-ahead and un-zip it as we will be using the files in the next section. Once you unzip, it should look something as shown below.

9.Unzipped ARM Template files

Import the ARM Template to the Stating ADF service

We are now ready to promote services to the next higher environment. Navigate to the ADF service of your next Environment (in my case it is adf-stg-eus-adf) and click on ARM Template and then Import ARM Template as shown below.

10.Import ARM Template

You will be taken back to Azure Management Portal in a new browser window to the Custom deployment as shown below.

11.Build your own Template

As shown in the above screenshot, click on Build your own template in the editor button which takes you to the below screenshot.

12.Load File

Click on the Load File button as shown in the above step and choose the Template file that we have exported from the Development environment’s ADF service.

13.LoadFile-Select Template

As soon as you select the template file it will be loaded as shown below.

14.Loaded the ARM Template file

Once the file is loaded, click on the Save button as shown in the above screenshot. And then, you will be taken to the next step where you can provide Environment specific settings like Storage and Database connection strings of your staging environment as shown below.

Configure Staging properties

As shown in the below screenshot, provide the Environment specific settings.

15.Stage Environment Settings

Once you provide the environment specific configurations, review and then click on Purchase button available in the above step. If everything is configured properly, the ARM templates will be published immediately.

Below is the screenshot of the status once the ARM gets published.

16.Publish ARM Template

And, if you now navigate to the Data Factory instance of Staging Environment then you should see all the pipeline and it’s related components as shown below.

17.Completed Azure Data Factory Pipeline - Staging

Summary:

In this article, we have learned how to promote the ADF Pipeline and related components (but not the ADF service itself) from Development environment to another higher environment say Staging Environment

  1. We have built a simple ADF pipeline
  2. We have created Linked Services and Datasets for Azure SQL database and Storage Account
  3. Exported the ARM Template from the Development Environment
  4. Configure Stage Environment setting in the ARM Template
  5. Import the ARM Template in the Staging Environment.

Next Steps:

  1. You can save the credentials in Key Vault
  2. You can automate the process of creating of the Azure Services (like ADF, Storage and SQL Databases) using ARM Templates.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s