Azure Data Factory – 3 ways to Integrate ADF Pipeline with Azure DevOps – Git

In this article, we will learn how to integrate the Azure Data Factory pipeline with git hosted in Azure DevOps.

Prerequisites:

  1. We need to create an Azure Data Factory service
  2. We need to create the following Azure DevOps components
    • Create or use an existing Organization / Account
    • Create or use an existing Project
    • Create or use an existing Repository

Below are the three different ways in which you can integrate Azure Data Factory with git.

Method1:

ADF Get's Started Page - Set up code repository
ADF Get’s Started Page – Set up code repository

Method2:

ADF Pipeline Page - Set up code repository
ADF Pipeline Page – Set up code repository

Method3:

3. Management Hub Page - Set up code Repository
Azure Data Factory – Management Hub Page – Set up code Repository

After choosing any one of the above methods, you will be taken to the below popup where you need to configure all the settings related to git integration.

Component name Description
Repository Type As of this writing, ADF supports git hosted in the following

  1. Github
  2. Azure DevOps (I will select this for this article)
Azure DevOps Account For every Organization that you work with, you will have one Azure DevOps Account / Organization created. Choose an existing one or create a new one.
Project Name Choose an existing Project or a new one.
Git repository name Choose an existing Repository or a new one.
Collaboration Branch Default – master. Feel free to change it to any other branch.
Root Folder Path of the folder within the repository where the JSON of the Data Factory components are saved.
Import existing Data Factory resources to repository

 

If you have already authored the pipeline then you can check this checkbox which will import the JSON of the components into the branch specified in the Branch to Import resources into field
Branch to Import resources into This field is visible only if Import existing Data Factory resources to repository is checked. Otherwise, this field is not visible.

Once you specify all the fields mentioned in the above table in the popup, it should look something as shown below.

4. Repository Settings
Azure Data Factory – Repository Settings

Review all the details in the above screenshot and click on Apply button. Once you click on the Apply button, you will be taken to the below screenshot where you can view all the setting applied and you can also Edit the information or even get disconnected with the Source Control settings.

And, if you navigate to the Azure DevOps repository (in my case it is adf-git-integration-project) then you would see an empty branch master is created as shown below.

6. ADF - Repository View
ADF – Repository View

That’s it. You have learnt 3 ways to Integrate ADF Pipeline with Azure DevOps – Git

Points to Ponder:

  • We can link multiple ADF pipelines with the same Repository.
  • We can have only one Collaboration branch (master branch is commonly used as collaboration branch)
  • We can’t have a separate collaboration branch per environment. One common practice in applications is to have separate branches per environment.

Next Steps:

In real-time projects, developers need to work collaboratively as team. You can learn how to achieve that while developing ADF pipelines by reading my article at Azure Data Factory – Collaborative development of ADF pipelines using Azure DevOps – Git

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s