Azure Data Factory – 3 ways to Integrate ADF Pipeline with Azure DevOps – Git

In this article, we will learn how to integrate the Azure Data Factory pipeline with git hosted in Azure DevOps.

Prerequisites:

  1. We need to create an Azure Data Factory service
  2. We need to create the following Azure DevOps components
    • Create or use an existing Organization / Account
    • Create or use an existing Project
    • Create or use an existing Repository

Below are the three different ways in which you can integrate Azure Data Factory with git.

Method1:

ADF Get's Started Page - Set up code repository
ADF Get’s Started Page – Set up code repository
Advertisements

Method2:

ADF Pipeline Page - Set up code repository
ADF Pipeline Page – Set up code repository

Method3:

3. Management Hub Page - Set up code Repository
Azure Data Factory – Management Hub Page – Set up code Repository

After choosing any one of the above methods, you will be taken to the below popup where you need to configure all the settings related to git integration.

Component nameDescription
Repository TypeAs of this writing, ADF supports git hosted in the following

 

  1. Github
  2. Azure DevOps (I will select this for this article)
Azure DevOps AccountFor every Organization that you work with, you will have one Azure DevOps Account / Organization created. Choose an existing one or create a new one.
Project NameChoose an existing Project or a new one.
Git repository nameChoose an existing Repository or a new one.
Collaboration BranchDefault – master. Feel free to change it to any other branch.
Root FolderPath of the folder within the repository where the JSON of the Data Factory components are saved.
Import existing Data Factory resources to repository

 

 

If you have already authored the pipeline then you can check this checkbox which will import the JSON of the components into the branch specified in the Branch to Import resources into field
Branch to Import resources intoThis field is visible only if Import existing Data Factory resources to repository is checked. Otherwise, this field is not visible.

Once you specify all the fields mentioned in the above table in the popup, it should look something as shown below.

4. Repository Settings
Azure Data Factory – Repository Settings
Advertisements

Review all the details in the above screenshot and click on Apply button. Once you click on the Apply button, you will be taken to the below screenshot where you can view all the setting applied and you can also Edit the information or even get disconnected with the Source Control settings.

And, if you navigate to the Azure DevOps repository (in my case it is adf-git-integration-project) then you would see an empty branch master is created as shown below.

6. ADF - Repository View
ADF – Repository View

That’s it. You have learnt 3 ways to Integrate ADF Pipeline with Azure DevOps – Git

Points to Ponder:

  • We can link multiple ADF pipelines with the same Repository.
  • We can have only one Collaboration branch (master branch is commonly used as collaboration branch)
  • We can’t have a separate collaboration branch per environment. One common practice in applications is to have separate branches per environment.

Next Steps:

In real-time projects, developers need to work collaboratively as team. You can learn how to achieve that while developing ADF pipelines by reading my article at Azure Data Factory – Collaborative development of ADF pipelines using Azure DevOps – Git

Advertisements

Do you like this article? If you want to get more updates about these kind of articles, you can join my Learning Groups

WhatsApp

Telegram

4 comments

  1. There are two options available “setup git repository” and “Azure Data Factory” what does this azure data factory option works. How do we make changes when there is no git setup done. Only using Azure data factory option. Then how the pipeline flow works

    Like

Leave a Reply to Deep G Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s