Azure Data Factory – All about publish branch adf_publish

1. ADF - Azure Devops - Task Branches

In the Azure Data Factory – Collaborative development of ADF pipelines using Azure DevOps – Git  article, we have learned how to collaborate with different team members while working with Azure Data Factory.
Each developer creates an individual branch for each of their tasks as shown below.

In the above screenshot, you have Task1 and Task2 branches that were created for two different tasks. And, you also see the other two branches.

Advertisements
  1. master – master is the collaboration branch that is used to merge the code developed by all the developers.
  2. adf_publish – this branch is specific to Azure Data Factory which gets created automatically by the Azure Data Factory service. The adf-publish branch, as the name suggest, it contains the code, specifically, the json code related to all the ADF pipeline and it’s components that are published to the Data Factory service.

Please make sure you publish all the work that you have done in developing the ADF pipeline. If you have not yet published all your work then you might see something as shown below.

1.Azure Data Factory - Azure DevOps - adf_Publish - Partial ARM Templates
Azure Data Factory – Azure DevOps – adf_Publish – Partial ARM Templates

The above screenshot is a version which has the code related to Linked Services. Linked Services will get published to the adf_publish branch immediately (only when you integrate ADF Service with git) after they are created.
And, when you want to publish all the work, then you need to click on the Publish button which will show a popup for confirmation as shown below.

3.Azure Data Factory - Azure DevOps - adf_Publish - Publish Pending Changes
Azure Data Factory – Azure DevOps – adf_Publish – Publish Pending Changes
Advertisements

Once, you click on the Ok button, It will generate the ARM templates, saves them into the adf_publish branch and then all the ADF components will be published to the ADF Instance. Below are the notifications that you will see.

2.Azure Data Factory - Azure DevOps - adf_Publish - Publish Complete
Azure Data Factory – Azure DevOps – adf_Publish – Publish Complete

And, If you have published all the work, then the adf_publish branch should contain all the files as shown below.

Azure Data Factory - Azure DevOps - adf_Publish - Generated ARM Templates
Azure Data Factory – Azure DevOps – adf_Publish – Generated ARM Templates

ARMTemplateForFactory.json – The ARM template file that consists of ALL the resources that we have in the ADF pipeline.

Advertisements

ARMTemplateParameterForFactory.json – The parameters that the ARM Template would need. The parameter files contains all the names and configurations of the services that are environment specific.

FAQs about adf_publish branch in Azure Data Factory.

  1. What is adf_publish?
    1. It is a special branch that get’s created automatically.
  2. What does it contain?
    1. It contains the code (in json format) of all the services that are published to Azure Data Factory service.
  3. Why is adf_publish branch is empty?
    1. The adf_publish branch would contain files in the below scenarios
      1. When you create Linked Services, they will be published immediately
      2. When you publish the Pipeline, all it’s components will be created in the adf_publish branch.
    2. I have not published anything yet why do I see some JSON code already created?
      1. You might have created Linked Services which will be published immediately after you create them.
    3. What is the folder named PartialArmTemplates inside the adf_publish branch?
      1. The PartialArmTemplates folder contains multiple json files each created for individual Linked Services that you create.
    4. Can I use some other branch to save published contents instead of adf_publish?
      1. By default, adf_publish is the default Publish Branch. However you can use any of your branches as the Publish branch.
Advertisements

One comment

  1. Do I ever need to merge adf_publish into master? in VSO i get a notice ‘you have updated adf_publish just now’ and ‘create a pull request’ button exists. Should i periodically do this? I see that adf_publish is a growing list of everything I have done so far. I publish to adf via the UI, in case that matters.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s