In this article, we are going to learn about returning a value from a child pipeline and consume the same in the Parent Pipeline using a new feature of Set Variable activity called Pipeline return value feature which is currently in preview

Introduction

Azure Data Factory (ADF) is a cloud-based data integration service that allows you to create, schedule, and orchestrate data workflows at scale. One of the powerful features of ADF is the ability to create nested pipelines, also known as child pipelines, which can be called from within a parent pipeline. Child pipelines allow you to organize and reuse complex sets of activities within your data integration workflows.

In this blog post, we will explore how to return a value from a child pipeline to a parent pipeline in Azure Data Factory. This functionality can be useful when you need to pass data or metadata between pipelines, or when you want to control the execution of the parent pipeline based on the result of a child pipeline. We will walk through a step-by-step tutorial on how to configure your pipelines and activities to return a value from a child pipeline to a parent pipeline.

So, let’s get started!

Return a value from the Pipeline

To return a value from a child pipeline to a parent pipeline in Azure Data Factory, you can use a combination of the Set Variable and Execute Pipeline activities. The Set Variable activity is used to assign a value to a variable, and the Execute Pipeline activity is used to call the child pipeline.

The process of returning a value from a child pipeline to a parent pipeline involves the following steps:

Child Pipeline

1. Use the Set Variable activity in the child pipeline to assign a value to the variable called   strReturnVariable whose variable type is Pipeline return value.

Parent Pipeline

1. Create a variable in the parent pipeline to store the returned value.
2. Use the Execute Pipeline activity in the parent pipeline to call the child pipeline.
3. Map the output of the child pipeline to the variable in the parent pipeline.

In the following sections, we will walk through each of these steps in detail, with screenshots and code examples to help you understand how to return a value from a child pipeline to a parent pipeline in Azure Data Factory.

Step by Step walk through

Create a Child Pipeline named PL_ChildPipeline, add a SetVariable Activity, navigate to the Settings tab, choose Variable type as Pipeline return value as shown below.

As shown in the above screen capture, click on the NEW button to create a return value as shown below.

Now, run the pipeline and view the output of the SetActivity which displays the return value as shown below.

Let’s now create the Parent pipeline named PL_ParentPipeline with an ExecutePipeline activity to invoke the Child Pipeline as shown below.

Let’s now create a Pipeline variable named CaptureReturnValue as shown below. We are going to use this variable to capture the output of the ChildPipeline

Let’s now create a SetVariable Activity to set the value to the CaptureReturnValue variable as shown below.

Now, click on the Value textbox and click on Add Dynamic Content link which opens up the Pipeline expression builder popup where you provide the below expression.

@activity('Execute Child Pipeline').output.pipelineReturnValue.strReturnVariable 

Let’s now view the value of the CaptureReturnValue variable as shown below.

The value “Value to Return” was the value set in the Child Pipeline which can be now access in the Parent Pipeline.

Summary

In this article, we have now learnt a new simple technique that could be used to retrieve a value from the Child Pipeline. As of this writing, the feature is in preview.

Thanks for reading.

Do you like this article? If you want to get more updates about these kind of articles, you can join my Learning Groups

WhatsApp

Telegram