Azure Data Factory – Assign values to Pipeline Arrays in ForEach activity using Append Variable

In this blob post, we will learn how to create a Pipeline variable that can be access anywhere in the pipeline.

In order to create a variable, click anywhere in the Azure Data Factory canvas which opens up the properties of the ADF Pipeline as shown below.

Now, in the Variables tab of the above screen capture, click on the +New button to create a new Variable. As shown below, a new variable is created as shown below.

Provide a meaningful name and choose the Type as shown in the above screenshot. As of this writing, Azure Data Factory supports the following three types of variable

1. String
2. Boolean
3. Array

This variable filesList can be accessed anywhere in the Pipeline. Let’s assume you have a ForEach activity that gets input of some elements from another activity and you want to view the list of all the values that ForEach activity would get. In order to achieve the functionality, we need to add the Append Variable activity inside he ForEach activity.

Click on the Pencil icon which is available in the ForEach activity which opens up an empty canvas as shown below.

Now, Drag & Drop the Append Variable Activity into the ForEach activity canvas as shown below. The next step is to choose the variable from the Variables section as shown below.

Once you configure all the properties and choose the filesList properties, a new textbox with the label Value will be displayed as shown below.

The next step is to supply the value to this activity by extracting the input value which is passed to the ForEach activity. It can be done by clicking on the Add dynamic content button which is shown in the above screen capture.

Once you click on the Finish button in the above popup, you would see something as shown below in the value textbox.

Note: Inside the ForEach activity, if you want to use the inputs passed to ForEach activity, you need to use @item() which provides you the entire object. However, if you want to only access a property of the object you need to refer that property by accessing the property name. In this example, we want to use the name property.

Let’s validate the functionality by clicking on the Debug button. If everything is configured properly, we should see something as shown below.

We have implemented the following.

1. Created a Pipeline Variable of type Array
2. Inside a ForEach activity, created an AppendVariable activity
3. Assign the Pipeline variable to the AppendVariable activity
4. Assign the ForEach input value to the Pipeline array variable
5. Test the Functionality using the Debug button.

This is a sub-post of Azure Data Factory – Implement UpSert using Dataflow Alter Row Transformation

Do you like this article? If you want to get more updates about these kind of articles, you can join my Learning Groups

WhatsApp

Telegram