Azure for Data Engineers

Azure Basics

1. Introduction to Cloud Computing
2. IaaS Vs PaaS vs SaaS and Serverless
3. Understand Azure Management Portal

Data Engineer – Roles & Responsibilities

1. Overview of the Roles and Responsibilities of Data Engineers
2. Role of DevOps for automation of Data and other components
3. Understand the Data Engineering Processes

Introduction to Azure Data Services

1. Relation Databases
2. NoSQL Databases
3. Storage Services
4. ETL Services
5. Big Data Services

Database in the Cloud

1. Introduction to various options for hosting SQL server on Azure
2. Create Logical SQL Server and Sql Database
3. Learn about various Pricing Tiers (DTU vs vCores)
4. Learn about Backup & Restore methods (Point-in-Time and Long Term Backup Retention)
5. Understand how to scale-up and scale-down the capacity of SQL Database
6. Configure Geo-Replication for disaster recovery.
7. Configure Firewall to whitelist the required IP Address for secure access at both Server level and Database Level.
8. Manage Sensitive Data with Dynamic Data Masking and Data Encryption

Azure Storage

1. Create a Storage Account
2. Create Containers and Blobs
3. Install & Explore Azure Storage Explorer
4. Create SAS tokens and understand the benefits of SAS tokens

Azure Data Lake Gen 2

1. Overview of Azure Data Lake Service ( ADLS ) Gen2
2. Comparison with Azure Storage Account.
3. Quick History of ADLS Service
4. Lab: Creating an Azure Data Lake Store Gen2 with Portal
5. Convert Raw JSON files into Parquet files and store into ADLS
6. Understand Modern Data Analytics Solution Architecture
7. Lab: Azure Data Factory integration with ADLS

Azure NoSQL – Cosmos DB

1. Overview of Cosmos DB
2. Understand the Cosmos DB architecture & Hierarchy
3. Create Cosmos DB Account & Database
4. Create Cosmos DB Containers and explore Partitions & Indexes
5. Working with Cosmos DB Capacity (Request Units) and Cost Estimations
6. Learn how to query data from Cosmos DB

Azure – Deployment as Code using Bicep

1. Overview of Bicep
2. Benefits of Bicep
3. Provision Azure Resources using Bicep
4. Integrate Bicep Scripts to Azure DevOps

Azure Data Factory

• Overview of Azure Data Factory
  • Real-time use cases of Azure Data Factory
  • Lab – Create Azure Data Factory Instance
  • Azure Data Factory Core Concepts (Pipelines, Linked Services, Datasets, Activities, Triggers, Integration Runtime (Azure, Self-Hosted & SSIS IR)
  • Lab – A tour of Azure Data Factory User Experience & Interface (Pipeline UI Components, Data Flow UI Components, Monitor UI components, Debug Components, Trigger UI Components, Management Hub)
  • Quiz

• Explore ADF
  • Lab – Create the first ADF Pipeline
  • Understand the Real-Time Use Case for a Fictions Company and it’s Data
  • Lab – Copy data from Storage Account to Azure SQL Database
  • Lab – Learn different ways of Executing a Stored Procedure (Lookup & Stored Procedure)
  • Lab – Filter the data using Filter Activity
  • Lab – Retrieve the Storage Blobs using GetMetaData Activity
  • Lab – Loop Activities – Iterate through the results of executing a Stored Procedure
  • Lab – Set Variable – Learn how to create, configure and use the Variables in the Pipeline
  • Lab – If Condition Activity – Process records based on a Flag
  • Lab – Fail – Learn how to throw an exception in the Pipeline intentionally
  • Lab – Logic Apps – Send Email using Logic App with Outlook action
  • Lab – Logic Apps – Validate the Logic App Send mail Functionality
  • Lab – Web Activity – Integrate and Invoke Logic Apps for sending E-mails
  • Lab – Azure Data Factory – Learn how to dynamically pass values using Parameters
  • Lab – Execute Pipeline Activity – Learn how to invoke nested Pipeline
  • Scheduled Jobs
  • Understand the differences between Tumbling Schedule and Schedule
  • Lab – Execute Data Factory Pipeline based on Blob Creation Event
  • Lab – Execute Data Factory Pipeline based on Custom Event
  • Lab – Perform Data Transformations using Data Flows
  • Quiz
• Data Transformations using Data Flows
  • Introduction to Data Flows & Power Query
  • Explore various types of Data Flow Transformations
  • Lab – Remove Nulls
  • Lab – Error Row Handling
• Data Transformations using Power Query
• Production Grade pipelines
  • Lab – Copy data from multiple files (stored in Storage Account) to Azure SQL Database
  • Lab- Optimize the performance of
  • Lab – Promote ADF Pipelines to multiple Environments

Azure Data Bricks

1. Overview of Apache Spark, DataBricks and Azure Data Bricks
2. Create an Azure Data Bricks service.
3. Explore Core Components of Azure Data Bricks
4. Create a Spark cluster in Azure Data Bricks.
5. Create Notebook to develop ETL using Data Bricks
6. Azure DataBricks Utilities
   • File System Utilities
   • Notebook Utilities
   • Secret Utilities
   • Library Utilities
   • Jobs Utilities
7. Learn transformation of data in Azure Data Bricks using PySpark
8. Create Mount Points to retrieve data from a Data Lake Gen2 Account
9. Learn how to write Re-usable code using UDF
10. Integrate Azure Key Vault to securely store and access sensitive information from PySpark
11. Integrate and Persist Data to SQL Database using JDBC Drivers
12. Integrate and Persist Data to Azure Synapse Dedicated Pool using Synapse Connector.
13. Understand how to parameterize Data Bricks pipelines.
14. Invoke Data Bricks pipelines with Azure Data Factory.
15. Integrate Azure Data Bricks Notebooks with Azure DevOps Git for Version Control
16. Implement CI/CD process for promoting Code with Higher Environments (Staging and Production) using Azure DevOps

Azure Synapse Analytics

1. Overview of Synapse Analytics
   • Traditional approach to Data Analytics
   • Overview of Data Lakehouse
   • Overview of Synapse Analytics
   • Understand Synapse Analytics Components (Dedicated, Serverless, Spark Pools)
   • Lab: Create Synapse Analytics Workspace
   • Lab: Explore Synapse Components: Compute Options, Storage, RBAC Permissions
   • Understand Synapse SQL Architecture – Control Node, Compute Node, Data Movement Service
   • Quiz
2. Analyze data with SQL Serverless
   • Overview of SQL Serverless Pools
   • Lab: Working with Database & External Tables with SQL Serverless Pools
   • Lab: Explore & Analyze the data from the SQL Serverless Pool
   • Quiz
3. Analyze data with dedicated SQL pools
   • Overview of SQL Dedicated Pools
   • Lab: Create a dedicated SQL Pool
   • Lab: Load the data into SQL Pool using COPY command
   • Lab: Explore & Analyze the data from the SQL Pool
   • Lab: Pause & Resume SQL Dedicated Pool
   • Lab: Load the data into SQL Pool using Polybase 
   • Quiz
4. Analyze data with Apache Spark Pools
   • Overview of Apache Spark Pools and it’s environment
   • Lab: Create an Apache Spark Pool
   • Lab: Load the data into Apache Spark Pool
   • Lab: Explore & Analyze the data using Spark and Notebooks
   • Quiz
5. Overview of Synapse Pipelines
   • Overview of Synapse Pipelines
   • Understand the components of the Synapse Pipelines
   • Lab: Transform Data using mapping Data Flows
   • Lab: Orchestrating, running and Monitor the Pipelines
   • Quiz

Do you like this article? If you want to get more updates about these kind of articles, you can join my Learning Groups

WhatsApp

Telegram

What People Say

---