Azure for Data Engineers

Advertisements

Azure Basics

  1. Introduction to Cloud Computing
  2. IaaS Vs PaaS vs SaaS and Serverless
  3. Understand Azure Management Portal

Data Engineer – Roles & Responsibilities

  1. Overview of the Roles and Responsibilities of Data Engineers
  2. Role of DevOps for automation of Data and other components
  3. Understand the Data Engineering Processes

Introduction to Azure Data Services

  1. Relation Databases
  2. NoSQL Databases
  3. Storage Services
  4. ETL Services
  5. Big Data Services
  6. Stream Analytics

Database in the Cloud

  1. Introduction to various options for hosting SQL server on Azure
  2. Create Logical SQL Server and Sql Database
  3. Learn about various Pricing Tiers (DTU vs vCores)
  4. Learn about Backup & Restore methods (Point-in-Time and Long Term Backup Retention)
  5. Understand how to scale-up and scale-down the capacity of SQL Database
  6. Configure Geo-Replication for disaster recovery.
  7. Configure Firewall to whitelist the required IP Address for secure access at both Server level and Database Level.
  8. Manage Sensitive Data with Dynamic Data Masking and Data Encryption
Advertisements

Azure Storage

  1. Create a Storage Account
  2. Create Containers and Blobs
  3. Install & Explore Azure Storage Explorer
  4. Create SAS tokens and understand the benefits of SAS tokens

Azure Data Lake Gen 2

  1. Overview of Azure Data Lake
  2. Understand Azure Data Lake Architecture
  3. Creating an Azure Data Lake Store Gen2 with Portal
  4. Managing Data with Azure Data Lake Store Gen2

Azure NoSQL – Cosmos DB

  1. Overview of Cosmos DB
  2. Understand the Cosmos DB architecture & Hierarchy
  3. Create Cosmos DB Account & Database
  4. Create Cosmos DB Containers and explore Partitions & Indexes
  5. Working with Cosmos DB Capacity (Request Units) and Cost Estimations
  6. Learn how to query data from Cosmos DB

Azure – Deployment as Code using Bicep

  1. Overview of Bicep
  2. Benefits of Bicep
  3. Provision Azure Resources using Bicep
  4. Integrate Bicep Scripts to Azure DevOps
Advertisements

Azure Data Factory

  • Overview of Azure Data Factory
    • Real-time use cases of Azure Data Factory
    • Lab – Create Azure Data Factory Instance
    • Azure Data Factory Core Concepts (Pipelines, Linked Services, Datasets, Activities, Triggers, Integration Runtime (Azure, Self-Hosted & SSIS IR)
    • Lab – A tour of Azure Data Factory User Experience & Interface (Pipeline UI Components, Data Flow UI Components, Monitor UI components, Debug Components, Trigger UI Components, Management Hub)
    • Quiz
  • Explore ADF
    • Lab – Create the first ADF Pipeline
    • Understand the Real-Time Use Case for a Fictions Company and it’s Data
    • Lab – Copy data from Storage Account to Azure SQL Database
    • Lab – Learn different ways of Executing a Stored Procedure (Lookup & Stored Procedure)
    • Lab – Filter the data using Filter Activity
    • Lab – Retrieve the Storage Blobs using GetMetaData Activity
    • Lab – Loop Activities – Iterate through the results of executing a Stored Procedure
    • Lab – Set Variable – Learn how to create, configure and use the Variables in the Pipeline
    • Lab – If Condition Activity – Process records based on a Flag
    • Lab – Fail – Learn how to throw an exception in the Pipeline intentionally
    • Lab – Logic Apps – Send Email using Logic App with Outlook action
    • Lab – Logic Apps – Validate the Logic App Send mail Functionality
    • Lab – Web Activity – Integrate and Invoke Logic Apps for sending E-mails
    • Lab – Azure Data Factory – Learn how to dynamically pass values using Parameters
    • Lab – Execute Pipeline Activity – Learn how to invoke nested Pipeline
    • Scheduled Jobs
    • Understand the differences between Tumbling Schedule and Schedule
    • Lab – Execute Data Factory Pipeline based on Blob Creation Event
    • Lab – Execute Data Factory Pipeline based on Custom Event
    • Lab – Perform Data Transformations using Data Flows
    • Quiz
  • Data Transformations using Data Flows
    • Introduction to Data Flows & Power Query
    • Explore various types of Data Flow Transformations
    • Lab – Remove Nulls
    • Lab – Error Row Handling
  • Data Transformations using Power Query
  • Production Grade pipelines
    • Lab – Copy data from multiple files (stored in Storage Account) to Azure SQL Database
    • Lab- Optimize the performance of
    • Lab – Promote ADF Pipelines to multiple Environments
Advertisements

Azure Data Bricks

  1. Overview of Azure Data Bricks
  2. Create an Azure Data Bricks service.
  3. Create a Spark cluster in Azure Data Bricks.
  4. Learn transformation of data in Azure Data Bricks using Scala
  5. Create Notebook to develop ETL using Data Bricks
  6. Understand how to parameterize Data Bricks pipelines
  7. Invoke Data Bricks pipelines with Azure Data Factory.

Azure Synapse Analytics

  1. Overview of Synapse Analytics
    • Traditional approach to Data Analytics
    • Overview of Data Lakehouse
    • Overview of Synapse Analytics
    • Understand Synapse Analytics Components (Dedicated, Serverless, Spark Pools)
    • Lab: Create Synapse Analytics Workspace
    • Lab: Explore Synapse Components: Compute Options, Storage, RBAC Permissions
    • Understand Synapse SQL Architecture – Control Node, Compute Node, Data Movement Service & Sharding Patterns
    • Quiz
  2. Analyze data with SQL Serverless
    • Overview of SQL Serverless Pools
    • Lab: Working with Database & External Tables with SQL Serverless Pools
    • Lab: Explore & Analyze the data from the SQL Serverless Pool
    • Lab: Create a Power BI report and integrate with SQL Serverless Pools to query data from Data Lakes
    • Quiz
  3. Analyze data with dedicated SQL pools
    • Overview of SQL Dedicated Pools
    • Lab: Create a dedicated SQL Pool
    • Lab: Load the data into SQL Pool using COPY command
    • Lab: Explore & Analyze the data from the SQL Pool
    • Lab: Pause & Resume SQL Dedicated Pool
    • Lab: Load the data into SQL Pool using Polybase 
    • Understand the Sharding Patterns
      • Hash Algorithm
      • Round-Robin
      • Replicate Tables
    • Best Practices
    • Quiz
  4. Analyze data with Apache Spark Pools
    • Overview of Apache Spark Pools and it’s environment
    • Lab: Create an Apache Spark Pool
    • Lab: Load the data into Apache Spark Pool
    • Lab: Explore & Analyze the data using Spark and Notebooks
    • Quiz
  5. Overview of Synapse Pipelines
    • Overview of Synapse Pipelines
    • Understand the components of the Synapse Pipelines
    • Lab: Transform Data using mapping Data Flows
    • Lab: Orchestrating, running and Monitor the Pipelines
    • Quiz

For getting more updates you can join my Telegram group at https://t.me/+s_AafIg0OgpmY2E9

Praveen Sreeram


What People Say

“It was really great experience learning Azure for data engineers. Very informative and highly recommend this course. He has in-depth knowledge of Azure. Most importantly all sessions are practical with real-time use cases. “.

Prasad

Learning from Praveen does not stop after classes are over but an ongoing process. Explains concepts very patiently.

Omkar Gundlur

The course was very informative. He is very patient at answering each and every question. I would recommend Praveen for anyone who is seeking to learn Azure

Chanikya Reddy
Advertisements

Let’s build something together.


One comment

Leave a Reply to Anil Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s