Azure Data Factory Training In Hyderabad
(adf training in Hyderabad)
with
100% Placement Assistance

Azure Data Factory Training in Hyderabad - Online Classes

Azure Data Factory (ADF) Curriculum

Azure Data Engineer Services Introduction Part 1

Azure Data Engineer Services Introduction Part 2

Azure Data Engineer Services Introduction Part 3

Azure Data Engineer Services Introduction Part 4

Azure Data Engineer Services Introduction Part 5

Azure Data Engineer Services Introduction Part 6

ADF introduction 

Difference between ADF version 1 and Version 2. 

  • DataFlows and Power Query are new features in Version 2
  • DataFlows are for data transformations
  • PowerQuery is for Data preparations and Data Wrangling Activities.
  • Building blocks of ADF
  • -> PipeLine
  • -> Activities
  • -> Datasets
  • -> Linked Service
  • -> Integration
  • RunTime
  •   Auto Integration Runtime
  • Self Hosted Integration runtime
  • ADF Session 8:SSIS Integration Runtime. 

 More on Technical differences between ADF version 1 and Version 2 – Part 1

  • More on Technical differences between ADF version 1 and Version 2 – Part 2
  • Introduction to Azure Subscriptions
  • Types of Subscriptions

 -> Free Trial

-> Pay-AS-You-Go

  •  Why Multiple subscriptions are required?
  • What are resources and Resource Groups? 
  • Resource Group Advantages
  • Why Multiple resource Groups need to be created ?
  • What are regions
  • Region Advantages.
  •  Create  Storage Account with Blob Storage feature
  • Converting Blob storage feature as DataLake Gen2 feature
  •  Create Storage Account with Azure Data Lake Gen2 features.
  • How to Enable Hierarchical name space. 
  • Creating Containers
  • Creating sub directories in the Container of Blob Storage. 
  • Creating sub directories in Container of DataLake Gen2 Storage
  • Uploading local files into Container/sub directories.
  • When  is ADF required?
  • Create Azure SQL and Play with Azure Sql – Part 1
  •  Azure sql as OLTP
  •  Create Azure Sql Database 
  •  Create Azure Sql Server
  •  Assign Username and Password for Authentication
  • Launching Query Editor
  • Adding Client IP address to FireWall Rule settings
  • Create table
  • Insert rows into table
  • Default schema in azure sql
  • Create schema
  • Create table in user created schema
  • Loading query data into to table
  • Information_schema.tables
  • Fetching all tables and views from database
  • Columns of Information_schema.tables

-> TABLE_CATALOG

-> TABLE_SCHEMA

-> TABLE_NAME

-> TABLE_TYPE

  • Fetching Only tables from database
  • How to create Linked Service for Azure data lake
  • Possible errors while creating  linked service for Azure datalake account.
  • How to solve errors for Linked Service for Azure data Lake
  • Two ways to Solve Linked Service Connection Error.

 -> Enable Heirarchical namespace for Storage Account

 -> Disable Soft Delete options BLOB and CONTAINER

  • How to create datasets for Azure data lake file and containers
  • Your First ADF pipeline for Datalake to data lake  file loading
  • COPY DATA  activity used in pipeline
  •  Configuring Source dataset  for Copy Data Activity
  • Configuring Sink dataset for Copy Data Activity
  • Run the pipeline:
  • Two ways to Run a Pipeline. 

 -> Debug mode

-> Trigger mode

  • Two Options in Trigger mode

 -> On demand

-> Scheduling

  • How to load  data from Azure data lake to Azure sql table. 
  • Create linked service for Azure sql Database 
  • Error resolving , while creating Linked service for Azure Sql Database 
  • Create a dataset for Azure Sql Table.
  • Create a pipeline to load data from Azure data lake to Azure sql Table
  • Helped activities 
  • Copy activity 
  •  If   data lake file schema, and azure sql table schema are different,    How to load Using Copy Data Activity. 
  • Perform ETL With “Copy Data” Activity
  • Copy Data Activity With “Query” Option
  • Loading Selected Columns and Matching Rows with Given Condition , From Azure Sql Table to Azure Data Lake
  • Creating new fields based on existed Columns  of table and Load into Azure Data Lake 
  • Problem Statement :  A file  has   n fields in the header , but data has n+1 field values . How to Solve this problem in Copy Data Activity
  • Solution to Above Problem Statement. Practical implementation. 
  • Get Metadata – part 1
  • Get Metadata Field List Options for Folder
  •  “Exists” Option of Field List 
  • Data type of Exists field in  “Get Metadata”  output as Boolean(true/false)
  • “Item Name” Option of Field List
  • “Item Type “ Option of Field List
  • “Last Modified” Option of Field List
  • “Child Items” Option of Field List
  • Data Type of “child Items” field  in Json Output.
  • What is each element of “Child Items” called?
  • Data type of  item of “ChildItems” Field
  • What are subfields of each item in “childItems” Field
  • Get Metadata Activity – Part 2
  • If Input dataset is file, what are Options of “Field List” of Get Metadata.
  • How to Get the number of Columns in a file?
  • How do you make sure the given file exists ?
  • How to get the file name, which is configured for the dataset. ?
  • How to get a dataset  back end data object type(file or folder)  ?
  • How to Know when a file was last modified?
  • How to get File size ?
  • How to get File structure(schema) ?
  • How to get all above information of a file/folder/table  with a single pipeline run
  • If the input dataset is an RDBMS table, what are options of “Field List” of Get Metadata Activity. ?
  • How to get the number of columns of a table ?
  • How to make sure the table exists in the database ?
  • How to get the structure of a table ?
  • In this Session, you will learn answers for all above questions practically.
  • Introduction to “Get Metadata” activity. 
  • How to fetch File System information Using “ Get metadata” Activity. 
  • How “Get metadata” activity writes output in “JSON” format. 
  • How to Configure Input dataset for “Get Metadata” activity. 
  • What is the “Field List” for “Get Metadata” activity?
  • Small introduction to “Field List” options. 
  • Importance of “Child Items” option of “Field List” in “Get Metadata” activity.
  • How to Check and understand  output of “Get Metadata” activity.
  • “childItems” field as JSON output of “Get Metadata”.
  • “childItems” data type as Collection  as data type As Array.
  • What is each element of “childItems” output
  • What is the “exists” option in “Field List”. 
  • Introduction to “Filter”  activity. 
  • Problem statement: in a container of a Azure Datalake storage account , there are 100s of files, some files related with adf , and some files related with employees, and some files related with sales and some files related with others like log.  
  • How to fetch only required files from Output of  “get metadata” activity  ?
  • How to place “Filter”  activity in the pipeline. 
  • How to Connect “Get metadata” activity and “Filter” activity. 
  • What happens , If we don’t  connect two activities? (bcoz, this is the first scenario with multiple activities in a single pipeline). 
  • How to pass output of “Get metadata” Activity to “Filter Activity”
  • “Items” field of “Filter” Activity. 
  • What is the “@activity()” function  and “@activity().output” .
  • Above Output produces a lot of fields , How to take specific fields as input to filter activity. 

Example :  @activity(“get metadata1”).output.childItems

  • How to avoid unnecessary information  to next  subsequence Activity
  • Configuring “Condition” field of “Filter” activity. 
  •  How to access each element of the output of “childItems” field is @item() function. 
  • @item() output as nested json record.  How to access each field. 

Example,    @item().name,    @item().type  

  • @startswith()   function  example
  • How to check output of “Filter” activity   
  • What is the Field name of  “Filter” activity , in which required information is available. 
  • @not() function example 
  • ——————————————–
  • @item() output as nested json record.  How to access each field. 

Example,    @item().name,    @item().type  

get metadata + filter activities and how to apply single condition how to  apply multiple conditions all the things  

there are functions  

Functions  

@equals 

@greater( ) 

@greaterOrEquals( ) 

@or (C1,C2 …….) 

@and (C1,C2 …..) 

@not (equals( ))

  • Task : Get Metadata – fetch only files , and reject folders of Given Container with “Get Metadata” and “Filter”  Activity.
  • steps   to achieve Above Task.
  • Step1:  Create a Pipeline and Drag Get MetaData Activity
  • Step2 : Configure Input DataSet of Container
  • Step3:  Add a Field List  With “child Items” Option.
  • Step4:  Add “Filter” activity to “Get Metadata”
  • Step5:  Configure “Items”  Field , Which is input for “Filter” activity  From Output of “Get Metadata”
  • Step6:  Configure Filter Condition  to  take only Files.
  • Step7: Run the Pipeline
  • Step8:  Understand Output of  “Filter” Activity. 
  • In which  field of Json, Filter output is available ?
  • What is data type of  Filtered  output 
  • In this task you will work with below  ADF expressions
  • @ activity()  Function .. 
  • @ activity(‘Get Metadata1’).output
  • @ activity(‘Get Metadata1).output.childItems
  • @ item().type
  • @ equals(item().type, ‘File’)
  • After Completion of this session you will be able to  implement all above steps  , and can know answers for above questions , and  able to use adf expressions practically. 
  • Scenario :  with  GetMetadata and Filter Activities – Part 1

Scenario :  with  GetMetadata and Filter Activities – Part 2

  • Task : bulk load of files one storage account to another storage account   (from one container to other container )
  •  Helped activities 

 -> get metadata 

 -> foreach

 -> copy data

  • Bulk load of files from one storage account to Other storage account with “Wild card “ option.
  • Bulk load of files from  one storage account to Other Storage account with “Get Metadata” and “ForEach” and “Copy Data” Activities.
  • Data Set Parameterization 
  • Bulk load of files  into Other Storage Account  with “Parameterization”
  • copy only files which starts emp into target container Using Get MetaData , Filter , ForEach, Copy Data Activities – [ in target container, file name should be same as source ] 
  • load multiple files into the table.(Azure Sql) 
  • Helped activities 
    • get metadata + filter + foreach  + copy data activities
  • Conditional split . 
  • Conditionally distributing files (data) into different targets (sinks) two ways.
    • using filter 
    • using if condition activity  
    • Conditional Split implementation with “Filter” activity
    • Problem with Filter , explained. 
  • Conditional split with “if condition”. 
  • Helped activities
    • get metadata  + filter +  for each + if condition
  • Lookup activity 
  • migrate all tables of the database into a data lake. with a single pipeline. 
  • Helped activities

lookup activity for each activity  COPY DATA ACTIVITY WITH PARAMETERIZATION

  • Data Flows Introduction. 
  • ELT (Extract Load and Transform)  
  • Two DataFlows in ADF 
  • Dataflow activity in pipeline.
  • Mapping Data Flows  
  • Configuring Mapping Dataflows as Data Flow Activity in PipeLine
  • Introduction to Transformations
  • Source 
  • Sink
  • Union
  • Filter
  • Select
  • Derived Column
  • Join etc. 
  • Difference between source , sink of “Copy Data” activity  And source , sink of “Mapping Data Flows”.
  • Source as (data lake) Sink as (SQL table) ——-> by using dataflow  source and sink transformation 
  • Extract data from RDBMS to Datalake ———–> Apply Filter ——> sink
  • Assignment1 
    • SQL Table to Datalake
    • Helped Transformation (Souce) 

Sink (Dataflow Activity)

  • Filter transformations 
  • Helped Transformations

(Source) Filter Sink (Dataflow Activity)

  • Select transformations 
  1. Rename columns 
  2. Drop columns 
  3. Reoder the columns 
  • Helped Transformation 

(Source ) Select Sink (Dataflow Activity)

  • Derived column Transformation Part1 

1.You can generate new column with given expression 

2.You can update existed column values 

  • Helped Transformation 

Derived column 

Select Sink (Dataflow Activity) 

  •  How to clean/handle null values in data.
  • why should we clean nulls 

1.computational errors 

2.data loading errors into target table synapse table.

  • Cleaning names(cleaning means not always replace nulls with some  constant value,  cleaning means transform data according to business tool).formating of  the data (names) 
  • Helped transformation 

Derived column

  • Generate new columns,with conditional transformation.  two options: 

 1.iif() —->nested ifs, 

 2.case() 

  • Helped transformation 

Derived column 

  • Conditional transformation with case() function. 

emp—->derived columns—–>select—–sink 

  • Helped transformation 

derived column 

select  sink  (Dataflow activity) 

  • Union transformation  part 1.
  • Merging Two different files with the same schema by using two sources using the union transformation, And writes output into a single file. 
  • Helped transformation 

Two sources

Union 

Sink 

(Dataflow activity)

  • Merging three different files with different schemas by using derived column transformation, select transformation and union transformation,  finally we get a single file with a common schema. 
  • Helped transformation 

Derived column 

union 

select 

(source) 

sink (Dataflow activity)

  • Two different files by using derived column transformation,  union transformation and aggregate transformation,  to get branch1 total and branch 2 total. 
  • Helped transformations 

(Source) 

Derived column  

Union 

Aggregate 

*(After this Watch session 70, for more on Unions).

  • Joins transformations 5 types  

 1.Inner join  

 2.Left outer join 

 3.Right outer join 

 4.Full outer join 

 (5.Cross jons) 

  • Helped transformation 

Joins 

Select 

Sink 

(Dataflow activity) 

ADF session 47:

Full Outer Join Bug fix

Full Outer Join Bug fix

How to join more than two datasets (example 3 datasets).

  • Inter linked scenario related to 25th session. 

treat, dept as project 

task is ” summary Report” 

active employes (who already engaged into project -> 11,12,13  projects,-> these projects total salary budget (bench team 20,21 total salary ,bench project ) 

  • Helped transformations 

Joins 

Select  

Derived column 

Aggregate 

Sink 

(Dataflow activity)

  • To use full outer join advantage (complete information,no information  missing) 

task~1 : Monthly sales report by using derived column transformation,  aggregate transformation. 

  • Helped transformation 

Derived column  

Aggregate 

sink

  • Task~2 : Quarterly sales report by using derived column transformation,  aggregate transformation. 
  • Helped transformation 

Derived column  

Aggregate 

  • Task~3: Year as primary group,quarterly as sub group,sales report.
    • Helped transformation 

Filter activity 

Derived column 

Aggregate 

Sort 

Sink 

(Dataflow activity)

  • Task~4 : Comparing Quartely sales report: Comparing Current Quarter sales with its Previous Quarter Sales. 
  • Helped transformations 

Source  

Join  

Select 

Derived column

  • A real time scenario on Sales data Analytics – Part 1

A real time Scenario on Sales data Analytics – Part 2

  • More on Aggregation Transformation.
  •  Configuration Required  for  Aggregation Transformation
  • Entire Column aggregations.
  • Entire Column Multiple Aggregations.

-> Sum()

-> Count()

-> max()

-> min()

-> avg()

  • Single grouping and Single Aggregation
  • Single Grouping and Multiple Aggregations
  • Grouping by Multiple Columns with Single aggregation
  • Grouping by Multiple Columns with Multiple Aggregations.
  • Finding range Aggregation by adding  “Derived Column” transformation to “Aggregation” Transformation. 
  • Conditional Split of data. 
  • Distributing data into multiple datasets based on given condition
  • Split on options:
  1.  First Matching conditions
  2. All Matching conditions
  • When to use “First Matching Conditions”  and When to use “All Matching Conditions”
  •  An Example.        
  • An Use Case on “First Matching Condition” option  of “Conditional Split” Transformation in Mapping Data Flows with Sales data.

An Use Case on “Matching All Conditions” option of “Conditional Split” Transformation in Mapping Data flows with Sales data.

Conditional split with cross join transformation   by using matrimony example.

  • Lookup with multiple datasets 
  • Helped transformations 

Source 

Lookup 

Sink 

(Dataflow activity) 

  • Lookup with more options
  • products as primary stream 
  • transformation as lookup stream 
  • Helped transformations 

Products 

Lookup

Broadcasting

Partitioning part 1

Partitioning part2

Partitioning part3

Exists transformation – part 1

  • Helped transformation 

(Customers) 

Exists 

 Exists transformations  – part 2

  • Helped transformation 

(Source) 

(Customers) 

Exists 

Sink 

(Dataflow activity)

  •  Finding Common Records, Only Records available in first dataset, Only Records available in Second Dataset. 
  • All Records except Common records in First and 2nd Datasets  in Single DataFlow
  • How to capture data changes from source systems to Target Data warehouse  Systems. 
  • Introduction to SCD (Slowly Changing Dimensions)
  • What is SCD Type 0 and Its Limitation. 
  • What is Delta in Data of Source System.
  • What is SCD Type 1 and Its Limitation 
  • What is SCD Type 2 and How it tracks History of specific attributes of source data. 
  • Problem with SCD Type 2
  • Introduction to t SCD Type 3
  • How It solves problem of SCD Type 2
  • How SCD type 3 maintains recent History track
  • Limitation of SCD Type 3. 
  • Introduction to SCD Type 4 
  • How SCD Type 4 will provide complete Solution to SCD type 2.
  • (Remember no SCD 5)
  • Introduction to SCD Type 6 and Its benefits.
  • How  data transformations are done in ADF1 (with out dataflows)
  • Aggregate Transformation , Sort Transformation with following examples
  1. Single grouping with single aggregation 
  2. Single Grouping with multiple aggregations
  3. Multi Grouping with Multiple Aggregations. 
  4. Sort with single column
  5. Sort with multiple columns. 
  •  What is pivot transformation?
  • Difference between Aggregate transformation and Pivot Transformation 
  •   Implement pivot transformation in dataflow
  •     How to clean pivot output
  •    How to call multiple dataflows  in a single pipeline. 
  •  Why we used multiple dataflows in one single pipeline. 
  • Assignment on   Join, aggregate, pivot transformations. 
  • Finding Occupancy based on salary. 
  • Unpivot transformation. 
  • What is should be the input for Unpivot (pivoted output file). 
  • 3   configurations : 

  -> Ungrouping column

 -> Unpivoted column.  (column names to as column values)

 →  aggregated column expression (   which row aggregated values to be turned  as column values) 

  • Difference between  output of  “aggregate” and “unpivot” transformation. 
  • What additionally unpivot  produces. 
  • Use case of Unpivot
  • Surrogate Key transformation, 
  • Why we should use Surrogate key.
  • Configuring Starting Integer Number for Surrogate Key
  • Scenario :  your bank is “ICICI”, for every record Unique CustomerKey should be generated as ICICI101 for the first customer, ICICI102 for 2nd customer . But the Surrogate key gives only integer value as 101 for first and 102 for 2nd. How to handle the given scenario , which is a combination of string and integer.
  • Rank Transformation in dataflows – Part 1
  • Why sorting data is required for Rank transformation 
  • Sorting options as “Ascending and Descending”
  • When to use the “Ascending” option for Ranking. 
  • When to use the “Descending” option for Ranking. 
  •  Dense Rank and How it Works
  • Non Dense Rank (Normal Rank) and How it Works. 
  • Why we should not use “surrogate key” for Ranking. 
  • What is difference between Dense Rank and Non Dense Rank(Normal Rank)
  • Limitations of Rank Transformation
  • Implementing Custom Ranking With a Real time scenario
  • Custom Ranking  implementation with Sort,  Surrogate key, new Branch , aggregate, Join transformations. 
  • Scenario : 

   -> problem statement:  if  a school has 100 students, one student got 90 marks , remaining all 99 students failed and scored  10 marks. The Rank transformation of ADF dataflows gives 1 rank for those who scored 90 , and 2 rank for failed students , who scored 10 marks.  But the school management wanted to give a gift to the top 2 Rankers ( 2nd rankers failed and got equal least score , so that all students will get a gift).  How to handle this scenario. 

  • Window transformation part 1
  • Window transformation. 
  • Cumulative Average
  • Cumulative Sum
  • Cumulative Max
  • Dense Rank for each partition. 
  • Making all rows as Single Window and apply Cumulative Aggregation. 

Window transformation part 2

Window transformation part 3

Window transformation part 4

Window transformation Part 5

Window transformation Part 6

Custom Rank Implementation with Window Transformation,

  • Parse transformation in Mapping DataFlows. 
  • How to handle and parse string collection ( delimited string values )
  • How to handle and parse xml data. 
  • How to handle and parse json data
  • Converting Complex Json nested structures into CSV/Text file.
  • Complex data processing  (transaformations). 
  • How  parse nested Json records
  • How to flatten  array of values into multiple rows. 
  • If data has complex structures, what are supportive data store formats .
  • How to write into json format. 
  • How to write into flatten file format( csv).
  • Transformation used to process data. 
  1. Parse
  2. Flatten
  3. Derived Column
  4. Select 
  5. Sink.
  • Reading Json data (form vertical format)
  • Reading a single Json record(document)
  • Reading array of documents . 
  • Converting complex types into String using Strigify and derived column transformations. 
  • Used transformations in Data Flow. 

-> source 

->stringify

->derived column

Expression :  toString(complexColumn)

         → select

         -> sink.

  • Assert Transformation
  • Setting Validation Rules for Data. 
  • Types of Assert:

-> Expect True

-> Expect Unique

-> Expect Exists

  • isError()   function to  validate a record as   “Valid”   or “Invalid”.
  • Assert Transformation part 2.
  • Assert Type “Expect Unique”
  • Rule for Null values
  • Rule for ID ranges.
  • Rule for ID uniqueness
  • hasError() function  to  identify  which rule is  failed. 
  • Difference Between isError() and hasError()
  • Assert Transformation part 3.
  • How  and why to configure Additional streams
  • Assert type “Expect Exists”
  • How to validate a record reference available in any one of multiple additional streams. (A scenario implementation).
  • How to Validate a record reference available in all multiple additional streams.(A scenario Implementation),

Incremental data loading (part 1)

Incremental data loading (part 2)

  • Incremental data loading (part 3).
  • Implementation of Incremental load(delta load) for multiple tables with a single pipeline.

Deduplication part 1

Deduplication part 2

Deduplication Part 3 

Solving Bugs of Incremental Loading . – Part1

Solving Bugs of Incremental Loading – Part2

  • AlterRow Transformation Part – 1 
  •  Removing List of Records  from Sink of a Specific criteria (Table)
  • AlterRow Transformation part- 2
  • Removing a given List of Records from Sink of no specific Criteria .
  • AlterRow Transformation Part – 3. 
  • Update given List of Records in Sink. 
  • SCD (Slowly Changing Dimensions Type 1 ) Implementation. 
  • Alter Row Transformation with UpSert Action.
  • A realtime Assignment (Assignment1 and Assignment 2) on SCD. 
  • Incremental Load and SCD combination 
  • How to implement SCD before “Alter Row” Transformation not introduced in DataFlows. 
  • How exists transformation helpful. 
  • How to implement SCD before ADF version 2 features of DataFlows. 
  • How Stored procedures helpful and implement SCD.
  • Combining rows of multiple tables with different schemas with a single Union Transformation. 
  • Problem with Multiple Unions in DataFlow. 
  • Types of Sources and Sinks
  • Source Types:

-> Dataset

-> Inline

  • Sink Types:

-> Dataset

-> Inline

-> Cache

  • When to use dataset and Inline
  • Advantage of  Cache as Reusage of output of Transfomation 
  • How to write output into Cache. 
  • How to Reuse the Cached output. 
  • Scenario :   Generating Incremental IDs based on Maximum ID of Sink DataSet – Part 1
  • Scenario: Generating Incremental IDs based on Maximum ID of Sink DataSet – Part 2
  • Writing outputs into JSon format  
  •  Load Azure SQL data into Json File.
  • Cached Lookup    in Sink Cache .   Part 1
  • Scenario : 

 Two Source Files, 1. Employee 2. Department

  Common Column in both dno(department number) which is as Joining Column or Key Column. 

  • Task: Without using Join Transformation , Lookup Transformation  , we need to join two datasets to increase performance using “Cache Lookup” of Cache Sink Type. 
  • Other alternatives of this Cached Lookup.
  • How to configure Key Column for Cached Lookup
  • How to access values of Lookup key in Expression Builder. 
  • lookup() function  in Expressions
  • sink#lookup(key).column
  • Cached Lookup in Sink Cache – Part 2
  • Below Scenario will teach you, how cached output is used by multiple transformations. 
  • Scenario:

    Single Input file : Employee,

            Two Transformations required.

  1.   Foreach employee , find his salary occupancy in his department
  2. Foreach employee, find his average salary status in his department as 

“Above Average” or “Below Average”.

             For these two transformations    common input sum(), avg() aggregations with 

group by department number . This should be sent to Cache Sink   as Cached

 Lookup.

   Each transformation output should be in separate Output file. 

  • In above Dataflow, What flows are executed in  parallel and What flows executed in sequence. 
  • How spark knows dependency between flows. [Using DAG (Direct Acyclic Graph ) engine ]
  • Scope of “Sink Cache” output. [ problem statement ]
  • Using “  Multiple Cache Sink  “ outputs in Single Transformation. 
  • Scenario:

Employee table has id, name, salary, gender, dno, dname, location columns with two records 101 and 102 ids.

New employees are placed in datalake file  newdata.txt with  name,salary, gender, dno  fields.    

Department name, department location fields are available in department.txt file of datalake. 

 Insert all new employees into Employee table of azure SQL , with incremental ID based on maximum id of the Table  along with department name and location  .

  • Sink1 : write max(id) from Employee table  (cache sink)
  • Sink2 :  write all rows of department.txt into  Cache Sink. and Configure dno as Key Column of Lookup
  • Read data from new employees file and generate next employee id with help of surrogate key and sink1 output. 
  • Generate lookup columns department name, location from Sink2 and Load into Target Employee table Azure SQL. 
  • How two Different DataFlows  exchanging Values – Part 1. 
  • Scope of Cache Sink of a DataFlow.
  • What is Variable
  • What is Difference between Parameter and Variables
  • How to Create a Variable in Pipeline
  • Variable Data Types:

-> String

-> Boolean

-> Array

  • Create a  DataFlow with Following Transformation Sequence to find Average of Column
  1.  Source 
  2.  Aggregate (for finding Average)
  3. Sink Cache (to write Average of column)
  • How to pass Cache Sink output to Activity Output. How to Enable this feature
  • Create a Pipeline to Call this Data Flow which writes into Sink Cache
  • How to Understand Output of DataFlow Activity which writes in Sink Cache and Writes as Activity Output
  • Fields of  DataFlow Activity output.
  1. runStatus
  2. Output
  3. Sink
  4. value as Array
  • “Set Variable” Activity 
  • How to Assign Value to Variable using “Set Variable” Activity.
  • How two Different DataFlows  exchanging Values – Part 2
  • How to Assign DataFlow Sink Cache Output  to Variable .
  • Dynamic Expression to Assign Value to Variable Using “Set Variable” Activity.
  • @activity(‘dataflow’).output
  • runStatus.output.sink.value[].field
  • Understand Output of “Set Variable” Activity.
  • How two Different DataFlows  exchanging Values – Part 3
  • DataFlow Parameters
  • How to create Parameter for DataFlow
  • Fixing Default Value for DataFlow Parameter
  • How to access DataFlow Parameter
  • $<parameter_name>
  • Accessing variable value  from pipeline into DataFlow Parameter
  • @variables(‘<variable_name>’)
  • Dynamic expression to access variable value in DataFlow Parameter
  • From  above 3 sessions, you will be learning how to pass output of cache sink output to other dataflows .
  • Categorical Distribution of Data into Multiple Files (foreach data group one separate file Generation ). → Part 1
  • Scenario:   There is department name column with “Marketing”, “Hr”, “Finance”   values.    But these values are duplicated.  All rows related with these 3 categories . We need to distribute data  dynamically into 3 categories.
  • Example:   all rows related with marketing department into “marketing.txt” file.
  • Find Unique Values in Categorical Column and Eliminate Duplicate Values
  • Convert Unique Column into Array of String
  • collect() function
  • Aggregate Transformation using Collect() function
  • Write Array of String into Sink Cache
  • Enable  Cache  to  “Write to Activity Output” 
  • Create a pipeline, to Call this Dataflow. 
  • Create a variable in pipeline with Array Data type 
  • Access    DataFlow Cache  Array   into   Array Variable  with “Set Variable” activity Using “Dynamic Expressions”.
  • Categorical Distribution of Data into Multiple Files (foreach data group one separate file Generation ). → Part 2
  • Array variable to Foreach
  • Configuring  input (value of Array Variable)  for Foreach Activity 
  • DataFlow sub Activity under Foreach.
  • Passing Foreach Current item to Dataflow parameter 
  • Apply Filter  with DataFlow Parameter. 
  • DataFlow Parameter as file name in Sink Transformation,
  • SCD (Slowly Changing Dimensions ) Type 2 – Part 1
  • Difference between SCD Type 1 and SCD Type 2
  • What is Delta of Source data
  • How to Capture Delta (Solution : Incremental loading)
  • Behavior of SCD Type 1
  • If Record of Delta existed in  Target, what happens for SCD Type1
  • If Record of Delta does not exists in Target, What happens for SCD Type 1
  • How to capture Deleted records from source into Delta. 
  • Behavior of SCD Type 2
  • What should happen for Delta of Source in Target
  • What should happen for old records of Target when Delta is inserted in Target
  • Importance of Surrogate Key in Target Key table. 
  • What is version of a record
  • How to recognize a record is active or inactive in Target Table
  • Importance of Active Status Column in Target table to implement SCD Type 2
  • What additional columns are required for Target table, to implement 

SCD Type 2

  • Preparing data objects and data for SCD Type 2
  • Create target table  @ Azure sql or Synapse 
  • What is identity column is azure sql 
  • Identity column as surrogate key in target table
  • Possible values for active status column in target table
  • Preparing data lake file for delta

SCD (Slowly Changing Dimensions ) Type 2 – Part 2

SCD (Slowly Changing Dimensions) Type 2 – Part 3

SCD (Slowly Changing Dimensions) Type 3 – Part 1

SCD (Slowly Changing Dimensions) Type 3 – Part 2

SCD (Slowly Changing Dimensions ) Type 4 – Part 1

SCD (Slowly Changing Dimensions) Type 4 – Part 2

SCD  (Slowly Changing Dimensions) Type 6 – part 1

SCD (Slowly Changing Dimensions) Type 6 – Part2

SCD (Slowly Changing Dimensions) Type 6 – Part 3

How to write activity output into file: ( datalake).  – part1

How to write activity output into file: ( datalake).  – part2

How to write activity output into file: ( datalake).  – part3

How to write activity output into file: ( datalake).  – part4

More Information on Integration Runtime. – part 1

More Information on Integration Runtime – Part 2

Until Activity Part 1

Until Activity Part 2.

Web activity Part 1

Web Activity Part 2

Switch Activity  

Script Activity

Accordion Content

More on Lookup Activity With Set Variable  Part – 2

More On Lookup Activity With Append Variable Activity

 Importing Data From  SNOWFLAKE   to  Azure Blob Storage   Part – 1

Importing Data From  SNOWFLAKE  to  Azure Blob Storage  Part – 2

 Most advanced Azure Data Factory Training Program in India

Total: 151 Sessions of Video Lectures on
Azure Data Factory Training By Bharat Sriram sir

Azure+Data+Factory+Training+In+Hyderabad

Key points of Azure Data Factory Training (ADF Training) in Hyderabad

Azure Data Factory Training Modes - Hyderabad, Banglore, Chennai, online Training & Video course

ADF virtual training

The ADF online training in Hyderabad includes all inclusive live training sessions conducted by our trainer, Mr.Bharat Sriram. The training is conducted in an interactive way to help participants understand and absorb the concepts easily. The course content is designed based on industry standards and real time projects. The online training sessions will be held via e-learning platforms such as Webex, Zoom and other similar platforms. The training sessions are conducted on weekends or on working days during evening hours so that participants can attend the sessions conveniently.

Personalized ADF Video training

Learning is made so much easier with our ADF video course training packed with an assortment of live trainer demonstrated classroom recordings and online assessments. We have a vast array of adf video training modules designed to help you improve your skills and knowledge. Our comprehensive training program covers all the aspects of Adf training in Hyderabad to promote easy learning with effective course material.

ADF Classroom training

Our ADF training in Hyderabad classroom study sessions are designed to provide the student with a quality training experience. The classroom course includes both in-class and self-study notes. The class format is designed to be both interactive and informative, while some of the class activities include a combination of lecture, discussion and hands-on practice.

Why choose our Azure Data Factory training program?

With our Azure Data Factory Online course, you will gain valuable hands-on skills and competence. Here’s why you should choose us :

Student portal

Our student portal gives each student a personalized chance to participate and be involved in our Adf course training. We provide Learning Management System (LMS) accounts to the students to learn, track and further improve your skills and cognitive abilities in the course modules that are taught by the trainer.

Group learning sessions

We focus on every student’s growth and improvement throughout the training period by implementing various practices that will help the students grow and learn. Our group learning sessions are extremely penetrative and functional where students can get to communicate and coordinate with each other to trade knowledge and check their personal development growth in the Adf training in Hyderabad.

Career guidance

We prioritize every student and work on providing them with a complete package of counselling and training in our Azure Data Factory (ADF) course. Our top expert trainer will conduct the training, Mr.Bharat Sreeram, and a separate team of experts will manage the counselling. They help the students with all kinds of queries and clarifications regarding the course, its outlook and other instrumental perquisites to give our students a complete counselling experience.

Placement Support

We provide the best placement assistance in Hyderabad by including a series of highly constructive practices such as mock tests, mock interviews, resume preparation, frequently asked interview questions and more for our students to stay one step ahead from the lot. We at AzureTrainings, also recommend and refer our students to companies and startups that are tied up with our institute.

Students reviews

What do students say about Azure Data Factory Training In Hyderabad

Bharat sir is the best trainer in hyderabad and Institute offer quality azure data factory training in Hyderabad. I really enjoyed the online classes and I am happy to be a part of this great training. I am really impressed by the quality and content of there tutorial.
Kaushik
5/5
Bharat sir is extremely helpful and kind. He has provided professional guidance and knowledge to us by helping us in every step of the way and we are extremely grateful for his training! I would definitely recommend AzureTrainings to anyone who is looking to learn Azure data factory training in Hyderabad.
Mrunal
4.5/5
AzureTrainings is one of the best training institutes in Hyderabad for ADF. I really enjoyed the video courses; they were not only informative but also very engaging to watch. I found the training content very useful and the trainer, Bharat sir, was very friendly and easy to approach, he always made sure everyone understood what was being taught.
Sally D'Silva
4.7/5
The ADF online classes were super fun and interesting unlike the other training institutes. I enjoyed the quizzes and interactions with our tutor, he was really cordial. The classes were very interactive and I was able to ask questions when I needed help. The material was easy to understand and the assignments helped me practice what I learned in class.
Ragini
5/5
The ADF course was good and worth the money. It was a great learning experience for me. I am so glad I chose ADF for my online training. The course was easy to understand and the tutors were very helpful. They were always there to answer questions and provided feedback on our assignments.
Geeta
4/5
I learned a lot from my experience at AzureTrainings. The trainer was knowledgeable and used good examples to explain the concepts of ADF to us, which made it easier for us all to understand. He also gave each student extra materials that we could use later for our own projects in this field!
Ramya
4.8/5
AzureTrainings is one of the best training institutes I have seen in Hyderabad. The video courses are very helpful, and Bharat sir's classes were always enjoyable as he made sure we understood all aspects of our work.
Saloni
5/5
AzureTrainings is a great place for anyone who wants to learn about Azure and get certified. Bharat sir is a great trainer and makes sure that everyone understands the concepts before moving onto the next one. He also ensures that all our doubts are cleared before we leave class.
Janvi
5/5
The course material was always up-to-date, and we were given the latest updates on Azure every week. The trainer, Mr.bharat was very friendly and supportive throughout my time at AzureTrainings. The hands-on training was very helpful, and the instructor gave me detailed explanations of all concepts. The trainer is very knowledgeable about Azure and explained things well.
Sooraj
4.5/5

About ADF Training in Hyderabad, online (Azure Data Factory Training)

Azure data factory (ADF) is a cloud-based ETL and Data Integration service created by Azure. It eases the path for an intuitive authoring experience, fully automating management tasks such as monitoring. Adf allows you to schedule and time the workflow according to your preferences. 

We provide the most comprehensive Microsoft Azure Data Factory training (ADF) in Hyderabad, with a conceptual curriculum and hands-on exercises that prepare students for real-world applications.

We train our students by giving them practical experience and knowledge so that learning will be more effective. The training course provides comprehensive and in-depth coverage of Microsoft Azure Data Factory.

The course will cover the following topics: Introduction to Azure Data Factory, Overview of AzureData Factory, Architecture Creating an Azure Data Factory, and Creating a data pipeline using Azure Resource, Manager Templates Using PowerShell for configuration and management, Monitoring your pipelines Troubleshooting Errors. 

AzureTrainings is a subsidiary of Brolly Academy and aims to provide individualized Azure Data factory specialized training in Hyderabad. We also certify students who complete the course. The certificate is suitable for beginners and more advanced learners and can be used to prove your competence in a CV or job application. Our instructor-led ADF training in Hyderabad will prepare you for the Azure certifications by imparting the necessary knowledge and skills to pass the exams. 

You can also opt for self-paced video tutorial courses, where you learn at your own pace and don’t have to worry about deadlines or exam dates. The course material is available online and on mobile devices with an app version of the course. Our training institute has the best ADF trainer in India, whose experience working with multiple organizations and companies across the country makes him well suited to coach our trainees and students. 

We at AzureTrainings, conduct our classes at a level of quality that other institutes unmatch. Batch sizes are kept small to ensure each student’s best possible training experience. Join our training program and learn from the best trainers, with dedicated placement opportunities in some of Hyderabad’s top companies.

Our Azure Data Factory Course (adf course) Includes:

About Our Trainer

Mr.Bharat Sriram will be your Adf trainer and will be leading you through the entire process of the Azure Data Factory training program. He is a veteran in Azure and holds an excellent track record of training students to become experts in Azure. He has trained over 30,000+ students in the past few years with 25+ years of competent experience. 

His passion for technology and teaching makes him even more inspiring to learn from, as he is one of the most sought-after trainers in Hyderabad with an impressive reputation. His expertise in Azure is unparalleled, and his teaching style makes it easy to understand even the most complex concepts. 

He is well versed with the latest technologies, tools, and trends in the industry. Moreover, he has worked with some of the top companies and institutions in Hyderabad as a lead Adf trainer and has a vast experience in training and developing students. He is an excellent trainer, has a deep understanding of technology with patience, and takes extra effort to explain each topic in detail, ensuring that the student enjoys a satisfactory quality learning experience. 

He ensures that every student gets adequate attention and time during the training sessions. His experience as a consultant helps him understand what students need to learn to become successful professionals in their careers. His approach to teaching is convenient and has helped many students pass their certification exams. He has also delivered training in Data Science, MLops, Azure DevOps, and more. 

Get certified with Adf course certification

The Azure Data factory certification program is designed to provide a credential that professionals can use to demonstrate technical skills and knowledge in Azure Data Factory. ADF Certification training provides a comprehensive overview of Data Factory, including its architecture and components. 

It also provides an introductory level of skills in Azure Data Factory, which will help professionals to use the tool effectively. Our trainer, Mr.Bharat, will train you with all the concepts in ADF by giving you ample knowledge of the course subjects. He will also guide you on using the software and all its features with his competent 25+ years of experience in the field. 

Upon completing the course duration with our expert guidance, you will also be able to attend the ADF certification exams and get yourself certified. 

Azure Data Factory certifications

The Azure Data Factory certification program currently offers four different certifications:

  1. The Azure Data Factory Certified Professional certification is the entry-level certification for Azure Data Factory. It’s designed to validate your knowledge of the basics of Microsoft Azure Data Factory and its associated tools, including Azure SQL Database and HDInsight.
  2. The second level is the Architect certification. This credential validates your ability to design, develop and deploy an enterprise-level solution.
  3. The third level is the Administrator certification, which validates your ability to manage and troubleshoot a data factory environment.
  4.  Finally, the Data Factory Consultant certification validates your ability to design and implement an Azure Data Factory solution for an enterprise.

About Azure Data Factory Training institute in Hyderabad - Online Classes

Azure Trainings is one of the best ADF training institutes in Ameerpet, Hyderabad, with industry experts trainers. We have been training in Azure over the past 3 years and have successfully trained 800+ students across Hyderabad and Telangana. We provide Azure placement assistance along with Azure certification and support. Enroll yourself in our Azure Training program to gain our e-book and the right skill set to make it in the IT industry.

Benefits of this Azure Data Factory (adf) Training Program

AzureTrainings offers comprehensive, hands-on Azure Data Factory training in Hyderabad that will prepare you for success with ADF’s technical and advanced concepts. We offer a full range of Adf training services, including an industry-aligned curriculum that will qualify you for Microsoft Azure Data Factory careers. AzureTrainings is the most convenient way to learn Azure Data Factory. Moreover, we offer flexible scheduling options, including weekdays and weekend classes. 

  • You will be able to create a data pipeline that processes data from various sources and stores it in a destination database.
  • You will be able to use Azure Data Factory to schedule and monitor your data pipelines.
  • You will be able to manage the security of your Data Factory account using Azure Active Directory.
  • You will be able to use Azure Data Factory to move data between on-premises and cloud environments.
  • You will be able to test your data pipelines using the Azure Data Factory designer.

Job Opportunities in Azure Data Factory

The most common job roles in Azure Data Factory include:

Skills you will gain from our Adf training in Hyderabad

Frequently Asked Questions

Who can opt for Azure Data Factory course?

Azure Data Factory is designed for many users, including data scientists and analysts who want to automate their data processing tasks. It’s also well-suited for IT professionals who need to deploy and maintain enterprise-scale workloads. Even students can take up our Adf training in Hyderabad.

Do I need any prior technical Knowledge to learn ADF?

ADF is a simple programming language that anyone can learn. You don’t need prior programming or software development knowledge to learn ADF.

Who will be my trainer?

Mr. Bharat Sriram, a renowned ADF trainer in Hyderabad, will be your trainer.

How do I practice Azure Data Factory?

There are several ways to practice Azure Data Factory. You can go through the tutorials and hands-on labs in the Data Factory documentation or use a tool that automatically generates a pipeline based on your data, among the other options.

Is coding required for Azure Data Factory?

Basic coding knowledge would be required to implement some of the features, but it is not necessary to use all of them. If you do not have coding skills, then it is recommended that you learn the basics.

Will I get any course completion certificate from Azure Online training in Hyderabad?

Yes, you will get a course completion certificate after completing the Azure Online Training in Hyderabad.

How to register for the ADF Training Program?

You can first try our demo session conducted by an expert from our team and join the course as you wish.

How long does it take to learn Azure Data Factory?

The time required to learn Azure Data Factory depends on your level of experience and prior knowledge. We will provide a complete Azure data factory training covering all the essential concepts and skills.

Which certification is best for ADF?

The MCSA: Data Management and Analytics is the most popular certification for Azure Data Factory. This certification covers the entire platform and includes both ADF and SQL Server Data Warehouse.

Book Free Demo Class

Register for The Live Demo Class

*By filling the form you are giving us the consent to receive emails from us regarding all the updates.