Azure Data Factory Training In Hyderabad
with
100% Placement Assistance
- Most Experienced Trainer in India
- 15+ Real time scenarious covered
- LMS Access for 180+ recorded sessions
- Life Time LMS Access
- 100% Job Oriented Training Program
- Certfication Dumps
- Practice Session
- Upto Date Notes
- Resume Creation
- Interview Preparation
Azure Data Factory Training in Hyderabad - Online Classes
Azure Data Factory (ADF) Curriculum
Azure Data Engineer Services Introduction Part 1
Azure Data Engineer Services Introduction Part 2
Azure Data Engineer Services Introduction Part 3
Azure Data Engineer Services Introduction Part 4
Azure Data Engineer Services Introduction Part 5
Azure Data Engineer Services Introduction Part 6
ADF introduction
Difference between ADF version 1 and Version 2.
- DataFlows and Power Query are new features in Version 2
- DataFlows are for data transformations
- PowerQuery is for Data preparations and Data Wrangling Activities.
- Building blocks of ADF
- -> PipeLine
- -> Activities
- -> Datasets
- -> Linked Service
- -> Integration
- RunTime
- Auto Integration Runtime
- Self Hosted Integration runtime
- ADF Session 8:SSIS Integration Runtime.
More on Technical differences between ADF version 1 and Version 2 – Part 1
- More on Technical differences between ADF version 1 and Version 2 – Part 2
- Introduction to Azure Subscriptions
- Types of Subscriptions
-> Free Trial
-> Pay-AS-You-Go
- Why Multiple subscriptions are required?
- What are resources and Resource Groups?
- Resource Group Advantages
- Why Multiple resource Groups need to be created ?
- What are regions
- Region Advantages.
- Create Storage Account with Blob Storage feature
- Converting Blob storage feature as DataLake Gen2 feature
- Create Storage Account with Azure Data Lake Gen2 features.
- How to Enable Hierarchical name space.
- Creating Containers
- Creating sub directories in the Container of Blob Storage.
- Creating sub directories in Container of DataLake Gen2 Storage
- Uploading local files into Container/sub directories.
- When is ADF required?
- Create Azure SQL and Play with Azure Sql – Part 1
- Azure sql as OLTP
- Create Azure Sql Database
- Create Azure Sql Server
- Assign Username and Password for Authentication
- Launching Query Editor
- Adding Client IP address to FireWall Rule settings
- Create table
- Insert rows into table
- Default schema in azure sql
- Create schema
- Create table in user created schema
- Loading query data into to table
- Information_schema.tables
- Fetching all tables and views from database
- Columns of Information_schema.tables
-> TABLE_CATALOG
-> TABLE_SCHEMA
-> TABLE_NAME
-> TABLE_TYPE
- Fetching Only tables from database
- How to create Linked Service for Azure data lake
- Possible errors while creating linked service for Azure datalake account.
- How to solve errors for Linked Service for Azure data Lake
- Two ways to Solve Linked Service Connection Error.
-> Enable Heirarchical namespace for Storage Account
-> Disable Soft Delete options BLOB and CONTAINER
- How to create datasets for Azure data lake file and containers
- Your First ADF pipeline for Datalake to data lake file loading
- COPY DATA activity used in pipeline
- Configuring Source dataset for Copy Data Activity
- Configuring Sink dataset for Copy Data Activity
- Run the pipeline:
- Two ways to Run a Pipeline.
-> Debug mode
-> Trigger mode
- Two Options in Trigger mode
-> On demand
-> Scheduling
- How to load data from Azure data lake to Azure sql table.
- Create linked service for Azure sql Database
- Error resolving , while creating Linked service for Azure Sql Database
- Create a dataset for Azure Sql Table.
- Create a pipeline to load data from Azure data lake to Azure sql Table
- Helped activities
- Copy activity
- If data lake file schema, and azure sql table schema are different, How to load Using Copy Data Activity.
- Perform ETL With “Copy Data” Activity
- Copy Data Activity With “Query” Option
- Loading Selected Columns and Matching Rows with Given Condition , From Azure Sql Table to Azure Data Lake
- Creating new fields based on existed Columns of table and Load into Azure Data Lake
- Problem Statement : A file has n fields in the header , but data has n+1 field values . How to Solve this problem in Copy Data Activity
- Solution to Above Problem Statement. Practical implementation.
- Get Metadata – part 1
- Get Metadata Field List Options for Folder
- “Exists” Option of Field List
- Data type of Exists field in “Get Metadata” output as Boolean(true/false)
- “Item Name” Option of Field List
- “Item Type “ Option of Field List
- “Last Modified” Option of Field List
- “Child Items” Option of Field List
- Data Type of “child Items” field in Json Output.
- What is each element of “Child Items” called?
- Data type of item of “ChildItems” Field
- What are subfields of each item in “childItems” Field
- Get Metadata Activity – Part 2
- If Input dataset is file, what are Options of “Field List” of Get Metadata.
- How to Get the number of Columns in a file?
- How do you make sure the given file exists ?
- How to get the file name, which is configured for the dataset. ?
- How to get a dataset back end data object type(file or folder) ?
- How to Know when a file was last modified?
- How to get File size ?
- How to get File structure(schema) ?
- How to get all above information of a file/folder/table with a single pipeline run
- If the input dataset is an RDBMS table, what are options of “Field List” of Get Metadata Activity. ?
- How to get the number of columns of a table ?
- How to make sure the table exists in the database ?
- How to get the structure of a table ?
- In this Session, you will learn answers for all above questions practically.
- Introduction to “Get Metadata” activity.
- How to fetch File System information Using “ Get metadata” Activity.
- How “Get metadata” activity writes output in “JSON” format.
- How to Configure Input dataset for “Get Metadata” activity.
- What is the “Field List” for “Get Metadata” activity?
- Small introduction to “Field List” options.
- Importance of “Child Items” option of “Field List” in “Get Metadata” activity.
- How to Check and understand output of “Get Metadata” activity.
- “childItems” field as JSON output of “Get Metadata”.
- “childItems” data type as Collection as data type As Array.
- What is each element of “childItems” output
- What is the “exists” option in “Field List”.
- Introduction to “Filter” activity.
- Problem statement: in a container of a Azure Datalake storage account , there are 100s of files, some files related with adf , and some files related with employees, and some files related with sales and some files related with others like log.
- How to fetch only required files from Output of “get metadata” activity ?
- How to place “Filter” activity in the pipeline.
- How to Connect “Get metadata” activity and “Filter” activity.
- What happens , If we don’t connect two activities? (bcoz, this is the first scenario with multiple activities in a single pipeline).
- How to pass output of “Get metadata” Activity to “Filter Activity”
- “Items” field of “Filter” Activity.
- What is the “@activity()” function and “@activity().output” .
- Above Output produces a lot of fields , How to take specific fields as input to filter activity.
Example : @activity(“get metadata1”).output.childItems
- How to avoid unnecessary information to next subsequence Activity
- Configuring “Condition” field of “Filter” activity.
- How to access each element of the output of “childItems” field is @item() function.
- @item() output as nested json record. How to access each field.
Example, @item().name, @item().type
- @startswith() function example
- How to check output of “Filter” activity
- What is the Field name of “Filter” activity , in which required information is available.
- @not() function example
- ——————————————–
- @item() output as nested json record. How to access each field.
Example, @item().name, @item().type
get metadata + filter activities and how to apply single condition how to apply multiple conditions all the things
there are functions
Functions
@equals
@greater( )
@greaterOrEquals( )
@or (C1,C2 …….)
@and (C1,C2 …..)
@not (equals( ))
- Task : Get Metadata – fetch only files , and reject folders of Given Container with “Get Metadata” and “Filter” Activity.
- steps to achieve Above Task.
- Step1: Create a Pipeline and Drag Get MetaData Activity
- Step2 : Configure Input DataSet of Container
- Step3: Add a Field List With “child Items” Option.
- Step4: Add “Filter” activity to “Get Metadata”
- Step5: Configure “Items” Field , Which is input for “Filter” activity From Output of “Get Metadata”
- Step6: Configure Filter Condition to take only Files.
- Step7: Run the Pipeline
- Step8: Understand Output of “Filter” Activity.
- In which field of Json, Filter output is available ?
- What is data type of Filtered output
- In this task you will work with below ADF expressions
- @ activity() Function ..
- @ activity(‘Get Metadata1’).output
- @ activity(‘Get Metadata1).output.childItems
- @ item().type
- @ equals(item().type, ‘File’)
- After Completion of this session you will be able to implement all above steps , and can know answers for above questions , and able to use adf expressions practically.
- Scenario : with GetMetadata and Filter Activities – Part 1
Scenario : with GetMetadata and Filter Activities – Part 2
- Task : bulk load of files one storage account to another storage account (from one container to other container )
- Helped activities
-> get metadata
-> foreach
-> copy data
- Bulk load of files from one storage account to Other storage account with “Wild card “ option.
- Bulk load of files from one storage account to Other Storage account with “Get Metadata” and “ForEach” and “Copy Data” Activities.
- Data Set Parameterization
- Bulk load of files into Other Storage Account with “Parameterization”
- copy only files which starts emp into target container Using Get MetaData , Filter , ForEach, Copy Data Activities – [ in target container, file name should be same as source ]
- load multiple files into the table.(Azure Sql)
- Helped activities
- get metadata + filter + foreach + copy data activities
- Conditional split .
- Conditionally distributing files (data) into different targets (sinks) two ways.
- using filter
- using if condition activity
- Conditional Split implementation with “Filter” activity
- Problem with Filter , explained.
- Conditional split with “if condition”.
- Helped activities
- get metadata + filter + for each + if condition
- Lookup activity
- migrate all tables of the database into a data lake. with a single pipeline.
- Helped activities
lookup activity for each activity COPY DATA ACTIVITY WITH PARAMETERIZATION
- Data Flows Introduction.
- ELT (Extract Load and Transform)
- Two DataFlows in ADF
- Dataflow activity in pipeline.
- Mapping Data Flows
- Configuring Mapping Dataflows as Data Flow Activity in PipeLine
- Introduction to Transformations
- Source
- Sink
- Union
- Filter
- Select
- Derived Column
- Join etc.
- Difference between source , sink of “Copy Data” activity And source , sink of “Mapping Data Flows”.
- Source as (data lake) Sink as (SQL table) ——-> by using dataflow source and sink transformation
- Extract data from RDBMS to Datalake ———–> Apply Filter ——> sink
- Assignment1
- SQL Table to Datalake
- Helped Transformation (Souce)
Sink (Dataflow Activity)
- Filter transformations
- Helped Transformations
(Source) Filter Sink (Dataflow Activity)
- Select transformations
- Rename columns
- Drop columns
- Reoder the columns
- Helped Transformation
(Source ) Select Sink (Dataflow Activity)
- Derived column Transformation Part1
1.You can generate new column with given expression
2.You can update existed column values
- Helped Transformation
Derived column
Select Sink (Dataflow Activity)
- How to clean/handle null values in data.
- why should we clean nulls
1.computational errors
2.data loading errors into target table synapse table.
- Cleaning names(cleaning means not always replace nulls with some constant value, cleaning means transform data according to business tool).formating of the data (names)
- Helped transformation
Derived column
- Generate new columns,with conditional transformation. two options:
1.iif() —->nested ifs,
2.case()
- Helped transformation
Derived column
- Conditional transformation with case() function.
emp—->derived columns—–>select—–sink
- Helped transformation
derived column
select sink (Dataflow activity)
- Union transformation part 1.
- Merging Two different files with the same schema by using two sources using the union transformation, And writes output into a single file.
- Helped transformation
Two sources
Union
Sink
(Dataflow activity)
- Merging three different files with different schemas by using derived column transformation, select transformation and union transformation, finally we get a single file with a common schema.
- Helped transformation
Derived column
union
select
(source)
sink (Dataflow activity)
- Two different files by using derived column transformation, union transformation and aggregate transformation, to get branch1 total and branch 2 total.
- Helped transformations
(Source)
Derived column
Union
Aggregate
*(After this Watch session 70, for more on Unions).
- Joins transformations 5 types
1.Inner join
2.Left outer join
3.Right outer join
4.Full outer join
(5.Cross jons)
- Helped transformation
Joins
Select
Sink
(Dataflow activity)
ADF session 47:
Full Outer Join Bug fix
Full Outer Join Bug fix
How to join more than two datasets (example 3 datasets).
- Inter linked scenario related to 25th session.
treat, dept as project
task is ” summary Report”
active employes (who already engaged into project -> 11,12,13 projects,-> these projects total salary budget (bench team 20,21 total salary ,bench project )
- Helped transformations
Joins
Select
Derived column
Aggregate
Sink
(Dataflow activity)
- To use full outer join advantage (complete information,no information missing)
task~1 : Monthly sales report by using derived column transformation, aggregate transformation.
- Helped transformation
Derived column
Aggregate
sink
- Task~2 : Quarterly sales report by using derived column transformation, aggregate transformation.
- Helped transformation
Derived column
Aggregate
- Task~3: Year as primary group,quarterly as sub group,sales report.
- Helped transformation
Filter activity
Derived column
Aggregate
Sort
Sink
(Dataflow activity)
- Task~4 : Comparing Quartely sales report: Comparing Current Quarter sales with its Previous Quarter Sales.
- Helped transformations
Source
Join
Select
Derived column
- A real time scenario on Sales data Analytics – Part 1
A real time Scenario on Sales data Analytics – Part 2
- More on Aggregation Transformation.
- Configuration Required for Aggregation Transformation
- Entire Column aggregations.
- Entire Column Multiple Aggregations.
-> Sum()
-> Count()
-> max()
-> min()
-> avg()
- Single grouping and Single Aggregation
- Single Grouping and Multiple Aggregations
- Grouping by Multiple Columns with Single aggregation
- Grouping by Multiple Columns with Multiple Aggregations.
- Finding range Aggregation by adding “Derived Column” transformation to “Aggregation” Transformation.
- Conditional Split of data.
- Distributing data into multiple datasets based on given condition
- Split on options:
- First Matching conditions
- All Matching conditions
- When to use “First Matching Conditions” and When to use “All Matching Conditions”
- An Example.
- An Use Case on “First Matching Condition” option of “Conditional Split” Transformation in Mapping Data Flows with Sales data.
An Use Case on “Matching All Conditions” option of “Conditional Split” Transformation in Mapping Data flows with Sales data.
Conditional split with cross join transformation by using matrimony example.
- Lookup with multiple datasets
- Helped transformations
Source
Lookup
Sink
(Dataflow activity)
- Lookup with more options
- products as primary stream
- transformation as lookup stream
- Helped transformations
Products
Lookup
Broadcasting
Partitioning part 1
Partitioning part2
Partitioning part3
Exists transformation – part 1
- Helped transformation
(Customers)
Exists
Exists transformations – part 2
- Helped transformation
(Source)
(Customers)
Exists
Sink
(Dataflow activity)
- Finding Common Records, Only Records available in first dataset, Only Records available in Second Dataset.
- All Records except Common records in First and 2nd Datasets in Single DataFlow
- How to capture data changes from source systems to Target Data warehouse Systems.
- Introduction to SCD (Slowly Changing Dimensions)
- What is SCD Type 0 and Its Limitation.
- What is Delta in Data of Source System.
- What is SCD Type 1 and Its Limitation
- What is SCD Type 2 and How it tracks History of specific attributes of source data.
- Problem with SCD Type 2
- Introduction to t SCD Type 3
- How It solves problem of SCD Type 2
- How SCD type 3 maintains recent History track
- Limitation of SCD Type 3.
- Introduction to SCD Type 4
- How SCD Type 4 will provide complete Solution to SCD type 2.
- (Remember no SCD 5)
- Introduction to SCD Type 6 and Its benefits.
- How data transformations are done in ADF1 (with out dataflows)
- Aggregate Transformation , Sort Transformation with following examples
- Single grouping with single aggregation
- Single Grouping with multiple aggregations
- Multi Grouping with Multiple Aggregations.
- Sort with single column
- Sort with multiple columns.
- What is pivot transformation?
- Difference between Aggregate transformation and Pivot Transformation
- Implement pivot transformation in dataflow
- How to clean pivot output
- How to call multiple dataflows in a single pipeline.
- Why we used multiple dataflows in one single pipeline.
- Assignment on Join, aggregate, pivot transformations.
- Finding Occupancy based on salary.
- Unpivot transformation.
- What is should be the input for Unpivot (pivoted output file).
- 3 configurations :
-> Ungrouping column
-> Unpivoted column. (column names to as column values)
→ aggregated column expression ( which row aggregated values to be turned as column values)
- Difference between output of “aggregate” and “unpivot” transformation.
- What additionally unpivot produces.
- Use case of Unpivot
- Surrogate Key transformation,
- Why we should use Surrogate key.
- Configuring Starting Integer Number for Surrogate Key
- Scenario : your bank is “ICICI”, for every record Unique CustomerKey should be generated as ICICI101 for the first customer, ICICI102 for 2nd customer . But the Surrogate key gives only integer value as 101 for first and 102 for 2nd. How to handle the given scenario , which is a combination of string and integer.
- Rank Transformation in dataflows – Part 1
- Why sorting data is required for Rank transformation
- Sorting options as “Ascending and Descending”
- When to use the “Ascending” option for Ranking.
- When to use the “Descending” option for Ranking.
- Dense Rank and How it Works
- Non Dense Rank (Normal Rank) and How it Works.
- Why we should not use “surrogate key” for Ranking.
- What is difference between Dense Rank and Non Dense Rank(Normal Rank)
- Limitations of Rank Transformation
- Implementing Custom Ranking With a Real time scenario
- Custom Ranking implementation with Sort, Surrogate key, new Branch , aggregate, Join transformations.
- Scenario :
-> problem statement: if a school has 100 students, one student got 90 marks , remaining all 99 students failed and scored 10 marks. The Rank transformation of ADF dataflows gives 1 rank for those who scored 90 , and 2 rank for failed students , who scored 10 marks. But the school management wanted to give a gift to the top 2 Rankers ( 2nd rankers failed and got equal least score , so that all students will get a gift). How to handle this scenario.
- Window transformation part 1
- Window transformation.
- Cumulative Average
- Cumulative Sum
- Cumulative Max
- Dense Rank for each partition.
- Making all rows as Single Window and apply Cumulative Aggregation.
Window transformation part 2
Window transformation part 3
Window transformation part 4
Window transformation Part 5
Window transformation Part 6
Custom Rank Implementation with Window Transformation,
- Parse transformation in Mapping DataFlows.
- How to handle and parse string collection ( delimited string values )
- How to handle and parse xml data.
- How to handle and parse json data
- Converting Complex Json nested structures into CSV/Text file.
- Complex data processing (transaformations).
- How parse nested Json records
- How to flatten array of values into multiple rows.
- If data has complex structures, what are supportive data store formats .
- How to write into json format.
- How to write into flatten file format( csv).
- Transformation used to process data.
- Parse
- Flatten
- Derived Column
- Select
- Sink.
- Reading Json data (form vertical format)
- Reading a single Json record(document)
- Reading array of documents .
- Converting complex types into String using Strigify and derived column transformations.
- Used transformations in Data Flow.
-> source
->stringify
->derived column
Expression : toString(complexColumn)
→ select
-> sink.
- Assert Transformation
- Setting Validation Rules for Data.
- Types of Assert:
-> Expect True
-> Expect Unique
-> Expect Exists
- isError() function to validate a record as “Valid” or “Invalid”.
- Assert Transformation part 2.
- Assert Type “Expect Unique”
- Rule for Null values
- Rule for ID ranges.
- Rule for ID uniqueness
- hasError() function to identify which rule is failed.
- Difference Between isError() and hasError()
- Assert Transformation part 3.
- How and why to configure Additional streams
- Assert type “Expect Exists”
- How to validate a record reference available in any one of multiple additional streams. (A scenario implementation).
- How to Validate a record reference available in all multiple additional streams.(A scenario Implementation),
Incremental data loading (part 1)
Incremental data loading (part 2)
- Incremental data loading (part 3).
- Implementation of Incremental load(delta load) for multiple tables with a single pipeline.
Deduplication part 1
Deduplication part 2
Deduplication Part 3
Solving Bugs of Incremental Loading . – Part1
Solving Bugs of Incremental Loading – Part2
- AlterRow Transformation Part – 1
- Removing List of Records from Sink of a Specific criteria (Table)
- AlterRow Transformation part- 2
- Removing a given List of Records from Sink of no specific Criteria .
- AlterRow Transformation Part – 3.
- Update given List of Records in Sink.
- SCD (Slowly Changing Dimensions Type 1 ) Implementation.
- Alter Row Transformation with UpSert Action.
- A realtime Assignment (Assignment1 and Assignment 2) on SCD.
- Incremental Load and SCD combination
- How to implement SCD before “Alter Row” Transformation not introduced in DataFlows.
- How exists transformation helpful.
- How to implement SCD before ADF version 2 features of DataFlows.
- How Stored procedures helpful and implement SCD.
- Combining rows of multiple tables with different schemas with a single Union Transformation.
- Problem with Multiple Unions in DataFlow.
- Types of Sources and Sinks
- Source Types:
-> Dataset
-> Inline
- Sink Types:
-> Dataset
-> Inline
-> Cache
- When to use dataset and Inline
- Advantage of Cache as Reusage of output of Transfomation
- How to write output into Cache.
- How to Reuse the Cached output.
- Scenario : Generating Incremental IDs based on Maximum ID of Sink DataSet – Part 1
- Scenario: Generating Incremental IDs based on Maximum ID of Sink DataSet – Part 2
- Writing outputs into JSon format
- Load Azure SQL data into Json File.
- Cached Lookup in Sink Cache . Part 1
- Scenario :
Two Source Files, 1. Employee 2. Department
Common Column in both dno(department number) which is as Joining Column or Key Column.
- Task: Without using Join Transformation , Lookup Transformation , we need to join two datasets to increase performance using “Cache Lookup” of Cache Sink Type.
- Other alternatives of this Cached Lookup.
- How to configure Key Column for Cached Lookup
- How to access values of Lookup key in Expression Builder.
- lookup() function in Expressions
- sink#lookup(key).column
- Cached Lookup in Sink Cache – Part 2
- Below Scenario will teach you, how cached output is used by multiple transformations.
- Scenario:
Single Input file : Employee,
Two Transformations required.
- Foreach employee , find his salary occupancy in his department
- Foreach employee, find his average salary status in his department as
“Above Average” or “Below Average”.
For these two transformations common input sum(), avg() aggregations with
group by department number . This should be sent to Cache Sink as Cached
Lookup.
Each transformation output should be in separate Output file.
- In above Dataflow, What flows are executed in parallel and What flows executed in sequence.
- How spark knows dependency between flows. [Using DAG (Direct Acyclic Graph ) engine ]
- Scope of “Sink Cache” output. [ problem statement ]
- Using “ Multiple Cache Sink “ outputs in Single Transformation.
- Scenario:
Employee table has id, name, salary, gender, dno, dname, location columns with two records 101 and 102 ids.
New employees are placed in datalake file newdata.txt with name,salary, gender, dno fields.
Department name, department location fields are available in department.txt file of datalake.
Insert all new employees into Employee table of azure SQL , with incremental ID based on maximum id of the Table along with department name and location .
- Sink1 : write max(id) from Employee table (cache sink)
- Sink2 : write all rows of department.txt into Cache Sink. and Configure dno as Key Column of Lookup
- Read data from new employees file and generate next employee id with help of surrogate key and sink1 output.
- Generate lookup columns department name, location from Sink2 and Load into Target Employee table Azure SQL.
- How two Different DataFlows exchanging Values – Part 1.
- Scope of Cache Sink of a DataFlow.
- What is Variable
- What is Difference between Parameter and Variables
- How to Create a Variable in Pipeline
- Variable Data Types:
-> String
-> Boolean
-> Array
- Create a DataFlow with Following Transformation Sequence to find Average of Column
- Source
- Aggregate (for finding Average)
- Sink Cache (to write Average of column)
- How to pass Cache Sink output to Activity Output. How to Enable this feature
- Create a Pipeline to Call this Data Flow which writes into Sink Cache
- How to Understand Output of DataFlow Activity which writes in Sink Cache and Writes as Activity Output
- Fields of DataFlow Activity output.
- runStatus
- Output
- Sink
- value as Array
- “Set Variable” Activity
- How to Assign Value to Variable using “Set Variable” Activity.
- How two Different DataFlows exchanging Values – Part 2
- How to Assign DataFlow Sink Cache Output to Variable .
- Dynamic Expression to Assign Value to Variable Using “Set Variable” Activity.
- @activity(‘dataflow’).output
- runStatus.output.sink.value[].field
- Understand Output of “Set Variable” Activity.
- How two Different DataFlows exchanging Values – Part 3
- DataFlow Parameters
- How to create Parameter for DataFlow
- Fixing Default Value for DataFlow Parameter
- How to access DataFlow Parameter
- $<parameter_name>
- Accessing variable value from pipeline into DataFlow Parameter
- @variables(‘<variable_name>’)
- Dynamic expression to access variable value in DataFlow Parameter
- From above 3 sessions, you will be learning how to pass output of cache sink output to other dataflows .
- Categorical Distribution of Data into Multiple Files (foreach data group one separate file Generation ). → Part 1
- Scenario: There is department name column with “Marketing”, “Hr”, “Finance” values. But these values are duplicated. All rows related with these 3 categories . We need to distribute data dynamically into 3 categories.
- Example: all rows related with marketing department into “marketing.txt” file.
- Find Unique Values in Categorical Column and Eliminate Duplicate Values
- Convert Unique Column into Array of String
- collect() function
- Aggregate Transformation using Collect() function
- Write Array of String into Sink Cache
- Enable Cache to “Write to Activity Output”
- Create a pipeline, to Call this Dataflow.
- Create a variable in pipeline with Array Data type
- Access DataFlow Cache Array into Array Variable with “Set Variable” activity Using “Dynamic Expressions”.
- Categorical Distribution of Data into Multiple Files (foreach data group one separate file Generation ). → Part 2
- Array variable to Foreach
- Configuring input (value of Array Variable) for Foreach Activity
- DataFlow sub Activity under Foreach.
- Passing Foreach Current item to Dataflow parameter
- Apply Filter with DataFlow Parameter.
- DataFlow Parameter as file name in Sink Transformation,
- SCD (Slowly Changing Dimensions ) Type 2 – Part 1
- Difference between SCD Type 1 and SCD Type 2
- What is Delta of Source data
- How to Capture Delta (Solution : Incremental loading)
- Behavior of SCD Type 1
- If Record of Delta existed in Target, what happens for SCD Type1
- If Record of Delta does not exists in Target, What happens for SCD Type 1
- How to capture Deleted records from source into Delta.
- Behavior of SCD Type 2
- What should happen for Delta of Source in Target
- What should happen for old records of Target when Delta is inserted in Target
- Importance of Surrogate Key in Target Key table.
- What is version of a record
- How to recognize a record is active or inactive in Target Table
- Importance of Active Status Column in Target table to implement SCD Type 2
- What additional columns are required for Target table, to implement
SCD Type 2
- Preparing data objects and data for SCD Type 2
- Create target table @ Azure sql or Synapse
- What is identity column is azure sql
- Identity column as surrogate key in target table
- Possible values for active status column in target table
- Preparing data lake file for delta
SCD (Slowly Changing Dimensions ) Type 2 – Part 2
SCD (Slowly Changing Dimensions) Type 2 – Part 3
SCD (Slowly Changing Dimensions) Type 3 – Part 1
SCD (Slowly Changing Dimensions) Type 3 – Part 2
SCD (Slowly Changing Dimensions ) Type 4 – Part 1
SCD (Slowly Changing Dimensions) Type 4 – Part 2
SCD (Slowly Changing Dimensions) Type 6 – part 1
SCD (Slowly Changing Dimensions) Type 6 – Part2
SCD (Slowly Changing Dimensions) Type 6 – Part 3
How to write activity output into file: ( datalake). – part1
How to write activity output into file: ( datalake). – part2
How to write activity output into file: ( datalake). – part3
How to write activity output into file: ( datalake). – part4
More Information on Integration Runtime. – part 1
More Information on Integration Runtime – Part 2
Until Activity Part 1
Until Activity Part 2.
Web activity Part 1
Web Activity Part 2
Switch Activity
Script Activity
More on Lookup Activity With Set Variable Part – 2
More On Lookup Activity With Append Variable Activity
Importing Data From SNOWFLAKE to Azure Blob Storage Part – 1
Importing Data From SNOWFLAKE to Azure Blob Storage Part – 2
Most advanced Azure Data Factory Training Program in India
Total: 151 Sessions of Video Lectures on
Azure Data Factory Training By Bharat Sriram sir
- Industry Expert with 25+ years Exp.
- #1 Leading Trainer - Trained more than 30,000 professionals.
- 126 Sessions on Azure Data Factory
- 12 Sessions on Azure Synapse Analytics
- 13 Sessions on Azure Data Bricks & more...
Key points of Azure Data Factory Training In Hyderabad
-
Gain in-depth knowledge about Azure Data Factory operations and data processes with hands-on training techniques - Understand the Azure synapse architecture from table creation to data monitoring scripts with trainer assistance
- Get 100+ hours of Azure Data Factory video recordings for free as a part of your course training program
- Get advanced Azure Data Factory Training In Hyderabd from top trainers with 20+ years of professional experience
- Join the course and get 2 day Free access to exclusive Adf LMS curated by our expert trainer.
- Learn about mapping data flow and wrangling data flow with live examples and case studies
- 200+ trainees trained with 100+ successful placements in the last 4 months by AzureTrainings
- Create Azure data lake storage and manage task schedules + tumbling windows as a part of your project
- Attend our 40+ hours of live intuitive learning with free access to updated course curriculum and study guides
- Get 70% practical + 30% theory based Azure Data Factory Training with cost-effective course fee and installments facility
- Attend our unparalleled Azure Data Factory Training In Hyderabad that is primarily focused on Industrial concepts and techniques
- Get additional career counseling with interview prep support and job placement assistance
- Get everyday class recordings and curated Adf LMS with lifetime access
Azure Data Factory Training In Hyderabad & Video course
Azure Data Factory virtual training
The Azure Data Factory Training in Hyderabad includes all inclusive live training sessions conducted by our trainer, Mr.Bharat Sriram. The training is conducted in an interactive way to help participants understand and absorb the concepts easily. The course content is designed based on industry standards and real time projects. The online training sessions will be held via e-learning platforms such as Webex, Zoom and other similar platforms. The training sessions are conducted on weekends or on working days during evening hours so that participants can attend the sessions conveniently.
Personalized Azure Data Factory Video Training
Learning is made so much easier with our Azure Data Factory(ADF) video course packed with an assortment of live trainer demonstrated classroom recordings and online assessments. We have a vast array of adf video training modules designed to help you improve your skills and knowledge. Our comprehensive Azure Data Factory training program covers all the aspects of Azure Data factory Training in Hyderabad to promote easy learning with effective course material.
Azure Data Factory ( ADF ) ClassroomTraining
Our Azure Data Factory Training in Hyderabad classroom study sessions are designed to provide the student with a quality training experience. The classroom course includes both in-class and self-study notes. The class format is designed to be both interactive and informative, while some of the class activities include a combination of lecture, discussion and hands-on practice.
Why choose our Azure Data Factory Training program?
With our Azure Data Factory Training Online course, you will gain valuable hands-on skills and competence. Here’s why you should choose us :
Student portal
Our student portal gives each student a personalized chance to participate and be involved in our Azure Data Factory Training In Hyderabad. We provide Learning Management System (LMS) accounts to the students to learn, track and further improve your skills and cognitive abilities in the course modules that are taught by the trainer.
Group learning sessions
We focus on every student’s growth and improvement throughout the Azure Data Factory training period by implementing various practices that will help the students grow and learn. Our group learning sessions are extremely penetrative and functional where students can get to communicate and coordinate with each other to trade knowledge and check their personal development growth in the Azure Data FactoryTraining In Hyderabad.
Career guidance
We prioritize every student and work on providing them with a complete package of counselling and our Azure Data Factory Training In Hyderabad course. Our top expert trainer will conduct the training, Mr.Bharat Sreeram, and a separate team of experts will manage the counselling. They help the students with all kinds of queries and clarifications regarding the course, its outlook and other instrumental perquisites to give our students a complete counselling experience.
Placement Support
We provide the best placement assistance in Hyderabad by including a series of highly constructive practices such as mock tests, mock interviews, resume preparation, frequently asked interview questions and more for our students to stay one step ahead from the lot. We at Azure Trainings, also recommend and refer our students to companies and startups that are tied up with our institute.
Students reviews
What do students say about Azure Data Factory Training In Hyderabad
About Azure Data Factory Training in Hyderabad (ADF)
Azure data factory (ADF) is a cloud-based ETL and Data Integration service created by Azure. It eases the path for an intuitive authoring experience, fully automating management tasks such as monitoring. Adf allows you to schedule and time the workflow according to your preferences.
We provide the most comprehensive Microsoft Azure Data Factory training in Hyderabad, with a conceptual curriculum and hands-on exercises that prepare students for real-world applications.
We train our students by giving them practical experience and knowledge so that learning will be more effective. The training course provides comprehensive and in-depth coverage of Microsoft Azure Data Factory.
The course will cover the following topics: Introduction to Azure Data Factory, Overview of Azure Data Factory, Architecture Creating an Azure Data Factory Training in Hyderabad, and Creating a data pipeline using Azure Resource, Manager Templates Using PowerShell for configuration and management, Monitoring your pipelines Troubleshooting Errors.
AzureTrainings is a subsidiary of Brolly Academy and aims to provide individualized Azure Data Factory Training in Hyderabad . We also certify students who complete the course. The certificate is suitable for beginners and more advanced learners and can be used to prove your competence in a CV or job application. Our instructor-led Azure Data Factory Training in Hyderabad will prepare you for the Azure certifications by imparting the necessary knowledge and skills to pass the exams.
You can also opt for self-paced video tutorial courses, where you learn at your own pace and don’t have to worry about deadlines or exam dates. The course material is available online and on mobile devices with an app version of the course.
Our training institute has the best ADF trainer in India, whose experience working with multiple organizations and companies across the country makes him well suited to coach our trainees and students.
We at AzureTrainings, conduct our classes at a level of quality that other institutes unmatch. Batch sizes are kept small to ensure each student’s best possible training experience. Join our training program and learn from the best trainers, with dedicated placement opportunities in some of Hyderabad’s top companies.
Our Azure Data Factory Course (adf course) Includes:
- Hands-on lab sessions
- Practical demonstration of ADF concepts
- Additional Adf counseling by our expert trainer
- Job based training
- Azure certification training
- Live doubt clearing sessions
- Disciplined + Consistent Adf weekly scheduled classes
- One-on-one personalized interaction with Bharat Sriram sir.
About Our Trainer
Mr.Bharat Sriram will be your Adf trainer and will be leading you through the entire process of the Azure Data Factory training program. He is a veteran in Azure and holds an excellent track record of training students to become experts in Azure. He has trained over 30,000+ students in the past few years with 25+ years of competent experience.
His passion for technology and teaching makes him even more inspiring to learn from, as he is one of the most sought-after trainers in Hyderabad with an impressive reputation. His expertise in Azure is unparalleled, and his teaching style makes it easy to understand even the most complex concepts.
He is well versed with the latest technologies, tools, and trends in the industry. Moreover, he has worked with some of the top companies and institutions in Hyderabad as a lead Adf trainer and has a vast experience in training and developing students. He is an excellent trainer, has a deep understanding of technology with patience, and takes extra effort to explain each topic in detail, ensuring that the student enjoys a satisfactory quality learning experience.
He ensures that every student gets adequate attention and time during the training sessions. His experience as a consultant helps him understand what students need to learn to become successful professionals in their careers. His approach to teaching is convenient and has helped many students pass their certification exams. He has also delivered training in Data Science, MLops, Azure DevOps, and more.
Get Certified With Azure Data Factory Course Certification
The Azure Data factory certification program is designed to provide a credential that professionals can use to demonstrate technical skills and knowledge in Azure Data Factory. ADF Certification training provides a comprehensive overview of Data Factory, including its architecture and components.
It also provides an introductory level of skills in Azure Data Factory, which will help professionals to use the tool effectively. Our trainer, Mr.Bharat, will train you with all the concepts in ADF by giving you ample knowledge of the course subjects. He will also guide you on using the software and all its features with his competent 25+ years of experience in the field.
Upon completing the course duration with our expert guidance, you will also be able to attend the ADF certification exams and get yourself certified.
Azure Data Factory certifications
The Azure Data Factory certification program currently offers four different certifications:
- Azure Data Factory Certified Professional (DCV)
- Azure Data Factory Architect (DCA)
- Azure Data Factory Administrator (DA)
- Azure Data Factory Consultant (DCC)
- The Azure Data Factory Certified Professional certification is the entry-level certification for Azure Data Factory. It’s designed to validate your knowledge of the basics of Microsoft Azure Data Factory and its associated tools, including Azure SQL Database and HDInsight.
- The second level is the Architect certification. This credential validates your ability to design, develop and deploy an enterprise-level solution.
- The third level is the Administrator certification, which validates your ability to manage and troubleshoot a data factory environment.
- Finally, the Data Factory Consultant certification validates your ability to design and implement an Azure Data Factory solution for an enterprise.
About Azure Data Factory Training institute in Hyderabad - Online Classes
Azure Trainings is one of the best Azure Data Factory Training in Hyderabad, Ameerpet with industry experts trainers. We have been training in Azure over the past 3 years and have successfully trained 800+ students across Hyderabad and Telangana.
We provide Azure placement assistance along with Azure certification and support. Enroll yourself in our Azure Training program to gain our e-book and the right skill set to make it in the IT industry.
Benefits of this Azure Data Factory (adf) Training Program
AzureTrainings offers comprehensive, hands-on Azure Data Factory training in Hyderabad that will prepare you for success with ADF’s technical and advanced concepts. We offer a full range of Adf training services, including an industry-aligned curriculum that will qualify you for Microsoft Azure Data Factory careers.
AzureTrainings is the most convenient way to learn Azure Data Factory. Moreover, we offer flexible scheduling options, including weekdays and weekend classes.
- You will be able to create a data pipeline that processes data from various sources and stores it in a destination database.
- You will be able to use Azure Data Factory to schedule and monitor your data pipelines.
- You will be able to manage the security of your Data Factory account using Azure Active Directory.
- You will be able to use Azure Data Factory to move data between on-premises and cloud environments.
- You will be able to test your data pipelines using the Azure Data Factory designer.
Job Opportunities in Azure Data Factory
The most common job roles in Azure Data Factory include:
- Data Factory Architect
- Azure Data Factory Solution Engineer
- Azure Data Factory Consultant
- Azure Data Factory Developer
- Azure Data Factory Administrator
- Azure Data Factory Manager
Skills you will gain from our Azure Data Factory Training in Hyderabad
- ADF operations
- Hybrid data ingestion and orchestration
- Managing data pipelines, flows and wrangling
- Data mashup + ETL in Azure
- Azure Synapse Architecture
- Azure Data factory architecture
- Azure data lake with ADF
- Incremental Loads with ADF and more.
Frequently Asked Questions
Who can opt for Azure Data Factory course?
Azure Data Factory is designed for many users, including data scientists and analysts who want to automate their data processing tasks. It’s also well-suited for IT professionals who need to deploy and maintain enterprise-scale workloads. Even students can take up our Azure Data Factory Training in Hyderabad.
Do I need any prior technical Knowledge to learn Azure Data Factory?
ADF is a simple programming language that anyone can learn. You don’t need prior programming or software development knowledge to learn Azure Data Factory.
Who will be my trainer?
Mr. Bharat Sriram, a renowned Azure Data Factory trainer, will be your trainer.
How do I practice Azure Data Factory?
There are several ways to practice Azure Data Factory. You can go through the tutorials and hands-on labs in the Data Factory documentation or use a tool that automatically generates a pipeline based on your data, among the other options.
Is coding required for Azure Data Factory?
Basic coding knowledge would be required to implement some of the features, but it is not necessary to use all of them. If you do not have coding skills, then it is recommended that you learn the basics.
Will I get any course completion certificate from Azure Online training in Hyderabad?
Yes, you will get a course completion certificate after completing the Azure Data Factory Training in Hyderabad.
How to register for the Azure Data Factory Training Program?
You can first try our demo session conducted by an expert from our team and join the course as you wish.
How long does it take to learn Azure Data Factory?
The time required to learn Azure Data Factory depends on your level of experience and prior knowledge. We will provide a complete Azure data factory training covering all the essential concepts and skills.
Which certification is best for ADF?
The MCSA: Data Management and Analytics is the most popular certification for Azure Data Factory. This certification covers the entire platform and includes both ADF and SQL Server Data Warehouse.