Question

What are the steps involved in the ETL process?

Answer 1

ETL (Extract, Transform, Load) process follows four main steps:

Connect and Collect – helps in moving the data on-premises and cloud source data stores

• Transform – lets users collect the data by using compute services such as HDInsight Hadoop, Spark etc.
• Publish – Helps in loading the data into Azure data warehouse, Azure SQL database, and Azure Cosmos DB etc
• Monitor – It helps support the pipeline monitoring via Azure Monitor, API and PowerShell, Log Analytics, and health panels on the Azure Portal.

Answer 2

The ETL (Extract, Transform, Load) process follows four main steps:

i) Connect and Collect: Connect to the data source/s and move data to local and crowdsource data storage.

ii) Data transformation using computing services such as HDInsight, Hadoop, Spark etc.

iii) Publish: To load data into Azure data lake storage, Azure SQL data warehouse, Azure SQL databases, Azure Cosmos DB, etc.

iv)Monitor: Azure Data Factory has built-in support for pipeline monitoring via Azure Monitor, API, PowerShell, Azure Monitor logs, and health panels on the Azure portal.

Build a unique job-winning data engineer resume with big data mini projects.