Microsoft Certified Azure Data Engineer Associate · Free Practice Question Medium
Question 8
Azure Databricks is an amalgamation of multiple technologies that enable you to work with data at scale.
Which of the following is best described by “This an open-source technology used to define a relational abstraction layer of tables over file-based data. The tables can then be queried using SQL syntax. The table definitions and details of the file system locations on which they're based is stored in the this for a Spark cluster. This is created for each cluster when it's created, but you can configure a cluster to use an existing one if necessary”?
-
A
SQL Warehouses
-
B
Notebook
-
C
Delta Lake
-
D
Databricks File System
-
E
Hive metastore
-
F
Apache Spark clusters
Reveal correct answer
Correct answer: E
Explanation
Understand key concepts
Azure Databricks is an amalgamation of multiple technologies that enable you to work with data at scale. Before using Azure Databricks, there are some key concepts that you should understand.
Apache Spark clusters - Spark is a distributed data processing solution that makes use of clusters to scale processing across multiple compute nodes. Each Spark cluster has a driver node to coordinate processing jobs, and one or more worker nodes on which the processing occurs. This distributed model enables each node to operate on a subset of the job in parallel; reducing the overall time for the job to complete.
https://learn.microsoft.com/en-us/azure/databricks/clusters/
Databricks File System (DBFS) - While each cluster node has its own local file system (on which operating system and other node-specific files are stored), the nodes in a cluster have access to a shared, distributed file system in which they can access and operate on data files. The Databricks File System (DBFS) enables you to mount cloud storage and use it to work with and persist file-based data.
https://learn.microsoft.com/en-us/azure/databricks/data/databricks-file-system
Notebooks - One of the most common ways for data analysts, data scientists, data engineers, and developers to work with Spark is to write code in notebooks. Notebooks provide an interactive environment in which you can combine text and graphics in Markdown format with cells containing code that you run interactively in the notebook session.
https://learn.microsoft.com/en-us/azure/databricks/notebooks/
Hive metastore - Hive is an open-source technology used to define a relational abstraction layer of tables over file-based data. The tables can then be queried using SQL syntax. The table definitions and details of the file system locations on which they're based is stored in the metastore for a Spark cluster. A Hive metastore is created for each cluster when it's created, but you can configure a cluster to use an existing external metastore if necessary.
https://learn.microsoft.com/en-us/azure/databricks/data/metastores/
Delta Lake - Delta Lake builds on the relational table schema abstraction over files in the data lake to add support for SQL semantics commonly found in relational database systems. Capabilities provided by Delta Lake include transaction logging, data type constraints, and the ability to incorporate streaming data into a relational table.
https://learn.microsoft.com/en-us/azure/databricks/delta/
SQL Warehouses - SQL Warehouses are relational compute resources with endpoints that enable client applications to connect to an Azure Databricks workspace and use SQL to work with data in tables. The results of SQL queries can be used to create data visualizations and dashboards to support business analytics and decision making. SQL Warehouses are only available in premium tier Azure Databricks workspaces.
https://learn.microsoft.com/en-us/azure/databricks/sql/admin/sql-endpoints
A. SQL Warehouses are databases optimized for analytical workloads in Azure Databricks. While they are used for storing and querying data, they are not specifically used to define a relational abstraction layer of tables over file-based data.
B. Notebooks are interactive documents that contain code, visualizations, and narrative text. While they are commonly used in Azure Databricks for data analysis and collaboration, they are not specifically used to define a relational abstraction layer of tables over file-based data.
C. Delta Lake is a technology that provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing in Azure Databricks. It is not specifically related to defining a relational abstraction layer of tables over file-based data.
D. Databricks File System (DBFS) is a distributed file system that is part of Azure Databricks. It is used for storing data and files, but it is not specifically used to define a relational abstraction layer of tables over file-based data.
E. The description provided matches the functionality of a Hive metastore, which is an open-source technology used to define a relational abstraction layer of tables over file-based data in Azure Databricks. It stores table definitions and file system locations for querying with SQL syntax.
F. Apache Spark clusters are the computing engines that process data in Azure Databricks. While they are essential for data processing, they are not specifically used to define a relational abstraction layer of tables over file-based data.
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
