Microsoft Certified Azure Data Engineer Associate · Free Practice Question Medium
Question 40
Scenario: Mercs for Money is a collection of mercenaries gathered together by Wade Wilson who organized the group and is now a successful Professional Services enterprise. Looking to improve the operations of the organization, Wade has contracted you to assist with several IT projects.
The current project is using Spark Structured Streaming. Wade wants to use a Delta Lake table as a source.
A Delta Lake table can serve as which of the following?
-
A
Only a source
-
B
Only a sink
-
C
None of the available responses are correct
-
D
Either a source or a sink
Reveal correct answer
Correct answer: D
Explanation
A Delta Lake table can be a source or a sink.
Delta Lake is an open-source storage layer that adds relational database semantics to Spark-based data lake processing. Delta Lake is supported in Azure Synapse Analytics Spark pools for PySpark, Scala, and .NET code.
The benefits of using Delta Lake in a Synapse Analytics Spark pool include:
Relational tables that support querying and data modification. With Delta Lake, you can store data in tables that support CRUD (create, read, update, and delete) operations. In other words, you can select, insert, update, and delete rows of data in the same way you would in a relational database system.
Support for ACID transactions. Relational databases are designed to support transactional data modifications that provide atomicity (transactions complete as a single unit of work), consistency (transactions leave the database in a consistent state), isolation (in-process transactions can't interfere with one another), and durability (when a transaction completes, the changes it made are persisted). Delta Lake brings this same transactional support to Spark by implementing a transaction log and enforcing serializable isolation for concurrent operations.
Data versioning and time travel. Because all transactions are logged in the transaction log, you can track multiple versions of each table row and even use the time travel feature to retrieve a previous version of a row in a query.
Support for batch and streaming data. While most relational databases include tables that store static data, Spark includes native support for streaming data through the Spark Structured Streaming API. Delta Lake tables can be used as both sinks (destinations) and sources for streaming data.
Standard formats and interoperability. The underlying data for Delta Lake tables is stored in Parquet format, which is commonly used in data lake ingestion pipelines. Additionally, you can use the serverless SQL pool in Azure Synapse Analytics to query Delta Lake tables in SQL.
A. A Delta Lake table can indeed serve as a source in Spark Structured Streaming, providing data for processing in the streaming pipeline. However, it can also serve as a sink, storing the processed data back to the Delta Lake table or another storage system.
B. While a Delta Lake table can be used as a sink in Spark Structured Streaming to store the processed data, it can also serve as a source, providing data for processing in the streaming pipeline. Therefore, it is not accurate to say that it can only be a sink.
C.
A Delta Lake table can serve as either a source or a sink in Spark Structured Streaming. As a source, it can provide data for processing, and as a sink, it can store the processed data back to the Delta Lake table or another storage system.
D. A Delta Lake table can serve as either a source or a sink in Spark Structured Streaming. As a source, it can provide data for processing, and as a sink, it can store the processed data back to the Delta Lake table or another storage system.
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
