Microsoft Certified Azure Data Engineer Associate · Free Practice Question Medium
Question 26
Identify the missing word(s) in the following sentence within the context of Microsoft Azure.
Apache Spark Structured Streaming is a fast, scalable, and fault-tolerant stream processing API. You can use it to perform analytics on your streaming data in [ ? ].
-
A
Real-time
-
B
Near real-time
-
C
Historic batches
-
D
Prediction mode
Reveal correct answer
Correct answer: B
Explanation
Apache Spark Structured Streaming is a fast, scalable, and fault-tolerant stream processing API. You can use it to perform analytics on your streaming data in near real time.
With Structured Streaming, you can use SQL queries to process streaming data in the same way that you would process static data. The API continuously increments and updates the final data.
Event Hubs and Spark Structured Streaming
Azure Event Hubs is a scalable real-time data ingestion service that processes millions of data in a matter of seconds. It can receive large amounts of data from multiple sources and stream the prepared data to Azure Data Lake or Azure Blob storage.
Azure Event Hubs can be integrated with Spark Structured Streaming to perform processing of messages in near real time. You can query and analyze the processed data as it comes by using a Structured Streaming query and Spark SQL.
Streaming concepts
Stream processing is where you continuously incorporate new data into Data Lake storage and compute results. The streaming data comes in faster than it can be consumed when using traditional batch-related processing techniques. A stream of data is treated as a table to which data is continuously appended. Examples of such data include bank card transactions, Internet of Things (IoT) device data, and video game play events.
A streaming system consists of:
Input sources such as Kafka, Azure Event Hubs, IoT Hub, files on a distributed system, or TCP-IP sockets
Stream processing using Structured Streaming, forEach sinks, memory sinks, etc.
A. While Apache Spark Structured Streaming can process data in real-time, the more common use case is for near real-time processing, making "Near real-time" the more appropriate choice in the sentence.
B. The missing word in the sentence should be "Near real-time" as Apache Spark Structured Streaming is commonly used for processing data in near real-time, allowing for fast and scalable analytics on streaming data.
C. "Historic batches" do not align with the capabilities of Apache Spark Structured Streaming, which is designed for stream processing and not batch processing of historical data.
D. "Prediction mode" does not accurately describe the functionality of Apache Spark Structured Streaming, which is focused on real-time or near real-time stream processing rather than predictive analytics.
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
