Databricks Certified Data Engineer Professional · Free Practice Question Medium
Question 2
A junior data engineer is using the following code to de-duplicate raw streaming data and insert them in a target Delta table
- spark.readStream
- .table("orders_raw")
- .dropDuplicates(["order_id", "order_timestamp"])
- .writeStream
- .option("checkpointLocation", "dbfs:/checkpoints")
- .table("orders_unique")
A senior data engineer pointed out that this approach is not enough for having distinct records in the target table when there are late-arriving, duplicate records.
Which of the following could explain the senior data engineer’s remark?
-
A
Watermarking is also needed to only track state information for a window of time in which we expect records could be delayed.
-
B
A ranking function is also needed to ensure processing only the most recent records
-
C
A window function is also needed to apply deduplication for each non-overlapping interval.
-
D
The new records need also to be deduplicated against previously inserted data into the table.
-
E
More information is needed to determine the correct response
Reveal correct answer
Correct answer: D
Explanation
To perform streaming deduplication, we use dropDuplicates() function to eliminate duplicate records within each new micro batch. In addition, we need to ensure that records to be inserted are not already in the target table. We can achieve this using insert-only merge.
Reference:
https://docs.databricks.com/delta/merge.html#data-deduplication-when-writing-into-delta-tables
Study materials from our exam preparation course on Udemy:
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
