Databricks Certified Associate Developer For Apache Spark 30 · Free Practice Question Easy
Question 2
- A Failure of driver node
- B Shuffle
- C Failure of worker node
- D Failure of network
Reveal correct answer
Correct answer: B
Explanation
A shuffle occurs when a Spark job needs to perform an operation that cannot be executed on a single machine, such as a groupByKey or a join. In these cases, the data must be redistributed across the partitions of the RDD so that the operation can be performed in parallel on different machines.
The shuffle process involves two stages: a map stage and a reduce stage. The map stage processes the input data and produces a set of intermediate key-value pairs, which are then shuffled and sorted by key. The reduce stage consumes the sorted key-value pairs and produces the final output data.
The map and reduce stages of the shuffle are separated by a stage boundary, which indicates that the data is being redistributed and that the next stage of the job cannot be executed until the shuffle is complete.
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
