Databricks Certified Data Engineer Professional · Free Practice Question Medium
Question 1
The data engineering team maintains the following join logic between three Delta tables:
- df_students = spark.table("students")
- df_courses = spark.table("courses")
- df_enrollments = spark.table("enrollments")
- df_join_1 = (df_students.join(df_enrollments, df_students.student_id == df_enrollments.student_id)
- .select(df_students.student_id,
- df_students.student_name,
- df_enrollments.course_id)
- )
- df_join_2 = (df_join_1.join(df_courses, df_join_1.course_id == df_courses.course_id)
- .select(df_join_1.student_id,
- df_join_1.student_name,
- df_courses.course_name)
- )
- (df_join_2.write
- .mode("overwrite")
- .table("students_courses_details"))
Which statement describes the result of this code block each time it is executed ?
-
A
All records in the current version of the source tables will be considered in the join operations. The matched records will overwrite the students_courses_details table.
-
B
All records in the current version of the source tables will be considered in the join operations. The unmatched records will overwrite the students_courses_details table.
-
C
Only newly added records to any of the source tables will be considered in the join operations. The matched records will overwrite the students_courses_details table.
-
D
Only newly added records to any of the source tables will be considered in the join operations. The unmatched records will overwrite the students_courses_details table.
-
E
These join operations are stateful, meaning that they will wait for unmatched records to be added to the source tables prior to calculating the results.
Reveal correct answer
Correct answer: A
Explanation
The query reads three static Delta tables using spark.table() function, which means that all records in the current version of these tables will be read and considered in the join operations.
There is no difference between spark.table() and spark.read.table() function. Actually, spark.read.table() internally calls spark.table().
The pyspark.sql.DataFrame.join() function performs inner join operation by default, so the matched records will be written to the target table. In our case, the query writes the data in mode “overwrite” to the target table, which completely overwrites the table.
Reference:
https://spark.apache.org/docs/3.1.2/api/python/reference/api/pyspark.sql.DataFrame.join.html
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
