Databricks Certified Data Engineer Associate · Free Practice Question Medium
Question 5
A data engineer at an HR analytics company is developing a PySpark pipeline to analyze salary metrics across departments. They wrote the following line of code to compute the total, average, and count of salaries per department:
result_df = df.groupBy("department").agg({"salary": "sum", "salary": "avg", "salary": "count"})
After running the code, they observed that the resulting DataFrame only contains one aggregated value instead of the three expected metrics.
What is the most probable cause of this issue?
-
A
The agg() method only supports one aggregation function at a time.
-
B
Python dictionaries do not allow duplicate keys, so only the last aggregation is applied.
-
C
The agg() method must use a list of tuples rather than a dictionary when aggregating multiple functions.
-
D
The groupBy() method must be preceded by a select() method for columns used in aggregation.
Reveal correct answer
Correct answer: B
Explanation
In this implementation, a dictionary is passed to the agg() method to specify multiple aggregation functions for the "salary" column. However, Python dictionaries must have unique keys, and using "salary" as a key multiple times results in the earlier entries being silently overwritten. In the example provided, only the last aggregation ("salary": "count") is actually executed.
To correctly compute multiple metrics on the same column, the appropriate method is to use functions from the pyspark.sql.functions module, with each aggregation defined explicitly:
- from pyspark.sql import functions as F
- result_df = df.groupBy("department").agg(
- F.sum("salary").alias("total_salary"),
- F.avg("salary").alias("average_salary"),
- F.count("salary").alias("salary_count")
- )
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
