Databricks Certified Data Engineer Professional · Free Practice Question Medium
Question 3
A data engineer has a MLFlow model logged in a given “model_url”. They have registered the model as a Spark UDF using the following code:
predict_udf = mlflow.pyfunc.spark_udf(spark, "model_url")
The data engineer wants to apply this model UDF to a test dataset loaded in the “test_df” DataFrame in order to calculate predictions in a new column “prediction”
Which of the following code blocks allows the data engineer to accomplish this task ?
-
A
test_df.apply(predict_udf, *column_list).select(“record_id”, “prediction")
-
B
test_df.select(“record_id”, predict_udf(*column_list).alias("prediction"))
-
C
predict_udf(“record_id”, test_df).select(“record_id”, “prediction")
-
D
mlflow.pyfunc.map(predict_udf, test_df, “record_id”).alias("prediction")
Reveal correct answer
Correct answer: B
Explanation
In PySpark Dataframe, you can create a new column based on function return value. This can be achieved by calling the function using either:
Dataframe.withColumn method:
test_df.withColumn("prediction", predict_udf(*column_list))
Or using Dataframe.select method:
test_df.select( predict_udf(*column_list).alias("prediction") )
Dataframe.select allows also to select one or more columns:
test_df.select(“record_id”, predict_udf(*column_list).alias("prediction"))
or it can be expanded to include all columns using “*”
test_df.select(“*”, predict_udf(*column_list).alias("prediction"))
Reference:
https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.select.html
https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrame.withColumn.html
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
