AWS Certified Machine Learning Specialty · Free Practice Question Hard
Question 10
A data science team at your company is planning to utilize Amazon SageMaker to train an XGBoost model to predict customer churn. The dataset comprises millions of rows, necessitating significant pre-processing to ensure model accuracy. To handle this task efficiently, the team has decided to leverage Apache Spark due to its capability for large-scale data processing. As the lead architect, you are tasked with designing a solution that integrates Apache Spark for data pre-processing while optimizing for simplicity and scalability.
What is the simplest architecture that allows the team to pre-process the data at scale using Apache Spark before training the model with XGBoost on SageMaker?
-
A
Configure Livy in SageMaker, preprocess data on EMR Spark via SageMaker notebook, train XGBoost PySpark Estimator, and host on SageMaker for inference.
-
B
Use SageMaker Spark to preprocess data, train with XGBoostSageMakerEstimator, and host on SageMaker for inference.
-
C
Configure Sparkmagic in SageMaker, preprocess data on EMR Spark via SageMaker notebook, train with SageMaker XGBoost, and host on SageMaker for inference.
-
D
Preprocess data on EMR Spark, save in S3, use SageMaker to train XGBoost, and host on SageMaker for inference.
Reveal correct answer
Correct answer: B
Explanation
Consider the integration points between EMR Spark and SageMaker. Choose based on where your processing and model training will primarily occur. The simplest architecture is one that minimizes maintenance and leverages AWS SageMaker's built-in features to the fullest.
Be aware that you can utilize the SageMaker Spark library to invoke SageMaker from an EMR Spark cluster, or alternatively, use Sparkmagic or Livy to access EMR Spark from a SageMaker notebook. The decision on which approach to use hinges on whether your workflow involves an EMR batch pipeline requiring integration with SageMaker, or vice versa.
Regarding model selection, several options are available:
SageMaker Spark offers an XGBoostEstimator
SageMaker features the SageMaker XGBoost algorithm
XGBoost PySpark Estimator
Correct Choice: Use SageMaker Spark to preprocess data, train with XGBoostSageMakerEstimator, and host on SageMaker.
The SageMaker Spark library facilitates the execution of Spark jobs as part of the machine learning pipeline within SageMaker, without the user needing to set up and manage an EMR cluster or deal with the intricacies of Spark cluster configuration and scaling.
Incorrect Choice: Preprocess data on EMR Spark, save in S3, use SageMaker to train XGBoost, and host for inference.
Valid solution; however, it overlooks AWS integration between EMR and SageMaker, and adds an unnecessary intermediary step via S3.
Incorrect Choice: Configure Sparkmagic in SageMaker, preprocess data on EMR Spark via SageMaker notebook, train with SageMaker XGBoost, and host on SageMaker for inference.
Valid solution; however, it necessitates provisioning an EMR Spark cluster.
Incorrect Choice: Configure Livy in SageMaker, preprocess data on EMR Spark via SageMaker notebook, train XGBoost PySpark Estimator, and host on SageMaker for inference.
Valid solution; however, it necessitates provisioning an EMR Spark cluster.
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
