AWS Certified Machine Learning Engineer Associate · Free Practice Question Medium
Question 36
You are a Data Scientist working for an e-commerce platform that uses a machine learning model to recommend products to customers. The model has been in production for over a year and was initially performing well. However, you have recently noticed a decrease in the model's accuracy, particularly when recommending products to new customers. This decline suggests that the model may be experiencing drift due to changing customer preferences and market trends. To address this issue, you need to implement a strategy for detecting and managing model drift using Amazon SageMaker.
Which of the following strategies should you implement to effectively detect and manage model drift in your product recommendation model using Amazon SageMaker? (Select two)
-
A
Deploy multiple versions of the model simultaneously using Amazon SageMaker multi-model endpoints, and switch between them based on performance metrics
-
B
Retrain the model with the new training data, ensuring that the model remains up to date with new customer preferences
-
C
Use Amazon SageMaker Clarify to continuously monitor and mitigate bias in the model, and initiate model retraining due to changes in data distribution
-
D
Use Amazon SageMaker Model Monitor to set up monitoring for data quality and data drift, enabling you to receive alerts and initiate model retraining when the distribution of input data changes significantly from the training data
-
E
Manually review model performance every quarter and initiate retraining only if a significant drop in accuracy is observed, minimizing unnecessary retraining costs
Reveal correct answers
Correct answers: B, D
Explanation
Correct options:
Use Amazon SageMaker Model Monitor to set up monitoring for data quality and data drift, enabling you to receive alerts and initiate model retraining when the distribution of input data changes significantly from the training data
Retrain the model with the new training data, ensuring that the model remains up to date with new customer preferences
Amazon SageMaker Model Monitor allows you to continuously monitor the input data for data quality and data drift. By setting up alerts, you can detect when the input data distribution has changed significantly from the training data, which is a key indicator of model drift. This proactive approach helps in taking timely action, such as retraining the model or adjusting it to account for new patterns in the data.
With SageMaker Model Monitor, you can set alerts that notify you when there are deviations in the model quality. Early and proactive detection of these deviations lets you to take corrective actions. You can take actions like retraining models, auditing upstream systems, or fixing quality issues without having to monitor models manually or build additional tooling.
Incorrect options:
Deploy multiple versions of the model simultaneously using Amazon SageMaker multi-model endpoints, and switch between them based on performance metrics - Deploying multiple versions of the same model using Amazon SageMaker multi-model endpoints is resource-intensive and cost-inefficient. So, this option is ruled out.
Manually review model performance every quarter and initiate retraining only if a significant drop in accuracy is observed, minimizing unnecessary retraining costs - Manually reviewing performance every quarter is reactive and might not catch drift early enough, leading to prolonged periods of suboptimal model performance. Automated monitoring with Model Monitor provides a more timely and systematic approach to detecting drift.
Use Amazon SageMaker Clarify to continuously monitor and mitigate bias in the model, and initiate model retraining due to changes in data distribution - While Amazon SageMaker Clarify is useful for monitoring and mitigating bias, it is not specifically designed for detecting model drift caused by changes in data distribution. Model Monitor is the more appropriate tool for addressing drift in this context.
References:
https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html
https://docs.aws.amazon.com/machine-learning/latest/dg/retraining-models-on-new-data.html
https://aws.amazon.com/blogs/architecture/detecting-data-drift-using-amazon-sagemaker/
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
