Question 14

Explanation

Correct Answer:
Use MLflow to package the model and deploy it as a REST API endpoint.

Explanation:

The most robust and scalable approach for deploying a Databricks MLlib model in production is to use MLflow to:

1. Package the Model:

   • Log the trained model using mlflow.spark.log_model().

   • MLflow automatically captures dependencies (e.g., PySpark, Python version).

2. Deploy as a REST API:

   • Use MLflow Model Serving or deploy to Azure ML/AWS SageMaker for real-time inference.

   • Example:

     import mlflow
     from pyspark.ml import PipelineModel
      
     # Log the model
     model = PipelineModel.load("path/to/model")
     mlflow.spark.log_model(model, "spark-model")
      
     # Deploy (e.g., via Databricks Model Serving UI)

   • This provides a scalable, low-latency endpoint for predictions.

Why Other Options Are Incorrect:

1. Export as a serialized file and deploy on a separate server:

   • Manual deployment is error-prone (dependency mismatches, scaling issues).

   • Lacks the monitoring and scalability of MLflow.

2. Save to a Delta table and query for predictions:

   • Delta tables are for data storage, not model serving.

   • Real-time predictions require low-latency APIs, not batch queries.

3. Schedule a Databricks Job for periodic runs:

   • Jobs are for batch processing, not real-time inference.

Key Benefits of MLflow Deployment:

✅ Dependency Management: Auto-captures Python/Spark versions.
✅ Scalability: Built-in load balancing (e.g., via Databricks Model Serving).
✅ Monitoring: Track latency, errors, and usage metrics.

Example Workflow:

1. Train → Log model with MLflow.

2. Deploy → Enable "Model Serving" in Databricks UI or export to cloud platforms.

3. Call API:

   import requests
   response = requests.post("https://<model-endpoint>/invocations", json=input_data)