Question 17

Explanation

Correct Answer:
Workflow orchestration

Explanation:

1. Why This Is Correct?

   • Workflow orchestration (e.g., Airflow, Databricks Workflows, Kubeflow) is essential for managing multi-task ML pipelines, including:

     • Dependencies: Ensuring tasks (data prep → training → deployment) run in order.

     • Scheduling: Automating recurring runs (e.g., nightly model retraining).

     • Error Handling: Retries, alerts, and logging for robustness.

   • Example:

     # Databricks Workflows example  
     from databricks.sdk import WorkspaceClient  
     w = WorkspaceClient()  
     w.jobs.submit(run_name="retrain_model", tasks=[data_task, train_task, deploy_task])  

2. Impact on Multi-Task ML:

   • Reproducibility: Track inputs/outputs across tasks.

   • Scalability: Parallelize tasks across clusters.

Why Other Options Are Less Critical?

• Efficient cluster creation: Important, but handled by infra teams/orchestrators.

• Performance optimization: Task-specific (e.g., model tuning), not pipeline-wide.

• Feature engineering: A single task within the workflow.

Key Takeaway:

Master orchestration tools (e.g., Airflow, MLflow Pipelines, Prefect) to design resilient, automated ML workflows.

Pro Tip: Use MLflow Projects + Databricks Jobs for seamless orchestration with experiment tracking.