Question 8

Explanation

Correct Answer:
✅ Adding /databricks/python/bin/pip install newpackage to the cluster’s bash init script

Explanation:

To make the newpackage library available to all notebooks on a shared cluster, the best approach is to install it via the cluster’s init script.

• Why?

  • Init scripts run automatically when the cluster starts, ensuring the package is installed for all users and notebooks.

  • This eliminates the need for reinstalling the package in each session.

  • Works across all users and all projects using the same cluster.

How to Add an Init Script to a Databricks Cluster

1. Navigate to Clusters in the Databricks workspace.

2. Select the cluster you want to modify.

3. Click Edit and scroll down to Advanced Options → Init Scripts.

4. Add the following command to install newpackage:

   Example Init Script (Bash)

   #!/bin/bash
   /databricks/python/bin/pip install newpackage

5. Save the changes and restart the cluster.

6. The package will now be available in all notebooks attached to this cluster.

Why Other Options Are Incorrect?

1. "Edit the cluster to use the Databricks Runtime for Machine Learning"

   • Incorrect, because Databricks ML Runtime includes pre-installed ML libraries but does not automatically install custom packages like newpackage.

2. "Set the runtime-version variable in their Spark session to 'ml'"

   • Incorrect, because there is no runtime-version variable in Spark that controls package availability.

   • Databricks runtime selection happens at the cluster level, not in a notebook session.

3. "Running %pip install newpackage once on any notebook attached to the cluster"

   • Incorrect, because installing with %pip install only applies to the current session.

   • Once the cluster restarts, the package will be removed, requiring reinstallation.

4. "There is no way to make the newpackage library available on a cluster"

   • Incorrect, because Databricks provides multiple ways to install persistent libraries (e.g., init scripts, cluster libraries, workspace libraries).

Final Answer:

✅ Using an init script (/databricks/python/bin/pip install newpackage) ensures the package is installed persistently on the cluster for all users and notebooks.