Databricks Certified Data Engineer Associate · Free Practice Question Medium
Question 22
You noticed that a team member started using an all-purpose cluster to develop a notebook and used the same all-purpose cluster to set up a job that can run every 30 mins so they can update underlying tables which are used in a dashboard. What would you recommend for reducing the overall cost of this approach?
-
A
Reduce the size of the cluster
-
B
Reduce the number of nodes and enable auto scale
-
C
Enable auto termination after 30 mins
-
D
Change the cluster all-purpose to job cluster when scheduling the job
-
E
Change the cluster mode from all-purpose to single-mode
Reveal correct answer
Correct answer: D
Explanation
While using an all-purpose cluster is ok during development but anytime you don't need to interact with a notebook, especially for a scheduled job it is less expensive to use a job cluster. Using an all-purpose cluster can be twice as expensive as a job cluster.
Please note: The compute cost you pay the cloud provider for the same cluster type and size between an all-purpose cluster and job cluster is the same the only difference is the DBU cost.
The total cost of cluster = Total cost of VM compute(Azure or AWS or GCP) + Cost per DBU
The per DBU cost varies between all-purpose and Job Cluster
Here is the recent cost estimate from AWS between Jobs Cluster and all-purpose Cluster, for jobs compute its $0.15 cents per DBU v$0.55 cents per DBU for all-purpose
How do I check how much the DBU cost for my cluster?
When you click on an exister cluster or when you look at the cluster details you will see this in the top right corner
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
