Microsoft Certified Azure Data Engineer Associate · Free Practice Question Medium
Question 5
Because the Databricks API is declarative, a large number of optimizations are available to us. Among the most powerful components of Spark are Spark SQL. At its core lies the Catalyst optimizer.
When you execute code, Spark SQL uses Catalyst's general tree transformation framework in four phases, as shown below:
analyzing a logical plan to resolve references
logical plan optimization
physical planning
code generation to compile parts of the query to Java bytecode
In the physical planning phase, Catalyst may generate multiple plans and compare them based on [?].
-
A
Region
-
B
Rules
-
C
Cost
-
D
Permissions
Reveal correct answer
Correct answer: C
Explanation
Because the Databricks API is declarative, a large number of optimizations are available to us.
Some of the examples include:
Optimizing data type for storage
Rewriting queries for performance
Predicate push downs
Among the most powerful components of Spark are Spark SQL. At its core lies the Catalyst optimizer. This extensible query optimizer supports both rule-based and cost-based optimization.
When you execute code, Spark SQL uses Catalyst's general tree transformation framework in four phases, as shown below:
analyzing a logical plan to resolve references
logical plan optimization
physical planning
code generation to compile parts of the query to Java bytecode
In the physical planning phase, Catalyst may generate multiple plans and compare them based on cost. All other phases are purely rule-based.
Catalyst is based on functional programming constructs in Scala and designed with these key two purposes:
Easily add new optimization techniques and features to Spark SQL
Enable external developers to extend the optimizer (e.g. adding data source specific rules, support for new data types, etc.)
A. Region is not a factor used by Catalyst to compare and select execution plans in the physical planning phase. Regions are typically used in cloud computing to define geographical locations for data storage and processing, not in the context of query optimization in Spark SQL.
B. Rules are used in the logical plan optimization phase of Spark SQL, not in the physical planning phase where Catalyst compares multiple plans based on cost. Rules are transformations applied to the logical plan to optimize it before generating the physical plan.
C. In the physical planning phase, Catalyst may generate multiple plans and compare them based on cost. Cost-based optimization is crucial in determining the most efficient execution plan for a query by estimating the cost of different execution strategies and selecting the one with the lowest cost.
D. Permissions are not used to compare and select execution plans in the physical planning phase of Spark SQL. Permissions are related to access control and security, not query optimization.
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
