Question 5

Explanation

Correct Answer:
The employment of an internalFrame to maintain metadata

Explanation:

The pandas API on Spark (Koalas) introduces an internalFrame layer to bridge pandas-like operations with Spark's distributed execution. While this enables familiar syntax, it adds overhead due to:

1. Metadata Management:

   • The internalFrame tracks pandas-like indices, column names, and data types, requiring extra bookkeeping.

2. Conversion Costs:

   • Pandas operations are translated to Spark plans via this layer, which can slow down execution compared to native Spark DataFrames.

Example Impact:

import pyspark.pandas as ps
 
# pandas API on Spark (uses internalFrame)
kdf = ps.DataFrame(...)
result = kdf.groupby("col1").sum()  # Slower due to metadata handling
 
# Native Spark (direct execution)
sdf = spark.createDataFrame(...)
result = sdf.groupBy("col1").sum()  # Faster

Why Other Options Are Incorrect:

1. "Increased code amount":

   • Irrelevant; performance is about execution, not code volume.

2. "Dependence on CSV files":

   • Unrelated; data source format doesn’t affect API performance.

3. "Immediate evaluation":

   • Both APIs use lazy evaluation.

4. "Absence of data distribution":

   • False; pandas API on Spark does distribute data (unlike vanilla pandas).

Key Takeaway:

For large datasets, prefer native Spark DataFrames when:
✅ Performance is critical (avoid internalFrame overhead).
✅ Advanced Spark optimizations (e.g., predicate pushdown) are needed.

Use pandas API on Spark for:
✅ Pandas familiarity on small-to-medium distributed data.