Question 20

Explanation

Correct Answer:
transform_batch requires the length of input and output to be the same, while apply_batch does not have this restriction.

Explanation:

1. Why This Is Correct?

   • transform_batch():

     • Input/Output Length Must Match: The function must return a DataFrame/Series with the same number of rows as the input. Ideal for row-wise transformations (e.g., scaling, filtering).

     • Example:

       
       

       def scale(df):
           return df * 2  # Output length = input length
       df.pandas_on_spark.transform_batch(scale)

   • apply_batch():

     • Flexible Output Length: Can return a DataFrame/Series of any size (e.g., aggregations, group-wise stats).

     • Example:

       
       

       def summary(df):
           return df.mean()  # Output length ≠ input length (aggregation)
       df.pandas_on_spark.apply_batch(summary)

2. Key Difference:

   MethodInput/Output LengthUse Casetransform_batchMust matchRow-wise operationsapply_batchCan differAggregations, summaries

Why Other Options Are Incorrect?

• "Interchangeable":

  • They serve different purposes (fixed vs. flexible output size).

• "apply_batch has length restriction":

  • Incorrect—apply_batch allows variable output.

• "Same restriction":

  • False—only transform_batch enforces length matching.

Key Takeaway:

• Use transform_batch for 1:1 row transformations.

• Use apply_batch for aggregations or variable-length outputs.

Pro Tip: For grouped operations, combine with groupby.apply_batch().