Question 4

Explanation

Correct Answer:
compute.shortcut_limit

Explanation:

1. Why This Is Correct?

   • compute.shortcut_limit in pandas API on Spark determines how many rows are computed locally (on the driver) to infer schema or perform quick checks, avoiding full distributed computation.

     • Default: Typically 1000 rows.

     • Use Case: Speeds up operations like df.head() or schema inference by sampling.

   • Example:

     import pyspark.pandas as ps
     ps.set_option("compute.shortcut_limit", 500)  # Only compute 500 rows for schema checks

2. Key Impact:

   • Performance: Reduces latency for metadata operations.

   • Trade-off: Too low → schema inference errors; too high → slower driver ops.

Why Other Options Are Incorrect?

• display.max_rows:

  • Controls how many rows print/show (e.g., df.display()).

• compute.ops_on_diff_frames:

  • Allows operations between unrelated DataFrames (unrelated to sampling).

• compute.default_index_type:

  • Sets index type (e.g., "distributed" vs. "sequence").

Key Takeaway:

For efficient schema checks in pandas-on-Spark:

1. Adjust compute.shortcut_limit based on data size.

2. Use df.spark.schema() for full schema without sampling.

Pro Tip: Increase this value if you encounter schema mismatches in large DataFrames.