Question 7

Explanation

Correct Answer:
A new default index is created.

Explanation:

1. Why This Is Correct?

   • When converting a Spark DataFrame to a pandas-on-Spark DataFrame, the library automatically generates a new default index (sequential integers) unless explicitly specified.

   • This is because:

     • Spark DataFrames are distributed and do not inherently have row indices.

     • pandas-on-Spark mimics pandas behavior, where an index is fundamental.

   • Example:

     
     

     import pyspark.pandas as ps
     spark_df = spark.createDataFrame([(1, "A"), (2, "B")], ["id", "value"])
     ps_df = ps.DataFrame(spark_df)  # New default index (0, 1, ...) is created

2. Key Implications:

   • Performance Overhead: Index creation requires shuffling data to ensure uniqueness.

   • Data Integrity: The new index does not preserve the original Spark row order.

Why Other Options Are Incorrect?

• "Remains unchanged":

  • Spark DataFrames lack a default index, so nothing to preserve.

• "Depends on dataset size":

  • Index creation is consistent (always happens).

• "Set to 'index_col'":

  • Only occurs if you explicitly set index_col during conversion.

Key Takeaway:

To avoid surprises:

• Explicitly set an index if needed (e.g., ps.DataFrame(spark_df, index="id")).

• Use spark_df.to_pandas_on_spark() for clarity.

Pro Tip: For large DataFrames, avoid default indices—use existing columns as indices to minimize shuffling.