Question 10

Explanation

Correct Answer:
To reduce the number of input features

Explanation:

1. Why This Is Correct?

   • Dimensionality reduction (e.g., PCA, Feature Selection) in Spark ML aims to:

     • Eliminate redundant/irrelevant features: Focus on the most informative ones.

     • Reduce computational cost: Fewer features speed up training and inference.

     • Improve model performance: Mitigates the "curse of dimensionality" (noise from irrelevant features).

   • Example (PCA in Spark ML):

     
     

     from pyspark.ml.feature import PCA
     pca = PCA(k=10, inputCol="features", outputCol="pca_features")
     model = pca.fit(scaled_df)  # Reduces 100D features → 10D

2. Key Benefits:

   • Faster training: Less data to process.

   • Better generalization: Removes noise.

Why Other Options Are Incorrect?

• "Increase model complexity":

  • Dimensionality reduction simplifies models.

• "Add irrelevant features":

  • Counterproductive—reduction removes irrelevance.

• "Speed up preprocessing":

  • A side effect, not the primary goal.

Key Takeaway:

For efficient Spark ML workflows:

1. Use PCA for linear dependencies.

2. Use ChiSqSelector for categorical features.

3. Validate with cross-validation to choose optimal dimensions.

Pro Tip: Pair with VectorSlicer to manually drop low-importance features.