Question 18

Explanation

Correct Answer:
✅ Gradient descent

Explanation:

For extraordinarily large datasets, Spark ML tackles linear regression problems using Gradient Descent (GD) instead of the traditional Least Squares Method.

• Why Gradient Descent?

  • The Least Squares Method requires inverting a large matrix, which is computationally expensive and infeasible for big data.

  • Gradient Descent scales efficiently because it updates model weights iteratively instead of solving an explicit equation.

  • Spark ML implements Stochastic Gradient Descent (SGD) and L-BFGS (Limited-memory BFGS) for optimization.

Example: Linear Regression in Spark ML Using Gradient Descent

from pyspark.ml.regression import LinearRegression
from pyspark.sql import SparkSession
 
# Initialize Spark session
spark = SparkSession.builder.appName("LinearRegressionExample").getOrCreate()
 
# Load data
data = [(1, 2.0, 4.0), (2, 3.0, 6.0), (3, 4.0, 8.0)]
df = spark.createDataFrame(data, ["id", "feature", "label"])
 
# Train linear regression model using gradient descent
lr = LinearRegression(featuresCol="feature", labelCol="label", solver="gd")  # Gradient Descent
 
# Fit model
model = lr.fit(df)
 
print("Coefficients:", model.coefficients)
print("Intercept:", model.intercept)

• solver="gd" ensures that gradient descent is used for optimization.

• Spark also supports L-BFGS solver, which is an improvement over basic GD.

Why Other Options Are Incorrect?

1. Brute Force Algorithm

   • Incorrect, because brute force is not computationally feasible for large datasets.

   • Spark uses optimization algorithms like GD instead of exhaustive search.

2. Matrix Decomposition

   • Incorrect, because matrix decomposition methods (e.g., Cholesky, LU decomposition) require storing large matrices, which is impractical for large-scale ML.

3. Singular Value Decomposition (SVD)

   • Incorrect, because SVD is used for dimensionality reduction and PCA, not for training regression models efficiently.

4. Least Squares Method

   • Incorrect, because the Least Squares Method requires inverting a large matrix, which is computationally expensive for big data.

   • Spark avoids direct matrix inversion and uses Gradient Descent instead.

Final Answer:

✅ Spark ML tackles large-scale linear regression problems using Gradient Descent, making it scalable and efficient for big data processing.