[SPARK-7222][ML] Added mathematical derivation in comment and compressed the model, removed the correction terms in LinearRegression with ElasticNet

[dbtsai]

Added detailed mathematical derivation of how scaling and LeastSquaresAggregator work. Refactored the code so the model is compressed based on the storage. We may try compression based on the prediction time.

Also, I found that diffSum will be always zero mathematically, so no corrections are required.

[SparkQA]

Test build #31249 has finished for PR 5767 at commit f135c2b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #31253 has finished for PR 5767 at commit 63f7d1e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Repartition(numPartitions: Int, shuffle: Boolean, child: LogicalPlan)
  • case class RepartitionByExpression(partitionExpressions: Seq[Expression], child: LogicalPlan)
  • case class Repartition(numPartitions: Int, shuffle: Boolean, child: SparkPlan)
• This patch does not change any dependencies.

[viirya]

...the intercept is yMean...

[dbtsai]

thanks.

[SparkQA]

Test build #31292 has finished for PR 5767 at commit 5929e49.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[mengxr]

Is it really rare? If yStd is 0.0 and then the optimal model would be empty with intercept yMean. In this case, a warning would be proper. Having this giant if ... else block making the code hard to read.

if (yStd == 0.0) {
  logWarning(...)
  if (handlePersistence) ...
  return new LinearRegressionModel(...)
}

// actual implementation

[dbtsai]

fair!

[SparkQA]

Test build #31316 has finished for PR 5767 at commit 69757b8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[mengxr]

minor: x.foreach { v =>

[mengxr]

LGTM. Merged into master. Thanks! @dbtsai Please address the comment in a separate PR.

[SparkQA]

Test build #31321 has finished for PR 5767 at commit fc9f582.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #31329 has finished for PR 5767 at commit 5e346c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.