[SPARK-11207][ML] Add test cases for solver selection of LinearRegres…

[Lewuathe]

…sion as followup. This is the follow up work of SPARK-10668.

• Fix miner style issues.
• Add test case for checking whether solver is selected properly.

[SparkQA]

Test build #43981 has finished for PR 9180 at commit 11cd9c1.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43983 has finished for PR 9180 at commit 28427d2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

WithBigFeature -> WithManyFeatures or WithLargeFeatureSize

[mengxr]

@Lewuathe The added test took 30 seconds to run, which might be too long. Shall we try to reduce the number of iterations?

[info] - linear regression model with l-bfgs with big feature datasets (29 seconds, 82 milliseconds)

[dbtsai]

Or you can make them sparse by randomly choosing most of the features zeros.

[mengxr]

+1 on @dbtsai 's suggestion

[SparkQA]

Test build #44067 has finished for PR 9180 at commit 22ba64e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Lewuathe]

- linear regression model with l-bfgs with big feature datasets (14 seconds, 524 milliseconds)

It takes about the half of the initial one. Could review this again? > @dbtsai @mengxr

[dbtsai]

extra line.

[SparkQA]

Test build #44116 has finished for PR 9180 at commit f6b2256.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Lewuathe]

@dbtsai Could you check again please?

[dbtsai]

How about consolidate with LinearDataGenerator, and add sparsity = 1.0 as param to control if it's sparse feature?

[Lewuathe]

Yes, I also thought it is good idea. But LinearDataGenerator is used as static object, then we have to pass sparsity as parameter to generateLinearInput. This method seems to be used a lot of suites. It is necessary to change a lot of method reference.
Therefore it might be better to do in separate JIRA. What do you thing about?

[dbtsai]

Let's modify the JIRA and do it here. Basically, you can create a LinearDataGenerator with old signature calling new API for compatibility issue.

[SparkQA]

Test build #44279 has finished for PR 9180 at commit 2082d47.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44308 has finished for PR 9180 at commit 003d3bd.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44313 has finished for PR 9180 at commit 0a43033.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44317 has finished for PR 9180 at commit 59383fd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Lewuathe]

@dbtsai Sorry for bothering many times but could check again please?

[dbtsai]

The formatting in pervious method

  def generateLinearInput(
      intercept: Double,
      weights: Array[Double],
      nPoints: Int,
      seed: Int,
      eps: Double = 0.1): Seq[LabeledPoint] = {
    generateLinearInput(intercept, weights,
      Array.fill[Double](weights.length)(0.0),
      Array.fill[Double](weights.length)(1.0 / 3.0),
      nPoints, seed, eps)}

looks weird for me. Can you fix in this PR? Thanks.

[dbtsai]

Sorry for the delay. The current implementation of creating sparse features is not efficient since we need to create dense feature first. Let's do it as it. But if you are interested in, let's create another JIRA such that the sparse features can be generated without doing dense one. Thanks.

[Lewuathe]

@dbtsai Thank you so much for reviewing even you would be busy in Spark Summit. I'll update.

[SparkQA]

Test build #44667 has finished for PR 9180 at commit 97c76c9.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44669 has finished for PR 9180 at commit 74de81e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

make seed = seed into just seed

[dbtsai]

LGTM except the small styling issues.

[SparkQA]

Test build #44673 has finished for PR 9180 at commit 241ec72.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

Thanks. Merged into master.