[SPARK-25623][SPARK-25624][SPARK-25625][TEST] Reduce test time of LogisticRegressionSuite

[shahidki31]

...with intercept with L1 regularization

What changes were proposed in this pull request?

In the test, "multinomial logistic regression with intercept with L1 regularization" in the "LogisticRegressionSuite", taking more than a minute due to training of 2 logistic regression model.
However after analysing the training cost over iteration, we can reduce the computation time by 50%.
Training cost vs iteration for model 1

So, model1 is converging after iteration 150.

Training cost vs iteration for model 2

After around 100 iteration, model2 is converging.
So, if we give maximum iteration for model1 and model2 as 175 and 125 respectively, we can reduce the computation time by half.

How was this patch tested?

Computation time in local setup :
Before change:
~53 sec
After change:
~26 sec

Please review http://spark.apache.org/contributing.html before opening a pull request.

[HyukjinKwon]

ok to test

[SparkQA]

Test build #97087 has finished for PR 22659 at commit 2040ada.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shahidki31]

In the test, "multinomial logistic regression with intercept with elasticnet regularization" in the "LogisticRegressionSuite", taking around 1 minute to train 2 logistic regression model.
However after analyzing the training cost over iteration, we can reduce the computation time by 50%.
Training cost vs iteration for model 1

So, model1 is converging after iteration 200.

Training cost vs iteration for model 2:
image

After around 50 iteration, model2 is converging.
So, if we give maximum iteration for model1 and model2 as 220 and 90 respectively, we can reduce the computation time by half.

Computation time in local setup :
Before change:
~54 sec
After change:
~35 sec

[shahidki31]

In the test "binary logistic regression with intercept with ElasticNet regularization", taking around 30sec to run. But we can reduce the time to 15 sec by reducing the iteration.

model1 converges after 100 iteration,

model2 converges after 20 iterations.
So, if we make maxIter of model1 and model2 as 120 and 30 respectively, we can reduce the time to ~15 sec.

In the test "multinomial logistic regression without intercept with elasticnet regularization", taking around 30 sec to run. This also can be reduced to 15 sec by reducing number of iteration.

model1 converges after 50 iteration.

model2 converges after 30 iteration.
So, if we make maxIter of model1 and model2 as 75 and 50 respectively, we can reduce the computation time less than 15sec

[shahidki31]

Before the changes:
Running time of logistic regression suite: 4min 35 sec
After the changes:
Running time of logistic regression suite: 3min 22 sec

cc @srowen @HyukjinKwon . Kindly review

[SparkQA]

Test build #97093 has finished for PR 22659 at commit 3d9673e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #97094 has finished for PR 22659 at commit c28fd05.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shahidki31]

In Jenkins CI, testing time of logisticRegressionSuite without the PR is 5 min 10 sec and with the PR, 4 min 21 sec

[srowen]

Merged to master

[shahidki31]

Thank you @srowen for merging.