[SPARK-20762][ML]Make String Params Case-Insensitive #17995

zhengruifeng · 2017-05-16T06:09:05Z

What changes were proposed in this pull request?

Make String Params Case-Insensitive:
solver, modelType, initMode, metricName, handleInvalid, strategy, stringOrderType, coldStartStrategy, impurity, lossType, featureSubsetStrategy, intermediateStorageLevel, finalStorageLevel

Leave alone ChiSqSelector & StringIndexer, for they are not easy to handle like others.

How was this patch tested?

existing tests and added tests

SparkQA · 2017-05-16T16:01:03Z

Test build #76962 has finished for PR 17995 at commit abac904.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-17T03:38:49Z

Test build #76999 has finished for PR 17995 at commit eecc1b0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-17T04:48:38Z

Test build #77004 has finished for PR 17995 at commit 97b8df6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-17T04:57:35Z

Test build #77009 has started for PR 17995 at commit bed4c41.

SparkQA · 2017-05-17T05:03:56Z

Test build #77005 has finished for PR 17995 at commit 6b1fed5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zhengruifeng · 2017-05-18T03:11:06Z

Jenkins, retest this please

SparkQA · 2017-05-18T04:11:33Z

Test build #77038 has finished for PR 17995 at commit bed4c41.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zhengruifeng · 2017-05-18T05:18:30Z

ping @yanboliang

SparkQA · 2017-05-31T03:38:49Z

Test build #77568 has finished for PR 17995 at commit 0db3a52.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yanboliang · 2017-06-25T02:03:15Z

mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala

Could we organize as

val AreaUnderROC: String = "areaUnderROC".toLowerCase val AreaUnderPR: String = "areaUnderPR".toLowerCase val supportedMetricNames = Set(AreaUnderROC, AreaUnderPR)

in object BinaryClassificationEvaluator? This should be more clear.

yanboliang · 2017-06-26T02:34:00Z

mllib/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala

yanboliang · 2017-06-26T02:34:38Z

mllib/src/main/scala/org/apache/spark/ml/evaluation/RegressionEvaluator.scala

yanboliang · 2017-06-26T02:36:35Z

mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala

Supported selector types should always be stored as lower case, please update corresponding code snippet in mllib.feature.ChiSqSelector from:

private[spark] val NumTopFeatures: String = "numTopFeatures" ......

to

private[spark] val NumTopFeatures: String = "numTopFeatures".toLowerCase ......

yanboliang · 2017-06-26T02:46:57Z

mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala

Is this really necessary?

yanboliang · 2017-06-26T02:48:03Z

mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala

yanboliang · 2017-06-26T02:55:55Z

mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala

Actually we can't do this, since MLlib supports set params via other entrances. Currently we can leave as it is, until we resolved #16028.

yanboliang · 2017-06-26T03:00:23Z

mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala

What do you think of adding a new function in object ParamValidators as

def inStringArray(allowed: Array[String]): String => Boolean = { (value: String) => allowed.contains(value.toLowerCase(java.util.Locale.ROOT)) }

to facilitate similar check here and other place. cc @hhbyyh @sethah

I think it a good idea.

zhengruifeng · 2017-06-26T05:58:53Z

@yanboliang I update this PR and revert changes on setSolver in GLR and LiR. Thanks for your reviewing.

SparkQA · 2017-06-26T06:26:43Z

Test build #78607 has finished for PR 17995 at commit 28941f3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-26T08:14:25Z

Test build #78619 has finished for PR 17995 at commit 70e479b.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-26T09:12:31Z

Test build #78623 has finished for PR 17995 at commit a4fa44e.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-26T09:26:30Z

Test build #78621 has finished for PR 17995 at commit c377208.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? 1, make param support non-final with `finalFields` option 2, generate `HasSolver` with `finalFields = false` 3, override `solver` in LiR, GLR, and make MLPC inherit `HasSolver` ## How was this patch tested? existing tests Author: Ruifeng Zheng <[email protected]> Author: Zheng RuiFeng <[email protected]> Closes #16028 from zhengruifeng/param_non_final.

SparkQA · 2017-07-03T04:01:44Z

Test build #79060 has finished for PR 17995 at commit c58614f.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-03T04:14:10Z

Test build #79066 has finished for PR 17995 at commit 1715131.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-03T04:59:03Z

Test build #79065 has finished for PR 17995 at commit 6557b37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-03T05:15:00Z

Test build #79068 has finished for PR 17995 at commit 1997cd1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yanboliang · 2017-07-05T06:41:22Z

mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala

${getModelType} -> $getModelType

yanboliang · 2017-07-05T10:14:03Z

mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala

It's better to keep the original style, since we may add new metric which is not comply with isLargerBetter.

yanboliang · 2017-07-05T10:16:03Z

mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala

Can we make it private[BinaryClassificationEvaluator]?

yanboliang · 2017-07-05T10:19:31Z

mllib/src/main/scala/org/apache/spark/ml/param/params.scala

Check for string value in an allowed set of string values in a case-insensitive way.

yanboliang · 2017-07-05T10:19:58Z

mllib/src/main/scala/org/apache/spark/ml/param/params.scala

I think this function is useless and involves extra computing cost.

yanboliang · 2017-07-05T13:30:18Z

mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala

Please keep the original output format for supported family names.

yanboliang · 2017-07-05T13:50:06Z

mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala

Please revert this change.

yanboliang · 2017-07-05T13:53:07Z

mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala

NORMAL -> Normal

SparkQA · 2017-07-06T03:49:11Z

Test build #79258 has finished for PR 17995 at commit 4a6682f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-06T04:24:34Z

Test build #79260 has finished for PR 17995 at commit 8256530.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-06T06:01:10Z

Test build #79262 has finished for PR 17995 at commit 6d89c00.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zhengruifeng force-pushed the str_get_set branch from abac904 to eecc1b0 Compare May 17, 2017 03:04

zhengruifeng force-pushed the str_get_set branch from bed4c41 to 0db3a52 Compare May 31, 2017 02:35

yanboliang reviewed Jun 26, 2017

View reviewed changes

zhengruifeng force-pushed the str_get_set branch from 0db3a52 to 28941f3 Compare June 26, 2017 05:54

yanboliang mentioned this pull request Jul 1, 2017

[SPARK-18518][ML] HasSolver supports override #16028

Closed

zhengruifeng force-pushed the str_get_set branch from c58614f to 6557b37 Compare July 3, 2017 03:52

yanboliang reviewed Jul 5, 2017

View reviewed changes

zhengruifeng added 3 commits July 6, 2017 10:16

create pr

a689eb3

fix glr

1f7a908

update tests

c91c449

zhengruifeng added 16 commits July 6, 2017 10:16

create pr x

2146d88

fix some nits

79da304

fix some nits

1afc31f

fix bugs

a3d5d4d

fix chisq

bfa9a1a

fix stringindexer

37428fd

fix chisq fdr

dd67b67

many fixes

e13e6fa

add functions

f6b899a

fix one bug

577dc7b

two nits

00727eb

fix conflicts

4541439

fix nits

d8359e2

fix nit

1c67542

fix wrong private

bf75d99

add private scope && revert chi && update other nits

4a6682f

zhengruifeng force-pushed the str_get_set branch from 1997cd1 to 4a6682f Compare July 6, 2017 03:39

update scope in LoR

8256530

revert StringIndexer

6d89c00

zhengruifeng closed this Aug 10, 2017

zhengruifeng deleted the str_get_set branch August 10, 2017 05:33

[SPARK-20762][ML]Make String Params Case-Insensitive #17995

[SPARK-20762][ML]Make String Params Case-Insensitive #17995

Uh oh!

Conversation

zhengruifeng commented May 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 16, 2017

Uh oh!

SparkQA commented May 17, 2017

Uh oh!

SparkQA commented May 17, 2017

Uh oh!

SparkQA commented May 17, 2017

Uh oh!

SparkQA commented May 17, 2017

Uh oh!

zhengruifeng commented May 18, 2017

Uh oh!

SparkQA commented May 18, 2017

Uh oh!

zhengruifeng commented May 18, 2017

Uh oh!

SparkQA commented May 31, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yanboliang Jun 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengruifeng commented Jun 26, 2017

Uh oh!

SparkQA commented Jun 26, 2017

Uh oh!

SparkQA commented Jun 26, 2017

Uh oh!

SparkQA commented Jun 26, 2017

Uh oh!

SparkQA commented Jun 26, 2017

Uh oh!

SparkQA commented Jul 3, 2017

Uh oh!

SparkQA commented Jul 3, 2017

Uh oh!

SparkQA commented Jul 3, 2017

Uh oh!

SparkQA commented Jul 3, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

zhengruifeng commented May 16, 2017 •

edited

Loading

yanboliang Jun 26, 2017 •

edited

Loading