SPARK-9690 Adding the possibility to set the seed of the rand in the …

[mmenestret]

…CrossValidator fold

The fold in the ML CrossValidator depends on a rand whose seed is set to 0 and it leads the sql.functions rand to call sc._jvm.functions.rand() with no seed.
In order to be able to unit test a Cross Validation it would be a good idea to be able to set this seed so the output of the cross validation (with a featureSubsetStrategy set to "all") would always be the same.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

that seems reasonable -- @davies ?

[davies]

LGTM. cc @mengxr Should we merge this into 1.5?

[jkbradley]

Please update this line too.

[jkbradley]

Please make CrossValidator inherit from HasSeed.
Also, update setParams.
See RandomForestClassifier or other existing classes for examples.

[mengxr]

@mmenestret We should first add seed to Scala's CrossValidator and then add wrappers to Python. You can use https://github.com/apache/spark/blob/master/python/pyspark/ml/clustering.py#L37 as a reference implementation.

[mengxr]

This is also not necessary if we make CrossValidator extend HasSeed.

[jkbradley]

Ping!

[jkbradley]

Ping. Please let me know if you don't have time to push an update.

[mmenestret]

Im sorry im on holyday right now, i'll be back in a week, is it ok ?

Envoyé depuis mon HTC

----- Reply message -----
De : "jkbradley" [email protected]
Pour : "apache/spark" [email protected]
Cc : "mmenestret" [email protected]
Objet : [spark] SPARK-9690 Adding the possibility to set the seed of the rand in the … (#7997)
Date : mar., sept. 22, 2015 10:39

Ping. Please let me know if you don't have time to push an update.

—
Reply to this email directly or view it on GitHub.

[jkbradley]

Oh OK no problem. Enjoy your holiday!

[jkbradley]

Ping! Btw, the 1.6 code freeze is scheduled for the end of this month.

[jkbradley]

@mmenestret I'm going to create a new PR based on yours. You'll still be the primary author.

Please close this issue/PR, thanks.