[SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs #12938

holdenk · 2016-05-05T21:56:22Z

What changes were proposed in this pull request?

Mark ml.classification algorithms as experimental to match Scala algorithms, update PyDoc for for thresholds on LogisticRegression to have same level of info as Scala, and enable mathjax for PyDoc.

How was this patch tested?

Built docs locally & PySpark SQL tests

…hat way in scala and update the docs for logisitc regression threshold

…ce it doesn't throw an an error

SparkQA · 2016-05-05T22:26:24Z

Test build #57925 has finished for PR 12938 at commit c72fa46.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…tigate how to handle the weights param being none

…layers stay same but weights change

SparkQA · 2016-05-09T18:30:28Z

Test build #58153 has finished for PR 12938 at commit 64942b7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-10T18:19:04Z

Test build #58249 has finished for PR 12938 at commit a73913b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2016-05-10T19:26:07Z

mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala

-    " Allows setting the solver: minibatch gradient descent (gd) or l-bfgs. " +
-      " l-bfgs is the default one.",
+    "Allows setting the solver: minibatch gradient descent (gd) or l-bfgs. " +
+      "(Default l-bfgs)",


indentation is off here. Also prefer (Default: l-bfgs)

sure, kept original indentation but will update.

SparkQA · 2016-05-10T19:36:34Z

Test build #58255 has finished for PR 12938 at commit 9e38ddf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-10T21:18:56Z

Test build #58264 has finished for PR 12938 at commit f4df8f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-19T20:04:00Z

Test build #58895 has finished for PR 12938 at commit 5df5a93.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-19T21:15:38Z

Test build #58901 has finished for PR 12938 at commit 2eec947.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-23T22:59:12Z

Test build #59158 has finished for PR 12938 at commit e11dbf8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-05-26T07:10:05Z

ping?

MLnick · 2016-06-14T13:27:31Z

@holden does f7288e1 need to be reverted?

holden · 2016-06-14T17:51:20Z

I think you meant @holdenk

Otherwise the pr looks ok to me. ;-)

SparkQA · 2016-06-14T19:41:17Z

Test build #60499 has finished for PR 12938 at commit 4431daa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-06-14T22:53:42Z

@MLnick I think its your call - as @rxin pointed out on another PR - during a release we normally revert rather than waiting to fix - but since I made a fix in the process of figuring out what the issue was I think it would also just be OK to merge this - but your call.

MLnick · 2016-06-15T17:41:47Z

@holdenk master passes PySpark tests for me locally. Are we sure it is f7288e1? Is it intermittent? Because #13489 did pass Jenkins (including PySpark tests).

holdenk · 2016-06-17T08:05:37Z

@MLnick - I think it just a timing condition - if the file is available in the resource path its all good but if it isn't then its sad. So f7288e1 would have passed, but this PR probably changes how long the pylint check takes (adds mathjax) which surfaces this.

MLnick · 2016-06-17T12:53:07Z

@holdenk ok weird - now master is failing for me locally with the same error and I confirmed your fix does work.

Failure is consistent for me locally. But it doesn't appear this is impacting open/recent PySpark SQL PRs... e.g. #13717, #13558. This PR is the only place it's failing on Jenkins, which is really weird to me, since even an intermittent failure should show up on someone else's PR. So for now I have not reverted f7288e1 ... cc @rxin @sameeragarwal

In any case let's rather isolate the issue/fix in a separate PR for SPARK-15954.

holdenk · 2016-06-17T14:29:20Z

Sounds good will make that PR :)

On Friday, June 17, 2016, Nick Pentreath [email protected] wrote:

@holdenk https://github.com/holdenk ok weird - now master is failing
for me locally with the same error and I confirmed your fix does work.

Failure is consistent for me locally. But it doesn't appear this is
impacting open/recent PySpark SQL PRs... e.g. #13717
#13717, #13558
#13558. This PR is the only place
it's failing on Jenkins, which is really weird to me, since even an
intermittent failure should show up on someone else's PR. So for now I have
not reverted f7288e1
f7288e1
... cc @rxin https://github.com/rxin @sameeragarwal
https://github.com/sameeragarwal

In any case let's rather isolate the issue/fix in a separate PR for
SPARK-15954.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12938 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AADp9V03SpADeLij0leDvXuHD3lIlODuks5qMpjngaJpZM4IYbBk
.

Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau

MLnick · 2016-06-21T15:15:19Z

@holdenk can you back out 4431daa from this PR when you get a chance?

holdenk · 2016-06-21T15:46:59Z

Sure thing.

On Tuesday, June 21, 2016, Nick Pentreath [email protected] wrote:

@holdenk https://github.com/holdenk can you back out 4431daa
4431daa
from this PR when you get a chance?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12938 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AADp9dSasgfL9zhu21irWOwssmXi0SNsks5qOABBgaJpZM4IYbBk
.

Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau

This reverts commit 4431daa.

holdenk · 2016-06-21T17:33:39Z

So this reverts the support both, but leaves in the revert of the breaking change. Did you also want that removed @MLnick ?

mengxr · 2016-06-21T18:41:57Z

@holdenk @MLnick Could this PR be split into smaller ones? I don't see a good reason to put changes to HiveTest and pyspark.ml Experimental annotations under the same PR. Keeping PRs minimal is a good practice and it also helps locate issues and backport bug fixes.

MLnick · 2016-06-21T18:45:45Z

I did mean by my earlier comment that this PR should focus on the doc
changes and the other new (linked) one is handling the HiveTest issue.
On Tue, 21 Jun 2016 at 20:42, Xiangrui Meng [email protected]
wrote:

@holdenk https://github.com/holdenk @MLnick https://github.com/MLnick
Could this PR be split into smaller ones? I don't see a good reason to put
changes to HiveTest and pyspark.ml Experimental annotations under the
same PR. Keeping PRs minimal is a good practice and it also helps locate
issues and backport bug fixes.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12938 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AA_SB5KowL5vQrTp_YBBbovlqU27UnI2ks5qODCugaJpZM4IYbBk
.

holdenk · 2016-06-21T18:46:52Z

Good, I'll revert the revert then as well :)

…r reading resource files in HiveTests" as it was causing Jenkins failures." This reverts commit 2be8cdf.

holdenk · 2016-06-21T18:50:10Z

Stripped down to just the doc changes. I personally very much like having small PRs - the fix just happened to end up in here since it this doc change exposed the underlying issue (and it seemed like the doc change its self was small enough it would be ok to package the fix with it) as some people have asked that I group things together more.

SparkQA · 2016-06-21T19:20:02Z

Test build #60943 has finished for PR 12938 at commit d842309.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-21T20:04:38Z

Test build #60949 has finished for PR 12938 at commit de63f9f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2016-06-22T09:51:44Z

Ok it's passing now :) LGTM

## What changes were proposed in this pull request? Mark ml.classification algorithms as experimental to match Scala algorithms, update PyDoc for for thresholds on `LogisticRegression` to have same level of info as Scala, and enable mathjax for PyDoc. ## How was this patch tested? Built docs locally & PySpark SQL tests Author: Holden Karau <[email protected]> Closes #12938 from holdenk/SPARK-15162-SPARK-15164-update-some-pydocs. (cherry picked from commit d281b0b) Signed-off-by: Nick Pentreath <[email protected]>

MLnick · 2016-06-22T09:56:23Z

Merged to master/branch-2.0.

holdenk added 5 commits May 5, 2016 13:24

Mark a number of alogrithms and models experimental that are marked t…

8389280

…hat way in scala and update the docs for logisitc regression threshold

Add the rest

1fa57e5

Use mathjax for formula in PyDoc

b1ce817

Switch to math highlighting and update legostic regresion get doc sin…

8125c8c

…ce it doesn't throw an an error

Long line fix

c72fa46

holdenk added 3 commits May 5, 2016 16:40

Start adding the missing params to mutli-layer perceptron, also inves…

3fd1dce

…tigate how to handle the weights param being none

Or wait we just don't need to support None

c7caa43

Update the doc string for weights param and add doctest that verifys …

4776221

…layers stay same but weights change

holdenk changed the title ~~[SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs~~ [SPARK-15136][SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs May 6, 2016

holdenk changed the title ~~[SPARK-15136][SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs~~ [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs May 6, 2016

Merge in master

64942b7

holdenk added 3 commits May 10, 2016 10:38

mini fix

2397004

Merge branch 'master' into SPARK-15162-SPARK-15164-update-some-pydocs

130d05f

more pydoc fix

a73913b

holdenk added 2 commits May 10, 2016 11:42

Merge branch 'master' into SPARK-15162-SPARK-15164-update-some-pydocs

50b41ae

Remove flaky doctet component

9e38ddf

MLnick reviewed May 10, 2016
View reviewed changes

Add a : as requested

f4df8f0

Merge in master

5df5a93

Back out some unrelated changes that are in a seperate PR anyways

2eec947

Merge branch 'master' into SPARK-15162-SPARK-15164-update-some-pydocs

e11dbf8

Support both methods

4431daa

holdenk changed the title ~~[WIP][SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs~~ [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs Jun 14, 2016

holdenk changed the title ~~[SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs~~ [SPARK-15954][SPARK-15162][SPARK-15164][PySpark][DOCS][ML][SQL] update some pydocs and fix test issue Jun 14, 2016

holdenk mentioned this pull request Jun 19, 2016

[SPARK-15954][SQL][PySpark][TEST] Fix TestHiveContext interaction with PySpark issue #13737

Closed

Revert "Support both methods"

d842309

This reverts commit 4431daa.

Revert "Revert "[SPARK-15745][SQL] Use classloader's getResource() fo…

de63f9f

…r reading resource files in HiveTests" as it was causing Jenkins failures." This reverts commit 2be8cdf.

holdenk changed the title ~~[SPARK-15954][SPARK-15162][SPARK-15164][PySpark][DOCS][ML][SQL] update some pydocs and fix test issue~~ [SPARK-15162][SPARK-15164][PySpark][DOCS][ML][SQL] update some pydocs and fix test issue Jun 21, 2016

holdenk changed the title ~~[SPARK-15162][SPARK-15164][PySpark][DOCS][ML][SQL] update some pydocs and fix test issue~~ [SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs Jun 21, 2016

asfgit closed this in d281b0b Jun 22, 2016

[SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs #12938

[SPARK-15162][SPARK-15164][PySpark][DOCS][ML] update some pydocs #12938

Uh oh!

Conversation

holdenk commented May 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

SparkQA commented May 9, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

MLnick May 10, 2016

Choose a reason for hiding this comment

Uh oh!

holdenk May 10, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

SparkQA commented May 19, 2016

Uh oh!

SparkQA commented May 19, 2016

Uh oh!

SparkQA commented May 23, 2016

Uh oh!

holdenk commented May 26, 2016

Uh oh!

MLnick commented Jun 14, 2016

Uh oh!

holden commented Jun 14, 2016

Uh oh!

SparkQA commented Jun 14, 2016

Uh oh!

holdenk commented Jun 14, 2016

Uh oh!

MLnick commented Jun 15, 2016

Uh oh!

holdenk commented Jun 17, 2016

Uh oh!

MLnick commented Jun 17, 2016

Uh oh!

holdenk commented Jun 17, 2016

Uh oh!

MLnick commented Jun 21, 2016

Uh oh!

holdenk commented Jun 21, 2016

Uh oh!

holdenk commented Jun 21, 2016

Uh oh!

mengxr commented Jun 21, 2016

Uh oh!

MLnick commented Jun 21, 2016

Uh oh!

holdenk commented Jun 21, 2016

Uh oh!

holdenk commented Jun 21, 2016

Uh oh!

SparkQA commented Jun 21, 2016

Uh oh!

SparkQA commented Jun 21, 2016

Uh oh!

MLnick commented Jun 22, 2016

Uh oh!

MLnick commented Jun 22, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

holdenk commented May 5, 2016 •

edited

Loading