[SPARK-11381][DOCS] Replace example code in mllib-linear-methods.md using include_example #11320

dongjoon-hyun · 2016-02-23T11:26:47Z

What changes were proposed in this pull request?

This PR replaces example codes in mllib-linear-methods.md using include_example
by doing the followings:

Extracts the example codes(Scala,Java,Python) as files in example module.
Merges some dialog-style examples into a single file.
Hide redundant codes in HTML for the consistency with other docs.

How was the this patch tested?

manual test.
This PR can be tested by document generations, SKIP_API=1 jekyll build.

noprom · 2016-02-23T11:44:22Z

So many changes in this commit.

dongjoon-hyun · 2016-02-23T18:07:21Z

Thank you for reviewing, @noprom . This PR is similar to #11053 (merged yesterday.)

Hi, @yinxusen, @mengxr .
Could you review this PR?

mengxr · 2016-02-23T20:52:52Z

ok to test

mengxr · 2016-02-23T20:52:57Z

cc @yinxusen

dongjoon-hyun · 2016-02-23T21:38:38Z

Oh, thanks. I'll check the Jenkins in detail.

dongjoon-hyun · 2016-02-23T21:42:07Z

Actually, the error was Github timeout. I pushed force again. Could you ask Jenkins to retest please?

+refs/pull/11320/*:refs/remotes/origin/pr/11320/* # timeout=15
ERROR: Timeout after 15 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git

dongjoon-hyun · 2016-02-23T21:46:34Z

To be sure, I rebased this PR to the master, too.

SparkQA · 2016-02-23T21:55:15Z

Test build #51805 has finished for PR 11320 at commit 3e56717.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaL1UpdaterExample
- public class JavaLinearRegressionWithSGDExample
- public class JavaLogisticRegressionWithLBFGSExample
- public class JavaSVMWithSGDExample

dongjoon-hyun · 2016-02-23T22:12:22Z

Let me fix the Scala style. I thought I passed since build/mvn -DskipTests checkstyle:check ends with 'SUCCESS'. Now, I see that there were printed '[ERROR]' lines. Sorry for missing that.

yinxusen · 2016-02-23T22:15:20Z

@dongjoon-hyun You can use dev/scalastyle, dev/lint-python, etc. to simply the style check.

dongjoon-hyun · 2016-02-23T22:16:48Z

Thank you so much, @yinxusen ! I'll fix soon and let you know here.

dongjoon-hyun · 2016-02-23T22:57:10Z

I've done!

dongjoon-hyun · 2016-02-23T22:59:07Z

Test build 51808 is running now. Let's see the result. :)

dongjoon-hyun · 2016-02-23T23:14:33Z

Hmm, it fails again due to Github.

ERROR: Timeout after 15 minutes
ERROR: Error fetching remote repo 'origin'

dongjoon-hyun · 2016-02-23T23:17:10Z

According to Jenkins, other PRs also suffer from this. I think retriggering is not helpful at this time.

yinxusen · 2016-02-23T23:18:06Z

We can wait for a while. No worry.

…sing include_example This PR replaces example codes in mllib-linear-methods.md using `include_example` by doing the followings: * Extracts the example codes(Scala,Java,Python) as files in `example` module. * Merges some dialog-style examples into a single file. * Hide redundant codes in HTML for the consistency with other docs. * Move the output directory into 'target/tmp'.

SparkQA · 2016-02-24T01:24:05Z

Test build #51825 has finished for PR 11320 at commit 12e40a0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaL1UpdaterExample
- public class JavaLinearRegressionWithSGDExample
- public class JavaLogisticRegressionWithLBFGSExample
- public class JavaSVMWithSGDExample

dongjoon-hyun · 2016-02-24T01:35:20Z

Finally! Now, it's ready to be reviewed again. :)
Thank you, all.

dongjoon-hyun · 2016-02-24T23:11:20Z

Hi, @yinxusen and @mengxr .
Could you review this if you squeeze some time out of your schedule?

yinxusen · 2016-02-25T08:54:43Z

I'll do it tomorrow. Thx

On Wed, Feb 24, 2016 at 3:12 PM, Dongjoon Hyun [email protected]
wrote:

Hi, @yinxusen https://github.com/yinxusen and @mengxr
https://github.com/mengxr .
Could you review this if you squeeze some time out of your schedule?

—
Reply to this email directly or view it on GitHub
#11320 (comment).

Cheers

Xusen Yin (尹绪森)
LinkedIn: https://cn.linkedin.com/in/xusenyin

dongjoon-hyun · 2016-02-25T17:52:20Z

Thank you, @yinxusen !

yinxusen · 2016-02-26T08:06:51Z

examples/src/main/java/org/apache/spark/examples/mllib/JavaL1UpdaterExample.java

+    // $example on$
+    SVMWithSGD svmAlg = new SVMWithSGD();
+    svmAlg.optimizer()
+            .setNumIterations(200)


@dongjoon-hyun All Java files should follow 2-indent style.

@yinxusen . My bad.
I updated the PR to fix it.

yinxusen · 2016-02-26T08:25:07Z

@dongjoon-hyun

Let's remove those examples:

L1UpdaterExample.scala
JavaL1UpdaterExample.java

For these small pieces of codes, we can leave it untouched for now. Otherwise, there will be too many example codes.

dongjoon-hyun · 2016-02-26T08:28:48Z

Okay. No problem! I will update soon.

yinxusen · 2016-02-26T08:29:02Z

@dongjoon-hyun No need to rush, you can wait until I finish the round of reviewing.

dongjoon-hyun · 2016-02-26T08:31:23Z

Oh, I see. Then, let me know when you finish. :)

yinxusen · 2016-02-26T08:34:13Z

examples/src/main/java/org/apache/spark/examples/mllib/JavaLinearRegressionWithSGDExample.java

+
+package org.apache.spark.examples.mllib;
+
+// $example on$


Update the import block in this way:

import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaSparkContext; // $example on$ import scala.Tuple2; import org.apache.spark.api.java.JavaDoubleRDD; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.function.Function; import org.apache.spark.mllib.linalg.Vectors; import org.apache.spark.mllib.regression.LabeledPoint; import org.apache.spark.mllib.regression.LinearRegressionModel; import org.apache.spark.mllib.regression.LinearRegressionWithSGD; // $example off$

yinxusen · 2016-02-26T08:46:16Z

examples/src/main/python/mllib/svm_with_sgd_example.py

+from pyspark.mllib.classification import SVMWithSGD, SVMModel
+from pyspark.mllib.regression import LabeledPoint
+# $example off$
+from pyspark import SparkContext


This should be above of # $example on$

But other python codes keep from pyspark import SparkContext outside, doesn't it?

Ah.. I see.

SparkQA · 2016-02-26T08:50:35Z

Test build #52040 has finished for PR 11320 at commit 239e09e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yinxusen · 2016-02-26T08:54:18Z

...ples/src/main/scala/org/apache/spark/examples/mllib/LogisticRegressionWithLBFGSExample.scala

+// $example on$
+import org.apache.spark.mllib.classification.{LogisticRegressionModel, LogisticRegressionWithLBFGS}
+import org.apache.spark.mllib.evaluation.MulticlassMetrics
+import org.apache.spark.mllib.linalg.Vectors


yinxusen · 2016-02-26T09:14:54Z

@dongjoon-hyun For the section of streaming linear regression: http://spark.apache.org/docs/latest/mllib-linear-methods.html#streaming-linear-regression, we should change the following sentence:

Anytime a text file is placed in /training/data/dir the model will update. Anytime a text file is placed in /testing/data/ dir you will see predictions. As you feed more data to the training directory, the predictions will get better!

Substituting the bold words with args(0) and args(1) in Scala; sys.argv[1] and sys.argv[2] for Python.

yinxusen · 2016-02-26T09:16:51Z

examples/src/main/python/mllib/streaming_linear_regression_example.py

+"""
+from __future__ import print_function
+
+import sys


change the line to:

# $example on$ import sys # $example off$ Since the truncated code uses the sys.argv[1].

yinxusen · 2016-02-26T09:21:34Z

@dongjoon-hyun One more thing, try to make the output dirs different for different examples. Like there are 3 examples (Scala/Java/Python versions) use target/tmp/myLinearRegressionWithSGDModel. We cannot run those examples once a time if we don't delete the dir generated before.

yinxusen · 2016-02-26T09:22:19Z

@dongjoon-hyun I finished the review. Thanks for working on this!

SparkQA · 2016-02-26T09:22:52Z

Test build #52037 has finished for PR 11320 at commit 92a3aff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-02-26T09:22:55Z

Thank you, @yinxusen !

dongjoon-hyun · 2016-02-26T09:49:03Z

I pushed the commit.
I've learn many things from you. Thank you very much, @yinxusen .

SparkQA · 2016-02-26T10:04:03Z

Test build #52043 has finished for PR 11320 at commit 9000405.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yinxusen · 2016-02-26T10:31:13Z

@mengxr LGTM

Note that we consolidate the original streaming linear regression example from small pieces into a complete example and modify some sentences.

mengxr · 2016-02-26T16:32:09Z

Merged into master. Thanks!

…sing include_example ## What changes were proposed in this pull request? This PR replaces example codes in `mllib-linear-methods.md` using `include_example` by doing the followings: * Extracts the example codes(Scala,Java,Python) as files in `example` module. * Merges some dialog-style examples into a single file. * Hide redundant codes in HTML for the consistency with other docs. ## How was the this patch tested? manual test. This PR can be tested by document generations, `SKIP_API=1 jekyll build`. Author: Dongjoon Hyun <[email protected]> Closes apache#11320 from dongjoon-hyun/SPARK-11381.

dongjoon-hyun · 2016-02-26T16:40:46Z

Thank you so much, @mengxr and @yinxusen !

dongjoon-hyun · 2016-02-26T17:55:46Z

Hi, @mengxr .
Could you close this PR?
It is merged to the master successfully, but is not closed until now.

dongjoon-hyun · 2016-03-01T09:31:58Z

Hi, @mengxr .
Could you close this PR? :)

srowen · 2016-03-01T11:18:29Z

@dongjoon-hyun normally the Apache ASF bot closes the PRs automatically, but I don't know why this one was not closed. We actually can't close PRs. You would have to close it manually.

dongjoon-hyun · 2016-03-01T19:52:41Z

Thank you for closing this, @srowen and @mengxr .

yinxusen reviewed Feb 26, 2016
View reviewed changes

Fix indentation.

239e09e

yinxusen reviewed Feb 26, 2016
View reviewed changes

Fix according to the comments.

9000405

asfgit closed this in c37bbb3 Mar 1, 2016

dongjoon-hyun deleted the SPARK-11381 branch March 3, 2016 20:40

[SPARK-11381][DOCS] Replace example code in mllib-linear-methods.md using include_example #11320

[SPARK-11381][DOCS] Replace example code in mllib-linear-methods.md using include_example #11320

Uh oh!

Conversation

dongjoon-hyun commented Feb 23, 2016

What changes were proposed in this pull request?

How was the this patch tested?

Uh oh!

noprom commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

mengxr commented Feb 23, 2016

Uh oh!

mengxr commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

SparkQA commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

yinxusen commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

dongjoon-hyun commented Feb 23, 2016

Uh oh!

yinxusen commented Feb 23, 2016

Uh oh!

SparkQA commented Feb 24, 2016

Uh oh!

dongjoon-hyun commented Feb 24, 2016

Uh oh!

dongjoon-hyun commented Feb 24, 2016

Uh oh!

yinxusen commented Feb 25, 2016

Cheers

Uh oh!

dongjoon-hyun commented Feb 25, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yinxusen commented Feb 26, 2016

Uh oh!

dongjoon-hyun commented Feb 26, 2016

Uh oh!

yinxusen commented Feb 26, 2016

Uh oh!

dongjoon-hyun commented Feb 26, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 26, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yinxusen commented Feb 26, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yinxusen commented Feb 26, 2016