[MINOR][DOCS] Fix some links for python api doc #28017

yaooqinn · 2020-03-25T08:45:20Z

What changes were proposed in this pull request?

the link for partition discovery is malformed, because for releases, there will contains /docs/<version>/ in the full URL.

Why are the changes needed?

fix doc

Does this PR introduce any user-facing change?

no

How was this patch tested?

SKIP_SCALADOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 jekyll serve locally verified

yaooqinn · 2020-03-25T08:48:45Z

cc @HyukjinKwon

SparkQA · 2020-03-25T08:54:35Z

Test build #120327 has finished for PR 28017 at commit f6bd4ec.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-25T09:15:53Z

python/pyspark/sql/functions.py

    pattern letters of `datetime pattern`_. can be used.

-    .. _datetime pattern: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+    .. _datetime pattern: /docs/latest/sql-ref-datetime-pattern.html


@yaooqinn you can just use the http://... links.

Seems the relative links work in your local build but doesn't work when it's released because the documentation root is rebased.

thanks,fixed

SparkQA · 2020-03-25T09:48:01Z

Test build #120331 has finished for PR 28017 at commit 2fd8b3b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-25T10:30:51Z

Test build #120339 has finished for PR 28017 at commit b4d0fc7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit a959d6a.

SparkQA · 2020-03-25T12:00:07Z

Test build #120344 has finished for PR 28017 at commit 5346870.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-03-25T12:08:29Z

retest this please

SparkQA · 2020-03-25T12:33:02Z

Test build #120347 has finished for PR 28017 at commit 5346870.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-25T12:37:36Z

Test build #120348 has finished for PR 28017 at commit 5346870.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-25T12:44:08Z

retest this please

SparkQA · 2020-03-25T13:08:24Z

Test build #120350 has finished for PR 28017 at commit 5346870.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-25T13:09:31Z

Hm?

HyukjinKwon · 2020-03-25T13:21:24Z

Seems like different avro is being picked up during unidoc for some reasons:

[info] Main Java API documentation to /home/jenkins/workspace/SparkPullRequestBuilder@6/target/javaunidoc...
[error] /home/jenkins/workspace/SparkPullRequestBuilder@6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData
[error]     writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema))
[error]                                                         ^
[info] No documentation generated with unsuccessful compiler run
[error] one error found

I faced a long ago. I had to just skip it (#17669) when it's built against hadoop-2.6 profile because explicitly setting dependencies didn't work (#17642, #17651)

HyukjinKwon · 2020-03-25T13:29:00Z

Can you try the below?

diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 4e1badb5e90..60319d81e16 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -625,7 +625,8 @@ object DependencyOverrides {
     dependencyOverrides += "com.google.guava" % "guava" % "14.0.1",
     dependencyOverrides += "commons-io" % "commons-io" % "2.4",
     dependencyOverrides += "xerces" % "xercesImpl" % "2.12.0",
-    dependencyOverrides += "jline" % "jline" % "2.14.6")
+    dependencyOverrides += "jline" % "jline" % "2.14.6",
+    dependencyOverrides += "org.apache.avro" % "avro" % "1.8.2")
 }

 /**

HyukjinKwon · 2020-03-25T14:14:40Z

@yaooqinn, can you make a separate PR to pin the avro version specifically in SBT? seems it fixes the issue. FYI @HeartSaVioR and @cloud-fan from the mailing list.

HyukjinKwon · 2020-03-25T14:14:50Z

Oops, mistake.

SparkQA · 2020-03-25T16:27:30Z

Test build #120359 has finished for PR 28017 at commit aaa1594.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit aaa1594.

### What changes were proposed in this pull request? add arvo dep in SparkBuild ### Why are the changes needed? fix sbt unidoc like #28017 (comment) ```scala [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [info] Main Scala API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/scala-2.12/unidoc... [info] Main Java API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/javaunidoc... [error] /home/jenkins/workspace/SparkPullRequestBuilder6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData [error] writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema)) [error] ^ [info] No documentation generated with unsuccessful compiler run [error] one error found ``` ### Does this PR introduce any user-facing change? no ### How was this patch tested? pass jenkins and verify manually with `sbt dependencyTree` ```scala kentyaohulk  ~/spark   dep  build/sbt dependencyTree | grep avro | grep -v Resolving [info] +-org.apache.avro:avro-mapred:1.8.2 [info] | +-org.apache.avro:avro-ipc:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] +-org.apache.avro:avro:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] org.apache.spark:spark-avro_2.12:3.1.0-SNAPSHOT [S] [info] | | | +-org.apache.avro:avro-mapred:1.8.2 [info] | | | | +-org.apache.avro:avro-ipc:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 [info] | | | +-org.apache.avro:avro:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 ``` Closes #28020 from yaooqinn/dep. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

### What changes were proposed in this pull request? add arvo dep in SparkBuild ### Why are the changes needed? fix sbt unidoc like #28017 (comment) ```scala [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [info] Main Scala API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/scala-2.12/unidoc... [info] Main Java API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/javaunidoc... [error] /home/jenkins/workspace/SparkPullRequestBuilder6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData [error] writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema)) [error] ^ [info] No documentation generated with unsuccessful compiler run [error] one error found ``` ### Does this PR introduce any user-facing change? no ### How was this patch tested? pass jenkins and verify manually with `sbt dependencyTree` ```scala kentyaohulk  ~/spark   dep  build/sbt dependencyTree | grep avro | grep -v Resolving [info] +-org.apache.avro:avro-mapred:1.8.2 [info] | +-org.apache.avro:avro-ipc:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] +-org.apache.avro:avro:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] org.apache.spark:spark-avro_2.12:3.1.0-SNAPSHOT [S] [info] | | | +-org.apache.avro:avro-mapred:1.8.2 [info] | | | | +-org.apache.avro:avro-ipc:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 [info] | | | +-org.apache.avro:avro:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 ``` Closes #28020 from yaooqinn/dep. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 336621e) Signed-off-by: HyukjinKwon <[email protected]>

add arvo dep in SparkBuild fix sbt unidoc like #28017 (comment) ```scala [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [info] Main Scala API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/scala-2.12/unidoc... [info] Main Java API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/javaunidoc... [error] /home/jenkins/workspace/SparkPullRequestBuilder6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData [error] writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema)) [error] ^ [info] No documentation generated with unsuccessful compiler run [error] one error found ``` no pass jenkins and verify manually with `sbt dependencyTree` ```scala kentyaohulk  ~/spark   dep  build/sbt dependencyTree | grep avro | grep -v Resolving [info] +-org.apache.avro:avro-mapred:1.8.2 [info] | +-org.apache.avro:avro-ipc:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] +-org.apache.avro:avro:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] org.apache.spark:spark-avro_2.12:3.1.0-SNAPSHOT [S] [info] | | | +-org.apache.avro:avro-mapred:1.8.2 [info] | | | | +-org.apache.avro:avro-ipc:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 [info] | | | +-org.apache.avro:avro:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 ``` Closes #28020 from yaooqinn/dep. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 336621e) Signed-off-by: HyukjinKwon <[email protected]>

yaooqinn · 2020-03-26T01:52:14Z

retest this please

SparkQA · 2020-03-26T02:24:11Z

Test build #120384 has finished for PR 28017 at commit a1fbefe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-26T02:27:58Z

Test build #120385 has finished for PR 28017 at commit a1fbefe.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-26T02:43:12Z

retest this please

SparkQA · 2020-03-26T03:41:17Z

Test build #120388 has finished for PR 28017 at commit a1fbefe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-26T04:06:32Z

Merged to master and branch-3.0.

### What changes were proposed in this pull request? the link for `partition discovery` is malformed, because for releases, there will contains` /docs/<version>/` in the full URL. ### Why are the changes needed? fix doc ### Does this PR introduce any user-facing change? no ### How was this patch tested? `SKIP_SCALADOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 jekyll serve` locally verified Closes #28017 from yaooqinn/doc. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit b024a8a) Signed-off-by: HyukjinKwon <[email protected]>

### What changes were proposed in this pull request? add arvo dep in SparkBuild ### Why are the changes needed? fix sbt unidoc like apache#28017 (comment) ```scala [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [info] Main Scala API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/scala-2.12/unidoc... [info] Main Java API documentation to /home/jenkins/workspace/SparkPullRequestBuilder6/target/javaunidoc... [error] /home/jenkins/workspace/SparkPullRequestBuilder6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData [error] writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema)) [error] ^ [info] No documentation generated with unsuccessful compiler run [error] one error found ``` ### Does this PR introduce any user-facing change? no ### How was this patch tested? pass jenkins and verify manually with `sbt dependencyTree` ```scala kentyaohulk  ~/spark   dep  build/sbt dependencyTree | grep avro | grep -v Resolving [info] +-org.apache.avro:avro-mapred:1.8.2 [info] | +-org.apache.avro:avro-ipc:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] +-org.apache.avro:avro:1.8.2 [info] | | +-org.apache.avro:avro:1.8.2 [info] org.apache.spark:spark-avro_2.12:3.1.0-SNAPSHOT [S] [info] | | | +-org.apache.avro:avro-mapred:1.8.2 [info] | | | | +-org.apache.avro:avro-ipc:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 [info] | | | +-org.apache.avro:avro:1.8.2 [info] | | | | | +-org.apache.avro:avro:1.8.2 ``` Closes apache#28020 from yaooqinn/dep. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

### What changes were proposed in this pull request? the link for `partition discovery` is malformed, because for releases, there will contains` /docs/<version>/` in the full URL. ### Why are the changes needed? fix doc ### Does this PR introduce any user-facing change? no ### How was this patch tested? `SKIP_SCALADOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 jekyll serve` locally verified Closes apache#28017 from yaooqinn/doc. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

[MINOR][DOCS] Fix some links for python api doc

f6bd4ec

style fix

2fd8b3b

HyukjinKwon reviewed Mar 25, 2020

View reviewed changes

address cmts

b4d0fc7

yaooqinn added 3 commits March 25, 2020 19:31

fix style

a959d6a

Revert "fix style"

6d45125

This reverts commit a959d6a.

fix link

5346870

dep

aaa1594

HyukjinKwon closed this Mar 25, 2020

HyukjinKwon reopened this Mar 25, 2020

yaooqinn mentioned this pull request Mar 25, 2020

[SPARK-31258][BUILD] Pin the avro version in SBT #28020

Closed

Revert "dep"

a1fbefe

This reverts commit aaa1594.

HyukjinKwon approved these changes Mar 26, 2020

View reviewed changes

HyukjinKwon closed this in b024a8a Mar 26, 2020

[MINOR][DOCS] Fix some links for python api doc #28017

[MINOR][DOCS] Fix some links for python api doc #28017

Uh oh!

Conversation

yaooqinn commented Mar 25, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yaooqinn commented Mar 25, 2020

Uh oh!

SparkQA commented Mar 25, 2020

Uh oh!

HyukjinKwon Mar 25, 2020

Choose a reason for hiding this comment

Uh oh!

yaooqinn Mar 25, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 25, 2020

Uh oh!

SparkQA commented Mar 25, 2020

Uh oh!

SparkQA commented Mar 25, 2020

Uh oh!

yaooqinn commented Mar 25, 2020

Uh oh!

SparkQA commented Mar 25, 2020

Uh oh!

SparkQA commented Mar 25, 2020

Uh oh!

HyukjinKwon commented Mar 25, 2020

Uh oh!

SparkQA commented Mar 25, 2020

Uh oh!

HyukjinKwon commented Mar 25, 2020

Uh oh!

HyukjinKwon commented Mar 25, 2020

Uh oh!

HyukjinKwon commented Mar 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Mar 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Mar 25, 2020

Uh oh!

SparkQA commented Mar 25, 2020

Uh oh!

yaooqinn commented Mar 26, 2020

Uh oh!

SparkQA commented Mar 26, 2020

Uh oh!

SparkQA commented Mar 26, 2020

Uh oh!

HyukjinKwon commented Mar 26, 2020

Uh oh!

SparkQA commented Mar 26, 2020

Uh oh!

HyukjinKwon commented Mar 26, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HyukjinKwon commented Mar 25, 2020 •

edited

Loading

HyukjinKwon commented Mar 25, 2020 •

edited

Loading