[SPARK-32150][BUILD] Upgrade to ZStd 1.4.5-4 #28969

williamhyun · 2020-07-01T16:24:11Z

What changes were proposed in this pull request?

This PR aims to upgrade to ZStd 1.4.5-4.

Why are the changes needed?

ZStd 1.4.5-4 fixes the following.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins.

SparkQA · 2020-07-01T20:43:03Z

Test build #124805 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-02T00:57:30Z

Looks okay to me. cc: @dongjoon-hyun @srowen

srowen · 2020-07-02T01:34:26Z

Looks OK to backport to 3.0.x right?

maropu · 2020-07-02T01:49:38Z

branch-3.0 still uses zstd v1.4.4 because we didn't merge the last zstd upgrade commit (#28682) into branch-3.0.

// zstd version in branch-3.0

spark/pom.xml

Line 668 in bdbfe6e

If there is no compatiblity issue and the fixes are somewhat critical, I think merging this into branch-3.0 looks fine.

SparkQA · 2020-07-02T01:59:08Z

Test build #124821 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-02T02:02:38Z

retest this please

SparkQA · 2020-07-02T02:07:15Z

Test build #124843 has finished for PR 28969 at commit 7abf2b5.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-02T02:13:02Z

retest this please

SparkQA · 2020-07-02T02:15:11Z

Test build #124844 has finished for PR 28969 at commit 7abf2b5.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-02T02:23:42Z

Test build #124845 has finished for PR 28969 at commit 7abf2b5.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM.

SparkQA · 2020-07-02T14:59:37Z

Test build #124914 has finished for PR 28969 at commit 7abf2b5.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-02T18:53:32Z

Test build #124918 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-02T22:15:32Z

retest this please

HyukjinKwon · 2020-07-03T01:18:14Z

retest this please

SparkQA · 2020-07-05T19:56:38Z

Test build #124934 has started for PR 28969 at commit 7abf2b5.

maropu · 2020-07-06T07:08:40Z

retest this please

SparkQA · 2020-07-06T09:43:39Z

Test build #125041 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-06T10:18:50Z

retest this please

SparkQA · 2020-07-06T10:36:32Z

Test build #125062 has finished for PR 28969 at commit 7abf2b5.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-06T11:06:34Z

retest this please

SparkQA · 2020-07-06T11:11:33Z

Test build #125069 has started for PR 28969 at commit 7abf2b5.

SparkQA · 2020-07-06T21:41:22Z

Test build #125133 has finished for PR 28969 at commit 7abf2b5.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-07T00:44:23Z

retest this please

SparkQA · 2020-07-07T04:36:50Z

Test build #125154 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? This PR aims to disable SBT `unidoc` generation testing in Jenkins environment because it's flaky in Jenkins environment and not used for the official documentation generation. Also, GitHub Action has the correct test coverage for the official documentation generation. - #28848 (comment) (amp-jenkins-worker-06) - #28926 (comment) (amp-jenkins-worker-06) - #28969 (comment) (amp-jenkins-worker-06) - #28975 (comment) (amp-jenkins-worker-05) - #28986 (comment) (amp-jenkins-worker-05) - #28992 (comment) (amp-jenkins-worker-06) - #28993 (comment) (amp-jenkins-worker-05) - #28999 (comment) (amp-jenkins-worker-04) - #29010 (comment) (amp-jenkins-worker-03) - #29013 (comment) (amp-jenkins-worker-04) - #29016 (comment) (amp-jenkins-worker-05) - #29025 (comment) (amp-jenkins-worker-04) - #29042 (comment) (amp-jenkins-worker-03) ### Why are the changes needed? Apache Spark `release-build.sh` generates the official document by using the following command. - https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh#L341 ```bash PRODUCTION=1 RELEASE_VERSION="$SPARK_VERSION" jekyll build ``` And, this is executed by the following `unidoc` command for Scala/Java API doc. - https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L30 ```ruby system("build/sbt -Pkinesis-asl clean compile unidoc") || raise("Unidoc generation failed") ``` However, the PR builder disabled `Jekyll build` and instead has a different test coverage. ```python # determine if docs were changed and if we're inside the amplab environment # note - the below commented out until *all* Jenkins workers can get `jekyll` installed # if "DOCS" in changed_modules and test_env == "amplab_jenkins": # build_spark_documentation() ``` ``` Building Unidoc API Documentation ======================================================================== [info] Building Spark unidoc using SBT with these arguments: -Phadoop-3.2 -Phive-2.3 -Pspark-ganglia-lgpl -Pkubernetes -Pmesos -Phadoop-cloud -Phive -Phive-thriftserver -Pkinesis-asl -Pyarn unidoc ``` ### Does this PR introduce _any_ user-facing change? No. (This is used only for testing and not used in the official doc generation.) ### How was this patch tested? Pass the Jenkins without doc generation invocation. Closes #29017 from dongjoon-hyun/SPARK-DOC-GEN. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

SparkQA · 2020-07-08T23:26:16Z

Test build #125396 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-08T23:43:52Z

retest this please

SparkQA · 2020-07-09T02:46:56Z

Test build #125418 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-09T02:47:37Z

retest this please

SparkQA · 2020-07-09T05:32:56Z

Test build #125430 has finished for PR 28969 at commit 7abf2b5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-09T06:06:12Z

retest this please

SparkQA · 2020-07-09T07:05:02Z

Test build #125438 has finished for PR 28969 at commit 7abf2b5.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-09T07:14:52Z

retest this please

SparkQA · 2020-07-09T11:47:18Z

Test build #125444 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-09T11:49:48Z

retest this please

SparkQA · 2020-07-09T11:50:56Z

Test build #125467 has started for PR 28969 at commit 7abf2b5.

SparkQA · 2020-07-09T22:03:08Z

Test build #125504 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-10T02:18:36Z

Test build #125519 has finished for PR 28969 at commit 7abf2b5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-10T02:23:10Z

retest this please

SparkQA · 2020-07-10T06:02:24Z

Test build #125541 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-10T06:06:59Z

retest this please

SparkQA · 2020-07-10T07:05:01Z

Test build #125560 has finished for PR 28969 at commit 7abf2b5.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-10T07:16:59Z

retest this please

SparkQA · 2020-07-10T11:22:33Z

Test build #125575 has finished for PR 28969 at commit 7abf2b5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-10T12:23:45Z

retest this please

SparkQA · 2020-07-10T17:20:39Z

Test build #125607 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-10T22:55:50Z

Test build #125629 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-11T03:45:36Z

Test build #125652 has finished for PR 28969 at commit 7abf2b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-11T08:10:27Z

retest this please

SparkQA · 2020-07-11T11:08:53Z

Test build #125673 has finished for PR 28969 at commit 7abf2b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-11T15:45:37Z

Merged to master.

### What changes were proposed in this pull request? This PR aims to upgrade to ZStd 1.4.5-4. ### Why are the changes needed? ZStd 1.4.5-4 fixes the following. - luben/zstd-jni@3d16e51 - luben/zstd-jni@3d51bdc ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the Jenkins. Closes apache#28969 from williamhyun/zstd2. Authored-by: William Hyun <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

[SPARK-XXX][BUILD] Upgrade to ZStd 1.4.5-4

7abf2b5

probot-autolabeler bot added the BUILD label Jul 1, 2020

williamhyun changed the title ~~[SPARK-XXX][BUILD] Upgrade to ZStd 1.4.5-4~~ [SPARK-32150][BUILD] Upgrade to ZStd 1.4.5-4 Jul 1, 2020

HyukjinKwon approved these changes Jul 2, 2020

View reviewed changes

dongjoon-hyun approved these changes Jul 2, 2020

View reviewed changes

dongjoon-hyun mentioned this pull request Jul 6, 2020

[SPARK-32233][TESTS] Disable SBT unidoc generation testing in Jenkins #29017

Closed

HyukjinKwon approved these changes Jul 11, 2020

View reviewed changes

HyukjinKwon closed this in 10a65ee Jul 11, 2020

[SPARK-32150][BUILD] Upgrade to ZStd 1.4.5-4 #28969

[SPARK-32150][BUILD] Upgrade to ZStd 1.4.5-4 #28969

Uh oh!

Conversation

williamhyun commented Jul 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Jul 1, 2020

Uh oh!

maropu commented Jul 2, 2020

Uh oh!

srowen commented Jul 2, 2020

Uh oh!

maropu commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

maropu commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

maropu commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

maropu commented Jul 2, 2020

Uh oh!

HyukjinKwon commented Jul 3, 2020

Uh oh!

SparkQA commented Jul 5, 2020

Uh oh!

maropu commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

HyukjinKwon commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

HyukjinKwon commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

HyukjinKwon commented Jul 7, 2020

Uh oh!

SparkQA commented Jul 7, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

maropu commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

HyukjinKwon commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

HyukjinKwon commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

HyukjinKwon commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

williamhyun commented Jul 1, 2020 •

edited

Loading