[SPARK-24529][BUILD][test-maven][FOLLOW-UP] Set spotbugs-maven-plugin's fork to true #21785

wangyum · 2018-07-16T14:52:24Z

What changes were proposed in this pull request?

Set spotbugs-maven-plugin's fork to true, otherwise will throw exception when make distribution:

./dev/make-distribution.sh --name SPARK-24529  --tgz  -Phadoop-2.7 -Phive -Phive-thriftserver -Pyarn -Phadoop-provided

exception:

...
[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  8.753 s]
[INFO] Spark Project Tags ................................. SUCCESS [  9.334 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 12.029 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 13.641 s]
[INFO] Spark Project Networking ........................... FAILURE [10:10 min]
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Unsafe ............................... SUCCESS [ 16.415 s]
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SUCCESS [  8.750 s]
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] Spark Project YARN ................................. SKIPPED
[INFO] Spark Project Hive Thrift Server ................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:29 min (Wall Clock)
[INFO] Finished at: 2018-07-16T21:39:46+08:00
[INFO] Final Memory: 61M/885M
[INFO] ------------------------------------------------------------------------
Timeout: sub-process interrupted
[ERROR] Failed to execute goal com.github.spotbugs:spotbugs-maven-plugin:3.1.3:spotbugs (spotbugs) on project spark-network-common_2.11: Execution spotbugs of goal com.github.spotbugs:spotbugs-maven-plugin:3.1.3:spotbugs failed: Timeout: killed the sub-process -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-network-common_2.11
org.apache.tools.ant.ExitException: Permission ("java.lang.RuntimePermission" "exitVM") was not granted.
        at org.apache.tools.ant.types.Permissions$MySM.checkExit(Permissions.java:194)
        at java.lang.Runtime.exit(Runtime.java:107)
        at java.lang.System.exit(System.java:971)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:358)
Exception in thread "main" org.apache.tools.ant.ExitException: Permission ("java.lang.RuntimePermission" "exitVM") was not granted.
        at org.apache.tools.ant.types.Permissions$MySM.checkExit(Permissions.java:194)
        at java.lang.Runtime.exit(Runtime.java:107)
        at java.lang.System.exit(System.java:971)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:364)
Timeout: sub-process interrupted

How was this patch tested?

manual tests

SparkQA · 2018-07-16T19:05:07Z

Test build #93114 has finished for PR 21785 at commit 9d87160.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-07-17T01:39:57Z

@wangyum, I think we better test the build against maven by adding [test-maven] in the PR title.

HyukjinKwon · 2018-07-17T02:34:09Z

retest this please

HyukjinKwon · 2018-07-17T04:10:33Z

cc @kiszk

SparkQA · 2018-07-17T04:48:49Z

Test build #93149 has finished for PR 21785 at commit 9d87160.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-07-17T06:17:45Z

retest this please

kiszk · 2018-07-17T06:29:20Z

Now, I am checking make-distribution.sh in my environment. If my memory is correct, the error in spark-sql_2.11 or spark-catalyst_2.11 was a motivation to stop forking of spotbugs.

SparkQA · 2018-07-17T07:05:02Z

Test build #93154 has finished for PR 21785 at commit 9d87160.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-07-17T08:29:29Z

retest this please

skonto · 2018-07-17T12:41:07Z

+1 I saw this with my build. I reported on the other PR.

kiszk · 2018-07-17T13:01:05Z

In my environment, make-distribution.sh works well without fork.

...
[INFO] 
[INFO] --- maven-source-plugin:3.0.1:jar-no-fork (create-source-jar) @ spark-assembly_2.11 ---
[INFO] 
[INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork (create-source-jar) @ spark-assembly_2.11 ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.8:run (default) @ spark-assembly_2.11 ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork (create-source-jar) @ spark-examples_2.11 ---
[INFO] Building jar: /home/ishizaki/Spark/PR/overflow/SPARK-24529/spark/examples/target/spark-examples_2.11-2.4.0-SNAPSHOT-test-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  3.749 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 19.141 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 24.832 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 19.311 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 53.422 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 30.715 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 33.014 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 22.917 s]
[INFO] Spark Project Core ................................. SUCCESS [32:29 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [01:15 min]
[INFO] Spark Project GraphX ............................... SUCCESS [01:24 min]
[INFO] Spark Project Streaming ............................ SUCCESS [04:32 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [14:08 min]
[INFO] Spark Project SQL .................................. SUCCESS [12:10 min]
[INFO] Spark Project ML Library ........................... SUCCESS [04:59 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 16.323 s]
[INFO] Spark Project Hive ................................. SUCCESS [08:14 min]
[INFO] Spark Project REPL ................................. SUCCESS [  7.546 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 17.865 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 59.089 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 38.509 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  1.320 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 46.277 s]
[INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [ 27.301 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 44.678 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  7.238 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:09 h (Wall Clock)
[INFO] Finished at: 2018-07-17T16:23:18+09:00
[INFO] Final Memory: 78M/1967M
[INFO] ------------------------------------------------------------------------
...

SparkQA · 2018-07-17T13:08:11Z

Test build #93161 has finished for PR 21785 at commit 9d87160.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-07-17T13:30:19Z

@kiszk What is your environment? My environment is:

Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 on Mac OS X 10.12.6

kiszk · 2018-07-17T17:12:58Z

@wangyum my environment is

$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

kiszk · 2018-07-17T19:39:48Z

I got the same error in my Mac. I understand that the motivation of forking is this solution.

Unfortunately, I cannot access my Mac until this weekend. For now, let me retest again.

kiszk · 2018-07-17T19:39:55Z

retest this please

SparkQA · 2018-07-18T00:29:06Z

Test build #93198 has finished for PR 21785 at commit 9d87160.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-07-18T01:58:27Z

I can reproduce this and tested it fixes in my local as well.
@kiszk, let me just push this in since it fixes an actual issue and the build passes; however, please make a followup if you see something else to fix.

HyukjinKwon · 2018-07-18T02:00:59Z

Merged to master.

kiszk · 2018-07-21T16:41:15Z

@HyukjinKwon @wangyum thank you. I took a vacation.
If there is no error in amp build infra, I think that this is a better way.

Fix ./dev/make-distribution.sh Permission error

9d87160

wangyum changed the title ~~[SPARK-24529][BUILD][FOLLOW-UP] Set spotbugs-maven-plugin's fork to true~~ [SPARK-24529][BUILD][test-maven][FOLLOW-UP] Set spotbugs-maven-plugin's fork to true Jul 17, 2018

HyukjinKwon approved these changes Jul 18, 2018

View reviewed changes

asfgit closed this in fc2e189 Jul 18, 2018

[SPARK-24529][BUILD][test-maven][FOLLOW-UP] Set spotbugs-maven-plugin's fork to true #21785

[SPARK-24529][BUILD][test-maven][FOLLOW-UP] Set spotbugs-maven-plugin's fork to true #21785

Uh oh!

Conversation

wangyum commented Jul 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jul 16, 2018

Uh oh!

HyukjinKwon commented Jul 17, 2018

Uh oh!

HyukjinKwon commented Jul 17, 2018

Uh oh!

HyukjinKwon commented Jul 17, 2018

Uh oh!

SparkQA commented Jul 17, 2018

Uh oh!

wangyum commented Jul 17, 2018

Uh oh!

kiszk commented Jul 17, 2018

Uh oh!

SparkQA commented Jul 17, 2018

Uh oh!

wangyum commented Jul 17, 2018

Uh oh!

skonto commented Jul 17, 2018

Uh oh!

kiszk commented Jul 17, 2018

Uh oh!

SparkQA commented Jul 17, 2018

Uh oh!

wangyum commented Jul 17, 2018

Uh oh!

kiszk commented Jul 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kiszk commented Jul 17, 2018

Uh oh!

kiszk commented Jul 17, 2018

Uh oh!

SparkQA commented Jul 18, 2018

Uh oh!

HyukjinKwon commented Jul 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Jul 18, 2018

Uh oh!

kiszk commented Jul 21, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wangyum commented Jul 16, 2018 •

edited

Loading

kiszk commented Jul 17, 2018 •

edited

Loading

HyukjinKwon commented Jul 18, 2018 •

edited

Loading