[SPARK-29991][INFRA] Support Hive 1.2 and Hive 2.3 (default) in PR builder #26710

HyukjinKwon · 2019-11-29T07:29:45Z

What changes were proposed in this pull request?

Currently, Apache Spark PR Builder is using hive-1.2 for hadoop-2.7 and hive-2.3 for hadoop-3.2. This PR aims to support

[test-hive1.2] in PR builder
[test-hive2.3] in PR builder to be consistent and independent of the default profile
After this PR, all PR builders will use Hive 2.3 by default (because Spark uses Hive 2.3 by default as of c98e5eb)
Use default profile in AppVeyor build.

Note that this was reverted due to unexpected test failure at ThriftServerPageSuite, which was investigated in #26706 . This PR fixed it by letting it use their own forked JVM. There is no explicit evidence for this fix and it was just my speculation, and thankfully it fixed at least.

Why are the changes needed?

This new tag allows us more flexibility.

Does this PR introduce any user-facing change?

No. (This is a dev-only change.)

How was this patch tested?

Check the Jenkins triggers in this PR.

Default:

========================================================================
Building Spark
========================================================================
[info] Building Spark using SBT with these arguments:  -Phadoop-2.7 -Phive-2.3 -Phive-thriftserver -Pmesos -Pspark-ganglia-lgpl -Phadoop-cloud -Phive -Pkubernetes -Pkinesis-asl -Pyarn test:package streaming-kinesis-asl-assembly/assembly

[test-hive1.2][test-hadoop3.2]:

========================================================================
Building Spark
========================================================================
[info] Building Spark using SBT with these arguments:  -Phadoop-3.2 -Phive-1.2 -Phadoop-cloud -Pyarn -Pspark-ganglia-lgpl -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Pkinesis-asl test:package streaming-kinesis-asl-assembly/assembly

[test-maven][test-hive2.3]:

========================================================================
Building Spark
========================================================================
[info] Building Spark using Maven with these arguments:  -Phadoop-2.7 -Phive-2.3 -Pspark-ganglia-lgpl -Pyarn -Phive -Phadoop-cloud -Pkinesis-asl -Pmesos -Pkubernetes -Phive-thriftserver clean package -DskipTests

…lder"" This reverts commit 9351e3e.

HyukjinKwon · 2019-11-29T07:30:48Z

cc @dongjoon-hyun, @wangyum, and @srowen

HyukjinKwon · 2019-11-29T07:37:28Z

project/SparkBuild.scala

    "org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite",
-    "org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite"
+    "org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite",
+    "org.apache.spark.sql.hive.thriftserver.ui.ThriftServerPageSuite"


This fixes the test failure against Hive 2.3 (tested in #26706); however, I have no explicit evidence. Just given my speculation and it worked.

I'm still seeing failures like in https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4953/testReport/ - should we make all the thriftserver tests single-threaded?

Ah yeah. Maybe that's a better idea. I'll monitor a bit more and make a PR soon

HyukjinKwon · 2019-11-29T08:29:42Z

dev/run-tests-jenkins.py

+    if "test-hive1.2" in ghprb_pull_title:
+        os.environ["AMPLAB_JENKINS_BUILD_HIVE_PROFILE"] = "hive1.2"
+    if "test-hive2.3" in ghprb_pull_title:
+        os.environ["AMPLAB_JENKINS_BUILD_HIVE_PROFILE"] = "hive2.3"


cc @shaneknapp, maybe we can use it this environment variable (?) later when we consider setting the Jenkins jobs for Hive 1.2/2.3 + Hadoop 2.7/3.2 + JDK 8/11 combinations.

The env name looks a bit odd given AMPLAB_JENKINS_BUILD_PROFILE. We might have to rename it later.

sorry, i missed this over the november holiday. yes, i've had a desire for years to rename the AMPLAB_* variables as the amplab project ended in december 2016. :)

i'll do this after the 3.0 cut. no time like the present!

SparkQA · 2019-11-29T09:55:33Z

Test build #114614 has finished for PR 26710 at commit 2135955.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-29T10:44:39Z

Test build #114618 has finished for PR 26710 at commit 2135955.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen

Looks reasonable, esp. the last part that may fix the test problem

SparkQA · 2019-11-29T15:15:52Z

Test build #114621 has finished for PR 26710 at commit 2135955.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-11-29T15:41:33Z

retest this please

HyukjinKwon · 2019-11-29T15:42:04Z

Yeah, seems it fixed. Let me merge this tomorrow if this isn't merged

dongjoon-hyun

Thank you for the fix!

SparkQA · 2019-11-29T21:30:17Z

Test build #114635 has finished for PR 26710 at commit 2135955.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-11-30T02:10:43Z

Do we need to support testing [test-hive1.2][test-hadoop3.2]?

HyukjinKwon · 2019-11-30T03:47:06Z

No because it doesn't work but let's leave it since our build itself already allows such profile combinations.

HyukjinKwon · 2019-11-30T03:47:51Z

If you feel strongly, I will make a followup to disallow that specific matrix. Let me know @wangyum.

Merged to master.

wangyum · 2019-11-30T04:09:54Z

OK. Thank you @HyukjinKwon LGTM

shaneknapp · 2019-11-30T19:10:46Z

um, sure... maybe? i'll take a closer look at all of this stuff next week when the holiday is over (including adding/renaming the jenkins jobs).

…

On Fri, Nov 29, 2019 at 12:29 AM Hyukjin Kwon ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In dev/run-tests-jenkins.py <#26710 (comment)>: > @@ -182,6 +182,11 @@ def main(): os.environ["AMPLAB_JENKINS_BUILD_PROFILE"] = "hadoop2.7" if "test-hadoop3.2" in ghprb_pull_title: os.environ["AMPLAB_JENKINS_BUILD_PROFILE"] = "hadoop3.2" + # Switch the Hive profile based on the PR title: + if "test-hive1.2" in ghprb_pull_title: + os.environ["AMPLAB_JENKINS_BUILD_HIVE_PROFILE"] = "hive1.2" + if "test-hive2.3" in ghprb_pull_title: + os.environ["AMPLAB_JENKINS_BUILD_HIVE_PROFILE"] = "hive2.3" cc @shaneknapp <https://github.com/shaneknapp>, maybe we can use it this environment variable (?) later when we consider setting the Jenkins jobs for Hive 1.2/2.3 + Hadoop 2.7/3.2 + JDK 8/11 combinations. The env name looks a bit odd given AMPLAB_JENKINS_BUILD_PROFILE. We might have to rename it later. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26710?email_source=notifications&email_token=AAMIHLDTVQOMNYD344ZKMALQWDHHTA5CNFSM4JS35F5KYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCNMKFAY#pullrequestreview-324575875>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMIHLAVWZBXFCZ5ADOAVVTQWDHHTANCNFSM4JS35F5A> .

…ilder ### What changes were proposed in this pull request? Currently, Apache Spark PR Builder using `hive-1.2` for `hadoop-2.7` and `hive-2.3` for `hadoop-3.2`. This PR aims to support - `[test-hive1.2]` in PR builder - `[test-hive2.3]` in PR builder to be consistent and independent of the default profile - After this PR, all PR builders will use Hive 2.3 by default (because Spark uses Hive 2.3 by default as of apache@c98e5eb) - Use default profile in AppVeyor build. Note that this was reverted due to unexpected test failure at `ThriftServerPageSuite`, which was investigated in apache#26706 . This PR fixed it by letting it use their own forked JVM. There is no explicit evidence for this fix and it was just my speculation, and thankfully it fixed at least. ### Why are the changes needed? This new tag allows us more flexibility. ### Does this PR introduce any user-facing change? No. (This is a dev-only change.) ### How was this patch tested? Check the Jenkins triggers in this PR. Default: ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-2.3 -Phive-thriftserver -Pmesos -Pspark-ganglia-lgpl -Phadoop-cloud -Phive -Pkubernetes -Pkinesis-asl -Pyarn test:package streaming-kinesis-asl-assembly/assembly ``` `[test-hive1.2][test-hadoop3.2]`: ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using SBT with these arguments: -Phadoop-3.2 -Phive-1.2 -Phadoop-cloud -Pyarn -Pspark-ganglia-lgpl -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Pkinesis-asl test:package streaming-kinesis-asl-assembly/assembly ``` `[test-maven][test-hive-2.3]`: ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using Maven with these arguments: -Phadoop-2.7 -Phive-2.3 -Pspark-ganglia-lgpl -Pyarn -Phive -Phadoop-cloud -Pkinesis-asl -Pmesos -Pkubernetes -Phive-thriftserver clean package -DskipTests ``` Closes apache#26710 from HyukjinKwon/SPARK-29991. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

gatorsmile

@wangyum @HyukjinKwon could you help update the website https://spark.apache.org/developer-tools.html for including these new tags?

dongjoon-hyun · 2020-01-09T21:49:01Z

+1 for @gatorsmile 's advice.

HyukjinKwon · 2020-01-10T01:23:56Z

Sure, will do.

…ranch-3.0-test-sbt-hadoop-2.7-hive-2.3 ### What changes were proposed in this pull request? This PR tries #26710 (comment) way to fix the test. ### Why are the changes needed? To make the tests pass. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Jenkins will test first, and then `on spark-branch-3.0-test-sbt-hadoop-2.7-hive-2.3` will test it out. Closes #27513 from HyukjinKwon/test-SPARK-30756. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

…ranch-3.0-test-sbt-hadoop-2.7-hive-2.3 ### What changes were proposed in this pull request? This PR tries #26710 (comment) way to fix the test. ### Why are the changes needed? To make the tests pass. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Jenkins will test first, and then `on spark-branch-3.0-test-sbt-hadoop-2.7-hive-2.3` will test it out. Closes #27513 from HyukjinKwon/test-SPARK-30756. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 8efe367) Signed-off-by: HyukjinKwon <[email protected]>

…ranch-3.0-test-sbt-hadoop-2.7-hive-2.3 ### What changes were proposed in this pull request? This PR tries apache#26710 (comment) way to fix the test. ### Why are the changes needed? To make the tests pass. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Jenkins will test first, and then `on spark-branch-3.0-test-sbt-hadoop-2.7-hive-2.3` will test it out. Closes apache#27513 from HyukjinKwon/test-SPARK-30756. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 8efe367) Signed-off-by: HyukjinKwon <[email protected]>

HyukjinKwon added 2 commits November 29, 2019 15:23

Revert "Revert "[SPARK-29991][INFRA] Support test-hive1.2 in PR Bui…

158af66

…lder"" This reverts commit 9351e3e.

Support test-hive1.2 and test-hive2.3 (default) in PR builder

2135955

HyukjinKwon changed the title ~~[SPARK-29991][INFRA] Support test-hive1.2 and test-hive2.3 (default) in PR builder~~ [SPARK-29991][INFRA][test-hive2.3] Support test-hive1.2 and test-hive2.3 (default) in PR builder Nov 29, 2019

HyukjinKwon changed the title ~~[SPARK-29991][INFRA][test-hive2.3] Support test-hive1.2 and test-hive2.3 (default) in PR builder~~ [SPARK-29991][INFRA][test-hive1.2] Support test-hive1.2 and test-hive2.3 (default) in PR builder Nov 29, 2019

HyukjinKwon commented Nov 29, 2019

View reviewed changes

HyukjinKwon changed the title ~~[SPARK-29991][INFRA][test-hive1.2] Support test-hive1.2 and test-hive2.3 (default) in PR builder~~ [SPARK-29991][INFRA][test-hive1.2][test-hadoop3.2] Support test-hive1.2 and test-hive2.3 (default) in PR builder Nov 29, 2019

HyukjinKwon changed the title ~~[SPARK-29991][INFRA][test-hive1.2][test-hadoop3.2] Support test-hive1.2 and test-hive2.3 (default) in PR builder~~ [WIP][SPARK-29991][INFRA][test-hive1.2][test-hadoop3.2] Support test-hive1.2 and test-hive2.3 (default) in PR builder Nov 29, 2019

This comment has been minimized.

Sign in to view

HyukjinKwon changed the title ~~[WIP][SPARK-29991][INFRA][test-hive1.2][test-hadoop3.2] Support test-hive1.2 and test-hive2.3 (default) in PR builder~~ [WIP][SPARK-29991][INFRA][test-hive1.2][test-hadoop3.2] Support Hive 1.3 and Hive 2.3 (default) in PR builder Nov 29, 2019

HyukjinKwon force-pushed the SPARK-29991 branch from 47e7c42 to 2135955 Compare November 29, 2019 08:09

This comment has been minimized.

Sign in to view

HyukjinKwon changed the title ~~[WIP][SPARK-29991][INFRA][test-hive1.2][test-hadoop3.2] Support Hive 1.3 and Hive 2.3 (default) in PR builder~~ [SPARK-29991][INFRA][test-hive1.2][test-hadoop3.2] Support Hive 1.3 and Hive 2.3 (default) in PR builder Nov 29, 2019

HyukjinKwon mentioned this pull request Nov 29, 2019

[SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column #26697

Closed

HyukjinKwon changed the title ~~[SPARK-29991][INFRA][test-hive1.2][test-hadoop3.2] Support Hive 1.3 and Hive 2.3 (default) in PR builder~~ [SPARK-29991][INFRA] Support Hive 1.3 and Hive 2.3 (default) in PR builder Nov 29, 2019

HyukjinKwon commented Nov 29, 2019

View reviewed changes

HyukjinKwon changed the title ~~[SPARK-29991][INFRA] Support Hive 1.3 and Hive 2.3 (default) in PR builder~~ [SPARK-29991][INFRA][test-maven][hive-2.3] Support Hive 1.3 and Hive 2.3 (default) in PR builder Nov 29, 2019

HyukjinKwon changed the title ~~[SPARK-29991][INFRA][test-maven][hive-2.3] Support Hive 1.3 and Hive 2.3 (default) in PR builder~~ [SPARK-29991][INFRA] Support Hive 1.3 and Hive 2.3 (default) in PR builder Nov 29, 2019

srowen reviewed Nov 29, 2019

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-29991][INFRA] Support Hive 1.3 and Hive 2.3 (default) in PR builder~~ [SPARK-29991][INFRA] Support Hive 1.2 and Hive 2.3 (default) in PR builder Nov 29, 2019

dongjoon-hyun approved these changes Nov 29, 2019

View reviewed changes

HyukjinKwon closed this in 4a73bed Nov 30, 2019

gatorsmile reviewed Jan 9, 2020

View reviewed changes

HyukjinKwon mentioned this pull request Feb 11, 2020

[SPARK-30756][SQL] Fix ThriftServerWithSparkContextSuite on spark-branch-3.0-test-sbt-hadoop-2.7-hive-2.3 #27513

Closed

HyukjinKwon deleted the SPARK-29991 branch March 3, 2020 01:17

[SPARK-29991][INFRA] Support Hive 1.2 and Hive 2.3 (default) in PR builder #26710

[SPARK-29991][INFRA] Support Hive 1.2 and Hive 2.3 (default) in PR builder #26710

Uh oh!

Conversation

HyukjinKwon commented Nov 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HyukjinKwon commented Nov 29, 2019

Uh oh!

HyukjinKwon Nov 29, 2019

Choose a reason for hiding this comment

Uh oh!

srowen Nov 30, 2019

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Nov 30, 2019

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

HyukjinKwon Nov 29, 2019

Choose a reason for hiding this comment

Uh oh!

shaneknapp Feb 11, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 29, 2019

Uh oh!

SparkQA commented Nov 29, 2019

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 29, 2019

Uh oh!

HyukjinKwon commented Nov 29, 2019

Uh oh!

HyukjinKwon commented Nov 29, 2019

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 29, 2019

Uh oh!

wangyum commented Nov 30, 2019

Uh oh!

HyukjinKwon commented Nov 30, 2019

Uh oh!

HyukjinKwon commented Nov 30, 2019

Uh oh!

wangyum commented Nov 30, 2019

Uh oh!

shaneknapp commented Nov 30, 2019 via email

Uh oh!

gatorsmile left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 9, 2020

Uh oh!

HyukjinKwon commented Jan 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

HyukjinKwon commented Nov 29, 2019 •

edited

Loading

gatorsmile left a comment •

edited

Loading