[SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT #20222

gatorsmile · 2018-01-10T15:03:33Z

What changes were proposed in this pull request?

This patch bumps the master branch version to 2.4.0-SNAPSHOT.

How was this patch tested?

N/A

gatorsmile · 2018-01-10T15:04:00Z

cc @yhuai @sameeragarwal @JoshRosen @rxin @cloud-fan

srowen · 2018-01-10T15:17:28Z

Yeah ideally that's part of the branching process. I think it could be documented in the "Preparing Spark for Release" section of release-process.md in spark-website. Open up a separate PR for that if you like, or I can do it if you're not set up for it.

gatorsmile · 2018-01-10T15:31:59Z

@srowen Yeah. I am still in travel mode. : ) Please help do it. Thanks!

srowen · 2018-01-10T15:42:10Z

R/pkg/DESCRIPTION needs an update too.
I'll update the release process.

@JoshRosen or @shaneknapp looks like we also need to have set up the branch-2.3 tests in Jenkins. Right now it's not tested. I might have access to do it. Is it safe to copy and paste existing jobs and change the branch reference?

JoshRosen · 2018-01-10T17:04:44Z

We should already be set up for 2.3.x builds in AMPLab Jenkins. For example: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-branch-2.3-test-maven-hadoop-2.6/

JoshRosen · 2018-01-10T17:09:23Z

project/MimaExcludes.scala

 */
 object MimaExcludes {

+  // Exclude rules for 2.4.x


This reminds me that we should probably bump previousSparkVersion in MimaBuild.scala to be 2.2.0: https://github.com/apache/spark/blame/f340b6b3066033d40b7e163fd5fb68e9820adfb1/project/MimaBuild.scala#L91. I think this should happen for both master and branch-2.3.

See #15061 for an example of a similar change when 2.1.x was being prepared (it looks like we missed this step for 2.2.0). We may also need to un-exclude any new subprojects / artifacts that were added in 2.2.0 since they now need to be backwards-compatibility-tested for 2.3.x.

We might want to make my proposed MiMa changes in a separate patch so the same set of reviewed changes can go to both master and branch-2.3.

srowen · 2018-01-10T17:11:38Z

Oh I see, they're just not listed on https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/ . I believe I can fix that.

felixcheung · 2018-01-10T19:02:07Z

Wait dev/create-release/release-tag.sh does this automatically though. I just want to make sure we are not missing things (like R/pkg/DESCRIPTION) I think maybe we should run a subset of release-tag

SparkQA · 2018-01-10T19:14:51Z

Test build #85922 has finished for PR 20222 at commit 3e1d6d6.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-01-10T20:17:50Z

Test build #85923 has finished for PR 20222 at commit 9107f9f.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2018-01-10T21:46:37Z

argh. i'll bump the job timeout now.

…

On Wed, Jan 10, 2018 at 12:19 PM, UCB AMPLab ***@***.***> wrote: Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85923/ Test FAILed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20222 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABiDrAOQSJRXi3DeWcIaxSmSuX7ayuawks5tJRtYgaJpZM4RZdH7> .

jiangxb1987 · 2018-01-10T23:20:25Z

retest this please

HyukjinKwon

Can we update python/pyspark/version.py too?

SparkQA · 2018-01-11T03:32:56Z

Test build #85935 has finished for PR 20222 at commit 9107f9f.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

felixcheung

by looking at release-tag,

python/pyspark/version.py is another one we need to update, note it should say 2.4.0.dev0

https://github.com/apache/spark/blob/master/dev/create-release/release-tag.sh#L87

gatorsmile · 2018-01-11T06:53:11Z

Thanks for your review!

SparkQA · 2018-01-11T08:05:01Z

Test build #85952 has finished for PR 20222 at commit 5eded03.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2018-01-11T08:05:59Z

Jenkins, retest this please.

SparkQA · 2018-01-11T12:17:48Z

Test build #85955 has finished for PR 20222 at commit 5eded03.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-01-11T12:26:45Z

@shaneknapp Could we bump the job timeout?

HyukjinKwon · 2018-01-11T13:14:11Z

retest this please

shaneknapp · 2018-01-11T17:12:28Z

@gatorsmile -- the timeout is currently at six hours. i am loathe to bump it any higher.

SparkQA · 2018-01-11T17:27:55Z

Test build #85966 has finished for PR 20222 at commit 5eded03.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2018-01-11T17:39:44Z

confused:

anyways, i will change the timeout. again.

shaneknapp · 2018-01-11T17:40:03Z

test this please

SparkQA · 2018-01-11T21:52:51Z

Test build #85980 has finished for PR 20222 at commit 5eded03.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2018-01-11T22:57:02Z

retest this please

HyukjinKwon · 2018-01-12T00:40:02Z

I think we should fix:

spark/dev/run-tests-jenkins.py

Lines 183 to 185 in 7c7bc8f

    
           # format: http://linux.die.net/man/1/timeout 
        
           # must be less than the timeout configured on Jenkins (currently 300m) 
        
           tests_timeout = "250m"

too.

cc @JoshRosen, since you increased time before IIRC.

SparkQA · 2018-01-12T03:08:08Z

Test build #85996 has finished for PR 20222 at commit 5eded03.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-01-12T05:04:52Z

dev/run-tests-jenkins.py

-    # must be less than the timeout configured on Jenkins (currently 300m)
-    tests_timeout = "250m"
+    # must be less than the timeout configured on Jenkins (currently 350m)
+    tests_timeout = "300m"


Ah, this was the root causes. Thanks, @HyukjinKwon .
Sorry, @shaneknapp . I sent an annoying email before knowing this here.

SparkQA · 2018-01-12T05:46:51Z

Test build #86008 has finished for PR 20222 at commit 440d11f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-01-12T05:49:27Z

retest this please

SparkQA · 2018-01-12T08:05:02Z

Test build #86021 has finished for PR 20222 at commit 440d11f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-01-12T08:06:22Z

retest this please

HyukjinKwon · 2018-01-12T12:27:22Z

BTW, does anyone know why it suddently takes logner time?

HyukjinKwon · 2018-01-12T12:38:12Z

dev/run-tests-jenkins.py

    # format: http://linux.die.net/man/1/timeout
-    # must be less than the timeout configured on Jenkins (currently 300m)
-    tests_timeout = "250m"
+    # must be less than the timeout configured on Jenkins (currently 350m)


Not a big deal but seems currently we set it to 361m in Jenkins now - cc @shaneknapp.

I think this comment is OK.

SparkQA · 2018-01-12T12:40:48Z

Test build #86026 has finished for PR 20222 at commit 440d11f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-01-12T13:19:49Z

Unlike the other PRs, this PR changes all the POMs. It requires running all the test cases. Thus, this hit the limit.

dongjoon-hyun · 2018-01-12T13:45:47Z

No, all the others seems to suffer together.
I think this might be related to the intel patch done by Shane a few days ago.

cloud-fan · 2018-01-12T16:22:32Z

is it good to go?

cloud-fan · 2018-01-12T16:38:08Z

LGTM, merging to master!

cloud-fan · 2018-01-12T16:39:17Z

BTW don't forget #20222 (comment) :)

gatorsmile added 2 commits January 10, 2018 22:57

bump

932d93e

bump

3e1d6d6

bump

9107f9f

JoshRosen reviewed Jan 10, 2018

View reviewed changes

HyukjinKwon reviewed Jan 11, 2018

View reviewed changes

felixcheung requested changes Jan 11, 2018

View reviewed changes

bump

5eded03

test

440d11f

dongjoon-hyun reviewed Jan 12, 2018

View reviewed changes

HyukjinKwon reviewed Jan 12, 2018

View reviewed changes

asfgit closed this in 651f761 Jan 12, 2018

maropu mentioned this pull request Jul 19, 2018

[WIP][SPARK-24498][SQL] Add JDK compiler for runtime codegen #21777

Closed

[SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT #20222

[SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT #20222

Uh oh!

Conversation

gatorsmile commented Jan 10, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

gatorsmile commented Jan 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Jan 10, 2018

Uh oh!

gatorsmile commented Jan 10, 2018

Uh oh!

srowen commented Jan 10, 2018

Uh oh!

JoshRosen commented Jan 10, 2018

Uh oh!

JoshRosen Jan 10, 2018

Choose a reason for hiding this comment

Uh oh!

JoshRosen Jan 10, 2018

Choose a reason for hiding this comment

Uh oh!

srowen commented Jan 10, 2018

Uh oh!

felixcheung commented Jan 10, 2018 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Jan 10, 2018

Uh oh!

SparkQA commented Jan 10, 2018

Uh oh!

shaneknapp commented Jan 10, 2018 via email

Uh oh!

jiangxb1987 commented Jan 10, 2018

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 11, 2018

Uh oh!

felixcheung left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jan 11, 2018

Uh oh!

SparkQA commented Jan 11, 2018

Uh oh!

ueshin commented Jan 11, 2018

Uh oh!

SparkQA commented Jan 11, 2018

Uh oh!

gatorsmile commented Jan 11, 2018

Uh oh!

HyukjinKwon commented Jan 11, 2018

Uh oh!

shaneknapp commented Jan 11, 2018

Uh oh!

SparkQA commented Jan 11, 2018

Uh oh!

shaneknapp commented Jan 11, 2018

Uh oh!

shaneknapp commented Jan 11, 2018

Uh oh!

SparkQA commented Jan 11, 2018

Uh oh!

jiangxb1987 commented Jan 11, 2018

Uh oh!

HyukjinKwon commented Jan 12, 2018

Uh oh!

SparkQA commented Jan 12, 2018

Uh oh!

dongjoon-hyun Jan 12, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 12, 2018

Uh oh!

gatorsmile commented Jan 10, 2018 •

edited

Loading

felixcheung commented Jan 10, 2018 via email •

edited

Loading

felixcheung left a comment •

edited

Loading

cloud-fan commented Jan 12, 2018 •

edited

Loading