[SPARK-24886][INFRA] Fix the testing script to increase timeout for Jenkins build (from 340m to 400m)

[HyukjinKwon]

What changes were proposed in this pull request?

This PR targets to increase the timeout from 340 to 400m. Please also see #21845 (comment)

How was this patch tested?

N/A

[HyukjinKwon]

cc @shaneknapp

[SparkQA]

Test build #94722 has finished for PR 22098 at commit bffce5e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gengliangwang]

One suggestion: can we add a environment variable for the script execution timeout in Jenkins, so that we don't need to modify the hardcode value here?

[HyukjinKwon]

Hm.. makes sense. @shaneknapp WDYT? I can also log the timeout as well and default to 400m if not set just in case for now. Would this make the things complicated? If so, let me just leave it as is.

[shaneknapp]

to be honest, i have *no* idea why this was done in the first place... the jenkins build already has a timeout. i'll dig around today and see if i can't find out who put this in the repo in the first place.

…

On Tue, Aug 14, 2018 at 12:58 AM, Hyukjin Kwon ***@***.***> wrote: Hm.. makes sense. @shaneknapp <https://github.com/shaneknapp> WDYT? I can also log the timeout as well and default to 400m if not set just in case for now. Would this make the things complicated? If so, let me just leave it as is. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22098 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABiDrLAUnGOCZM2w7r__4XIM4eBf7JxVks5uQoMNgaJpZM4V7rKC> .

[HyukjinKwon]

Ah, @shaneknapp, probably one possibility about the motivation is that we don't touch build configuration often and control the build time within the codebase so that committers don't have to bother you every time it needs. For instance, although I know how to control the build time too, I don't touch the build configuration itself.

Shall we just keep the current way and discuss if we happen to increase it again? I think 400m looks very enough as a limit probably at least for an year I guess, and we will put some efforts to reduce or keep the current limit I believe.

[shaneknapp]

Ah, @shaneknapp <https://github.com/shaneknapp>, probably one possibility about the motivation is that we don't touch build configuration often and control the build time within the codebase so that committers don't have to bother you every time it needs. For instance, although I know how to control the build time too, I don't touch them.

that makes sense, but at the same time, it does require coordination to happen between the timeout in the test framework and setting the jenkins (absolute timeout)... which literally is just adding another step to the process.

Shall we just keep the current way and discuss if we happen to increase it again? I think 400m looks very enough as a limit, and we will put some efforts to reduce or keep the current limit I believe.

sure, this works for me. and seeing that our builds are now taking upwards of ***6*** hours, i'd rather have a conversation about build strategy before revisiting this. :)

…

[HyukjinKwon]

@shaneknapp, seems this was first introduced in https://issues.apache.org/jira/browse/SPARK-3076 / #1974 for a good reason fwiw.

One thing I am not sure is if we need to have env or not ..

[shaneknapp]

oh wow, nice find @HyukjinKwon! code archaeology ftw! :)

anyways: that PR went through about a week after i started here @ the amplab and i don't think i even had access to the build system at that point.

[HyukjinKwon]

haha, it's more then 4 years ago .. if we are unsure on the env, let me just push this in.

[HyukjinKwon]

Let me just push this in.

Merged to master.