[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

dongjoon-hyun · 2020-05-14T21:46:25Z

What changes were proposed in this pull request?

This PR makes test-dependencies.sh detect the version string correctly by ignoring all the other lines.

Why are the changes needed?

Currently, all SBT jobs are broken like the following.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/476/console

[error] running /home/jenkins/workspace/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/dev/test-dependencies.sh ; received return code 1
Build step 'Execute shell' marked build as failure

The reason is that the script detects the old version like Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT when build/mvn did fallback.

Specifically, in the script, OLD_VERSION became Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT instead of 3.1.0-SNAPSHOT if build/mvn did fallback. Then, pom.xml file is corrupted like the following at the end and the exit code become 1 instead of 0. It causes Jenkins jobs fails

-    <version>3.1.0-SNAPSHOT</version>
+    <version>Falling</version>

NO FALLBACK

$ build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec
Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn
3.1.0-SNAPSHOT

FALLBACK

$ build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn
3.1.0-SNAPSHOT

In the script

$ echo $(build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec)
Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn
Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT

This PR will prevent irrelevant logs like Falling back to archive.apache.org to download Maven.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the PR Builder.

…correctly

dongjoon-hyun · 2020-05-14T21:49:41Z

Hi, @srowen . Could you review this PR? This PR will remove the weird test-dependency failures on the PR builder and Jenkins jobs.

dongjoon-hyun · 2020-05-14T21:50:57Z

dev/test-dependencies.sh

    -Dexec.args='${project.version}' \
    --non-recursive \
-    org.codehaus.mojo:exec-maven-plugin:1.6.0:exec)
+    org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -e '[0-9]\.[0-9]\.[0-9]')


I used [0-9] because \d is accepted only on Mac OS.

This would still match 3.1.0-SNAPSHOT wouldn't it - was the idea not to match it?
You could exclude "SNAPSHOT" matches but that might be just a band-aid.

You might need [0-9]+ in each case to account for double-digit versions.

\d might work with -E but not worth it.

Thanks . Yes. It matches SNAPSHOT, too.

This is only grepping lines with version like string.

Oh, just to be clear, what are you trying to match or not match here?

In the script, $(...) merges two lines into one.

BTW, is the following working in grep on Mac?

You might need [0-9]+ in each case to account for double-digit versions.

It works but you need -E as that is 'extended' regex syntax, like many things. I think -E is standard across GNU / BSD grep.

OK if you tell me this works, then I'm missing something, but it seems like this grep does not exclude the line I think you are trying to exclude, unless you want to print only the matching version from that line, and that's -o I understand it's picking up "Falling" so I am assuming something needs to ignore this line or only extract the version. grep -v Falling would just ignore that line

Thanks. BTW, grep -v Falling is not robust enough. If someone adds another warning output someday, this situation will repeat again.

I'll update to use grep -E '[0-9]+\.[0-9]+\.[0-9]+'.

dongjoon-hyun · 2020-05-15T00:16:05Z

The falling back is happening in this PR's PR Builder like the following, but this PR builder passed. Please see Falling back to archive.apache.org to download Maven messages.

========================================================================
Running Apache RAT checks
========================================================================
Attempting to fetch rat
RAT checks passed.

========================================================================
Running build tests
========================================================================
exec: curl --silent --show-error -L https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz
exec: curl --silent --show-error -L https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.tgz
exec: curl --silent --show-error -L https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven install for hadoop-2.7-hive-1.2
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven validate for hadoop-2.7-hive-1.2
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Generating dependency manifest for hadoop-2.7-hive-1.2
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven install for hadoop-2.7-hive-2.3
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven validate for hadoop-2.7-hive-2.3
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Generating dependency manifest for hadoop-2.7-hive-2.3
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven install for hadoop-3.2-hive-2.3
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven validate for hadoop-3.2-hive-2.3
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Generating dependency manifest for hadoop-3.2-hive-2.3
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn

========================================================================
Building Spark
========================================================================
...

SparkQA · 2020-05-15T02:01:03Z

Test build #122636 has finished for PR 28532 at commit b96806a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-15T02:02:45Z

Oh.. In Jenkins server, the new pattern seems not working.

srowen · 2020-05-15T02:04:42Z

do you mean grep -E "^[0-9]+\.[0-9]+\.[0-9]+$"? that would force it to match the whole line and not accept anything else

dongjoon-hyun · 2020-05-15T02:05:16Z

Ah, it's different issue. There exists some corrupted ~/.m2/repository in some machines.

Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install (default-cli) on project spark-parent_2.12: ArtifactInstallerException: 
Failed to install metadata org.apache.spark:spark-parent_2.12/maven-metadata.xml: 
Could not parse metadata /home/jenkins/.m2/repository/org/apache/spark/spark-parent_2.12/maven-metadata-local.xml: 
in epilog non whitespace content is not allowed but got >
(position: END_TAG seen ...</metadata>\n>... @13:2) -> [Help 1]

dongjoon-hyun · 2020-05-15T02:06:12Z

It seems that we need to clean up ./m2/repository at all Jenkins machine. During recent network outage, the directory might be corrupted.

dongjoon-hyun · 2020-05-15T02:07:10Z

Retest this please.

SparkQA · 2020-05-15T02:12:29Z

Test build #122635 has finished for PR 28532 at commit e9bdb59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-15T02:17:15Z

Test build #122638 has finished for PR 28532 at commit b96806a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-15T02:17:59Z

amp-jenkins-worker-05 has the corrupted Maven repo. Last two runs were assigned into the same machine.

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532 (comment)
[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532 (comment)

dongjoon-hyun · 2020-05-15T02:18:47Z

@srowen . If you don't mind, can we merge this? This PR passed already on amp-jenkins-worker-03 with a fallback situation (#28532 (comment)). We don't have a configuration to assign this PR into another Jenkins job machine.

srowen · 2020-05-15T02:26:17Z

I trust your judgment; it can't really make things worse either if most PR builds are failing.

HyukjinKwon

Okay, same logic. LGTM

dongjoon-hyun · 2020-05-15T02:28:06Z

Thank you, @srowen and @HyukjinKwon !

HyukjinKwon · 2020-05-15T02:28:20Z

Should we clean that specific correct corrupted one in .m2, @shaneknapp?

…correctly ### What changes were proposed in this pull request? This PR makes `test-dependencies.sh` detect the version string correctly by ignoring all the other lines. ### Why are the changes needed? Currently, all SBT jobs are broken like the following. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/476/console ``` [error] running /home/jenkins/workspace/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/dev/test-dependencies.sh ; received return code 1 Build step 'Execute shell' marked build as failure ``` The reason is that the script detects the old version like `Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT` when `build/mvn` did fallback. Specifically, in the script, `OLD_VERSION` became `Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT` instead of `3.1.0-SNAPSHOT` if build/mvn did fallback. Then, `pom.xml` file is corrupted like the following at the end and the exit code become `1` instead of `0`. It causes Jenkins jobs fails ``` - <version>3.1.0-SNAPSHOT</version> + <version>Falling</version> ``` **NO FALLBACK** ``` $ build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn 3.1.0-SNAPSHOT ``` **FALLBACK** ``` $ build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec Falling back to archive.apache.org to download Maven Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn 3.1.0-SNAPSHOT ``` **In the script** ``` $ echo $(build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec) Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT ``` This PR will prevent irrelevant logs like `Falling back to archive.apache.org to download Maven`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the PR Builder. Closes #28532 from dongjoon-hyun/SPARK-31713. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit cd5fbcf) Signed-off-by: Dongjoon Hyun <[email protected]>

HyukjinKwon · 2020-05-15T02:29:44Z

Shall we file a JIRA and assign @shaneknapp? I think he asked this before :-).

dongjoon-hyun · 2020-05-15T02:30:19Z

I left a comment on the existing JIRA. But, new JIRA sounds good.

https://issues.apache.org/jira/browse/SPARK-31693?focusedCommentId=17107856&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17107856

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string …

e9bdb59

…correctly

probot-autolabeler bot added the BUILD label May 14, 2020

dongjoon-hyun commented May 14, 2020

View reviewed changes

Address comment

b96806a

dongjoon-hyun mentioned this pull request May 15, 2020

[SPARK-31387] Handle unknown operation/session ID in HiveThriftServer2Listener #28155

Closed

HyukjinKwon approved these changes May 15, 2020

View reviewed changes

dongjoon-hyun closed this in cd5fbcf May 15, 2020

dongjoon-hyun deleted the SPARK-31713 branch May 15, 2020 02:30

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

Uh oh!

Conversation

dongjoon-hyun commented May 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented May 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun May 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun May 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented May 15, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

srowen commented May 15, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

SparkQA commented May 15, 2020

Uh oh!

SparkQA commented May 15, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented May 15, 2020

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

HyukjinKwon commented May 15, 2020

Uh oh!

HyukjinKwon commented May 15, 2020

Uh oh!

dongjoon-hyun commented May 15, 2020

Uh oh!

Reviewers

Assignees

Labels

dongjoon-hyun commented May 14, 2020 •

edited

Loading

dongjoon-hyun commented May 14, 2020 •

edited

Loading

dongjoon-hyun May 14, 2020 •

edited

Loading

dongjoon-hyun May 15, 2020 •

edited

Loading

dongjoon-hyun commented May 15, 2020 •

edited

Loading

dongjoon-hyun commented May 15, 2020 •

edited

Loading

dongjoon-hyun commented May 15, 2020 •

edited

Loading

dongjoon-hyun commented May 15, 2020 •

edited

Loading