Skip to content

SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 16 commits into from
Closed

SPARK-9545, SPARK-9547: Use Maven in PRB if title contains "[test-maven]" #7878

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
6184a00
SPARK-9545: Use Maven in prb if title contains "[maven-test]"
pwendell Aug 2, 2015
a15f53c
Wording change to test-maven
pwendell Aug 2, 2015
e83b49f
Adding support for SPARK-9547
pwendell Aug 2, 2015
5d15d31
Adding randomized zinc ports
pwendell Aug 2, 2015
7f1ff75
Fixing up python lint issues
pwendell Aug 2, 2015
c77f951
Small style fix
pwendell Aug 2, 2015
69dd9af
Push Zinc support down into exec_maven
pwendell Aug 2, 2015
0d3944a
Small fixes
pwendell Aug 2, 2015
05d7720
Bug fix to kill command
pwendell Aug 3, 2015
3d36d0a
Temporarily disabling mima
pwendell Aug 3, 2015
2e9edcc
Increasing timeout
pwendell Aug 3, 2015
acf1e4f
Merge remote-tracking branch 'apache/master' into maven-tests
pwendell Aug 4, 2015
a938de3
Only do compatibility tests in sbt
pwendell Aug 4, 2015
603f891
removing unncessary spacing changes
pwendell Aug 4, 2015
87d4d19
Whitepsace issues
pwendell Aug 4, 2015
cf58c49
Merge remote-tracking branch 'apache/master' into maven-tests
pwendell Aug 27, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions dev/run-tests-jenkins
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,9 @@ pr_message=""
current_pr_head="`git rev-parse HEAD`"

echo "HEAD: `git rev-parse HEAD`"
echo "GHPRB: $ghprbActualCommit"
echo "SHA1: $sha1"
echo "\$ghprbActualCommit: $ghprbActualCommit"
echo "\$sha1: $sha1"
echo "\$ghprbPullTitle: $ghprbPullTitle"

# Run pull request tests
for t in "${PR_TESTS[@]}"; do
Expand All @@ -189,6 +190,19 @@ done
{
# Marks this build is a pull request build.
export AMP_JENKINS_PRB=true
if [[ $ghprbPullTitle == *"test-maven"* ]]; then
export AMPLAB_JENKINS_BUILD_TOOL="maven"
fi
if [[ $ghprbPullTitle == *"test-hadoop1.0"* ]]; then
export AMPLAB_JENKINS_BUILD_PROFILE="hadoop1.0"
elif [[ $ghprbPullTitle == *"test-hadoop2.0"* ]]; then
export AMPLAB_JENKINS_BUILD_PROFILE="hadoop2.0"
elif [[ $ghprbPullTitle == *"test-hadoop2.2"* ]]; then
export AMPLAB_JENKINS_BUILD_PROFILE="hadoop2.2"
elif [[ $ghprbPullTitle == *"test-hadoop2.3"* ]]; then
export AMPLAB_JENKINS_BUILD_PROFILE="hadoop2.3"
fi

timeout "${TESTS_TIMEOUT}" ./dev/run-tests
test_result="$?"

Expand Down
28 changes: 26 additions & 2 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import itertools
from optparse import OptionParser
import os
import random
import re
import sys
import subprocess
Expand Down Expand Up @@ -227,11 +228,32 @@ def build_spark_documentation():
os.chdir(SPARK_HOME)


def get_zinc_port():
"""
Get a randomized port on which to start Zinc
"""
return random.randrange(3030, 4030)
Copy link
Contributor

@vanzin vanzin Aug 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope this doesn't lead to builds failing because zinc can't bind to the chosen port.

Copy link
Contributor Author

@pwendell pwendell Aug 31, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is identical to that hard coded in the bash scripts that run the maven builds, in the past they've never failed for this reason... but yeah it could happen.



def kill_zinc_on_port(zinc_port):
"""
Kill the Zinc process running on the given port, if one exists.
"""
cmd = ("/usr/sbin/lsof -P |grep %s | grep LISTEN "
"| awk '{ print $2; }' | xargs kill") % zinc_port
subprocess.check_call(cmd, shell=True)


def exec_maven(mvn_args=()):
"""Will call Maven in the current directory with the list of mvn_args passed
in and returns the subprocess for any further processing"""

run_cmd([os.path.join(SPARK_HOME, "build", "mvn")] + mvn_args)
zinc_port = get_zinc_port()
os.environ["ZINC_PORT"] = "%s" % zinc_port
zinc_flag = "-DzincPort=%s" % zinc_port
flags = [os.path.join(SPARK_HOME, "build", "mvn"), "--force", zinc_flag]
run_cmd(flags + mvn_args)
kill_zinc_on_port(zinc_port)


def exec_sbt(sbt_args=()):
Expand Down Expand Up @@ -495,7 +517,9 @@ def main():
build_apache_spark(build_tool, hadoop_version)

# backwards compatibility checks
detect_binary_inop_with_mima()
if build_tool == "sbt":
# Note: compatiblity tests only supported in sbt for now
detect_binary_inop_with_mima()

# run the test suites
run_scala_tests(build_tool, hadoop_version, test_modules)
Expand Down