Skip to content

Commit ca4898e

Browse files
Brian Urizzcclp
authored andcommitted
[SPARK-16257][BUILD] Update spark_ec2.py to support Spark 1.6.2 and 1.6.3.
## What changes were proposed in this pull request? - Adds 1.6.2 and 1.6.3 as supported Spark versions within the bundled spark-ec2 script. - Makes the default Spark version 1.6.3 to keep in sync with the upcoming release. - Does not touch the newer spark-ec2 scripts in the separate amplabs repository. ## How was this patch tested? - Manual script execution: export AWS_SECRET_ACCESS_KEY=_snip_ export AWS_ACCESS_KEY_ID=_snip_ $SPARK_HOME/ec2/spark-ec2 \ --key-pair=_snip_ \ --identity-file=_snip_ \ --region=us-east-1 \ --vpc-id=_snip_ \ --slaves=1 \ --instance-type=t1.micro \ --spark-version=1.6.2 \ --hadoop-major-version=yarn \ launch test-cluster - Result: Successful creation of a 1.6.2-based Spark cluster. This contribution is my original work and I license the work to the project under the project's open source license. Author: Brian Uri <[email protected]> Closes apache#13947 from briuri/branch-1.6-bug-spark-16257. (cherry picked from commit ccc7fa3)
1 parent 46bb68e commit ca4898e

File tree

1 file changed

+6
-2
lines changed

1 file changed

+6
-2
lines changed

ec2/spark_ec2.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@
5151
raw_input = input
5252
xrange = range
5353

54-
SPARK_EC2_VERSION = "1.6.1"
54+
SPARK_EC2_VERSION = "1.6.3"
5555
SPARK_EC2_DIR = os.path.dirname(os.path.realpath(__file__))
5656

5757
VALID_SPARK_VERSIONS = set([
@@ -77,6 +77,8 @@
7777
"1.5.2",
7878
"1.6.0",
7979
"1.6.1",
80+
"1.6.2",
81+
"1.6.3",
8082
])
8183

8284
SPARK_TACHYON_MAP = {
@@ -96,14 +98,16 @@
9698
"1.5.2": "0.7.1",
9799
"1.6.0": "0.8.2",
98100
"1.6.1": "0.8.2",
101+
"1.6.2": "0.8.2",
102+
"1.6.3": "0.8.2",
99103
}
100104

101105
DEFAULT_SPARK_VERSION = SPARK_EC2_VERSION
102106
DEFAULT_SPARK_GITHUB_REPO = "https://github.com/apache/spark"
103107

104108
# Default location to get the spark-ec2 scripts (and ami-list) from
105109
DEFAULT_SPARK_EC2_GITHUB_REPO = "https://github.com/amplab/spark-ec2"
106-
DEFAULT_SPARK_EC2_BRANCH = "branch-1.5"
110+
DEFAULT_SPARK_EC2_BRANCH = "branch-1.6"
107111

108112

109113
def setup_external_libs(libs):

0 commit comments

Comments
 (0)