[SPARK-8821] [EC2] Switched to binary mode for file reading #7215

reactormonk · 2015-07-03T20:37:33Z

Otherwise the script will crash with

- Downloading boto...
Traceback (most recent call last):
  File "ec2/spark_ec2.py", line 148, in <module>
    setup_external_libs(external_libs)
  File "ec2/spark_ec2.py", line 128, in setup_external_libs
    if hashlib.md5(tar.read()).hexdigest() != lib["md5"]:
  File "/usr/lib/python3.4/codecs.py", line 319, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

In case of an utf8 env setting.

AmplabJenkins · 2015-07-03T20:38:09Z

Can one of the admins verify this patch?

shivaram · 2015-07-03T23:52:41Z

Could you open a JIRA for this ? See https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark for more details

cc @nchammas

reactormonk · 2015-07-04T01:00:21Z

Looks too trivial to jump through the hoops of JIRA.

JoshRosen · 2015-07-04T01:25:53Z

I think the motivations for JIRA tickets are:

JIRA helps us track where a fix has been applied; this is important if a fix needs to be applied to multiple maintenance branches and it also helpful when a fix is reverted.
The contributor credits in our release notes are automatically generated from JIRA.

reactormonk · 2015-07-04T05:39:40Z

https://issues.apache.org/jira/browse/SPARK-8821

Otherwise the script will crash with - Downloading boto... Traceback (most recent call last): File "ec2/spark_ec2.py", line 148, in <module> setup_external_libs(external_libs) File "ec2/spark_ec2.py", line 128, in setup_external_libs if hashlib.md5(tar.read()).hexdigest() != lib["md5"]: File "/usr/lib/python3.4/codecs.py", line 319, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte In case of an utf8 env setting.

nchammas · 2015-07-06T16:43:01Z

LGTM

shivaram · 2015-07-06T16:49:12Z

Could you add [SPARK-8821] [EC2] to the PR title ?

JoshRosen · 2015-07-06T17:22:12Z

Jenkins, this is ok to test.

AmplabJenkins · 2015-07-06T17:23:12Z

Merged build triggered.

AmplabJenkins · 2015-07-06T17:23:20Z

Merged build started.

SparkQA · 2015-07-06T17:24:36Z

Test build #36590 has started for PR 7215 at commit e86957a.

SparkQA · 2015-07-06T18:56:09Z

Test build #36590 has finished for PR 7215 at commit e86957a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-06T18:56:20Z

Merged build finished. Test FAILed.

JoshRosen · 2015-07-06T20:20:06Z

Jenkins, retest this please.

AmplabJenkins · 2015-07-06T20:23:12Z

Merged build triggered.

AmplabJenkins · 2015-07-06T20:23:22Z

Merged build started.

SparkQA · 2015-07-06T20:24:40Z

Test build #36601 has started for PR 7215 at commit e86957a.

SparkQA · 2015-07-06T21:48:58Z

Test build #36601 has finished for PR 7215 at commit e86957a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-06T21:49:11Z

Merged build finished. Test FAILed.

reactormonk · 2015-07-06T23:26:33Z

There doesn't even seem to be any reference to ec2 in the test output.

shivaram · 2015-07-06T23:28:18Z

Jenkins, retest this please

AmplabJenkins · 2015-07-06T23:33:12Z

Merged build triggered.

AmplabJenkins · 2015-07-06T23:33:17Z

Merged build started.

SparkQA · 2015-07-06T23:36:31Z

Test build #36617 has started for PR 7215 at commit e86957a.

SparkQA · 2015-07-07T00:58:06Z

Test build #36617 has finished for PR 7215 at commit e86957a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-07T00:58:18Z

Merged build finished. Test FAILed.

srowen · 2015-07-07T08:29:12Z

Yeah it's not related. You can see the failure is related to SQL. I don't think ec2 is tested. In fact I think this whole bit is moving out of apache/spark? So, LGTM

shivaram · 2015-07-07T16:41:21Z

Yeah EC2 is not tested by jenkins -- Merging this

Otherwise the script will crash with - Downloading boto... Traceback (most recent call last): File "ec2/spark_ec2.py", line 148, in <module> setup_external_libs(external_libs) File "ec2/spark_ec2.py", line 128, in setup_external_libs if hashlib.md5(tar.read()).hexdigest() != lib["md5"]: File "/usr/lib/python3.4/codecs.py", line 319, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte In case of an utf8 env setting. Author: Simon Hafner <[email protected]> Closes #7215 from reactormonk/branch-1.4 and squashes the following commits: e86957a [Simon Hafner] [SPARK-8821] [EC2] Switched to binary mode

reactormonk changed the title ~~Switched to binary mode for file reading~~ [SPARK-8821] [EC2] Switched to binary mode for file reading Jul 6, 2015

asfgit closed this in 70beb80 Jul 7, 2015

reactormonk deleted the branch-1.4 branch July 7, 2015 18:10

[SPARK-8821] [EC2] Switched to binary mode for file reading #7215

[SPARK-8821] [EC2] Switched to binary mode for file reading #7215

Uh oh!

Conversation

reactormonk commented Jul 3, 2015

Uh oh!

AmplabJenkins commented Jul 3, 2015

Uh oh!

shivaram commented Jul 3, 2015

Uh oh!

reactormonk commented Jul 4, 2015

Uh oh!

JoshRosen commented Jul 4, 2015

Uh oh!

reactormonk commented Jul 4, 2015

Uh oh!

nchammas commented Jul 6, 2015

Uh oh!

shivaram commented Jul 6, 2015

Uh oh!

JoshRosen commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

JoshRosen commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

reactormonk commented Jul 6, 2015

Uh oh!

shivaram commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

AmplabJenkins commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 6, 2015

Uh oh!

SparkQA commented Jul 7, 2015

Uh oh!

AmplabJenkins commented Jul 7, 2015

Uh oh!

srowen commented Jul 7, 2015

Uh oh!

shivaram commented Jul 7, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants