Skip to content

Conversation

@ueshin
Copy link
Member

@ueshin ueshin commented Oct 18, 2016

What changes were proposed in this pull request?

SerializationUtils.clone() of commons-lang3 (<3.5) has a bug that breaks thread safety, which gets stack sometimes caused by race condition of initializing hash map.
See https://issues.apache.org/jira/browse/LANG-1251.

How was this patch tested?

Existing tests.

@SparkQA
Copy link

SparkQA commented Oct 18, 2016

Test build #67107 has finished for PR 15525 at commit f318dff.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ueshin
Copy link
Member Author

ueshin commented Oct 18, 2016

Jenkins, retest this please.

@srowen
Copy link
Member

srowen commented Oct 18, 2016

I think this is OK. The general issue is whether it will be compatible with other transitive dependencies, but I believe it would remain so.

@SparkQA
Copy link

SparkQA commented Oct 18, 2016

Test build #67111 has finished for PR 15525 at commit f318dff.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Oct 18, 2016

Merging in master.

@asfgit asfgit closed this in bfe7885 Oct 18, 2016
@rxin
Copy link
Contributor

rxin commented Oct 18, 2016

Actually I'm seeing the following exceptions locally as well as on Jenkins for Hadoop 2.2:

[error] /scratch/rxin/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:1489: value read is not a member of object org.apache.commons.io.IOUtils
[error]       var numBytes = IOUtils.read(gzInputStream, buf)
[error]                              ^
[error] /scratch/rxin/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:1492: value read is not a member of object org.apache.commons.io.IOUtils
[error]         numBytes = IOUtils.read(gzInputStream, buf)
[error]                            ^

I'm going to revert the patch for now.

@ueshin
Copy link
Member Author

ueshin commented Oct 19, 2016

@rxin I think the failure is not related to this PR.
The builds for hadoop-2.2 are still failing after reverted.
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.2/1948/console

Looks IOUtils is introduced by 231f39e (#15285) and fixing is in progress at #15537.

@rxin
Copy link
Contributor

rxin commented Oct 19, 2016

Ah alright thanks. Can you submit a new pr so I can merge it again?

@ueshin
Copy link
Member Author

ueshin commented Oct 19, 2016

I see. I'll submit soon.

@ueshin
Copy link
Member Author

ueshin commented Oct 19, 2016

@rxin I submitted a new pr #15548. Thanks.

robert3005 pushed a commit to palantir/spark that referenced this pull request Nov 1, 2016
## What changes were proposed in this pull request?

`SerializationUtils.clone()` of commons-lang3 (<3.5) has a bug that breaks thread safety, which gets stack sometimes caused by race condition of initializing hash map.
See https://issues.apache.org/jira/browse/LANG-1251.

## How was this patch tested?

Existing tests.

Author: Takuya UESHIN <[email protected]>

Closes apache#15525 from ueshin/issues/SPARK-17985.
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
## What changes were proposed in this pull request?

`SerializationUtils.clone()` of commons-lang3 (<3.5) has a bug that breaks thread safety, which gets stack sometimes caused by race condition of initializing hash map.
See https://issues.apache.org/jira/browse/LANG-1251.

## How was this patch tested?

Existing tests.

Author: Takuya UESHIN <[email protected]>

Closes apache#15525 from ueshin/issues/SPARK-17985.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants