Skip to content

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Feb 27, 2019

What changes were proposed in this pull request?

After upgrading cloudpickle to 0.6.1 at #20691, one regression was found. Cloudpickle had a critical cloudpipe/cloudpickle#240 for that.

Basically, it currently looks existing globals would override globals shipped in a function's, meaning:

Before:

>>> def hey():
...     return "Hi"
...
>>> spark.range(1).rdd.map(lambda _: hey()).collect()
['Hi']
>>> def hey():
...     return "Yeah"
...
>>> spark.range(1).rdd.map(lambda _: hey()).collect()
['Hi']

After:

>>> def hey():
...     return "Hi"
...
>>> spark.range(1).rdd.map(lambda _: hey()).collect()
['Hi']
>>>
>>> def hey():
...     return "Yeah"
...
>>> spark.range(1).rdd.map(lambda _: hey()).collect()
['Yeah']

Therefore, this PR upgrades cloudpickle to 0.8.0.

Note that cloudpickle's release cycle is quite short.

Between 0.6.1 and 0.7.0, it contains minor bug fixes. I don't see notable changes to double check and/or avoid.

There is virtually only this fix between 0.7.0 and 0.8.1 - other fixes are about testing.

How was this patch tested?

Manually tested, tests were added. Verified unit tests were added in cloudpickle.

@HyukjinKwon
Copy link
Member Author

@SparkQA
Copy link

SparkQA commented Feb 27, 2019

Test build #102812 has finished for PR 23904 at commit 3e0afd1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 27, 2019

Test build #102811 has finished for PR 23904 at commit 746051e.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

retest this please

@SparkQA
Copy link

SparkQA commented Feb 27, 2019

Test build #102818 has finished for PR 23904 at commit 3e0afd1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@srowen srowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like an important fix to pick up.

global global_func
self.assertEqual(self.sc.parallelize([1]).map(lambda _: global_func()).first(), "Hi")
global_func = lambda: "Yeah"
self.assertEqual(self.sc.parallelize([1]).map(lambda _: global_func()).first(), "Yeah")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran a test locally:

>>> global_func = lambda: "Hi"
>>> sc.parallelize([1]).map(lambda _: global_func()).first()
'Hi'
>>> global_func = lambda: "Yeah"
>>> sc.parallelize([1]).map(lambda _: global_func()).first()
'Hi'
>>> sc.parallelize([1]).map(lambda _: global_func()).first()
'Yeah'
>>> sc.parallelize([1]).map(lambda _: global_func()).first()
'Hi'

Seems it outputs Hi or Yeah randomly. Is it caused by this cloudpickle issue?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it looks so. After the fix, the result becomes consistent:

>>> global_func = lambda: "Hi"
>>> sc.parallelize([1]).map(lambda _: global_func()).first()
'Hi'
>>> global_func = lambda: "Yeah"
>>> sc.parallelize([1]).map(lambda _: global_func()).first()
'Yeah'
>>> sc.parallelize([1]).map(lambda _: global_func()).first()
'Yeah'
>>> sc.parallelize([1]).map(lambda _: global_func()).first()
'Yeah'

@HyukjinKwon
Copy link
Member Author

Let me get this in. The pickle protocol change at #20691 was closely reviewed and I think it's still important change to keep than revert for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants