[SPARK-31231][BUILD] Explicitly setuptools version as 46.0.0 in pip package test #27995

HyukjinKwon · 2020-03-24T02:03:36Z

What changes were proposed in this pull request?

For a bit of background,
PIP packaging test started to fail (see this logs) as of setuptools 46.1.0 release. In pypa/setuptools#1424, they decided to don't keep the modes in package_data.

In PySpark pip installation, we keep the executable scripts in package_data

spark/python/setup.py

Lines 199 to 200 in fc4e56a

    
           'pyspark.bin': ['*'], 
        
           'pyspark.sbin': ['spark-config.sh', 'spark-daemon.sh',

, and expose their symbolic links as executable scripts.

So, the symbolic links (or copied scripts) executes the scripts copied from package_data, which doesn't have the executable permission in its mode:

/tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: Permission denied
/tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: exec: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: cannot execute: Permission denied

The current issue is being tracked at pypa/setuptools#2041

For what this PR proposes:
It sets the setuptools version in PR builder for now to unblock other PRs. This PR does not solve the issue yet. I will make a fix after monitoring pypa/setuptools#2041

Why are the changes needed?

It currently affects users who uses the latest setuptools. So, users seem unable to use PySpark with the latest setuptools. See also pypa/setuptools#2041 (comment)

Does this PR introduce any user-facing change?

It makes CI pass for now. No user-facing change yet.

How was this patch tested?

Jenkins will test.

SparkQA · 2020-03-24T02:20:05Z

Test build #120230 has finished for PR 27995 at commit db53aa5.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T02:53:48Z

Test build #120231 has finished for PR 27995 at commit 857cdb4.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T03:21:39Z

Test build #120232 has finished for PR 27995 at commit fa9b0e5.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T03:42:58Z

Test build #120234 has finished for PR 27995 at commit 37c32d1.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T05:01:11Z

Test build #120237 has finished for PR 27995 at commit aa5647f.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T05:23:30Z

Test build #120238 has finished for PR 27995 at commit 570a08e.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T06:48:39Z

Test build #120245 has finished for PR 27995 at commit cee6d51.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T07:26:59Z

Test build #120248 has finished for PR 27995 at commit a1fdad8.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T07:55:32Z

Test build #120252 has finished for PR 27995 at commit f5a60c6.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T08:21:24Z

Test build #120255 has finished for PR 27995 at commit 01a1cad.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-03-24T08:43:31Z

Test build #120256 has finished for PR 27995 at commit 0c83437.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-03-24T08:53:27Z

Okay .. it passed. I'll clean up and merge to unblock other PRs.

dev/run-pip-tests

HyukjinKwon · 2020-03-24T08:59:30Z

Merged to master, branch-3.0, and branch-2.4.

I am not going to resolve the JIRA yet - this PR is a temp workaround only for CI at this momoent.

…ackage test ### What changes were proposed in this pull request? For a bit of background, PIP packaging test started to fail (see [this logs](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120218/testReport/)) as of setuptools 46.1.0 release. In pypa/setuptools#1424, they decided to don't keep the modes in `package_data`. In PySpark pip installation, we keep the executable scripts in `package_data` https://github.com/apache/spark/blob/fc4e56a54c15e20baf085e6061d3d83f5ce1185d/python/setup.py#L199-L200, and expose their symbolic links as executable scripts. So, the symbolic links (or copied scripts) executes the scripts copied from `package_data`, which doesn't have the executable permission in its mode: ``` /tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: Permission denied /tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: exec: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: cannot execute: Permission denied ``` The current issue is being tracked at pypa/setuptools#2041 </br> For what this PR proposes: It sets the upper bound in PR builder for now to unblock other PRs. _This PR does not solve the issue yet. I will make a fix after monitoring https://github.com/pypa/setuptools/issues/2041_ ### Why are the changes needed? It currently affects users who uses the latest setuptools. So, _users seem unable to use PySpark with the latest setuptools._ See also pypa/setuptools#2041 (comment) ### Does this PR introduce any user-facing change? It makes CI pass for now. No user-facing change yet. ### How was this patch tested? Jenkins will test. Closes #27995 from HyukjinKwon/investigate-pip-packaging. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit c181c45) Signed-off-by: HyukjinKwon <[email protected]>

HyukjinKwon · 2020-03-24T09:04:11Z

@nchammas, @srowen, @dongjoon-hyun, @BryanCutler, FWIW, I think this is one example of the trade-off we discussed at #27928.

setuptools is dev-only dependency but seems it found a pretty critical issue. Currently seems users can't use pip-installed PySpark assuming from pypa/setuptools#2041. However, of course it broke the PR builders.

SparkQA · 2020-03-24T14:30:17Z

Test build #120259 has finished for PR 27995 at commit b8926bd.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

srowen · 2020-03-24T15:13:38Z

Was there an issue in 46.1.0? wouldn't pinning to 46.0.0 have been a good thing then? or else I misunderstand what broke.

dongjoon-hyun · 2020-03-24T18:55:16Z

Hi, @HyukjinKwon .
This broke branch-2.4 due to the following. I'll revert this from branch-2.4.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.4-test-sbt-hadoop-2.6/804/

UnsatisfiableError: The following specifications were found to be in conflict:
  - python=3.5
  - setuptools=46.0.0 -> python[version='>=3.8,<3.9.0a0'] -> openssl[version='>=1.1.1d,<1.1.2a']
Use "conda search <package> --info" to see the dependencies for each package.

dongjoon-hyun · 2020-03-24T19:00:27Z

Since Apache Spark 2.4 still supports Python 2.7+/3.4+, setuptools 46.0.0 seems to be incompatible to that. Please make a separate PR for branch-2.4.

nchammas · 2020-03-24T19:11:52Z

Perhaps the requirement should instead be of the form setuptools < 46.1.0, to allow flexibility for older versions of Python to still find a release of setuptools that satisfies their requirements.

HyukjinKwon · 2020-03-24T23:30:20Z

Yeah, maybe I should better set upperbound instead. Thanks @dongjoon-hyun and @nchammas

HyukjinKwon · 2020-03-24T23:31:33Z

@srowen, seems this can be the problem in client side with the latest setuptools (see pypa/setuptools#2041). So, I think using lower version now fixed the test but the issue itself persists.

srowen · 2020-03-25T07:09:34Z

I'm still confused. This is an argument for pinning versions, right? Because a new version of setuptools caused a problem:

HyukjinKwon · 2020-03-25T09:28:33Z

But we found that issue from the broken PR builder, which seems to be pretty critical.

HyukjinKwon · 2020-03-25T09:29:23Z

I simply meant this is an example of the trade-off. It caused a problem and broke the PR builder vs it found a problem early.

…or setuptools in pip package test ## What changes were proposed in this pull request? This PR is a followup of #27995. Rather then pining setuptools version, it sets upper bound so Python 3.5 with branch-2.4 tests can pass too. ## Why are the changes needed? To make the CI build stable ## Does this PR introduce any user-facing change? No, dev-only change. ## How was this patch tested? Jenkins will test. Closes #28005 from HyukjinKwon/investigate-pip-packaging-followup. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

…or setuptools in pip package test ## What changes were proposed in this pull request? This PR is a followup of #27995. Rather then pining setuptools version, it sets upper bound so Python 3.5 with branch-2.4 tests can pass too. ## Why are the changes needed? To make the CI build stable ## Does this PR introduce any user-facing change? No, dev-only change. ## How was this patch tested? Jenkins will test. Closes #28005 from HyukjinKwon/investigate-pip-packaging-followup. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 178d472) Signed-off-by: HyukjinKwon <[email protected]>

…or setuptools in pip package test This PR is a followup of #27995. Rather then pining setuptools version, it sets upper bound so Python 3.5 with branch-2.4 tests can pass too. To make the CI build stable No, dev-only change. Jenkins will test. Closes #28005 from HyukjinKwon/investigate-pip-packaging-followup. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 178d472) Signed-off-by: HyukjinKwon <[email protected]>

…ackage test ### What changes were proposed in this pull request? For a bit of background, PIP packaging test started to fail (see [this logs](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120218/testReport/)) as of setuptools 46.1.0 release. In pypa/setuptools#1424, they decided to don't keep the modes in `package_data`. In PySpark pip installation, we keep the executable scripts in `package_data` https://github.com/apache/spark/blob/fc4e56a54c15e20baf085e6061d3d83f5ce1185d/python/setup.py#L199-L200, and expose their symbolic links as executable scripts. So, the symbolic links (or copied scripts) executes the scripts copied from `package_data`, which doesn't have the executable permission in its mode: ``` /tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: Permission denied /tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: exec: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: cannot execute: Permission denied ``` The current issue is being tracked at pypa/setuptools#2041 </br> For what this PR proposes: It sets the upper bound in PR builder for now to unblock other PRs. _This PR does not solve the issue yet. I will make a fix after monitoring https://github.com/pypa/setuptools/issues/2041_ ### Why are the changes needed? It currently affects users who uses the latest setuptools. So, _users seem unable to use PySpark with the latest setuptools._ See also pypa/setuptools#2041 (comment) ### Does this PR introduce any user-facing change? It makes CI pass for now. No user-facing change yet. ### How was this patch tested? Jenkins will test. Closes apache#27995 from HyukjinKwon/investigate-pip-packaging. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

…or setuptools in pip package test ## What changes were proposed in this pull request? This PR is a followup of apache#27995. Rather then pining setuptools version, it sets upper bound so Python 3.5 with branch-2.4 tests can pass too. ## Why are the changes needed? To make the CI build stable ## Does this PR introduce any user-facing change? No, dev-only change. ## How was this patch tested? Jenkins will test. Closes apache#28005 from HyukjinKwon/investigate-pip-packaging-followup. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

Investigate PIP package failure

db53aa5

show permissions

857cdb4

typo

fa9b0e5

typo :(

37c32d1

HyukjinKwon force-pushed the investigate-pip-packaging branch from cee6d51 to a1fdad8 Compare March 24, 2020 07:12

set upperbound

f5a60c6

HyukjinKwon force-pushed the investigate-pip-packaging branch from a1fdad8 to f5a60c6 Compare March 24, 2020 07:37

HyukjinKwon changed the title ~~[DO-NOT-MERGE] Investigate PIP package failure~~ [SPARK-31231][BUILD] Set the upper bound (before 46.1.0) for setuptools in pip package test Mar 24, 2020

Explicitly set

01a1cad

HyukjinKwon changed the title ~~[SPARK-31231][BUILD] Set the upper bound (before 46.1.0) for setuptools in pip package test~~ [SPARK-31231][BUILD] Explicitly setuptools version as 46.0.0 in pip package test Mar 24, 2020

Remove comment

0c83437

cloud-fan reviewed Mar 24, 2020

View reviewed changes

dev/run-pip-tests Outdated Show resolved Hide resolved

Revert other changes

b8926bd

HyukjinKwon closed this in c181c45 Mar 24, 2020

HyukjinKwon mentioned this pull request Mar 25, 2020

[SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test #28005

Closed

HyukjinKwon deleted the investigate-pip-packaging branch July 27, 2020 07:45

	'pyspark.bin': ['*'],
	'pyspark.sbin': ['spark-config.sh', 'spark-daemon.sh',

[SPARK-31231][BUILD] Explicitly setuptools version as 46.0.0 in pip package test #27995

[SPARK-31231][BUILD] Explicitly setuptools version as 46.0.0 in pip package test #27995

Uh oh!

Conversation

HyukjinKwon commented Mar 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

HyukjinKwon commented Mar 24, 2020

Uh oh!

Uh oh!

HyukjinKwon commented Mar 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Mar 24, 2020

Uh oh!

SparkQA commented Mar 24, 2020

Uh oh!

srowen commented Mar 24, 2020

Uh oh!

dongjoon-hyun commented Mar 24, 2020

Uh oh!

dongjoon-hyun commented Mar 24, 2020

Uh oh!

nchammas commented Mar 24, 2020

Uh oh!

HyukjinKwon commented Mar 24, 2020

Uh oh!

HyukjinKwon commented Mar 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Mar 25, 2020

Uh oh!

HyukjinKwon commented Mar 25, 2020

Uh oh!

HyukjinKwon commented Mar 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

HyukjinKwon commented Mar 24, 2020 •

edited

Loading

HyukjinKwon commented Mar 24, 2020 •

edited

Loading

HyukjinKwon commented Mar 24, 2020 •

edited

Loading