Skip to content

Commit 1530bc0

Browse files
committed
[SPARK-31231][BUILD] Explicitly setuptools version as 46.0.0 in pip package test
### What changes were proposed in this pull request? For a bit of background, PIP packaging test started to fail (see [this logs](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120218/testReport/)) as of setuptools 46.1.0 release. In pypa/setuptools#1424, they decided to don't keep the modes in `package_data`. In PySpark pip installation, we keep the executable scripts in `package_data` https://github.com/apache/spark/blob/fc4e56a54c15e20baf085e6061d3d83f5ce1185d/python/setup.py#L199-L200, and expose their symbolic links as executable scripts. So, the symbolic links (or copied scripts) executes the scripts copied from `package_data`, which doesn't have the executable permission in its mode: ``` /tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: Permission denied /tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: exec: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: cannot execute: Permission denied ``` The current issue is being tracked at pypa/setuptools#2041 </br> For what this PR proposes: It sets the upper bound in PR builder for now to unblock other PRs. _This PR does not solve the issue yet. I will make a fix after monitoring https://github.com/pypa/setuptools/issues/2041_ ### Why are the changes needed? It currently affects users who uses the latest setuptools. So, _users seem unable to use PySpark with the latest setuptools._ See also pypa/setuptools#2041 (comment) ### Does this PR introduce any user-facing change? It makes CI pass for now. No user-facing change yet. ### How was this patch tested? Jenkins will test. Closes #27995 from HyukjinKwon/investigate-pip-packaging. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit c181c45) Signed-off-by: HyukjinKwon <[email protected]>
1 parent dad4d6a commit 1530bc0

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

dev/run-pip-tests

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ for python in "${PYTHON_EXECS[@]}"; do
7676
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
7777
rm -rf "$VIRTUALENV_PATH"
7878
if [ -n "$USE_CONDA" ]; then
79-
conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
79+
conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools=46.0.0
8080
source activate "$VIRTUALENV_PATH"
8181
else
8282
mkdir -p "$VIRTUALENV_PATH"

0 commit comments

Comments
 (0)