Skip to content

[SPARK-26083][k8s] Add Copy pyspark into corresponding dir cmd in pyspark Dockerfile #23037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-26083][k8s] Add Copy pyspark into corresponding dir cmd in pyspark Dockerfile #23037

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c2f782b
Copy python/pyspark to ${SPARK_HOME}/python/pyspark to make bin/pyspa…
AzureQ Nov 14, 2018
4bf6bc6
Added test for client mode pyspark shell into PythonTestsSuite
AzureQ Nov 19, 2018
c626bd1
Correct pyspark shell test's driver image
AzureQ Nov 20, 2018
149bed4
Remove erroring test case for pyspark shell
AzureQ Nov 29, 2018
ff8d0ba
Merge remote-tracking branch 'upstream/master'
AzureQ Dec 1, 2018
5f6aaf8
Updated docker-image-tool.sh with copying pyspark
AzureQ Dec 1, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions bin/docker-image-tool.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ function create_dev_build_context {(
"$PYSPARK_CTX/kubernetes/dockerfiles"
mkdir "$PYSPARK_CTX/python"
cp -r "python/lib" "$PYSPARK_CTX/python/lib"
cp -r "python/pyspark" "$PYSPARK_CTX/python/pyspark"

local R_CTX="$CTX_DIR/sparkr"
mkdir -p "$R_CTX/kubernetes"
Expand Down
1 change: 1 addition & 0 deletions resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ RUN apk add --no-cache python && \
# Removed the .cache to save space
rm -r /root/.cache

COPY python/pyspark ${SPARK_HOME}/python/pyspark
COPY python/lib ${SPARK_HOME}/python/lib
ENV PYTHONPATH ${SPARK_HOME}/python/lib/pyspark.zip:${SPARK_HOME}/python/lib/py4j-*.zip

Expand Down