Skip to content

[SPARK-25682][k8s] Package example jars in same target for dev and distro images. #22681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-25682][k8s] Package example jars in same target for dev and distro images. #22681

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
97e0fe2
[SPARK-25682][k8s] Package example jars in same target for dev and di…
Oct 9, 2018
1e0c081
Add examples srcs to image.
Oct 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions bin/docker-image-tool.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,11 @@ function build {

if [ ! -f "$SPARK_HOME/RELEASE" ]; then
# Set image build arguments accordingly if this is a source repo and not a distribution archive.
#
# Note that this will copy all of the example jars directory into the image, and that will
# contain a lot of duplicated jars with the main Spark directory. In a proper distribution,
# the examples directory is cleaned up before generating the distribution tarball, so this
# issue does not occur.
IMG_PATH=resource-managers/kubernetes/docker/src/main/dockerfiles
BUILD_ARGS=(
${BUILD_PARAMS}
Expand All @@ -55,6 +60,8 @@ function build {
--build-arg
spark_jars=assembly/target/scala-$SPARK_SCALA_VERSION/jars
--build-arg
example_jars=examples/target/scala-$SPARK_SCALA_VERSION/jars
--build-arg
k8s_tests=resource-managers/kubernetes/integration-tests/tests
)
else
Expand All @@ -78,14 +85,23 @@ function build {
docker build $NOCACHEARG "${BUILD_ARGS[@]}" \
-t $(image_ref spark) \
-f "$BASEDOCKERFILE" .
if [[ $? != 0 ]]; then
error "Failed to build Spark docker image."
fi

docker build $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" \
-t $(image_ref spark-py) \
-f "$PYDOCKERFILE" .
if [[ $? != 0 ]]; then
error "Failed to build PySpark docker image."
fi

docker build $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" \
-t $(image_ref spark-r) \
-f "$RDOCKERFILE" .
if [[ $? != 0 ]]; then
error "Failed to build SparkR docker image."
fi
}

function push {
Expand Down
5 changes: 4 additions & 1 deletion resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
FROM openjdk:8-alpine

ARG spark_jars=jars
ARG example_jars=examples/jars
Copy link
Member

@felixcheung felixcheung Oct 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we make this optional? if someone wants to build a smaller image without example

Copy link
Contributor Author

@vanzin vanzin Oct 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Different change? ATM the existing code does not make this optional.

Copy link
Member

@rvesse rvesse Oct 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the K8S integration tests currently assume that the examples JAR is present since they rely upon running code from the examples

ARG img_path=kubernetes/dockerfiles
ARG k8s_tests=kubernetes/tests

Expand All @@ -32,6 +33,7 @@ RUN set -ex && \
apk upgrade --no-cache && \
apk add --no-cache bash tini libc6-compat linux-pam && \
mkdir -p /opt/spark && \
mkdir -p /opt/spark/examples && \
mkdir -p /opt/spark/work-dir && \
touch /opt/spark/RELEASE && \
rm /bin/sh && \
Expand All @@ -43,7 +45,8 @@ COPY ${spark_jars} /opt/spark/jars
COPY bin /opt/spark/bin
COPY sbin /opt/spark/sbin
COPY ${img_path}/spark/entrypoint.sh /opt/
COPY examples /opt/spark/examples
COPY ${example_jars} /opt/spark/examples/jars
COPY examples/src /opt/spark/examples/src
COPY ${k8s_tests} /opt/spark/tests
COPY data /opt/spark/data

Expand Down