Skip to content

[SPARK-43171][K8S] Support custom Unix username in Pod #40831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

[SPARK-43171][K8S] Support custom Unix username in Pod #40831

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
20f1aff
[SPARK-43171][K8S] Support dynamic changing unix user in Pod
pan3793 Apr 18, 2023
bec5f3f
IT
pan3793 Apr 28, 2023
049a984
Revert "IT"
pan3793 Apr 28, 2023
6fecd5c
IT 2
pan3793 Apr 28, 2023
3ee42bb
nit
pan3793 Apr 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,14 @@ set -e
# If there is no passwd entry for the container UID, attempt to create one
if [ -z "$uidentry" ] ; then
if [ -w /etc/passwd ] ; then
echo "$myuid:x:$myuid:$mygid:${SPARK_USER_NAME:-anonymous uid}:$SPARK_HOME:/bin/false" >> /etc/passwd
echo "${SPARK_USER_NAME:-$myuid}:x:$myuid:$mygid:${SPARK_USER_NAME:-anonymous uid}:$SPARK_HOME:/bin/false" >> /etc/passwd
else
echo "Container ENTRYPOINT failed to add passwd entry for anonymous UID"
fi
fi

echo "login user is `id -un`"

if [ -z "$JAVA_HOME" ]; then
JAVA_HOME=$(java -XshowSettings:properties -version 2>&1 > /dev/null | grep 'java.home' | awk '{print $3}')
fi
Expand Down
18 changes: 17 additions & 1 deletion ...on-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import org.scalatest.matchers.should.Matchers._
import org.scalatest.time.{Seconds, Span}

import org.apache.spark.{SparkFunSuite, TestUtils}
import org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.SPARK_PI_MAIN_CLASS
import org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.{usernameTestTag, SPARK_PI_MAIN_CLASS}
import org.apache.spark.launcher.SparkLauncher

private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
Expand Down Expand Up @@ -168,6 +168,22 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
.exists(envVar => envVar.getName == "SPARK_DRIVER_POD_IP")
})
}

test("SPARK-43171: Support custom Unix username in Pod", k8sTestTag, usernameTestTag) {
sparkAppConf
.set("spark.kubernetes.driverEnv.SPARK_USER_NAME", "spark-kube")
.set("spark.executorEnv.SPARK_USER_NAME", "spark-kube")
runSparkApplicationAndVerifyCompletion(
containerLocalSparkDistroExamplesJar,
SPARK_PI_MAIN_CLASS,
Seq("login user is spark-kube", "Pi is roughly 3"),
Seq("login user is spark-kube"),
appArgs = Array.empty,
doBasicDriverPodCheck,
doBasicExecutorPodCheck,
isJVM = true
)
}
}

private[spark] object BasicTestsSuite extends SparkFunSuite {
Expand Down
1 change: 1 addition & 0 deletions ...on-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -623,6 +623,7 @@ private[spark] object KubernetesSuite {
val decomTestTag = Tag("decom")
val rTestTag = Tag("r")
val MinikubeTag = Tag("minikube")
val usernameTestTag = Tag("username")
Copy link
Member Author

@pan3793 pan3793 Apr 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Yikun A new tag is added, you can use it to exclude this test for apache/spark-docker GA

val SPARK_PI_MAIN_CLASS: String = "org.apache.spark.examples.SparkPi"
val SPARK_DFS_READ_WRITE_TEST = "org.apache.spark.examples.DFSReadWriteTest"
val SPARK_MINI_READ_WRITE_TEST = "org.apache.spark.examples.MiniReadWriteTest"
Expand Down