Skip to content

Commit a06768e

Browse files
committed
[SPARK-31780][K8S][TESTS] Add R test tag to exclude R K8s image building and test
### What changes were proposed in this pull request? This PR aims to skip R image building and one R test during integration tests by using `--exclude-tags r`. ### Why are the changes needed? We have only one R integration test case, `Run SparkR on simple dataframe.R example`, for submission test coverage. Since this is rarely changed, we can skip this and save the efforts required for building the whole R image and running the single test. ``` KubernetesSuite: ... - Run SparkR on simple dataframe.R example Run completed in 10 minutes, 20 seconds. Total number of tests run: 20 ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the K8S integration test and do the following manually. (Note that R test is skipped) ``` $ resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --deploy-mode docker-for-desktop --exclude-tags r --spark-tgz $PWD/spark-*.tgz ... KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage - Launcher client dependencies - Test basic decommissioning Run completed in 10 minutes, 23 seconds. Total number of tests run: 19 Suites: completed 2, aborted 0 Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #28594 from dongjoon-hyun/SPARK-31780. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 26bc690 commit a06768e

File tree

4 files changed

+15
-3
lines changed

4 files changed

+15
-3
lines changed

resource-managers/kubernetes/integration-tests/pom.xml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,9 @@
121121

122122
<argument>--spark-tgz</argument>
123123
<argument>${spark.kubernetes.test.sparkTgz}</argument>
124+
125+
<argument>--test-exclude-tags</argument>
126+
<argument>"${test.exclude.tags}"</argument>
124127
</arguments>
125128
</configuration>
126129
</execution>

resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ IMAGE_TAG="N/A"
2626
JAVA_IMAGE_TAG="8-jre-slim"
2727
SPARK_TGZ="N/A"
2828
MVN="$TEST_ROOT_DIR/build/mvn"
29+
EXCLUDE_TAGS=""
2930

3031
# Parse arguments
3132
while (( "$#" )); do
@@ -58,6 +59,10 @@ while (( "$#" )); do
5859
SPARK_TGZ="$2"
5960
shift
6061
;;
62+
--test-exclude-tags)
63+
EXCLUDE_TAGS="$2"
64+
shift
65+
;;
6166
*)
6267
break
6368
;;
@@ -99,7 +104,10 @@ then
99104
LANGUAGE_BINDING_BUILD_ARGS="-p $DOCKER_FILE_BASE_PATH/bindings/python/Dockerfile"
100105

101106
# Build SparkR image
102-
LANGUAGE_BINDING_BUILD_ARGS="$LANGUAGE_BINDING_BUILD_ARGS -R $DOCKER_FILE_BASE_PATH/bindings/R/Dockerfile"
107+
tags=(${EXCLUDE_TAGS//,/ })
108+
if [[ ! ${tags[@]} =~ "r" ]]; then
109+
LANGUAGE_BINDING_BUILD_ARGS="$LANGUAGE_BINDING_BUILD_ARGS -R $DOCKER_FILE_BASE_PATH/bindings/R/Dockerfile"
110+
fi
103111

104112
# Unset SPARK_HOME to let the docker-image-tool script detect SPARK_HOME. Otherwise, it cannot
105113
# indicate the unpacked directory as its home. See SPARK-28550.

resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -476,6 +476,7 @@ class KubernetesSuite extends SparkFunSuite
476476

477477
private[spark] object KubernetesSuite {
478478
val k8sTestTag = Tag("k8s")
479+
val rTestTag = Tag("r")
479480
val MinikubeTag = Tag("minikube")
480481
val SPARK_PI_MAIN_CLASS: String = "org.apache.spark.examples.SparkPi"
481482
val SPARK_DFS_READ_WRITE_TEST = "org.apache.spark.examples.DFSReadWriteTest"

resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/RTestsSuite.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,9 @@ package org.apache.spark.deploy.k8s.integrationtest
1919
private[spark] trait RTestsSuite { k8sSuite: KubernetesSuite =>
2020

2121
import RTestsSuite._
22-
import KubernetesSuite.k8sTestTag
22+
import KubernetesSuite.{k8sTestTag, rTestTag}
2323

24-
test("Run SparkR on simple dataframe.R example", k8sTestTag) {
24+
test("Run SparkR on simple dataframe.R example", k8sTestTag, rTestTag) {
2525
sparkAppConf.set("spark.kubernetes.container.image", rImage)
2626
runSparkApplicationAndVerifyCompletion(
2727
appResource = SPARK_R_DATAFRAME_TEST,

0 commit comments

Comments
 (0)