Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Reorganize packages between v1 work and v2 work #220

Merged
merged 40 commits into from
Apr 21, 2017
Merged

Reorganize packages between v1 work and v2 work #220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
71b825a
Staging server for receiving application dependencies.
mccheah Apr 3, 2017
e005213
Merge remote-tracking branch 'apache-spark-on-k8s/branch-2.1-kubernet…
mccheah Apr 3, 2017
929d880
Move packages around to split between v1 work and v2 work
mccheah Apr 4, 2017
6e40c4c
Add unit test for file writing
mccheah Apr 4, 2017
e4daf56
Merge branch 'submission-v2-file-server' into branch-packages
mccheah Apr 4, 2017
89c295e
Remove unnecessary main
mccheah Apr 5, 2017
876bb4a
Add back license header
mccheah Apr 6, 2017
7d00f07
Minor fixes
mccheah Apr 6, 2017
0597b4c
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 6, 2017
ea76823
Merge branch 'branch-2.1-kubernetes' into submission-v2-file-server
mccheah Apr 6, 2017
ac0b841
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 6, 2017
baf10aa
Fix integration test with renamed package for client. Fix scalastyle.
mccheah Apr 6, 2017
d20537d
Force json serialization to consider the different package.
mccheah Apr 10, 2017
2ca73a3
Revert extraneous log
mccheah Apr 10, 2017
ffe8c6c
Fix scalastyle
mccheah Apr 10, 2017
3dd3504
Remove getting credentials from the API
mccheah Apr 11, 2017
3cbcadf
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 11, 2017
9bfb085
Merge branch 'branch-2.1-kubernetes' into submission-v2-file-server
ash211 Apr 12, 2017
df8e0c8
Generalize to resource staging server outside of Spark
mccheah Apr 12, 2017
24452ec
Update code documentation
mccheah Apr 12, 2017
f597171
Val instead of var
mccheah Apr 12, 2017
c20e461
Merge remote-tracking branch 'apache-spark-on-k8s/submission-v2-file-…
mccheah Apr 12, 2017
47704f1
Merge remote-tracking branch 'apache-spark-on-k8s/submission-v2-file-…
mccheah Apr 12, 2017
2e6fe07
Fix build
mccheah Apr 12, 2017
e5f26aa
Fix naming, remove unused import
mccheah Apr 13, 2017
a09d935
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 13, 2017
c408ff9
Move suites from integration test package to core
mccheah Apr 13, 2017
928f3a5
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 13, 2017
a527847
Merge branch 'branch-2.1-kubernetes' into submission-v2-file-server
mccheah Apr 18, 2017
0f2f847
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 18, 2017
64eddc1
Use TrieMap instead of locks
mccheah Apr 19, 2017
8f79802
Address comments
mccheah Apr 19, 2017
31fd3b6
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 19, 2017
04099d6
Fix imports
mccheah Apr 19, 2017
cac485c
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 19, 2017
d713c27
Change paths, use POST instead of PUT
mccheah Apr 20, 2017
91df628
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 20, 2017
720c38d
Use a resource identifier as well as a resource secret
mccheah Apr 20, 2017
89b032e
Merge branch 'submission-v2-file-server' into split-v1-and-v2-submiss…
mccheah Apr 20, 2017
b6da23e
Merge branch 'branch-2.1-kubernetes' into split-v1-and-v2-submission-…
ash211 Apr 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -619,7 +619,7 @@ object SparkSubmit {
}

if (isKubernetesCluster) {
childMainClass = "org.apache.spark.deploy.kubernetes.Client"
childMainClass = "org.apache.spark.deploy.kubernetes.submit.v1.Client"
childArgs += args.primaryResource
childArgs += args.mainClass
childArgs ++= args.childArgs
Expand Down
2 changes: 1 addition & 1 deletion dev/.rat-excludes
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,4 +103,4 @@ org.apache.spark.scheduler.ExternalClusterManager
org.apache.spark.deploy.yarn.security.ServiceCredentialProvider
spark-warehouse
structured-streaming/*
org.apache.spark.deploy.rest.kubernetes.DriverServiceManager
org.apache.spark.deploy.kubernetes.submit.v1.DriverServiceManager
2 changes: 2 additions & 0 deletions ...urces/META-INF/services/org.apache.spark.deploy.kubernetes.submit.v1.DriverServiceManager
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
org.apache.spark.deploy.kubernetes.submit.v1.ExternalSuppliedUrisDriverServiceManager
org.apache.spark.deploy.kubernetes.submit.v1.NodePortUrisDriverServiceManager
2 changes: 0 additions & 2 deletions .../resources/META-INF/services/org.apache.spark.deploy.rest.kubernetes.DriverServiceManager
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion ...e-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/kubernetes/config.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ package org.apache.spark.deploy.kubernetes
import java.util.concurrent.TimeUnit

import org.apache.spark.{SPARK_VERSION => sparkVersion}
import org.apache.spark.deploy.rest.kubernetes.NodePortUrisDriverServiceManager
import org.apache.spark.deploy.kubernetes.submit.v1.NodePortUrisDriverServiceManager
import org.apache.spark.internal.config.ConfigBuilder
import org.apache.spark.network.util.ByteUnit

Expand Down
5 changes: 2 additions & 3 deletions ...ache/spark/deploy/kubernetes/Client.scala → .../deploy/kubernetes/submit/v1/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import java.io.File
import java.security.SecureRandom
Expand All @@ -32,8 +32,7 @@ import scala.collection.JavaConverters._
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.kubernetes.constants._
import org.apache.spark.deploy.rest.{AppResource, ContainerAppResource, KubernetesCreateSubmissionRequest, KubernetesCredentials, RemoteAppResource, UploadedAppResource}
import org.apache.spark.deploy.rest.kubernetes._
import org.apache.spark.deploy.rest.kubernetes.v1.{AppResource, ContainerAppResource, HttpClientUtil, KubernetesCreateSubmissionRequest, KubernetesCredentials, KubernetesFileUtils, KubernetesSparkRestApi, RemoteAppResource, UploadedAppResource}
import org.apache.spark.internal.Logging
import org.apache.spark.util.{ShutdownHookManager, Utils}

Expand Down
4 changes: 2 additions & 2 deletions ...oy/rest/kubernetes/CompressionUtils.scala → ...bernetes/submit/v1/CompressionUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import java.io.{ByteArrayInputStream, File, FileInputStream, FileOutputStream}
import java.util.zip.{GZIPInputStream, GZIPOutputStream}
Expand All @@ -26,7 +26,7 @@ import org.apache.commons.compress.utils.CharsetNames
import org.apache.commons.io.IOUtils
import scala.collection.mutable

import org.apache.spark.deploy.rest.TarGzippedData
import org.apache.spark.deploy.rest.kubernetes.v1.TarGzippedData
import org.apache.spark.internal.Logging
import org.apache.spark.util.{ByteBufferOutputStream, Utils}

Expand Down
4 changes: 2 additions & 2 deletions ...verPodKubernetesCredentialsProvider.scala → ...verPodKubernetesCredentialsProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,15 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import java.io.File

import com.google.common.io.{BaseEncoding, Files}

import org.apache.spark.SparkConf
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.rest.KubernetesCredentials
import org.apache.spark.deploy.rest.kubernetes.v1.KubernetesCredentials
import org.apache.spark.internal.config.OptionalConfigEntry

private[spark] class DriverPodKubernetesCredentialsProvider(sparkConf: SparkConf) {
Expand Down
3 changes: 1 addition & 2 deletions ...est/kubernetes/DriverServiceManager.scala → ...etes/submit/v1/DriverServiceManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import io.fabric8.kubernetes.api.model.{Service, ServiceBuilder}
import io.fabric8.kubernetes.client.KubernetesClient
Expand Down
4 changes: 2 additions & 2 deletions ...riverSubmitSslConfigurationProvider.scala → ...riverSubmitSslConfigurationProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import java.io.{File, FileInputStream}
import java.security.{KeyStore, SecureRandom}
Expand All @@ -29,7 +29,7 @@ import scala.collection.JavaConverters._
import org.apache.spark.{SecurityManager => SparkSecurityManager, SparkConf, SparkException, SSLOptions}
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.kubernetes.constants._
import org.apache.spark.deploy.rest.kubernetes.{KubernetesFileUtils, PemsToKeyStoreConverter}
import org.apache.spark.deploy.rest.kubernetes.v1.{KubernetesFileUtils, PemsToKeyStoreConverter}
import org.apache.spark.util.Utils

/**
Expand Down
2 changes: 1 addition & 1 deletion ...nalSuppliedUrisDriverServiceManager.scala → ...nalSuppliedUrisDriverServiceManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import java.util.concurrent.TimeUnit

Expand Down
2 changes: 1 addition & 1 deletion ...ubernetes/KubernetesResourceCleaner.scala → ...submit/v1/KubernetesResourceCleaner.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import io.fabric8.kubernetes.api.model.HasMetadata
import io.fabric8.kubernetes.client.KubernetesClient
Expand Down
5 changes: 2 additions & 3 deletions .../kubernetes/LoggingPodStatusWatcher.scala → ...s/submit/v1/LoggingPodStatusWatcher.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import java.util.concurrent.{CountDownLatch, Executors, TimeUnit}

import scala.collection.JavaConverters._

import io.fabric8.kubernetes.api.model.Pod
import io.fabric8.kubernetes.client.{KubernetesClientException, Watcher}
import io.fabric8.kubernetes.client.Watcher.Action
import scala.collection.JavaConverters._

import org.apache.spark.internal.Logging

Expand Down
2 changes: 1 addition & 1 deletion ...es/NodePortUrisDriverServiceManager.scala → ...v1/NodePortUrisDriverServiceManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.kubernetes.submit.v1

import io.fabric8.kubernetes.api.model.{Service, ServiceBuilder}
import scala.collection.JavaConverters._
Expand Down
2 changes: 1 addition & 1 deletion ...ploy/rest/kubernetes/HttpClientUtil.scala → ...y/rest/kubernetes/v1/HttpClientUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.rest.kubernetes.v1

import java.io.IOException
import java.net.{InetSocketAddress, ProxySelector, SocketAddress, URI}
Expand Down
2 changes: 1 addition & 1 deletion ...rest/kubernetes/KubernetesFileUtils.scala → ...t/kubernetes/v1/KubernetesFileUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.rest.kubernetes.v1

import org.apache.spark.util.Utils

Expand Down
12 changes: 10 additions & 2 deletions ...rest/KubernetesRestProtocolMessages.scala → ...s/v1/KubernetesRestProtocolMessages.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,13 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest
package org.apache.spark.deploy.rest.kubernetes.v1

import com.fasterxml.jackson.annotation.{JsonSubTypes, JsonTypeInfo}
import com.fasterxml.jackson.annotation.{JsonIgnore, JsonSubTypes, JsonTypeInfo}

import org.apache.spark.SPARK_VERSION
import org.apache.spark.deploy.rest.{SubmitRestProtocolRequest, SubmitRestProtocolResponse}
import org.apache.spark.util.Utils

case class KubernetesCredentials(
oauthToken: Option[String],
Expand All @@ -35,6 +37,9 @@ case class KubernetesCreateSubmissionRequest(
driverPodKubernetesCredentials: KubernetesCredentials,
uploadedJarsBase64Contents: TarGzippedData,
uploadedFilesBase64Contents: TarGzippedData) extends SubmitRestProtocolRequest {
@JsonIgnore
override val messageType: String = s"kubernetes.v1.${Utils.getFormattedClassName(this)}"
Copy link

@ash211 ash211 Apr 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this to support compat across a span of versions? Maybe a new rest service with an old launcher?

Copy link
Author

@mccheah mccheah Apr 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is necessary due to the fact that SubmitRestProtocolRequest expects all subclasses to live in the package org.apache.spark.deploy.rest. The deserialization fails otherwise. This is a workaround to keep the package names consistent for now until we remove this class entirely.

override val action = messageType
message = "create"
clientSparkVersion = SPARK_VERSION
}
Expand Down Expand Up @@ -68,5 +73,8 @@ class PingResponse extends SubmitRestProtocolResponse {
val text = "pong"
message = "pong"
serverSparkVersion = SPARK_VERSION
@JsonIgnore
override val messageType: String = s"kubernetes.v1.${Utils.getFormattedClassName(this)}"
override val action: String = messageType
}

4 changes: 2 additions & 2 deletions ...t/kubernetes/KubernetesSparkRestApi.scala → ...ubernetes/v1/KubernetesSparkRestApi.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.rest.kubernetes.v1

import javax.ws.rs.{Consumes, GET, Path, POST, Produces}
import javax.ws.rs.core.MediaType

import org.apache.spark.deploy.rest.{CreateSubmissionResponse, KubernetesCreateSubmissionRequest, PingResponse}
import org.apache.spark.deploy.rest.CreateSubmissionResponse

@Path("/v1/submissions/")
trait KubernetesSparkRestApi {
Expand Down
3 changes: 2 additions & 1 deletion ...ubernetes/KubernetesSparkRestServer.scala → ...rnetes/v1/KubernetesSparkRestServer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.rest.kubernetes.v1

import java.io.{File, FileOutputStream, StringReader}
import java.net.URI
Expand All @@ -34,6 +34,7 @@ import scala.collection.mutable.ArrayBuffer
import org.apache.spark.{SecurityManager, SPARK_VERSION => sparkVersion, SparkConf, SparkException, SSLOptions}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.kubernetes.submit.v1.CompressionUtils
import org.apache.spark.deploy.rest._
import org.apache.spark.internal.config.OptionalConfigEntry
import org.apache.spark.util.{ShutdownHookManager, ThreadUtils, Utils}
Expand Down
2 changes: 1 addition & 1 deletion ...t/kubernetes/MultiServerFeignTarget.scala → ...ubernetes/v1/MultiServerFeignTarget.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.rest.kubernetes.v1

import feign.{Request, RequestTemplate, RetryableException, Retryer, Target}
import scala.reflect.ClassTag
Expand Down
2 changes: 1 addition & 1 deletion .../kubernetes/PemsToKeyStoreConverter.scala → ...bernetes/v1/PemsToKeyStoreConverter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.rest.kubernetes
package org.apache.spark.deploy.rest.kubernetes.v1

import java.io.{File, FileInputStream, FileOutputStream, InputStreamReader}
import java.nio.file.Paths
Expand Down
2 changes: 1 addition & 1 deletion ...re/src/main/scala/org/apache/spark/deploy/rest/kubernetes/v2/ResourceStagingService.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import javax.ws.rs.core.{MediaType, StreamingOutput}

import org.glassfish.jersey.media.multipart.FormDataParam

import org.apache.spark.deploy.rest.KubernetesCredentials
import org.apache.spark.deploy.rest.kubernetes.v1.KubernetesCredentials

/**
* Service that receives application data that can be retrieved later on. This is primarily used
Expand Down
2 changes: 1 addition & 1 deletion ...rc/main/scala/org/apache/spark/deploy/rest/kubernetes/v2/ResourceStagingServiceImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import com.google.common.io.{BaseEncoding, ByteStreams, Files}
import scala.collection.concurrent.TrieMap

import org.apache.spark.SparkException
import org.apache.spark.deploy.rest.KubernetesCredentials
import org.apache.spark.deploy.rest.kubernetes.v1.KubernetesCredentials
import org.apache.spark.internal.Logging
import org.apache.spark.util.Utils

Expand Down
2 changes: 1 addition & 1 deletion .../kubernetes/KubernetesClientBuilder.scala → .../kubernetes/KubernetesClientBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.kubernetes
package org.apache.spark.scheduler.cluster.kubernetes

import java.io.File

Expand Down
1 change: 0 additions & 1 deletion ...ala/org/apache/spark/scheduler/cluster/kubernetes/KubernetesClusterSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ import scala.collection.JavaConverters._
import scala.concurrent.{ExecutionContext, Future}

import org.apache.spark.{SparkContext, SparkException}
import org.apache.spark.deploy.kubernetes.KubernetesClientBuilder
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.kubernetes.constants._
import org.apache.spark.rpc.RpcEndpointAddress
Expand Down
2 changes: 1 addition & 1 deletion ...rc/test/scala/org/apache/spark/deploy/rest/kubernetes/v2/ResourceStagingServerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import org.scalatest.BeforeAndAfterAll
import retrofit2.Call

import org.apache.spark.SparkFunSuite
import org.apache.spark.deploy.rest.KubernetesCredentials
import org.apache.spark.deploy.rest.kubernetes.v1.KubernetesCredentials
import org.apache.spark.util.Utils

/**
Expand Down
2 changes: 1 addition & 1 deletion ...st/scala/org/apache/spark/deploy/rest/kubernetes/v2/ResourceStagingServiceImplSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import java.nio.file.Paths
import com.google.common.io.Files

import org.apache.spark.SparkFunSuite
import org.apache.spark.deploy.rest.KubernetesCredentials
import org.apache.spark.deploy.rest.kubernetes.v1.KubernetesCredentials
import org.apache.spark.util.Utils

/**
Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/docker-minimal-bundle/src/main/docker/driver/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ CMD SSL_ARGS="" && \
if ! [ -z ${SPARK_SUBMISSION_KEYSTORE_KEY_PASSWORD_FILE+x} ]; then SSL_ARGS="$SSL_ARGS --keystore-key-password-file $SPARK_SUBMISSION_KEYSTORE_KEY_PASSWORD_FILE"; fi && \
if ! [ -z ${SPARK_SUBMISSION_KEY_PEM_FILE+x} ]; then SSL_ARGS="$SSL_ARGS --key-pem-file $SPARK_SUBMISSION_KEY_PEM_FILE"; fi && \
if ! [ -z ${SPARK_SUBMISSION_CERT_PEM_FILE+x} ]; then SSL_ARGS="$SSL_ARGS --cert-pem-file $SPARK_SUBMISSION_CERT_PEM_FILE"; fi && \
exec bin/spark-class org.apache.spark.deploy.rest.kubernetes.KubernetesSparkRestServer \
exec bin/spark-class org.apache.spark.deploy.rest.kubernetes.v1.KubernetesSparkRestServer \
--hostname $HOSTNAME \
--port $SPARK_SUBMISSION_SERVER_PORT \
--secret-file $SPARK_SUBMISSION_SECRET_LOCATION \
Expand Down
3 changes: 1 addition & 2 deletions ...s/src/test/scala/org/apache/spark/deploy/kubernetes/integrationtest/KubernetesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,13 @@ import scala.collection.JavaConverters._

import org.apache.spark.{SparkConf, SparkException, SparkFunSuite}
import org.apache.spark.deploy.SparkSubmit
import org.apache.spark.deploy.kubernetes.Client
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.kubernetes.constants._
import org.apache.spark.deploy.kubernetes.integrationtest.docker.SparkDockerImageBuilder
import org.apache.spark.deploy.kubernetes.integrationtest.minikube.Minikube
import org.apache.spark.deploy.kubernetes.integrationtest.restapis.SparkRestApiV1
import org.apache.spark.deploy.kubernetes.integrationtest.sslutil.SSLUtils
import org.apache.spark.deploy.rest.kubernetes.ExternalSuppliedUrisDriverServiceManager
import org.apache.spark.deploy.kubernetes.submit.v1.{Client, ExternalSuppliedUrisDriverServiceManager}
import org.apache.spark.status.api.v1.{ApplicationStatus, StageStatus}
import org.apache.spark.util.Utils

Expand Down
2 changes: 1 addition & 1 deletion ...src/test/scala/org/apache/spark/deploy/kubernetes/integrationtest/minikube/Minikube.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import io.fabric8.kubernetes.client.internal.SSLUtils
import scala.collection.mutable.ArrayBuffer
import scala.reflect.ClassTag

import org.apache.spark.deploy.rest.kubernetes.HttpClientUtil
import org.apache.spark.deploy.rest.kubernetes.v1.HttpClientUtil
import org.apache.spark.internal.Logging
import org.apache.spark.util.Utils

Expand Down