Fail running integration test with ClassCastException

[yonzhang]

Run integration-test with the following command:
build/mvn integration-test -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests -am

And get the following class cast exception

ClientSuite:

• The client should configure the pod with the submission steps.
• The client should create the secondary Kubernetes resources.
• The client should attach the driver container with the appropriate JVM options.
• Waiting for app completion should stall on the watcher
  ResourceStagingServerSuite:
• Accept file and jar uploads and downloads
• Enable SSL on the server *** FAILED ***
  java.lang.ClassCastException: org.spark_project.jetty.util.ssl.SslContextFactory cannot be cast to org.eclipse.jetty.util.ssl.SslContextFactory
  at org.apache.spark.deploy.rest.kubernetes.ResourceStagingServer$$anonfun$1.apply(ResourceStagingServer.scala:61)
  at scala.Option.map(Option.scala:146)
  at org.apache.spark.deploy.rest.kubernetes.ResourceStagingServer.start(ResourceStagingServer.scala:61)
  at org.apache.spark.deploy.rest.kubernetes.ResourceStagingServerSuite.org$apache$spark$deploy$rest$kubernetes$ResourceStagingServerSuite$$startServer(ResourceStagingServerSuite.scala:145)
  at org.apache.spark.deploy.rest.kubernetes.ResourceStagingServerSuite$$anonfun$4.apply$mcV$sp(ResourceStagingServerSuite.scala:92)
  at org.apache.spark.deploy.rest.kubernetes.ResourceStagingServerSuite$$anonfun$4.apply(ResourceStagingServerSuite.scala:78)
  at org.apache.spark.deploy.rest.kubernetes.ResourceStagingServerSuite$$anonfun$4.apply(ResourceStagingServerSuite.scala:78)
  at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
  at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  ...

[mccheah]

This is strange - you probably don't want to be running ResourceStagingServerSuite in the integration tests, as this is actually a unit test of sorts. But we've been running into some problems with this test in particular before so it might perhaps just be better to disable it.

[sahilprasad]

@mccheah @yonzhang I've also been having issues with this test. I normally disable it when testing.

[yonzhang]

Thanks @mccheah @sahilprasad
Integration-tests phase will run all previous phases like unit test, that is why all spark core tests are run and this takes very long time.

So finally I simply run only KubernetesSuite directly by the following commands, which will save much time.
build/mvn integration-test -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests -am -DwildcardSuites=org.apache.spark.deploy.kubernetes.integrationtest.KubernetesSuite

[mccheah]

This is very easy to hit and the error is very ambiguous, especially since the test itself that is failing is very much like an integration test as opposed to a unit test. I'm very much in favor of eliminating this test entirely.

That being said - there might be something still missing with Jetty shading in our modules. All of the other Spark modules include the Jetty dependencies in their pom.xml - see Mesos and spark-core in particular, possibly others also that I haven't looked at. We probably also want to include Jetty in our pom.xml so that the Jetty references are shaded accordingly.

[echarles]

I am hitting that cast expection (org.spark_project conflicting with org.jetty), not only in the ResourceStagingServerSuite unit test but also trying to compile the single /resource-managers/kubernetes/core module.

With the -am flag, other modules and the maven voodoo comes into action (see list hereafter), but compiling a single module should work.

[INFO] Spark Project Parent POM
[INFO] Spark Project Tags
[INFO] Spark Project Networking
[INFO] Spark Project Shuffle Streaming Service
[INFO] Spark Project Unsafe
[INFO] Spark Project Launcher
[INFO] Spark Project Core
[INFO] Spark Project Kubernetes

The parent spark pom declares the jetty deps as provided, the kubernetes/core pom does not declare it and relies on the shaded jetty deps present in the spark-core jar. The issue with the unit test is linked with those parameters.

[echarles]

So far, SSLOptions.createJettySslContextFactory is only used in spark/core module and now we use it in spark/resource-manager/kubernetes/core.

The shading maven plugin is configured to shade jetty and guava dependencies to org.spark_project:

• When I run the test in my IDE with only spark/resource-manager/kubernetes/core loaded, the test fails because we don't apply any shading (we work with org.eclipse.jetty packages) and the spark-core.jar works with ``org.spark_project.jetty` packages, which give the ClassCastException.
• When I run the test in the IDE with both spark/resource-manager/kubernetes/core and spark/core modules loaded, everything work well, which is expected and shows the kubernetes code is fine.

I have tried different strategies:

1. Hacking the poms to solve in various ways (maven-dependency-plugin...) the issue - no luck
2. Hacking the code to test the class name of the returned object, cast... - no luck
3. Rebuild the spark/core module with -Dspark.shade.packageName=org.eclipse - With this property, the spark-core.jar puts the jetty classes in their original namespace (strangely I didn't find any guava class).

Listening to the Spark community, shading seems like a common and loved approach (personally, I am not a fan of this).

Unless a maven expert can solve this at pom level, the next step would be, or we talk with the upstream spark on the shading approach and expose them our issue, or we document the -Dspark.shade.packageName=org.eclipse property to build spark.