Configure Travis to run K8s integration test #74

cvpatel · 2017-02-02T01:14:26Z

Modify Travis configuration to run k8s integration test with every pull request. In addition, made a change to run lint-scala along with lint-java.

The integration test seems to be failing currently due to an un-related issue. We'll need to fix that before this ratchet is enabled.

cvpatel · 2017-02-02T01:30:05Z

FYI, @ssuchter

lins05

The integration tests make use of minikube, which would fire up a virtual machine. AFAIK travis ci runs the build in a vm. So I doubt trying to create a vm inside a vm would not work.

lins05 · 2017-02-03T16:16:28Z

.travis.yml

nit: "lint-java SKIPPED"

lins05 · 2017-02-03T16:16:36Z

.travis.yml

lins05 · 2017-02-03T16:17:47Z

.travis.yml

semicolons are not necessary for calling scripts or shell commands

Travis munges that whole block onto one line during execution, which would cause the problem for the shell command.

Something I had tried on my test repo (here the lint-java has the semi-colons and lint-scala doesn't):

0.00s$ if [[ $TEST_TO_RUN == "lint-java" ]] ; then dev/lint-java; else echo "SKIPPED"; fi
SKIPPED
The command "if [[ $TEST_TO_RUN == "lint-java" ]] ; then dev/lint-java; else echo "SKIPPED"; fi" exited with 0.

0.00s$ if [[ $TEST_TO_RUN == "lint-scala" ]] ; then dev/lint-scala else echo "SKIPPED" fi
/home/travis/build.sh: eval: line 60: syntax error: unexpected end of file
The command "if [[ $TEST_TO_RUN == "lint-scala" ]] ; then dev/lint-scala else echo "SKIPPED" fi" exited with 1.

cvpatel · 2017-02-03T18:43:35Z

Thanks @lins05 for taking a look. Have addressed the comments.

I haven't had a chance to go investigate the integration test failure yet, but more than likely I could be doing something wrong. I ran it locally and it seemed to fail similarly.

ash211 · 2017-02-08T01:00:10Z

Looks like we need to change the evaluation order of some modules, since this is failing on line 50:

 46 private[spark] class KubernetesSuite extends SparkFunSuite with BeforeAndAfter {
 47
 48   private val EXAMPLES_JAR = Paths.get("target", "integration-tests-spark-jobs")
 49     .toFile
 50     .listFiles()(0)
 51     .getAbsolutePath

Looks like the target/integration-tests-spark-jobs/ folder is empty when starting the KubernetesSuite tests

kimoonkim · 2017-02-08T18:08:46Z

@ash211, Yes that's what I saw too.

It seems that resource-managers/kubernetes/integration-tests/pom.xml specifies a plugin execution for copying the test jobs.

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
      <execution>
        <id>copy-test-spark-jobs</id>
        <phase>pre-integration-test</phase>
        <goals>
          <goal>copy</goal>
        </goals>
        <configuration>
          <artifactItems>
            <artifactItem>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-kubernetes-integration-tests-spark-jobs_${scala.binary.version}</artifactId>
              <version>${project.version}</version>
              <type>jar</type>
              <outputDirectory>${project.build.directory}/integration-tests-spark-jobs</outputDirectory>
            </artifactItem>
            <artifactItem>
              <groupId>org.apache.spark</groupId>
              <artifactId>spark-kubernetes-integration-tests-spark-jobs-helpers_${scala.binary.version}</artifactId>
              <version>${project.version}</version>
              <type>jar</type>
              <outputDirectory>${project.build.directory}/integration-tests-spark-jobs-helpers</outputDirectory>
            </artifactItem>
          </artifactItems>
        </configuration>
      </execution>

But somehow it's not happening. I'm looking into why.

kimoonkim · 2017-02-08T20:07:17Z

I think I found the root cause. KubernetesSuite.scala is running in the maven test phase, which happens much earlier than the pre-integration-test phase that will trigger the copy-test-spark-jobs execution above.

One can override the scalatest plugin, in charge of running tests, to trigger in the integration-test phase. Trying this now.

kimoonkim · 2017-02-08T21:06:30Z

Ok. I was able to pass the previous failing point with scalatest maven plugin triggering in the integration test phase. I see a different error, which is better.

Here's the target dir showing the copied jars:

ls resource-managers/kubernetes/integration-tests/target/*.jar
resource-managers/kubernetes/integration-tests/target/original-spark-kubernetes-integration-tests_2.11-2.2.0-SNAPSHOT.jar
resource-managers/kubernetes/integration-tests/target/spark-kubernetes-integration-tests_2.11-2.2.0-SNAPSHOT-sources.jar
resource-managers/kubernetes/integration-tests/target/spark-kubernetes-integration-tests_2.11-2.2.0-SNAPSHOT-test-sources.jar
resource-managers/kubernetes/integration-tests/target/spark-kubernetes-integration-tests_2.11-2.2.0-SNAPSHOT-tests.jar
resource-managers/kubernetes/integration-tests/target/spark-kubernetes-integration-tests_2.11-2.2.0-SNAPSHOT.jar

Here's the new error:

[INFO] --- scalatest-maven-plugin:1.0:test (integration-test) @ spark-kubernetes-integration-tests_2.11 ---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Discovery starting.
Discovery completed in 266 milliseconds.
Run starting. Expected test count is: 6
KubernetesSuite:
Exception encountered when invoking run on a nested suite - Unknown status There is a newer version of minikube available (v0.16.0). Download it here: *** ABORTED ***
java.lang.IllegalStateException: Unknown status There is a newer version of minikube available (v0.16.0). Download it here:
at org.apache.spark.deploy.kubernetes.integrationtest.minikube.Minikube$$anonfun$getMinikubeStatus$2.apply(Minikube.scala:69)
at org.apache.spark.deploy.kubernetes.integrationtest.minikube.Minikube$$anonfun$getMinikubeStatus$2.apply(Minikube.scala:69)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.deploy.kubernetes.integrationtest.minikube.Minikube$.getMinikubeStatus(Minikube.scala:69)
at org.apache.spark.deploy.kubernetes.integrationtest.minikube.Minikube$.startMinikube(Minikube.scala:51)
at org.apache.spark.deploy.kubernetes.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:82)
at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
at org.apache.spark.SparkFunSuite.beforeAll(SparkFunSuite.scala:31)
at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
at org.apache.spark.deploy.kubernetes.integrationtest.KubernetesSuite.org$scalatest$BeforeAndAfter$$super$run(KubernetesSuite.scala:46)
...

I'll send a PR with the scalatest plugin fix.

kimoonkim · 2017-02-08T23:31:55Z

The minikube version issue was due to my Macbook having an old version of minikube pre-installed. Removed that and the integration test passed (with the patch of #93).

$ build/mvn -B clean integration-test -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests -am -Dtest=none -DwildcardSuites=org.apache.spark.deploy.kubernetes.integrationtest.KubernetesSuite

...

Discovery starting.
Discovery completed in 244 milliseconds.
Run starting. Expected test count is: 6
KubernetesSuite:
- Run a simple example
- Run using spark-submit
- Run using spark-submit with the examples jar on the docker image
- Run with custom labels
- Enable SSL on the driver submit server
- Added files should exist on the driver.
Run completed in 5 minutes, 16 seconds.
Total number of tests run: 6
Suites: completed 2, aborted 0
Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

...

[INFO] Spark Project Kubernetes Integration Tests Spark Jobs Helpers SUCCESS [  3.513 s]
[INFO] Spark Project Kubernetes Integration Tests Spark Jobs SUCCESS [  3.172 s]
[INFO] Spark Project Kubernetes Integration Tests ......... SUCCESS [05:37 min]

kimoonkim · 2017-02-09T17:10:02Z

I sent PR #93 with the scalatest plugin ordering fix. I wonder who would be the best person to review the fix. @lins05, or @ash211, maybe one of you can please take a look? Thanks.

cvpatel · 2017-02-24T19:05:05Z

Have started working on this again and created issue #158 to discuss the options. Based on the finding discussed there, this PR doesn't make sense anymore.

kimoonkim · 2018-03-28T16:28:48Z

Probably this came too late, but now it seems possible to run minikube on travis CI: https://blog.travis-ci.com/2017-10-26-running-kubernetes-on-travis-ci-with-minikube

…park-on-k8s#74)

cvpatel mentioned this pull request Feb 2, 2017

Reduce pre-integration-test runtime #68

Closed

lins05 reviewed Feb 3, 2017

View reviewed changes

Configure Travis to run K8s integration test

62fa890

kimoonkim mentioned this pull request Feb 8, 2017

Trigger scalatest plugin in the integration-test phase #93

Merged

cvpatel closed this Feb 24, 2017

cvpatel deleted the add-travis-k8s-integration-test branch February 28, 2017 22:17

ifilonenko pushed a commit to ifilonenko/spark that referenced this pull request Feb 25, 2019

Fix inset for large queries. Disable record level filtering (apache-s…

a1a17b5

…park-on-k8s#74)

Configure Travis to run K8s integration test #74

Configure Travis to run K8s integration test #74

Uh oh!

Conversation

cvpatel commented Feb 2, 2017

Uh oh!

cvpatel commented Feb 2, 2017

Uh oh!

lins05 left a comment

Choose a reason for hiding this comment

Uh oh!

lins05 Feb 3, 2017

Choose a reason for hiding this comment

Uh oh!

lins05 Feb 3, 2017

Choose a reason for hiding this comment

Uh oh!

lins05 Feb 3, 2017

Choose a reason for hiding this comment

Uh oh!

cvpatel Feb 3, 2017

Choose a reason for hiding this comment

Uh oh!

cvpatel commented Feb 3, 2017

Uh oh!

ash211 commented Feb 8, 2017

Uh oh!

kimoonkim commented Feb 8, 2017

Uh oh!

kimoonkim commented Feb 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kimoonkim commented Feb 8, 2017

Uh oh!

kimoonkim commented Feb 8, 2017

Uh oh!

kimoonkim commented Feb 9, 2017

Uh oh!

cvpatel commented Feb 24, 2017

Uh oh!

kimoonkim commented Mar 28, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kimoonkim commented Feb 8, 2017 •

edited

Loading