Setup a CI environment to run k8s integration test

[cvpatel]

I'm in the process of building an environment to run the k8s integration test; wanted to provide an update on the what's attempted so far and get some opinions from the community on the next steps.

What's attempted so far?
Tried to run on Travis with VirtualBox and KVM as Minikube vm-driver. Also have setup a Jenkins environment on standalone instances but that is not going to work as well. The main issue is the Minikube and its supported hypervisors' system requirement of VT-x/AMD-v support. As of now, the commonly used cloud providers (GCE, AWS) don't support nested virtualization or provide bare-metal instances.

Available options:

1. Find a cloud provider that supports nested virtualization (https://www.ravellosystems.com/ seems like an option).
2. Setup an on-premise bare-metal node (but would require public access to look at the results). Contacted the jenkins maintainer at Amplab and they offered to host the project, but depending our requirements for the environment and getting proper approval it may have a longer lead time.
3. Minikube seems to be adding a no-vm driver for linux CI environment, maybe we can use some of the early builds of that and try it with our test.
4. Setup a k8s cluster just for integration test in a jenkins environment as being discussed Run tests against any specified k8s cluster #100.

Want to get a opinion from the community on where to focus our efforts.

In parallel, I'm starting to explore option 1 as it seems to be path to least resistance...

[cvpatel]

Just to summarize the conclusion, Option 1 was no go for Virtualbox (at least out of the box). We went with a bare-metal setup and have made it publicly accessible.

For all subsequent commits, @spark-k8s-jenkins user will add an "Integration Test" check to each PR similar to what Travis CI does.

• There are two jobs configured to run for this project:
  • PR-spark-k8s-integraion-test - This is triggered on every commit or by adding a comment "rerun integration test please" to the PR.
  • SCHEDULED-spark-k8s-integration-test - This is scheduled to run every 3 hours after syncing with k8s-support-alternate-incremental branch.
• Each test runs the following command:
  ./build/mvn -B clean integration-test -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests -am -Dtest=none -DwildcardSuites=org.apache.spark.deploy.kubernetes.integrationtest.KubernetesSuite
• For debugging purposes, the "resource-managers/kubernetes/integration-tests/target/integration-tests.log" is appended to the jenkins log on failure.

Known issues:

• When integration test ends abruptly, the subsequent tests are impacted. For now, jenkins will be attempt to stop and delete the minikube if it is running; this may or may not be enough. Over time we can make the integration test setup more robust by ensuring it starts from a sane environment.
• Only one minikube can run at time; given the current volume of PRs should be enough for the time being.

[foxish]

Awesome work on setting up the integration tests. Thanks @cvpatel

[cvpatel]

Thanks @foxish; glad to help.

Looks like the tests are running fine now, going to close this issue.

[ash211]

Thanks again for setting up integration tests @cvpatel !