Reduce pre-integration-test runtime

[mccheah]

docker-minimal-bundle now only includes the core, kubernetes, and sql modules of the
Spark project. The Dockerfiles are still valid for the main distribution
which includes all of the modules.

A small bug was found and fixed on the way in using jars that are mounted on the
docker image.

Closes #66

[mccheah]

Build failure seems transient.

[ash211]

@ssuchter this is the PR I mentioned that might make integration tests faster

[ssuchter]

Yep, thanks Andrew. I mentioned your comment to Chai from our team who is testing out the Travis<->integration test integration (heh) and he was already way ahead of me and had seen this. :) On February 1, 2017 at 11:09:32 AM, Andrew Ash ([email protected]) wrote: @ssuchter <https://github.com/ssuchter> this is the PR I mentioned that might make integration tests faster — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2E2_Hqzcg-GXd6GJqtbRRS8LY7QlDqks5rYNhkgaJpZM4LzQX_> .

[mccheah]

Note this isn't primarily for reducing docker image size - the removal of a few jars is negligible. This primarily focuses on removing compilation steps from the Maven flow, so running pre-integration-test to prepare the environment for running tests in an IDE is faster. I think we can get even faster if we remove the dependency on SQL and just use raw SparkContext operations.

[cvpatel]

Thanks @mccheah, this will also help in reducing the integration test run time :). Also, I had had to apply this patch locally to get the tests working.

I'll submit a PR with the travis changes I was trying out so we continue the discussion there on what to include.

[mccheah]

I'm not sure if the overall build time will reduce by a significant amount since the full build over all modules needs to run anyways. We could conceivably decide to make our Travis CI only run the kubernetes/* and core modules though since those are the only things we will be changing.

[cvpatel]

Have created #74, PTAL. We can discuss further modifications to what we run with Travis there.

[mccheah]

This is unfortunately out of date and probably would be easier to revisit from the ground up later on. I'm going to close this but I'll revisit this in post-alpha.