[SPARK-8316] Upgrade to Maven 3.3.3

[nchammas]

Versions of Maven older than 3.3.0 apparently have a bug in how they handle transitive dependencies.

I confirmed that upgrading to Maven 3.3.3 resolves at least the particular manifestation of this bug that I ran into.

[nchammas]

I removed this method because I found it unnecessary and hard to understand.

CC @brennonyork who I believe wrote this, in case I missed something.

[nchammas]

cc @srowen

[SparkQA]

Test build #34732 has finished for PR 6770 at commit 6bed2d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I believe we'd also need to require Maven 3.3 in the build, and in enforcer plugin? to actually fail if used with a local version that's not high enough?

3.3 is fairly new and most dev environments won't have it. I suppose that's why build/mvn exists too. It may still cause some surprise since most mvn ... would fail then.

Not enforcing this risks people hitting the issue that prompted this, I suppose, without knowing. then again, otherwise, earlier versions of Maven appear fine.

Hm. What do you guys think about the tradeoffs?

[nchammas]

I believe we'd also need to require Maven 3.3 in the build, and in enforcer plugin? to actually fail if used with a local version that's not high enough?

@srowen - We can always install Maven at this step and ignore any existing Maven installs. Would that be sufficient?

[nchammas]

@pwendell may have some thoughts on this.

As I noted in a line comment, we can always install and use our own version of Maven, ignoring any existing installs on the user's machine.

That should, I believe, guarantee that we're always using the version of Maven that we want.

[pwendell]

Has this issue ever affected anyone before now? If it's a relatively obscure issue I think failing builds with Maven < 3.3 might be a big inconvenience for developers.

[srowen]

Heh, well we have this problem right now: #6492
The reason it's popped up is that Hadoop 1 is now controlled by a profile, and profile handling behavior is at issue here.

Of course if we didn't support Hadoop 1, this would go away. (See what I did there?)
Nah, it's really nothing to do with Hadoop 1 per se, but profile handling, but I do wonder if this would eventually bite again anyway for a different profile.

I think the only reasonable way forward on this one right now is to, yes, download and use the latest Maven when that is requested, but not actually fail the build if 3.3 isn't used. This does mean that someone with 3.2 on their machine can run into #6492, but hey, that only affects people building for Hadoop 1. Maybe close enough for now.

[brennonyork]

We could also set a warning to print if a user already has mvn installed, but version < 3.3. That seems the least intrusive to the dev community without mandating they install the latest version. Just my 2c.

[nchammas]

Is it an inconvenience if we just always download (if not already downloaded) and use Maven, instead of checking for a local install first? We already do that for Zinc and Scala, so I don't see that we need to ever fail the build for using an out of date version of Maven.

As long as we always download our own version of Maven, then the only time a developer might still use an out-of-date version is if they already downloaded one using build/mvn before we make this update. Not a big deal as they can just delete the one downloaded to build/ and rerun build/mvn.

[nchammas]

If it's a relatively obscure issue

I dunno if it counts as obscure, but the issue described here means at the very least that we will generate broken Spark builds for Hadoop 1 when using versions of Maven older than 3.3.

[srowen]

Other discussion aside, I think this is OK to merge, since it merely updates the version of Maven that is downloaded to a new version, which should be a generally good thing.