SPARK-2332 [build] add exclusion for old servlet-api on hadoop-client in core

[pdmack]

Fix for class of test suite failures in jenkins

[srowen]

I've been wrestling with this dependency for a while. However I thought we specifically wanted to retain servlet-api 3.0. What problem are you seeing? It's possible there's a different answer.

[pdmack]

Well, the build seems generally set up for jetty 8 which makes use of servlet-api 3.0.x. So I'm assuming we don't want the old 2.5 one since there are numerous explicit excludes littered through the poms for that version.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Yeah, saw your detail on the JIRA. Yes 2.5 should not be in there, I'm just surprised it is. The concern is knocking out 3.0 accidentally. Have you checked that is all OK still? I'll look too just because I've been staring at this particular dependency hell for a while now.

[pdmack]

mvn -X -Dhadoop.version=2.0.0-mr1-cdh4.1.2 -Dlabel=centos test

I didn't see any jetty test failures and certainly none of the previous FilterRegistration security exceptions. Also 3.0 remains in tact:

DEBUG] Invalidating /home/pmackinn/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar: class javax.servlet.http.HttpServletRequest now provided by /home/pmackinn/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/3.0.0.v201112011016/javax.servlet-3.0.0.v201112011016.jar

[srowen]

Hey @pdmack as far as I can see, you're correct. javax.servlet:servlet-api is not currently a dependency for most (?) Hadoop versions. It seems to come in with versions of Hadoop 2.0.x via hadoop-hdfs. Excluding it should be fine since it is not desirable where it is brought in, and not brought in elsewhere.

Just check that it all works, that the web UI starts. I have managed to accidentally exclude all copies of the servlet API accidentally before!

[rxin]

Thanks for sending this. Do you mind updating the SBT build file as well (project/SparkBuild.scala)?

[srowen]

On a very related note, have a look at #906 -- SPARK-1949. It addresses something very similar and I fixed the problem in it vis-a-vis tests.

[pdmack]

@rxin, I added requested change to project/SparkBuild.scala

[rxin]

Jenkins, retest this please.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16278/

[rxin]

Thanks. I'm merging this in master.