Maven build: Simplify the way transitive dependency resolution is disabled, and increase the minimum supported Maven from 2.2.1 to 3.2.1 [LUCENE-8616]

[asfimport]

Right now the Maven build disables transitive dependency resolution for all dependencies by listing each dependency's dependencies in a nested <exclusions> section in the grandparent POM's <dependencyManagement> section.

As of Maven 3.2.1, it's possible to use a wildcard syntax in nested <exclusions> sections to exclude all transitive deps: MNG-2315. This would make the grandparent POM much smaller and make the intent much clearer.

We would have to increase our minimum supported Maven (currently 2.2.1), but that seems reasonable, given that Maven 2.x has been EOL'd: https://maven.apache.org/maven-2.x-eol.html

---

Migrated from LUCENE-8616 by Steven Rowe (@sarowe), updated Mar 19 2019
Attachments: LUCENE-8616.patch
Linked issues:

• SOLR-11579

[asfimport]

Daniel Collins (@dcollins53) (migrated from JIRA)

https://issues.apache.org/jira/browse/SOLR-11579 was a related ticket for upgrading all the maven plug-ins, will try to update the patches for that tonight.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Thanks @dcollins53, I'll add SOLR-11579 as a linked issue here.

I had considered upgrading all maven plugins as part of this ticket, but since you already have an open issue, let's do it over there. I do think it makes sense to delay that work until this issue has been finished though, since IIRC the Versions Maven Plugin will only suggest upgrades that are compatible with the minimum Maven version, which will be raised by this issue.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Related: Elasticsearch had wildcard exclusions for a while and then put back their explicit exclusions, because 3rd-party tools like Ant didn't understand wildcard exclusions, see elastic/elasticsearch#24879

Ivy 2.5.0-rc1, released 8 months ago, includes wildcard exclusions support (IVY-1531). There has been no discussion on the Ant dev mailing list about a final 2.5.0 release, but I found an ivy-user@ post from an Ant dev ( https://lists.apache.org/thread.html/e911b9ea3b8c415de2927f3762ba0866465d95ee75aed6c6b336276a@%3Civy-user.ant.apache.org%3E ) that says:

2.5.0-rc1 has a major issue that's being tracked as part of IVY-1580. I have been working on it but haven't had enough time to get a right solution for it. A fix for that would be necessary to release rc2 of Final of 2.5.0. I don't have an exact time frame for the release, but I'll send a note on the dev list once I have something.

Unfortunately there has been no activity on IVY-1580 since then, and Elasticsearch's latest POMs still have explicit exclusions.

Should we delay implementing wildcard exclusions until after Ivy 2.5.0 final has been released?

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Attaching patch that increases minimum supported Maven from 2.2.1 to 3.2.1, and that switches exclusions to wildcards in the grandparent POM's <dependencyManagement> section.

I tested using Maven 3.5.3 on macOS.

Maven compilation works, and all Lucene tests pass. None of the Solr modules pass all their tests, but that's also true of the unpatched Maven build, for me anyway.

I had to upgrade the maven-remote-resources-plugin version, from (default) 1.4 to 1.6.0, because 1.4 apparently ignores wildcard exclusions.

I looked at differences between the generated grandparent POMs, and as expected, all dependencies' <exclusions> sections are converted to wildcards. One other notable difference: in the unpatched Maven build's grandparent POM, 82/259 dependencies have zero exclusions (and hence no <exclusions> section), while with the patch every dependency, including these 82, includes wildcard exclusions. I looked at a random handful of these, and they legitimately have no compile-time non-optional dependencies. Declaring them with an exclusions section (as in the patched Maven build) looks better though, since its presence makes explicit the intent to disable transitive dependency resolution.

I also looked at the difference in output between the patched and the unpatched Maven build for mvn dependency:tree, and there was only one difference (repeated for each module with this dependency) - the unpatched Maven build shows jdk.tools:jdk.tools as a transitive dependency of hadoop-annotations:

[INFO] +- org.apache.hadoop:hadoop-annotations:jar:2.7.4:compile
[INFO] |  \- jdk.tools:jdk.tools:jar:1.8:system

^^ looks like a failure of the dependency discovery mechanism used by the mvndeps task (via the Ivy API); the patched Maven build does not include this transitive dependency, so that would be an improvement.

This patch also includes a fix for a WARNING in the unpatched build:

[WARNING] Some problems were encountered while building the effective model for org.apache.solr:solr-test-framework:jar:8.0.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.lucene:lucene-test-framework:jar -> duplicate declaration of version ? `@ org`.apache.solr:solr-test-framework:[unknown-version], /Users/sarowe/git/lucene-solr-3/maven-build/solr/test-framework/pom.xml, line 131, column 17

^^ This is caused by the lucene-test-framework dependency being listed twice in the generated POM for solr-test-framework, once from the POM template (to order it at the top of the dependency list, since it must precede the lucene-core dependency) and another mention from the dependency discovery mechanism. The included fix just adjusts the regex for the logic to exclude some internal dependencies, so that it matches properly in this case and causes lucene-test-framework not to be added by the dependency discovery mechanism.

I won't commit until I get other opinions about the wisdom of doing so, given the Ant support issues mentioned in my previous comment on this issue.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

About the size of the grandparent POM generated with the patch: it's 75% smaller: 101KB vs. 386KB; and 3K lines vs. 11K lines.

[asfimport]

Kevin Risden (@risdenk) (migrated from JIRA)

@sarowe anymore progress on this? What else is left?

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

@risdenk: it's ready to go (might have grown stale since...), but as I wrote in a post above:

I won't commit until I get other opinions about the wisdom of doing so, given the Ant support issues mentioned in my previous comment on this issue.

There has been no activity on IVY-1580 since then, and a query about Ivy 2.5.0 release timing on the ant-dev list was responded to two weeks ago:

The only thing that's stopping me from
releasing 2.5.0 is IVY-1580 issue. I will send out a mail this week to
dev list to explain what that issue is and what are our options on
getting past it. Once we have a decision, I'll go ahead with a release
proposal.

[asfimport]

Kevin Risden (@risdenk) (migrated from JIRA)

Sounds good - came here from SOLR-11579 after the email about master potentially being JDK 11. We are going to have to upgrade plugins to make the maven build continue to work on master.