Shade client jars and build uber jar

[rjernst]

We currently have a shared version of http client and its dependencies (eg various apache commons libs). This version must be used across any plugin/module that uses the ES test framework. Note that we have no such dependency in core, yet still the version must alight because the test framework is pulled in everywhere, and the rest client is part of the test framework. This is particularly troublesome as we sometimes force a version of these dependencies which does not align with what the plugin requires from other dependencies (eg see discussion in #24067).

In order to allow each plugin to use their own version of httpclient and its deps, we should shade http client and all other deps of the rest client, and package it in an uber jar. This will make the rest client standalone, so that no conflict will occur when the client is on the test classpath with any plugin also depending on http client.

[javanna]

This came up in some comments on #23331. I think that shading REST client dependencies (both low level and high level) will help a lot users, as they are going to have conflicts with jackson, http-client etc.

[rjernst]

To be clear, I am only talking about shading the low level client deps, ie http client/apache commons stuff. We should not try shading ES or lucene, that has all kinds of issues which we should solve by removing those deps altogether at a later time.

[javanna]

I think we should consider shading dependencies for the high level REST client too. That is going to be a huge help for users and will certainly help adoption too. Shading the Elasticsearch dep is not needed, but I would consider doing it for lucene, and of course jackson and friends. What issues are there with shading lucene? I agree that we should remove those deps altogether, but given that it will take time, shading would be an ok temporary solution till we get there.

[dadoonet]

If technically feasible, I'd fully agree with Luca here.
Not sure if we can test when people are embedding as well another Lucene version in our tests though. (Like what Hibernate search does: embedding elasticsearch client and a Lucene version IIRC).

[rjernst]

Shading becomes difficult with SPI, which Lucene uses. It may not matter for how Lucene is found within the high level client, but I think we should start simple. Shading the low level client deps will solve a large portion of the problems (and solve many of our test problems with httpclient as well).

[javanna]

We briefly discussed this in FixItFriday and we said there isn't much to discuss after all. We should start with shading low level clients deps and then see whether we need to do more or not.

[tlrx]

I tried to look at this and to implement it (spoiler: with no success)

I looked at the Shadow Gradle plugin which is a port of the well known Apache Maven Shade Plugin for Gradle. The documentation is good and it's easy to configure it on a project and to build a shadow jar (in addition to the normal Jar) that includes all compile dependencies. It adds a "shadow" configuration and a "shadowJar" task to the project and also allows to relocate packages by manipulating the bytecode of the classes (using asm) before writing them in the jar file.

It is also possible to use Transformers during the shadowing to do various things like removing duplicate license/notice files or merge SPI service files (that can help to shade Lucene dependencies). I still think that we'll have to write our own transformers just to not mess up the license files.

In multiprojects build, it is possible to use the 'shadow' configuration in a project that depends on the shadow jar with something like this:

dependencies {
  compile project(path: ':client:rest', configuration: 'shadow')
}

Sadly I didn't manage to get it working correctly with the low level REST client and the high level REST client.

Also, I don't really like the fact that classes are modified just before being packaged as a Jar while the tests are executed using the non shaded classes. Same thing for debugging code. Some dependencies might rely on hard package names to load a resource or a factory and I'd be more comfortable if our tests ran with already shaded classes so that we can catch such things more easily.

I ended up writing some code to try to shade the dependencies using a custom plugin (see tlrx@b552340). It basically unzip the jar files of dependencies marked as 'shaded' and modify classes
to relocate packages before the compileJava task is executed. This way project code can rely on already shaded classes, but it brings a lot of complexity to make it work
correctly with IDEs and sub projects too.

That's where my little knowledge of gradle and our build brought me, I'd be happy if someone can have a quick look and give his thoughts?

[rjernst]

Also, I don't really like the fact that classes are modified just before being packaged as a Jar while the tests are executed using the non shaded classes.

I would not do this, for that exact reason. I think we could instead do this by modifying the dependencies before the code is even compiled. I'll be happy to take a look.

[tlrx]

@rjernst I agree, that's why I tried another solution. I'd be happy to help implement this.