Initialization of LoggingDeprecationHandler fails because of missing log4j core dependency

[fl0rian-p]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version):
6.3.0
Plugins installed: []

JVM version (java -version):
1.8.0
OS version (uname -a if on a Unix-like system):
AIX ***** 1 7 00C7EE054C00
Description of the problem including expected versus actual behavior:

When trying to index a document using the rest high level client, we have the following error message
java.lang.NoClassDefFoundError: org.apache.logging.log4j.core.Appender because as we are using the logback we do not have the log4j-core dependency.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Use logback with log4j-to-slf4j
2. Use the index method on RestHighLevelClient with and IndexRequest:

      IndexRequest request = new IndexRequest(index, "doc", id);
      request.source(source, XContentType.JSON);
      highElasticClient.index(request);

Provide logs (if relevant):

java.lang.NoClassDefFoundError: org.apache.logging.log4j.core.Appender
        at org.elasticsearch.common.xcontent.LoggingDeprecationHandler.<clinit>(LoggingDeprecationHandler.java:45)
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:651)
        at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$2(RestHighLevelClient.java:508)
        at org.elasticsearch.client.RestHighLevelClient$$Lambda$203.000000002585ACF0.apply(Unknown Source)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:539)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508)
        at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:348)

We have tried using this but as some of the classes in Loggers.java are actually coming from log4j-core they are nowhere to be found.

Any ideas on something we can try to have these operations working and logs redirected to logback ?
Thanks in advance

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

@nik9000 or @dakrone Can one of you pick this one up?

[nik9000]

@dakrone I think we should use forbidden apis to ban LoggingDeprecationHandler from the high level rest client. What do you think?

[dakrone]

@nik9000 I think that sounds reasonable

[nik9000]

I'm gonna do this.

[nik9000]

@florian-perrotti - I'll push a fix and it'll release the next time we cut a release.

What is happening is that the high level rest client shares a bunch of stuff with the Elasticsearch server so it includes the server. We'd like to only include the request and responses but that'll take some doing. Anyway, because it is available, we started using a portion of the server that logs deprecation warnings. It needs log4j's implementation, not log4j's API. This was a mistake. Luckily I spent a while refactoring things so this implementation is pluggable so I'm plugging something else in and I'm using one of our static analysis tools to forbid this particular implementation in the high level client. It isn't a guarantee that we won't accidentally drag log4j's implementation in again, but it is something.