Cannot configure logging for rest client.

[tsachev]

This affects the 5.6.0-SNAPSHOT version where commons-logging dependency is shaded (#25208).
The problem with commons logging is that it depends on service loader SPI (META-INF/services),
So if I want to provide an alternative logging backend like log4j-jcl, jcl-over-slf4j or other I cannot do it since it searches for the relocated LogFactory.

Here is the log if I enable diagonstics with

System.setProperty(org.elasticsearch.client.commons.logging.LogFactory.DIAGNOSTICS_DEST_PROPERTY, "STDERR");

[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [ENV] Extension directories (java.ext.dir): null
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [ENV] Application classpath (java.class.path): /usr/lib/jvm/java-8-jdk/jre/lib/charsets.jar:/usr/lib/jvm/java-8-jdk/jre/lib/deploy.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-jdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-jdk/jre/lib/javaws.jar:/usr/lib/jvm/java-8-jdk/jre/lib/jce.jar:/usr/lib/jvm/java-8-jdk/jre/lib/jfr.jar:/usr/lib/jvm/java-8-jdk/jre/lib/jfxswt.jar:/usr/lib/jvm/java-8-jdk/jre/lib/jsse.jar:/usr/lib/jvm/java-8-jdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-jdk/jre/lib/plugin.jar:/usr/lib/jvm/java-8-jdk/jre/lib/resources.jar:/usr/lib/jvm/java-8-jdk/jre/lib/rt.jar:/home/tsachev/tick42/metrics-viewer/es-rest-client-demo/target/classes:/home/tsachev/.m2/repository/org/apache/logging/log4j/log4j-api/2.8.2/log4j-api-2.8.2.jar:/home/tsachev/.m2/repository/org/apache/logging/log4j/log4j-core/2.8.2/log4j-core-2.8.2.jar:/home/tsachev/.m2/repository/org/apache/logging/log4j/log4j-jcl/2.8.2/log4j-jcl-2.8.2.jar:/home/tsachev/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/home/tsachev/.m2/repository/org/elasticsearch/client/elasticsearch-rest-client/5.6.0-SNAPSHOT/elasticsearch-rest-client-5.6.0-SNAPSHOT.jar:/home/tsachev/apps/idea-IU-171.3780.107/lib/idea_rt.jar
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [ENV] Class org.elasticsearch.client.commons.logging.LogFactory was loaded via classloader sun.misc.Launcher$AppClassLoader@414493378
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [ENV] Ancestry of classloader which loaded org.elasticsearch.client.commons.logging.LogFactory is sun.misc.Launcher$AppClassLoader@414493378 == 'sun.misc.Launcher$AppClassLoader@18b4aac2'
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [ENV] Ancestry of classloader which loaded org.elasticsearch.client.commons.logging.LogFactory is ClassLoader tree:sun.misc.Launcher$AppClassLoader@414493378 (SYSTEM)  --> sun.misc.Launcher$ExtClassLoader@1706377736 --> BOOT
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] BOOTSTRAP COMPLETED
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] LogFactory implementation requested for the first time for context classloader sun.misc.Launcher$AppClassLoader@414493378
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] sun.misc.Launcher$AppClassLoader@414493378 == 'sun.misc.Launcher$AppClassLoader@18b4aac2'
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] ClassLoader tree:sun.misc.Launcher$AppClassLoader@414493378 (SYSTEM)  --> sun.misc.Launcher$ExtClassLoader@1706377736 --> BOOT
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] No properties file of name 'commons-logging.properties' found.
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] Looking for system property [org.apache.commons.logging.LogFactory] to define the LogFactory subclass to use...
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] No system property [org.apache.commons.logging.LogFactory] defined.
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] Looking for a resource file of name [META-INF/services/org.apache.commons.logging.LogFactory] to define the LogFactory subclass to use...
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP]  Creating an instance of LogFactory class org.apache.logging.log4j.jcl.LogFactoryImpl as specified by file 'META-INF/services/org.apache.commons.logging.LogFactory' which was present in the path of the context classloader.
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] Factory class org.apache.logging.log4j.jcl.LogFactoryImpl loaded from classloader sun.misc.Launcher$AppClassLoader@414493378 does not extend 'org.elasticsearch.client.commons.logging.LogFactory' as loaded by this classloader.
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [BAD CL TREE] sun.misc.Launcher$AppClassLoader@414493378 == 'sun.misc.Launcher$AppClassLoader@18b4aac2'
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [BAD CL TREE] ClassLoader tree:sun.misc.Launcher$AppClassLoader@414493378 (SYSTEM)  --> sun.misc.Launcher$ExtClassLoader@1706377736 --> BOOT
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [CUSTOM LOG FACTORY] sun.misc.Launcher$AppClassLoader@414493378 == 'sun.misc.Launcher$AppClassLoader@18b4aac2'
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [CUSTOM LOG FACTORY] ClassLoader tree:sun.misc.Launcher$AppClassLoader@414493378 (SYSTEM)  --> sun.misc.Launcher$ExtClassLoader@1706377736 --> BOOT

>>> THE PROBLEM

[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [CUSTOM LOG FACTORY] org.apache.logging.log4j.jcl.LogFactoryImpl does not implement LogFactory.
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] The application has specified that a custom LogFactory implementation should be used but Class 'org.apache.logging.log4j.jcl.LogFactoryImpl' cannot be converted to 'org.elasticsearch.client.commons.logging.LogFactory'. Please check the custom implementation. Help can be found @http://commons.apache.org/logging/troubleshooting.html.
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] Unable to create LogFactory instance.
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] An error occurred while loading the factory class:The chosen LogFactory implementation does not extend LogFactory. Please check your configuration. (Caused by java.lang.ClassCastException: The application has specified that a custom LogFactory implementation should be used but Class 'org.apache.logging.log4j.jcl.LogFactoryImpl' cannot be converted to 'org.elasticsearch.client.commons.logging.LogFactory'. Please check the custom implementation. Help can be found @http://commons.apache.org/logging/troubleshooting.html.)


[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] A security exception occurred while trying to create an instance of the custom factory class: [The chosen LogFactory implementation does not extend LogFactory. Please check your configuration. (Caused by java.lang.ClassCastException: The application has specified that a custom LogFactory implementation should be used but Class 'org.apache.logging.log4j.jcl.LogFactoryImpl' cannot be converted to 'org.elasticsearch.client.commons.logging.LogFactory'. Please check the custom implementation. Help can be found @http://commons.apache.org/logging/troubleshooting.html.)]. Trying alternative implementations...
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] No properties file available to determine LogFactory subclass from..
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] [LOOKUP] Loading the default LogFactory implementation 'org.apache.commons.logging.impl.LogFactoryImpl' via the same classloader that loaded this LogFactory class (ie not looking in the context classloader).
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] Loaded class org.elasticsearch.client.commons.logging.impl.LogFactoryImpl from classloader sun.misc.Launcher$AppClassLoader@414493378
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Instance created.
[LogFactory from sun.misc.Launcher$AppClassLoader@414493378] Created object org.elasticsearch.client.commons.logging.impl.LogFactoryImpl@2065951873 to manage classloader sun.misc.Launcher$AppClassLoader@414493378
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Discovering a Log implementation...
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] Trying to get configuration for item org.elasticsearch.client.commons.logging.Log.allowFlawedContext
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No LogFactory attribute found for org.elasticsearch.client.commons.logging.Log.allowFlawedContext
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No system property found for property org.elasticsearch.client.commons.logging.Log.allowFlawedContext
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No configuration defined for item org.elasticsearch.client.commons.logging.Log.allowFlawedContext
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] Trying to get configuration for item org.elasticsearch.client.commons.logging.Log.allowFlawedDiscovery
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No LogFactory attribute found for org.elasticsearch.client.commons.logging.Log.allowFlawedDiscovery
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No system property found for property org.elasticsearch.client.commons.logging.Log.allowFlawedDiscovery
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No configuration defined for item org.elasticsearch.client.commons.logging.Log.allowFlawedDiscovery
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] Trying to get configuration for item org.elasticsearch.client.commons.logging.Log.allowFlawedHierarchy
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No LogFactory attribute found for org.elasticsearch.client.commons.logging.Log.allowFlawedHierarchy
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No system property found for property org.elasticsearch.client.commons.logging.Log.allowFlawedHierarchy
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [ENV] No configuration defined for item org.elasticsearch.client.commons.logging.Log.allowFlawedHierarchy
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Trying to get log class from attribute 'org.apache.commons.logging.Log'
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Trying to get log class from attribute 'org.apache.commons.logging.log'
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Trying to get log class from system property 'org.apache.commons.logging.Log'
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Trying to get log class from system property 'org.apache.commons.logging.log'
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] No user-specified Log implementation; performing discovery using the standard supported logging implementations...
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Attempting to instantiate 'org.elasticsearch.client.commons.logging.impl.Log4JLogger'
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Trying to load 'org.elasticsearch.client.commons.logging.impl.Log4JLogger' from classloader sun.misc.Launcher$AppClassLoader@414493378
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Class 'org.elasticsearch.client.commons.logging.impl.Log4JLogger' was found at 'jar:file:/home/tsachev/.m2/repository/org/elasticsearch/client/elasticsearch-rest-client/5.6.0-SNAPSHOT/elasticsearch-rest-client-5.6.0-SNAPSHOT.jar!/org/elasticsearch/client/commons/logging/impl/Log4JLogger.class'
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] The log adapter 'org.elasticsearch.client.commons.logging.impl.Log4JLogger' is missing dependencies when loaded via classloader sun.misc.Launcher$AppClassLoader@414493378: org/elasticsearch/client/log4j/Category
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Attempting to instantiate 'org.elasticsearch.client.commons.logging.impl.Jdk14Logger'
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Trying to load 'org.elasticsearch.client.commons.logging.impl.Jdk14Logger' from classloader sun.misc.Launcher$AppClassLoader@414493378
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Class 'org.elasticsearch.client.commons.logging.impl.Jdk14Logger' was found at 'jar:file:/home/tsachev/.m2/repository/org/elasticsearch/client/elasticsearch-rest-client/5.6.0-SNAPSHOT/elasticsearch-rest-client-5.6.0-SNAPSHOT.jar!/org/elasticsearch/client/commons/logging/impl/Jdk14Logger.class'
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] [INFO] 'org.elasticsearch.client.commons.logging.impl.Jdk14Logger' from classloader sun.misc.Launcher$AppClassLoader@414493378 does not declare optional method setLogFactory(LogFactory)
[LogFactoryImpl@2065951873 from sun.misc.Launcher$AppClassLoader@414493378] Log adapter 'org.elasticsearch.client.commons.logging.impl.Jdk14Logger' from classloader sun.misc.Launcher$AppClassLoader@414493378 has been selected for use.

Maybe its better to leave commons-logging un-shaded. WDYT?

[colings86]

@javanna Could you take a look at this one?

[javanna]

@hub-cap any ideas how we can work around this?

[hub-cap]

Hmm, ill look into this asap. It might be that we unshade the o.a.logging and only shade the o.a.http.

-Edit: Hah, I just noticed at the bottom of the initial comment Maybe its better to leave commons-logging un-shaded. WDYT? ... exactly, that was my first thought too :P

[jasontedor]

I think one of the primary drivers for shading was commons logging (see #24067), so if we are not going to be shading that do we still need to shade?

[javanna]

We do need shading for the apache http client deps as well.

[hub-cap]

I am closing this in favor of #26328 , which has a summary and a few open questions on it. Thank you for using our snapshot and finding this @tsachev. I will make sure we have a solution for 5.6 and beyond.