Wrong ClassLoader set within JMX Endpoint

[Dav1dde]

Simple Scenario: Spring Boot 2.0.0.RC1 Application with Spring-Cloud Config is built as WAR-File and deployed into a Tomcat 8.5. Trying to refresh the context using the JMX Refresh endpoint results in an Exception:

java.lang.IllegalStateException: Cannot load configuration class: org.springframework.cloud.context.refresh.ContextRefresher$Empty
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:416) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanFactory(ConfigurationClassPostProcessor.java:254) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:284) ~[spring-context-5.0.3.RELEASE.jar:2.5.5]
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:128) ~[spring-context-5.0.3.RELEASE.jar:2.5.5]
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:693) ~[spring-context-5.0.3.RELEASE.jar:2.5.5]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:531) ~[spring-context-5.0.3.RELEASE.jar:2.5.5]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:752) ~[spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:388) ~[spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) ~[spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
        at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:136) [spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
        at org.springframework.cloud.context.refresh.ContextRefresher.addConfigFilesToEnvironment(ContextRefresher.java:82) [spring-cloud-context-2.0.0.M5.jar:2.0.0.M5]
        at org.springframework.cloud.context.refresh.ContextRefresher.refresh(ContextRefresher.java:61) [spring-cloud-context-2.0.0.M5.jar:2.0.0.M5]
        at org.springframework.cloud.endpoint.RefreshEndpoint.refresh(RefreshEndpoint.java:41) [spring-cloud-context-2.0.0.M5.jar:2.0.0.M5]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
        at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:223) [spring-core-5.0.3.RELEASE.jar:2.5.5]
        at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74) [spring-boot-actuator-2.0.0.RC1.jar:2.0.0.RC1]
        at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:62) [spring-boot-actuator-2.0.0.RC1.jar:2.0.0.RC1]
        at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:99) [spring-boot-actuator-2.0.0.RC1.jar:2.0.0.RC1]
        at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:91) [spring-boot-actuator-2.0.0.RC1.jar:2.0.0.RC1]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [na:1.8.0_131]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [na:1.8.0_131]
        at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:468) [na:1.8.0_131]
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) [na:1.8.0_131]
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) [na:1.8.0_131]
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) [na:1.8.0_131]
        at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_131]
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408) [na:1.8.0_131]
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) [na:1.8.0_131]
        at sun.reflect.GeneratedMethodAccessor934.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346) [na:1.8.0_131]
        at sun.rmi.transport.Transport$1.run(Transport.java:200) [na:1.8.0_131]
        at sun.rmi.transport.Transport$1.run(Transport.java:197) [na:1.8.0_131]
        at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_131]
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196) [na:1.8.0_131]
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) [na:1.8.0_131]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) [na:1.8.0_131]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) [na:1.8.0_131]
        at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_131]
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) [na:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.ClassNotFoundException: org.springframework.cloud.context.refresh.ContextRefresher$Empty
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_131]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_131]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[na:1.8.0_131]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_131]
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:264) ~[spring-core-5.0.3.RELEASE.jar:2.5.5]
        at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass(AbstractBeanDefinition.java:437) ~[spring-beans-5.0.3.RELEASE.jar:2.5.5]
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:403) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
        ... 46 common frames omitted

After some digging it turns out the ConfigurationClassPostProcessor gets the Class Loader injected which resolves to sun.misc.Launcher$AppClassLoader instead of the one used to start the application -ParallelWebappClassLoader.

To resolve the class loader
org.springframework.util.ClassUtils#getDefaultClassLoader is used, which resolves the class loader using Thread.currentThread().getContextClassLoader(), now we are still within the JMX Thread -> You get the wrong class loader.

When starting with an embedded Tomcat this problem does not happen because the Application uses the same class loader when starting as during the JMX call (always sun.misc.Launcher$AppClassLoader)

[wilkinsona]

Support for refreshing the context is part of Spring Cloud Commons.

[Dav1dde]

This exact problem is a spring cloud problem, but the class loader issue exists for all endpoints.

Basically the bug is that the wrong class loader is used in JMX endpoints.

[wilkinsona]

This exact problem is a spring cloud problem, but the class loader issue exists for all endpoints.

Thanks. That wasn't clear to me from your original description.

Basically the bug is that the wrong class loader is used in JMX endpoints.

Perhaps, but I'm not sure that we should try to do that in Boot. I would expect the MBean server to set the thread's context class loader appropriately before it invokes the MBean. This is analogous to a servlet container setting the TCCL of a request-handling thread before dispatching to the application. If MBean Servers don't generally do that we may have to try to do something, but I'm a rather wary of breaking something. I wonder what @snicoll thinks. He knows more about JMX than I do.

When starting with an embedded Tomcat this problem does not happen because the Application uses the same class loader when starting as during the JMX call (always sun.misc.Launcher$AppClassLoader)

I don't understand that. Assuming that you were using embedded Tomcat launched from a fat jar, the application will have been loaded using Boot's LaunchedURLClassLoader and app class loader won't be able to see the application's classes.

[Dav1dde]

Sorry, I should have phrased that better.

I don't understand that. Assuming that you were using embedded Tomcat launched from a fat jar, the application will have been loaded using Boot's LaunchedURLClassLoader and app class loader won't be able to see the application's classes.

You are right, I was launching the Application through IntelliJ's Spring-Boot "launcher". Using mvn spring-boot:run the same issue occurs:

Perhaps, but I'm not sure that we should try to do that in Boot. I would expect the MBean server to set the thread's context class loader appropriately before it invokes the MBean.

After a quick google search it seems that this is a known problem, suggested is to override the currents thread class loader with getClass().getClassLoader() (StackOverflow). No idea if that is the "right" thing though.

Should I open an issue for Spring-Cloud as well?