Fat jars cause classloading problems with JDK11 and Security Manager

[piotrlitwin]

We have faced some classloading problems after migrating to Spring Boot 2.1.6 and Spring Framework 5.1.8. The WAR we build is built under JDK8 but the runtime environment is JDK11.
We are trying to deploy our application, but then we get ClassNotFoundException and the Spring context fails to initialize.

We have double checked WAR file. The "missing" class is there.

Now, the really interesting part: we are able to reproduce this one only with Security Manager. It seems like it's not a problem with our own custom security policy, as we have also tried granting access to everything. The problem persists even in this, rather extreme, case.

Therefore, to run our application we have only two options:

1. Run it under JDK8
2. Run it under JDK11, but without the Security Manager

Another description of (apparently) the same problem: https://stackoverflow.com/questions/54063602/springboot-on-open-jdk-11-classnotfound-errors-when-securitymanager-is-activ

[philwebb]

Can you please provide a stacktace and if possible a sample application that we can run to debug the issue?

[piotrlitwin]

I have generated a simple app with https://start.spring.io/. When running WAR file with Security Manager I get:

2019-08-08 14:00:26.573  WARN 13016 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during
context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web
server; nested exception is java.lang.NoClassDefFoundError: org/apache/catalina/Lifecycle$SingleUse
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
        at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:58)
Caused by: java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
        at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:119)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.log(Logger.java:765)
        at org.apache.commons.logging.LogAdapter$Slf4jLocationAwareLog.error(LogAdapter.java:410)
        at org.springframework.boot.SpringApplication.reportFailure(SpringApplication.java:822)
        at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:797)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1214)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1203)
        at com.example.demo.DemoApplication.main(DemoApplication.java:10)
        ... 8 more

It is not actually ClassNotFoundException, which happens in conjunction with another framework we use in our project. This NoClassDefFoundError I get from empty Spring Boot application seems to be the root cause of our problems.

[snicoll]

Thanks for sharing the stacktrace. I've edited your comment to improve the formatting. You might want to check out this Mastering Markdown guide for future reference.

There are several reference to ThrowableProxy and there is nothing at this point that indicates this is a Spring Boot bug. See #6151 that points ultimately to this thread. Can you share what makes you think this might be an issue in Spring Boot?

[piotrlitwin]

The #6151 is not the best example, because it is the runtime problem. In our case Spring Boot app does not even start.
Ok, so I have generated a demo app and built it with mvn clean install. Then I tried to run it with java -jar -Djava.security.manager demo-0.0.1-SNAPSHOT.war
As previously stated, it happens only under JDK11 with -Djava.security.manager. Without Security Manager everything works fine. I expect, that an empty application which I have generated will run with Security Manager as well. Which unfortunately didn't happened.
To be sure, I have put this statement in Security Policy:

grant {
permission java.security.AllPermission;
}

and the problem persists, which shows us clearly, that this is not a problem with Security Policy itself, but with Spring Boot or, potentially, JDK.
Moreover
Caused by: java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
makes no sense, as we are able to localize it in generated WAR file.

It is also noteworthy, that this clearly is a problem with compatibility with JDK11. I have tried:

1. Build it under JDK8, run it under JDK8 -> no problem
2. Build it under JDK8, run it under JDK11 -> fail
3. Build it under JDK11, run it under JDK11 -> fail

[piotrlitwin]

I have also tried to do the same with Spring Boot 1.5.1.RELEASE and did not observed any problems with classloading.

[snicoll]

I did a few tries, excluding logback and including log4j (got a log4j class) then excluding the logging system entirely and got the same error on a core Spring Boot class.

The fact classloading is different with or without a security manager feels like a bug in the JDK at a first glance but we'll keep investigating to narrow down the scope of the problem.

[hohwille]

As the issue also comes if you are using a policy with AllPermissions granted it apprears to be a bug or side-effect that the security manager is causing in Java11 negatively influencing the classloading. If this would be a bug in spring boot itself e.g. in the bootstrap classloader implementation this would IMHO imply that some exceptions from security manager are caught consumed and not propagated in any way but instead leading to the class not found exception. However, neigther the stacktrace of the NoClassDefFoundError nor the fact that it works with JDK8 seems to make this likely.
I also ran the app with -Djava.security.debug="access,failure" but got no indication that a rejection from the Java Security Manager is actually causing the classloader to fail. Any ideas how to debug or trace down this issue to the root? I tried with some breakpoints but this is extremely tricky. I always get my breakpoint when it is too late or I end up debugging endless times of generic classlaoder stuff even with conditional breakpoints. Some expert hint anybody?