Understand Gradle 4.8 test failure around setup of security policies

[alpar-t]

With Gradle 4.8 running on JDK 10 this happens (reformatted for readability):

java.lang.RuntimeException: unable to install test security manager
	at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:170)
	at org.elasticsearch.test.ESTestCase.<clinit>(ESTestCase.java:197)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:374)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:592)
Caused by: java.lang.IllegalStateException: codebase property already set: codebase.plugin-classloader -> 
file:/home/alpar/work/elastic/elasticsearch_master_ci/libs/plugin-classloader/build/distributions/plugin-classloader-7.0.0-alpha1-SNAPSHOT.jar, cannot set to 
file:/home/alpar/work/elastic/elasticsearch_master_ci/libs/plugin-classloader/build/distributions/plugin-classloader-7.0.0-alpha1-SNAPSHOT.jar
	at org.elasticsearch.bootstrap.Security.readPolicy(Security.java:233)
	at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:145)
	... 4 more

then most other tests fail with something like:

Exception in thread "Thread-3" java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.test.ESTestCase
    at java.base/java.lang.Thread.run(Thread.java:844)

The reading of policies from Security is done trough a static {} block in BootstrapForTesting which should guarantee that it is only initialized once. There doesn't seem to be any other code paths that exercise this, the only time Security.readPolicy is in BootstrapForTesting and Bootstrap that starts Elasticsearch, even if the tests were to do that, it doesn't have a chance to run before the tests are bootstrap.

It must be some strange intimacy between Gradle 4.8 and JDK 10 OpenJDK Runtime Environment (build 10.0.1+10) in my case, as the issue only reproduces with:
./gradlew clean check
It does not reproduce with:

• ./gradlew :server:test
• ./gradlew :server:test ran 100 times
• Gradle 4.7
• Gradle 4.8 with JDK 11ea (OpenJDK Runtime Environment 18.9 (build 11-ea+17))

Gradle seems to change the order of tasks a lot and will run many other tasks, including some tests before the :server:tests ( which probably doesn't make sense ). Not sure this is relevant.

In order to be able to move to the new Gradle version #31271 has a work around.
I'm writhing this up in the hopes of revisiting it to provide more clarity.
Note that as of this writing the Gradle 4.8 PR mentioned above is not merged.

Relates to #31230.

[elasticmachine]

Pinging @elastic/es-core-infra

[alpar-t]

the problem is not reproducible with a change such as to alter task ordering:

diff --git a/build.gradle b/build.gradle
index 43d59e7b249..de1facab3ca 100644
--- a/build.gradle
+++ b/build.gradle
@@ -585,3 +585,15 @@ compareGradleBuilds {
     arguments = ["-Dbuild.compare_friendly=true"]
   }
 }
+
+allprojects {
+    afterEvaluate {
+        tasks.all {
+            try {
+                if (it.getPath().startsWith(":server") == false && it.getPath().startsWith(":client") == false && it.getPath().startsWith(":libs") == false && it.getPath().startsWith(":test") ==false && it.getPath().startsWith(":build-tools") == false ) {
+                    it.mustRunAfter { project(':server').test  }
+                }
+            } catch (e) {println e}
+        }
+    }
+}

[alpar-t]

@nik9000 Has some comments on #31271 around this: not reproducible with older versions of Gradle, we may be running the tests with the Gradle runner instead of the RandomizedTestRunner, definitely a must-fix to get this in.

[alpar-t]

I can confirm the root issue is indeed that we are running the tests with the Gradle runner. In fact all the test tasks are with the Gradle runner instead of RandomizedTestingTask.

[nik9000]

Really!? Some of them looked right to me. Weird!

[alpar-t]

I extended the randomized test runner plugin to include a sanity check (commit is d6e4c56, not pushed as of this writing) with this, the problem reproduces in ~15 seconds. I then tried to isolate it in https://github.com/atorok/gradle48_jdk11_replace_task and with this I found that it also reproduces on JDK 10, but apparently less frequently. That means we should have seen it in CI already. I'm not sure why we didn't. Maybe it was such a rare occurrence that it was discounted, or the frequency at which it occurs for the ES build vs. my isolated build is much lower. I'll submit an issue for Gradle, but from the forums I take it that the task replacement is fuzzy and bound to break, there are some APIs but it's not quite a finished feature.

@nik9000 My previous comment is not quite right. I assumed many = all, but it's the case, e.x. from 3 runs I just did, first one had 44 wrong test task, the second one had none, the third one had 42.

[alpar-t]

The reason we are not seeing this in CI is that it's specific to Gradle 4.8