TestCluster hits AccessControlException in *rules.RamUsageEstimator

[albertzaharovits]

Builds
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+intake/1185/
and
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-darwin-compatibility/127/

Unreproducible:

/gradlew :modules:reindex:test --tests "org.elasticsearch.index.reindex.ReindexFailureTests.null" \
  -Dtests.seed=BDBD6D0301981618 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=UTC \
  -Dcompiler.java=12 \
  -Druntime.java=8

junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?) and null them, your test still has references to classes of which the sizes cannot be measured due to security restrictions or Java 9 module encapsulation:
  - private static org.elasticsearch.test.TestCluster org.elasticsearch.test.ESIntegTestCase.currentCluster
	at com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.java:146)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.nio.fs")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1564)
	at java.lang.Class.checkPackageAccess(Class.java:2372)
	at java.lang.Class.checkMemberAccess(Class.java:2351)
	at java.lang.Class.getDeclaredFields(Class.java:1915)
	at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator$2.run(RamUsageEstimator.java:585)
	at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator$2.run(RamUsageEstimator.java:582)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.createCacheEntry(RamUsageEstimator.java:582)
	at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.measureSizeOf(RamUsageEstimator.java:545)
	at com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.sizeOfAll(RamUsageEstimator.java:387)
	at com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.java:129)
	... 10 more

[elasticmachine]

Pinging @elastic/es-core-infra

[jaymode]

We discussed this in the core infra meeting. It appears as something is holding onto to a file/filesystem reference in a static field. We do not want to grant this level of access to the RamUsageEstimator. In order to find out what is holding said reference, we need to look at making an improvement in the RamUsageEstimator and/or the test rule to output the class and even the field name when available.

[jaymode]

we need to look at making an improvement in the RamUsageEstimator and/or the test rule to output the class and even the field name when available.

This may not be needed as RamUsageEstimator already does output the top level object that triggers the exception, but the information is easily overlooked. In the cases of these failures, the RamUsageEstimator output - private static org.elasticsearch.test.TestCluster org.elasticsearch.test.ESIntegTestCase.currentCluster as the offending static object. This makes sense as the InternalTestCluster class maintains a reference to a set of Path objects that represent data directories and the actual implementation of the Path objects are in the sun.nio.fs package.

In order to reproduce the issue, I was able to create a test that indexed 1000+ documents asynchronously, then performed a full cluster restart using internalCluster().fullRestart(), while indexing was ongoing, and modified the InternalTestCluster to call Node#awaitClose with a value of 0 seconds instead of 10 seconds, which will trigger an exception to be thrown during the execution of ESIntegTestCase#afterClass, which ultimately leads to the static fields not being cleared. JDK8 is necessary for this as LuceneTestCase disables the StaticFieldsInvariantRule if Java 9 or newer is used.