[TEST] SecurityRealmSettingsTests failing on Windows

[ywelsch]

Build scan: https://scans.gradle.com/s/o3smubtjyn4gu

Relevant output:

org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests > classMethod FAILED
    java.io.IOException: Could not remove the following files (in the order of attempts):
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.2+multijob-windows-compatibility\os\windows-2012-r2\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_E46537D453124597-001\jwkset-005json: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.2+multijob-windows-compatibility\os\windows-2012-r2\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_E46537D453124597-001\jwkset-005json: The process cannot access the file because it is being used by another process.

       C:\Users\jenkins\workspace\elastic+elasticsearch+7.2+multijob-windows-compatibility\os\windows-2012-r2\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_E46537D453124597-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.2+multijob-windows-compatibility\os\windows-2012-r2\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_E46537D453124597-001

[elasticmachine]

Pinging @elastic/es-security

[jkakavas]

Seems similar to #44006 and #44552

[droberts195]

This happened again in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.3+multijob-windows-compatibility/os=windows-2016/69/console:

2> java.io.IOException: Could not remove the following files (in the order of attempts):
     C:\Users\jenkins\workspace\elastic+elasticsearch+7.3+multijob-windows-compatibility\os\windows-2016\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_C3CCAFF8D4417098-001\jwkset-001json: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.3+multijob-windows-compatibility\os\windows-2016\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_C3CCAFF8D4417098-001\jwkset-001json: The process cannot access the file because it is being used by another process.

Seems similar to #44006 and #44552

Maybe #44006 is a bit different, because that's "directory not empty" and probably caused by async code creating a file in a directory during the recursive deletion of the temporary directory hierarchy after the loop thinks it's deleted all the files in the directory.

I agree this is similar to #44552 in that both are breaches of the Windows rule that you can't delete an open file (unless it was specifically opened in a mode that allows it to be deleted while open). However, unfortunately every place where we're deleting open files could require a separate code change to fix - it's not like there's a single magic line of code we can change to make all these Windows problems go away.

[tvernum]

Another failure
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-windows-compatibility/os=windows-2019/110/console

java.io.IOException
: 
Could not remove the following files (in the order of attempts):
   C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2019\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_4CD7B58A56A26CBC-001\jwkset-001json: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2019\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_4CD7B58A56A26CBC-001\jwkset-001json: The process cannot access the file because it is being used by another process.


   C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2019\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_4CD7B58A56A26CBC-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2019\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_4CD7B58A56A26CBC-001
at __randomizedtesting.SeedInfo.seed([4CD7B58A56A26CBC]:0)
at org.apache.lucene.util.IOUtils.rm(IOUtils.java:319)
at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)

[alpar-t]

Another one: https://gradle-enterprise.elastic.co/s/b25nyp3zemcxc/tests/g5u3xltc4r2du-tn72fxt3goumw

[romseygeek]

This has failed again on the 7.5 branch on Windows, which is deeply weird because it ought to be muted - the 7.5 branch has an assumeFalse check for Constants.WINDOWS:

elasticsearch/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/SecurityRealmSettingsTests.java

Line 131 in c39c1f4

assumeFalse("https://github.com/elastic/elasticsearch/issues/44942", Constants.WINDOWS);

[romseygeek]

Relevant build scan: https://gradle-enterprise.elastic.co/s/y56ncmvtmm46q/

[alpar-t]

at __randomizedtesting.SeedInfo.seed([171B268676DCC25C]:0)
at org.apache.lucene.util.IOUtils.rm(IOUtils.java:319)
at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)

Looks like the failure is coming from a cleanup rule that is executed despite of the test method being skipped. We could AFAIK skip the suite by moving the assumeFalse in a @BeforeClass