[ci] corrupted keystore on java8 fips build

[andyb-elastic]

7.x periodic fips builds have been failing since 19 April with this, unsure if the master equivalent is also effected as it has not run since 7 April

org.elasticsearch.gradle.BuildExamplePluginsIT testCurrentExamplePlugin
org.elasticsearch.gradle.BuildExamplePluginsIT testCurrentExamplePlugin
org.elasticsearch.gradle.BuildExamplePluginsIT testCurrentExamplePlugin
org.elasticsearch.gradle.BuildExamplePluginsIT testCurrentExamplePlugin
org.elasticsearch.gradle.BuildExamplePluginsIT testCurrentExamplePlugin
org.elasticsearch.gradle.BuildExamplePluginsIT testCurrentExamplePlugin
org.elasticsearch.gradle.BuildExamplePluginsIT testCurrentExamplePlugin

  2> REPRODUCE WITH: ./gradlew :integTestRunner --tests "org.elasticsearch.example.painlesswhitelist.PainlessWhitelistClientYamlTestSuiteIT.test {yaml=painless_whitelist/30_static/custom static imported method}" -Dtests.seed=927E287605F73A70 -Dtests.security.manager=true -Dtests.locale=mt -Dtests.timezone=Egypt -Dcompiler.java=12 -Druntime.java=8FIPS
  2> java.lang.IllegalStateException: could not create the default ssl context

        Caused by:
        java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

            Caused by:
            java.security.KeyStoreException: problem accessing trust store

                Caused by:
                java.io.IOException: BCFKS KeyStore corrupted: MAC calculation failed.

Earliest https://scans.gradle.com/s/pvn346idu54go/tests/orjfcz7hu5lqq-o3tyn5hvqpucy
Most recent (slightly different stack trace) https://scans.gradle.com/s/kzcihhmmfxx54/tests/orjfcz7hu5lqq-axhth7m5yd5sq

[elasticmachine]

Pinging @elastic/es-core-infra

[elasticmachine]

Pinging @elastic/es-security

[jkakavas]

This error most probably means that the system property with the keystore/truststore password is not added to the JVM argline (as it should when running in a FIPS 140-2 JVM) when spinning up the test cluster, and not that something is wrong with the failing test itself.

I'll ping @atorok in case he has an idea from the top of his head as to whether one of the recent changes might have caused this.

[alpar-t]

This was cause by #41199 . The example plugins when built stand alone do not get the FIPS specific configuration. The realistic scenario that the test would represent is a testing an external ( as in not included in the elasticsearch project ) plugin with a FIPS enabled JVM.

[danielmitterdorfer]

We have further test failures (e.g. https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8fips,nodes=immutable&&linux&&docker/157/console). As the failure is identical and the reason is known I will not include additional detail because it would just add noise.

@atorok as this failure is systematic is it possible to skip this test when running in a FIPS JVM for the time being? In the build-tools subproject we don't have the usual infrastructure (assumeFalse) in place so I am unsure how it can be achieved best.

[tvernum]

The fix doesn't seem to have resolved the problem.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8fips,nodes=immutable&&linux&&docker/191/

04:33:37 org.elasticsearch.gradle.JdkDownloadPluginIT > testWindowsExtractionOldVersion FAILED
04:33:37     java.security.KeyStoreException at JdkDownloadPluginIT.java:92
04:33:37         Caused by: java.io.IOException at JdkDownloadPluginIT.java:92
04:33:37             Caused by: java.security.UnrecoverableKeyException at JdkDownloadPluginIT.java:92
04:33:50 
04:33:50 org.elasticsearch.gradle.BuildExamplePluginsIT > testCurrentExamplePlugin {p0=/var/lib/jenkins/workspace/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA/openjdk12/ES_RUNTIME_JAVA/java8fips/nodes/immutable&&linux&&docker/plugins/examples/custom-suggester} FAILED
04:33:50     org.gradle.testkit.runner.UnexpectedBuildFailure at BuildExamplePluginsIT.java:91
04:33:51 

* What went wrong:
Execution failed for task ':dummyPlugin:compileJava'.
 > Could not resolve all files for configuration ':dummyPlugin:compileClasspath'.
   > Could not resolve org.locationtech.spatial4j:spatial4j:0.7.
     Required by:
         project :dummyPlugin
      > Could not resolve org.locationtech.spatial4j:spatial4j:0.7. 
         > Could not get resource 'https://jcenter.bintray.com/org/locationtech/spatial4j/spatial4j/0.7/spatial4j-0.7.pom'. 
            > org.apache.http.ssl.SSLInitializationException: Keystore was tampered with, or password was incorrect