Elasticsearch version 6.8.6 oss unable to start

[nnbrandon]

Elasticsearch version (bin/elasticsearch --version): Version: 6.8.6, Build: oss/rpm/3d9f765/2019-12-13T17:11:52.013738Z, JVM: 1.8.0_181

Plugins installed: []

JVM version (java -version):
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

OS version (uname -a if on a Unix-like system): Linux

Description of the problem including expected versus actual behavior:
Hi all,

I am trying to do a fresh install of elasticsearch-oss-6.8.6 with OpenJDK version 1.8.0_181. An elasticsearch-oss-6.5 was able to successfully install before but I am receiving errors related to the keystore and trust manager. There were no issues related to the trust manager and keystore before. It seems like for some reason, elasticsearch is trying to do SSL configurations but we never specified it. There is no SSL options or configurations in the elasticsearch.yml file.

Steps to reproduce:

1. Download the https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-6.8.6.rpm
2. I had to rebuild the rpm file with "Requires: java-1.8.0-openjdk"
3. Build and install elasticsearch

Provide logs (if relevant):

[2020-01-13T19:11:29,780][ERROR][o.e.b.Bootstrap          ] [xSBU6-2] Exception
org.elasticsearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore
	at org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:70) ~[?:?]
	at org.elasticsearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:109) ~[?:?]
	at org.elasticsearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:145) ~[?:?]
	at org.elasticsearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:115) ~[?:?]
	at org.elasticsearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:88) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$11(Node.java:472) ~[elasticsearch-6.8.6.jar:6.8.6]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_181]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_181]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_181]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_181]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_181]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_181]
	at org.elasticsearch.node.Node.<init>(Node.java:475) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.6.jar:6.8.6]
Caused by: java.security.KeyStoreException: problem accessing trust storejava.io.IOException: Invalid keystore format
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:74) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_181]
	at org.elasticsearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:151) ~[?:?]
	at org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:68) ~[?:?]
	... 24 more

Log File:
elasticsearch.log

[nnbrandon]

Based off of this call stack,

Version 6.8: https://github.com/elastic/elasticsearch/blob/6.8/server/src/main/java/org/elasticsearch/node/Node.java#L472, elasticsearch is trying to create a component for ReindexPlugin.

Version 6.8: https://github.com/elastic/elasticsearch/blob/6.8/modules/reindex/src/main/java/org/elasticsearch/index/reindex/ReindexPlugin.java#L88, it calls a function ReindexSslConfig() and I am unsure as to why it is trying to config it with SSL.

Version 6.5, https://github.com/elastic/elasticsearch/blob/6.5/modules/reindex/src/main/java/org/elasticsearch/index/reindex/ReindexPlugin.java, there is no mentions of ReindexSslConfig() and this is mainly due to version 6.5 not supporting the library ssl-config.

Why is elasticsearch trying to configure out ReindexPlugin with SSL even though this is an OSS version?

[nnbrandon]

@tvernum, do you possibly know what is going on here? I saw that you created this PR wanted to get your input if possible. We did not introduce any SSL settings to reindex from remote.

[tvernum]

Why is elasticsearch trying to configure out ReindexPlugin with SSL even though this is an OSS version?

Open Source Elasticsearch has always (well, for a very long time) supported reindexing over https. In 6.7 we added support for configuring the SSL parameters for that connection.

So it makes sense that you see a differne between 6.8 and 6.5

Can you give an exact OS version and/or JDK package name?
OS vendors repackage openjdk and CA Certs is one of the areas that they sometimes fiddle with, so if I'm going to reproduce this I'll need some more info on the environment.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Reindex)

[elasticmachine]

Pinging @elastic/es-security (:Security/Network)

[tvernum]

In your log file I see

 -Djava.security.properties==/ci/hubble/elasticsearch-java.security

My strong suspicion is that your custom security properties are the root cause of this issue.
Are you able to provide a copy of that file?

[nnbrandon]

In your log file I see

 -Djava.security.properties==/ci/hubble/elasticsearch-java.security

My strong suspicion is that your custom security properties are the root cause of this issue.
Are you able to provide a copy of that file?

ssl.KeyManagerFactory.algorithm=SunX509
ssl.TrustManagerFactory.algorithm=PKIX

I believe that these two lines in there may be the reason why. I'll give it an attempt without it and will ping the results.

[nnbrandon]

Why is elasticsearch trying to configure out ReindexPlugin with SSL even though this is an OSS version?

Open Source Elasticsearch has always (well, for a very long time) supported reindexing over https. In 6.7 we added support for configuring the SSL parameters for that connection.

So it makes sense that you see a differne between 6.8 and 6.5

Can you give an exact OS version and/or JDK package name?
OS vendors repackage openjdk and CA Certs is one of the areas that they sometimes fiddle with, so if I'm going to reproduce this I'll need some more info on the environment.

OS Version:

cat /etc/os-release
NAME="MgmtOS"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="MgmtOS 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

JDK:

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

Thank you @tvernum

[nnbrandon]

@tvernum I didn't have much luck removing the lines I indicated. Here are the contents of the security properties file. Thank you ahead of time for taking a look.

security-properties.txt

[tvernum]

I can't reproduce this using the security properties you posted, so it looks like it will be something more subtle.
I'll try and work out a reproduction but I don't have any promises on timeframe.

[nnbrandon]

@tvernum, I was able to find out the problem and it was from the security properties file that you had suspicions on. It was mainly due to a keystore mismatch with Java's keystore. I'll go ahead and close this issue. Thank you for taking a look at it.