Pushing auto-follow-pattern results in a Null pointer exception on elasticsearch side #44172
Description
Copy - paste from here https://discuss.elastic.co/t/pushing-auto-follow-pattern-results-in-a-null-pointer-exception-on-elasticsearch-side/189666/2
Elasticsearch version (bin/elasticsearch --version):
6.8.1
Plugins installed: []
none.
JVM version (java -version):
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b04, mixed mode)
OS version (uname -a if on a Unix-like system):
Linux ip-10-176-68-180 4.14.62-65.117.amzn1.x86_64 #1 SMP Fri Aug 10 20:03:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
So, we installed elasticsearch 6.8.1 on 6 nodes and are trying to setup CCR with two clusters - 3 nodes each. License is activated and valid, remote settings are set (seed, nodes and cluster name). But now, somehow when I try to push the autofollow pattern like this
curl -X PUT "localhost:9200/_ccr/auto_follow/AutoFollowerPatterns" -H 'Content-Type: application/json' -d'
{
"remote_cluster" : "elastic-uswest",
"leader_index_patterns" :
[
"visit*"
],
"follow_index_pattern" : "ccr-{{leader_index}}"
}
'
it results into that response:
{
"error" : {
"root_cause" : [
{
"type" : "remote_transport_exception",
"reason" : "[10.176.70.184][10.176.70.184:9300][cluster:admin/xpack/ccr/auto_follow_pattern/put]"
}
],
"type" : "null_pointer_exception",
"reason" : null
},
"status" : 500
}
Digging a bit deeper into the logs of elasticsearch, we can see this message:
[2019-07-09T10:10:49,510][WARN ][r.suppressed ] [10.176.68.180] path: /_ccr/auto_follow/AutoFollowerPatterns, params: {name=AutoFollowerPatterns}
org.elasticsearch.transport.RemoteTransportException: [10.176.69.160][10.176.69.160:9300][cluster:admin/xpack/ccr/auto_follow_pattern/put]
Caused by: java.lang.NullPointerException
at org.elasticsearch.xpack.ccr.CcrLicenseChecker.hasPrivilegesToFollowIndices(CcrLicenseChecker.java:318) ~[?:?]
at org.elasticsearch.xpack.ccr.action.TransportPutAutoFollowPatternAction.lambda$masterOperation$2(TransportPutAutoFollowPatternAction.java:88) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1104) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:985) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:977) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:952) [elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:763) [elasticsearch-6.8.1.jar:6.8.1]
...
Our initial thought was, that the security module could cause the error - because of this word in the exception: hasPrivilegesToFollowIndices. So, we removed all unnecessary modules and the error still occurs.
Installed modules:
Steps to reproduce:
Sadly, I cannot reproduce the bug. When I tried it, it always worked fine. It happened on this one particular environment and now we cannot fix it.