5.4.0 transport client failed to get local cluster state while using 5.3.0 to connect to 5.4.0 servers works

[fangqiao]

Hi, I have a rest service using Netty as basis and connecting to ElasticSearch backend via java transport client API.
It worked very well with Netty 4.1.8 and ES 5.3.0.
Now I tried to upgrade ES backend and transport client to 5.4.0, and also Netty to 4.1.9. Then following problems happened:

10 May 2017;17:01:59.645 Developer linux-68qh [elasticsearch[client][generic][T#3]] INFO o.e.c.t.TransportClientNodesService - failed to get local cluster state for {#transport#-1}{WlTQjgcGQ1uqyNNsw4ZnAw}{127.0.0.1}{127.0.0.1:9300}, disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][127.0.0.1:9300][cluster:monitor/state] request_id [7] timed out after [5001ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:925)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I roll back the transport client to 5.3.0 but keep backend 5.4.0.

Then it is able to connect to Es backend.
I use SBT and the build dependencies for the error are:

"io.netty" % "netty-all" % "4.1.9.Final"
"org.elasticsearch" % "elasticsearch" % "5.4.0"
"org.elasticsearch.client" % "transport" % "5.4.0",
and "io.netty" % "netty-transport-native-epoll" % "4.1.9.Final" classifier "linux-x86_64"

Environment:

openjdk version "1.8.0_121"
OpenJDK Runtime Environment (IcedTea 3.3.0) (suse-3.3-x86_64)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

Linux linux-68qh 4.10.13-1-default #1 SMP PREEMPT Thu Apr 27 12:23:31 UTC 2017 (e5d11ce) x86_64 x86_64 x86_64 GNU/Linux

Thanks

[tlrx]

It looks like a bug to me. Is sniffing enabled on your transport client?

[fangqiao]

Yes it is enabled.

[jloisel]

Same issue here. We have:

• Spring Boot v1.5.3.RELEASE,
• Switched from Elasticsearch 5.3.2 to 5.4.0,
• using Transport Client with sniff enabled.

Client and Elasticsearch both on the same machine, connecting through localhost:

• When using TransportClient 5.3.2 to connect to Elastic 5.4.0 => OK,
• 5.4.0 to 5.4.0 => KO.

The exception we have on startup:

org.elasticsearch.transport.ReceiveTimeoutTransportException: [][127.0.0.1:9300][cluster:monitor/state] request_id [7] timed out after [5000ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:925) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

[nwadams]

I am seeing this issue as well on some nodes connecting to ES. We run a service that has multiple machines that each connect to ES, some of them are able to connect successfully and others do not.

[tlrx]

Thanks for reporting, I think I know where the issue is.

[nwadams]

Thanks @tlrx. I'm not sure if you are also aware, but I also saw errors that looked like the following when I disabled sniffing.

20:38:44.935 [elasticsearch[_client_][generic][T#2]] DEBUG - failed to connect to discovered node [{i-0562d98cb14e42358}{Gzbd-MEzRo-OHMUoEajvXA}{x6V2--f3SS-NzVk5wAQQYg}{10.178.212.242}{127.0.0.1:4374}{aws_availability_zone=us-east-1a}]
ConnectTransportException[[i-0562d98cb14e42358][127.0.0.1:4374] handshake failed. unexpected remote node {i-01bae8d9b0f31ac54}{MUjAv_3JR5KmzEdn-eJeSA}{qJdTT_oaSRCJ1TLO1W2A6w}{10.158.100.27}{10.158.100.27:9300}{aws_availability_zone=us-east-1b}]
	at org.elasticsearch.transport.TransportService.lambda$connectToNode$3(TransportService.java:319)
	at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:466)
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:315)
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:302)
	at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.validateNewNodes(TransportClientNodesService.java:374)
	at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:442)
	at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:358)
	at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:391)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

If it helps we have a service discovery framework to discover services: (https://medium.com/airbnb-engineering/smartstack-service-discovery-in-the-cloud-4b8a080de619). We "randomly" pick an ES box to connect to and then use sniffing (if enabled) to discover the rest. Even though ES is running on 9200/9300 we use a different port on our client machines because of the service discovery framework does the correct routing. Both the service discovery port and the "direct access" port are reachable over the network.

I am rolling back our transport client version to 5.3.2 and will report back on the results.
Update: 5.3.2 works great

[githubdoramon]

Same here... 5.4.0 to 5.4.0 fails.... but 5.3.0 to 5.4.0 works