Networking: Node did not recover from recovered networking issue

[spinscale]

I have one node, that continouosly spit a "NoRouteToHostException" until the JVM is restarted, then everything is fine.

This exception floods the logs, even though the network connectivity has been restored in the meantime (about 5 min outage). Maybe the JVM is caching something here, but not sure where to look.

[2015-XX-YY AA:BB:07,790][WARN ][transport.netty          ] [NODENAME] exception caught on transport layer [[id: 0x9634070f]], closing connection
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Happened on elasticsearch 1.4.x

[rmuir]

There is a 5 minute cache in the JDK for localhost.

[rmuir]

Sorry, i meant 5 second :) Look @ InetAddress.java

[kimchy]

this is really weird, since you can see we do full connection all the way to the SocketImpl every time... .

[spinscale]

@rmuir how do you draw the connection to localhost? This cluster was running on several hosts, no localhost involved in discovery... could the cache still be an issue?

[rmuir]

You mentioned a JVM cache, so I gave you one. I wouldn't dismiss the problem: i see ES calling InetAddress.getLocalHost in inconsistent and buggy ways. In one place it even caches it in a static block as if it can never change. broken...

[kimchy]

@spinscale here is some reference to the caching I was referring to, it revolves around DNS caching done in the JVM:

• http://wiki.apache.org/hadoop/NoRouteToHost
• http://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html
• http://serverfault.com/questions/459814/clearing-tomcats-dns-cache
• http://docs.aws.amazon.com/AWSSdkDocsJava/latest//DeveloperGuide/java-dg-jvm-ttl.html

We might want to consider setting different ttl in our config, unsure yet though, but at least it can explain whats going on.

Regarding the local address caching, its mainly used during startup and for logging. I think we can clean this up, for startup, its ok to simply not use the cache, and conceptually we can do the same for logging, since we set all the loggers anyhow.

[kimchy]

Btw, another reason why this can happen, and this is more connected to how nodes are serialized over the network. When a node starts up, it gets initialized with an address, and when the cluster state is published it gets streamed to the other node, and it uses the actual IP address to construct back the inet address, and not perform another lookup based on host name. So the IP address of a node is set on startup, and then published so it can be connected to.

This can result in this failure as well, and to be honest, its a tricky tradeoff here... . There is a way to force serialization to serialize host name, and then do lookup on each deserialization, its in InetSocketTransportAddress.

Most other systems I have looked at when investigating this were doing the same thing, which is serializing the IP address itself, I think thats why what you see when a change happens is the common recommendation is to restart the JVM (the combination of the JVM cache and this serialization mode for networked systems)

[clintongormley]

Has this been resolved by recent changes?

[jasontedor]

Closed by #21630