Unicast hostname failing to resolve blocks cluster from starting

[wkruse]

Hi folks,

there seems to be a regression from 1.7.3 to 2.0.0 regarding the gateway.recover_after_time setting.

docker run --name="elasticsearch" -p 9200:9200 -p 9300:9300 -e "SERVICE_ID=$hostname" -e "SERVICE_NAME=elasticsearch" elasticsearch:2.0.0 -Des.index.number_of_shards=3 -Des.index.number_of_replicas=2 -Des.gateway.recover_after_nodes=2 -Des.gateway.expected_nodes=3 -Des.gateway.recover_after_time=5m -Des.discovery.zen.ping.multicast.enabled=false -Des.discovery.zen.ping.unicast.hosts="01.elasticsearch,02.elasticsearch,03.elasticsearch" -Des.cluster.name="logging-test" -Des.network.publish_host="${PUBLIC_IP}"

I run Elasticsearch on three nodes (Docker hosts). The DNS names are not immediatly available. In 1.7.3 the cluster stabilizes after a couple of minutes. In 2.0.0 the container crashes directly.

Exception in thread "main" java.lang.IllegalArgumentException: Failed to resolve address for [02.elasticsearch]
Likely root cause: java.net.UnknownHostException: 02.elasticsearch: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.elasticsearch.transport.netty.NettyTransport.parse(NettyTransport.java:668)
at org.elasticsearch.transport.netty.NettyTransport.addressesFromString(NettyTransport.java:620)
at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:398)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.<init>(UnicastZenPing.java:141)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at <<<guice>>>
at org.elasticsearch.node.Node.<init>(Node.java:198)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

[dakrone]

@wkruse it looks like this doesn't have anything to do with the gateway.recover_after_time setting. In 2.0.0 the networking is more strict regarding networking settings. In your case, one of your unicast hosts, 02.elasticsearch fails to resolve to an IP address.

[meatheadmike]

Is there an option to turn off the exception that occurs when host resolution fails? In 1.7 I used to set up a list of hosts and then add them as needed (it is "elastic" search after all). But now in 2.0 I have to have all of my hosts preconfigured with DNS or I get an exception and ES wont even start :(

[wkruse]

I've had it similar to @meatheadmike. CoreOS cluster and DNS service discovery per Elasticsearch container. The current solution is to use the hostnames for unicast.

[dakrone]

I think the unicast hosts not resolving shouldn't fail the node starting up, what do you think @bleskes ?

[meatheadmike]

Here's the current scenario and why it's an issue to me...

I spin up my nodes in Openstack. When I spin up a new node, I name it right then and there. So the DNS name doesn't exist prior. Neither does the IP address.

My current solution is to feed a list of potential node names in to a bash script, cycle through them and remove the ones that don't respond prior to creating the discovery.zen.ping.unicast.hosts entry in elasticsearch.yml.

It's an ugly hack, but it works. It allows me to spin up new nodes without worrying about what currently exists and what doesn't. But this used to not be necessary :(

[bleskes]

I think the unicast hosts not resolving shouldn't fail the node starting up, what do you think @bleskes ?

I'm a bit wary of this as it may also mean a configuration typo and which will cause the node from joining the cluster and one needs to go grep the logs to find out what's going on...

Normally the answer would be to make the unicast host dynamically updatable. This doesn't work cleanly here because those settings can not be persisted in the cluster state - they are used before a node joins the cluster and has no access to it.

I'll munch on this some more. Ideas welcome.

[majormoses]

This poses a problem for anyone using CM such as chef or puppet. With the way it currently is if a node is rebooting that means that you must deregister it from CM and bootstrap on launch again as well as every node to converge to remove it from its list. Otherwise the ES service on any node cant be (re)started until the machine comes back up. This seems to me at least to be far more common an issue than a misconfiguration of a list of hosts or ips.

[rmuir]

The same logic to fail on invalid host was in 1.7, but it was just buggy:

https://github.com/elastic/elasticsearch/blob/1.7/src/main/java/org/elasticsearch/discovery/zen/ping/unicast/UnicastZenPing.java#L141-L143

From what I remember, hostname resolution at a lower level had several bugs, in such a way that unknown hosts wouldn't always deliver exceptions, e.g. be silently treated as a wildcard address (0.0.0.0) instead.

[rmuir]

It might have been silent conversion to 127.0.0.1 instead, can't remember which. The problems were something sneaky, like an unexpected null value -> InetAddress.getByName(null) -> 127.0.0.1 or something like that. http://docs.oracle.com/javase/7/docs/api/java/net/InetAddress.html#getByName%28java.lang.String%29

[rmuir]

The other big problem was use of https://docs.oracle.com/javase/7/docs/api/java/net/InetSocketAddress.html#InetSocketAddress%28java.lang.String,%20int%29, but then not dealing with the 'unresolved' case correctly or at all across the codebase. Then the return method of null from the unresolved address would be treated the same as 0.0.0.0 for some methods.

If we want to change the behavior for pings, lets please not go back to using that trappy ctor (which is banned in forbidden apis for a reason). I would also greatly prefer if we didn't create unresolved InetSocketAddresses in general (so lets ban https://docs.oracle.com/javase/7/docs/api/java/net/InetSocketAddress.html#createUnresolved%28java.lang.String,%20int%29 too).

The reason is, I think we should always explicitly do DNS lookups -> addresses ourselves, this is an important operation, and need not to be "mixed in" with a convenience holder class that combines an IP address with a port number (InetSocketAddress), especially in the very sneaky lenient and trappy way it handles the case.

So for pings, it would mean that we'd just use String and ping would explicitly do DNS lookup itself if that is what we want to do.