elastic 5.1.1 master node always pinged time out, failed to form a cluster

[sinsonglew]

Elasticsearch version: 5.1.1

Plugins installed: [] only default

JVM version: Java(TM) SE Runtime Environment (build 1.8.0_111-b14)

OS version: 2.6.32-504.12.2.02.el6.x86_64 #1 SMP Tue May 12 11:44:09 CST 2015 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
if nodes greater than 3, the cluster is constantly add and remove nodes, leading to unavailable status.
From the following attached log, nodes ping each other timed out often
expected: a full-nodes started cluster

Provide logs (if relevant):

[2016-12-15T19:39:17,648][INFO ][o.e.c.s.ClusterService   ] [node244] removed {{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: master_failed ({node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371})
[2016-12-15T19:39:42,556][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: NodeDisconnectedException[[node244][ee.ff.gg.hh:9371][internal:discovery/zen/join/validate] disconnected]; ]
[2016-12-15T19:40:02,565][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: ConnectTransportException[[node244][ee.ff.gg.hh:9371] connect_timeout[30s]]; nested: IOException[Connection timed out: ee.ff.gg.hh/ee.ff.gg.hh:9371]; ]
[2016-12-15T19:40:25,433][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: NodeDisconnectedException[[node244][ee.ff.gg.hh:9371][internal:discovery/zen/join/validate] disconnected]; ]
[2016-12-15T19:40:42,442][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [NodeDisconnectedException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join] disconnected]]
[2016-12-15T19:41:03,531][INFO ][o.e.c.s.ClusterService   ] [node244] detected_master {node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}, added {{node241}{gx3rLPdIQvSm_u6mLoIbtQ}{80MXRZOjSN6RAQ7gZGeUcQ}{xx.x.xx.x}{x.x.x.x:9371},{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: zen-disco-receive(from master [master {node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371} committed version [115]])
[2016-12-15T19:41:09,541][INFO ][o.e.d.z.ZenDiscovery     ] [node244] master_left [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [failed to ping, tried [3] times, each with  maximum [30s] timeout]
[2016-12-15T19:41:09,541][WARN ][o.e.d.z.ZenDiscovery     ] [node244] master left (reason = failed to ping, tried [3] times, each with  maximum [30s] timeout), current nodes: nodes: 
   {node243}{tEv4PrXSTdyecz2nClog4Q}{KQsJYnauSe2W0hxWPte4Iw}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}
   {node239}{oWLoFJf-Rs-3q-kK1Y_2BQ}{dekHpx5zQASxKgIO21n1Yg}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}
   {node244}{stop1h4jQc-zCvM2hBS56Q}{iV8QrIRiQ5y3cvW_en0sMA}{ee.ff.gg.hh}{ee.ff.gg.hh:9371}, local
   {node241}{gx3rLPdIQvSm_u6mLoIbtQ}{80MXRZOjSN6RAQ7gZGeUcQ}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}
   {node242}{k5vCKHcjTSKTK-uhK_MtLw}{DMGU0D-8T-Si-ZpfqmiDTQ}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}

[2016-12-15T19:41:09,542][INFO ][o.e.c.s.ClusterService   ] [node244] removed {{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: master_failed ({node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371})
[2016-12-15T19:41:28,555][INFO ][o.e.c.s.ClusterService   ] [node244] master {new {node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}}, removed {{node241}{gx3rLPdIQvSm_u6mLoIbtQ}{80MXRZOjSN6RAQ7gZGeUcQ}{xx.xx.xx.xx}{x.x.x.x:9371},}, added {{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: zen-disco-receive(from master [master {node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371} committed version [120]])
[2016-12-15T19:41:34,564][INFO ][o.e.d.z.ZenDiscovery     ] [node244] master_left [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [failed to ping, tried [3] times, each with  maximum [30s] timeout]
[2016-12-15T19:41:34,564][WARN ][o.e.d.z.ZenDiscovery     ] [node244] master left (reason = failed to ping, tried [3] times, each with  maximum [30s] timeout), current nodes: nodes: 
   {node243}{tEv4PrXSTdyecz2nClog4Q}{KQsJYnauSe2W0hxWPte4Iw}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}
   {node239}{oWLoFJf-Rs-3q-kK1Y_2BQ}{dekHpx5zQASxKgIO21n1Yg}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}
   {node244}{stop1h4jQc-zCvM2hBS56Q}{iV8QrIRiQ5y3cvW_en0sMA}{ee.ff.gg.hh}{ee.ff.gg.hh:9371}, local
   {node242}{k5vCKHcjTSKTK-uhK_MtLw}{DMGU0D-8T-Si-ZpfqmiDTQ}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}

[2016-12-15T19:41:34,565][INFO ][o.e.c.s.ClusterService   ] [node244] removed {{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: master_failed ({node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371})
[2016-12-15T19:41:55,575][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: ConnectTransportException[[node244][ee.ff.gg.hh:9371] connect_timeout[30s]]; nested: IOException[Connection timed out: ee.ff.gg.hh/ee.ff.gg.hh:9371]; ]
[2016-12-15T19:42:15,586][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: ConnectTransportException[[node244][ee.ff.gg.hh:9371] connect_timeout[30s]]; nested: IOException[Connection timed out: ee.ff.gg.hh/ee.ff.gg.hh:9371]; ]

[clintongormley]

Looks like you have networking problems. I don't know if you have any customs settings, or where you're running Elasticsearch etc. More info would be helpful.

[sinsonglew]

@clintongormley , total 8 nodes, three of them are master eligible. we run elastic on virtual machines. It's definitely that network is fine. we tried elastic 1.x, it works very well. elasticsearch.yml posted as follows:

cluster.name: demo
node.name: node237

network.host: 10.xx.85.xx
transport.tcp.port: 9371
http.port: 9200

discovery.zen.ping.unicast.hosts: ["10.xx.85.aa:9371", "10.xx.85.bb:9371", "10.xx.85.cc:9371"]
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping_timeout: 17s

gateway.recover_after_nodes: 8

[clintongormley]

1.x used multicast, you may well find that it was using different network cards etc than the ones configured. It still looks to me like you have a networking issue here, or possibly different config on one of the nodes?

[georgecao]

@clintongormley , I work with @sinsonglew on the same team. Does ElasticSearch 5.1.1 require specific network cards? What's your concerns and how we check it out? If we have 10 nodes to form one cluster, what's elasticsearch.yml should look like? It's wired that we reduce the number of nodes from 10 to 3, then the cluster becomes stable, but increase the number of nodes from 3 to 4 or 5, it happens again.

Any idea?
Thanks in advance.

[bleskes]

@georgecao I think there are a couple of things going on.

First you say you have 10 nodes, I presume you don't run dedicated master nodes. If that's the case then setting discovery.zen.minimum_master_nodes to 2 is dangerous as your 10 nodes can split to form 3 clusters of 3 nodes each. You should set it to 6 (a majority). That said, it's not clear to me why this explains network disconnects.

It seems the nodes discover the master fine (and try to join it), so I don't think it's a question of multicast based discovery.

It seems something is off with the network there are a lot of timeouts and disconnect errors in the log you shared. Since you masked the IPs (good!) it's hard for me to check they all make sense. In ES each node has the local ip it binds to and the IP it publishes (i.e., the IP people should use to connect to it). Can you check those make sense?

Finally, I was wondering how you ended up with 17s in discovery.zen.ping_timeout: 17s ?

[sinsonglew]

@boaz, at first, 10 nodes without dedicated roles settings were experimented, the minimum master nodes set up with 6, we can't get a full started cluster, so we have to decrease the total nodes to 3, it works, then 4, it failed. We have another dc, so 8 nodes with 3 master eligible got tried again, it repeated. We can't figure out where there is something wrong. Thanks! 发自我的 Windows Phone

…

________________________________ From: Boaz Leskes<mailto:notifications@github.com> Sent: ‎2016/‎12/‎17 4:43 To: elastic/elasticsearch<mailto:elasticsearch@noreply.github.com> Cc: sinsonglew<mailto:sinsonglew@live.com>; Mention<mailto:mention@noreply.github.com> Subject: Re: [elastic/elasticsearch] elastic 5.1.1 master node always pinged time out, failed to form a cluster (#22189) @georgecao<https://github.com/georgecao> I think there are a couple of things going on. First you say you have 10 nodes, I presume you don't run dedicated master nodes. If that's the case then setting discovery.zen.minimum_master_nodes to 2 is dangerous as your 10 nodes can split to form 3 clusters of 3 nodes each. You should set it to 6 (a majority). That said, it's not clear to me why this explains network disconnects. It seems the nodes discover the master fine (and try to join it), so I don't think it's a question of multicast based discovery. It seems something is off with the network there are a lot of timeouts and disconnect errors in the log you shared. Since you masked the IPs (good!) it's hard for me to check they all make sense. In ES each node has the local ip it binds to and the IP it publishes (i.e., the IP people should use to connect to it). Can you check those make sense? Finally, I was wondering how you ended up with 17s in discovery.zen.ping_timeout: 17s ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#22189 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMTJ-MTYkyb5ivp6gt3i9wUbXxWWbTAnks5rIvfvgaJpZM4LOBBu>.

[bleskes]

@sinsonglew everything points at network issues. Can you check the IPs of the published address?

[sinsonglew]

@bleskes, @clintongormley, we downgrade netty from 4 to 3 by setting transport.type=netty3
http.type=netty3 , problem sovled, it's not a network issue.

[bleskes]

we downgrade netty from 4 to 3 by setting transport.type=netty3
http.type=netty3 , problem sovled, it's not a network issue.

Thank you for this clue. We will investigate and get back to you as soon as possible.

[jasontedor]

Can you share some more details:

• node specs (especially heap size but general specs as well)
• from your kernel version I think that you're running RHEL 6, can you confirm?
• logs from a node and the master (there is no need to mask the IP addresses, from your configuration I see that you're in 10.0.0.0/8 which is a private IP address space; masking will just make it harder-than-necessary to read)
• are the nodes accessible over HTTP (e.g., GET /)?
• what is going on on the cluster when this happens (heavy indexing, searching, etc.)?