fatal error on the network layer on es + kibana dashboard system

[lks21c]

Hi, I'm running 11 physical nodes with es and kibana to serve dashboard system using my company's big data.

I indexed about 500GB data into es and maintain 3 masters, 11 data nodes and 2 client nodes(to avoid cluster failure and to make aggr efficent).

My problem is that es client node(which connects to kibana) frequently dies with below logs.

I suspect client node drops after serving heavy aggregation request.
But, I'm suspicious why fatal error on the network layer occurs because of heavy aggr.

So I want to know the root cause and some configuration to avoid this issue.
I already set memory lock in elasticsearch.yml but is there anything else that I can do?

Thanks for reading this.

Elasticsearch version (bin/elasticsearch --version):
5.5.0

Plugins installed: [x-pack]

JVM version (java -version):oracle jdk 1.8.0_72

OS version (uname -a if on a Unix-like system):centos 6.8

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. I think this happens when heavy aggr from kibana is requested.

Provide logs (if relevant):

[2017-08-07T12:41:03,612][INFO ][o.e.d.z.ZenDiscovery     ] [client-mpalyes06] master_left [{master-mpalyes01}{0eS18FIcSsSPWBAiMEqQLA}{VmvWR8a2SXKJsIdJ_5QCYQ}{mpalyes01}{172.18.100.209:19010}], reason [failed to ping, tried [3] times, each with  maximum [30s] timeout]
[2017-08-07T12:41:03,621][WARN ][o.e.d.z.ZenDiscovery     ] [client-mpalyes06] master left (reason = failed to ping, tried [3] times, each with  maximum [30s] timeout), current nodes: nodes: 
   {master-mpalyes01}{0eS18FIcSsSPWBAiMEqQLA}{VmvWR8a2SXKJsIdJ_5QCYQ}{mpalyes01}{172.18.100.209:19010}, master
   {data-mpalyes04}{HHak7ledRhSnbc3X5H6obA}{TNp7jFQ5QJqaUeag5yl9GA}{mpalyes04}{172.18.100.212:19110}
   {data-mpalyes08}{SfKsA6_mT2S6Tfx8cgKnAw}{YLEYX3nvQpmLQBxBoGc7Hw}{mpalyes08}{172.18.100.192:19110}
   {data-mpalyes06}{BkPlDj4sTPqgWRB7pfCmgQ}{vRks3Oy6SKW7nd9c-P7RyA}{mpalyes06}{172.18.100.214:19110}
   {data-mpalyes10}{5zQG38wnRcuC3ga4YoBbMQ}{I1bUw2eLRju58Ar1DPLTcw}{mpalyes10}{172.18.100.193:19110}
   {client-mpalyes05}{HIYj1dn8S8mURRiXJNoVQg}{YtmkHe0zRYepYuqdDwAfgQ}{mpalyes05}{172.18.100.213:19210}
   {data-mpalyes03}{DfTF2Ti4S_WqP4MxWyRMHA}{882rm-0hSRKZnxUM1XRzsA}{mpalyes03}{172.18.100.211:19110}
   {data-mpalyes11}{mHQnmXiBS_WVn4Pu9OlFhA}{gAaGWX7WSbSYI6lVdxkngQ}{mpalyes11}{172.18.100.195:19110}
   {data-mpalyes01}{fOyjaXFwStC84D5ulRSjaQ}{k42YkNN7QAe2oe8dzQtvxg}{mpalyes01}{172.18.100.209:19110}
   {data-mpalyes09}{gQN9UN2XSm2-VtLhGlt1BA}{UXYy3a9yRDC5y6QihliHnQ}{mpalyes09}{172.18.100.194:19110}
   {data-mpalyes05}{RpR888-jR36OkUVekNC9pg}{ciWDC1epSEyf7-xImHaEaA}{mpalyes05}{172.18.100.213:19110}
   {client-mpalyes06}{RFsf_xhhRaC2l9RQ1NfYrw}{MlekE8HlRrKsRYdkIUF_jg}{mpalyes06}{172.18.100.214:19210}, local
   {data-mpalyes07}{C-CsxnULShObXi6m7G-V_Q}{aQb2kg2BQgS8lzMyrfyMXA}{mpalyes07}{172.18.100.131:19110}
   {data-mpalyes02}{Qf7U_EK5QaewJAeEASzpCA}{mc8FfITVSzaSFr7cJ9yZbQ}{mpalyes02}{172.18.100.210:19110}
   {master-mpalyes03}{r53e2-F3T42tjgKJuTF8sg}{_rGpkV6vScmpCVczu6QCnw}{mpalyes03}{172.18.100.211:19010}
   {master-mpalyes02}{ZdhNZbDbQJCWEazRDav_6w}{6rfj1cwhT6SztvdQsQmwqQ}{mpalyes02}{172.18.100.210:19010}

[2017-08-07T12:41:03,712][ERROR][o.e.t.n.Netty4Utils      ] fatal error on the network layer
	at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:179)
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:83)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
	at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:745)
[2017-08-07T12:41:03,714][ERROR][o.e.t.n.Netty4Utils      ] fatal error on the network layer
	at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:179)
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:83)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
	at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:745)
[2017-08-07T12:41:03,715][ERROR][o.e.t.n.Netty4Utils      ] fatal error on the network layer
	at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:179)
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:83)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
	at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:745)
[2017-08-07T12:41:03,731][ERROR][o.e.t.n.Netty4Utils      ] fatal error on the network layer
	at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:179)
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:83)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
	at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:745)
[2017-08-07T12:41:03,729][ERROR][o.e.t.n.Netty4Utils      ] fatal error on the network layer
	at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:179)
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:83)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
	at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:745)
[2017-08-07T12:41:03,789][WARN ][c.f.s.s.t.SearchGuardSSLNettyTransport] [client-mpalyes06] Someone (/172.18.97.58:56650) speaks transport plaintext instead of ssl, will close the channel
[2017-08-07T12:41:03,724][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [client-mpalyes06] fatal error in thread [Thread-8], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-08-07T12:41:17,707][WARN ][o.e.m.j.JvmGcMonitorService] [client-mpalyes06] [gc][old][8507][53] duration [13.6s], collections [1]/[13.9s], total [13.6s]/[7.6m], memory [19.7gb]->[19.8gb]/[19.8gb], all_pools {[young] [1.4gb]->[1.4gb]/[1.4gb]}{[survivor] [81.6mb]->[180.8mb]/[191.3mb]}{[old] [18.1gb]->[18.1gb]/[18.1gb]}
[2017-08-07T12:41:17,709][WARN ][o.e.m.j.JvmGcMonitorService] [client-mpalyes06] [gc][8507] overhead, spent [13.6s] collecting in the last [13.9s]
[2017-08-07T12:42:33,964][WARN ][o.e.m.j.JvmGcMonitorService] [client-mpalyes06] [gc][old][8508][55] duration [26.7s], collections [2]/[13.9s], total [26.7s]/[8.1m], memory [19.8gb]->[19.8gb]/[19.8gb], all_pools {[young] [1.4gb]->[1.4gb]/[1.4gb]}{[survivor] [180.8mb]->[189.9mb]/[191.3mb]}{[old] [18.1gb]->[18.1gb]/[18.1gb]}
[2017-08-07T12:42:33,964][WARN ][o.e.m.j.JvmGcMonitorService] [client-mpalyes06] [gc][8508] overhead, spent [26.7s] collecting in the last [13.9s]
[2017-08-07T12:42:33,964][INFO ][o.e.d.z.ZenDiscovery     ] [client-mpalyes06] failed to send join request to master [{master-mpalyes01}{0eS18FIcSsSPWBAiMEqQLA}{VmvWR8a2SXKJsIdJ_5QCYQ}{mpalyes01}{172.18.100.209:19010}], reason [ElasticsearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.]; ]
[2017-08-07T13:30:55,801][INFO ][o.e.n.Node               ] [client-mpalyes06] initializing ...
[2017-08-07T13:30:56,014][INFO ][o.e.e.NodeEnvironment    ] [client-mpalyes06] using [23] data paths, mounts [[/data23 (/dev/sdx1), /data1 (/dev/sdb1), /data20 (/dev/sdu1), /data2 (/dev/sdc1), /data3 (/dev/sdd1), /data19 (/dev/sdt1), /data21 (/dev/sdv1), /data22 (/dev/sdw1), /data9 (/dev/sdj1), /data13 (/dev/sdn1), /data12 (/dev/sdm1), /data11 (/dev/sdl1), /data8 (/dev/sdi1), /data14 (/dev/sdo1), /data7 (/dev/sdh1), /data15 (/dev/sdp1), /data4 (/dev/sde1), /data18 (/dev/sds1), /data5 (/dev/sdf1), /data17 (/dev/sdr1), /data6 (/dev/sdg1), /data16 (/dev/sdq1), /data10 (/dev/sdk1)]], net usable_space [19.4tb], net total_space [20.5tb], spins? [possibly], types [ext4]
[2017-08-07T13:30:56,015][INFO ][o.e.e.NodeEnvironment    ] [client-mpalyes06] heap size [19.8gb], compressed ordinary object pointers [true]
[2017-08-07T13:30:56,036][INFO ][o.e.n.Node               ] [client-mpalyes06] node name [client-mpalyes06], node ID [RFsf_xhhRaC2l9RQ1NfYrw]
[2017-08-07T13:30:56,036][INFO ][o.e.n.Node               ] [client-mpalyes06] version[5.5.0], pid[35121], build[260387d/2017-06-30T23:16:05.735Z], OS[Linux/2.6.32-642.15.1.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_72/25.72-b15]
[2017-08-07T13:30:56,036][INFO ][o.e.n.Node               ] [client-mpalyes06] JVM arguments [-Xms20g, -Xmx20g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/home/hadoop/elasticsearch/elasticsearch-client]
[2017-08-07T13:30:57,715][INFO ][c.f.s.SearchGuardPlugin  ] Clustername: alyes
[2017-08-07T13:30:57,716][WARN ][c.f.s.SearchGuardPlugin  ] 

[jpountz]

The node decided to stop because it hit an OutOfMemoryError. Unfortunately, your logs don't tell where the error occured. The fact that a client node failed (as opposed to a data or master node) suggests that it might be because too many shards were queried at once. Can you check how many shards you were querying in total? If you have time-based data, maybe you can shrink old indices?

I am closing this issue now since out-of-memory errors caused by too many queried shards is something that we are addressing in the upcoming 6.0 release. #23253