[CI] multiple tests started failing with NoNodeAvailableException

[alpar-t]

Seems to be triggered by the first test to run for the class.

Example build failure

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8,nodes=virtual&&linux/170/console

Reproduction line

does not reproduce locally

./gradlew :x-pack:plugin:security:unitTest \
  -Dtests.seed=7030014DD5B5A0EB \
  -Dtests.class=org.elasticsearch.xpack.security.authc.TokenAuthIntegTests \
  -Dtests.method="testRefreshAsDifferentUser" \
  -Dtests.security.manager=true \
  -Dtests.locale=th \
  -Dtests.timezone=Pacific/Apia \
  -Dcompiler.java=12 \
  -Druntime.java=8

Example relevant log:

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{tZDl3ofZSsybqfW_cz1-iw}{127.0.0.1}{127.0.0.1:35230}]]
	at __randomizedtesting.SeedInfo.seed([7030014DD5B5A0EB:E8D192FF96C2C519]:0)
	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:351)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:249)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:386)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:382)
	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1217)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45)
	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:52)
	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:113)
	at org.elasticsearch.test.TestCluster.wipeIndices(TestCluster.java:142)
	at org.elasticsearch.test.TestCluster.wipe(TestCluster.java:79)
	at org.elasticsearch.test.ESIntegTestCase.lambda$beforeInternal$0(ESIntegTestCase.java:378)
	at com.carrotsearch.randomizedtesting.RandomizedContext.runWithPrivateRandomness(RandomizedContext.java:187)
	at com.carrotsearch.randomizedtesting.RandomizedContext.runWithPrivateRandomness(RandomizedContext.java:211)
	at org.elasticsearch.test.ESIntegTestCase.beforeInternal(ESIntegTestCase.java:386)
	at org.elasticsearch.test.ESIntegTestCase.setupTestCluster(ESIntegTestCase.java:2217)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)

Frequency

Up to 5 times per day.

[elasticmachine]

Pinging @elastic/es-security

[alpar-t]

There seem to be a bunch of other tests failing recently with this exception too.
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-90d,mode:relative,to:now))&_a=(columns:!(branch,build-id,build_url,error-type,test,stacktrace,class),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!t,index:e58bf320-7efd-11e8-bf69-63c8ef516157,key:branch,negate:!f,params:(query:master,type:phrase),type:phrase,value:master),query:(match:(branch:(query:master,type:phrase))))),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:d,query:(language:kuery,query:'stacktrace:%22NoNodeAvailableException%22'),sort:!(time,desc)) [auth required]

[elasticmachine]

Pinging @elastic/es-core-infra

[ywelsch]

The interesting log lines here seem to indicate a problem with establishing a connection. @tbrooks8 do you have an idea what might be going on here?

1> [2019-01-10T07:55:41,225][INFO ][o.e.x.s.a.l.LoggingAuditTrail] [node_sd1] action="cluster:monitor/nodes/liveness" event.action="access_granted" event.type="transport" host.ip="127.0.0.1" origin.address="127.0.0.1:56657" origin.type="transport" request.id="uog1J730SHSoWN1AV5tU-g" request.name="LivenessRequest" user.name="test_trans_client_user" user.realm="file" user.roles="["transport_client","user"]"
  1> [2019-01-10T07:55:51,830][WARN ][o.e.t.TcpTransport       ] [transport_client_node_sd1] send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:56675, remoteAddress=null}]
  1> java.nio.channels.ClosedChannelException: null
  1> 	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
  1> [2019-01-10T07:55:51,846][WARN ][o.e.t.TcpTransport       ] [node_sd1] exception caught on transport layer [TcpNioSocketChannel{localAddress=0.0.0.0/0.0.0.0:35230, remoteAddress=/127.0.0.1:56674}], closing connection
  1> javax.net.ssl.SSLException: Closed engine without completely sending the close alert message.
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLDriver.close(SSLDriver.java:165) ~[main/:?]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLChannelContext.closeFromSelector(SSLChannelContext.java:164) ~[main/:?]
  1> 	at org.elasticsearch.nio.EventHandler.handleClose(EventHandler.java:208) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.EventHandler.postHandling(EventHandler.java:168) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:233) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:158) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:123) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2019-01-10T07:55:51,848][WARN ][o.e.t.TcpTransport       ] [node_sd1] exception caught on transport layer [TcpNioSocketChannel{localAddress=0.0.0.0/0.0.0.0:35230, remoteAddress=/127.0.0.1:56678}], closing connection
  1> javax.net.ssl.SSLException: Closed engine without completely sending the close alert message.
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLDriver.close(SSLDriver.java:165) ~[main/:?]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLChannelContext.closeFromSelector(SSLChannelContext.java:164) ~[main/:?]
  1> 	at org.elasticsearch.nio.EventHandler.handleClose(EventHandler.java:208) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.EventHandler.postHandling(EventHandler.java:168) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:233) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:158) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:123) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2019-01-10T07:55:51,848][WARN ][o.e.t.TcpTransport       ] [node_sd1] exception caught on transport layer [TcpNioSocketChannel{localAddress=0.0.0.0/0.0.0.0:35230, remoteAddress=/127.0.0.1:56676}], closing connection
  1> javax.net.ssl.SSLException: Closed engine without completely sending the close alert message.
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLDriver.close(SSLDriver.java:165) ~[main/:?]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLChannelContext.closeFromSelector(SSLChannelContext.java:164) ~[main/:?]
  1> 	at org.elasticsearch.nio.EventHandler.handleClose(EventHandler.java:208) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.EventHandler.postHandling(EventHandler.java:168) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:233) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:158) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:123) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2019-01-10T07:55:51,849][WARN ][o.e.t.TcpTransport       ] [node_sd1] exception caught on transport layer [TcpNioSocketChannel{localAddress=0.0.0.0/0.0.0.0:35230, remoteAddress=/127.0.0.1:56672}], closing connection
  1> javax.net.ssl.SSLException: Closed engine without completely sending the close alert message.
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLDriver.close(SSLDriver.java:165) ~[main/:?]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLChannelContext.closeFromSelector(SSLChannelContext.java:164) ~[main/:?]
  1> 	at org.elasticsearch.nio.EventHandler.handleClose(EventHandler.java:208) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.EventHandler.postHandling(EventHandler.java:168) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:233) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:158) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:123) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2019-01-10T07:55:51,849][WARN ][o.e.t.TcpTransport       ] [node_sd1] exception caught on transport layer [TcpNioSocketChannel{localAddress=0.0.0.0/0.0.0.0:35230, remoteAddress=/127.0.0.1:56696}], closing connection
  1> javax.net.ssl.SSLException: Closed engine without completely sending the close alert message.
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLDriver.close(SSLDriver.java:165) ~[main/:?]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLChannelContext.closeFromSelector(SSLChannelContext.java:164) ~[main/:?]
  1> 	at org.elasticsearch.nio.EventHandler.handleClose(EventHandler.java:208) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.EventHandler.postHandling(EventHandler.java:168) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:233) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:158) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:123) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2019-01-10T07:55:51,852][WARN ][o.e.t.TcpTransport       ] [node_sd1] exception caught on transport layer [TcpNioSocketChannel{localAddress=0.0.0.0/0.0.0.0:35230, remoteAddress=/127.0.0.1:56675}], closing connection
  1> javax.net.ssl.SSLException: Closed engine without completely sending the close alert message.
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLDriver.close(SSLDriver.java:165) ~[main/:?]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.security.transport.nio.SSLChannelContext.closeFromSelector(SSLChannelContext.java:164) ~[main/:?]
  1> 	at org.elasticsearch.nio.EventHandler.handleClose(EventHandler.java:208) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.EventHandler.postHandling(EventHandler.java:168) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:233) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:158) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:123) ~[elasticsearch-nio-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2019-01-10T07:55:51,870][WARN ][o.e.x.s.t.n.SecurityNioTransport] [node_sd1] connection TcpNioSocketChannel{localAddress=/127.0.0.1:35230, remoteAddress=/127.0.0.1:56682} closed during handshake
  1> [2019-01-10T07:55:51,911][WARN ][o.e.x.s.t.n.SecurityNioTransport] [node_sd1] connection TcpNioSocketChannel{localAddress=/127.0.0.1:35230, remoteAddress=/127.0.0.1:56680} closed during handshake
  1> [2019-01-10T07:55:51,911][WARN ][o.e.x.s.t.n.SecurityNioTransport] [node_sd1] connection TcpNioSocketChannel{localAddress=/127.0.0.1:35230, remoteAddress=/127.0.0.1:56695} closed during handshake
  1> [2019-01-10T07:55:51,911][WARN ][o.e.x.s.t.n.SecurityNioTransport] [node_sd1] connection TcpNioSocketChannel{localAddress=/127.0.0.1:35230, remoteAddress=/127.0.0.1:56681} closed during handshake

[s1monw]

@tbrooks8 I see some SSL issues here can this be related to #37246