RemoteClusterConnectionTests failing on 6.8

[przemekwitek]

Build scan:

https://gradle-enterprise.elastic.co/s/egmexshkcwg6i
https://gradle-enterprise.elastic.co/s/t6mwtrmt6e24k
https://gradle-enterprise.elastic.co/s/74hh3etfidq6g

Repro line:

REPRODUCE WITH: ./gradlew ':server:unitTest' \
  -Dtests.seed=B592A0D1EC8BF9A8 \
  -Dtests.class=org.elasticsearch.transport.RemoteClusterConnectionTests \
  -Dtests.method="testPendingConnectListeners" \
  -Dtests.security.manager=true \
  -Dtests.locale=sl \
  -Dtests.timezone=Pacific/Auckland \
  -Dcompiler.java=12 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':server:unitTest' \
  -Dtests.seed=B592A0D1EC8BF9A8 \
  -Dtests.class=org.elasticsearch.transport.RemoteClusterConnectionTests \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=UTC \
  -Dcompiler.java=12 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':server:unitTest' \
  -Dtests.seed=B592A0D1EC8BF9A8 \
  -Dtests.class=org.elasticsearch.transport.RemoteClusterConnectionTests \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=UTC \
  -Dcompiler.java=12 \
  -Druntime.java=8

Reproduces locally?:
No

Applicable branches:
6.8

Failure excerpt:

15:40:48   1> org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [connect handler is closed]
15:40:48   1> 	at org.elasticsearch.common.util.CancellableThreads.checkForCancel(CancellableThreads.java:65) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:130) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:518) [main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.access$1000(RemoteClusterConnection.java:408) [main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:506) [main/:?]
15:40:48   1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?]
15:40:48   1> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
15:40:48   1> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
15:40:48   1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?]
15:40:48   1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
15:40:48   1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
15:40:48   1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
15:40:48   1> 	Suppressed: java.lang.IllegalStateException: Future got interrupted
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:60) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:34) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:209) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.transport.ConnectionManager.openConnection(ConnectionManager.java:80) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:524) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:108) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:518) [main/:?]
15:40:48   1> 		at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.access$1000(RemoteClusterConnection.java:408) [main/:?]
15:40:48   1> 		at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:506) [main/:?]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?]
15:40:48   1> 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
15:40:48   1> 		at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?]
15:40:48   1> 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
15:40:48   1> 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
15:40:48   1> 		at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
15:40:48   1> 	Caused by: java.lang.InterruptedException
15:40:48   1> 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_241]
15:40:48   1> 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_241]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:57) ~[main/:?]
15:40:48   1> 		... 15 more
15:40:48   1> [2020-07-21T01:20:43,145][INFO ][o.e.t.RemoteClusterConnectionTests] [testLazyResolveTransportAddress] after test
15:40:48   1> [2020-07-21T01:20:43,148][INFO ][o.e.t.RemoteClusterConnectionTests] [testSlowNodeCanBeCancelled] before test
15:40:48   1> [2020-07-21T01:20:43,151][INFO ][o.e.t.TransportService   ] [testSlowNodeCanBeCancelled] publish_address {127.0.0.1:11800}, bound_addresses {[::1]:11800}, {127.0.0.1:11800}
15:40:48   1> [2020-07-21T01:20:43,154][WARN ][o.e.t.RemoteClusterConnection] [org.elasticsearch.transport.RemoteClusterConnectionTests] fetching nodes from external cluster [test-cluster] failed
15:40:48   1> org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [connect handler is closed]
15:40:48   1> 	at org.elasticsearch.common.util.CancellableThreads.checkForCancel(CancellableThreads.java:65) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:130) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:518) [main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.access$1000(RemoteClusterConnection.java:408) [main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:506) [main/:?]
15:40:48   1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?]
15:40:48   1> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
15:40:48   1> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
15:40:48   1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?]
15:40:48   1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
15:40:48   1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
15:40:48   1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
15:40:48   1> 	Suppressed: java.lang.IllegalStateException: Future got interrupted
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:60) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:34) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:209) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.transport.ConnectionManager.openConnection(ConnectionManager.java:80) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:524) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:108) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:518) [main/:?]
15:40:48   1> 		at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.access$1000(RemoteClusterConnection.java:408) [main/:?]
15:40:48   1> 		at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:506) [main/:?]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?]
15:40:48   1> 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
15:40:48   1> 		at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?]
15:40:48   1> 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
15:40:48   1> 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
15:40:48   1> 		at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
15:40:48   1> 	Caused by: java.lang.InterruptedException
15:40:48   1> 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_241]
15:40:48   1> 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_241]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[main/:?]
15:40:48   1> 		at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:57) ~[main/:?]
15:40:48   1> 		... 15 more
15:40:48   1> [2020-07-21T01:20:43,158][INFO ][o.e.t.RemoteClusterConnectionTests] [testSlowNodeCanBeCancelled] after test
15:40:48   1> [2020-07-21T01:20:43,161][INFO ][o.e.t.RemoteClusterConnectionTests] [testFetchShardsSkipUnavailable] before test
15:40:48   1> [2020-07-21T01:20:43,163][INFO ][o.e.t.TransportService   ] [testFetchShardsSkipUnavailable] publish_address {127.0.0.1:11800}, bound_addresses {[::1]:11800}, {127.0.0.1:11800}
15:40:48   1> [2020-07-21T01:20:43,165][INFO ][o.e.t.TransportService   ] [testFetchShardsSkipUnavailable] publish_address {127.0.0.1:11801}, bound_addresses {[::1]:11801}, {127.0.0.1:11801}
15:40:48   1> [2020-07-21T01:20:43,184][WARN ][o.e.t.RemoteClusterConnection] [org.elasticsearch.transport.RemoteClusterConnectionTests] fetching nodes from external cluster [test-cluster] failed
15:40:48   1> org.elasticsearch.transport.ConnectTransportException: [seed_node][127.0.0.1:11800] DISCONNECT: simulated
15:40:48   1> 	at org.elasticsearch.test.transport.MockTransportService.lambda$addFailToSendNoConnectRule$3(MockTransportService.java:228) ~[framework-6.8.11-SNAPSHOT.jar:6.8.11-SNAPSHOT]
15:40:48   1> 	at org.elasticsearch.test.transport.StubbableTransport.openConnection(StubbableTransport.java:153) ~[framework-6.8.11-SNAPSHOT.jar:6.8.11-SNAPSHOT]
15:40:48   1> 	at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.transport.ConnectionManager.openConnection(ConnectionManager.java:80) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:524) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:108) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:518) [main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.access$1000(RemoteClusterConnection.java:408) [main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:506) [main/:?]
15:40:48   1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?]
15:40:48   1> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
15:40:48   1> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
15:40:48   1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?]
15:40:48   1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
15:40:48   1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
15:40:48   1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
15:40:48   1> [2020-07-21T01:20:43,186][WARN ][o.e.t.RemoteClusterConnection] [org.elasticsearch.transport.RemoteClusterConnectionTests] fetching nodes from external cluster [test-cluster] failed
15:40:48   1> org.elasticsearch.transport.ConnectTransportException: [seed_node][127.0.0.1:11800] DISCONNECT: simulated
15:40:48   1> 	at org.elasticsearch.test.transport.MockTransportService.lambda$addFailToSendNoConnectRule$3(MockTransportService.java:228) ~[framework-6.8.11-SNAPSHOT.jar:6.8.11-SNAPSHOT]
15:40:48   1> 	at org.elasticsearch.test.transport.StubbableTransport.openConnection(StubbableTransport.java:153) ~[framework-6.8.11-SNAPSHOT.jar:6.8.11-SNAPSHOT]
15:40:48   1> 	at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.transport.ConnectionManager.openConnection(ConnectionManager.java:80) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:524) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:108) ~[main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:518) [main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.access$1000(RemoteClusterConnection.java:408) [main/:?]
15:40:48   1> 	at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:506) [main/:?]
15:40:48   1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?]
15:40:48   1> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
15:40:48   1> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
15:40:48   1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [main/:?]
15:40:48   1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
15:40:48   1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
15:40:48   1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[cbuescher]

More of these today, only on 6.8, e.g.: https://gradle-enterprise.elastic.co/s/hg6d6bkvdz5pg

I think we should mute this test on the 6.8 branch
Looking at the history, this seems to have picked up starting sometime on July 15th, https://gradle-enterprise.elastic.co/s/cwvrkl6barm4w

[cbuescher]

I muted this test on 6.8 with 04f24ac

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Network)

[dnhatn]

Fixed in 673dd65