[CI] org.elasticsearch.node.NodeTests times out repeatedly on Windows

[alpar-t]

Example build failure

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/404/
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/405/
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/406/

Reproduction line

gradlew :server:test --tests "org.elasticsearch.node.NodeTests" -Dtests.seed=6F75492D3C1D3E0 -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=GMT -Dcompiler.java=12 -Druntime.java=11

Example relevant log:

java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).

[elasticmachine]

Pinging @elastic/es-core-infra

[jaymode]

We looked at the build scans and saw that these nodes were not closed possibly due to blocked network connection attempts, https://scans.gradle.com/s/vhno45shnhyz6/console-log#L667-L688:

2> "elasticsearch[mock_ node][generic][T#6]" ID=356 WAITING on org.elasticsearch.common.util.concurrent.BaseFuture$Sync@65e79455
--
2> 	at java.base@11.0.3/jdk.internal.misc.Unsafe.park(Native Method)
2> 	- waiting on org.elasticsearch.common.util.concurrent.BaseFuture$Sync@65e79455
2> 	at java.base@11.0.3/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
2> 	at java.base@11.0.3/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
2> 	at java.base@11.0.3/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
2> 	at java.base@11.0.3/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
2> 	at app//org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:252)
2> 	at app//org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:87)
2> 	at app//org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:57)
2> 	at app//org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:34)
2> 	at app//org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:209)
2> 	at app//org.elasticsearch.transport.ConnectionManager.openConnection(ConnectionManager.java:80)
2> 	at app//org.elasticsearch.transport.TransportService.openConnection(TransportService.java:368)
2> 	at app//org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:82)
2> 	at app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)
2> 	at app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
2> 	at java.base@11.0.3/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2> 	at java.base@11.0.3/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2> 	at java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
2> 	Locked synchronizers:
2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@20da04c0

Similar stacktraces can be found in https://scans.gradle.com/s/cmnynlo6dzt7w/console-log#L702-L721. I do wonder if some of the work in #42636 would make this better. @tbrooks8 @original-brownbear do you have any thoughts?

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

It seems like we're swallowing the PlainActionFuture we're stuck on here somewhere (since the threads waiting on them are all that's left in the thread-dump as far as I can see). I guess #42636 could remove the blocking here but still we shouldn't just lose track of an ActionListener.
I'll see if I can find out where this is happening.

[original-brownbear]

I fixed a few spots of unsafe handling of the listener that we get stuck on here in #42781, could be that some of the IO timing is different on Windows and e.g. org.elasticsearch.nio.NioSelector#ensureSelectorOpenForEnqueuing throws a RuntimeException which we never handle and hence never resolve the future.