RemoteClusterConnectionTests failing with "transport has been stopped"

[DaveCTurner]

I've seen a couple of failures of different tests from RemoteClusterConnectionTests failing like this:

org.elasticsearch.transport.RemoteClusterConnectionTests > testConnectedNodesConcurrentAccess FAILED
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=449, name=elasticsearch[org.elasticsearch.transport.RemoteClusterConnectionTests][generic][T#3], state=RUNNABLE, group=TGRP-RemoteClusterConnectionTests]
 
Caused by:
java.lang.IllegalStateException: transport has been stopped
REPRODUCE WITH: ./gradlew :server:test --tests "org.elasticsearch.transport.RemoteClusterConnectionTests.testConnectedNodesConcurrentAccess" -Dtests.seed=E63A7B3434778207 -Dtests.security.manager=true -Dtests.locale=es-CR -Dtests.timezone=Australia/Tasmania -Dcompiler.java=12 -Druntime.java=8

This particular one is testConnectedNodesConcurrentAccess but I've also seen testRenderConnectionInfoXContent fail in the same way. I've seen this on a branch off master at f2a6dd7 (so after #44805 was merged) - these CI failures look like the same issue too:

• https://scans.gradle.com/s/mxhoe2ylcjrys/console-log?task=:server:test
• https://scans.gradle.com/s/ib5fikbrhhobu/console-log?task=:server:test

[elasticmachine]

Pinging @elastic/es-distributed

[jdconrad]

Another:
https://scans.gradle.com/s/svd3em7mf74i6/console-log#L713

[jdconrad]

Another:
https://scans.gradle.com/s/76rujrh2qy2ku/console-log