[Tests] Occasional failures because of "still open connections" in MockTransportService.doClose

[cbuescher]

On teardown of the MockTransportService used in unit tests we check that all connections that were opened are closed correctly again. This seems to fail occasionally with errors that look like:

  1> [2017-01-25T22:30:19,038][INFO ][o.e.a.s.RemoteClusterConnectionTests] [testNodeDisconnected]: after test
  2> REPRODUCE WITH: gradle :core:test -Dtests.seed=8A5ADED8D62B415D -Dtests.class=org.elasticsearch.action.search.RemoteClusterConnectionTests -Dtests.method="testNodeDisconnected" -Dtests.security.manager=true -Dtests.locale=ru -Dtests.timezone=America/Tijuana
FAILURE 0.12s J0 | RemoteClusterConnectionTests.testNodeDisconnected <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: still open connections: {{BtOlW7kRTK6YUTDmeeI-cg}{uiX1VgzsQmmqjLTVuqH2HQ}{127.0.0.1}{127.0.0.1:9301}=[org.elasticsearch.test.transport.MockTransportService$5@2cc22be3]}
   > 	at __randomizedtesting.SeedInfo.seed([8A5ADED8D62B415D:430ECAC96D5DA0DF]:0)
   > 	at org.elasticsearch.test.transport.MockTransportService.doClose(MockTransportService.java:763)
   > 	at org.elasticsearch.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:106)
   > 	at org.elasticsearch.action.search.RemoteClusterConnectionTests.testNodeDisconnected(RemoteClusterConnectionTests.java:212)
   > 	at java.lang.Thread.run(Thread.java:745)

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/1516/console

Unfortunately I cannot reproduce with the given seed, so just opening this to track this or similar build failures for now.

[s1monw]

I think I know what the issue is. there is a very very small race that is not relevant in production but the tests require more strickness. I will try to improve the code to make sure these failures won't happen anymore