[CI] RemoteClusterServiceTests/TransportSearchActionTests#testCollectSearchShards times out waiting in latch #33852
Description
Just had this failure in a PR build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/311/console
But I found several instances over time in the CI mails, e.g.
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+release-tests/922/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/6789/console
and some a while ago in February.
It seems a rare timing issue, so this doesn't reproduce for me locally:
./gradlew :server:test -Dtests.seed=F3E20416D514B7F4 -Dtests.class=org.elasticsearch.transport.RemoteClusterServiceTests -Dtests.method="testCollectSearchShards" -Dtests.security.manager=true -Dtests.locale=es-MX -Dtests.timezone=HST -Dcompiler.java=10 -Druntime.java=8
The failure shows the test is checking a latch after waiting for a second, maybe that is not long enough and should be increased.
11:20:48 > Throwable #1: java.lang.AssertionError
11:20:48 > at __randomizedtesting.SeedInfo.seed([F3E20416D514B7F4:BDFDA55350DA8054]:0)
11:20:48 > at org.elasticsearch.transport.RemoteClusterServiceTests.testCollectSearchShards(RemoteClusterServiceTests.java:581)
11:20:48 > at java.lang.Thread.run(Thread.java:748)