[CI] ConcurrentModificationException in ClusterDisruptionIT#testAckedIndexing

[droberts195]

There are already two open issues for ClusterDisruptionIT#testAckedIndexing, #36789 and #36042, but a new type of failure occurred in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1480/console

ERROR   12.1s J0 | ClusterDisruptionIT.testAckedIndexing <<< FAILURES!
   > Throwable #1: java.util.ConcurrentModificationException
   > 	at __randomizedtesting.SeedInfo.seed([FB2F1C569F462B5B:71EEA8A5C33BCD10]:0)
   > 	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
   > 	at java.util.ArrayList$Itr.next(ArrayList.java:859)
   > 	at org.elasticsearch.discovery.ClusterDisruptionIT.testAckedIndexing(ClusterDisruptionIT.java:219)
   > 	at java.lang.Thread.run(Thread.java:748)

This was thrown in the finally block of the test (while iterating exceptedExceptions), which caused the underlying assertion failure exception to be lost:

        } finally {
            if (exceptedExceptions.size() > 0) {
                StringBuilder sb = new StringBuilder();
                for (Exception e : exceptedExceptions) {

However, looking at the log messages the underlying assertion failure must have been the one from line 183, between these two log messages:

                logger.info("waiting for indexing requests to complete");
                assertTrue(countDownLatchRef.get().await(docsPerIndexer * seconds * 1000 + 2000, TimeUnit.MILLISECONDS));

                logger.info("stopping disruption");

As a result of the exception in the finally block the background threads kicked off by the test were leaked.

The repro command is:

./gradlew :server:integTest \
  -Dtests.seed=FB2F1C569F462B5B \
  -Dtests.class=org.elasticsearch.discovery.ClusterDisruptionIT \
  -Dtests.method="testAckedIndexing" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-CO \
  -Dtests.timezone=America/Martinique \
  -Dcompiler.java=11 \
  -Druntime.java=8

Unsurprisingly given the error this problem does not reproduce locally.

[elasticmachine]

Pinging @elastic/es-distributed

[droberts195]

I muted the test on the master branch in bcf5a4c

[droberts195]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1473/consoleText didn't suffer the ConcurrentModificationException, but failed on line 183 and got as far as reporting that exception:

FAILURE 5.57s J1 | ClusterDisruptionIT.testAckedIndexing <<< FAILURES!
   > Throwable #1: java.lang.AssertionError
   > 	at __randomizedtesting.SeedInfo.seed([4EA2B795D970C7FE:C4630366850D21B5]:0)
   > 	at org.elasticsearch.discovery.ClusterDisruptionIT.testAckedIndexing(ClusterDisruptionIT.java:183)
   > 	at java.lang.Thread.run(Thread.java:748)

[markharwood]

The ConcurrentModificationException should be fixed by adding synchronization around the loop on the exceptions list in the finally clause:

synchronized(expectedExceptions) {
	for (Exception e : exceptedExceptions) {
		sb.append("\n").append(e.getMessage());
	}
}