ClusterDisruptionIT#testSendingShardFailure fails on CI

[javanna]

This does not reproduce for me. I can see that the whole suite has trace logging on for more than a year, which is a signal that it was failing in the past. I didn't find this specific issue though in our repo.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=sles/1206

REPRODUCE WITH: ./gradlew :server:integTest -Dtests.seed=E77D94CFECE59A17 -Dtests.class=org.elasticsearch.discovery.ClusterDisruptionIT -Dtests.method="testSendingShardFailure" -Dtests.security.manager=true -Dtests.locale=sq -Dtests.timezone=PST8PDT

With all the trace logging I have not identified the exact cause of the failure. I did notice quite some "Connection reset by peer" errors in the logs, bu they may actually be expected. Somebody more familiar with this test is definitely going to understand more.

[elasticmachine]

Pinging @elastic/es-distributed

[javanna]

@ywelsch kindly pointed out to me that the failure is

08:42:18    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3973, name=elasticsearch[node_t2][write][T#1], state=RUNNABLE, group=TGRP-ClusterDisruptionIT]
08:42:18    > 	at __randomizedtesting.SeedInfo.seed([E77D94CFECE59A17:4127A25150EDF080]:0)
08:42:18    > Caused by: java.lang.AssertionError: shard term already update.  op term [2], shardTerm [3]
08:42:18    > 	at __randomizedtesting.SeedInfo.seed([E77D94CFECE59A17]:0)
08:42:18    > 	at org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$9(IndexShard.java:2269)
08:42:18    > 	at org.elasticsearch.index.shard.IndexShardOperationPermits.doBlockOperations(IndexShardOperationPermits.java:177)
08:42:18    > 	at org.elasticsearch.index.shard.IndexShardOperationPermits.blockOperations(IndexShardOperationPermits.java:114)
08:42:18    > 	at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2268)
08:42:18    > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:633)
08:42:18    > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
08:42:18    > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:510)
08:42:18    > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:490)
08:42:18    > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
08:42:18    > 	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1605)
08:42:18    > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
08:42:18    > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
08:42:18    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
08:42:18    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
08:42:18    > 	at java.lang.Thread.run(Thread.java:748)

This is the same failure as #32304 and #32118 although it is triggered in a different test.

[ywelsch]

Closed by #32442. If this still occurs, please reopen.