IndexFollowingIT.testUpdateRemoteConfigsDuringFollowing fails on master

[mayya-sharipova]

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/4093/console
Build Scans: https://gradle-enterprise.elastic.co/s/tp2q6hwdkzcom

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ccr:internalClusterTest' --tests "org.elasticsearch.xpack.ccr.IndexFollowingIT.testUpdateRemoteConfigsDuringFollowing"
-Dtests.seed=6CB3F4DF8251DFB5
-Dtests.security.manager=true
-Dtests.locale=es-AR
-Dtests.timezone=Asia/Aden
-Dcompiler.java=13

Doesn't reproduce for me.
No other failures of this test for this year.

Stack trace:

java.lang.AssertionError: incorrect global checkpoint {"remote_cluster":"leader_cluster","follow_shard_index":"index2","follow_shard_index_uuid":"56HGhnidRZOl-eRO5HtDUw","follow_shard_shard":0,"leader_shard_index":"index1","leader_shard_index_uuid":"mlnxKU3BSPqE8AGe9MQt9A","leader_shard_shard":0,"max_read_request_operation_count":5120,"max_write_request_operation_count":5120,"max_outstanding_read_requests":12,"max_outstanding_write_requests":9,"max_read_request_size":"32mb","max_write_request_size":"9223372036854775807b","max_write_buffer_count":2147483647,"max_write_buffer_size":"512mb","max_retry_delay":"10ms","read_poll_timeout":"10ms","headers":{}}
Expected: <229L>
     but: was <-1L>
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.elasticsearch.xpack.ccr.IndexFollowingIT.lambda$assertTask$63(IndexFollowingIT.java:1476)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:881)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:854)
	at org.elasticsearch.xpack.ccr.IndexFollowingIT.testUpdateRemoteConfigsDuringFollowing(IndexFollowingIT.java:1348)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/CCR)

[dnhatn]

  2> mar 06, 2020 5:25:28 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
  2> WARNING: Uncaught exception in thread: Thread[elasticsearch[followerd3][ccr][T#22],5,TGRP-IndexFollowingIT]
  2> org.elasticsearch.transport.NoSuchRemoteClusterException: no such remote cluster: [leader_cluster]
  2> 	at __randomizedtesting.SeedInfo.seed([6CB3F4DF8251DFB5]:0)
  2> 	at org.elasticsearch.transport.RemoteClusterService.getRemoteClusterConnection(RemoteClusterService.java:205)
  2> 	at org.elasticsearch.transport.RemoteClusterService.ensureConnected(RemoteClusterService.java:188)
  2> 	at org.elasticsearch.transport.RemoteClusterAwareClient.doExecute(RemoteClusterAwareClient.java:48)
  2> 	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:377)
  2> 	at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:661)
  2> 	at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.state(AbstractClient.java:691)
  2> 	at org.elasticsearch.xpack.ccr.action.CcrRequests.getIndexMetadata(CcrRequests.java:59)
  2> 	at org.elasticsearch.xpack.ccr.action.ShardFollowTasksExecutor$1.innerUpdateMapping(ShardFollowTasksExecutor.java:144)
  2> 	at org.elasticsearch.xpack.ccr.action.ShardFollowNodeTask.updateMapping(ShardFollowNodeTask.java:481)
  2> 	at org.elasticsearch.xpack.ccr.action.ShardFollowNodeTask.lambda$updateMapping$17(ShardFollowNodeTask.java:482)
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
  2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

I will take a closer look tomorrow.

[pgomulka]

@dnhatn by any chance this was not backported to 6.8 ? Do you think it is worth a backport?
there was a very similar failure in that test on that branch

java.lang.AssertionError: incorrect global checkpoint {"remote_cluster":"leader_cluster","follow_shard_index":"index2","follow_shard_index_uuid":"6PI3qVcLS12o7i-_cGFeEw","follow_shard_shard":1,"leader_shard_index":"index1","leader_shard_index_uuid":"OiYzpvIbThCOK_FB8uuSCA","leader_shard_shard":1,"max_read_request_operation_count":9016,"max_write_request_operation_count":5120,"max_outstanding_read_requests":12,"max_outstanding_write_requests":9,"max_read_request_size":"30573350b","max_write_request_size":"9223372036854775807b","max_write_buffer_count":2147483647,"max_write_buffer_size":"512mb","max_retry_delay":"10ms","read_poll_timeout":"10ms","headers":{}}
Expected: <171L>
     but: was <-1L>
	at __randomizedtesting.SeedInfo.seed([3C886A4EEAD67EB9:FA33AFE3CB28BE11]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.elasticsearch.xpack.ccr.IndexFollowingIT.lambda$assertTask$53(IndexFollowingIT.java:1386)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:906)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:880)
	at org.elasticsearch.xpack.ccr.IndexFollowingIT.testUpdateRemoteConfigsDuringFollowing(IndexFollowingIT.java:1257)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)

https://gradle-enterprise.elastic.co/s/ysedpzc6m6ilc

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ccr:internalClusterTest' \
  -Dtests.seed=3C886A4EEAD67EB9 \
  -Dtests.class=org.elasticsearch.xpack.ccr.IndexFollowingIT \
  -Dtests.method="testUpdateRemoteConfigsDuringFollowing" \
  -Dtests.security.manager=true \
  -Dtests.locale=ro \
  -Dtests.timezone=SystemV/PST8PDT \
  -Dcompiler.java=12 \
  -Druntime.java=12

interestingly SystemV/PST8PDT is used. This is 6.8 so I guess joda is used, which does not support that timezone (it is only supported by java.time).
could this be that there are nodes in 7 and 6.8 in this test?