[CI] LocalIndexFollowingIT#testRemoveRemoteConnection failure

[mayya-sharipova]

Test failed on 7.x. Doesn't reproduce for me.

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8,nodes=general-purpose/161/console
Build Scans: https://gradle-enterprise.elastic.co/s/2evjqejst7hay

./gradlew :x-pack:plugin:ccr:internalClusterTest --tests "org.elasticsearch.xpack.ccr.LocalIndexFollowingIT.testRemoveRemoteConnection"
-Dtests.seed=AE0AA9FBFD8C5280
-Dtests.security.manager=true
-Dtests.locale=en-NZ
-Dtests.timezone=Eire
-Dcompiler.java=12
-Druntime.java=8

Relevant stack trace:

java.lang.AssertionError: 

Expected: <1L>
     but: was <0L>
..
2019-08-23T19:33:52,919][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [[]] to [["127.0.0.1:32200"]]
[2019-08-23T19:33:52,870][WARN ][o.e.x.c.a.AutoFollowCoordinator] [node_s_0] failure occurred while fetching cluster state for auto follow pattern [my_pattern]
org.elasticsearch.ElasticsearchStatusException: can not fetch remote cluster state as the license state of the remote cluster [local] could not be determined
	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.clusterStateUnknownRemoteLicense(CcrLicenseChecker.java:444) ~[main/:?]
	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.lambda$checkRemoteClusterLicenseAndFetchClusterState$5(CcrLicenseChecker.java:176) ~[main/:?]
	at org.elasticsearch.xpack.ccr.CcrLicenseChecker$1.onFailure(CcrLicenseChecker.java:227) [main/:?]
	at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onFailure(RemoteClusterLicenseChecker.java:188) [x-pack-core-7.4.0-SNAPSHOT.jar:7.4.0-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-7.4.0-SNAPSHOT.jar:7.4.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.4.0-SNAPSHOT.jar:7.4.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) [elasticsearch-7.4.0-SNAPSHOT.jar:7.4.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$8.run(TransportService.java:982) [elasticsearch-7.4.0-SNAPSHOT.jar:7.4.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0-SNAPSHOT.jar:7.4.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
Caused by: org.elasticsearch.ElasticsearchException: could not determine the license type for cluster [local]
	... 9 more
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [node_s_0][127.0.0.1:32200][cluster:monitor/xpack/info] disconnected
[2019-08-23T19:33:52,921][INFO ][o.e.x.c.a.AutoFollowCoordinator] [node_s_0] AutoFollower for cluster [local] has stopped, because remote connection is gone

[elasticmachine]

Pinging @elastic/es-distributed

[bizybot]

Another failure
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.3+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=openjdk13,nodes=general-purpose/106/console

Could not reproduce locally with the following command:

./gradlew :x-pack:plugin:ccr:internalClusterTest --tests "org.elasticsearch.xpack.ccr.LocalIndexFollowingIT.testRemoveRemoteConnection" \
  -Dtests.seed=C1509FBC009515F2 \
  -Dtests.security.manager=true \
  -Dtests.locale=fr-CF \
  -Dtests.timezone=Etc/GMT+1 \
  -Dcompiler.java=12 \
  -Druntime.java=13

failed with a different error:

10:53:20   2> java.lang.AssertionError: On Linux and MacOSX fsyncing a directory should not throw IOException, we just don't want to rely on that in production (undocumented). Got: java.nio.file.NoSuchFileException: /var/lib/jenkins/workspace/elastic+elasticsearch+7.3+matrix-java-periodic/ES_BUILD_JAVA/openjdk12/ES_RUNTIME_JAVA/openjdk13/nodes/general-purpose/x-pack/plugin/ccr/build/testrun/internalClusterTest/temp/org.elasticsearch.xpack.ccr.LocalIndexFollowingIT_C1509FBC009515F2-001/tempDir-002/data/nodes/0/indices/jGUuzRJWSUWeAuofKRr-jg/0/_state
10:53:20   2> 	at __randomizedtesting.SeedInfo.seed([C1509FBC009515F2]:0)
10:53:20   2> 	at org.apache.lucene.util.IOUtils.fsync(IOUtils.java:464)
10:53:20   2> 	at org.apache.lucene.store.FSDirectory.syncMetaData(FSDirectory.java:310)
10:53:20   2> 	at org.elasticsearch.gateway.MetaDataStateFormat.performStateDirectoriesFsync(MetaDataStateFormat.java:172)
10:53:20   2> 	at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:246)
10:53:20   2> 	at org.elasticsearch.gateway.MetaDataStateFormat.writeAndCleanup(MetaDataStateFormat.java:185)
10:53:20   2> 	at org.elasticsearch.index.seqno.ReplicationTracker.persistRetentionLeases(ReplicationTracker.java:389)
10:53:20   2> 	at org.elasticsearch.index.shard.IndexShard.persistRetentionLeases(IndexShard.java:2100)
10:53:20   2> 	at org.elasticsearch.index.shard.StoreRecovery.writeEmptyRetentionLeasesFile(StoreRecovery.java:446)
10:53:20   2> 	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:490)

[andrershov]

Two comments above represent different kind of issues.
As for original issue opened by @mayya-sharipova, I've tried to reproduce the issue with 10K and failed to do so. I've also noticed that the stack trace reported sometimes happens during the test execution, but test passes fine. I've also checked other builds for this error occurrence and this is the only one. Given that it's not reproducible and has happened just once, I suggest to close it and reopen if it occurs in the future.
For the second issue reported by @bizybot, I'm creating a separate ticket - the reason for the failure is clear, but we won't fix it in older versions.