TEST: CoreFullClusterRestartIT failing

[ywelsch]

Build failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+bwc-tests/191/consoleText

Relevant log lines:

[2018-12-10T11:32:52,051][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-0] fatal error in thread [elasticsearch[node-0][clusterApplierService#updateTask][T#1]], exiting
java.lang.AssertionError: existing values: [[299 Elasticsearch-7.0.0-SNAPSHOT-e160346 "[search.remote.foo.seeds] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." "Mon, 10 Dec 2018 11:32:52 GMT", 299 Elasticsearch-7.0.0-SNAPSHOT-e160346 "[search.remote.foo.skip_unavailable] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." "Mon, 10 Dec 2018 11:32:52 GMT", 299 Elasticsearch-7.0.0-SNAPSHOT-e160346 "[search.remote.foo.seeds] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." "Mon, 10 Dec 2018 11:32:51 GMT", 299 Elasticsearch-7.0.0-SNAPSHOT-e160346 "[search.remote.foo.skip_unavailable] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." "Mon, 10 Dec 2018 11:32:51 GMT"]], existing unique values [[[search.remote.foo.skip_unavailable] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version., [search.remote.foo.seeds] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.]]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ThreadContextStruct.putResponse(ThreadContext.java:528) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ThreadContextStruct.access$1100(ThreadContext.java:403) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext.addResponseHeader(ThreadContext.java:331) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.logging.DeprecationLogger.deprecated(DeprecationLogger.java:313) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.logging.DeprecationLogger.deprecatedAndMaybeLog(DeprecationLogger.java:148) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting.checkDeprecation(Setting.java:470) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting.getRaw(Setting.java:449) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting.lambda$listSetting$33(Setting.java:1208) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting$ListSetting.lambda$new$0(Setting.java:1282) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting$ListSetting.innerGetRaw(Setting.java:1292) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting.getRaw(Setting.java:450) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting$Updater.hasChanged(Setting.java:951) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting$AffixSetting$1.lambda$getValue$2(Setting.java:683) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
	at java.util.stream.DistinctOps$1$2.accept(DistinctOps.java:175) ~[?:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) ~[?:?]
	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
	at org.elasticsearch.common.settings.Setting$AffixSetting$1.getValue(Setting.java:677) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Setting$AffixSetting$1.getValue(Setting.java:666) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.apply(AbstractScopedSettings.java:547) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.AbstractScopedSettings$1.getValue(AbstractScopedSettings.java:283) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.AbstractScopedSettings$1.getValue(AbstractScopedSettings.java:255) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.updater(AbstractScopedSettings.java:561) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:184) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:460) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:418) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:162) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

[elasticmachine]

Pinging @elastic/es-core-infra

[ywelsch]

Another one:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+bwc-tests/191/console

[martijnvg]

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+bwc-tests/194/console

[ywelsch]

More failures (see #37189). Can someone have a look here?

[danielmitterdorfer]

Analyzing the logs provided in #37189, one thing that stands out is that cluster settings get updated multiple times. In upgraded-node0.log we can see this:

[2019-01-07T02:31:30,983][INFO ][o.e.c.s.ClusterSettings  ] [node-0] updating [search.remote.foo.seeds] from [[]] to [["localhost:9200"]]
[2019-01-07T02:31:30,986][INFO ][o.e.c.s.ClusterSettings  ] [node-0] updating [search.remote.foo.seeds] from [[]] to [["localhost:9200"]]
[2019-01-07T02:31:30,987][INFO ][o.e.c.s.ClusterSettings  ] [node-0] updating [search.remote.foo.seeds] from [[]] to [["localhost:9200"]]

And immediately afterwards, the assertion trips complaining that the deprecation headers for search.remote.foo.seeds are contained twice. I assume that we see the headers only twice because assertion trips immediately on duplication. If the assertion were not there, I expect to see the same deprecation header three times.

[matriv]

Another failure here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+bwc-tests/332/console

15:22:58 [2019-01-15T13:20:50,073][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [elasticsearch[node-1][clusterApplierService#updateTask][T#1]], exiting
15:22:58 java.lang.AssertionError: existing values: [[299 Elasticsearch-7.0.0-SNAPSHOT-b97245c "[search.remote.foo.seeds] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." "Tue, 15 Jan 2019 13:20:50 GMT", 299 Elasticsearch-7.0.0-SNAPSHOT-b97245c "[search.remote.foo.skip_unavailable] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." "Tue, 15 Jan 2019 13:20:50 GMT", 299 Elasticsearch-7.0.0-SNAPSHOT-b97245c "[search.remote.foo.seeds] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." "Tue, 15 Jan 2019 13:20:49 GMT", 299 Elasticsearch-7.0.0-SNAPSHOT-b97245c "[search.remote.foo.skip_unavailable] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." "Tue, 15 Jan 2019 13:20:49 GMT"]], existing unique values [[[search.remote.foo.skip_unavailable] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version., [search.remote.foo.seeds] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.]]
15:22:58 	at org.elasticsearch.common.util.concurrent.ThreadContext$ThreadContextStruct.putResponse(ThreadContext.java:528) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.ThreadContext$ThreadContextStruct.access$1100(ThreadContext.java:403) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.ThreadContext.addResponseHeader(ThreadContext.java:331) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.logging.DeprecationLogger.deprecated(DeprecationLogger.java:315) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.logging.DeprecationLogger.deprecatedAndMaybeLog(DeprecationLogger.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting.checkDeprecation(Setting.java:479) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting.getRaw(Setting.java:458) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting.lambda$listSetting$32(Setting.java:1226) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting$ListSetting.lambda$new$0(Setting.java:1300) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting$ListSetting.innerGetRaw(Setting.java:1310) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting.getRaw(Setting.java:459) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting$Updater.hasChanged(Setting.java:973) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting$AffixSetting$1.lambda$getValue$2(Setting.java:692) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
15:22:58 	at java.util.stream.DistinctOps$1$2.accept(DistinctOps.java:175) ~[?:?]
15:22:58 	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
15:22:58 	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
15:22:58 	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603) ~[?:?]
15:22:58 	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
15:22:58 	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
15:22:58 	at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) ~[?:?]
15:22:58 	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) ~[?:?]
15:22:58 	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
15:22:58 	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
15:22:58 	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
15:22:58 	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
15:22:58 	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
15:22:58 	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
15:22:58 	at org.elasticsearch.common.settings.Setting$AffixSetting$1.getValue(Setting.java:686) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.Setting$AffixSetting$1.getValue(Setting.java:675) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.apply(AbstractScopedSettings.java:547) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.AbstractScopedSettings$1.getValue(AbstractScopedSettings.java:283) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.AbstractScopedSettings$1.getValue(AbstractScopedSettings.java:255) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.updater(AbstractScopedSettings.java:561) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:184) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:465) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:422) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:166) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
15:22:58 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
15:22:58 	at java.lang.Thread.run(Thread.java:834) [?:?]
15:22:58 =========================================

There is also this part:

15:22:57 [2019-01-15T13:20:51,943][WARN ][o.e.r.b.FileRestoreContext] [node-0] [[restored_testsnapshotrestore][4]] [old_snap/wttV8rxnT7mjam1JdL3r_Q] Can't read metadata from store, will not reuse local files during restore
15:22:57 org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(ByteSizeCachingDirectory(HybridDirectory@/var/lib/jenkins/workspace/elastic+elasticsearch+master+bwc-tests/x-pack/qa/full-cluster-restart/with-system-key/build/cluster/v6.4.3#oldClusterTestCluster node0/elasticsearch-6.4.3/data/nodes/0/indices/kT1bicurQmaPNoDHsOe9Qw/4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d652784)): files: [write.lock]
15:22:57 	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:675) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
15:22:57 	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
15:22:57 	at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-8.0.0-snapshot-a1c6e642aa.jar:8.0.0-snapshot-a1c6e642aa a1c6e642aad90d3615b4c71bf261a5aad7e32369 - nknize - 2019-01-02 14:49:37]
15:22:57 	at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:200) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.store.Store.access$200(Store.java:128) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:839) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:772) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.store.Store.getMetadata(Store.java:286) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1172) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:121) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:862) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:464) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:279) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:277) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1682) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$7(IndexShard.java:2206) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
15:22:57 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
15:22:57 	at java.lang.Thread.run(Thread.java:834) [?:?]
15:22:57 [2019-01-15T13:20:51,980][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] adding template [.ml-state] for index patterns [.ml-state*]
15:22:57 [2019-01-15T13:20:52,026][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] adding template [.ml-meta] for index patterns [.ml-meta]
15:22:57 [2019-01-15T13:20:52,064][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] adding template [security_audit_log] for index patterns [.security_audit_log*]
15:22:57 [2019-01-15T13:20:52,111][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] adding template [.ml-anomalies-] for index patterns [.ml-anomalies-*]
15:22:57 [2019-01-15T13:20:52,166][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] adding template [.ml-notifications] for index patterns [.ml-notifications]
15:22:57 [2019-01-15T13:20:52,188][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] removing template [logstash-index-template]
15:22:57 [2019-01-15T13:20:52,211][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-0] removing template [security-index-template]
15:22:57 [2019-01-15T13:20:52,239][INFO ][o.e.c.m.TemplateUpgradeService] [node-0] Templates were upgraded successfully to version 7.0.0
15:22:57 [2019-01-15T13:20:52,301][WARN ][o.e.c.r.a.AllocationService] [node-0] [.triggered_watches][0] marking unavailable shards as stale: [UNeo_ImdTqubILWyJknZgA]
15:22:57 [2019-01-15T13:20:52,338][WARN ][o.e.s.RestoreService     ] [node-0] [old_snap/wttV8rxnT7mjam1JdL3r_Q] failed to restore snapshot
15:22:57 org.elasticsearch.snapshots.SnapshotRestoreException: [repo:old_snap/wttV8rxnT7mjam1JdL3r_Q] cannot restore index [restored_testsnapshotrestore] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name
15:22:57 	at org.elasticsearch.snapshots.RestoreService$1.validateExistingIndex(RestoreService.java:446) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.snapshots.RestoreService$1.execute(RestoreService.java:307) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:686) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:311) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:211) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:143) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:57 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
15:22:58 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
15:22:58 	at java.lang.Thread.run(Thread.java:834) [?:?]
15:22:58 [2019-01-15T13:20:52,340][WARN ][r.suppressed             ] [node-0] path: /_snapshot/repo/old_snap/_restore, params: {repository=repo, wait_for_completion=true, snapshot=old_snap}
15:22:58 org.elasticsearch.snapshots.SnapshotRestoreException: [repo:old_snap/wttV8rxnT7mjam1JdL3r_Q] cannot restore index [restored_testsnapshotrestore] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name
15:22:58 	at org.elasticsearch.snapshots.RestoreService$1.validateExistingIndex(RestoreService.java:446) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.snapshots.RestoreService$1.execute(RestoreService.java:307) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:686) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:311) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:211) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:143) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
15:22:58 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
15:22:58 	at java.lang.Thread.run(Thread.java:834) [?:?]
15:22:58 [2019-01-15T13:20:52,348][INFO ][o.e.c.r.DelayedAllocationService] [node-0] scheduling reroute for delayed shards in [58.7s] (65 delayed shards)
15:22:58 [2019-01-15T13:20:52,382][WARN ][o.e.c.r.a.AllocationService] [node-0] [bwc_watch_index][1] marking unavailable shards as stale: [H-SpIbDTSxuNG-I493aCxg]
15:22:58 [2019-01-15T13:20:52,439][WARN ][o.e.c.r.a.AllocationService] [node-0] [.watches][0] marking unavailable shards as stale: [1CLl4uM7Q8meu3flxrK4sA]
15:22:58 [2019-01-15T13:20:52,477][WARN ][o.e.c.r.a.AllocationService] [node-0] [.watcher-history-9-2019.01.15][0] marking unavailable shards as stale: [T1fVCuDQTOmLJgcju4dTUA]
15:22:58 [2019-01-15T13:20:52,623][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node-0] updating number_of_replicas to [1] for indices [testnewreplicaswork]
15:22:58 [2019-01-15T13:20:53,098][WARN ][o.e.c.r.a.AllocationService] [node-0] [bwc_watch_index][4] marking unavailable shards as stale: [oz89uc2aTfid5T3xznA2aA]
15:22:58 [2019-01-15T13:20:55,094][WARN ][o.e.c.r.a.AllocationService] [node-0] [bwc_watch_index][0] marking unavailable shards as stale: [-wR9VVlqQOmJCjIRlYYqyA]
15:22:58 [2019-01-15T13:20:58,098][WARN ][o.e.c.r.a.AllocationService] [node-0] [bwc_watch_index][2] marking unavailable shards as stale: [iEAVxjqVSludHLux6A1slw]
15:22:58 [2019-01-15T13:21:01,116][WARN ][o.e.c.r.a.AllocationService] [node-0] [bwc_watch_index][3] marking unavailable shards as stale: [t6yhXVKuTKmYNy5-dhzmnQ]
15:22:58 [2019-01-15T13:21:22,829][INFO ][o.e.c.s.IndexScopedSettings] [node-0] updating [index.refresh_interval] from [1s] to [-1]
15:22:58 [2019-01-15T13:21:22,850][INFO ][o.e.c.s.IndexScopedSettings] [node-0] updating [index.refresh_interval] from [1s] to [-1]
15:22:58 [2019-01-15T13:21:22,947][INFO ][o.e.c.s.IndexScopedSettings] [node-0] updating [index.refresh_interval] from [-1] to [1s]
15:22:58 [2019-01-15T13:21:22,973][INFO ][o.e.c.s.IndexScopedSettings] [node-0] updating [index.refresh_interval] from [-1] to [1s]
15:22:58 [2019-01-15T13:22:24,668][DEBUG][o.e.a.s.TransportSearchAction] [node-0] All shards failed for phase: [query]
15:22:58 [2019-01-15T13:22:24,669][WARN ][r.suppressed             ] [node-0] path: /queries/_search, params: {index=queries}
15:22:58 org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
15:22:58 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:297) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:138) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:258) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
15:22:58 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
15:22:58 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
15:22:58 	at java.lang.Thread.run(Thread.java:834) [?:?]

and this:

15:22:23 ERROR   30.0s | CoreFullClusterRestartIT.testEmptyShard <<< FAILURES!
15:22:23    > Throwable #1: java.io.IOException: listener timeout after waiting for [30000] ms
15:22:23    > 	at __randomizedtesting.SeedInfo.seed([C79798D73C9F726C:8BEC4F0FC0A65DF9]:0)
15:22:23    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:669)
15:22:23    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
15:22:23    > 	at org.elasticsearch.test.rest.ESRestTestCase.ensureGreen(ESRestTestCase.java:788)
15:22:23    > 	at org.elasticsearch.upgrades.FullClusterRestartIT.testEmptyShard(FullClusterRestartIT.java:723)
15:22:23    > 	at java.lang.Thread.run(Thread.java:748)
15:22:23   1> [2019-01-15T13:21:53,052][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testSoftDeletes] before test
15:22:23   1> [2019-01-15T13:21:53,059][WARN ][o.e.c.RestClient         ] [[I/O dispatcher 36]] request [GET http://[::1]:40835/info/doc/testsoftdeletes_doc_count?filter_path=_source] returned 1 warnings: [299 Elasticsearch-7.0.0-SNAPSHOT-b97245c "[types removal] Specifying types in document get requests is deprecated, use the /{index}/_doc/{id} endpoint instead." "Tue, 15 Jan 2019 13:21:53 GMT"]
15:22:23   1> [2019-01-15T13:21:53,075][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testSoftDeletes] There are still tasks running after this test that might break subsequent tests [cluster:monitor/health, indices:data/write/bulk, indices:data/write/bulk[s], indices:data/write/bulk[s][p], xpack/ml/datafeed[c], xpack/ml/job[c], xpack/rollup/job[c]].
15:22:23   1> [2019-01-15T13:21:53,076][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testSoftDeletes] after test
15:22:23   1> [2019-01-15T13:21:53,087][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testShrinkAfterUpgrade] before test
15:22:23   1> [2019-01-15T13:22:23,097][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testShrinkAfterUpgrade] There are still tasks running after this test that might break subsequent tests [cluster:monitor/health, xpack/ml/datafeed[c], xpack/ml/job[c], xpack/rollup/job[c]].
15:22:23   1> [2019-01-15T13:22:23,098][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testShrinkAfterUpgrade] after test
15:22:23 ERROR   30.0s | CoreFullClusterRestartIT.testShrinkAfterUpgrade <<< FAILURES!
15:22:23    > Throwable #1: java.io.IOException: listener timeout after waiting for [30000] ms
15:22:23    > 	at __randomizedtesting.SeedInfo.seed([C79798D73C9F726C:1D8E873143D52146]:0)
15:22:23    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:669)
15:22:23    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
15:22:23    > 	at org.elasticsearch.test.rest.ESRestTestCase.ensureGreen(ESRestTestCase.java:788)
15:22:23    > 	at org.elasticsearch.upgrades.FullClusterRestartIT.testShrinkAfterUpgrade(FullClusterRestartIT.java:432)
15:22:23    > 	at java.lang.Thread.run(Thread.java:748)
15:22:23   1> [2019-01-15T13:22:23,108][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testAliasWithBadName] before test
15:22:23   1> [2019-01-15T13:22:23,125][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testAliasWithBadName] There are still tasks running after this test that might break subsequent tests [cluster:monitor/health, xpack/ml/datafeed[c], xpack/ml/job[c], xpack/rollup/job[c]].
15:22:23   1> [2019-01-15T13:22:23,126][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testAliasWithBadName] after test
15:22:23 IGNOR/A 0.03s | CoreFullClusterRestartIT.testAliasWithBadName
15:22:23    > Assumption #1: Can only test bad alias name if old cluster is on 5.1.0 or before
15:22:23   1> [2019-01-15T13:22:23,141][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testRollover] before test
15:22:23   1> [2019-01-15T13:22:23,150][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testRollover] There are still tasks running after this test that might break subsequent tests [cluster:monitor/health, xpack/ml/datafeed[c], xpack/ml/job[c], xpack/rollup/job[c]].
15:22:23   1> [2019-01-15T13:22:23,151][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testRollover] after test
15:22:23 ERROR   0.02s | CoreFullClusterRestartIT.testRollover <<< FAILURES!
15:22:23    > Throwable #1: java.net.ConnectException: Connection refused
15:22:23    > 	at __randomizedtesting.SeedInfo.seed([C79798D73C9F726C:4C8A95AFD2331E93]:0)
15:22:23    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:713)
15:22:23    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
15:22:23    > 	at org.elasticsearch.upgrades.FullClusterRestartIT.testRollover(FullClusterRestartIT.java:501)
15:22:23    > 	at java.lang.Thread.run(Thread.java:748)
15:22:23    > Caused by: java.net.ConnectException: Connection refused
15:22:23    > 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
15:22:23    > 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
15:22:23    > 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
15:22:23    > 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
15:22:23    > 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
15:22:23    > 	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
15:22:23    > 	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
15:22:23    > 	... 1 more
15:22:23   1> [2019-01-15T13:22:23,161][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testRecovery] before test
15:22:23   1> [2019-01-15T13:22:23,170][WARN ][o.e.c.RestClient         ] [[I/O dispatcher 37]] request [GET http://[::1]:40835/info/doc/testrecovery_count?filter_path=_source] returned 1 warnings: [299 Elasticsearch-7.0.0-SNAPSHOT-b97245c "[types removal] Specifying types in document get requests is deprecated, use the /{index}/_doc/{id} endpoint instead." "Tue, 15 Jan 2019 13:22:23 GMT"]
15:22:23   1> [2019-01-15T13:22:23,175][WARN ][o.e.c.RestClient         ] [[I/O dispatcher 37]] request [GET http://[::1]:40835/info/doc/testrecovery_should_have_translog?filter_path=_source] returned 1 warnings: [299 Elasticsearch-7.0.0-SNAPSHOT-b97245c "[types removal] Specifying types in document get requests is deprecated, use the /{index}/_doc/{id} endpoint instead." "Tue, 15 Jan 2019 13:22:23 GMT"]
15:22:23   1> [2019-01-15T13:22:23,194][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testRecovery] There are still tasks running after this test that might break subsequent tests [cluster:monitor/health, xpack/ml/datafeed[c], xpack/ml/job[c], xpack/rollup/job[c]].
15:22:23   1> [2019-01-15T13:22:23,194][INFO ][o.e.x.r.CoreFullClusterRestartIT] [testRecovery] after test
15:22:23 Completed [2/5] in 93.99s, 13 tests, 6 errors, 1 skipped <<< FAILURES!

[jasontedor]

Closed by #37725