CI: testWatcherWithApiKey fails #63088
Description
Build scan: https://gradle-enterprise.elastic.co/s/aylzo4qv6qdaa
Repro line:
./gradlew ':x-pack:qa:full-cluster-restart:v7.6.0#upgradedClusterTest' \
--tests "org.elasticsearch.xpack.restart.FullClusterRestartIT.testWatcherWithApiKey" \
-Dtests.seed=63828CA06BFC16ED
Reproduces locally?: No such luck
Applicable branches:
master
Failure history:
https://build-stats.elastic.co/goto/092f99a325a05a8cd7f07aaac1223cf4
Failure excerpt:
org.elasticsearch.xpack.restart.FullClusterRestartIT > testWatcherWithApiKey FAILED
org.junit.ComparisonFailure: expected:<[execut]ed> but was:<[throttl]ed>
at __randomizedtesting.SeedInfo.seed([63828CA06BFC16ED:8AD946D694CF07C4]:0)
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.elasticsearch.xpack.restart.FullClusterRestartIT.lambda$testWatcherWithApiKey$1(FullClusterRestartIT.java:260)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:934)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:907)
at org.elasticsearch.xpack.restart.FullClusterRestartIT.testWatcherWithApiKey(FullClusterRestartIT.java:256)
REPRODUCE WITH: ./gradlew ':x-pack:qa:full-cluster-restart:v7.6.0#upgradedClusterTest' --tests "org.elasticsearch.xpack.restart.FullClusterRestartIT.testWatcherWithApiKey" -Dtests.seed=63828CA06BFC16ED -Dtests.security.manager=true -Dtests.locale=es-NI -Dtests.timezone=Canada/Yukon -Dtests.distribution=default -Druntime.java=11
Suite: Test class org.elasticsearch.xpack.restart.FullClusterRestartIT
This might be relevant, I'm not sure.
=== Standard output of node `node{:x-pack:qa:full-cluster-restart:v7.6.0-1}` ===
» ↓ errors and warnings from /dev/shm/elastic+elasticsearch+master+default-distro+bwc/BWC_VERSION/7.6.0/nodes/centos-7&&immutable/x-pack/qa/full-cluster-restart/build/testclusters/v7.6.0-1/logs/es.stdout.log ↓
» WARN ][o.e.b.BootstrapChecks ] [v7.6.0-1] HTTPS is required in order to use the API key service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the API key service using the [xpack.security.authc.api_key.enabled] setting
» ↑ repeated 2 times ↑
» WARN ][o.e.d.FileBasedSeedHostsProvider] [v7.6.0-1] expected, but did not find, a dynamic hosts list at [/dev/shm/elastic+elasticsearch+master+default-distro+bwc/BWC_VERSION/7.6.0/nodes/centos-7&&immutable/x-pack/qa/full-cluster-restart/build/testclusters/v7.6.0-1/config/unicast_hosts.txt]
» ↑ repeated 2 times ↑
» WARN ][o.e.c.m.MetadataIndexTemplateService] [v7.6.0-1] index template [.triggered_watches] has index patterns [.triggered_watches*] matching patterns from existing older templates [.triggered_watches] with patterns (.triggered_watches => [.triggered_watches*]); this template [.triggered_watches] will take precedence during new index creation
» WARN ][o.e.c.m.MetadataIndexTemplateService] [v7.6.0-1] index template [.watches] has index patterns [.watches*] matching patterns from existing older templates [.watches] with patterns (.watches => [.watches*]); this template [.watches] will take precedence during new index creation
» WARN ][o.e.c.r.a.AllocationService] [v7.6.0-1] [.triggered_watches][0] marking unavailable shards as stale: [hwrtSB9URSK7Y47XT0Givw]
» WARN ][o.e.c.r.a.AllocationService] [v7.6.0-1] [.watches][0] marking unavailable shards as stale: [8g4NUyMEQeSIG5GN1Ubvqg]
» WARN ][o.e.c.r.a.AllocationService] [v7.6.0-1] [bwc_watch_index][0] marking unavailable shards as stale: [626vP1RTREWSPLy2pzGjjQ]
» WARN ][o.e.c.r.a.AllocationService] [v7.6.0-1] [testshrink][0] marking unavailable shards as stale: [hMQpSsyERymdM5knIazURQ]
» WARN ][o.e.c.r.a.AllocationService] [v7.6.0-1] [testshrink][4] marking unavailable shards as stale: [sNQ_oiHoRmGExAT1yyBBMg]
» WARN ][o.e.c.r.a.AllocationService] [v7.6.0-1] [testshrink][2] marking unavailable shards as stale: [ITU2k_ypQr2BkNQifJUxNw]
» WARN ][o.e.s.RestoreService ] [v7.6.0-1] [snapshot-testforbiddisablesoftdeletesonrestore/GjAdNXcbSpKqODnuCU4JfQ] failed to restore snapshot
» org.elasticsearch.snapshots.SnapshotRestoreException: [repo:snapshot-testforbiddisablesoftdeletesonrestore/GjAdNXcbSpKqODnuCU4JfQ] cannot disable setting [index.soft_deletes.enabled] on restore
» at org.elasticsearch.snapshots.RestoreService$1.updateIndexSettings(RestoreService.java:530) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.snapshots.RestoreService$1.execute(RestoreService.java:300) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:674) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
» at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
» at java.lang.Thread.run(Thread.java:834) [?:?]
» WARN ][r.suppressed ] [v7.6.0-1] path: /_snapshot/repo/snapshot-testforbiddisablesoftdeletesonrestore/_restore, params: {repository=repo, wait_for_completion=true, snapshot=snapshot-testforbiddisablesoftdeletesonrestore}
» org.elasticsearch.snapshots.SnapshotRestoreException: [repo:snapshot-testforbiddisablesoftdeletesonrestore/GjAdNXcbSpKqODnuCU4JfQ] cannot disable setting [index.soft_deletes.enabled] on restore
» at org.elasticsearch.snapshots.RestoreService$1.updateIndexSettings(RestoreService.java:530) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.snapshots.RestoreService$1.execute(RestoreService.java:300) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:674) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
» at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
» at java.lang.Thread.run(Thread.java:834) [?:?]
» ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v7.6.0-1] uncaught exception in thread [elasticsearch[v7.6.0-1][generic][T#3]]
» java.lang.IllegalArgumentException: Malformed [mappings] section for type [dynamic], should include an inner object describing the mapping
» at org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest.source(PutIndexTemplateRequest.java:342) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest.source(PutIndexTemplateRequest.java:381) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.metadata.TemplateUpgradeService.upgradeTemplates(TemplateUpgradeService.java:145) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.cluster.metadata.TemplateUpgradeService.lambda$clusterChanged$1(TemplateUpgradeService.java:132) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:674) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
» at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
» at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
» at java.lang.Thread.run(Thread.java:834) [?:?]
» ↓ last 40 non error or warning messages from /dev/shm/elastic+elasticsearch+master+default-distro+bwc/BWC_VERSION/7.6.0/nodes/centos-7&&immutable/x-pack/qa/full-cluster-restart/build/testclusters/v7.6.0-1/logs/es.stdout.log ↓
» [2020-09-30T16:54:51,994][INFO ][o.e.x.c.t.IndexTemplateRegistry] [v7.6.0-1] upgrading legacy template [.ml-meta] for [ml] from version [7060099] to version [8000099]
```