[CI] SearchableSnapshotsRollingUpgradeIT.testMountPartialCopyAndRecoversCorrectly failing

[ywelsch]

Build scan: https://gradle-enterprise.elastic.co/s/s2xxiwwowgafs/tests/:x-pack:qa:rolling-upgrade:v7.12.0%23oneThirdUpgradedTest/org.elasticsearch.upgrades.SearchableSnapshotsRollingUpgradeIT/testMountPartialCopyAndRecoversCorrectly?top-execution=1

Repro line: ./gradlew ':x-pack:qa:rolling-upgrade:v7.12.0#oneThirdUpgradedTest' -Dtests.class="org.elasticsearch.upgrades.SearchableSnapshotsRollingUpgradeIT" -Dtests.method="testMountPartialCopyAndRecoversCorrectly" -Dtests.seed=BB551710FD94A05A -Dtests.bwc=true -Dtests.locale=fr-CH -Dtests.timezone=CST -Druntime.java=8

Reproduces locally?: yeah

Applicable branches: 7.x

Failure history:

Failure excerpt:

» [2021-10-20T06:44:48,088][WARN ][o.e.c.r.a.AllocationService] [v7.12.0-2] failing shard [failed shard, shard [mounted_index_shared_cache][0], node[fC5jsTcfSx25ArCUErRDxA], relocating [UwabCf2LQqCTvSorWaesTA], [P], recovery_source[peer recovery], s[INITIALIZING], a[id=Rp8Ufzh9QEKpyrRy2wjGjA, rId=lM4cONngRyy621rvbIbEXA], expected_shard_size[0], message [failed to create index], failure [IllegalArgumentException[only the [data_frozen] tier preference may be used for partial searchable snapshots (got: [data_frozen,data_cold,data_warm,data_hot])]], markAsStale [true]]
»  java.lang.IllegalArgumentException: only the [data_frozen] tier preference may be used for partial searchable snapshots (got: [data_frozen,data_cold,data_warm,data_hot])
»  	at org.elasticsearch.cluster.routing.allocation.DataTier$DataTierSettingValidator.validate(DataTier.java:255) ~[?:?]
»  	at org.elasticsearch.cluster.routing.allocation.DataTier$DataTierSettingValidator.validate(DataTier.java:223) ~[?:?]
»  	at org.elasticsearch.common.settings.Setting.get(Setting.java:505) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.common.settings.Setting.get(Setting.java:475) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:567) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:652) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:571) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:175) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:468) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:227) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:528) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:517) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:484) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:423) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.access$000(ClusterApplierService.java:56) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:153) ~[elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:259) [elasticsearch-7.12.0.jar:7.12.0]
»  	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:222) [elasticsearch-7.12.0.jar:7.12.0]
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
»  	at java.lang.Thread.run(Thread.java:748) [?:?]

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[fcofdez]

Another instance of this in https://gradle-enterprise.elastic.co/s/ivkskxvxuokpk and https://gradle-enterprise.elastic.co/s/z34e33e3xx6us

[tlrx]

It appears in https://gradle-enterprise.elastic.co/s/bg23lwaabm6vg

It looks like it is not only a test failure but rather something that can occur on production clusters.

[original-brownbear]

Tanguy raised this with me today, I think this is a somewhat serious bug introduced by pulling this setting into the IndexMetadata in #78668. I'll have a look at this one asap.

[original-brownbear]

closed via #80288