Updating index metadata does an expensive validation of the mapping even if unchanged #89309
Description
Updating index settings on a large number of indices can take many minutes and the resulting cluster state update might fail to cleanly publish without warning. A significant part of this slowness is that the index metadata validation that is run for each index. This validation deserialises + reserializes the mapping for every index that got updated, which for large mappings combined with a large number of updates indices can take many minutes.
100.0% [cpu=98.6%, other=1.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elasticsearch-5][masterService#updateTask][T#1]'
10/10 snapshots sharing following 26 elements
app/[email protected]/org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:150)
app/[email protected]/org.elasticsearch.index.mapper.ObjectMapper$Builder.build(ObjectMapper.java:171)
app/[email protected]/org.elasticsearch.index.mapper.ObjectMapper$Builder.build(ObjectMapper.java:64)
app/[email protected]/org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:150)
app/[email protected]/org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:110)
app/[email protected]/org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:99)
app/[email protected]/org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94)
app/[email protected]/org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370)
app/[email protected]/org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347)
app/[email protected]/org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337)
app/[email protected]/org.elasticsearch.indices.IndicesService.verifyIndexMetadata(IndicesService.java:810)
app/[email protected]/org.elasticsearch.cluster.metadata.MetadataUpdateSettingsService$1.execute(MetadataUpdateSettingsService.java:247)
app/[email protected]/org.elasticsearch.cluster.metadata.MetadataUpdateSettingsService.lambda$new$0(MetadataUpdateSettingsService.java:79)
app/[email protected]/org.elasticsearch.cluster.metadata.MetadataUpdateSettingsService$$Lambda$3405/0x00000008014c4400.execute(Unknown Source)
app/[email protected]/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:908)
app/[email protected]/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:878)
app/[email protected]/org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:248)
app/[email protected]/org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:156)
app/[email protected]/org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:110)
app/[email protected]/org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:148)
app/[email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:709)
app/[email protected]/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:260)
app/[email protected]/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[email protected]/java.lang.Thread.run(Thread.java:833)
We should find a way to skip unnecessary mapping validation when nothing about the mappings has changed, and to make use of mapping deduplication here too.