Conflicting doc values in mappings causes shard initialization failures

[KeithTr]

I found a way to create conflicting mappings using doc values in Elasticsearch 2.2.0.

Steps to reproduce:

# Create index with doc values enabled
curl -XPUT localhost:9200/index -d '{
    "mappings": {
        "type1": {
            "properties": {
                "field_name": {
                    "type": "string",
                    "index": "not_analyzed",
                    "doc_values": true
                }
            }
        }
    }
}'

# Add type with doc values disabled
# These must be done as separate requests, otherwise ES recognizes that they are conflicting
curl -XPUT localhost:9200/index/_mapping/type2 -d '{
    "properties": {
        "field_name": {
            "type": "string",
            "index": "not_analyzed",
            "doc_values": false
        }
    }
}'

Getting the mappings returns conflicting doc values for this index: one field has doc values disabled, the other has doc values enabled.

{
   "index": {
      "mappings": {
         "type2": {
            "properties": {
               "field_name": {
                  "type": "string",
                  "index": "not_analyzed",
                  "doc_values": false
               }
            }
         },
         "type1": {
            "properties": {
               "field_name": {
                  "type": "string",
                  "index": "not_analyzed"
               }
            }
         }
      }
   }
}

This does not cause problems for the cluster until the nodes are restarted. When the nodes try to initialize the index with conflicting mappings it can cause nodes to enter an endless loop of trying (and failing) to allocate shards:

[2016-02-09 16:04:56,313][WARN ][indices.cluster          ] [Fantasia] [index] failed to add mapping [type1], source [{"type1":{"properties":{"field_name":{"type":"string","index":"not_analyzed"}}}}]
java.lang.IllegalArgumentException: Mapper for [field_name] conflicts with existing mapping in other types:
[mapper [field_name] has different [doc_values] values, cannot change from disabled to enabled]
    at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:170)
    at org.elasticsearch.index.mapper.MapperService.checkMappersCompatibility(MapperService.java:402)
    at org.elasticsearch.index.mapper.MapperService.checkMappersCompatibility(MapperService.java:411)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:314)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:272)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:388)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:348)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:164)
    at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:600)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:762)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-02-09 16:04:56,740][WARN ][indices.cluster          ] [Fantasia] [[index][4]] marking and sending shard failed due to [failed to update mappings]
java.lang.IllegalArgumentException: Mapper for [field_name] conflicts with existing mapping in other types:
[mapper [field_name] has different [doc_values] values, cannot change from disabled to enabled]
    at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:170)
    at org.elasticsearch.index.mapper.MapperService.checkMappersCompatibility(MapperService.java:402)
    at org.elasticsearch.index.mapper.MapperService.checkMappersCompatibility(MapperService.java:411)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:314)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:272)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:388)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:348)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:164)
    at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:600)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:762)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-02-09 16:04:56,743][WARN ][cluster.action.shard     ] [Fantasia] [index][4] received shard failed for [index][4], node[tf4WmuoCQHKoHctABxTNwg], [P], v[89], s[INITIALIZING], a[id=zgBVat8RTKaWToeKvEQ9tg], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-02-09T22:04:54.620Z], details[failed to update mappings, failure IllegalArgumentException[Mapper for [field_name] conflicts with existing mapping in other types:
[mapper [field_name] has different [doc_values] values, cannot change from disabled to enabled]]]], indexUUID [Nn0u7aW-RNiSyba5s1YrsQ], message [failed to update mappings], failure [IllegalArgumentException[Mapper for [field_name] conflicts with existing mapping in other types:
[mapper [field_name] has different [doc_values] values, cannot change from disabled to enabled]]]
java.lang.IllegalArgumentException: Mapper for [field_name] conflicts with existing mapping in other types:
[mapper [field_name] has different [doc_values] values, cannot change from disabled to enabled]
    at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:170)
    at org.elasticsearch.index.mapper.MapperService.checkMappersCompatibility(MapperService.java:402)
    at org.elasticsearch.index.mapper.MapperService.checkMappersCompatibility(MapperService.java:411)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:314)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:272)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:388)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:348)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:164)
    at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:600)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:762)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-02-09 16:04:56,745][WARN ][indices.cluster          ] [Fantasia] [[index][3]] marking and sending shard failed due to [master [{Fantasia}{tf4WmuoCQHKoHctABxTNwg}{127.0.0.1}{127.0.0.1:9300}] marked shard as started, but shard has not been created, mark shard as failed]
[2016-02-09 16:04:56,746][WARN ][cluster.action.shard     ] [Fantasia] [index][3] received shard failed for [index][3], node[tf4WmuoCQHKoHctABxTNwg], [P], v[140], s[STARTED], a[id=LW-dtt99Sq6fDsfAMy7qBw], indexUUID [Nn0u7aW-RNiSyba5s1YrsQ], message [master [{Fantasia}{tf4WmuoCQHKoHctABxTNwg}{127.0.0.1}{127.0.0.1:9300}] marked shard as started, but shard has not been created, mark shard as failed], failure [Unknown]
[2016-02-09 16:04:56,746][WARN ][indices.cluster          ] [Fantasia] [[index][1]] marking and sending shard failed due to [master [{Fantasia}{tf4WmuoCQHKoHctABxTNwg}{127.0.0.1}{127.0.0.1:9300}] marked shard as started, but shard has not been created, mark shard as failed]
[2016-02-09 16:04:56,746][WARN ][cluster.action.shard     ] [Fantasia] [index][1] received shard failed for [index][1], node[tf4WmuoCQHKoHctABxTNwg], [P], v[134], s[STARTED], a[id=xuQ5AlXZR9SoR-5zeb161Q], indexUUID [Nn0u7aW-RNiSyba5s1YrsQ], message [master [{Fantasia}{tf4WmuoCQHKoHctABxTNwg}{127.0.0.1}{127.0.0.1:9300}] marked shard as started, but shard has not been created, mark shard as failed], failure [Unknown]
[2016-02-09 16:04:56,747][WARN ][cluster.action.shard     ] [Fantasia] [index][4] received shard failed for [index][4], node[tf4WmuoCQHKoHctABxTNwg], [P], v[89], s[INITIALIZING], a[id=zgBVat8RTKaWToeKvEQ9tg], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-02-09T22:04:54.620Z], details[failed to update mappings, failure IllegalArgumentException[Mapper for [field_name] conflicts with existing mapping in other types:
[mapper [field_name] has different [doc_values] values, cannot change from disabled to enabled]]]], indexUUID [Nn0u7aW-RNiSyba5s1YrsQ], message [master {Fantasia}{tf4WmuoCQHKoHctABxTNwg}{127.0.0.1}{127.0.0.1:9300} marked shard as initializing, but shard is marked as failed, resend shard failure], failure [Unknown]
[2016-02-09 16:04:56,747][WARN ][cluster.action.shard     ] [Fantasia] [index][0] received shard failed for [index][0], node[tf4WmuoCQHKoHctABxTNwg], [P], v[117], s[INITIALIZING], a[id=L2V8OQcFSZmi8X55QereqQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-02-09T22:04:54.620Z], details[failed to update mappings, failure IllegalArgumentException[Mapper for [field_name] conflicts with existing mapping in other types:
[mapper [field_name] has different [doc_values] values, cannot change from disabled to enabled]]]], indexUUID [Nn0u7aW-RNiSyba5s1YrsQ], message [master {Fantasia}{tf4WmuoCQHKoHctABxTNwg}{127.0.0.1}{127.0.0.1:9300} marked shard as initializing, but shard is marked as failed, resend shard failure], failure [Unknown]

The above failure to allocate shards is not guaranteed, but I have always been able to get it to happen by restarting the node several times.

[rjernst]

This was just fixed in 2.x: #16264. Currently that will be 2.3.