Conflicting mappings causes runaway shard failures on upgrade

[clintongormley]

Start Elasticsearch 1.6.0, and run the following commands:

DELETE *

PUT bad
{
  "mappings": {
    "x": {
      "properties": {
        "foo": {
          "type": "string"
        }
      }
    },
    "y": {
      "properties": {
        "foo": {
          "type": "date"
        }
      }
    }
  }
}

PUT bad/x/1
{"foo": "bar"}

PUT bad/y/1
{"foo": "2015-01-01"}

POST _flush

Then shutdown and start ES compiled from master. The bad index fails to recover because of the conflicting mappings, and the failures just keep being repeated endlessly:

[2015-06-24 18:22:46,644][WARN ][indices.cluster          ] [Crossbones] [bad] failed to add mapping [y], source [{"y":{"properties":{"foo":{"t
ype":"date","format":"dateOptionalTime"}}}}]
java.lang.IllegalArgumentException: Mapper for [foo] conflicts with existing mapping in other types[mapper [foo] cannot be changed from type [s
tring] to [date]]
        at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:117)
        at org.elasticsearch.index.mapper.MapperService.checkNewMappersCompatibility(MapperService.java:350)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:305)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:255)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:422)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:376)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:181)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:484)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThr
eadPoolExecutor.java:209)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolE
xecutor.java:179)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
[2015-06-24 18:22:46,701][WARN ][indices.cluster          ] [Crossbones] [[bad][0]] marking and sending shard failed due to [failed to update m
appings]
java.lang.IllegalArgumentException: Mapper for [foo] conflicts with existing mapping in other types[mapper [foo] cannot be changed from type [s
tring] to [date]]
        at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:117)
        at org.elasticsearch.index.mapper.MapperService.checkNewMappersCompatibility(MapperService.java:350)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:305)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:255)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:422)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:376)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:181)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:484)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThr
eadPoolExecutor.java:209)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolE
xecutor.java:179)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
[2015-06-24 18:22:46,703][WARN ][cluster.action.shard     ] [Crossbones] [bad][0] received shard failed for [bad][0], node[_DucCUykRy6JHLfcchxVVg], [P], s[INITIALIZING], unassigned_info[[reason=CLUSTER_RECOVERED], at[2015-06-24T16:22:46.149Z]], indexUUID [RET0PNdWQQWhX_fglK4FIw], reason [shard failure [failed to update mappings][IllegalArgumentException[Mapper for [foo] conflicts with existing mapping in other types[mapper [foo] cannot be changed from type [string] to [date]]]]]
[2015-06-24 18:22:46,705][WARN ][indices.cluster          ] [Crossbones] [[bad][1]] marking and sending shard failed due to [failed to update mappings]
java.lang.IllegalArgumentException: Mapper for [foo] conflicts with existing mapping in other types[mapper [foo] cannot be changed from type [string] to [date]]
        at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:117)
        at org.elasticsearch.index.mapper.MapperService.checkNewMappersCompatibility(MapperService.java:350)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:305)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:255)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:422)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:376)
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:181)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:484)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:209)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:179)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
[2015-06-24 18:22:46,706][WARN ][cluster.action.shard     ] [Crossbones] [bad][0] received shard failed for [bad][0], node[_DucCUykRy6JHLfcchxVVg], [P], s[INITIALIZING], unassigned_info[[reason=CLUSTER_RECOVERED], at[2015-06-24T16:22:46.149Z]], indexUUID [RET0PNdWQQWhX_fglK4FIw], reason [master [Crossbones][_DucCUykRy6JHLfcchxVVg][Slim-2.local][inet[/127.0.0.1:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]

[szroland]

A simpler way to reproduce this is to try put this mapping to ES master, and it will fail.
The ability to have a field with the same name in different types with uncompatible mapping seems to have been deliberately removed in #11812 by @rjernst.

[clintongormley]

@szroland the change is intentional. This issue is about what happens when you upgrade to 2.0 with conflicting mappings. We should fail once and stop, instead of trying over and over again.