Aggregations using "missing" lead to AggregationExecutionException because they use wrong datatype if the field was never stored in an index #20163
Description
Elasticsearch version: 2.1.2 and 2.3.5
Plugins installed: [none]
JVM version:1.8.0_80
OS version:Windows 10
Description of the problem including expected versus actual behavior:
We are working on a feature where different fields can be added to documents and we want them to be handled as double value.
However we cannot say upfront which datatype these items should have and thus cannot define datatypes directly, but rather need to use dynamic templates in the mapping, so our mapping is something like the following:
{
"testtype": {
"dynamic_templates": [{
"num_fields": {
"match": "amount*",
"mapping": {
"type": "double"
}
}
}]
}
}
When using aggregations together with specifying a missing-value, we found that the returned aggregation-key is different depending upon if the field was ever stored in the index or not.
Also if we query across two indices, one which contains a value for that field and one which does not, we get the following exceptions when doing a Bucket-Term-Aggregation together with a missing-attribute:
AggregationExecutionException[Merging/Reducing the aggregations failed when computing the aggregation [ Name: testagg, Type: terms ] because: the field you gave in the aggregation query existed as two different types in two different indices]
In the test-case below, the 2nd query returns a string "-1", although ideally we would get a Double value -1.0. In the 3rd query we get the exception.
Steps to reproduce:
- See test-case below
public class OldIndexWithoutMappingTest extends ESIntegTestCase {
@Test
public void testTwoIndicesOneWithoutMapping() throws Exception {
initializeIndex();
// aggregate by terms with a missing-value set
final TermsBuilder agg = AggregationBuilders.terms("testagg").field("amount-high").missing(-1);
// first try with first index only, here the "missing" value is returned as double
SearchResponse searchResponse = client().prepareSearch("testindex-1").addAggregation(agg).execute().actionGet();
assertEquals(0, searchResponse.getFailedShards());
assertEquals(2, searchResponse.getHits().getTotalHits());
assertEquals(2, ((Terms)searchResponse.getAggregations().get("testagg")).getBuckets().size());
assertEquals(1234.0, ((Terms)searchResponse.getAggregations().get("testagg")).getBucketByKey("1234.0").getKey());
assertEquals(-1.0, ((Terms)searchResponse.getAggregations().get("testagg")).getBucketByKey("-1.0").getKey());
// then with second index only, here the missing value is returned as string!
searchResponse = client().prepareSearch("testindex-2").addAggregation(agg).execute().actionGet();
assertEquals(0, searchResponse.getFailedShards());
assertEquals(1, searchResponse.getHits().getTotalHits());
assertEquals(1, ((Terms)searchResponse.getAggregations().get("testagg")).getBuckets().size());
assertEquals("-1", ((Terms)searchResponse.getAggregations().get("testagg")).getBucketByKey("-1").getKey());
// and finally with both indices, here it fails because we take the missing value from both indices and these are different!
searchResponse = client().prepareSearch("testindex*").addAggregation(agg).execute().actionGet();
assertEquals(0, searchResponse.getFailedShards());
assertEquals(2, searchResponse.getHits().getTotalHits());
assertEquals(2, ((Terms)searchResponse.getAggregations().get("testagg")).getBuckets().size());
}
private void initializeIndex() throws Exception {
String testMapping = getMapping(XContentFactory.jsonBuilder().startObject().startObject("testtype")).string();
System.out.println("Mapping: " + testMapping);
assertAcked(prepareCreate("testindex-1").addMapping("testtype", testMapping));
assertAcked(prepareCreate("testindex-2").addMapping("testtype", testMapping));
List<IndexRequestBuilder> indexBuilders = new ArrayList<>();
indexBuilders.add(client().prepareIndex("testindex-1", "testtype", "index1-1").setSource("{\"amount-high\":1234}"));
indexBuilders.add(client().prepareIndex("testindex-1", "testtype", "index1-2").setSource("{}"));
indexBuilders.add(client().prepareIndex("testindex-2", "testtype", "index2-1").setSource("{}"));
indexRandom(true, indexBuilders);
}
private XContentBuilder getMapping(XContentBuilder testtype) throws IOException {
return testtype.startArray("dynamic_templates")
.startObject()
.startObject("num_fields")
.field("match", "amount*")
.startObject("mapping")
.field("type", "double").endObject()
.endObject()
.endObject()
.endArray();
}
}