An item with the same key has already been added. (ES 5.2.0/Json: 9.0.1)

[amirxnz]

NEST/Elasticsearch.Net version:
5.2.0

Elasticsearch version:
5.2.0

json.net version
9.0.1

Description of the problem including expected versus actual behavior:
I run a query with aggregation and piped aggregation with a script.
It works fine on one field, but if i change the field i am initially aggregating against, I get the error
"An item with the same key has already been added."
I see that in the past this has been a json.net compatibility issue, however i am on 9.0.1.

I have taken the query produced by the request and run it against the ES directly and it works fine.

This works:

var searchResults =
                    client.Search<object>(s => s
                        .Size(0)
                        .Aggregations(
                            f =>
                                f.Terms("Truck",
                                    t =>
                                        t.Field("deviceId")
                                            .Aggregations(
                                                a =>a.Sum("Breaking Points",
                                                sm => sm.Field("number_70035_BreakingPoints"))
                                                          .Sum("Acceleration Points",
                                                            sm => sm.Field("number_70034_AccelerationPoints"))
                                                          .BucketScript("TotalPoints",
                                                            bs =>
                                                                bs.BucketsPath(
                                                                        bp =>
                                                                            bp.Add("BreakingPointsSumPath",
                                                                                    "Breaking Points")
                                                                                .Add("AccelerationPointsSumPath",
                                                                                    "Acceleration Points")
                                                                                    )
                                                                    .Script(
                                                                   "params.BreakingPointsSumPath+params.AccelerationPointsSumPath"))))));

This does not work:

 var searchResults =
                    client.Search<object>(s => s
                        .Size(0)
                        .Aggregations(
                            f =>
                                f.Terms("Truck",
                                    t =>
                                        t.Field("string_998_Driver")
                                            .Aggregations(
                                                a => a.Sum("Breaking Points",
                                                            sm => sm.Field("number_70035_BreakingPoints"))
                                                        .Sum("Acceleration Points",
                                                            sm => sm.Field("number_70034_AccelerationPoints"))
                                                         .BucketScript("TotalPoints",
                                                            bs =>
                                                                bs.BucketsPath(
                                                                        bp =>
                                                                            bp.Add("BreakingPointsSumPath",
                                                                                    "Breaking Points")
                                                                                .Add("AccelerationPointsSumPath",
                                                                                    "Acceleration Points")
                                                                                    )
                                                                    .Script(
                                                                        "params.BreakingPointsSumPath+params.AccelerationPointsSumPath"))))));

The only thing i changed was the top aggregation from truck to driver.

Thanks for the help!

Provide DebugInformation (if relevant):
FailureReason: Unrecoverable/Unexpected BadResponse while attempting POST http://blablabla:9200/ax/ax/_search

• [1] BadResponse: Node: http://blablabla:9200/ Exception: ArgumentException Took: 00:00:00.8928303
  Audit exception in step 1 BadResponse:
  System.ArgumentException: An item with the same key has already been added.
  at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
  at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add) at Nest.AggregateJsonConverter.GetSubAggregates(JsonReader reader, JsonSerializer serializer) in C:\code\elasticsearch-net\src\Nest\Aggregations\AggregateJsonConverter.cs:line 434 at Nest.AggregateJsonConverter.GetKeyedBucket(JsonReader reader, JsonSerializer serializer) in C:\code\elasticsearch-net\src\Nest\Aggregations\AggregateJsonConverter.cs:line 675 at Nest.AggregateJsonConverter.ReadBucket(JsonReader reader, JsonSerializer serializer) in C:\code\elasticsearch-net\src\Nest\Aggregations\AggregateJsonConverter.cs:line 172 at Nest.AggregateJsonConverter.GetMultiBucketAggregate(JsonReader reader, JsonSerializer serializer) in C:\code\elasticsearch-net\src\Nest\Aggregations\AggregateJsonConverter.cs:line 490 at Nest.AggregateJsonConverter.ReadAggregate(JsonReader reader, JsonSerializer serializer) in C:\code\elasticsearch-net\src\Nest\Aggregations\AggregateJsonConverter.cs:line 131 at Nest.AggregateJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in C:\code\elasticsearch-net\src\Nest\Aggregations\AggregateJsonConverter.cs:line 18 at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader) at Nest.JsonNetSerializer.Deserialize[T](Stream stream) in C:\code\elasticsearch-net\src\Nest\CommonAbstractions\SerializationBehavior\JsonNetSerializer.cs:line 124 at Elasticsearch.Net.ResponseBuilder1.SetBody(ElasticsearchResponse1 response, Stream stream) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\ResponseBuilder.cs:line 83 at Elasticsearch.Net.ResponseBuilder1.ToResponse() in C:\code\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\ResponseBuilder.cs:line 37
  at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 158
  at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 413
  at Elasticsearch.Net.Transport1.Request[TReturn](HttpMethod method, String path, PostData1 data, IRequestParameters requestParameters) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs:line 69
  Inner Exception: An item with the same key has already been added.

[Mpdreamz]

Hi @amirxnz sorry for the late reply, we were all at elasticon last week + travelling.

Are you able to share the responses of both somehow? Privately is also fine just email ml >@< elastic.co

If you EnableDebugMode() on ConnectionSettings() and then share the responses DebugInformation with us we should be able to get to the root cause easier.

[amirxnz]

thanks @Mpdreamz!
I think I already put the DebugInformation in the original post?
(I just changed the url)
Is there any other information I can supply that will help?

[Mpdreamz]

The raw responses from the searches would help. Maybe you can snoop them using fiddler and sent them our way (provided you are at liberty to do so).

[amirxnz]

Ok, I'll email you privately

[kam99]

Hello, @Mpdreamz , @amirxnz

I have the same issue here.

Since it has not been fixed yet, maybe yo can suggest some workaround to get a "calculated field" over aggregations?

Thanks in advance

var result2 = elasticClient.Search<dynamic>(s => s
	.AllIndices()
	.AllTypes()
	.Take(0)
	.Aggregations(a => a
		.Terms("branch", t => t.Field("saleBranchName")
			.Aggregations(aa => aa
				.Sum("net", ss => ss.Field("sellNetAmount"))
				.Sum("cost", ss => ss.Field("sellCostAmount"))
				.BucketScript("margin", bs => bs
					.BucketsPath(bp => bp
						.Add("totalNet", "net")
						.Add("totalCost", "cost")
					)
					.Script("params.totalCost / params.totalNet * 100")
				)
			)
		)
	));

EnableDebugMode():

# FailureReason: Unrecoverable/Unexpected BadResponse while attempting POST on http://localhost:9200/_search
 - [1] BadResponse: Node: http://localhost:9200/ Exception: ArgumentException Took: 00:00:00.3670718
# Audit exception in step 1 BadResponse:
System.ArgumentException: An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Nest.AggregateJsonConverter.GetSubAggregates(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.GetKeyedBucket(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadBucket(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.GetMultiBucketAggregate(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadAggregate(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Nest.JsonNetSerializer.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder`1.SetBody(ElasticsearchResponse`1 response, Stream stream)
   at Elasticsearch.Net.ResponseBuilder`1.ToResponse()
   at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
# Inner Exception: An item with the same key has already been added.
System.ArgumentException: An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Nest.AggregateJsonConverter.GetSubAggregates(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.GetKeyedBucket(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadBucket(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.GetMultiBucketAggregate(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadAggregate(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Nest.JsonNetSerializer.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder`1.SetBody(ElasticsearchResponse`1 response, Stream stream)
   at Elasticsearch.Net.ResponseBuilder`1.ToResponse()
   at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
# Exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: An item with the same key has already been added. ---> System.ArgumentException: An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Nest.AggregateJsonConverter.GetSubAggregates(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.GetKeyedBucket(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadBucket(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.GetMultiBucketAggregate(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadAggregate(JsonReader reader, JsonSerializer serializer)
   at Nest.AggregateJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Nest.JsonNetSerializer.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder`1.SetBody(ElasticsearchResponse`1 response, Stream stream)
   at Elasticsearch.Net.ResponseBuilder`1.ToResponse()
   at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
   at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[T](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
   at Elasticsearch.Net.ElasticLowLevelClient.Search[T](PostData`1 body, Func`2 requestParameters)
   at Nest.LowLevelDispatch.SearchDispatch[T](IRequest`1 p, PostData`1 body)
   at Nest.ElasticClient.<>c__DisplayClass568_0`2.<Search>b__0(ISearchRequest p, PostData`1 d)
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 responseGenerator, Func`3 dispatch)
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 dispatch)
   at Nest.ElasticClient.Search[T,TResult](ISearchRequest request)
   at Nest.ElasticClient.Search[T](ISearchRequest request)
   at Nest.ElasticClient.Search[T,TResult](Func`2 selector)
   at Nest.ElasticClient.Search[T](Func`2 selector)

[kam99]

@Mpdreamz , @amirxnz

for those who interested - found the root of the issue

it's about bad response from ELASTIC itself
(not related t0 NEST/Elasticsearch.Net directly)

see dup values of margin fields:

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 6,
        "successful": 6,
        "failed": 0
    },
    "hits": {
        "total": 3980,
        "max_score": 0.0,
        "hits": []
    },
    "aggregations": {
        "branch": {
            "doc_count_error_upper_bound": -1,
            "sum_other_doc_count": 3142,
            "buckets": [{
                "key": "key1",
                "doc_count": 170,
                "cost": {
                    "value": 43
                },
                "net": {
                    "value": 68
                },
                "margin": {
                    "value": 62.934600543653104
                },
                "margin": {
                    "value": 62.934600543653104
                }
            }, {
                "key": "key2",
                "doc_count": 209,
                "cost": {
                    "value": 41
                },
                "net": {
                    "value": 64
                },
                "margin": {
                    "value": 63.98929828455564
                },
                "margin": {
                    "value": 63.98929828455564
                }
            }...]
        }
    }
}

can be worked around via custom JsonNetSerializer that forcibly ignores duplicate values
(and then feed it to NEST ElasticClient settings)

[Mpdreamz]

@kam99 awesome thank you for responding back 👍 , totally missed your update.

This has been fixed in Elasticsearch 6.0 as per: elastic/elasticsearch#22073