Dotted field names that conflict with objects

[jpountz]

Elasticsearch assumes that dots in fields names are an object separator. This means that a document such as this one:

{
  "metric.value.max": 42
}

is actually indexed as if it was formatted like below:

{
  "metric": {
    "value": {
      "max": 42
    }
  }
}

And in the mappings, this translates into two object fields called metric and metric.value and a long field called metric.value.max.

This proves problematic when ingesting metrics that come from external systems such as Micrometer or OpenTelemetry, as it's not rare to have both metric.value and metric.value.max as metric names:

{
  "metric.value": 10,
  "metric.value.max": 42
}

Such a document will always fail indexing because metric.value would need to be an object field because of metric.value.max and a long field at the same time, which is illegal.

Some workarounds have been developed, such as replacing dots with underscores, or adding suffixes, but this creates a bad user experience in Kibana as users are not seeing the field names that they expect.

We should look into ways to make this supported in Elasticsearch.

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[felixbarny]

Some workarounds have been developed, such as replacing dots with underscores, or adding suffixes, but this creates a bad user experience in Kibana as users are not seeing the field names that they expect.

Is there an estimated release version for this? To choose the workaround with the right tradeoffs, for the time being, it would help us to know when we could expect this enhancement to land. See also elastic/apm#347 (comment).

[zacharymorn]

Just a thought. Is it possible to do something similar to Java auto-boxing here? For example, when encountering definitions above

{
  "metric.value": 10,
  "metric.value.max": 42
}

ES detects the potential collision and wrap things into "super type", and produce these internally:

{
  "metric.value.__long_value": 10,
  "metric.value.max": 42
}

and the translation between external and internal representation get encapsulated somewhere and hidden from the outside world?

[SylvainJuge]

Adding a suffix if there is a conflict was the initial idea we had as a work-around.

However, that assumes that metrics that conflict are always sent together, or in a specific order, and we can't guarantee that in the context of APM agents as there might be more than one agent sending such data.

If we can guarantee that sending both metric.value then metric.value.max independently and in any order produces the same result (at least to the end-user, internal storage may differ), that could work though.

[zacharymorn]

Hmm I see that's a good call out. What if ES suffix every field user defines, like such:

{
  "metric.value.__long_value": 10,
  "metric.value.max.__long_value": 42
}

This basically cause every user defined field to be of object type, and thus order shouldn't matter?

[SylvainJuge]

Yes, but in that case, unless there is a way to make this transparent to the user, they will have to use __long_value to query their metrics.

[zacharymorn]

Yes exactly.

Just to be clear, what I meant above was that the user should use and see representation in the existing format

{
  "metric.value": 10,
  "metric.value.max": 42
}

but internally these representations could be converted effectively into the following that's hidden from the user (hence the __xxx notation here)

{
  "metric.value.__long_value": 10,
  "metric.value.max.__long_value": 42
}

I think since there's already some field process logic to handle the dotted field name to object conversion, ES can potentially piggyback on that to support this conversion as well?

[jimczi]

We discussed offline and agreed that we'd prefer to consider this case under the flattened field use case.
For solutions that cannot control the name or the shape of the fields, the flattened field is a simple and powerful choice.

Although we've spotted some limitations that we'll need to solve before switching to this solution:

1. We should ensure that flattened field accepts foo.value and foo.value.max explicitly.
2. We need to handle numerics, not only keyword.
3. Kibana needs to support flattened type.
4. We need a suggester for field names that are under a flattened field. They don't appear in the mapping so we need to help users when writing queries and aggregations. That should help for the integration in Kibana.