Flattened object fields design + implementation

[jtibshirani]

Main issue: #25312
Feature branch: https://github.com/elastic/elasticsearch/tree/object-fields

Note: this field type was previously called embedded_json, so many PRs + comments will refer to that name.

Motivation

Documents sometimes contain large objects, where only a small number of the fields are frequently used in searches. By default, we create dynamic mappings for all key-value pairs in the object, and index each one as a separate field. This has a number of downsides:

• We’re creating a large number of distinct fields in Lucene.
• Each field becomes its own entry in the mappings, which can lead to a large cluster state.
• From a UX perspective, the list of fields can appear quite cluttered, and it can be difficult to understand which fields are most critical.

In some cases, the number of field keys not just a large known number, but unbounded. Here, it can be difficult to successfully model the data at all.

Feature Summary

This feature will allow an entire JSON object to be indexed into a field, and provide limited search functionality over the field's contents. Given an object field header of the form {"content-type": "text/html", "referer": "https://google.com"}, its content will be analyzed into the individual tokens content-type\0text/html, referer\0https://google.com (where \0 is some suitable delimiter). Additionally, tokens are created for each value alone: text/html, https://google.com. Each leaf value in the object becomes its own token, and no further analysis is applied to the individual values.

In addition to being able to retrieve the JSON blob (through fetching source, or as a stored field), we plan to support queries of the following forms:

• key: header, value: application/json, for example {"term": {"header": "application/json"}}
• key: header.content-type, value: application/json, for example {"term": {"header.content-type": "application/json"}}

Note that it is not possible to search the prefixed tokens directly, i.e. the following query will not return results: {"term": {"header": "content-type\0application/json"}}.

As a first pass, the following query types will be allowed: term, terms, terms_set, range (without special support for numerics), prefix, match family (insofar as they work for keyword fields), query_string, simple_query_string, exists.

In this first version, it will not be possible to refer to field keys using wildcards, as in {"header.content-*": "application/json"}. Under the proposed API/ implementation, supporting field wildcards would add significant complexity and uncertainty around performance.

Potential Extensions

• Collect more feedback on the importance of numeric fields, and explore adding more targeted support for them. As an example, users may want to perform true range queries on numeric fields within the object.
• Introduce a way for certain JSON keys to be 'promoted' into individual fields. One approach we're considering is to extend copy_to to work on entire objects, so that the same JSON blob could be added both as a 'queryable object' field, and also as normal object with explicit subfield definitions.
• Add support for additional query types.
  • By performing proper escaping and taking advantage of prefix_length, we could likely support wildcard, regexp, and fuzzy queries.
  • As mentioned in the original issue, we could consider tokenizing values on whitespace. This could allow for better support of positional queries like match_phrase.
• Explore adding support for aggregations + sorting. This idea needs a lot more research, but could maybe be accomplished by creating additional 'doc value fields', then adding a filtering layer when fetching doc values that checks for the field prefix. Update: we've decided to include this in the first version.
• Explore adding support for highlighting, since with large JSON blobs it can be difficult to tell which key-value pairs matched the query.
• Potentially allow for the field contents to be specified as a JSON string, in addition to accepting an object embedded in the document source.

Implementation Plan
Core items:

• Create a new field type that accepts an object and indexes its leaf values. Verify that the object field can be used in queries of the form {"header": "application/json"}. Add a simple JSON field mapper. #33923
• Index prefixed tokens, and support searching for values based on key as in {"header.content-type": "application/json"}. When parsing JSON fields, also create tokens prefixed with the field key. #34207 Add support for querying JSON fields based on key. #34621
• Add support for storing the field by adding a single stored field containing the whole JSON blob. Add support for storing JSON fields. #34942
• Add a limit to the depth of the objects that will be indexed. Enforce a limit on the depth of the JSON object. #35063
• Add documentation. Add documentation for JSON fields. #35281
• Add tests for the supported query types. Add tests for the supported query types. #35319
• Revisit the field lookup logic with potential optimizations. Prevent slow field lookups when JSON fields are present. #39872
• Rename field type to embedded_json. Rename 'json' to 'embedded_json'. #40712
• Explore adding support for doc values, to allow for aggregations + sorting. Add doc values support for JSON fields. #40069
• Address issues in doc values implementation (Add doc values support for JSON fields. #40069 (comment)). Rebase keyed JSON ordinals to start from zero. #41282 Make sure to apply eager_global_ordinals for keyed JSON fields. #41319
• Perform benchmarks.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jpountz]

The initial example on #25312 suggests that object fields would be indexed like text since it indexes city: "New York" as [ "city:new", "city:york" ] but your plan here doesn't mention analysis and gives examples with fields that are usually indexed as keywords like referer and content-type. Which route do we want to follow? Keyword-like indexing could be achieved with a keyword analyzer, but I feel like there is more ask for actual keywords which might enable support for aggregations in the future like you mentioned.

Additionally, tokens are created for each value alone: text/html, https://google.com

Why do we plan to index individual tokens alone? I suspect most users won't want/need to search the entire object, meaning that this feature will double the size of the inverted index for a feature they don't need? Since you already mentioned having some sort of copy_to support, it could be a way that users who need this feature could copy the content of all fields to another catch-all field?

As a first pass, the following query types will be allowed: term, terms, terms_set, range (without special support for numerics), prefix, match family (insofar as they work for keyword fields), common, query_string, simple_query_string, exists. Highlighting will be supported.

If we go with keyword-style indexing, then we should probably skip highlighting, which is only useful with text fields? (matched queries are typically used instead for structured content)

Explore adding support for aggregations + sorting. This idea needs a lot more research, but could maybe be accomplished by creating additional 'doc value fields', then adding a filtering layer when fetching doc values that checks for the field prefix.

+1 I suspect it will be quite easy actually.

Create benchmarks for searching based on object keys.

That would be nice of course, but I'm not worried about it being slow since term queries on an indexed object would translate to a term query at the lucene level.

[jtibshirani]

Thanks @jpountz for your thoughts.

I feel like there is more ask for actual keywords which might enable support for aggregations in the future like you mentioned.

I agree — in the potential use cases we’ve seen, the data is better modelled as keywords than text. The most critical feature is the ability to filter by an exact match on key-value pairs, and performing aggregations and sorting on the values would also be nice. A couple examples of these use cases:

• A user can have a number of metricbeats, each of which contains a large set of unique metrics. Only a small number of these fields are regularly used for filters, aggregations, etc. while the rest are only useful for reference when looking at a document, or in a rare case when filtering down to dig into an unusual issue.
• Swiftype indexes logs of HTTP requests + responses, which contain a large number of arbitrary HTTP headers.

My sense is that we should focus on keywords for now, but in the future we could consider support for some simple analysis/ normalization, pending feedback on the feature.

Why do we plan to index individual tokens alone?

It could be nice if users were able search an entire object field (e.g. {"headers": "en.wikipedia.org"}), since the object's keys might be unknown or non-standardized. This is admittedly quite speculative, and partially just based on our thoughts on the original issue — I’ll try to collect more feedback here.

I'm not worried about it being slow since term queries on an indexed object would translate to a term query at the lucene level.

Right, that makes sense! I’ll just plan on a sanity check here.

[jtibshirani]

I also wanted to clarify a point that was a bit fuzzy to me until I did a prototype. Under the proposed implementation, we are planning to index the entire JSON blob as a lucene field, and apply a special analyzer to create tokens that resemble keywords. This is in contrast to an approach where we create a new field mapping for each key in the object (which I think would be messy and negate some of the benefit of the feature).

Taking this approach means that in kibana and other clients, the field will be displayed as a single block of JSON. I created a quick example using the prototype implementation:

• the data consists of a filebeat for a docker container, with id, name, and image mapped as keywords, and labels as an 'indexed object'
• we apply the search filter {"term": {"docker.container.labels.io.kubernetes.pod.name": "kafka-0"}}

As these JSON blobs can be quite large, highlighting seemed useful in showing where the match actually occurred. I also wonder if we should support highlighting for consistency with keyword fields, as some clients (like kibana) do depend on highlighting for displaying these matches?

[jpountz]

I'm curious how this will work in practice as mixing up pre/post tags with a JSON structure sounds challenging? The other thing that worries me a bit is that if we want to support any highligter that is able to use indexed offsets or term vectors, then when we extract the JSON object from the source document at search time, we must make sure that it produces exactly the same string as what was passed to the object mapper so that offsets are comparable, and any new/removed spaces and line breaks or reordering of keys would break highlighting?

[jpountz]

Now that I read my comment again, the latter doesn't make sense as we will not allow enabling term vectors or indexing offsets anyway, so the highlighter will have to recompute the matched offsets anyway.

[jtibshirani]

I don't think I've dug enough into the details of highlighting to understand the concerns, but my takeaway is that it may be tricky to find a robust approach (and we should be open to punting on highlighting for v1).

As for the question around non-prefixed tokens, what do you think about this plan? Whether or not to index non-prefixed tokens can be controlled through a flag, to give users the opportunity to try it out without forcing them to double their inverted index size. From my initial experiment, indexing the raw tokens doesn't add much more work/ complexity. We can mark the feature 'experimental' at first, to allow time to collect feedback about this flag, and also about analysis, a copy_to mechanism, etc. Different people I've spoken with have had different intuitions on this point, and it's been hard to come to a good decision a priori.

[jpountz]

Whether or not to index non-prefixed tokens can be controlled through a flag, to give users the opportunity to try it out without forcing them to double their inverted index size. From my initial experiment, indexing the raw tokens doesn't add much more work/ complexity.

I agree that complexity of the implementation is fine, I'm more concerned about the API as we should strive to have as few switches as possible, especially for a v1. To me the question of this switch boils down to the problem that we are trying to solve: either we want to allow users to actually index objects, in which case indexing raw values makes sense, or we want to allow users to avoid the overhead of mappings and Lucene fields when indexing keywords and then it makes less sense?

[jtibshirani]

That is a nice way to frame it! I am thinking of it as the former (providing a true 'object' field). I think it fits better with the use cases/ data we’ve seen, which center on indexing opaque JSON objects (metric beats, user-provided blobs of data, etc.). To me, the most compelling use for this feature is in being able to work with object data that is difficult to model otherwise, and not just saving on indexing cost when working with keywords. I will try to get some more consensus/ clarity on this point, and then loop back.

[jtibshirani]

We had a discussion offline, and came to the following conclusions:

• In this first version, we’ll create a new leaf field type instead of incorporating the functionality into the object mapping. It will be important to allow for certain keys to be 'promoted' into their own dedicated fields, and we need to find a good mechanism to do so. One approach which we liked was to extend copy_to to work on entire objects, so that the same JSON blob could be added both as a 'queryable object' field, and also as normal object with explicit subfield definitions.
• Raw values will be indexed in addition to key-value pairs. This will ensure consistent behavior for queryable object fields when the user performs a query across all mapped fields, or performs a wildcard search of the form header.*.