Move field mappings to the index level internally

[clintongormley]

Internally, field mappings should be stored at the index level, rather than at the type level. Types are essentially a way of grouping fields. While the mapping APIs will not change, an exception will be thrown if an attempt is made to map a field in one type in a way that conflicts with the mapping of a field with same name in another type.

The _parent field is the one exception - this field will remain at the type level.

Relates to #8870

[kiwigaffa]

We have a lot of indices, each of which has a large set of mappings. While no two sets of mappings are identical, for most of our indices there is considerable overlap. Being able to share mappings between indices would help us reduce the total cluster state size.

More concretely, imagine that we have 10 indices (a-j) all being written to, and we cycle each of those indices every hour (i.e. create a' to replace a, b' to replace b, and so on). If the 10 new indices (a'-j') could simply use the mappings from their predecessors, then we'd avoid the mappings storm as data is added to the empty indices.

This is possible today by simply setting the mappings when the new index is created, but every time that happens, we're increasing our total cluster state by the size of those mappings. We also have mappings "drift", where some percentage of all mappings are used in one index, but not in the next. Simply copying the mappings means we accumulate all of those "dead" mappings in each new index. If, instead, we could re-use a single set of mappings across multiple indices, then we minimize total cluster state size and avoid the mapping update storm. "Drift" is still an issue, but the cost is effectively zero, since those mappings are still required for the indices they were used in.

I guess what I'm asking for is to move the mappings to one level higher than the index, and allow that higher level mapping to be shared by multiple indices.

[clintongormley]

Hi @kiwigaffa

Something like this might be possible later on. I wouldn't do it as a user-facing option, but make it an internal optimisation instead. For instance, if field mappings are immutable, then when storing the cluster state we could have one in-memory representation for each unique field and just have a pointer for every use of that field.

For now, we just want to get the user-facing side sorted, and we can worry about these optimisations later on.

[kiwigaffa]

Thanks @clintongormley

Wasn't expecting it in the first cut, but did want to see whether it made sense, and get the gears turning for a later release :-)