Field aliases

[clintongormley]

It is hard to rename a field when using time-based indices - search and especially aggregations will only work on either the new or the old version, but there is a transition period where not all data will be seen.

We can introduce a new field type called alias which simply points to another field, eg:

PUT my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "host": {
          "properties": {
            "source_ip": {
              "type": "ip"
            }
          }
        },
        "sourceIP": {
          "type": "alias",
          "path": "host.source_ip"
        }
      }
    }
  }
}

This field type would work as follows:

• Attempts to index into the alias field would result in an exception - it is read only
• Queries, aggs, suggestions, scripts (using doc[]), highlighting, fielddata_fields, docvalue_fields, stored_fields would just get the data (and mapping) from the specified path
• Source filtering would not work with the aliased field

This also works for users who want to expose a nicer name for fields in Kibana

[skearns64]

++.

We may also want to consider supporting these field aliases in the field_stats and the (I think) upcoming field_capability APIs.

[colings86]

Discussed in FixItFriday and we said that we can see this as a useful feature for transitioning to a new field name but if the implementation is not as clean as it appears currently we should discuss this again since mappings are already complex and we should not do anything to increase that complexity too much

[rjernst]

If we are going to add (however much) complexity here, can we trade it for removal of some other complexity/leniency? One of the things that has bugged me for a long time is unmapped fields stuff. Could we be more strict about fields existing across indexes, and if there is not a concrete field, they can create an alias (maybe even have an "empty" type of alias which means "match anything against this field").

[clintongormley]

I really don't want to make this change contingent on anything else. It is a good solution in and of itself. Let's keep this issue on topic.

[mrec]

Another (non-transitional) use case for this: we have a lot of examples where the master datasets for some indices/types define separate FooVariant1 and FooVariant2 fields while others don't make a distinction and so only have data for one of these. To support consistent searching across indices we currently use copy_to rules, but the data duplication is obviously wasteful; zero-cost aliases like this would be much nicer.

[nik9000]

If we are going to add (however much) complexity here, can we trade it for removal of some other complexity/leniency? One of the things that has bugged me for a long time is unmapped fields stuff. Could we be more strict about fields existing across indexes, and if there is not a concrete field, they can create an alias (maybe even have an "empty" type of alias which means "match anything against this field").

@rjernst, are you saying that if we had field aliases we could clean up the code around unmapped fields by mapping them to a field that explicitly doesn't index them? Or are you thinking of something at query time?

@mrec I think this feature would work for what you want. As envisioned the fields would have to be the same types for this to work.

@clintongormley I wonder if duplicating the old structure with ingest is a technically better solution. It'd work in all cases without requiring new code, but it isn't a thing you could do after the fact and it isn't free from a storage standpoint. copy_to is a similar thing, but with less space used and working in fewer contexts. Both are more flexible than an alias in that you can define different mappings for the field so you can handle the cases where you changed mappings. I wonder if we're better off documenting a few "recipes" for migrating fields over time.

[rjernst]

are you saying that if we had field aliases we could clean up the code around unmapped fields by mapping them to a field that explicitly doesn't index them? Or are you thinking of something at query time?

@nik9000 Possibly. I just realized my original thought could be done now, regardless of aliases. That is, instead of having the current "unmapped fields" logic on the coordinating node, we could require adding a dummy empty field on older indexes for that name.

[clintongormley]

@clintongormley I wonder if duplicating the old structure with ingest is a technically better solution. It'd work in all cases without requiring new code, but it isn't a thing you could do after the fact and it isn't free from a storage standpoint. copy_to is a similar thing, but with less space used and working in fewer contexts. Both are more flexible than an alias in that you can define different mappings for the field so you can handle the cases where you changed mappings. I wonder if we're better off documenting a few "recipes" for migrating fields over time.

@nik9000 the point is that with copy_to and ingest you need to reindex. I'm trying to solve the case where you are transitioning from field Foo in old indices to field foo in new indices, and you want to be able to run aggs or searches across old and new indices. A field alias can be added after the fact (when you realise you have a problem) at zero cost.

[ppf2]

+1 For example, this would have helped Kibana users a lot if we had field aliases to handle the raw -> keyword field name changes between 2.x and 5.x. For there will be older indices with .raw references while newer indices will have .keyword.