PUT Mapping removes metadata from index

[pacer11]

Elasticsearch version (bin/elasticsearch --version): 5.3.0

Plugins installed: [repository-s3]

JVM version (java -version): 1.8.0_92

OS version (uname -a if on a Unix-like system): Linux 2.6

Description of the problem including expected versus actual behavior:

Steps to reproduce:
After we make a PUT or POST mapping to add additional fields, it removes our _meta Field from the mapping.
Is this normal? Can we prevent this?

Before:

{
"foo": {
"mappings": {
"bar": {
"_meta": {
"foo": "bar"
},
"properties": {
...

After:

{
"foo": {
"mappings": {
"bar": {
"properties": {
...

This was question was asked in the forum with no answer provided.
https://discuss.elastic.co/t/put-mapping-removes-metadata-from-index/91794

[colings86]

@pacer11 thanks or raising this, it looks to me like this is a bug. I managed to reproduce it on the master branch with the following script:

PUT test
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "doc": {
      "_meta": {
        "foo": "bar"
      },
      "properties": {
        "string_field": {
          "type": "text"
        }
      }
    }
  }
}

GET test/_mapping

#Dynamically mapped field maintain _meta object
PUT test/doc/1
{
  "string_field": "foo"
}

GET test/_mapping

# Adding a field with the PUT mapping API destroys the _meta object
PUT test/doc/_mapping
{
  "doc": {
    "properties": {
      "int_field": {
        "type": "integer"
      }
    }
  }
}

GET test/_mapping

Unfortunately I think the only workaround at the moment would be to make sure you include the _meta object on every call to the PUT mappings API.

Out of curiousity, what are you using the _meta field in the mappings for? i.e. what information do you store in it and how do you use that information?

[pacer11]

@colings86 Thanks for your confirmation. We are kinda already using the workaround you suggested, however it's one of those annoying things to remember especially when it's not necessarily intuitive.

We use the _meta field for a couple of reasons.

1. The date fields that we define in our mapping are all consistently stored as one format but when we retrieve from ES, we represent in various formats so we track the different formats in the _meta field so that as we are transforming the JSON we get from ES we can apply the applicable transformation before sending it further upstream in our application. I am sure there are other ways of achieving this but this seems to work well for us.
2. We have a few nested field mappings defined and we keep track of the nested paths within the _meta field. We have a query generator routine that dynamically generates the ES Queries based upon the fields passed in and we pull nested paths information from the _meta field so that we can determine if a nested path should be incorporated into the query or not.

Hope this helps!