[ML] Data frame transform _preview endpoint missing mapping fields

[walterra]

The decuded mappings introduced in #43742 seems to miss exposing mappings based on scripted fields. For example, using the following in Kibana Dev Console:

POST _data_frame/transforms/_preview
{
  "source": {
    "index": [
      "filebeat-nginx-elasticco-anon-2017"
    ]
  },
  "pivot": {
    "group_by": {
      "source.address": {
        "terms": {
          "field": "source.address"
        }
      }
    },
    "aggregations": {
      "@timestamp_max": {
        "max": {
          "field": "@timestamp"
        }
      },
      "@timestamp_min": {
        "min": {
          "field": "@timestamp"
        }
      },
      "duration": {
        "bucket_script": {
          "buckets_path": {
            "min": "@timestamp_min",
            "max": "@timestamp_max"
          },
          "script": "(params.max - params.min)/1000"
        }
      }
    }
  }
}

Will return:

{
  "preview" : [
    {
      "duration" : 3236636.783,
      "@timestamp_min" : "2017-02-01T06:02:10.303Z",
      "@timestamp_max" : "2017-03-10T17:06:07.086Z",
      "source" : {
        "address" : ""
      }
    },
    ...
   {
      "duration" : 0.0,
      "@timestamp_min" : "2017-02-09T08:07:07.963Z",
      "@timestamp_max" : "2017-02-09T08:07:07.963Z",
      "source" : {
        "address" : "0.104.130.123"
      }
    }
  ],
  "mappings" : {
    "properties" : {
      "@timestamp_min" : {
        "type" : "date"
      },
      "@timestamp_max" : {
        "type" : "date"
      },
      "source.address" : {
        "type" : "keyword"
      }
    }
  }
}

You can see that duration is part of the preview documents, but it's missing from the mappings part.

This affects the Kibana data frame transforms wizard in 7.3+: For the transforms preview table, the columns visible are based on the mappings, so the duration column will not show up at all.

[elasticmachine]

Pinging @elastic/ml-core

[benwtrent]

@walterra the mappings returned are the specific field mappings that we can actually deduce. When a field is not included, that means we defer to the Elasticsearch dynamic mapping logic.

For certain aggregations, there is no way for us to accurate deduce their field type. A specific example of this is scripted_metric.

I do not think we will ever include scripted_metric fields in this mapping.

Why not make the columns that are visible be the fields in the preview objects? We return a set number of them, so iterating through them and picking out the unique field names seems more reliable.

[walterra]

@benwtrent Thanks for clarifying! We can definitely fix it on the client side I was just not aware that there's a chance that fields would not be part of the mapping. I created a corresponding issue in the Kibana repo and will close this one.