Allow functions to work on unmapped fields

[sandstrom]

Problem

Function queries against unmapped fields doesn't work. Adding support for it would be very useful.

Example

We're running a top-level bool query with two should blocks, hitting two indices. Each should block is tailored to one index. This is restricted using filters. However, it will fail due to missing fields despite these filter restrictions.

• Each should-block target one index, via a filter on _index. This doesn't work, it will still fail with "unknown field [mentioned_at]".
• I've also tried setting a filter inside the functions block, with an exists restriction on the field. Doesn't work either.

Env

Version: 6.4.0

More details

This was originally discussed in #8069, and @jpountz asked me to open a new issue about this. Please let me know if I should elaborate on anything.

Full example query

⚠ Expand 'details' below, for the full query ⚠

{
  "query": {
    "bool": {
      "should": [{
        "bool": {
          "should": [{
            "function_score": {
              "query": {
                "multi_match": {
                  "query": "jane",
                  "type": "phrase_prefix",
                  "fields": ["name", "firm"]
                }
              },
              "functions": [{
                "filter": {
                  "exists": {
                    "field": "mentioned_at"
                  }
                },
                "gauss": {
                  "mentioned_at": {
                    "origin": "2018-05-07T13:49:33Z",
                    "scale": "180d",
                    "offset": "5d",
                    "decay": 0.5
                  }
                }
              }, {
                "filter": {
                  "term": {
                    "kinsman": true
                  }
                },
                "weight": 1.1
              }],
              "score_mode": "multiply"
            }
          }],
          "filter": {
            "bool": {
              "must": [{
                "term": {
                  "_index": "attendees"
                }
              }, {
                "term": {
                  "owner_id": "abc"
                }
              }]
            }
          }
        }
      }, {
        "bool": {
          "should": [{
            "multi_match": {
              "query": "jane",
              "type": "phrase_prefix",
              "fields": ["name", "company"],
              "boost": 1.1
            }
          }],
          "filter": {
            "bool": {
              "must": [{
                "term": {
                  "_index": "users"
                }
              }, {
                "term": {
                  "company_id": "abc"
                }
              }, {
                "bool": {
                  "should": [{
                    "range": {
                      "archived_at": {
                        "gte": "2018-02-07T13:49:33Z"
                      }
                    }
                  }, {
                    "bool": {
                      "must_not": {
                        "exists": {
                          "field": "archived_at"
                        }
                      }
                    }
                  }]
                }
              }]
            }
          }
        }
      }]
    }
  }
}

[elasticmachine]

Pinging @elastic/es-search-aggs

[heberuriegas]

Any workaround to set a default value in case that field is not mapped?

[mayya-sharipova]

I am closing this issue as we are deprecating function score query in favour of the new script score query.
In script_score query, it is possible to set up a filter to select documents for which script should be applied, and with the proper selection unmapped field exception will NOT be produced:

GET _search
{
  "query": {
    "script_score": {
      "query": {
        "bool": {
          "filter": {
            "term": {
              "_index": "index-date"
            }
          }
        }
      },
      "script": {
        "source": "doc['date'].value.getYear()"
      }
    }
  }
}

[sandstrom]

@mayya-sharipova Sounds good & thanks for writing an update here! 😄

One question:

In the issue above I had a filter in the function score, which was supposed to make sure that the function didn't run on anything that didn't match the filter. But that didn't work.

Is this new functionality different in that regard?

[mayya-sharipova]

@sandstrom Yes, indeed the script score is different. Objects for decay functions in function score query are evaluated and created in advance before any query is executed and if a field is missing mapping, an error is produced.
For script score, evaluation is done later. You can even test it today with script_score of the function_score query, you should't get unmapped field error if you put a proper filter. But script_score of the funciton_score query doesn't have built-in decay functions; the new script score query will have them.

[sandstrom]

Alright, sounds good! Thanks a lot @mayya-sharipova, much appreciated! 🏅