Unknown field error when function score filter filters out all records

[lukebergen]

I've created an example that demonstrates this issue here https://gist.github.com/lukebergen/1ddd897a1635f46c732b.

I've tried to pare this down to the bare essentials of the issue that I'm seeing but it's a little convoluted of a situation.

Basically, if I have a function score that I only want to apply if my field is present, so I add an exists filter to the function. This works great except in the use case where all records are filtered out by this existence filter. In which case an error comes back "Unknown field []".

I'm on ElasticSearch 1.3.4.

[clintongormley]

Hi @lukebergen

The problem is that you are relying on dynamic mapping. Your first two indexing requests don't create the field age (as we have no way of determining what datatype age will be from the value null).

If, instead, you map the age field before indexing, then your query works correctly:

DELETE /test 

PUT /test/
{
  "mappings": {
    "test": {
      "properties": {
        "age": {
          "type": "integer"
        }
      }
    }
  }
}

PUT /test/test/1
{
  "name": "Finn",
  "age": null
}

PUT /test/test/2
{
  "name": "Jake",
  "age": null
}


POST /test/_search
{
  "query": {
    "function_score": {
      "query": {
        "match_all": {}
      },
      "functions": [
        {
          "filter": {
            "exists": {
              "field": "age"
            }
          },
          "linear": {
            "age": {
              "origin": 10,
              "scale": 10,
              "decay": 0.9
            }
          }
        }
      ],
      "score_mode": "multiply"
    }
  }
}

[sandstrom]

@clintongormley Wouldn't it make more sense to not even attempt to run the function score if the filter removes the document? We've finally tracked down an issue caused by this, and it's very un-intuitive.

Or am I missing something obvious?

[sandstrom]

ping @clintongormley We're still having this issue under 6.2.

Basically, we're running a top-level bool query with two should blocks, hitting two indices. Each should block is tailored to one index. This is restricted using filters. However, it will fail due to missing fields despite these filter restrictions.

• Each should block target one index, via a filter on _index. This doesn't work, it will still fail with "unknown field [mentioned_at]".
• I've also tried setting a filter inside the functions block, with an exists restriction on the field. Doesn't work either.

Is this issue by design, or a bug?

If it's by design, are there other ways we could avoid it, and if not, do you have any planned enhancements around this?

⚠ Expand 'details' below, for the full query ⚠

{
  "query": {
    "bool": {
      "should": [{
        "bool": {
          "should": [{
            "function_score": {
              "query": {
                "multi_match": {
                  "query": "jane",
                  "type": "phrase_prefix",
                  "fields": ["name", "firm"]
                }
              },
              "functions": [{
                "filter": {
                  "exists": {
                    "field": "mentioned_at"
                  }
                },
                "gauss": {
                  "mentioned_at": {
                    "origin": "2018-05-07T13:49:33Z",
                    "scale": "180d",
                    "offset": "5d",
                    "decay": 0.5
                  }
                }
              }, {
                "filter": {
                  "term": {
                    "kinsman": true
                  }
                },
                "weight": 1.1
              }],
              "score_mode": "multiply"
            }
          }],
          "filter": {
            "bool": {
              "must": [{
                "term": {
                  "_index": "attendees"
                }
              }, {
                "term": {
                  "owner_id": "5acca8d1962d745523000000"
                }
              }]
            }
          }
        }
      }, {
        "bool": {
          "should": [{
            "multi_match": {
              "query": "jane",
              "type": "phrase_prefix",
              "fields": ["name", "company"],
              "boost": 1.1
            }
          }],
          "filter": {
            "bool": {
              "must": [{
                "term": {
                  "_index": "users"
                }
              }, {
                "term": {
                  "company_id": "5acca8d1962d745523000003"
                }
              }, {
                "bool": {
                  "should": [{
                    "range": {
                      "archived_at": {
                        "gte": "2018-02-07T13:49:33Z"
                      }
                    }
                  }, {
                    "bool": {
                      "must_not": {
                        "exists": {
                          "field": "archived_at"
                        }
                      }
                    }
                  }]
                }
              }]
            }
          }
        }
      }]
    }
  }
}

[sandstrom]

Friendly ping @original-brownbear @eskibars @jpountz 😄

Do you know if the "unknown field" issue described above is by design, or a bug?

(when looking through other issues it seems like the three of you all work on the mapping and query parts of Elastic Search, but feel free to loop in other people that are better positioned to answer this if needed)

[jpountz]

@sandstrom Can you open a new feature issue asking that we allow functions to work on unmapped fields?

[sandstrom]

@jpountz Certainly! #33558