Expressions scripts in filter contexts throw exception

[dakrone]

In the master branch, if I do a query with an expression script like:

{
  "query": {
    "function_score": {
      "query": {
        "constant_score": {
          "filter": {
            "bool": {
              "must": [
                {
                  "script": {
                    "script": {
                      "lang": "expression",
                      "inline": "birth_date >= doc[\"birth_date\"].value",
                      "params": {
                        "birth_date": 14
                      }}}}]}}}}}}}

I get the following error:

Caused by: java.lang.IllegalArgumentException: painless does not know how to handle context [filter]                                                         
        at org.elasticsearch.script.expression.ExpressionScriptEngine.compile(ExpressionScriptEngine.java:111) ~[?:?]                                                                                                                                                                     at org.elasticsearch.script.ScriptService.compile(ScriptService.java:296) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]                                                                                                                                        at org.elasticsearch.index.query.ScriptQueryBuilder.doToQuery(ScriptQueryBuilder.java:130) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]                                             
        at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]               
        at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:405) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:379) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.AbstractQueryBuilder.toFilter(AbstractQueryBuilder.java:119) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.ConstantScoreQueryBuilder.doToQuery(ConstantScoreQueryBuilder.java:136) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.functionscore.FunctionScoreQueryBuilder.doToQuery(FunctionScoreQueryBuilder.java:307) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$2(QueryShardContext.java:304) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:316) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:303) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:669) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]

The error also says "painless" instead of "expressions" from

elasticsearch/modules/lang-expression/src/main/java/org/elasticsearch/script/expression/ExpressionScriptEngine.java

Line 111 in c075323

throw new IllegalArgumentException("painless does not know how to handle context [" + context.name + "]");

[stacey-gammon]

@rjernst - It appears this bug might be causing Kibana build failures on master. Any ETA on a fix?

[epixa]

I'm going to mark this as a blocker for 6.1 since we disabled a sizable chunk of integration tests in Kibana in order to get builds passing. I assume this is a regression that needs to be fixed for 6.1 anyway, but I'm really just blocking on resolution one way or another.

[rjernst]

@epixa Looking back at this again, I'm actually not sure filters make sense for expressions. Expressions only know how to read numeric values, and return numeric values. There was previously hacky "treat 0 as false and anything else as true" code in filter scripts, but that was removed with my refactoring to create a filter script context.

Why can't Kibana use painless for filters? The same example script @dakrone gives in the issue description would work fine in painless.

[epixa]

@rjernst Seems like a reasonable question to me, especially since we want people to use painless instead of lucene expressions for this stuff since it's designed more for these specific use cases rather than relying on hacky type coercion.

That said, unless we can guarantee complete compatibility between the behaviors of expression-based filters and painless-based filters, this is going to be a breaking change for a lot of Kibana users, so I think we should preserve the existing behavior until 7.0.

People can filter on Kibana scripted fields, which can use either expressions or painless scripts. At the very least, we'll need to make changes to Kibana to make it so only painless scripted fields can be filtered on, we'll need to start throwing deprecation notices for the existing expression filters, and we probably want to add a migration mechanism to the upgrade assistant for people to convert their existing scripted fields over.

It's worth mentioning though, that we've never had any person (to my knowledge) that encountered unexpected behaviors with expression-based scripted fields in kibana. Was the 0->false coercion problematic from a performance or maintenance standpoint? Given the impact of the change on existing users and the amount of development that'll go into providing a bridge for those users going into 7.0, is it more practical for us to simply preserve the 0->false coercion as the documented behavior of how expressions work in a filter context?

[rjernst]

Part of the reason for the context work we have been doing in scripting is performance. When you do a coercion a million times (assuming one million docs being evaluated), the total time can be non-negligible. This is part of the reason expressions are currently faster in simple cases than painless. Once we have painless performance on par with expressions, I don't think there is any reason to keep expressions around. They were an early experiment in Lucene into doing scripted scoring, and will likely stay there for a long time. But having 2 languages in elasticsearch, especially one with limited functionality, is both confusing for users ("which one should I use?") and a maintenance burden on developers. Expanding on the latter, expressions require manual work for every new context we add. It is not simply a matter of "preserving coercion". There are a few classes necessary to be created and handled for every context expressions supports.

So I think beginning the journey to remove uses of expressions is well worth the time investment. I can add in a hack for 6.1, but I would like to remove it for 7.0 (ie remove filter script support for expressions then).

[jpountz]

+1 to add a workaround for now and removing filter support for expressions in 7.0 (or even remove expressions entirely?)