ES 6.0 regression: "query must be rewritten first" exception when using terms lookup filter in nested aggregation

[mrszg]

Elasticsearch version
6.0.0, 6.0.1

Plugins installed: []

JVM version :
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.17.10.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

OS version :
Ubuntu 17.10 with kernel 4.13.0-19-generic

Description of the problem including expected versus actual behavior:

Query with filter aggregation based on terms filter lookup nested below other aggregation terminates with error "query must be rewritten first". The same query executed on version 5.6.5 finishes with success.

This error is very similar to already reported and closed: #21301

The only difference is that now query fails when nested aggregation is used.

Steps to reproduce:
Recreation steps are based on those provided in issue #21301:

1. create index test_posts

PUT test_posts
{
  "mappings": {
    "post": {
      "properties": {
        "mentionIDs": {
          "type": "keyword"
        }
      }
    }
  }
}

2. create index test_users

PUT test_users
{
  "mappings": {
    "user": {
      "properties": {
        "notifications": {
          "type": "keyword"
        }
      }
    }
  }
}

3. index some user

PUT test_users/user/USR|CLIENTID|1234
{
  "notifications": ["abc"]
}

4. insert some post

PUT test_posts/post/POST|4321
{
  "mentionIDs": ["abc"]
}

5. execute search (this search finishes with success - it is similar to that from Terms lookup filter in filter aggregation complains that the query is not rewritten #21301)

GET test_posts/_search
{
  "aggs": {
    "itemsNotify": {
      "filter": {
        "terms": {
          "mentionIDs": {
            "index": "test_users",
            "type": "user",
            "id": "USR|CLIENTID|1234",
            "path": "notifications",
            "routing": "CLIENTID"
          }
        }
      }
    }
  }
}

6. This search fails with error on ES 6.0.0 and ES 6.0.1:

GET /test_posts/_search
{
  "aggs": {
    "facets": {
      "global": {},
      "aggs": {
        "filteredFacets": {
          "filter": {
            "terms": {
              "mentionIDs": {
                "index": "test_users",
                "type": "user",
                "id": "USR|CLIENTID|1234",
                "path": "notifications",
                "routing": "CLIENTID"
              }
            }
          }
        }
      }
    }
  }
}

Query result is:

{
  "error": {
    "root_cause": [
      {
        "type": "unsupported_operation_exception",
        "reason": "query must be rewritten first"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "test_posts",
        "node": "5ipdZJUdSqKf5gnOkwlkDg",
        "reason": {
          "type": "unsupported_operation_exception",
          "reason": "query must be rewritten first"
        }
      }
    ]
  },
  "status": 500
}

Above query finishes with success on ES 5.6.5

Fragment of logs from server:
es-query-must-be-rewritten.log

[jimczi]

Thanks @mrszg it's a bug in the rewrite phase of aggregations, only the root aggregation gets rewritten and sub aggregations are skipped. I'll take a look.

[marcmos]

It seems that version 6.1.0 is affected too.

[lbrzekowski]

@jimczi Do you know when this bug will be fixed?

[jimczi]

Unfortunately this bug requires a larger fix than I initially thought. The rewriting of aggregations has a bug when multiple rewrite of the query is needed, the terms query is rewritten in multiple rounds (create the fetch query to get the document, retrieve the document and creates the final query) but currently only one rewrite is called. We know how to fix this but it will require some time, I cannot give any date or version for the fix though @lbrzekowski , I'll update this issue when I have a pr ready for it.

[mrszg]

@jimczi Can you provide any update on this issue?