Separate metadata from the actual response in async search index

[jimczi]

Getting the status of an async search is fast when the query is running but can be very costly if the response is already stored.
When the task is gone/finished, we interrogate the async search index to retrieve the full response if it is still available. The metadata and statistics are stored inside a binary field that also contains the actual response so the cost to retrieve the status depends greatly on the size of the search response.
We should separate the actual response from the metadata in order to allow to query these informations separately.
That would make the cost of status calls more constant and cheaper than retrieving the full response each time.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[mayya-sharipova]

Currently _async_search/status for stored searches returns below:

for successfully completed searches:

{
  "id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
  "is_running" : false,
  "is_partial" : false,
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986,
  "_shards" : {
      "total" : 562,
      "successful" : 562,
      "skipped" : 0,
      "failed" : 0
  },
 "completion_status" : 200 
}

for unsuccessfully completed searches:

{
  "id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
  "is_running" : false,
  "is_partial" : true,
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986,
  "_shards" : {
      "total" : 562,
      "successful" : 450,
      "skipped" : 0,
      "failed" : 112
  },
 "completion_status" : 503 
}

Currently all stored responses are supposed to be completed. But in future, we may be storing partial results as well.

[mayya-sharipova]

Problem

To retrieve status from a stored response, we do:

• GET request to .async-search index with
• get result field from _source (which is object encoded with base64 encoding)
• decode result to AsyncSearchResponse
• create AsyncStatusResponse from AsyncSearchResponse and its SearchResponse.

If stored result is huge, retrieving it and decoding takes a lot of computer resources. Kibana may access status API thousands time per second.

The current mapping for .async-search is:

"properties" : {
  "expiration_time" : {
    "type" : "long"
  },
  "headers" : {
    "type" : "object",
    "enabled" : false
  },
  "response_headers" : {
    "type" : "object",
    "enabled" : false
  },
  "result" : {
    "type" : "object",
    "enabled" : false
  }
}

There are several ways we can separate metadata from the actual response:

Proposal 1: add status field as a base64 encoded stored only binary field:

Mapping:

"status": {
  "type" : "binary",
  "doc_values": false,
  "store": true
}

GET request will retrieve status as a separate source field, and decode it into AsyncStatusResponse object.
We only retrieve and decode "status" field without retrieving and decoding an expensive _source field.

GET .async-search/_doc/<ID>?stored_fields=status

Upgrade scenario and BWC:

• We either need to:
  • update mapping of an existing _async_search index to add status fields (is this a part of upgrade assistant ?)
  • or create a new index _async_search2 and use it.
• Older docs indexed in _async_search don't contain the status stored field, and a GET request with stored_fields will have stored fields empty (or GET request will return unfound if we are trying to find older docs in a new index). In this case, we need to reissue another GET request with _source (or older index).

Proposal 2: add status fields as separate stored only fields:

mapping

"status": {
  "type" : "object",
  "properties": {
      "is_running": {
        "type": "boolean",
        "store": true,
        "index": false,
        "doc_values" : false
      },
      "is_partial": {
        "type": "boolean",
        "store": true,
        "index": false,
        "doc_values" : false
      },
      "start_time": {
        "type": "long",
        "store": true,
        "index": false,
        "doc_values" : false
      },
     "expiration_time": {
        "type": "long",
        "store": true,
        "index": false,
        "doc_values" : false
      },
      "shards.total": {
        "type": "integer",
        "store": true,
        "index": false,
        "doc_values" : false
      },
      "shards.successful": {
        "type": "integer",
        "store": true,
        "index": false,
        "doc_values" : false
      },
      "shards.skipped": {
        "type": "integer",
        "store": true,
        "index": false,
        "doc_values" : false
      },
      "shards.failed": {
        "type": "integer",
        "store": true,
        "index": false,
        "doc_values" : false
      },
      "completion_status": {
        "type": "short",
        "store": true,
        "index": false,
        "doc_values" : false
      }
  }    
}

And this GET request will retrieve only these stored field without retrieving and decoding an expensive _source field:

GET .async-search/_doc/<ID>?stored_fields=status.is_running,status.is_partial,status.start_time,status.expiration_time,status.shards.total,status.shards.successful,status.shards.skipped,status.shards.failed,status.completion_status

• Upgrade scenario and BWC same as proposal 1
• Avoid encoding and decoding step that is needed for proposal 1, but needs to manage more fields.

Proposal 3: create a separate index for status updates: .async-search-status

Pros:

• Upgrade and BWC is easier than proposal 1 and 2
• Retrieval of status will be faster. For proposals 1 and 2, as Lucene keeps all stored fields together, Lucene stills needs to read stored _status and _source from disk.

Cons:

• double work on indexing, updating and deleting docs in .async-search and .async-search-status. It is possible that two indices could be out of synchronization.

[mayya-sharipova]

We had a team discussion and decided to go with the Proposal 1. So the plan is:

• Introduce status as a single binary field in existing .async-search indices
• status can be either:
  • a stored only field
  • a doc values only field
• Do performance benchmarking with around 1000 docs to compare stored VS doc values fields performances
  • some docs should have huge responses
  • 1000 requests per second, per 10 seconds

[dnhatn]

Older docs indexed in _async_search don't contain the status stored field, and a GET request with stored_fields will have stored fields empty (or GET request will return unfound if we are trying to find older docs in a new index). In this case, we need to reissue another GET request with _source (or older index).

@mayya-sharipova I think we can add a version to AsyncExecutionId(and encoded id) then make a decision based on the version so we can avoid issue two requests in a mixed cluster.

[mayya-sharipova]

@dnhatn Thanks, that's a great proposal. I will experiment with this idea

[mayya-sharipova]

We had another discussion on this topic today, and decided to go with GET response, and for a search response we need to worry about refresh interval.

We need to measure performance of retrieving doc values fields VS stored field fields for status, and if retrieving doc values fields turns out to be much faster than stored fields, we can think of adding support for retrieving doc value fields to GET api.

[mayya-sharipova]

I've benchmarked the time needed for retrieving stored fields through GET request VS retrieving doc values fields through SEARCH request depending on the size of result field. Here are the results:

GET request with stored fields VS SEARCH request with doc values fields

number of docs in the index: 1000

Result size	GET time with stored fields, ms	SEARCH time with doc values, ms	SEARCH time with doc values for first request, ms
100Kb	1.1	1.9	6.2
1Mb	2.4	3.0	6.0
10Mb	13.4	14.1	19.3
990 docs of 10Mb + 10 docs of 100 Mb	97.6 – for 100Mb doc 15 – for 10Mb doc	99.1 – for 100Mb doc 18.4 – for 10 Mb doc

It looks like retrieving just doc values fields through a search request doesn't bring significant benefits. It could be explained that while doing _search request we still need to access stored field of _id, and read and decompress stored fields.

GET request with _source VS GET request with stored fields

number of docs in the index: 1000
990 docs of 10Mb , 10 docs of 100 Mb

Result size	full _source ms	only stored status field ms
10Mb	30.8	13.4
100Mb	304.7	97.6

So the conclusion will be to proceed with using GET request where status field is a separate stored field, as this will allow us to have a request at least 3 times faster.

cc @jimczi

[jimczi]

Are you sure that you're benchmarking the doc values case with "stored_fields": ["_none_"] ? I guess that could explain why responses with doc values are not fast. 100ms for a status query is quite a lot, especially when testing in isolation. Is it only on the first retrieval or it's consistently slow ?