Unable to get progress in _async_search forever

[easyice]

Elasticsearch version (bin/elasticsearch --version):7.7.0

Plugins installed: []

JVM version (java -version):1.8

OS version (uname -a if on a Unix-like system):centos7.2

Description of the problem including expected versus actual behavior:

The environment：
1.two nodes in the cluster:node1,node2
2.only single shard,sales[0] on node1

Steps to reproduce:

1. submit a _async_search on node1,the request need took A long time，10 second for instance.
2. when the async search is running,move shard sales[0] to node2
3. after the async search is done, get the result: GET /_async_search/{id}

the get progress request will return error forever like this:

"is_partial" : true,
"is_running" : true,
"failures" : [
        {
          "shard" : 0,
          "index" : "sales",
          "node" : "xh5OKPitTMiArexFDzhT-w",
          "reason" : {
            "type" : "index_not_found_exception",
            "reason" : "no such index [sales]",
            "index_uuid" : "Fx1mX5_pS6WwpWerjFPNPw",
            "index" : "sales"
          }
        }
      ]

Provide logs (if relevant):

[2020-06-18T16:52:49,138][ERROR][o.e.t.TransportService ] [node-idea] failed to handle exception for action [indices:data/read/search[phase/query]], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler@2c182b7e/org.elasticsearch.action.search.SearchExecutionStatsCollector@783b1305]
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at org.elasticsearch.xpack.search.MutableSearchResponse.updateWithFailure(MutableSearchResponse.java:135) ~[?:?]
at org.elasticsearch.xpack.search.AsyncSearchTask$Listener.onFailure(AsyncSearchTask.java:398) ~[?:?]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:571) ~[classes/:?]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) ~[classes/:?]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) ~[classes/:?]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580) ~[classes/:?]
at
Code Bug:

in: org.elasticsearch.action.search.SearchPhaseExecutionException#guessRootCauses
the shardFailures array size is 0,so ,guessRootCauses funcation will return ElasticsearchException array with size 0

but,org.elasticsearch.xpack.search.MutableSearchResponse#updateWithFailure funcation will use the index 0 of the array:

this.failure = ElasticsearchException.guessRootCauses(exc)[0];

an exception happed on updateWithFailure() ，then,the executeCompletionListeners() funcation will not be call,then will not unregister from the taskManager

 public void onFailure(Exception exc) {
            if (searchResponse.get() == null) {
                // if the failure occurred before calling onListShards
                searchResponse.compareAndSet(null,
                    new MutableSearchResponse(-1, -1, null,
                        aggReduceContextSupplier, threadPool.getThreadContext()));
            }
            searchResponse.get().updateWithFailure(exc);
            executeInitListeners();
            executeCompletionListeners();
        }

when get asyc search progress，it still found the task in the taskmanager,and return the error response.

Can this just add a judgment when guessRootCauses return size 0?

[jimczi]

Thanks for reporting @easyice . There is an issue in how we handle exceptions in async search that will be solved by #57925. However I opened #58319 to fix the AOOBE explicitly since it's a bug in our code and not a normal failure.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[easyice]

Great !