Federated multi-cluster search + scroll

[ikonst]

When using scroll on a multi-cluster searching node, after successfully getting the first "page", the following error is returned when trying to use the scroll ID (using POST /_search/scroll):

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "node [SOME-NODE-ID] is not available"
      },
      ...

The root_cause element contains dozens of those "root causes", each concerning a different node. (The 'scroll_id' is pretty large, and when I base64-decode it, it seems to point to all those nodes.)

Version:

    "number": "5.5.2",
    "build_hash": "b2f0c09",
    "build_date": "2017-08-14T12:33:14.154Z",
    "build_snapshot": false,
    "lucene_version": "6.6.0"

[elasticmachine]

Pinging @elastic/es-search-aggs

[colings86]

@javanna would you mind taking a look into this?

[javanna]

hi @ikonst are you also sending ordinary search requests to multiple clusters without any issues? And scroll for you always gives problems? The error seems to indicate that some of the nodes are not available, have you checked that this is not the case and that your cluster is healthy?

[ikonst]

@javanna the cluster is otherwise healthy - it's used extensively for ELK in my organization (in a federated fashion).

When I reproduce this issue, the pattern is always:

1. First search request succeeds, returning N results and scroll_id.
2. Second attempt (using scroll_id) fails entirely, with a long array of "missing" nodes. Those nodes' IDs always belong to the clusters I federate to, and within those clusters those nodes are perfectly healthy.

[javanna]

hi @ikonst are all your nodes on the same version? It seems to me that the node that you are using as coordinating node for your scroll search is on 5.6 or later, while the remote nodes are on a previous version, probably 5.5 which is the version you mentioned in the description of this issue. Scroll searches are supported when using cross-cluster search only from 5.6 on, see #25094. I think that this is the problem here, can you please confirm?

[ikonst]

Confirming, our remote nodes are on 5.5. This will be taken care of as part of our update to 6.x. Thanks for looking into this!