CCS: Handling missing index in downstream clusters

[ppf2]

Elasticsearch version (bin/elasticsearch --version): 6.2.1

Steps to reproduce:

Not sure if it's the expected behavior or a documentation issue.

Create a ccs setup with a ccs cluster with a downstream cluster_one that has an index called test, and cluster_two without the same index name.

GET cluster_one:test,cluster_two:test/_search

Results in:

{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "index_uuid": "_na_",
        "resource.type": "index_or_alias",
        "resource.id": "test",
        "index": "test"
      }
    ],
    "type": "transport_exception",
    "reason": "unable to communicate with remote cluster [cluster_two]",
    "caused_by": {
      "type": "index_not_found_exception",
      "reason": "no such index",
      "index_uuid": "_na_",
      "resource.type": "index_or_alias",
      "resource.id": "test",
      "index": "test"
    }
  },
  "status": 500
}

Certainly, if I set "search.remote.cluster_two.skip_unavailable": true, it will work and skip over cluster_two without throwing the error.

Alternatively, without setting skip_unavailable, I can also simply add ignore_unavailable=true to the _search call to avoid this error.

In our current doc, we present skip_unavailable as a setting for dealing with disconnected clusters or when the search is against a downstream cluster(s) where no nodes belonging to them are available.

Is it the expected behavior for it to fail with a 500 error if the query is requesting data from an index from a set of downstream clusters where one of them may not have the index (but all the nodes in all downstream clusters are available)? If it is the expected behavior, it will be helpful to update our documentation to talk about skip_unavailable not only in the context of node availability/connection, but also for the no index use case.

[elasticmachine]

Pinging @elastic/es-search-aggs

[colings86]

@javanna could you take a look at this?

[javanna]

It's a bug that 500 is returned, but that has been fixed already in #30976 .

The fact that the whole request fails because one index is not present on one of the clusters can be worked around with skip_unavailable but as you guessed it is not really the purpose of skip_unavailable. You can more easily affect this behaviour using the usual indices options like ignore_unavailable and allow_no_indices, remote and local indices are treated the same by the search API. Default behaviour is strict though, meaning if one of the indices is not found, a hard error will be returned.