[CCS] improve response when node is not allowed to open remote connections in CCS

[LucaWintergerst]

If a node has cluster.remote.connect: false configured, that node will behave in the following way:

Request: GET remote_cluster:*/_search
Response: {}

The node also does not log anything that would indicate that the setting is the cause of the empty response.
CCS should give additional output why the response is empty so this can be handled by the downstream applications. Otherwise this misconfiguration can be easy to miss and very hard to troubleshoot.

@dimitris-athanasiou and @droberts195 are happy to discuss and elaborate more if necessary
We caught this when we were troubleshooting why dedicated ML nodes were not working for a user

[elasticmachine]

Pinging @elastic/es-search

[javanna]

heya @LucaWintergerst I don't see an empty response being returned. I rather get this (tested on 6.6.0):

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

The reason for this is that, when connecting to remote clusters is disabled, the node behaves as if the seeds were not configured and treats remote_cluster:* like a wildcard expression referring to local indices, which gets resolved to no indices. This is because there can be local indices matching that expression (see #26247 where we disallowed/deprecated using : in an index name), so we can't really assume that users are referring to a remote index and throw an error (we will be able to make that assumption in 8.0 only).

That said there are a couple of ways to detect that something is going wrong:

• check _shards.total: 0 means the query did not run on any shards, which looks suspicious
• use allow_no_indices=false,ignore_unavailable=false to have a strict behaviour, that way you get an exception whenever an index is mentioned that does not exist and the search fails

@dimitris-athanasiou @droberts195 could you let me know if this work for you?

[dimitris-athanasiou]

@javanna While we could probably fix the ML side using your suggestions, I still think other users could be bitten by this. Would it not be possible to throw an error instead? It seems like we know we're reaching a remote cluster and we know the node is disabled from doing so, so why not error?

[javanna]

@dimitris-athanasiou I explained above why we cannot do it.

This is because there can be local indices matching that expression (see #26247 where we disallowed/deprecated using : in an index name), so we can't really assume that users are referring to a remote index and throw an error (we will be able to make that assumption in 8.0 only).

[dimitris-athanasiou]

Ah, apologies, I misread that part.

[javanna]

No worries, I will take care of updating this behaviour once we move master to 8.0, for now I will close this issue.

[javanna]

Reopening, we moved master to 8.0.0 and it's now possible to make this change. PR coming.