Reindex API reports no errors when it fails against a cluster with skip_unavailable=true #106331
Description
Elasticsearch Version
8.14 SNAPSHOT
Installed Plugins
No response
Java Version
bundled
OS Version
Mac
Problem Description
In the PR to change the default value of skip_unavailable from false to true (#105792), I found that the reindex API is affected by the skip_unavailable setting when doing a remote reindexing operation.
When skip_unavailable=false and a cross-cluster reindex targets an index that does not exist, then IndexNotFoundException is returned with an HTTP status error code. (expected behavior)
However, when skip_unavailable=true it does not return an error status code, and worse it does not report any error in the response object:
{
"took": 36,
"timed_out": false,
"total": 0,
"updated": 0,
"created": 0,
"deleted": 0,
"batches": 0,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0,
"failures": []
}My guess as to what is happening is that reindex is doing a cross-cluster search action and when skip_unavailable=true, search will not return an error code (such as 404). Instead it returns a 200 OK and you have to parse the _cluster/details metadata section of the response to see that the remote cluster is marked as "skipped" and parse the failures section to determine why.
Should logic to handle skip_unavailable=true be added to reindex to parse errors and report them in the failures section of the response, or throw an error (such as IndexNotFoundException for this case) as it does when skip_unavailable=false? Should the behavior of reindex depend on the skip_unavailable cluster setting?
Steps to Reproduce
Run CrossClusterReindexIT with skip_unavailable=true. Change this code to do so: https://github.com/elastic/elasticsearch/pull/105792/files#diff-4c824afa4dd48c2429ae2c9711019301367429a8159a1d40cda8a22f6ef8032dR44
Logs (if relevant)
No response