ES|QL: inconsistent "index not found exception" scenario for "limit 0" queries

[astefan]

Description

Atm, if an ES|QL query targets an index that exists and a concrete index that doesn't, ie from test, inexistent an index_not_found_exception is thrown and reported back to the user at search time (after mappings discovery, analysis, optimization, planning happens).

However, when the query changes to from test, inexistent | limit 0, the optimization rule SkipQueryOnLimitZero will shortcut the execution and the search step is not performed and the query succeeds returning 0 rows.

Aiming to have a consistent behavior is not trivial because the mappings discovery step (using one _field_caps API call) doesn't have the capability to report the missing index. Some research is needed to investigate how this step can be improved and if it's feasible to do so:

• by having a second _field_caps call that will use ignore_unavailable=false. This should be a very light weight call (no fields reported back).
• by maybe using the exists API, but this has some particularities of its own: currently ES|QL does not report "index not found errors" on wildcard index patterns (some_index_or_alias_name* for example) unless the entire list of patterns in the from command doesn't have a match.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[kanoshiou]

Hi, @astefan

I spent some time digging into the code and found another bug that might be related.

When an ES|QL query targets both an empty index that exists and a nonexistent index, the response will never throw an index_not_found_exception, regardless of whether a LIMIT clause is present or if the limit is set to 0 or 1. This occurs because the mappings returned by the _field_caps call in lenient mode is considered empty if the index responseMaps are empty (see this line).

Therefore, this issue might be caused by the lenient mode of _field_caps. However, lenient mode is recommended for use (see this line).

[astefan]

@kanoshiou thank you for looking into this one. What is the actual mapping of the existent index and the exact ES|QL query that reproduces the other bug you mentioned?

[kanoshiou]

@astefan When an index mapping is set up like this or simply by putting an empty body, the bug I mentioned can be reproduced:

PUT test1
{
  "mappings": {
  }
}

POST /_query
{
  "query": "from test1, inexist | limit 1"	// with or without `limit` clause
}

Interestingly, when an index is created with the following body, the bug cannot be reproduced:

PUT test2
{
  "mappings": {
    "properties": {
    }
  }
}

[astefan]

Thanks @kanoshiou. This is known bug - #111545 - which is not ES|QL actually, but the way _field_caps deals with variants of "empty" mappings (and you caught the "variants" correctly). The issue is linked to one in ES and is blocked by the latter.

[kanoshiou]

@astefan I stumbled upon the variant by coincidence just now 😂. It seems there are two bugs in the areas I mentioned.

Please take a look at the return of _query:

{
    "took": 3,
    "columns": [
        {
            "name": "<no-fields>",
            "type": "null"
        }
    ],
    "values": []
}

The ESQL is ignoring the inexist index and returns <no-fields>. Shouldn’t an index_not_found_exception be thrown here too?

I think this screenshot illustrates it more clearly.

The reason is that the responseMap for test1 is empty, which causes it to return no-fields prematurely.